Guidelines for Informing Policy via Data

CHAPTER 3 - THE VALUE OF DATA AND INDICATORS AND THEIR ROLE IN POLICY MAKING (page 9)

3.6.2.2 Subjective and objective data

Just in the same way as quantitative and qualitative data can be seen as complementary tools, so can objective and subjective data. Objective data deal with observed/experienced facts or situations (such as levels of income and consumption, housing conditions, level of education, cases of corruption), whilst subjective data are linked to perceptions and assessments of the people being surveyed (such as degrees of satisfaction with living conditions, and opinions on how institutions operate and on the policies they implement). Subjective data is often considered “second best,” and are criticised as being less reliable than objective data. Those criticisms are worth taking into consideration.

The way in which a population perceives a given issue is extremely important in understanding local concerns, even if there is little “objective” justification for the perception. Perceptions can be early signs of significant events to come, such as conflict, condemnation, or the overthrow of a regime. They are indispensable when it comes to measuring democratic governance and human rights, as they give voice to local concerns and empower the populations in question.

Metagora projects illustrate the importance of considering objective and subjective data jointly. From May 2003 to the end of 2004, the perception among individuals in Peru (subjective indicator) was that corruption had worsened, although the incidence of small-scale corruption experienced by households (objective indicator) had not significantly changed. Through these two approaches, which produced different conclusions, it became clear that the assessments of the inhabitants themselves were based on a wider view of governance. The gap between the expectations of the population, generated by a proliferation of adverts for politicians advocating good governance, and the absence of reforms to translate those words into meaningful actions, had created a negative perception among the population on progress made in fighting corruption.

In South Africa, objective and subjective data were also collected, but the survey gave particular weight to the latter, focusing on individuals’ perceptions of agricultural reforms. An analysis of the results showed that the reform, as implemented, was not living up to popular expectations. This negative conclusion, partly based on subjective data, was at odds with the conclusion reached by a separate process that evaluated the objective impact of the reform, not whether the reform and its objectives responded to people’s demands.

3.6.2.3 Complementarity between bottom-up and top-down approaches

The collection of data and the building of indicators can be influenced by bottom-up or top-down approaches. It is useful to contrast collaborative approaches, characteristic of bottom-up processes, with top-down approaches, which apply global norms to the measurement of democratic governance and human rights without adapting to national or local contexts.

The obvious advantages of a global, top-down approach in selecting data and indicators is that it facilitates inter-country and large-scale international comparisons, thus providing international users, such as investors, donors and other bodies, with certain indicators on governance. (Those users do not usually require indicators that reflect context-specific features.) These top-down tools are relatively unsuited for effectively monitoring and evaluating national and local policies or strategies aimed at improving democratic governance and human rights. Often excessively aggregated and generalised, top-down indicators fail to take into account the specific contexts of individual countries, and thus the information available does not provide relevant data upon which targeted policies can be created and developed.

The Metagora project, which emphasises the need for a bottom-up approach, was not conceived as a competing alternative to top-down approaches, but as a necessary and useful complement to them. Indeed, both approaches have their advantages and disadvantages. To maximise efforts and initiatives to monitor democratic governance and human rights, the Metagora project has aimed to promote bottom-up approaches along with top-down approaches, with the belief that these approaches should enrich each other.

For instance, Metagora’s survey modules in West Africa and the Andes combine features characteristic of international initiatives with that of context-specific analysis. Some of the questions selected for the themed modules covering governance and democracy issues in Francophone countries of Africa were reproduced from other international initiatives. By doing so, the responses collected could be compared with responses to the same question obtained in other regions of the world.

3.6.2.4 Target populations: General or specific groups

The choice of units of analysis depends on the aim(s) of the study and on the nature of the information to be measured. Units of analysis can be individuals, households (or their heads), regions, countries, etc.; and the scope of the study might cover a general population or specific population groups, such as children, women, specific ethnic groups, AIDS victims, etc.

The scope and unit of analysis will be defined by the study and measurement exercise itself. When it was decided to conduct a survey into the rights of ethnic minorities in the Philippines, a specific questionnaire was produced and translated into several languages, taking into consideration the cultural diversity and the individual characteristics of the ethno-linguistic groups in question, and dealing with sensitive questions in the most appropriate manner. For a survey in South Africa, particular attention was paid to population groups likely to be affected by land reform. The decision made was to focus on the targeted populations’ views of land policies, given that the perception of gaps between their expectations and the policies implemented would provide useful information in assessing democratic governance and human rights issues.

In countries where no reliable quantitative information on democratic governance and human rights issues is available, it might be prudent to first define a few strategic problems and a series of indicators with which to monitor those problems over time. A complementary, in-depth approach, based on clearly identified themes, can later be introduced in order to explore the issues in question in greater detail.

The Metagora project used a variety of approaches. In Mexico, discussions were organised to define the population group to be surveyed. Given that the aim of the study was to promote overall accountability among public officials, it was decided that the survey should be conducted among the population of the Federal District as a whole, rather than among specific groups already known to be vulnerable to human rights violations. This approach does not preclude the subsequent study of specific sub-groups during analysis of the data.