Guidelines for Informing Policy via Data

CHAPTER 3 - THE VALUE OF DATA AND INDICATORS AND THEIR ROLE IN POLICY MAKING (page 10)

3.6.2.5 First-hand experience of the people or expert reviews

Numerous governance indicators available in international databases are constructed from the assessment of experts (see expert coding). Even though the number of surveys directly documenting the views and experiences of citizens is increasing rapidly, indicators based on that approach are still relatively scarce. The relevance of indicators based on the opinion of “experts” (e.g., consultants, researchers, development workers, decision-makers, senior civil servants, politicians, etc.) has to be assessed in relation to those based on surveys conducted among individuals and households.

As is the case with data-collection methodologies, there is a complementarity between the use of data collected by representative surveys and that derived from expert assessments. An example is the combination of governance and democracy modules attached to regular household surveys that were conducted between 2001 and 2003 in eight West African capitals with “mirror surveys.” The mirror surveys recorded the responses of some 250 specialists from the North and the South who were asked the same questions as those contained in the household surveys. A comparison between the data obtained from the two sources showed that experts consistently overestimated the level of corruption experienced by citizens and had a much more negative view of the situation than the general population. Given these results, and the strong discrepancies in the relative classification of countries, it can be assumed that experts do not have a good appreciation of the actual level of corruption. This finding, which is restricted to small-scale corruption and to the eight countries under consideration, prompts the question: what is actually being measured by the indicators produced by these surveys? It could be reasonably argued that it is precisely in countries where information is lacking that such indicators, based on perceptions, will be the furthest from reality. These results do not necessarily negate the relevance of these indicators, but they do illustrate that these indicators must be combined with another set of indicators that are based on objective measurements.

In the absence of household survey data, or in cases where such data are fragmentary, assessments provided by experts usefully complement existing information. The complementarity between these two types of information is illustrated in the construction of the database in Palestine.

3.7 CONCLUDING REMARKS

Data and indicators, in their many and diverse forms, and despite their respective limitations, can be extremely useful tools in measuring and monitoring democratic governance and human rights and in informing policy-making. There are a few points worth noting regarding their use in policy-making:

3.7.1 Who Collects the Data Matters

It is important to consider who is going to do collect the data and build the indicators. Indeed, the nature and reputation of the people and/or organisation(s) involved in the measurement exercise can have an enormous impact on the use, or lack thereof, of the data and indicators produced.

Who collects the information can have a double positive or negative impact: on people, when deciding to report events; and, as will be further seen in Chapter 13, on stakeholders and policy-makers, when considering whether or not to use the data produced for policy-making. Notions of trust, integrity and independence come into play and, if they are not handled properly, can lead to situations of under- or over-reporting.

Depending on the nature of the organisation, some individuals may either not respond to the questions at all or respond to them, but untruthfully because of fear or intimidation. This is particularly true with sensitive questions relating to democratic governance and human rights.

The subsequent use of the data and indicators by policy-makers depends on whether the data and indicators are considered to be the result of objective, impartial, and professional work. Thus, the overall reputation of the organisation(s) involved, along with the quality of the information, are extremely important.

National statistical offices (NSOs), non-governmental organisations (NGOs), Commissions on Human Rights (CHRs), governmental bodies and agencies, and academics may be involved at different stages and in different degrees during the exercise. A multi-disciplinary approach should be used whenever possible in order to enhance the comprehensiveness, usefulness and reliability of the exercises.

3.7.2 Disaggregation and Dissemination are Key Elements

Proper and sufficient disaggregation of data is essential; if the data are not sufficiently disaggregated, they may hide important differences related to regions, gender, age, etc. Aggregate data and indicators are useful to provide overall pictures and comparisons between countries, and within countries over time; however, they fail to reflect other significant differences. Information can be disaggregated along various lines, such as gender, age group, levels of income, regional locations, social groups, etc. Indicators that can be disaggregated along these lines help reveal differences within a given situation and are crucial for formulating pertinent and inequality-reducing policies and strategies.

In South Africa, data collection was based on the ethnic and social origin of the respondents, providing for specific analyses to be conducted with respect to each group identified. In Mexico, information was obtained that could later be analysed focusing on gender, specific age groups, etc. The possibility for data to be disaggregated along various dimensions depends on several factors. In the case of surveys, the size and design of the sample are crucial in defining whether the data can be disaggregated or not, and if so along which lines. In the Mexican activity, for example, one of the limits of the sample is that it did not allow for data to be disaggregated at a more precise level than the Federal District, thus making it impossible to analyse data at the borough level.

It is also extremely important to disclose and disseminate data. If this fails to happen, debates will fail to include relevant and updated information. As will be seen in Chapter 13, dissemination strategies are essential in promoting the use of data and indicators by policy-makers and relevant stakeholders.