Guidelines for Informing Policy via Data

CHAPTER 3 - THE VALUE OF DATA AND INDICATORS AND THEIR ROLE IN POLICY-MAKING

Indicators are an often-used but seldom understood tool. This chapter discusses what data are and how they are collected, what indicators are and how they are developed, and how the different methods for data collection, indicator-creation and statistical analysis can be used in the general context of the policy-making process, particularly in field of democratic governance and human rights. Emphasis is placed on the complementary roles of the policy-oriented professional and the statistician, and the difficulties inherent in data collection.

3.1 INTRODUCTION

The aim of this chapter is to describe how policy-making can benefit from the generation and use of data and indicators in their various forms. As stated by C. Scott, “Evidence-based policy-making in a democratic context means that, wherever possible, […] policy decisions should be reached after an open debate, which is informed by careful and rigorous analysis using sound and transparent data [1].” The potential impact of data and indicators on informing policy-making processes will only be as good as the quality of the information itself. It is therefore crucial to obtain and use the best possible data, and to analyse the potential difficulties and limits of different data-collection methods.

This chapter will focus on analysing what data are, what they consist of, and how they can be collected, and on what indicators are, how they are built, and how they can be useful. After providing a general approach to data, indicators, and policy-making, the chapter will focus on measuring and monitoring democratic governance and human rights.

3.2 DATA

3.2.1 Definition

"Data" can be defined as “factual information used as a basis for reasoning, discussion, calculation, or decisions [2].” The data collected can be quantitative or qualitative in nature. In broad terms, the main difference is that qualitative data use words to describe reality, whilst quantitative data use numbers and statistics. However, it is possible to modify and transfer qualitative data into numbers.

From these definitions, it follows that when data (pieces of information) are considered, they allow for informed dialogues or debates. A prerequisite is that the data be both relevant to the discussion and also solid, valid, and reliable (for further information on the validity and relevance of data for policy-making, please also refer to Chapter 13).

Therefore, before discussing the relevance of data in policy-making, it is important to understand where data come from, as well as how and by whom they have been/can be collected. A great variety of data-collection methods exist, and whether one or another method is used usually depends on factors such as time and financial constraints, and the existence and availability of adequate information.

3.2.2 General Data collection Methods

Data-collection methods have been classified in many different ways. Data can be quantitative or qualitative, subjective or objective, etc.; and data may be gathered in a variety of ways. The discussion below covers some of the most frequently used data-collection methods; it is by no means exhaustive. The advantages and limits of these different types of data will be addressed later in the chapter.

3.2.2.1 Pre-existing data

Whenever possible, it makes sense to use data of adequate quality that already exist, or “found data,” as the cost of collecting and analysing pre-existing data is usually far less than the cost of collecting data from scratch. Examples of pre-existing data include:

• Household survey or census data. Household survey data are almost always based on a random sample of a population or sub-population; in contrast, a census attempts to collect data from an entire population. The questionnaires are designed to capture the desired information. Both surveys and censuses can be focused on capturing factual data (e.g., household income), or perceptions/opinions (e.g., what the respondent believes is a reasonable wage in his/her community), or both. There are many different ways of carrying out a survey: face-to-face, on the telephone, by mail, email or the Internet, etc. Survey and census data are almost always quantitative, although a survey may contain a qualitative component.

• Administrative data. · In many countries, high-quality data have already been collected by the government in the pursuit of its administrative duties. For example, data are collected from citizens when they pay taxes, or when they register for social security or medical benefits. Though such data are not primarily collected to inform policies, they may be used as such—for example, medical data that include date of birth might be used to inform decisions regarding university scholarships designed to encourage geriatric specialists for an aging population.

• Expert or focus group interview data · can consist either of the transcript of an interview or of the summary of an interview. Interviews are usually aimed at reaching a specific section of the population, and can either be individual or organised in the form of focus groups. In addition, interviews can be more or less structured, depending on their specific objective and on the amount of information already in the hands of the people in charge of data collection. Such data are collected via very small population samples and are almost always qualitative.

• Events-based data consist in the recording of events. Such data include newspaper articles and other news sources, individual records collected by NGOs (testimonies, etc.), specialised private or public bodies, and information collected by independent researchers. Such data can be either quantitative or qualitative in nature. They might be similar to administrative data or provide less clear-cut information.

• Expert judgment. These data tend to be presented in the form of qualitative statements authored by the expert, and can be based on the legal situation of a country (i.e., data relating to the adoption of specific legislation and treaties, often used to measure statements of intent) and reports from civil society, the media or official sources.

• Desk studies. This type of study and data-collection method relies on pre-existing qualitative data and usually aims to find out what the general (i.e., legal, economic, political and social) situation of a country is like, based on existing material, such as domestic legislation and reports published by the media, government, civil society, and/or academia.

• Expert coding. · In some situations, existing qualitative data may be used more efficiently by being transformed into quantitative data. The process by which qualitative data (e.g., statements taken from the victims of human rights violations) can be used to create high-quality quantitative data is coding, using adequate classification and replicable judgment rules. Indeed, academics and experts around the world have developed a number of different indices and measures to assess and evaluate various practices, such as official practices affecting political freedom or a regime’s commitment to human rights.

3.2.2.2 Collecting new data

Individuals or organisations involved in a measurement exercise should aim to use quality found data. This has great advantages in terms of time and cost. Often, though, it is simply not possible to do so. The lack of available, relevant, and valid data will make it necessary to collect new data to meet the requirements of the measurement exercise to be carried out.

Since there are many data-collection methods to choose among, the ultimate selection will depend on the concept(s) to study, the research question(s), the intended coverage of the exercise, and the availability of some relevant quality data. No method is flawless; so when opting for a given method, consider its advantages and drawbacks, and remember that methods can be used alone or in concert with other methods.

The selection of a data-collection method follows consideration of several important aspects of the exercise:

• The target and focus of the study (general population or subgroups);

• The coverage of the exercise (entire national territory or specific areas only);

• The type of possible sampling (random, convenience, snowball);

• The type of data desired (objective versus subjective, qualitative versus quantitative);

• The mode of data collection (face-to-face, phone, mail-out/mail-back);

• The type of questions (open-ended, semi-structured, structured).

3.3 INDICATORS

3.3.1 Definition

H.O. Sano defines indicators as “...quantitative or qualitative statements that can be used to describe situations that exist and to measure changes or trends over a period of time. They are pieces of information that may provide insight into matters of larger significance, i.e., they may be seen as small windows that provide a glimpse of a bigger picture” [3].

Although indicators are used to measure given situations, they are not necessarily numeric. An example is the use of a “very well, well, badly, very badly, or do not know” scale (i.e., a Likert scale) as used by DIAL in Peru when enquiring about individuals’ perception of the way public administration operates, or when asking about people’s perception of the functioning of democracy.

Indicators are based on normative assumptions [4]. That is to say, every indicator needs to rely on the assumption that either more or less of whatever is being measured is a positive thing.

3.3.2 Building and Using Indicators

As determined through the work of H.O. Sano and that of the Vera Institute of Justice, indicators need to fulfil a certain number of criteria to be useful and useable. Indicators should be “valid (measure what they purport to measure), balanced (reducing ambiguity of measurement), sensitive (sensitive towards desired changes and towards specific groups), motivating (induce intended performance), practical (affordable, accurate and available), owned (legitimate in the eyes of those who are affected by them), and clear (target groups should be able to understand them)” [5].

Indicators are used to show the state or level of something. This implies that a crucial and basic step in building and using indicators is to have a clear definition and understanding of the concept to be measured. This is essential to reach relevant and useable measures.

Depending on the concept, right, or specific situation to be measured, information from one or more sources will be necessary. Indicators can therefore be “simple,” “composite” or “aggregate.” As defined by M. Sudders, a simple indicator is based on only one measure, whilst a composite indicator combines “different things into a single measure,” and an aggregate indicator combines “different measures of a similar thing into a single measure” [6]. The author also mentions that, although all indicators can be useful, the great advantage of aggregate indicators is that, in cases where the same concept is measured by different data sources, the coverage and reliability of the indicator is increased by combining such sources. However, it is essential that the different data included in the measurement be made available; if not, it might not be clear how the rating, or indicator, was reached. Lack of clarity tends to diminish the indicator’s significance and, as a consequence, its potential use.

Moreover, the author stresses the fact that there is not just one but several types of indicators that can serve many different purposes. For instance, indicators can be used for: “compliance assessment, diagnostic analysis, documentation, comparative and ranking assessment, check-point questions, and planning and performance” [7]. Experts and authors have developed different typologies of indicators reflecting this diversity of purpose.

Before focusing on the specificity of indicators in the field of democratic governance and human rights and on the variety of indicators that exist, we will first emphasise the importance of data and indicators in improving policy-making processes in general.

3.4 RATIONALE FOR USING DATA AND INDICATORS TO ENHANCE POLICY MAKING

3.4.1 Importance of Data and Indicators in Policy making in General

Evidence-based policy-making is essential for guaranteeing that policies are designed and actions taken to respond to the needs and demands of society, and that they are not based on assumptions and beliefs. As highlighted by C. Scott, evidence-based policy-making enhances the transparency and accountability of the policy process, making it more equitable and efficient [8]. Scott enumerates several alternatives to evidence-based policy-making, such as “policy processes dictated by power and influence of sectorial interests, corruption, political ideology, arbitrariness, and anecdotal information” [9], all of which can be detrimental if they do not promote or help establish policy-making procedures that are democratic, respectful of human rights, and accountable.

Data and indicators are necessary to the policy-making process. Based on C. Scott’s paper [10], there are five main ways in which data and indicators are useful to the policy process and allow for evidence-based policy-making. They can help: 1) identify relevant issues; 2) inform the design and choice of policies; 3) forecast the future (needs and problems); 4) monitor the implementation of policies; and 5) evaluate the impact of given policies. Clear parallels can be drawn between this list and the different stages of the policy process as presented in the previous chapter (see Chapter 2 and the stages model).

Policy-making’s dependence on valid and relevant data can be illustrated using both positive and negative examples. The use of data and indicators has proven useful in cases where it has allowed for specific problems or potential dangers to be identified and addressed in a timely manner. For instance, in 2000-2001, Uganda’s Demographic and Health Survey revealed stagnant infant mortality rates in a context of overall rapid economic growth and declining poverty. The government studied this question in depth, commissioning research on the causes of this phenomenon and, after a period of analysis, took the necessary measures to tackle the root causes identified [11]. On the other hand, there are examples of cases where, because data were not available or taken into account, no actions were taken to resolve or avoid a particular problem. For instance, as C.Scott documents, “In the second half of 2001, the Malawi office of Save the Children accumulated a growing body of evidence that serious food shortages were emerging in several regions of the country.” Despite active mobilisation and firm warnings by the organisation that the country was about to suffer a major food crisis, actors on the ground paid little attention and failed to prevent and limit the inevitable famine.

Whether data and indicators are actually used or not to fulfil the above-mentioned objectives depends on two main factors: the availability of data (this refers both to their quality and quantity) and their use by policy-makers (with the assumption that policy-makers want to base their policy actions and decisions on the data, and that the data actually reaches them). The author gives four possible scenarios [12]. The first scenario, the “vicious-circle countries,” is characterised by weak data systems that are not being used. In the second, the “data supply-constrained countries,” data, although weak, are increasingly being used. The third scenario, the “data demand-constrained countries,” is characterised by improving levels of data, but those data are not being used by policy-makers. The fourth scenario, the “virtuous-circle countries,” represents the best possible situation, one where the quality and quantity of data are improving and where they are increasingly being used. Depending on which scenario applies to a given country, priority may be on obtaining solid data or on getting governments and policy-makers to use the information collected. The use of data and indicators in informing policy is thus, to a great extent, context-specific.

3.4.2 The Use of Data and Indicators in Enhancing Democratic Governance and Human Rights-Related Policy Making

The philosophy behind democratic governance and human rights-measurement exercises – and the Metagora project – is that quantifying democratic governance and human rights phenomena allows policy-makers to better understand and address those phenomena. Current human rights-monitoring mechanisms are often incomplete and only reflect specific rights. Most often, there are no data available concerning both overall democratic governance and human rights situations, and the particular processes which led to these situations. What information is available is usually based on the reporting of individual cases to human rights organisations, or on judicial decisions. Thus the information does not apply to the entire population.

An argument for quantification is that individual cases, though irreducible in their importance, fail to provide evidence of policies or patterns of human rights abuse and violations, nor do they allow for human rights to be more respected., In the context of democratic governance and human rights monitoring, precise and quantified data can help establish the numbers, frequencies, distributions, magnitude, characteristics, and patterns of human rights violations.

The belief behind the Metagora approach is that human rights, being universal and indivisible, are obviously to be fought for whenever a violation is identified, since every case must be examined; but that only quantitative information can illustrate the magnitude and trends of human rights issues, be they civil, political, social, economic or cultural rights.

Quantification is thus essential for monitoring human rights and democratic governance, though it should not be misused to classify priorities according to the numbers obtained, that is, an erroneous hierarchy should not be established based on the idea that the higher the number, the more severe the problem. On the contrary, quantification should be used to identify each problem and allow for the elaboration and proposal of efficient measures to provide redress.

Quantifying the incidence of, say, human rights abuses serves as an efficacious means of informing the debate and contributing to the elaboration of more relevant policy-making. The core strategy of the Metagora pilot project is to help create monitoring tools by gathering evidence-based information on democratic governance and human rights issues of national importance [13]. As testimony to the power of such information, the Chairperson of the National Commission on Indigenous Peoples declared, in a preface to the final report of the Metagora activity in the Philippines: “May the findings of the [project] serve as a mirror of reflection among the other duty bearers, the institutions of government and the civil society, who have yet to awaken from their slumber to fully cater to the rights of Indigenous Peoples…”

Another example is found in the Mexican activity, carried out by Fundar. Prior to Fundar’s survey, there was no credible data available as to the frequency of abuses and ill-treatment in Mexico City, thus it was impossible to say whether the problem was severe, as claimed by human rights groups, or overstated, as some law enforcement officers averred. The findings of the survey put the debate on more solid ground. The survey provides information on the nature and quality of contacts between the population and law-enforcement officers, on the presence or absence of human rights violations, on the type of contact most likely to be associated with abuses and ill-treatment, the types of abuses and ill-treatment that are most common, and which law enforcement authorities are mainly responsible for what kind of abuse.

The impact of survey findings, and other kinds of data, on policy is crucial and involves dialogue with a number of relevant actors (i.e., members of government, the media, advisers and experts, civil society) to raise awareness on current democratic governance and human rights situations, and to document the public’s perceptions as to how well their expectations have been met by the policies implemented. Further information on this issue can be found in Chapter 13.

The DIAL project in Africa is also a good example of how precise and relevant information can help design properly targeted policies to address, for example, corruption. Indeed, surveys conducted in eight West African cities demonstrated that issues perceived as overwhelming problems are not necessarily as widespread as believed, even if very real. In the case of corruption, it appeared that, though regularly experienced by a significant proportion of the population, this phenomenon is not as prevalent for the general population as experts had thought. This understanding has led to better targeted corruption-related policies.

Despite these positive results, a number of human rights defenders have opposed the quantification of human rights violations, claiming it is unethical [14]. Quantifying human rights violations and commenting on their main features and evolution does not, in any way, question the common human rights approach. On the contrary, it reaffirms that any violation is unacceptable and wholly condemnable, and that full respect of the law and of individual rights is a basic obligation of the State. Quantifying situations helps provide human rights defenders, civil society and policy-makers with precise information about violations that can help them advocate for and frame policies and strategies aimed at reducing the gap between supposed respect for human rights and the reality on the ground.

The measurement and monitoring of democratic governance and human rights can encompass various aims and objectives. Just as H.O. Sano identifies different purposes for the use of indicators in democratic governance and human rights [15], T. Landman identifies four main functions associated with human rights measurement when considering political and civil rights. These are “(1) contextual description, monitoring and documentation of violations; (2) classification of different types of violations; (3) mapping and pattern recognition of violations over time and space; and (4) secondary analysis that provides explanations for violations and policy solutions for reducing them in the future [16].” The international human rights legal framework itself contains references to the need, indeed obligation, to monitor the realisation of human rights [17].

3.5 THE SPECIFICITY OF DATA AND INDICATORS FOR DEMOCRATIC GOVERNANCE AND HUMAN RIGHTS MEASUREMENT

3.5.1 Democratic Governance and Human Rights Data Sources

Many authors who have written about democratic governance and human rights monitoring have attempted to categorise the sources of data available for measuring a given State’s compliance with and promotion of human rights norms and principles, and its commitment to democratic governance. Although such categorisation varies from author to author, and although the primary focus can shift between human rights and governance [18], a number of key and recurrent elements can be identified. The sources presented below are not, in any way, exhaustive.

There are numerous potential sources of data for democratic governance and human rights exercises, many of which one might not necessarily think of at first. . Referring to the main data-collection methods presented above, we will now focus on highlighting ways in which they can be used:

• Household survey or census data. Household surveys and census data may be useful to democratic governance and human rights measurement exercises in two ways. As highlighted by R. Malhotra and N. Fasel, censuses allow for socio-economic data to be gathered, which can be extremely useful in providing information on the level of fulfillment of various human rights. “There is (…) an important body of work on criminal justice statistics, within the domain of socio-economic statistics that relates explicitly to the civil and political rights” [19]. . Censuses can also show the percentages of individuals entitled to vote, voter participation by age and gender, etc. [20]. Household surveys can reveal people’s perceptions of governmental bodies or of a specific policy, as well as their opinions on democratic governance and human rights.

  In addition, household surveys and censuses can be useful as an initial platform on which democratic governance and human rights measurement exercises can be developed. Based on the experience of the DIAL, regular household surveys or censuses can be used as the basis on which specific democratic governance and human rights modules can be added. This “piggy-backing” onto regularly implemented measurement instruments was successfully used by DIAL over the past few years to collect democratic governance and human rights information, especially poverty-related data, in several African and Andean countries.

• Administrative data can provide information on issues such as health, social security and welfare, and learning and educational services. Such data are collected together with information regarding the person’s origin, age, gender, etc. It is therefore possible, by combining these data, to measure the degree to which governments do or do not respect basic democratic governance and human rights principles. If unjustified differences appear regarding the access to such services, which should, according to fundamental human rights principles (e.g. non-discrimination) be accessible to all, then these administrative data point to inequalities and human rights violations.

• Expert or focus group interview data are also an extremely useful source of information for democratic governance and human rights measurement exercises. Whether they are individual or group interviews, data collected via this method allow for in-depth analyses of a large variety of democratic governance and human rights-related topics, and for understanding of the perceptions, needs and concerns of the population(s) under study. Interviews can either be carried out from scratch or previously carried out by other researchers, NGOs, experts, etc.

• Events-based data. In broad terms, this kind of data includes “testimonies of victims, witnesses, information provided by the media and reports of States, civil society organisations and other interest groups” [21], which are recorded in a standardised manner and compiled. In terms of democratic governance and human rights, events-based data are essential, since they provide data concerning the incidence of a series of elements (human rights violations) which, in turn, reveal the democratic governance and human rights situation in particular areas. This kind of data can be positive or negative. When a violation is recorded, it is negative data; but if what is recorded is the holding of an election or the fact that a victim of abuse was given reparation, then it should be considered as positive events-based data.

• Expert judgment · can be particularly useful. In the absence of other forms of data (i.e., household surveys, administrative data, or events-base data, etc.), expert judgment is an essential alternative source of information. Since this particular source of data is essentially qualitative, it is potentially very useful for measurement exercises in their preliminary and exploratory phases.

• Desk study. By providing important background information, desk studies can inform the preliminary and exploratory phases of various democratic governance and human rights exercises. Since they are, to a large extent, based on the analysis of domestic legislation, desk studies can provide information on the respect of “human rights, in principle,” as will be seen later in this chapter.

• Expert coding. Experts are sometimes asked to evaluate and score given democratic governance and human rights situations. Such data are useful for an overall impression of a given situation, and they make it possible to compare democratic governance and human rights records between countries or over time. An example of expert coding is the Corruption Perceptions Index (CPI), published annually by the NGO Transparency International (TI). However, the methods of obtaining, and using, such data have been criticised, as will be discussed later in this chapter.

  Another example of expert coding is that provided by the Sri Lankan experience. In a country where there are tremendous inconsistencies in facts and figures regarding ethnic and political violence and human rights violations, the sharing and pooling of human rights-violations data among NGOs were facilitated in order to support the development of a massive and objective statistical record of human rights violations. That record was then used to inject new, objective, and scientifically rigorous evidence into the peace process and transitional justice mechanisms. Expert coding therefore played an essential role in measuring democratic governance and human rights, and in truth and reconciliation work.

There are parallels between the use of these different sources of data and the specific objectives of measurement exercises. For example, desk studies relate particularly well to the measurement of “human rights, in principle,” whilst other sources refer more specifically to measuring “human rights, in practice” (see below for a more detailed explanation of the different purposes and objectives of measurement exercises).

3.5.2 Different Indicators to Fulfil Different Measurement Objectives

Different data-collection methods and sources are used depending on the objective of the measurement exercise, just as different types of indicators are used depending on the purposes of the exercise, the levels of measurement desired, and the rights that one intends to measure and assess.

3.5.2.1 Indicators are dependent on the purpose and level of measurement sought

Human rights can be measured at different levels. As mentioned by T. Landman, human rights can be measured “in principle (i.e., as they are laid out in national and international legal documents), in practice (i.e., as they are enjoyed by individuals and groups in nation states), and as outcomes of government policy that has a direct bearing on human rights protection” [22].

• Measuring rights in principle. References to human rights norms and principles can be found in international and national legislation and jurisprudence. Measuring such norms and principles shows States’ stated and formal levels of commitment to human rights. When this qualitative legal information is quantified, it enables us to compare these formal commitments with what is actually taking place.

• Measuring rights in practice. This measurement aims to capture the situation as it is actually experienced by groups and individuals. It is crucial for assessing, in a given country at a given time, the level of respect for human rights, for improving the human rights record, and for making governments accountable for their actions. Measurement of rights in practice is all the more interesting when compared to the measurement of rights in principle, that is, to a State’s official commitment to respect human rights. As mentioned by T. Landman, when such a comparison is made, three possible scenarios appear: the “negative gap” scenario, where actual practices fail to live up to the expectations generated by the State’s statement of intent with regard to human rights; the “zero gap” situation, where statements of intent and reality are in line; and the “positive gap,” where current practices are more favourable to human rights protection and promotion than the legal situation would suggest [23]. Such a measurement is important because experience has shown that regimes frequently make formal commitments to human rights that they only partially honour.

  Gathering comprehensive data on the respect, or violation, of human rights is an extremely difficult task. K.A. Bollen identifies six levels of information on human rights violations, which go from the ideal level, where all the characteristics of all the violations are recorded, to the most biased coverage of violations, where, for example, only specific violations are recorded, or only in specific areas, or only regarding particular groups of people [24].

• Measuring rights in government policies and outcomes. More indirect measures of human rights can also be obtained by using aggregate data relative to the outcomes of government actions and policies on human rights. Certain measures, though not providing direct information on the respect and protection of human rights, can be relevant to human rights monitoring if they show the degree to which governments support, or not, activities with a positive impact on human rights.

Though links between the nature and level of the measurement exercise and existing types of indicators are not automatic, parallels can sometimes be drawn. Before doing so, it is important to note the fact that the discussion on the different types of indicators available to monitor democratic governance and human rights has also been fed by another debate: that regarding the particular measuring requirements of the rights, or categories of rights, under scrutiny.

3.5.2.2 The nature of the rights to measure determine the data and indicators to use

It is common to separate human rights into different categories. Depending on the author and approach considered, human rights can be considered as negative or positive rights; first, second, or third generation rights; civil and political, or economic, social and cultural rights; or solidarity rights. There is also a debate in the field of human rights measurement and monitoring as to whether all such rights can be measured and, if so, whether they can be measured in the same way.

Civil and political rights are primarily designed to protect the individual against State inference. Such rights include the right to equality before the law, the right to a fair trial, the right to vote and political participation, etc. Historically, those rights have been seen as fundamental and are largely considered as negative rights due to the fact that they merely oblige States to refrain from engaging in certain activities, that is, these rights are fulfilled as long as they are not violated. On the other hand, social, economic and cultural rights are generally seen as less fundamental rights, and are considered as an obligation for the State to ensure some specified standard of living for all individuals, without discrimination. They are usually referred to as positive rights, i.e., they impose an obligation on States to take certain actions.

This distinction between different rights has been greatly challenged. A number of people active in the field of human rights claim that since all rights are indivisible and mutually reinforcing, rights should be considered as a whole body and not divided into types or categories. The dichotomy between negative and positive rights has also been questioned. If anything, many authors argue that it is possible to see all rights as having both positive and negative dimensions. As explained by T. Landman, positive dimensions include those actions taken by States to provide the resources and policies necessary for improving the protection of human rights, while negative dimensions are those actions deliberately taken by States that violate human rights, or those actions deliberately not taken by States to protect human rights. These positive and negative dimensions affect all rights.

Nonetheless, the debate between civil and political, and economic, social and cultural rights remains central to democratic governance and human rights measurement exercises, and has greatly influenced the methodological discussion behind them. Given the fundamental difference between the two categories of rights, i.e., immediate or progressive, different measurement methods and types of indicators may be best suited for each.

Interestingly, while some rights are over-represented in measurement exercises, others remain largely uncovered. Civil and political rights have tended to receive greater attention than socio-economic rights. Similarly, official data are usually focused on measuring “progressive” rights, whilst private data have tended to focus on measuring violations of “immediately applicable” rights [25]. The debate concerning the difference between civil and political, and economic, social and cultural rights, is based on the argument that second-generation rights are harder to measure, given that their progressive nature is linked to the government’s will and capacities, which are hard to assess. This reasoning is also much criticised; scholars such as A. Chapman [26] have called for more measurement exercises to be carried out in relation to economic, social and cultural rights.

3.5.3 A Presentation of Some Typologies of Democratic Governance and Human Rights Indicators

Three different typologies of indicators, which can be more or less directly associated with the main differences highlighted here, i.e., different levels and purposes of measurement, the rights to be studied, etc., are presented below.

A. First, as presented by H.O. Sano, indicators can be divided into two different kinds of instruments aimed to measure either performance or global compliance [27].

• Performance indicators are seen as referring to analyses “where assessment of policies, institutions, programmes, and business corporation performance is the central concern.”

• Global compliance indicators, on the other hand, refer to the analysis of “the global situation and changes therein,” and not to that of a specific policy or programme.

B. A second typology is suggested by M. Sudders in an attempt to go beyond the existing disagreement concerning terms and expressions and to provide a framework that reflects the most common terms. In doing so, the author divides indicators as follows [28]:

• Input/rights/commitment/de jure indicators, to measure the commitment made by countries regarding democratic governance and human rights. This could cover the inclusion, or exclusion, of principles on the respect and protection of human rights in national constitutions, the signing of international treaties, etc.

• Process/responsibility/de facto indicators, to measure whether parties are taking action to fulfil their responsibilities and commitments. This could include the existence, or non-existence, of institutions designed to ensure that the obligations are implemented.

• Output/outcome/enjoyment/performance/de facto indicators, to measure the proportion of people enjoying their rights versus those whose rights are being violated.

C. A third and alternative typology can be found in various World Bank papers that deal with the issue of building indicators to monitor poverty [29]. These authors distinguish two main types of indicators: intermediate and final indicators, each of which can be further divided. Thus, it appears that “indicators can be used to monitor progress at various stages: inputs into, outputs from policies and programmes (intermediate indicators); outcomes and impacts on households and individuals (final indicators)” [30].

• Intermediate indicators refer to indicators that measure factors that contribute to the process of achieving an outcome or impact (the focus is on policies and programmes). Such indicators can be further divided into input and output indicators, depending on the stage of the process. What is important is that inputs and outputs are not seen as objectives in themselves, but as elements that help achieve the chosen objectives, such as reducing poverty, eradicating torture, etc.
  • Input indicators measure the various financial and physical resources dedicated to achieve an objective.
  • Output indicators measure the goods and services produced by the inputs.

• Final indicators refer to indicators that measure, albeit indirectly, the effect of an intervention on an individual’s well-being and/or on the respect and protection of human rights. Such indicators can be separated into outcome and impact indicators.
  • Outcome indicators capture access to, use of, and satisfaction with public services (such as access to state-appointed lawyers, etc.). These indicators do not measure well-being and respect of human rights as such, but rather the key elements that contribute to protecting and promoting human rights.
  • Impact indicators measure key dimensions of well-being and respect for human rights such as freedom from hunger, freedom from torture, etc.

The authors provide an interesting illustration of the various types of indicators in the case of the goal of “improving child health.” Input indicators might consist of financial and physical resources, such as spending in primary health care. Output indicators could be the goods and services generated, such as the number of nurses or the availability of medicine. Outcome indicators might include access, use and satisfaction of users, and can be represented by the number of children vaccinated or the percentage within 5 km of a health centre. Impact indicators can consist of the effect on living standards and be covered by infant child mortality or the prevalence of a specific disease [31].

It is essential to monitor these different indicators, and not just one type, to understand the link between them and to identify where changes or additional efforts should be made in order to improve the human rights situation. Final indicators tend to change more slowly than intermediate ones, which should be included as much as possible to obtain swift information on a given process.

3.5.4 Further Reflections on Democratic Governance and Human Rights Indicators

The different typologies presented above focus on different aspects and priorities of measurement exercises; they do not contradict each other. In the case of the typologies presented by the World Bank and by H.O. Sano, some parallels can be drawn between performance assessments and intermediary indicators, and between global compliance and final indicators.

Indicators are not unique or irreplaceable. For instance, one right (for example, the right to a fair trial) may have several measures and indicators (such as the number of individuals who were given access to a lawyer, who were informed of the charges against them, or whose evidence and testimonies were accepted and included to the file, etc.). The use of one or another such measure will depend, among other things, on the availability of data. In cases where a specific measure is difficult to obtain, it might be useful to use proxy measures as substitute information, though this is not always acceptable [32].

In addition, though UN documents identify “possible objects of measurement (…) it does not define each right in sufficient detail to provide a basis for international measurement and comparison [33]”. The current lack of a common and universal understanding of concepts and rights creates problems for the use of indicators in the field of democratic governance and human rights; and this common understanding is only slowly making its way. Efforts to reach such an understanding and consensus include an OHCHR-based initiative. OHCHR, in consultation with a panel of experts, including committee members, special procedure mandate holders from the Human Rights Council, UN agencies, academics and non-governmental organisations, has developed a conceptual and methodological framework for identifying operationally feasible human rights indicators [34]. Building composite indicators (i.e. indicators that combine different elements into a single measure) can be difficult in the context of democratic governance and human rights issues. The formula to measure and regroup these different elements is neither easy to reach nor free from criticism. Indeed, two main methods are available: either attribute the same score to different elements or "weigh" them (i.e., give more or less importance to various elements). Neither method is perfect. When measuring respect for indigenous peoples’ rights, for example, shall all rights be given the same value or should some rights be considered as more important or vital than others? The use of composite indicators and complex indices has also been greatly questioned; they tend to be used for ranking purposes instead of focusing on assessing and evaluating the progress made by individual countries in specific areas.

Indicators are useful for measuring a given democratic governance and human rights situation and its evolution, and thus contributing to policy-making, but they are not to be used as the sole determinant in assessing given situations. Data and indicators are two of many tools available, including general knowledge of and judgment about the situation under study.

3.6 OVERCOMING THE INHERENT LIMITS OF DATA

Indicators can be based on different data. Be they found or new data, individuals using indicators need to know what the characteristic features of chosen data methods are in order to use the data well.

3.6.1 Limitations of Data-collection Methods

All data-collection methods have advantages and limitations. Though all the methods previously presented in this chapter will be reviewed briefly, special attention will be paid to the use of surveys, based on the experiences provided by three of the Metagora activities [35].

• Although the link with human rights is not always obvious, census data and administrative data can be useful for measuring democratic governance and human rights, with some limitations. This method provides no scope for in-depth analysis, thus no situation highlighted by census and administrative data that raises concern or curiosity can directly be explored further. This has serious restrictions on the establishment of strategies aimed to improve democratic governance and human rights and related policy-making.

• On the contrary, expert interviews are very useful, and relatively inexpensive, for both obtaining information and allowing for in-depth study and analysis of a given situation. A major limitation, however, is that results, however interesting, cannot be generalised, since they are neither representative nor comparable. In addition, good expert interviews rely heavily on the extensive and high-quality training of interviewers. As commented by M. Sudders [36], interviewers must be able to speak clearly, understand the data sought, and be acceptable to the target population. In turn, the interviewers must be supervised and cross-checked.

• Events-based data. One of the main limits associated with these type of data is that findings are usually based on individual records, such as a report of a human rights violation before a Human Rights Commission, and, since they are not based on representative samples, cannot be generalised to the entire population. Since the construction of these data depend on the filing of complaints/reports, the limitations are inherent in this method: the data are affected by selection bias and cannot correctly capture the full extent and magnitude of human rights violations or the characteristic features of victims. The data reported here are partial. In addition, such data tend towards either under-reporting or over-reporting of events.

• Expert judgment. A major restriction of these data is their subjective component. Indeed, when assessing a given democratic governance and human rights situation, experts do not usually rely purely on empirical quantitative data but rather on a mixture of factual (objective) and more subjective data. Moreover, expert judgments are often criticised for providing “summary to complex issues without providing systematic basis or examples justifying the assessments. Often they also lack transparency in the process of information selection, collection and evaluation” [37].

• Desk studies are extremely useful for collecting hard facts and existing data on the situation in a specific country. However, the main limit of this method is that it relies completely on the quality of the information already published, does not necessarily supply representative data, and rarely provides information on recent developments.

• Expert coding. · As seen above, expert coding is a particularly interesting method for comparing countries, areas or regions, though data are used to provide ratings, or codes, related to democratic governance and human rights issues on pre-defined scales, thus limiting the number of possible ratings. This phenomenon is commonly known as variance truncation. Furthermore, this method is highly dependent on the quality and neutrality of the primary sources on which it is based, and the codings must be cross-checked to ensure their validity.

• Surveys are very useful in producing data and statistics on the extent, size, and characteristics of observed phenomena; in determining the importance of given factors in influencing outcomes; and, since they use standardised procedures, in allowing the reproduction of results. However, there are limits to the effectiveness of surveys in measuring and monitoring democratic governance and human rights issues. Most of those limits affect the accuracy of the information reported. For example, translation or interpretation issues may arise, especially in the case of surveys with international or multi-national coverage. Indeed, surveys are prone to cultural biases, and when applied in more than one country, or where there are different languages used within a country, it is crucial to make sure that the concepts being measured, and the wording of the questions being asked, are accurately translated into the local language (for further information, please consult Chapter 7).

  Further limits include the fact that surveys, especiallyface-to-face surveys,can be extremely expensive. Surveys that involve writing or that are conducted via the telephone assume that the target population can read and write and/or that they have access to a phone. In many parts of the world, this is simply not true.

  In the case of surveys about democratic governance and human rights, there is also the challenge of conveying, asking about, and measuring concepts that might be new to the population. For example, a democratic governance and human rights survey might ask about abuses or violations that the population might not consider to be violations, due to the fact that they are commonplace (such as police requests for bribes in Mexico City).

  The characteristics of the sample behind any survey-based study are also important. As is the case for expert interviews and other data-collection methods, such as individual human rights violation reports collected by human rights commissions and civil society organisations, if survey samples are not designed to be representative of the population, the data produced by such surveys will not be representative, but will lead to over- or under-reporting of events and violations. Issues of selection bias are thus extremely important to consider. In general, there is usually a trade-off between smaller, less expensive samples and larger, more representative ones.

Most difficulties in data collection are not new or particular to human rights issues, but apply to statistics in general. Depending on the objective and constraints of a given measurement exercise, it will be more appropriate to resort to one method or another. The use of several complementary methods is also becoming more common. Based on the Metagora experience, we will now highlight some of these complementarities.

3.6.2 The Usefulness of Combining Methods and Approaches for Collecting Data and Building Indicators

3.6.2.1 Quantitative and qualitative data

As mentioned above, the data collected can be quantitative or qualitative in nature. In broad terms, qualitative data are comprised of words whilst quantitative data are comprised of numbers and statistics. Nevertheless, it is possible to modify and transform qualitative data into quantitative data. Each type has its advantages and limits [38].

Broadly speaking, qualitative data, such as data gathered from expert interviews and focus groups, are more flexible, allowing in-depth exploration of the meaning behind concepts and events, enabling a clear understanding of the situation, and exposing the motivations and patterns of association between factors. One main disadvantage, however is that sample sizes are often small and do not allow for representative data to be collected. The generalisation of findings can thus be a problem. In addition, expert interview methods rely heavily on respondents being reasonably articulate; and processes behind the analysis of the data are not always transparent or replicable.

Quantitative data, which are usually based on surveys, are extremely useful in producing statistics. Moreover, if random samples are used, estimates are precise and inferences can be made about the target population. Other advantages of quantitative data include: their capacity to measure the extent, size, and strength of observed phenomena; their usefulness in determining the importance of given factors in influencing the outcomes; and their use of standardised procedures, which allow for replication of results. However, quantitative data-collection methods can be costly, especially if the target population is hard to reach; sampling frames are not always available; the use of structured interviews can sometimes hinder a detailed exploration of the reasons behind specific actions or decisions; the use of standardised questionnaires means little flexibility; and key concepts must be clearly defined and translated into meaningful survey questions.

Rather than regard these methods as opposed and contradictory, it is constructive to see them as complementary. Indeed, although using a combination of these methods implies extended timeframes and increased costs, the results can be valuable, as can be illustrated by the Metagora project.

Gathering both quantitative and qualitative information helps refine methodologies and enriches analyses. The Metagora project has shown not only that assessment of democratic governance and human rights issues can build on the solidity of proper quantitative reporting, but also that the design and use of quantitative measurement methods and tools must be informed by accurate qualitative information, including in-depth documentation of the perceptions and experiences of target populations, and of the various assumptions and expectations of the different stakeholders.

Qualitative information is essential not only to ensure proper design of survey questionnaires, but also to focus statistical analysis on relevant issues and to provide appropriate contextual frameworks for an effective policy-oriented interpretation of quantitative data. In Metagora pilot activities, the design of survey questionnaires and databases was based on qualitative information gathered through in-depth narrative interviews with victims of rights infringements, focus group discussions with people belonging to the target populations, substantive reports of local experts, and large consultations with all relevant stakeholders.

As a result of this dual approach, the Metagora survey results provide qualitative, contextual information that ensure that policy-oriented reports are based on the broadest body of information possible. The pilot project of building quantitative data from records of narrative reports on human rights violations, and the attempt to merge quantitative and qualitative human rights data into a single database, are providing substantive lessons with obvious universal scope.

Metagora initiatives have thus demonstrated that quantitative and qualitative data can and should interrelate in order to properly and comprehensively inform democratic governance and human rights monitoring exercises. The combined use of quantitative and qualitative data contributed to providing richer analysis in various pilot experiences, which provide firm illustrations of the synergy between quantitative and qualitative information.

In the Philippines, the collection of these two types of data through a survey and focus group discussions (FGDs) has yielded rich results. The survey revealed a high level of awareness of indigenous populations’ rights to ancestral land and domains, while the FGDs elicited greater details, which made it possible to identify which types of rights were best understood by the local inhabitants.

In Mexico, the survey questionnaire was based on a series of in-depth interviews. The idea was to identify the key questions and problems to be measured, the way in which the relevant institutions operated, and the places or circumstances in which the population was exposed to the risk of abuse.

In Palestine, an examination into education rights was conducted with a database that contained both data from household surveys and qualitative assessments of the areas in which access to education needs improvement.

3.6.2.2 Subjective and objective data

Just in the same way as quantitative and qualitative data can be seen as complementary tools, so can objective and subjective data. Objective data deal with observed/experienced facts or situations (such as levels of income and consumption, housing conditions, level of education, cases of corruption), whilst subjective data are linked to perceptions and assessments of the people being surveyed (such as degrees of satisfaction with living conditions, and opinions on how institutions operate and on the policies they implement). Subjective data is often considered “second best,” and are criticised as being less reliable than objective data. Those criticisms are worth taking into consideration.

The way in which a population perceives a given issue is extremely important in understanding local concerns, even if there is little “objective” justification for the perception. Perceptions can be early signs of significant events to come, such as conflict, condemnation, or the overthrow of a regime. They are indispensable when it comes to measuring democratic governance and human rights, as they give voice to local concerns and empower the populations in question.

Metagora projects illustrate the importance of considering objective and subjective data jointly. From May 2003 to the end of 2004, the perception among individuals in Peru (subjective indicator) was that corruption had worsened, although the incidence of small-scale corruption experienced by households (objective indicator) had not significantly changed. Through these two approaches, which produced different conclusions, it became clear that the assessments of the inhabitants themselves were based on a wider view of governance. The gap between the expectations of the population, generated by a proliferation of adverts for politicians advocating good governance, and the absence of reforms to translate those words into meaningful actions, had created a negative perception among the population on progress made in fighting corruption.

In South Africa, objective and subjective data were also collected, but the survey gave particular weight to the latter, focusing on individuals’ perceptions of agricultural reforms. An analysis of the results showed that the reform, as implemented, was not living up to popular expectations. This negative conclusion, partly based on subjective data, was at odds with the conclusion reached by a separate process that evaluated the objective impact of the reform, not whether the reform and its objectives responded to people’s demands.

3.6.2.3 Complementarity between bottom-up and top-down approaches

The collection of data and the building of indicators can be influenced by bottom-up or top-down approaches. It is useful to contrast collaborative approaches, characteristic of bottom-up processes, with top-down approaches, which apply global norms to the measurement of democratic governance and human rights without adapting to national or local contexts.

The obvious advantages of a global, top-down approach in selecting data and indicators is that it facilitates inter-country and large-scale international comparisons, thus providing international users, such as investors, donors and other bodies, with certain indicators on governance. (Those users do not usually require indicators that reflect context-specific features.) These top-down tools are relatively unsuited for effectively monitoring and evaluating national and local policies or strategies aimed at improving democratic governance and human rights. Often excessively aggregated and generalised, top-down indicators fail to take into account the specific contexts of individual countries, and thus the information available does not provide relevant data upon which targeted policies can be created and developed.

The Metagora project, which emphasises the need for a bottom-up approach, was not conceived as a competing alternative to top-down approaches, but as a necessary and useful complement to them. Indeed, both approaches have their advantages and disadvantages. To maximise efforts and initiatives to monitor democratic governance and human rights, the Metagora project has aimed to promote bottom-up approaches along with top-down approaches, with the belief that these approaches should enrich each other.

For instance, Metagora’s survey modules in West Africa and the Andes combine features characteristic of international initiatives with that of context-specific analysis. Some of the questions selected for the themed modules covering governance and democracy issues in Francophone countries of Africa were reproduced from other international initiatives. By doing so, the responses collected could be compared with responses to the same question obtained in other regions of the world.

3.6.2.4 Target populations: General or specific groups

The choice of units of analysis depends on the aim(s) of the study and on the nature of the information to be measured. Units of analysis can be individuals, households (or their heads), regions, countries, etc.; and the scope of the study might cover a general population or specific population groups, such as children, women, specific ethnic groups, AIDS victims, etc.

The scope and unit of analysis will be defined by the study and measurement exercise itself. When it was decided to conduct a survey into the rights of ethnic minorities in the Philippines, a specific questionnaire was produced and translated into several languages, taking into consideration the cultural diversity and the individual characteristics of the ethno-linguistic groups in question, and dealing with sensitive questions in the most appropriate manner. For a survey in South Africa, particular attention was paid to population groups likely to be affected by land reform. The decision made was to focus on the targeted populations’ views of land policies, given that the perception of gaps between their expectations and the policies implemented would provide useful information in assessing democratic governance and human rights issues.

In countries where no reliable quantitative information on democratic governance and human rights issues is available, it might be prudent to first define a few strategic problems and a series of indicators with which to monitor those problems over time. A complementary, in-depth approach, based on clearly identified themes, can later be introduced in order to explore the issues in question in greater detail.

The Metagora project used a variety of approaches. In Mexico, discussions were organised to define the population group to be surveyed. Given that the aim of the study was to promote overall accountability among public officials, it was decided that the survey should be conducted among the population of the Federal District as a whole, rather than among specific groups already known to be vulnerable to human rights violations. This approach does not preclude the subsequent study of specific sub-groups during analysis of the data.

3.6.2.5 First-hand experience of the people or expert reviews

Numerous governance indicators available in international databases are constructed from the assessment of experts (see expert coding). Even though the number of surveys directly documenting the views and experiences of citizens is increasing rapidly, indicators based on that approach are still relatively scarce. The relevance of indicators based on the opinion of “experts” (e.g., consultants, researchers, development workers, decision-makers, senior civil servants, politicians, etc.) has to be assessed in relation to those based on surveys conducted among individuals and households.

As is the case with data-collection methodologies, there is a complementarity between the use of data collected by representative surveys and that derived from expert assessments. An example is the combination of governance and democracy modules attached to regular household surveys that were conducted between 2001 and 2003 in eight West African capitals with “mirror surveys.” The mirror surveys recorded the responses of some 250 specialists from the North and the South who were asked the same questions as those contained in the household surveys. A comparison between the data obtained from the two sources showed that experts consistently overestimated the level of corruption experienced by citizens and had a much more negative view of the situation than the general population. Given these results, and the strong discrepancies in the relative classification of countries, it can be assumed that experts do not have a good appreciation of the actual level of corruption. This finding, which is restricted to small-scale corruption and to the eight countries under consideration, prompts the question: what is actually being measured by the indicators produced by these surveys? It could be reasonably argued that it is precisely in countries where information is lacking that such indicators, based on perceptions, will be the furthest from reality. These results do not necessarily negate the relevance of these indicators, but they do illustrate that these indicators must be combined with another set of indicators that are based on objective measurements.

In the absence of household survey data, or in cases where such data are fragmentary, assessments provided by experts usefully complement existing information. The complementarity between these two types of information is illustrated in the construction of the database in Palestine.

3.7 CONCLUDING REMARKS

Data and indicators, in their many and diverse forms, and despite their respective limitations, can be extremely useful tools in measuring and monitoring democratic governance and human rights and in informing policy-making. There are a few points worth noting regarding their use in policy-making:

3.7.1 Who Collects the Data Matters

It is important to consider who is going to do collect the data and build the indicators. Indeed, the nature and reputation of the people and/or organisation(s) involved in the measurement exercise can have an enormous impact on the use, or lack thereof, of the data and indicators produced.

Who collects the information can have a double positive or negative impact: on people, when deciding to report events; and, as will be further seen in Chapter 13, on stakeholders and policy-makers, when considering whether or not to use the data produced for policy-making. Notions of trust, integrity and independence come into play and, if the