Guidelines for Informing Policy via Data

CHAPTER 3 - THE VALUE OF DATA AND INDICATORS AND THEIR ROLE IN POLICY MAKING (page 7)

3.6 OVERCOMING THE INHERENT LIMITS OF DATA

Indicators can be based on different data. Be they found or new data, individuals using indicators need to know what the characteristic features of chosen data methods are in order to use the data well.

3.6.1 Limitations of Data-collection Methods

All data-collection methods have advantages and limitations. Though all the methods previously presented in this chapter will be reviewed briefly, special attention will be paid to the use of surveys, based on the experiences provided by three of the Metagora activities [35].

• Although the link with human rights is not always obvious, census data and administrative data can be useful for measuring democratic governance and human rights, with some limitations. This method provides no scope for in-depth analysis, thus no situation highlighted by census and administrative data that raises concern or curiosity can directly be explored further. This has serious restrictions on the establishment of strategies aimed to improve democratic governance and human rights and related policy-making.

• On the contrary, expert interviews are very useful, and relatively inexpensive, for both obtaining information and allowing for in-depth study and analysis of a given situation. A major limitation, however, is that results, however interesting, cannot be generalised, since they are neither representative nor comparable. In addition, good expert interviews rely heavily on the extensive and high-quality training of interviewers. As commented by M. Sudders [36], interviewers must be able to speak clearly, understand the data sought, and be acceptable to the target population. In turn, the interviewers must be supervised and cross-checked.

• Events-based data. One of the main limits associated with these type of data is that findings are usually based on individual records, such as a report of a human rights violation before a Human Rights Commission, and, since they are not based on representative samples, cannot be generalised to the entire population. Since the construction of these data depend on the filing of complaints/reports, the limitations are inherent in this method: the data are affected by selection bias and cannot correctly capture the full extent and magnitude of human rights violations or the characteristic features of victims. The data reported here are partial. In addition, such data tend towards either under-reporting or over-reporting of events.

• Expert judgment. A major restriction of these data is their subjective component. Indeed, when assessing a given democratic governance and human rights situation, experts do not usually rely purely on empirical quantitative data but rather on a mixture of factual (objective) and more subjective data. Moreover, expert judgments are often criticised for providing “summary to complex issues without providing systematic basis or examples justifying the assessments. Often they also lack transparency in the process of information selection, collection and evaluation” [37].

• Desk studies are extremely useful for collecting hard facts and existing data on the situation in a specific country. However, the main limit of this method is that it relies completely on the quality of the information already published, does not necessarily supply representative data, and rarely provides information on recent developments.

• Expert coding. · As seen above, expert coding is a particularly interesting method for comparing countries, areas or regions, though data are used to provide ratings, or codes, related to democratic governance and human rights issues on pre-defined scales, thus limiting the number of possible ratings. This phenomenon is commonly known as variance truncation. Furthermore, this method is highly dependent on the quality and neutrality of the primary sources on which it is based, and the codings must be cross-checked to ensure their validity.

• Surveys are very useful in producing data and statistics on the extent, size, and characteristics of observed phenomena; in determining the importance of given factors in influencing outcomes; and, since they use standardised procedures, in allowing the reproduction of results. However, there are limits to the effectiveness of surveys in measuring and monitoring democratic governance and human rights issues. Most of those limits affect the accuracy of the information reported. For example, translation or interpretation issues may arise, especially in the case of surveys with international or multi-national coverage. Indeed, surveys are prone to cultural biases, and when applied in more than one country, or where there are different languages used within a country, it is crucial to make sure that the concepts being measured, and the wording of the questions being asked, are accurately translated into the local language (for further information, please consult Chapter 7).

  Further limits include the fact that surveys, especiallyface-to-face surveys,can be extremely expensive. Surveys that involve writing or that are conducted via the telephone assume that the target population can read and write and/or that they have access to a phone. In many parts of the world, this is simply not true.

  In the case of surveys about democratic governance and human rights, there is also the challenge of conveying, asking about, and measuring concepts that might be new to the population. For example, a democratic governance and human rights survey might ask about abuses or violations that the population might not consider to be violations, due to the fact that they are commonplace (such as police requests for bribes in Mexico City).

  The characteristics of the sample behind any survey-based study are also important. As is the case for expert interviews and other data-collection methods, such as individual human rights violation reports collected by human rights commissions and civil society organisations, if survey samples are not designed to be representative of the population, the data produced by such surveys will not be representative, but will lead to over- or under-reporting of events and violations. Issues of selection bias are thus extremely important to consider. In general, there is usually a trade-off between smaller, less expensive samples and larger, more representative ones.

Most difficulties in data collection are not new or particular to human rights issues, but apply to statistics in general. Depending on the objective and constraints of a given measurement exercise, it will be more appropriate to resort to one method or another. The use of several complementary methods is also becoming more common. Based on the Metagora experience, we will now highlight some of these complementarities.

35. Three of the seven activities of the Metagora I pilot project involved pilot surveys: the activity on irregularities, abuses of power, and acts of ill-treatment in Mexico, on the rights of indigenous peoples in the Philippines, and on land reform in South Africa.

36. Sudders, M., op.cit., p. 6.

37. Malhotra, R. and Fasel, N., op. cit., p. 17.