Guidelines for Informing Policy via Data

CHAPTER 3 - THE VALUE OF DATA AND INDICATORS AND THEIR ROLE IN POLICY MAKING (page 2)

3.2.2.2 Collecting new data

Individuals or organisations involved in a measurement exercise should aim to use quality found data. This has great advantages in terms of time and cost. Often, though, it is simply not possible to do so. The lack of available, relevant, and valid data will make it necessary to collect new data to meet the requirements of the measurement exercise to be carried out.

Since there are many data-collection methods to choose among, the ultimate selection will depend on the concept(s) to study, the research question(s), the intended coverage of the exercise, and the availability of some relevant quality data. No method is flawless; so when opting for a given method, consider its advantages and drawbacks, and remember that methods can be used alone or in concert with other methods.

The selection of a data-collection method follows consideration of several important aspects of the exercise:

• The target and focus of the study (general population or subgroups);

• The coverage of the exercise (entire national territory or specific areas only);

• The type of possible sampling (random, convenience, snowball);

• The type of data desired (objective versus subjective, qualitative versus quantitative);

• The mode of data collection (face-to-face, phone, mail-out/mail-back);

• The type of questions (open-ended, semi-structured, structured).

3.3 INDICATORS

3.3.1 Definition

H.O. Sano defines indicators as “...quantitative or qualitative statements that can be used to describe situations that exist and to measure changes or trends over a period of time. They are pieces of information that may provide insight into matters of larger significance, i.e., they may be seen as small windows that provide a glimpse of a bigger picture” [3].

Although indicators are used to measure given situations, they are not necessarily numeric. An example is the use of a “very well, well, badly, very badly, or do not know” scale (i.e., a Likert scale) as used by DIAL in Peru when enquiring about individuals’ perception of the way public administration operates, or when asking about people’s perception of the functioning of democracy.

Indicators are based on normative assumptions [4]. That is to say, every indicator needs to rely on the assumption that either more or less of whatever is being measured is a positive thing.

3.3.2 Building and Using Indicators

As determined through the work of H.O. Sano and that of the Vera Institute of Justice, indicators need to fulfil a certain number of criteria to be useful and useable. Indicators should be “valid (measure what they purport to measure), balanced (reducing ambiguity of measurement), sensitive (sensitive towards desired changes and towards specific groups), motivating (induce intended performance), practical (affordable, accurate and available), owned (legitimate in the eyes of those who are affected by them), and clear (target groups should be able to understand them)” [5].

Indicators are used to show the state or level of something. This implies that a crucial and basic step in building and using indicators is to have a clear definition and understanding of the concept to be measured. This is essential to reach relevant and useable measures.

Depending on the concept, right, or specific situation to be measured, information from one or more sources will be necessary. Indicators can therefore be “simple,” “composite” or “aggregate.” As defined by M. Sudders, a simple indicator is based on only one measure, whilst a composite indicator combines “different things into a single measure,” and an aggregate indicator combines “different measures of a similar thing into a single measure” [6]. The author also mentions that, although all indicators can be useful, the great advantage of aggregate indicators is that, in cases where the same concept is measured by different data sources, the coverage and reliability of the indicator is increased by combining such sources. However, it is essential that the different data included in the measurement be made available; if not, it might not be clear how the rating, or indicator, was reached. Lack of clarity tends to diminish the indicator’s significance and, as a consequence, its potential use.

Moreover, the author stresses the fact that there is not just one but several types of indicators that can serve many different purposes. For instance, indicators can be used for: “compliance assessment, diagnostic analysis, documentation, comparative and ranking assessment, check-point questions, and planning and performance” [7]. Experts and authors have developed different typologies of indicators reflecting this diversity of purpose.

Before focusing on the specificity of indicators in the field of democratic governance and human rights and on the variety of indicators that exist, we will first emphasise the importance of data and indicators in improving policy-making processes in general.

3. Sano, H.-O., Human Rights Indicators: Purpose and Validity, Paper for Turku/Abo Expert Meeting on Human Rights Indicators, 2005, p. 3.

4. Sudders, M., op. cit., p.13.

5. See Sano, H.-O., op. cit., p.4.

6. Sudders, M., op. cit., p.13.

7. See Sano, H.-O., op. cit., p. 1.