Guidelines for Informing Policy via Data

CHAPTER 3 - THE VALUE OF DATA AND INDICATORS AND THEIR ROLE IN POLICY MAKING (page 1)

Indicators are an often-used but seldom understood tool. This chapter discusses what data are and how they are collected, what indicators are and how they are developed, and how the different methods for data collection, indicator-creation and statistical analysis can be used in the general context of the policy-making process, particularly in field of democratic governance and human rights. Emphasis is placed on the complementary roles of the policy-oriented professional and the statistician, and the difficulties inherent in data collection.

3.1 INTRODUCTION

The aim of this chapter is to describe how policy-making can benefit from the generation and use of data and indicators in their various forms. As stated by C. Scott, “Evidence-based policy-making in a democratic context means that, wherever possible, […] policy decisions should be reached after an open debate, which is informed by careful and rigorous analysis using sound and transparent data [1].” The potential impact of data and indicators on informing policy-making processes will only be as good as the quality of the information itself. It is therefore crucial to obtain and use the best possible data, and to analyse the potential difficulties and limits of different data-collection methods.

This chapter will focus on analysing what data are, what they consist of, and how they can be collected, and on what indicators are, how they are built, and how they can be useful. After providing a general approach to data, indicators, and policy-making, the chapter will focus on measuring and monitoring democratic governance and human rights.

3.2 DATA

3.2.1 Definition

"Data" can be defined as “factual information used as a basis for reasoning, discussion, calculation, or decisions [2].” The data collected can be quantitative or qualitative in nature. In broad terms, the main difference is that qualitative data use words to describe reality, whilst quantitative data use numbers and statistics. However, it is possible to modify and transfer qualitative data into numbers.

From these definitions, it follows that when data (pieces of information) are considered, they allow for informed dialogues or debates. A prerequisite is that the data be both relevant to the discussion and also solid, valid, and reliable (for further information on the validity and relevance of data for policy-making, please also refer to Chapter 13).

Therefore, before discussing the relevance of data in policy-making, it is important to understand where data come from, as well as how and by whom they have been/can be collected. A great variety of data-collection methods exist, and whether one or another method is used usually depends on factors such as time and financial constraints, and the existence and availability of adequate information.

3.2.2 General Data collection Methods

Data-collection methods have been classified in many different ways. Data can be quantitative or qualitative, subjective or objective, etc.; and data may be gathered in a variety of ways. The discussion below covers some of the most frequently used data-collection methods; it is by no means exhaustive. The advantages and limits of these different types of data will be addressed later in the chapter.

3.2.2.1 Pre-existing data

Whenever possible, it makes sense to use data of adequate quality that already exist, or “found data,” as the cost of collecting and analysing pre-existing data is usually far less than the cost of collecting data from scratch. Examples of pre-existing data include:

• Household survey or census data. Household survey data are almost always based on a random sample of a population or sub-population; in contrast, a census attempts to collect data from an entire population. The questionnaires are designed to capture the desired information. Both surveys and censuses can be focused on capturing factual data (e.g., household income), or perceptions/opinions (e.g., what the respondent believes is a reasonable wage in his/her community), or both. There are many different ways of carrying out a survey: face-to-face, on the telephone, by mail, email or the Internet, etc. Survey and census data are almost always quantitative, although a survey may contain a qualitative component.

• Administrative data. · In many countries, high-quality data have already been collected by the government in the pursuit of its administrative duties. For example, data are collected from citizens when they pay taxes, or when they register for social security or medical benefits. Though such data are not primarily collected to inform policies, they may be used as such—for example, medical data that include date of birth might be used to inform decisions regarding university scholarships designed to encourage geriatric specialists for an aging population.

• Expert or focus group interview data · can consist either of the transcript of an interview or of the summary of an interview. Interviews are usually aimed at reaching a specific section of the population, and can either be individual or organised in the form of focus groups. In addition, interviews can be more or less structured, depending on their specific objective and on the amount of information already in the hands of the people in charge of data collection. Such data are collected via very small population samples and are almost always qualitative.

• Events-based data consist in the recording of events. Such data include newspaper articles and other news sources, individual records collected by NGOs (testimonies, etc.), specialised private or public bodies, and information collected by independent researchers. Such data can be either quantitative or qualitative in nature. They might be similar to administrative data or provide less clear-cut information.

• Expert judgment. These data tend to be presented in the form of qualitative statements authored by the expert, and can be based on the legal situation of a country (i.e., data relating to the adoption of specific legislation and treaties, often used to measure statements of intent) and reports from civil society, the media or official sources.

• Desk studies. This type of study and data-collection method relies on pre-existing qualitative data and usually aims to find out what the general (i.e., legal, economic, political and social) situation of a country is like, based on existing material, such as domestic legislation and reports published by the media, government, civil society, and/or academia.

• Expert coding. · In some situations, existing qualitative data may be used more efficiently by being transformed into quantitative data. The process by which qualitative data (e.g., statements taken from the victims of human rights violations) can be used to create high-quality quantitative data is coding, using adequate classification and replicable judgment rules. Indeed, academics and experts around the world have developed a number of different indices and measures to assess and evaluate various practices, such as official practices affecting political freedom or a regime’s commitment to human rights.

1. Scott, C., Measuring Up to the Measurement Problem: The Role of Statistics in Evidence-Based Policy-Making, London School of Economics, London, 2005, p. 5.

2. Definition based on the online edition of the Merriam Webster dictionary and modified by the author (see http://www.m-w.com/).