Guidelines for Informing Policy via Data

CHAPTER 1: INTRODUCTION (page 1)

Why does a policy-oriented professional need to think about data collection and analysis as a tool for policy creation? What does a survey-oriented professional or statistician need to understand about the policy-creation process in order to effectively collect or analyse data for the policy-maker's purposes? And how do both types of professionals ensure that accurate, high-quality data is interpreted well and used to inform effective policy? This chapter will introduce the concepts behind evidence-based policy creation and evaluation.

1.1 THE CONTEXT

The development of sound policy, particularly policy related to democratic governance and human rights, is crucial to the effective operation of governmental bodies. Given that it is the ethical and moral responsibility of governments to operate to serve their citizens, policy should, in part, work to remedy problems and issues related to those citizens' lives. In spite of rapid technological, scientific, and, arguably, governance development over the past century, a host of serious and overwhelming issues still affect all States: poverty, discrimination, mass human rights violations including genocide, a lack of universal primary education, a lack of basic health care, and global warming, to name just a few. The only method available to combat those universal issues is through sound policy enacted and monitored at the governmental, intra-governmental, and international level; only through combined effort can we hope to reduce and eliminate such problems and the suffering that they cause.

How is effective policy created? The Metagora community believes that effective policy is evidence-based policy - that is, policy informed by and based on valid empirical data that address the needs and preoccupations of both the population affected by the policy and also relevant stakeholders. We have therefore created these training materials as an introduction to the world of data, for policy-makers at all levels of government and in non-governmental organisations, and to the world of policy, for data-oriented professionals such as survey methodologists and statisticians.

Very often, policy-makers and statisticians do not see the need to communicate directly with each other. Policy-makers either do not see the relevance of empirical data for their work, preferring to rely on expert opinion, their personal or anecdotal experiences, or communication received from their constituency, or, since they are not experts on data-collection and analysis themselves, prefer to consider data only as it is presented in a final report. Conversely, data-oriented professionals may feel that as long as they are given a particular set of research questions, their role ends at the generation of a report responding to those questions, and understanding the policy process is not a requirement. Within the Metagora community, our experience has been that it is important for each type of professional, the policy-oriented and the data-oriented, to understand each other's field in order to create the most effective policy process. We therefore discuss below the advantages for each type of professional in learning the work of the other.

1.1.1 Why does a policy-oriented professional need to think about data-collection and analysis as a tool for policy creation?

Data are not a uniform product, collected via identical methods in different contexts. Rather, the practice of data-collection, and the subsequent quality of the data produced, varies significantly. Even when excellent methodology is used in the collection of data, different techniques of data-collection produce statistics and indicators that are subject to different interpretations.

An example will help to illustrate this concept. Policy-makers are often interested in the causal relationship between two variables - in order words, whether one phenomenon causes another. A classic historical example is the question of whether cigarette smoking causes cancer. Determining causality, however, is very difficult, if not impossible, unless data are collected via a randomised experiment.

In the most basic type of randomised experiment, a set of people is randomly split into two groups. One group receives the treatment, such as a pack of cigarettes to smoke each day, and the other group receives a placebo or no treatment. At the end of the experiment, the two groups are studied to see if the second variable of interest, such as cancer, is found in different rates in each of the two groups. If a statistically significant difference is found between the two groups, then a causal relationship has been established.

In most cases, the data available to policy-makers is non-experimental; no randomization has occurred, for example, to determine who will smoke and who will not. Very often such an experiment would be considered unethical and is simply not done. The issue is then one of confounding. In the historical example of cigarettes and cancer, the argument was that even if smokers as a group have higher cancer rates than non-smokers, there could be some other effect - a confounding variable - that caused both the higher rate of cancer and the tendency to smoke.

The policy-maker could not be sure, then, that policies intended to reduce the rate of smoking would indeed reduce the rate of cancer as well. The policy-maker would have to decide whether to implement a policy based on the evidence at hand about cigarette smoking and cancer, and implement that policy in a way that allowed the results of the policy to be monitored and evaluated. In the case of policies related to democratic governance and human rights, methods for monitoring and evaluating policies are discussed in Chapter 13 of this manual.

Data derived from sources other than a randomised experiment can be quite useful for informing policy creation, but not as direct measurements of causal relationships. One of the most basic distinctions between different types of non-experimental data is randomly collected data versus non-randomly collected data. When data are gathered via a random sample of a population, the resulting statistics represent the entire population, albeit with a margin of error. Clouding this distinction, however, is the case where data collected via a random sample are used to infer characteristics about a bigger population than the one from which the data were collected.

For example, if the sample frame for a survey is comprised of names of people who live in households in the federal district, then the homeless population has not been sampled and is not included in the resulting statistics. As a result, measurements of particular issues, such as poverty (and most certainly homelessness), will be biased. In other words, when interpreting the statistics created from these data, the policy-maker must be aware of the exclusion of the homeless from the sample and the potential impact on the data due to that exclusion.

1.1.2 What does a survey-oriented professional or statistician need to understand about the policy-creation process in order to effectively collect or analyse data for the policy-maker's purposes?

The policy process is complex and nuanced. If a data-oriented professional is interested in having an impact on the policy process, s/he must be aware of the role that politics can play in decision-making, and the need for transparent and well-explained methods. Situations such as upcoming elections, world events, and the relative bargaining positions of various politicians can all affect how data and/or statistics will be received. In such a situation, the role of the data-oriented professional may extend past simply collecting and/or analysing data of interest. S/he may be required to defend methods from political attacks, and even fight for the release of information that a superior within his/her organisation is withholding for political reasons.

As such, the most important reason that a data-oriented professional needs to be aware of the policy process is to maintain his/her own professional integrity and code of ethics. Understanding the potential barriers to collecting and releasing data that have policy or political impact, the political sensitivities that may influence general acceptance of the results of a data-collection process, and the potential misuses of data that are being collected all fall under the ethical responsibility of the data-oriented professional.

1.1.3 How do both types of professionals ensure that accurate, high-quality data is interpreted well and used to inform effective policy?

Data are analysed and interpreted correctly when:

• The method of data-collection is transparent and well-documented;
• The models and assumptions underlying the analysis of the data are clearly articulated;
• The professionals performing the analysis and interpretation of the data are aware of their own biases and analyse the data in as impartial a method as possible; and
• The public discussion about the data is focused on the best policy response to the results given by the data, rather than a discussion of the validity of the data and their subsequent interpretation.