Guidelines for Informing Policy via Data

CHAPTER 1: INTRODUCTION (page 2)

1.2 THE REMAINDER OF THESE GUIDELINES

The remaining thirteen chapters of these guidelines will encompass the following topics.

• Chapter 2: The policy process: Creating policy within governments and other types of organisations

  This chapter will review the policy process, exploring two different models: the stages model and the streams model.

• Chapter 3: The value of data and indicators

  Indicators are an often-used but seldom understood tool. This chapter discusses what data are and how they are collected, what indicators are and how they are developed, and how the different methods for data-collection, indicator-creation and statistical analysis can be used in the general context of the policy process, particularly in relation to policy-making in the field of democratic governance and human rights. Emphasis is placed on the complementary roles of the policy-oriented professional and the statistician, and the types of difficulties inherent in data-collection that must be considered.

• Chapter 4: Developing indicators and other statistics from pre-existing data

  Data are already out there -- on the United Nations web site, in the national statistical agencies, and housed within NGOs. This chapter discusses the advantages and dangers of using pre-existing data to develop indicators and other statistics.

• Chapter 5: Collecting data: Getting started

  If no pre-existing data exists, then data must be collected. In this case, several questions arise. What are the topics to be covered during the data collection? What research goals are part of the project? Are you measuring subjective or objective phenomena? Do you need quantitative or more qualitative data? Will you use a random sample survey or a more purposive method of data collection, such as focus group discussions? Are you best served by a survey instrument that contains open-ended questions or a highly structured questionnaire? Is there an existing data collection process to which you can add a module? What are the ethical implications and guidelines of your data-collection process? What confidentiality guarantees can you make? This chapter will explore all of these themes.

• Chapter 6: Collecting data: Finding the people to interview

  This chapter contains a discussion of sampling, highlighting the reasons for using a random sample or more purposive sample, determining a target population for data collection, finding a sampling frame, types of random samples, and other types of samples. Some of the pitfalls encountered in the sampling process will also be discussed, including incomplete sampling frames, under-coverage, biased samples, household-based populations, homeless and institutionalised populations, and the potential for non-response and incomplete responses. The errors associated with different sampling procedures and the errors introduced through non-response will also be explored.

• Chapter 7: Collecting data: Creating the statement/questionnaire

  Creating an instrument for the collection of data is a complex task within a single culture, and a phenomenal task when the instrument in question is to be used in multiple languages or cultures. This chapter reviews the questionnaire design process, with an emphasis on multi-language and multicultural surveys. After reading this section, the policy-oriented professional will gain a better understanding of the necessity for thorough pre-testing of questionnaires, and the survey-oriented professional will understand the basic steps required for a high-quality questionnaire-development process.

• Chapter 8: Collecting data: Logistics

  Once the data-collection instrument is designed and the people to be interviewed are picked, the real work of training interviewers and collecting the data begins. This chapter discusses all aspects of the logistics of data collection, including supervising field workers, maintaining high quality, budgeting, and the use of incentives.

• Chapter 9: Storing and processing data

  This chapter looks at what happens after the data are collected. Whether the data are qualitative or quantitative, they must be stored in a manner that facilitates analysis. In almost all modern data-collection processes, a computer is used to store the data. Quantitative data must be entered into a database, which involves a separate staff and the need to understand the underlying data structure. Qualitative data may be stored as-is or "coded" via a controlled vocabulary in order to extract quantitative elements. All of these approaches will be discussed.

• Chapter 10: Analysing data

  Whether data are qualitative or quantitative, they will need to be analysed before they will become useful for policy creation and monitoring. This chapter will discuss good practices of qualitative analysis, types of quantitative analysis, principles for the creation of statistics and statistical graphics, and how a combined qualitative/quantitative analysis can be approached.

• Chapter 11: Data presentation and the ethical use of data

  Presenting data well is not just a good idea, it is a matter of ethics. This chapter discusses best practices for data-presentation, gives examples of bad data-presentation, and discusses the issue of the use of data for unethical purposes.

• Chapter 12: Evaluating the data-collection and analysis process

  After the data collection is over and the analysis is done, it is important to evaluate the quality of these efforts. It is rare that a data-collection process is completed without unexpected setbacks. This chapter will discuss how a good assessment of a data-collection and analysis process can be achieved. Concepts of quality control during the process will also be explored.

• Chapter 13: Policy decisions and programmes based on data

  Now that the process by which indicators and data become available to the policy-oriented professional has been thoroughly discussed, this chapter will return to a more thorough review of the role that indicators and data play during the creation of policy and policy programmes.

• Chapter 14: Monitoring and evaluating policy programmes and decisions

  Sometimes the purpose of a data-collection project will be to allow the monitoring and/or evaluation of an existing policy. This chapter will discuss the issues specific to this type of data-collection and analysis process.

It is our hope that, after reading these materials, the policy-oriented professional will be better informed as to why an empirical approach to policy is an effective one and what pitfalls to avoid in using data to inform policy, and the data-oriented professional will better understand the needs and sensitivities of the policy process and tailor his/her field/analysis methods appropriately. Any questions about these materials are welcome and should be directed to us at metagora@oecd.org.