Guidelines for Informing Policy via Data

CHAPTER 5 - COLLECTING DATA: GETTING STARTED (page 1)

Despite the myriad possibilities for data collection, there is a common set of best practices that applies to all methods.

Once a clear set of research goals has been established, collecting data then involves two main sets of tasks: deciding from whom or what the data will be collected, and creating the set of questions that will be asked. For example, survey and census data may involve finding a sample/population frame and some type of complex sample design. Focus-group data or expert-interview data may also be based on a random sample, or may be based on the selection of particular informants or experts on the subject to be studied. Questionnaire design will involve determining the format and order of questions and rigourously testing those questions to ensure that they are interpreted correctly by respondents. The set of questions to be asked might be small, designed to be followed-up by expert probing by trained interviewers, or might serve as an exact script for the interviewer to follow. In either case, the same rigourous testing of the scripted questions must occur.

One general guiding principle is relevant to all of the above possibilities: keep the data-collection process as simple as possible to get the job done. The more complex a process, the more likely some small step in that process will not go smoothly.

The process by which data are collected from a respondent in a face-to-face interview is a case in point. There are many cognitive processes that can affect the accuracy of the information gathered:

1. Researcher designs question →Does the researcher ask the question s/he thinks s/he’s asking?

2. Interviewer reads question → Does the interviewer read the question accurately?

3. Respondent listens to interviewer → Does the respondent hear the question accurately?

4. Respondent interprets question → Does the respondent interpret the question as the researcher intended?

5. Respondent answers question → Does the respondent say what s/he means to say?

6. Interviewer listens to answer → Does the interviewer hear the answer accurately?

7. Interviewer interprets answer → Does the interviewer hear the answer accurately?

8. Interviewer records answer → Does the interviewer interpret the answer accurately?

9. Data-entry staff member reads answer → Does the data-entry staff member read the answer correctly?

10. Data-entry staff enters answer into a database → Does the data-entry staff member make an error?

A single misinterpretation at any of the steps might bias the data and the subsequent statistics. Every member of the data-collection team must therefore be aware of the importance of his/her role in the data-collection process, and the communication between the researcher and the respondent should be as direct as possible.

Creating a conceptual framework and research goals, and deciding on the mode of data collection are discussed below. Chapter 6 will review procedures for determining from whom or what the data are collected, and Chapter 7 will discuss questionnaire design.

For example, the Philippines Metagora project team engaged in a detailed development of a conceptual framework for their survey on access to land by indigenous peoples (IPs). In the process of creating that conceptual framework, the team explored what was meant by an "indigenous person," what was meant by "ancestral domain," and what the current law was pertaining to the right of IPs to ancestral lands. These three concepts had to be clarified in order to develop the sampling frame and the research topics for the survey. The team chose to use a rights-based approach for understanding the relationship between IPs and their traditional domains. They researched international human rights law, and designed the survey questionnaire to answer questions related to the IPs’ realisation of their human rights concerning their ancestral domains.

The conceptual framework for the Philippines Metagora project thus addressed perceptions and awareness of land rights, realisation of land rights and violations of those rights, and methods for measuring the rights and mechanisms for realising their rights to ancestral domains and lands. Further information is given in the Case Study associated with this manual.

5.2 DEVELOPMENT OF RESEARCH GOALS

Prior to making any decision as to who will be interviewed and in what manner, a clear set of research goals should be outlined. This step is often skipped in favour of diving right into developing a questionnaire. That is a mistake for several reasons. First, without a clear set of research goals it is difficult to ensure that the questions on the questionnaire will elicit the information required, and it will be difficult to determine if any of the questions on the questionnaire are extraneous. In order to increase the probability that any one respondent will complete the data-collection process, it is important to keep that process as short as possible. Only the data needed should be collected, and in as concise a way as possible.

Second, in framing research goals, a researcher may discover that a particular data-collection technique will work better than others. For example, if a researcher wants to know about the experiences of the homeless in an urban area, specialised sampling techniques will be required, and interviews will need to be kept very short if that population is transient and reluctant to be interviewed. Certain research questions lend themselves to the use of focus groups, others to individual interviews. Some research questions will require detailed qualitative data be collected from all respondents, while others are addressed readily via quantitative data. If the researcher starts with a clear set of research questions, then determining the type of data collection that is most appropriate will be easier.