Guidelines for Informing Policy via Data

CHAPTER 5 - COLLECTING DATA: GETTING STARTED (page 3)

5.2.2 An Example

The Metagora pilot projects all started by developing research goals. The pilot project in the Philippines serves as an exceptional example of that process. A great deal of time was spent by the project staff on considering their research interest and what local partners would be required for that focus to work.

Because the project staff was initially composed of members of the staff of the Human Rights Commission of the Philippines, with the National Statistical Coordination Board as a partner, the staff were interested in focusing on a human rights issue, and the issue of land rights of indigenous peoples of the Philippines was their priority. They quickly realised that such a research project would only be successful and ethical if it was of direct interest to the indigenous peoples themselves. They contacted the NCIP to determine its interest in the project, and the NCIP soon became an integral partner.

Other partners were needed for the project to be successful. The NSCB had no mandate to undertake data collection, but they knew that the National Statistical Office of the Philippines did. They approached that office, which, although initially reluctant, embraced the project as they became more involved in its planning. They also brought in the National Statistical Coordination Board as a partner.

• To determine the level of awareness and perceptions of IPs on their rights to ancestral domains and ancestral lands;
• To describe their enjoyment/realisation on one hand, and/or violations of rights to their ancestral domains and ancestral lands, on the other;
• To identify measures provided to IPs by governance stakeholders to enhance/facilitate the enjoyment of their rights to ancestral domains and ancestral lands; and
• To describe community and government mechanisms used by IPs to redress their grievances due to violations of their rights to ancestral domains and ancestral lands.

From a clear understanding of their objectives, the Philippines Metagora Pilot staff were able to determine that they would need both quantitative data in order to judge the awareness level among indigenous peoples, and also qualitative data in order to obtain detailed information on the problems they face. They therefore designed a project that began with a focus group discussion that would be used both to collect qualitative information and to gather information for use in designing a quantitative questionnaire form, and then progressed to a random sample survey.

The pilot project in the Philippines is an excellent example of the development of research goals. The staff were aware that it would only be ethical to collect data on the situation of indigenous peoples if that data were of interest to the indigenous peoples themselves and the indigenous peoples were involved in designing the project. The staff went through several iterations of developing research objectives before deciding on the specific method of data collection they would use. When they determined that they did not have the necessary expertise to implement the project themselves, they involved organisations that did. The final objectives were a joint product of all interested parties, and the data-collection methods employed were tailored towards meeting those objectives, not the other way around.

Data-collection mode is the method by which the questions of the survey will be delivered to the respondent pool, for example, via face-to-face interviews, or by telephone, or by another method. Each potential collection mode has both advantages and disadvantages, and the type of questions on the survey might indicate the use of one mode over another. For example, for more open-ended questions designed to collect qualitative data, a mail-out/mail-back mode will not work well as respondents are unlikely to take the time to write down good qualitative answers.

5.3.1 Mail-Out/Mail-Back

In countries with high literacy rates, and especially in situations where the whole population is to be interviewed (i.e., censuses), the use of mail-out/mail-back survey forms is common. For this technique, the sample frame is a list of addresses for the population of interest, and the questionnaire is designed to be self-administered. The survey forms are mailed to the entire population, or a sample of that population, and an addressed, stamped envelope is provided for the respondents to use to return their questionnaires.

The advantage of the mail-out/mail-back method is cost: it is a relatively inexpensive method for implementing a survey. The disadvantage, however, is that research has shown the response rate for mail-out/mail-back surveys is lower than for other modes of data collection, such as telephone interviews and face-to-face interviews.

In countries with extensive telephone coverage, surveys are often conducted by telephone. Telephone interviews are significantly less expensive than face-to-face interviews. While telephone surveys are typically more expensive than mail-out/mail-back surveys, they have been shown to yield higher response rates. Telephone surveys may also provide higher quality responses than mail-out/mail-back surveys, as trained interviewers ask the questions. If the interviewers are all in the same building during telephone calls, then supervision is significantly easier than when interviewers are speaking with respondents in their homes.

If the population of interest is a national or other large population, then telephone service must be mostly universal for a telephone survey to be feasible. One interesting aspect of telephone surveys is that a sample frame, or list of telephone numbers of the population, is not necessarily required. Due to a technique called random-digit dialing, a simple random sample of the population can be obtained relatively easily for a telephone survey.

When telephone surveys first began, the questionnaire itself was still a paper questionnaire that would be filled out by the interviewer during the telephone call. With the advent of affordable desktop computers, CATI has become a viable alternative. During CATI, the interviewer accesses the survey questionnaire in electronic format, on a desktop or laptop computer. As the respondent answers questions, the CATI interviewer enters those answers into the electronic survey questionnaire form.

CATI has two distinct advantages over traditional telephone interviewing. First, because data are entered directly into a computer, the data-entry step required of a pencil-and-paper survey is skipped, saving on time and cost. Second, for a complicated questionnaire with multiple skip patterns (places where the answer to a question determines which question is asked next), the CATI instrument can be programmed so that skips happen automatically, depending on the answer entered, greatly reducing interviewer error in following skip patterns and the time needed to complete the questionnaire. A CATI instrument, however, might take more time to prepare than a paper-and-pencil questionnaire, given the need to programme those skip patterns.