Guidelines for Informing Policy via Data

CHAPTER 3 - THE VALUE OF DATA AND INDICATORS AND THEIR ROLE IN POLICY MAKING (page 8)

3.6.2 The Usefulness of Combining Methods and Approaches for Collecting Data and Building Indicators

3.6.2.1 Quantitative and qualitative data

As mentioned above, the data collected can be quantitative or qualitative in nature. In broad terms, qualitative data are comprised of words whilst quantitative data are comprised of numbers and statistics. Nevertheless, it is possible to modify and transform qualitative data into quantitative data. Each type has its advantages and limits [38].

Broadly speaking, qualitative data, such as data gathered from expert interviews and focus groups, are more flexible, allowing in-depth exploration of the meaning behind concepts and events, enabling a clear understanding of the situation, and exposing the motivations and patterns of association between factors. One main disadvantage, however is that sample sizes are often small and do not allow for representative data to be collected. The generalisation of findings can thus be a problem. In addition, expert interview methods rely heavily on respondents being reasonably articulate; and processes behind the analysis of the data are not always transparent or replicable.

Quantitative data, which are usually based on surveys, are extremely useful in producing statistics. Moreover, if random samples are used, estimates are precise and inferences can be made about the target population. Other advantages of quantitative data include: their capacity to measure the extent, size, and strength of observed phenomena; their usefulness in determining the importance of given factors in influencing the outcomes; and their use of standardised procedures, which allow for replication of results. However, quantitative data-collection methods can be costly, especially if the target population is hard to reach; sampling frames are not always available; the use of structured interviews can sometimes hinder a detailed exploration of the reasons behind specific actions or decisions; the use of standardised questionnaires means little flexibility; and key concepts must be clearly defined and translated into meaningful survey questions.

Rather than regard these methods as opposed and contradictory, it is constructive to see them as complementary. Indeed, although using a combination of these methods implies extended timeframes and increased costs, the results can be valuable, as can be illustrated by the Metagora project.

Gathering both quantitative and qualitative information helps refine methodologies and enriches analyses. The Metagora project has shown not only that assessment of democratic governance and human rights issues can build on the solidity of proper quantitative reporting, but also that the design and use of quantitative measurement methods and tools must be informed by accurate qualitative information, including in-depth documentation of the perceptions and experiences of target populations, and of the various assumptions and expectations of the different stakeholders.

Qualitative information is essential not only to ensure proper design of survey questionnaires, but also to focus statistical analysis on relevant issues and to provide appropriate contextual frameworks for an effective policy-oriented interpretation of quantitative data. In Metagora pilot activities, the design of survey questionnaires and databases was based on qualitative information gathered through in-depth narrative interviews with victims of rights infringements, focus group discussions with people belonging to the target populations, substantive reports of local experts, and large consultations with all relevant stakeholders.

As a result of this dual approach, the Metagora survey results provide qualitative, contextual information that ensure that policy-oriented reports are based on the broadest body of information possible. The pilot project of building quantitative data from records of narrative reports on human rights violations, and the attempt to merge quantitative and qualitative human rights data into a single database, are providing substantive lessons with obvious universal scope.

Metagora initiatives have thus demonstrated that quantitative and qualitative data can and should interrelate in order to properly and comprehensively inform democratic governance and human rights monitoring exercises. The combined use of quantitative and qualitative data contributed to providing richer analysis in various pilot experiences, which provide firm illustrations of the synergy between quantitative and qualitative information.

In the Philippines, the collection of these two types of data through a survey and focus group discussions (FGDs) has yielded rich results. The survey revealed a high level of awareness of indigenous populations’ rights to ancestral land and domains, while the FGDs elicited greater details, which made it possible to identify which types of rights were best understood by the local inhabitants.

In Mexico, the survey questionnaire was based on a series of in-depth interviews. The idea was to identify the key questions and problems to be measured, the way in which the relevant institutions operated, and the places or circumstances in which the population was exposed to the risk of abuse.

In Palestine, an examination into education rights was conducted with a database that contained both data from household surveys and qualitative assessments of the areas in which access to education needs improvement.

38. For more complete information, see Government Chief Social Researcher’s Office, Policy Hub, Magenta Book, Chapter 6: How are the data collected?