Developing and Implementing a Triangulation Protocol for Qualitative ...

Qualitative Health Research http://qhr.sagepub.com

Developing and Implementing a Triangulation Protocol for Qualitative Health Research Tracy Farmer, Kerry Robinson, Susan J. Elliott and John Eyles Qual Health Res 2006; 16; 377 DOI: 10.1177/1049732305285708 The online version of this article can be found at: http://qhr.sagepub.com/cgi/content/abstract/16/3/377

Published by: http://www.sagepublications.com

Additional services and information for Qualitative Health Research can be found at: Email Alerts: http://qhr.sagepub.com/cgi/alerts Subscriptions: http://qhr.sagepub.com/subscriptions Reprints: http://www.sagepub.com/journalsReprints.nav Permissions: http://www.sagepub.com/journalsPermissions.nav Citations (this article cites 22 articles hosted on the SAGE Journals Online and HighWire Press platforms): http://qhr.sagepub.com/cgi/content/refs/16/3/377

Downloaded from http://qhr.sagepub.com at PENNSYLVANIA STATE UNIV on April 17, 2008 © 2006 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.

QUALITATIVE Farmer et al. / AHEALTH TRIANGULATION RESEARCHPROTOCOL / March 2006

ARTICLE 10.1177/1049732305285708

Developing and Implementing a Triangulation Protocol for Qualitative Health Research Tracy Farmer Kerry Robinson Susan J. Elliott John Eyles

In this article, the authors present an empirical example of triangulation in qualitative health research. The Canadian Heart Health Dissemination Project (CHHDP) involves a national examination of capacity building and dissemination undertaken within a series of provincial dissemination projects. The Project’s focus is on the context, processes, and impacts of health promotion capacity building and dissemination. The authors collected qualitative data within a parallel–case study design using key informant interviews as well as document analysis. Given the range of qualitative data sets used, it is essential to triangulate the data to address completeness, convergence, and dissonance of key themes. Although one finds no shortage of admonitions in the literature that it must be done, there is little guidance with respect to operationalizing a triangulation process. Consequently, the authors are feeling their way through the process, using this opportunity to develop, implement, and reflect on a triangulation protocol. Keywords: triangulation; qualitative methods; health promotion research

T

riangulation is a methodological approach that contributes to the validity of research results when multiple methods, sources, theories, and/or investigators are employed. Much of the literature dealing with qualitative modes of investigation within the health and social sciences cites the importance of triangulation (Erzerberger & Prein, 1997; Flick, 1992, 2002). There is little direction in this literature, however, regarding the nature or shape that this analytical process should take (National Institutes of Health, 1999). Indeed, a consistent criticism leveled at qualitative researchers in general has been a lack of detailed methodological description AUTHORS’ NOTE: We would like to note that this work would not be possible without the research, intervention efforts, and interpretative expertise of the provincial dissemination project teams and their stakeholders. In particular, we would like to thank the research advisory group members, who participated in the development of the project’s triangulation protocol. This paper is written on behalf of the CHHDP Investigative Team, Strategic and Research Advisory Groups. The CHHDP Investigative Team consists of Susan Elliott, Jennifer O’Loughlin, John Eyles, Roy Cameron, Dexter Harvey, and Kerry Robinson. The Strategic Advisory Group consists of Deborah Bradley, Catherine Donovan, Scott McLean, Kelly McQuillen, P. J. Naylor, Gilles Paradis, Brenda Perkins, and Kim Raine. The Research Advisory Group is made up of Lori Ebbesen, Ken Fowler, Ernest Khalema, Viviane Leaune, Murray McKay, Olive Moase, and Barb Riley. This research is funded by the Canadian Institutes of Health Research. QUALITATIVE HEALTH RESEARCH, Vol. 16 No. 3, March 2006 377-394 DOI: 10.1177/1049732305285708 © 2006 Sage Publications

377


378

QUALITATIVE HEALTH RESEARCH / March 2006

(Barbour & Barbour, 2003; Decrop, 1999; Morse, 1994). For example, Miles and Huberman (1994) remarked, “They [qualitative reports] are most often heavy on the ‘what’ (the findings, the description) and rather thin on the ‘how’ (how you got to the ‘what’)” (p. 262). In this article, we attempt to address some of these gaps by documenting a triangulation process developed as part of the Canadian Heart Health Dissemination Project (CHHDP). Given the range of qualitative data sets used in this project, we felt it was necessary to document the triangulation process that aided our development of an integrated set of findings. Specifically, our objectives in this article are 1. to explore qualitative triangulation within the social and health sciences, 2. to provide a worked example of the application of a qualitative triangulation protocol, and 3. to reflect on the methodological challenges inherent in triangulation.

QUALITATIVE TRIANGULATION The term triangulation originates from the sciences of land surveying and navigation, and refers to a simple method for determining the position of a point using observations from two additional points (Sharp, 1943). Triangulation was originally used in the health and social sciences by quantitative psychologists Campbell and Fiske (1959), who suggested using multiple tests to measure the same constructs to look for “convergent validity.” It has since been employed in a broad range of research related to both the social and health sciences, including political science (Arts & Verschuren, 1999; Davies, 2001), geography (Yeung, 2003), social work (Sheppard, Newstead, DiCaccavo, & Ryan, 2001), and nursing (Risjord, Dunbar, & Moloney, 2002). Recently, there has been a tremendous surge in the level of interest in triangulation within the realm of public health and health promotion (Aung et al., 2001; Goodson, Gottlieb, & Smith, 1999; Tones, 2000; von dem Knesebeck, Joksimovic, Badura, & Siegrist, 2002; Nakkash et al., 2003). This interest appears to reflect the belief that triangulation can lead to a multidimensional understanding of complex health issues. Regardless of the disciplinary background of the researchers or the research problem itself, the primary purpose of the triangulation process remains the same. Based on the work of Erzerberger and Prein (1997), the primary purposes of triangulation are to explore convergence, complementarity, and dissonance. Each of these, in turn, contributes to the overall goal of triangulation, that is, to enhance the validity of the research by increasing the likelihood that the findings and interpretations will be found credible and dependable (Lincoln & Guba, 1985). Researchers can employ various approaches to data collection to confirm or disconfirm previous research results. “The underlying assumption is that the validity of research results is enhanced if the different methodological approaches produce convergent findings about the same empirical domain” (Erzerberger & Prein, 1997, p. 144, emphases in original). Researchers can also choose to enhance validity by triangulating various approaches to form a more complete picture of the issue of interest. Through ascertaining the complementarity of various data sources, we can expose multiple dimensions of the same research issue and thereby


Farmer et al. / A TRIANGULATION PROTOCOL

379

increase our level of understanding (Fielding & Fielding, 1986). As a result of triangulation, researchers might also find dissonance in the findings; that is, two or more sources of data addressing one research problem can produce unexplainable divergences that might lead to a rejection of previous assumptions (Erzerberger & Prein, 1997). Dissonant findings can be considered constructive if they serve as a point of departure for the creation of a new hypothesis or more nuanced understanding (Miles & Huberman, 1994). Four types of triangulation techniques have been identified in Denzin’s (1978) seminal work. Methodological triangulation involves the use of more than one research method or data collection technique. This might involve employing several types of qualitative approaches (e.g., interviews, document analysis, focus group discussion). Data triangulation involves the use of multiple data sources (e.g., two types of reports) or respondent groups (e.g., professionals vs. lay). Theoretical triangulation involves using alternative disciplinary or substantive theoretical lenses to view research findings (e.g., stages of behavior change vs. health belief model). Finally, investigator triangulation entails involvement of two or more researchers in the analysis. The type of triangulation chosen and the decision to employ single or multiple triangulation techniques depend on the nature of the research question and should complement the methodological paradigm (e.g., phenomenology) that informs the question (Dootson, 1995). Like any analytical approach, there are advantages and disadvantages associated with adopting a multimethod approach to research. It appears as though triangulation has become the trendy way to help improve confidence in the research results and to overcome research bias (Murray, 1999). On the other hand, its application is problematic, because there might be incompatibility between the units of analysis and theoretical paradigms, and the process of triangulation might actually amplify sources of error and bias (Begley, 1996; Sim & Sharp, 1998). Although these advantages and disadvantages have been discussed in great detail (see also Blaikie, 1991; Kopinak, 1999; Lambert & McKevitt, 2002; Thurmond, 2001), our understanding of the triangulation process itself is limited by a scarcity of literature explaining how this technique is applied. This lack of detailed procedural information on triangulation was corroborated by a search of relevant literature that revealed a dearth of reference material.1 The majority of researchers merely discuss the need to triangulate findings, describe the various types of qualitative triangulation (e.g., investigator, methodological), and/ or remark on the type(s) of triangulation employed in a particular study. Although qualitative researchers argue for the need to collect and present in-depth, detailed, and contextualized information, this same level of richness is often missing from their own methodological descriptions. Basing their work on that of Smaling (1987), Meijer, Verloop, and Beijaard (2002) have described three approaches to the analysis of qualitative data that can be applied to multimethod triangulation: (a) the intuitive approach, in which a researcher intuitively relates information obtained from different instruments to each other; (b) the procedural approach, in which each comparative step taken in the triangulation process is documented to ensure transparency and replicability; and (c) the intersubjective approach, in which a group of researchers tries to reach agreement about the steps in their triangulation process. It is our observation that most researchers employ an intuitive approach and as such do not clearly articulate the triangulation procedures undertaken (see, e.g., Aung et al., 2001; Horne &


380


Costello, 2003). There are, however, exceptions among the researchers who endeavored to outline a triangulation process undertaken in qualitative research (see, e.g., Flick, 1992; Meijer et al., 2002). Yet, although these authors asserted that their respective articles contain a methodological framework for triangulation, they did not provide a systematic accounting of the process. The phenomenological study by Flick (1992) focused more on theory development and less on the triangulation process. Although the primary goals of triangulation are to explore levels of convergence, complementarity, and dissonance, Meijer et al. (2002) focused solely on complementarity. Moreover, although Meijer and colleagues identified three steps in the triangulation process, two of these steps (i.e., Step 1: analysis of semistructured interviews and concept maps; Step 2: analysis of stimulated recall interviews) merely reflect general qualitative analysis, whereas only the third (i.e., triangulating results of previous steps) addresses the triangulation process itself. Finally, neither article included a reflection on the challenges encountered during the triangulation process. Despite the challenges in undertaking triangulation, it has significant potential to expand the depth and breadth of our understanding of complex health and social issues. It allows for greater understanding by acknowledging that there is no longer one reality against which results can be verified or falsified, but that research is dealing with different versions of the world. Triangulation takes into account, that subjective knowledge and social interactions should be understood as parts of (social, local, institutional) contexts and on the historical backgrounds of those contexts. (Flick, 1992, p. 194)

As a result of the limitations in the literature, we set out to develop a triangulation protocol based on our experience as qualitative researchers, the existing literature, and substantial research experience of a national Project Advisory Group made up of provincial project researchers (through a 1-day workshop and electronic follow-up). We felt that the protocol should include the following: sorting (data set preparation), convergence or dissonance coding and assessment, completeness comparison, researcher comparison, and feedback.

CONTEXT FOR DOING TRIANGULATION: METHOD OF THE CHHDP CASE STUDY This article is based on research drawn from the Canadian Heart Health Dissemination Project (CHHDP), designed to investigate the learnings of the provincial projects involved in the Canadian Heart Health Initiative (CHHI) Dissemination Phase (Elliott et al., 2003). The CHHI is a 15-year program designed to address the epidemic of cardiovascular disease in Canada. The dissemination phase involves the extension of the best practices developed in a previous demonstration phase, as well as research initiatives to examine the factors affecting capacity for and dissemination of community-based health promotion. The triangulation exercise that follows is focused on a specific objective of the larger project: to examine contextual factors affecting health promotion dissemination, capacity building, and related research processes. The triangulation protocol involved two sources of qualitative data: key informant interviews with researchers



381

and project stakeholders from each of three provincial projects and analysis of project reports. Key informant interview respondents (N = 40 across three provincial projects, Prince Edward Island, Ontario, and Manitoba) were purposefully sampled to achieve maximum variation in opinions from diverse perspectives (researchers, government, community agencies) (Weiss, 1998). In fact, we conducted interviews beyond thematic saturation to ensure representation of voices from different respondent groups. The interview guide was developed to address the research objectives of the CHHDP and was pilot tested with three individuals representing researcher and stakeholder perspectives. Following pilot testing, we modified the interview guide to clarify question wording. All interviews were audiotaped, transcribed verbatim, and imported into NUD*IST for thematic analysis. In addition, 30 provincial project reports (e.g., funder reports, journal publications, final project reports) were analyzed in NUD*IST. We selected the documents to reflect content related to capacity and dissemination, a range of time periods in each project, and different audiences (academic, funders). The reports document research and intervention activities, provincial context, challenges and supports to capacity building and dissemination, and both qualitative and quantitative evaluation findings. We analyzed the interview transcripts and the project report text thematically to identify core consistencies and meanings using a coding scheme to index, search, summarize, and analyze the data (Patton, 2002). The coding schemes emerged through both inductive and deductive approaches based on the interview guide, report analysis questions, and a reading of the interviews and reports, as well as discussion and feedback among the research team. Analysis included searches of text both within and across the provincial cases based on thematic frequency, theme examples, and patterns of similarities and differences. Two researchers coded a subset of interviews (n = 10) and project reports (n = 3), showing approximately 70% agreement on very detailed coding, close to 90% agreement for main categories, and 80% code-recode dependability (from Time 1 to Time 2), indicating good coding dependability (Miles & Huberman, 1994). The provincial analysis summaries were validated through a member-checking process (Creswell, 2003), in which interview respondents and project representatives reviewed reports and provided feedback on the accuracy of interpretations. This study received ethics clearance from the McMaster University Research Ethics Board. All interview respondents provided informed consent for their participation as well as consent for their interviews to be taped and transcribed verbatim. No identifying information for respondents is presented; all quotes are anonymous and affiliated only with provincial projects.

DEVELOPING A TRIANGULATION PROTOCOL Given the project’s multiple data sets and the need to generate an integrated set of findings, the aim of employing triangulation was to help ensure complementarity and test for convergence and dissonance of ideas inherent therein. However, before findings were triangulated, we developed and reviewed decision rules guiding the analysis of results from each method to determine if particular perspectives were weighted differently from others, and why and how findings were reported (i.e.,


382


were all thematic areas represented, or did the findings represent the most frequent themes discussed in detail with less frequent themes listed with few examples). Limitations or differing purposes of data sets were acknowledged prior to undertaking the triangulation process (e.g., one data set, such as interviews, allowed for data collection on a priori research questions; the other data set, such as reports, included content on some, but not all, research questions). Following separate analysis of the interview and document analysis data respectively, we employed three types of triangulation: 1. Multiple investigators: Two researchers undertook independent applications of the triangulation protocol and compared results. 2. Methodological: Results were compared from two methods of data collection (key informant interviews, document analysis). 3. Data source: A range of project perspectives were represented in the interview analysis (researchers, government officials, regional health agencies, nongovernmental organizations ); and a range of project documents were reviewed (reports to funders, technical reports, peer reviewed publications).

Given the criticism that has been leveled at researchers employing triangulation without outlining how they went about the process, we present a comprehensive outline of the six steps taken (Table 1).

APPLICATION OF THE TRIANGULATION PROTOCOL To facilitate the application of the triangulation protocol, we have provided a detailed example of the process undertaken by two researchers (investigator triangulation). Independent testing of the triangulation protocol focused on the following research question: What contextual factors were found to influence health promotion capacity building and dissemination processes?

Step 1: Sorting The findings related to the research question from each data set were sorted and separated from the rest of the data into two files: interview findings and document analysis findings. We then reviewed the contents of both files to identify the key themes discussed in each data set to create a unified list of themes to compare for presence and frequency, meaning, and examples. These themes form the rows of the convergence coding matrix used to summarize similarities and differences between the two sets of data (Table 2).

Step 2: Convergence Coding We compared the two files of findings with respect to the meaning and interpretation of themes, the frequency and prominence of themes (e.g., number of documents or interview participants mentioning a theme), and specific examples provided to support or explain a particular theme. We then applied the Convergence Coding Scheme (see Table 1), first to determine convergence between the two sets of results on the essence of the meaning and prominence of themes presented and, sec-



TABLE 1:

383

Triangulation Protocol

Step

Activity

1. Sorting

2. Convergence coding

Sort findings from each data source or method into similarly categorized segments that address the research question(s) of interest to determine areas of content overlap and divergence. Identify the themes from each data source. Compare the findings to determine the degree of convergence of (a) essence of the meaning and prominence of the themes presented and (b) provincial coverage and specific examples provided in relation to each theme. Characterize the degree and type of convergence using the following typifications of concurrence (or nonconcurrence) within theme areas.

Convergence coding scheme Agreement There is full agreement between the sets of results on both elements of comparison (e.g., meaning and prominence are the same, provincial coverage and specific examples provided are the same). Partial agreement There is agreement on one but not both components (e.g., the meaning or prominence of themes is the same, provincial coverage or specific examples provided are the same). Silence One set of results covers the theme or example, whereas the other set of results is silent on the theme or example. Dissonance There is disagreement between the sets of results on both elements of comparison (e.g., meaning and prominence are different; provincial coverage and specific examples provided are different). 3. Convergence assessment Review all compared segments to provide a global assessment of the level of convergence. Document when and where researchers have different perspectives on convergence or dissonance of findings. 4. Completeness assessment Compare the nature and scope of the unique topic areas for each data source or method to enhance the completeness of the united set of findings and identify key differences in scope and/or coverage. 5. Researcher comparison Compare the assessments of convergence or dissonance and completeness of the united set of findings among multiple researchers to (a) clarify interpretations of the findings and (b) determine degree of agreement among researchers on triangulated findings. Plan for how disagreements will be handled and how final decisions on interpretations will be made. 6. Feedback Feedback of triangulated results to research team and/or stakeholders for review and clarification.

ond, on the convergence of the coverage and specific examples provided in relation to each theme. Based on a comparison between the two researchers’ coding and discussion to reach consensus, the resulting convergence coding matrix of themes and provincial examples for this research question is presented in Table 2. The following is a discussion of examples of agreement, partial agreement, silence, and dissonance along with illustrative quotes of the various contextual factors acting across the three provincial projects (Table 3).

Agreement Across the themes, there were only two instances in which there was agreement in both meaning and/or prominence and provincial examples. One of these theme


384


TABLE 2:

Contextual Factor Theme Frequencies and Quotes Number of Interview Respondents or Documents Mentioning Each Factor

Theme

Interviews

Documents

Provincial health reform or system

37

12

Geographic factors

20

10

Socioeconomic climate

21

7

Funders and funding process

23

0

Conference of Principal Investigators (COPI)/ Canadian Heart Health Initiative (CHHI) nationally

13

6

Political climate

15

2

Sample Quotes It began as Manitoba was experiencing a major shift in the structure of the provincial health care system. Decentralization had resulted in 11 new rural health regions, each with its own Regional Health Authority and each responsible for all health care planning and funding, albeit under the watchful eyes of the provincial health department: this department was experiencing a similar “sea change” in authority and direction. (Manitoba [MB] document) Two thirds of our population is in one city. We’ve got one other city that’s sort of 40,000 people, and that’s virtually nothing. . . . So, despite the fact that we have the majority of our population in an urban, we really are like a big town, our province is very, very rural. A lot of our communities are remote and limited resource-wise. (MB interview) People were quite conscious of going into an area where there is high unemployment, education levels are not as high as other parts of the province. So when you undertake a project, knowing you’re walking into that kind of situation, you’re always very sensitive towards the community. (Prince Edward Island [PEI] interview) Of course the federal government provided the bulk of the money also provided the connections to the national network. I think again, through NHRDP [National Health Research and Development Program], it certainly influenced the type of research and the direction of research and formulation of the research questions and brought together the expertise from throughout the world to assist us in developing research and research questions. (MB interview) The COPI meetings were a good opportunity to share what they were doing and sort of adapting things. Also we did have these Heart Health Initiative conferences, it was an opportunity to see what everybody else was doing very informally and almost incidentally, there was a diffusion piece of picking up what other people were doing. (Ontario [ON] interview) The public health system in Ontario is currently being challenged, however, by major structural changes. The first of these involves the “municipalisation” of funding responsibilities. In January 1997, the Ontario government announced a change in funding from the previous cost-shared arrangement for boards of health to a requirement that boards be 100% municipally funded as of January 1, 1998. (ON document) (continued)



385

TABLE 2 (continued) Number of Interview Respondents or Documents Mentioning Each Factor Theme

Interviews

Champions/leaders

Documents

15

1

Previous CHHI phases

8

8

Burden of cardiovascular disease (CVD) illness, risk factors Community cohesion/culture

6

9

12

0

Sample Quotes The Dissemination Phase in the Canadian Heart Health Initiative would not have happened without key players at the national level. . . . So at the national level there was predisposition and capacity. You had a national champion, who was Andreas who had the vision, and you had resources—the NHRDP [National Health Research and Development Program] to fund us. So I think it’s clearly evident without the national leadership, the Dissemination Phase would not have happened. (ON interview) The demonstration project (1990-1995) established that a community mobilization initiative aimed at promoting heart health could be established in an Island community with high community participation and acceptance. (PEI document) PEI competes with other Atlantic provinces for the highest rates of death due to CVD in factors the country and the highest prevalence of CVD risk factors. (PEI document) People were willing to listen. They didn’t miss anything. If something was going on, they wanted to know what it was and who was involved in it, what were they doing and why were they there. Who does she think she is? (laughter) So I think the nature of the community, that’s why it came out the way it was, because of the size of the community and the nature and the attitude of the people who live here. (PEI interview)

areas was the role of provincial health reform or system development as a key environmental influence. This theme focused primarily on the negative role of health system change, the resulting system instability, and its impacts on capacity building and dissemination efforts. It was ranked as the most frequently identified factor in both data sets. In terms of examples, all three provinces were represented in both data sets, and the types of factors described were virtually identical: the positive role of Ontario’s mandatory public health programs and negative effect of provincial downloading of funding to municipal levels; Manitoba’s new Regional Health Authority (RHA) structure as having potential for a greater focus on health promotion coupled, however, with system instability and reduced funds and staff time for community-driven health promotion; and the positive potential of system reorganization in Prince Edward Island (PEI) to support local interest in health promotion, combined with public resistance amid acute care cuts and low resources.


386


NOTE: AG = agreement; DA = dissonance; PA = partial agreement; S = silence.

3

•

• •

Provincial health reform/system development as an environmental force Geographic factors of communities and provinces (urban, rural, population) Socioeconomic climate as a varied influence across the provinces Funders and funding process influence on research and program agendas COPI/CHHI nationally as guiding network and framework Political climate including change in government-influenced priorities Champions/leaders playing roles in shaping projects and health promotion Previous CHHI phases (risk factor survey, demonstration) Burden of CVD illness, risk factors Community cohesion/culture Totals 5

• • • •

•

PA

• 2

•

S

0

DA

4

•

•

• •

AG

3

•

•

•

PA

• 3

•

•

S

Theme Provincial Example

Convergence Code Theme Meaning and Prominence AG

Convergence Coding Matrix for Contextual Factors

Contextual Theme

TABLE 3:

0

DA


387

Partial Agreement An example of agreement in the theme meaning or prominence but only partial agreement in provincial coverage or examples was in the theme relating to the Conference of Principal Investigators (COPI) and the CHHI as a whole. The meaning of the theme in both data sets was the same, with COPI/CHHI’s being characterized as acting as both a network for sharing among provincial projects and a guiding framework for objectives, intervention, and research activities. The prominence of the theme in each data set was also similar, with it being ranked based on frequency as a middle-to-low contextual factor in the document analysis. However, the theme coverage and examples were coded as a partial agreement, as the interviews provided examples from all three provinces, whereas the document analysis did not include any examples from the Manitoba project. In addition the document analysis provided a unique example, that is, a key aspect of the CHHI as a national framework was its emphasis on integrating heart health promotion research and interventions into existing provincial public health systems.

Silence and Dissonance Although instances occurred in which there was silence in one set of results compared to another in terms of both the appearance of a theme and provincial examples (e.g., the influence of funder agencies and the funding process on research and program agendas), there were no instances in which there was a full disagreement on both theme meaning or prominence and provincial coverage or examples between the two sets of results. In the example of the role of funding agencies or funding processes, it is not surprising that the document analysis findings were silent on this issue given the focus and audience of the project reports. These reports were often peer-reviewed publications, technical reports of findings, or final research reports to the funding agency. It is rare that these types of reports would include reflections on the support or lack of support of the funding process, as the primary objective is to communicate research and intervention results either to practice policy or scientific audiences. In these instances of silence, the purpose and nature of the data sets explain the absence of particular themes from one data set to another. If, however, silence of a theme cannot be explained by differences in the nature of the data sets, it is up to the researcher to review the context of the data collection, including composition of the respondent groups, timing, and the research questions asked to determine if one or more of these factors are responsible for the absence of a theme.

Step 3: Convergence Assessment Taking into consideration both the meaning and prominence of themes, and the provincial coverage and specific examples provided to support the findings, there was either partial or full agreement between the two data sets on 75% of the theme and example areas. Although there were no instances of disagreement in the comparison of these two data sets, 25% of theme and example areas showed an instance of presence in one data set and silence or absence in the other data set. This might be the closest thing to disagreement or dissonance of findings between the two data


388


sets. It is likely that these differences in findings relate to differences in the focus, scope, and nature of the two data sets and not likely to a chance finding in one data set.

Step 4: Completeness Comparison We then compared the two sets of results to highlight both similar and unique contributions to the research question, ultimately creating a summary of the unified findings of the two data sets. In this step, we aim to broaden the range of findings relevant to the research question to ensure completeness in perspective and in the ways in which a theme is characterized. Based on the convergence assessment, it is evident that there are many theme areas and examples in which the two data sets agree and confirm core themes. The interview data generally added factors that were not documented in the official reports of the provincial projects. For example, leadership, funders or funding processes, and community cohesion or culture are three factors that were identified in the interviews and not in the documents, in part because project reports tend not to contain reflections on process issues of the projects or those factors that perhaps are not easy to quantify, such as the social culture of a province or community. In addition, the interviews broadened the perspective of the nature of influence of a contextual factor, showing more variation and a more nuanced interpretation. For example, the interview findings pointed to a combination of positive and negative effects of health reform, or lack of consensus among stakeholders on the role of socioeconomic factors. Together, the two data sets confirm the central themes and the weight of evidence toward one position or another, while adding a richer view of variations in the themes by province. As an example, although the interviews reveal some perspective on the potential facilitating role of new regional structures for health, analyzed together, the interviews and documents confirm the overall negative effect of health reform on creating system instability, lack of resources, and decentralized systems. Thus, each data set on its own provides part of the story for this research question, but together, they contribute to a higher level of analysis and a broader understanding of the research question.

Step 5: Researcher Comparison One part of the exercise was to apply the triangulation protocol to determine the level of agreement between multiple researchers with respect to the degree of convergence between two data sets (in both meaning and prominence of themes) and the complementary and unique contributions of each data set to the research question. Using a method proposed by Miles and Huberman (1994) to calculate coder agreement in qualitative analysis, we observed 80% agreement between the two researchers on level of convergence (theme meaning, prominence, and supporting examples) between the interview and document analysis findings. An example of the type of disagreement between the researchers occurred when one coded the provincial examples of the roles of champions and leaders as a partial agreement (as one provincial example was consistent in both data sets, but the document analysis data set provided no discussion of the theme for the other two provinces), whereas the other labeled this instance as silence (because the document analysis did not include any substantive discussion of leadership issues at all, with the exception of



389

one brief reference to a supportive local CEO and partners). In the four instances of researcher disagreement, we discussed the rationale behind the coding to come to consensus on an agreed code. The level of agreement between the researchers on the common and unique contributions of each data set to the research question based on a comparison of each researcher’s summary of the unified findings was 70%. In the instances of differences in the researchers’ synthesis of findings from the two data sets, all but one was related to one researcher adding more detail in the examples supporting the themes than the other researcher. However, for each theme, at least one example was presented similarly by both researchers, with five themes having identical summaries by the two researchers. There was only one instance in which the researchers disagreed on one of two examples presented for a theme; for the socioeconomic theme, one researcher highlighted the varying influence of socioeconomic (SE) factors across the provinces, whereas the other highlighted the varying level at which the projects focused on SE factors (provincial vs. regional level). Both comments reflect the researchers’ agreement on the varying ways in which SE factors influenced the climate for capacity building and dissemination. Overall, there is reasonable confidence in the convergence coding and completeness exercise, as the level of agreement between the two researchers met or exceeded 70%, the level of agreement that is considered acceptable to ensure confidence in the coding process (Miles & Huberman, 1994).

Step 6: Feedback A further step to the triangulation process that was helpful was sharing the triangulated results with the research team for feedback/comment, discussion of any issues of significant disagreement and incorporation of research team changes into the data interpretation. Where appropriate, this should be followed by the distribution of a summary of triangulated key findings to key stakeholder groups for review and comment. A summary of the triangulated results was sent to four provincial advisory group members from Manitoba, Ontario, and PEI for their review.2 The project representatives indicated that the findings accurately reflected provincial level contextual influences on, and interprovincial differences in, dissemination and capacity-building processes.3 Consequently, member checking reinforced the validity of our research results. If, however, member checking resulted in different interpretations from those represented by our findings, we would first have to determine if these interpretations could be accommodated by our overall research results. If not, then we would have to present both sets of interpretations and reflect on the explanation for these differences in our discussion.

REFLECTIONS ON THE TRIANGULATION PROCESS Undertaking triangulation has the potential to add tremendously to the validity of interpretations based on large, rich qualitative data sets such as the one currently being analyzed by the CHHDP research team. Triangulation enabled us to bring together multiple perspectives on our particular research question. By triangulating the findings from different methodological approaches, we were able to tap into


390


different elements of the issue, providing complementary findings that contributed to achieving a more complete picture of the issue under study. In the current research, bringing together themes from the document analysis and interview results facilitated the identification of overriding metathemes that cut across provinces, respondents, and methods (e.g., the overall negative effect of health reform). This generated higher level interpretations of the data or a gestalt of the findings for our specific research question. Triangulating different data sources (types of respondents, stakeholders, etc.), case study contexts (multiple provincial projects under study), methods, and researcher perspectives allowed us to enhance both the credibility and the transferability of the findings. Findings that are consistent across diverse data sources and confirmed by multiple data sets provide greater confidence in the credibility of interpretations and the potential to transfer key learnings to other similar contexts. On the other hand, instances of dissonance between data sets provide an opportunity for further analysis to explore the source of differences. Despite the apparent simplicity of the concept of triangulation (i.e., developing knowledge of an issue by using multiple methods), it is a complex process to undertake, presenting several challenges. First, due to differences in the data sets (i.e., their purpose, focus, and intended audience), their content will likely vary. This can have implications for how the data sets are analyzed and the extent to which the content of the data sets directly or indirectly relates to the research question(s) of interest. There might be different coverage of questions or themes across different data sets. Some differences in the presence or absence of themes and the specific examples or content of themes between different data sets or methods might be due to these inherently different qualities of the data sets or methods themselves. This poses a challenge in determining the source of dissonance or silence on themes across data sets. For example, in the context of the CHHDP’s research, the provincial project reports document the key findings, learnings, methods, and interventions of the provincial projects. The content and coverage of these reports is not within the control of the CHHDP researchers. Alternatively, the interviews were guided by a set of questions that the CHHDP team aimed to explore, in particular factors affecting project interventions, research, and, ultimately, findings. Therefore, the coverage and nature of findings from the two data sets differs, and the nature of the data sets shapes to a certain extent the range of themes that will emerge. This is a strength, because it provides a broader set of perspectives and methods from which findings can be drawn, thereby allowing for a more comprehensive set of learnings. However, such differences in the data sets prove to be a challenge to the confirmatory aims of triangulation, as it might be difficult to determine agreement on findings. Instances in which themes are present in one data set and absent from another or where there are disagreements on findings between data sets should be examined to determine if these differences are related to the nature and/or focus of the data sets or represent legitimate differences in findings. This examination and explanation should be noted in the triangulation results as limitations of data sets or caveats in interpretation of triangulated findings. Second, because some data sets might be better suited to addressing a particular research question than others, there might be situations in which the findings of one data set are privileged, or weighted more than those of the other data sets, during triangulation. Specifically, interview data collection allows the researcher to have some influence on the content of the data set, whereas with a secondary data source, such as project reports, the researcher has no control over what data are reported.



391

For example, in the current analysis, interview questions probed the role of funding agencies as a contextual factor, whereas the project reports (e.g., peer-reviewed publications, annual reports) did not include any such reflections. In this case, the interview data set provided a more explicit focus on the role of contextual factors and therefore could have been weighted more heavily in the triangulation process. Decisions about whether to weight one data set more than another should be determined on a case-by-case basis considering the characteristics and aims of the data sets and the guiding research question. Third, a further challenge to triangulation is ensuring that the findings from each data set are presented at a similar level of detail. This helps to avoid the situation in which the broad themes of detailed data sets are compared to data sets with little detail, making it difficult to determine if the silence of a particular data set is a real silence or if the summary simply does not contain adequate detail to judge the presence of the theme. This need for adequate detail to judge the presence and meaning of themes and examples must be balanced with the level of triangulation that is meaningful in terms of contributing complementary or convergent findings. Feedback of findings to stakeholders is an important part of the triangulation process, as it can confirm the accuracy of findings and help determine the presence and source of different results. Last, a related challenge to synthesizing and comparing different data sets is potential differences in their analysis and presentation (Paterson, Thorne, Canam, & Jillings, 2001). For example, if frequency analysis of themes is done for one data set and not another, it is difficult to compare the prominence of themes across the two data sets. An earlier triangulation exercise came on this challenge, as the interview findings were organized along theme frequency to determine consensus among respondents, whereas the document analysis, whose primary focus was to document evolution in project activities and highlight main findings, did not initially include frequency counts across documents. To compare theme prominence, we undertook frequency counts for the document analysis and then compared them with the interview results. Given that no method, data set, or analysis process is without flaws, it is important for qualitative researchers to be up front in their acknowledgement and recognition of limitations of the sets of findings that they use as inputs into a triangulation process. In addition, given that the purpose and content of particular data sets will vary, researchers should be clear on how they have chosen to use the data sets and if they have weighted a particular data set within the triangulation process.

CONCLUSION In this article, we have outlined a detailed triangulation protocol and a worked example using findings from the Canadian Heart Health Dissemination Project in response to a gap in the literature related to the operationalization of triangulation in qualitative health research. Employing multiple triangulation strategies (data source, method, and investigator) has contributed to a more complete representation of the role of context in shaping dissemination and capacity building within and across provincial heart health programs. In addition, commonalities between findings of the different data sets have revealed a convergence in interpretation. Although this triangulation process did not identify any obvious discord between


392


the data sets, this experience has highlighted the importance of exploring and understanding disagreements or silence in data sets. Although Erzerberger and Prein (1997) remarked, “General rules for the use of triangulation strategies cannot be established” (p. 153), after having developed and carried out this protocol, we assert that there are basic steps that can and should be implemented in the application of triangulation in any context: sorting, convergence coding and assessment, review for completeness, researcher comparison, and feedback. As qualitative research is inherently interpretive, so is qualitative triangulation. However, this does not mean that we cannot aim for rigor in all research processes including triangulation. Our intent here has been to be explicit about what triangulation entails and how we have attempted to use it to enhance and validate our interpretations. This methodological transparency provides readers with insight into how the research was carried out and key assumptions. One significant consequence of having consistent findings between multiple data sets is an increased confidence in the credibility of study findings (Fielding & Fielding, 1986; Knafl & Breitmayer, 1991). It must be pointed out, however, that the fundamental strategy for ensuring rigor in qualitative research is still “systematic, self conscious research design, data collection, interpretation, and communication” (Mays & Pope, 2000, p. 52). In other words, triangulation is only as strong as the study’s underlying theoretical, methodological, and analytical paradigms and the researchers’ skills and abilities. It is only one way of ensuring rigor, as Sim and Sharp (1998) pointed out: “Triangulation in itself is not necessarily a methodological virtue” (p. 30). Triangulation (like all methods) possesses strengths and weaknesses, each of which must be critically acknowledged and addressed throughout the analytic process. The relatively scant literature addressing triangulation has highlighted some of these but our application of the process uncovered many more challenges. It is our hope that by sharing and debating the methodological processes and challenges of triangulation, qualitative researchers will not have to rely on intuitively “feeling our way” but, rather, can be guided by a set of basic triangulation procedures that aim to enhance the validity of research results.

NOTES 1. We conducted a literature search of the following databases: CINAHL (1990-2005), PubMed (1990-2005), WebSPIRS (Social Sciences Index (1990-2004), PsycINFO (1989-2004), Sociological Abstracts (1990-2005), MEDLINE (1996-2004), EMBASE (1996-2004), and ERIC (1990-2005). Keywords searched individually and in combination included triangulation, qualitative, protocol, within methods, application. 2. We asked the provincial advisory group members to respond to the following questions: (a) Do the findings reflect the main contextual factors acting during the time of your dissemination project? (b) Do the examples adequately show similarities and differences in the contextual factors across these provinces? and (c) Does the summary in any way misrepresent the nature of your dissemination project or its broader provincial health environment? 3. During this study, provincial project researchers regularly disseminated their findings to stakeholders to keep them up to date on overall project activities and findings. Consequently, both research members and stakeholders were in a position to verify or refute the triangulated results. Member checking occurred within provinces, and thus researcher members and stakeholders were not expected to confirm or contest findings outside their jurisdiction.



393

REFERENCES Arts, B., & Verschuren, P. (1999). Assessing political influence in complex decision-making: An instrument based on triangulation. International Political Science Review, 20(4), 411-424. Aung, M., Zhang, M., Farhat, H., Gan, W., Salameh, M., Wu, L., et al. (2001). An exploratory study of the smoking issue in restaurants. Management Decisions, 39(4), 279-284. Barbour, R. S., & Barbour, M. (2003). Evaluating and synthesizing qualitative research: The need to develop a distinctive approach. Journal of Evaluation in Clinical Practice, 9(2), 179-186. Begley, C. M. (1996). Using triangulation in nursing research. Journal of Advanced Nursing, 24(1), 122-128. Blaikie, N. W. H. (1991). A critique of the use of triangulation in social research. Quality and Quantity, 25, 115-136. Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validity by the multi-trait, multimethod matrix. Psychological Bulletin, 56, 81-105. Creswell, J. W. (2003). Research design: Qualitative, quantitative, and mixed methods approaches (2nd ed.). Thousand Oaks, CA: Sage. Davies, P. H. J. (2001). Spies as informants: Triangulation and interpretation of elite interview data in the study of intelligence and security services. Politics, 21(1), 73-80. Decrop, A. (1999). Triangulation in qualitative tourism research. Tourism Management, 20, 157-161. Denzin, N. K. (1978). The research act: A theoretical introduction to sociological methods (2nd ed.). New York: McGraw-Hill. Dootson, S. (1995). An in-depth study of triangulation. Journal of Advanced Nursing, 22(1), 183-187. Elliott, S., O’Loughlin, J., Robinson, K., Eyles, J., Cameron, R., Harvey, D., et al. (2003). Conceptualizing dissemination research and activity: The case of the Canadian Heart Health Initiative. Health Education & Behaviour, 30(3), 267-282. Erzerberger, C., & Prein, G. (1997). Triangulation: Validity and empirically based hypothesis construction. Quality and Quantity, 31, 141-154. Fielding, N. G., & Fielding, J. L. (1986). Linking data. Beverly Hills, CA: Sage. Flick, U. (1992). Triangulation revisited: Strategy of validation or alternative? Journal for the Theory of Social Behavior, 22(2), 175-197. Flick, U. (2002). Qualitative research: state of the art. Social Science Information, 41(1), 5-24. Goodson, P., Gottlieb, N. H., & Smith, M. M. (1999). Put prevention into practice: Evaluation of program initiation in nine Texas clinical sites. American Journal of Preventive Medicine, 17(1), 73-78. Horne, M., & Costello, J. (2003). A public health approach to health needs assessment at the interface of primary care and community development: Findings from an action research study. Primary Health Care Research and Development, 4, 340-352. Knafl, K. A., & Breitmayer, B. J. (1991). Triangulation in qualitative research: Issues of conceptual clarity and purpose. In J. M. Morse (Ed.), Qualitative nursing research: A contemporary dialogue (pp. 226-239). Newbury Park, CA: Sage. Kopinak, J. K. (1999). The use of triangulation in a study of refugee well-being. Quality & Quantity, 33, 169-183. Lambert, H., & McKevitt, C. (2002). Anthropology in health research: From qualitative methods to multidisciplinarity. British Medical Journal, 325(7357), 210-213. Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Beverly Hills, CA: Sage. Mays, N., & Pope, C. (2000). Assessing quality in qualitative research. British Medical Journal, 320(7226), 50-52. Meijer, P. C., Verloop, N., & Beijaard, D. (2002). Multi-method triangulation in a qualitative study on teachers’ practical knowledge: An attempt to increase internal validity. Quality & Quantity, 36, 145167. Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis (2nd ed.). Thousand Oaks, CA: Sage. Morse, J. M. (1994). Qualitative research: Fact or fantasy? In J. M. Morse (Ed.), Critical issues in qualitative research methods (pp. 1-9). Thousand Oaks, CA: Sage. Murray, J. S. (1999). Methodological triangulation in a study of social support for siblings of children with cancer. Journal of Pediatric Oncology Nursing, 16(4), 194-200. Nakkash, R., Soweid, R. A. A., Nehlawi, M. T., Shediac-Rizkallah, M. C., Hajjar, T. A., & Khogali, M. (2003). The development of a feasible community-specific cardiovascular disease prevention program: Triangulation of methods and sources. Health Education & Behavior, 30(6), 723-739.


394


National Institutes of Health. (1999). Qualitative methods in health research: Opportunities and considerations in application and review. Washington, DC: Office of Behavioral and Social Sciences Research, National Institutes of Health. Paterson, B. L., Thorne, S. E., Canam, C., & Jillings, C. (2001). Meta-study of qualitative health research: A practical guide to meta-analysis and meta-synthesis. Thousand Oaks, CA: Sage. Patton, M. Q. (2002). Qualitative research and evaluation methods (3rd ed.). Thousand Oaks, CA: Sage. Risjord, M. W., Dunbar S. B., & Moloney, M. F. (2002). A new foundation for methodological triangulation. Image: Journal of Nursing Scholarship, 34(3), 269-275. Sharp, H. O. (1943). Photogrammetry (3rd ed.). New York: John Wiley. Sheppard, M., Newstead, S., DiCaccavo, A., & Ryan, K. (2001). Comparative hypothesis assessment and quasi triangulation as process: Knowledge assessment strategies in social work practice. British Journal of Social Work, 31, 863-885. Sim, J., & Sharp, K. (1998). A critical appraisal of the role of triangulation in nursing research. International Journal of Nursing Studies, 35(1/2), 23-31. Smaling, A. (1987). Methodologische objectiviteit en kwalitatief onderzoek [Methodological objectivity and qualitative research]. Lisse, the Netherlands: Swets & Zeitlinger. Thurmond, V. A. (2001). The point of triangulation. Journal of Nursing Scholarship, 33(3), 253-258. Tones, K. (2000). Evaluating health promotion: A tale of three errors. Patient Education and Counseling, 39, 227-236. von dem Knesebeck, O., Joksimovic, L., Badura, B., & Siegrist, J. (2002). Evaluation of a community-level health policy intervention. Health Policy, 61, 111-122. Weiss, C. H. (1998). Evaluation (2nd ed.). Upper Saddle River, NJ: Prentice Hall. Yeung, H. (2003). Practicing new economic geographies: A methodological examination. Annals of the Association of American Geographers, 93(2), 442-462.

Tracy Farmer, PhD is a postdoctoral fellow with the Canadian Heart Health Dissemination Project at McMaster University in Hamilton, Ontario, Canada. Kerry Robinson, PhD candidate, is Project Director of the Canadian Heart Health Dissemination Project at McMaster University in Hamilton, Ontario, Canada. Susan J. Elliott, PhD, is a professor in the School of Geography and Earth Sciences, and Dean of Social Sciences at McMaster University in Hamilton, Ontario, Canada. John Eyles, PhD, is a professor in the School of Geography and Earth Sciences at McMaster University in Hamilton, Ontario, Canada.


Developing and Implementing a Triangulation Protocol for Qualitative ...

Developing and Implementing a Triangulation Protocol for Qualitative ...

Suggest Documents

Triangulation in Social Research: Qualitative and ... - HAREP

Triangulation in Social Research: Qualitative and ... - Federica.EU

A Protocol for Developing and Testing a Methodology for Linking

A protocol for a scoping and qualitative study to ... - BioMedSearch

Developing a Framework for Qualitative Evaluation of

DEVELOPING A WRITING OBSERVATION PROTOCOL 1 Developing ...

DEVELOPING A GEOSPATIAL PROTOCOL FOR CORAL ...

2001 - Developing and Implementing a Facilities Plan for a Freshmen ...

Developing and Implementing Strategies for Internationalisation

Developing and Implementing Online Laboratory for

Evaluation model for developing, implementing, and assessing

Developing and Implementing Models for the ...

Considerations for Developing and Implementing Patient-Centered ...

Developing, Implementing, and Evaluating a ... - Semantic Scholar

Developing, Implementing, and Evaluating a ... - Semantic Scholar

Implementing a Screening and Treatment Protocol for Latent ...

Implementing a protocol for selection and prioritisation ... - (SciELO) SA

Multi-Method Triangulation in a Qualitative Study on Teachers ...

Developing and implementing Entrustable ...

Developing and Implementing a Selection Model for ... - Springer Link

Designing developing and implementing a software tool for scenario

Designing developing and implementing a software tool for scenario ...

A Framework for Developing and Implementing an Online ... - ERIC

Developing and Implementing a Strategy for Technology Deployment