Message from the Editor - Journal Hosting and Publishing - University

Article

Current Mixed Methods Practices in Qualitative Research: A Content Analysis of Leading Journals Mandy M. Archibald, R.N., Ph.D. (cand) Faculty of Nursing, University of Alberta Edmonton, Canada Amanda I. Radil, M.Ed, Ph.D. (cand) Department of Educational Psychology Faculty of Education, University of Alberta Edmonton, Canada Xiaozhou Zhang, M.Ed., Ph.D. (cand) Department of Educational Psychology Faculty of Education, University of Alberta Edmonton, Canada William E. Hanson, Ph.D. Associate Professor, Department of Educational Psychology Faculty of Education, University of Alberta Edmonton, Canada © 2015 Archibald, Radil, Zhang, Hanson. This is an Open Access article distributed under the terms of the Creative Commons‐Attribution-NonCommercial-ShareAlike License 4.0 International (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and if transformed, the resulting work is redistributed under the same or similar license to this one.

121 5

Abstract Mixed methods research (MMR) has become increasingly popular in recent years. Yet, methodological challenges of mixing qualitative and quantitative data remain. Understanding how MMR is approached in qualitative research journals provides insights into lingering mixing issues. In this article, we content analyzed five leading qualitative research journals from 2003 to 2014, which represents the reflective period of MMR. Of the 5,254 articles published, 94, or 1.79%, were mixed methods in nature, comprising 44 theoretically oriented articles and 50 empirical articles. In terms of theoretical articles, five content-based themes were identified: (a) MMR advocacy, (b) philosophy issues, (c) procedural suggestions, (d) practical issues and best practices, and (e) future directions. In terms of empirical articles, 36% used exploratory sequential designs, primarily to develop instruments, and 52% explicitly identified as MMR. None of the studies included MMR questions, and development (21%) and complementarity (14%) were the primary rationales for mixing. In virtually all studies (98%), mixing occurred at the data interpretation stage through some comparison of qualitative and quantitative research. Qualitative data were prioritized in 86% of the studies. Based on these findings, it appears that MMR affects qualitative research most directly by influencing study design and study purpose; however, there is a strong tendency to conduct and publish qualitative and quantitative studies separately. Recommendations for publishing future MMR are discussed.

Keywords: mixed methods research, qualitative research, content analysis, systematic review, integration, multiple research methods, publishing Author Note: Archibald acknowledges the Canadian Child Health Clinician Scientist Program and Women and Children’s Health Research Institute for their support of her doctoral research. The research team acknowledges Btissam El Hassar for her assistance during preliminary journal screening.

121 65

International Journal of Qualitative Methods 2015, 14(2)

Mixed methods research (MMR) has become increasingly popular over the last 25 years (Creswell, 2015). However, collecting qualitative and quantitative data was commonplace in many social sciences throughout the first 60 years of the 20th Century. During the 1980’s, MMR re-emerged as a distinct approach, inducing a second wave of popularity (Creswell, 2015; Guest, 2013; Johnson, Onwuegbuzie & Turner, 2007). This period often is regarded as the beginning of contemporary MMR practice and the third research paradigm (Tashakkori & Teddlie, 2010). After gaining traction through this formative period and the ensuing era of paradigm debate, MMR expanded into a distinct methodology equipped with procedures and nomenclature (Creswell & Plano Clark, 2011). There is now little question of the legitimacy and utility of MMR in various sectors of inquiry (e.g., health and social sciences). Still, challenges remain (Bazeley, 2002). Such challenges exist at multiple levels, including procedural and methodological levels (Archibald, 2015; Bryman, 2006b). For instance, to what extent can qualitative and quantitative research be mixed without violating paradigmatic assumptions of each approach? To what extent should paradigmatic considerations infiltrate the procedural aspects of mixing? Questions of how mixing actually occurs within studies might trigger methodological concerns for certain practitioners while triggering procedural concerns for others. How MMR is understood and defined is fundamental to these considerations. Indeed, defining MMR is a debated topic in the field. Diverse definitions have been forwarded and refined as MMR studies and publications have emerged, and its procedural aspects established (Johnson et al., 2007; Tashakkori & Teddlie, 1998). Some purport that MMR involves the collection and integration of qualitative and quantitative data in a single or a series of studies (Johnson et al., 2007). Within this view, mixing can occur at the level of methods (e.g., Creswell, 2015), methodology (e.g., Tashakkori & Teddlie, 1998), or across disciplines (Bazeley, 2002). Others regard the mixing of diverse qualitative approaches as MMR (e.g., Morse & Niehaus, 2009), whereas a third group notes that the mixing metaphor can be a misnomer (Bazeley & Kemp, 2012; Johnson et al., 2007). Multiple quantitative research approaches in a single study have historically been termed multi-method research (Campbell & Fiske, 1959), whereas Tashakkori and Teddlie (1998) preferred the term mixed model research when integrating beyond the level of methods. In light of this diversity, it is imperative to clarify what is actually being mixed within MMR studies, and how this mixing has occurred (Bazeley, 2002). The variability in MMR definitions mirrors the diversity in its application. How MMR is applied across disciplines, geographic contexts, and within particular research traditions varies extensively. To date, analyses of actual MMR procedures across disciplines are scant (HesseBiber, 2010, p. 15). Recent work has uncovered practices that illuminate state-of-the-science of MMR in the reflective period, circa 2003 to the present (e.g., Bryman, 2006c; Creswell & Plano Clark, 2011; Zhang & Creswell, 2013). Although important strides towards delineating highquality MMR studies have been made, there are additional gains to be made in understanding how MMR is approached in journals aligned specifically with one research tradition (e.g., qualitative). Despite this, there has been no empirical study of MMR practices within qualitatively oriented research journals. As such, we conducted the present study on MMR practices in qualitative research journals. Qualitative and quantitative research can inform one another. For example, qualitatively driven MMR designs, such as those of an exploratory sequential nature, help identify variables for testing within quantitative research designs, contribute to modifying existing tools or to developing new tools or measures, and provide context and meaning to numerical values in a selfreport measure (e.g., Mason, 2006). However, investigating precisely how MMR is approached in qualitative research journals is necessary to provoke reflection upon beneficial and problematic practices that aid or impede the fields’ development and evolution. Understanding the actual practices of data integration, reporting of mixed methods sampling procedures, and using visual 7


displays to illustrate methodological complexities provides insights into the state-of-the-science of MMR. This information is important to investigators seeking publishing venues, to researchers seeking understanding of MMR procedures, and to MMR educators identifying domains where MMR skills-gaps persist and additional training might be required. Such inquiry also might facilitate understanding of pertinent issues to the field of MMR, such as those related to typological designs and integration, which are perennial and generate considerable discussion. Analyzing how MMR is approached in qualitative research journals is also relevant because MMR is increasingly published in non-mixed methods specific journals (i.e., Journal of Mixed Methods Research, International Journal of Multiple Research Approaches). Moreover, MMR is published in a wide array of venues, such as those with a largely qualitative or quantitative research orientation. Trainees and novice researchers who are conducting MMR might look to published articles for guidance, particularly because the formal MMR training might not be available (Archibald, 2015); however, whether such articles are worthy exemplars is not yet established. Therefore, we believe that this work provides the literature with a preliminary description of theoretical and empirical MMR being published in primarily qualitative research journals. Three research questions guided this study and set the stage for the results and discussion: Is MMR published in leading qualitative research journals? If so, how is it approached? How has MMR influenced qualitative research, more broadly, if at all? Additionally, we posed the following eight sub-questions: Do authors of published articles identify them as MMR? How is MMR defined? How are research questions approached? How are research designs approached? At the procedural level, what is the function of integrating? How does ‘mixing’ occur? What recommendations regarding MMR are made? And finally, to what extent are political dimensions discussed? We explored these questions differently, depending on whether articles were theoretical or empirical in nature. We integrated our findings with our interpretations under each associated question heading. Method Researcher-as-Instrument: Our Research Team Our research team consisted of three female doctoral candidates and one male faculty member. The graduate students were in nursing, school and child clinical psychology, and measurement and cognition. The faculty member was in counselling psychology. Two of us self-identified as White European Canadian, one of us self-identified as Chinese, and one other member selfidentified as White European American. All four of us were, at the time of the study, involved actively in various qualitative, quantitative, and MM research studies, as well as a campus-based Mixed Methods Interest Group (MMIG). Generally speaking, as a team, we value each methodology equally and do not adhere to a single ontology or epistemology, although one of us had statistically significantly more experience and training in quantitative methods. Before conducting the study, we discussed our research experiences, values, and biases as part of the MMIG. We also discussed community of practice issues regarding MMR at length. Further, prior to coding published studies, we discussed the rating sheet and potential challenges. Throughout, we employed a mixed methods (MM) approach to reviewing published articles (see Harden & Thomas, 2010). Ours constituted a MM approach in two ways: first, we included studies of various methodologies in the review (e.g., MMR of various typologies; qualitative research studies included in an overarching MM program of research; theoretical writings on MMR). Second, we approached the synthesis of different data sources using separate, and then combined, synthesis methods. Unlike Harden and Thomas (2010), who model multiple syntheses including statistical meta-analysis of quantitative findings, the nature of our review was not amendable to such statistical rendering. However, we still conducted three distinct syntheses, 8


comprising coding and descriptively analyzing domains from MMR empirical studies, conducting thematic analyses of recommendations and definitions provided in empirical and theoretical articles, and, when possible, integrating the two aforementioned syntheses at a comparative level. Sample / Published Articles The six qualitative research journals reviewed were identified through team discussion and comprised the following: International Journal of Qualitative Methods (IJQM), Qualitative Health Research (QHR), Qualitative Research (QR), Qualitative Inquiry (QI), The Qualitative Report (TQR), and Forum: Qualitative Social Research (FQS). These journals were selected based on the team’s awareness of their influence, their broad international audience (e.g., FQS; IJQM; QR), comparatively high Journal Impact Factors (e.g., QHR; QR) and generally, for their non-disciplinary orientations. Although we were aware that QHR is a health-oriented journal, it is not specific to one health discipline and was, therefore, included. A total of 5,254 articles were hand searched and reviewed. Initially, 105 articles were identified as eligible (n = 63 empirical and n = 54 theoretical, conceptual, methodological or empirical). Following primary and secondary screening procedures, 44 theoretical and 50 empirical articles met the inclusion criteria and were included in the content analysis (CA). Collectively, these journals published 94 MMR empirical studies or related theoretical articles between January 2003 and January 2014 (inclusive), representing 1.79% of the entire population of published articles (N = 5,254). A flow diagram of study inclusion is provided in Figure 1. During this screening process, we located other articles relevant to MMR that did not meet our inclusion criteria (e.g., Bryman, 2006b; O’Cathain, Murphy, & Nicholl, 2008). For instance, research syntheses that did not use a MM approach to reviewing the literature, articles where multiple qualitative methods or multiple quantitative approaches were used, and articles where the integration of qualitative and quantitative approaches were not discussed or demonstrated were all excluded from our review.

Figure 1. Flow chart of article inclusion. 9


Procedure We hand-searched journals for keywords of “mixed” and “qualitative and quantitative” published between January 2003 and January 2014, inclusive. Inclusivity was the guiding principle at this stage of article selection. Once potential articles were identified, they were reviewed by a second independent reviewer for preliminary eligibility. To be included at this stage, empirical articles needed to include both a qualitative and quantitative component or self-identify as MMR. Theoretical, conceptual, methodological, or editorial articles that mentioned MMR in any capacity, and book reviews about MMR also were included. This latter class of manuscripts are broadly referred to as theoretical throughout the remainder of the manuscript. Coding. Two reviewers independently reviewed all theoretical and empirical articles using preestablished coding forms to promote as much consistency as possible. For each identified MMR empirical study, we coded data, on a consensus basis, in terms of: research topic/problem, purpose/rationale/philosophy, research questions, design, method fidelity, and implementation, as well as overall mixing, interpretive rigor, and rhetoric/terminology. We coded and thematically analyzed MMR-related editorials, conceptual, and methodological articles in terms of: focus, conceptual position taken, and conclusions drawn/practice recommendations. Any discrepancies were resolved through discussion until 100% consensus was attained. Coding sheets are available from the last author. Results and Discussion Is MMR Published in Leading Qualitative Research Journals? Yes, but very infrequently. In this sample of studies, less than 2% of published articles were MMR. A total of 44 theoretical articles and 50 empirical articles were identified. The vast majority of theoretical articles (82%) were methodological or conceptual in nature. Four editorials (9%) and four reviews (9%) also were identified. The distribution of theoretical and editorial articles was not statistically significant among journals (χ2 = 5.09, p = 0.41); there was no association between journal and article type (theoretical or editorial). TQR was the most common venue, publishing 11 articles (nine theoretical and two reviews). QI published nine articles (eight theoretical and one editorial); FQS published eight articles (six theoretical and two reviews); QHR published seven articles (five theoretical and two editorials); QR published six articles (five theoretical and one editorial) and IJQM published three articles (all theoretical), respectively. Distinct peaks were identified when MMR articles were examined by year of publication. There was an overall increase in MMR published since 2003. Publishing of theoretical articles peaked in 2006 and, five years later, empirical articles peaked (see Figure 2). 20 15 10 5 0 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 Theoretical

Empirical

Figure 2. Mixed methods articles by year of publication. 10


Separate trend analyses were conducted to reveal patterns in the two types of articles (i.e., empirical and theoretical), with the number of articles as dependent variable and year (year; year and year square; year, year square and year cubic) as the independent variable. There was a statistically significant positive linear trend across the years among empirical articles (t = 2.86, p < .05; Cohen’s [1988] d = 2.02), indicating that frequencies for the empirical articles increase in a linear fashion over the time period of analysis (2003-2014). No statistically significant trend in any of the linear (t = -0.22, p = 0.83), quadratic (t = -1.84, p = 0.10) or cubic (t = -0.24, p = .82) patterns were present among theoretical articles, indicating that there is no trend pattern for publishing theoretical articles among these journals. The results of these analyses are presented in Table 1. Table 1 Summary of trend analyses of numbers of articles across the years (N = 22) Empirical articles

Theoretical articles

B

SE.B

R2

B

SE.B

R2

Linear trend

0.79*

0.28

0.44

-0.06

0.25

-0.10

Quadratic trend

-0.06

0.10

0.40

-0.14

0.08

0.12

Cubic trend

-0.05

0.03

0.54

-0.01

0.03

0.01

Note. * represents a statistically significant trend pattern; p < .05 How is MMR Approached in Leading Qualitative Research Journals? Do articles identify as MMR in qualitative research journals? Of the 50 empirical articles included in this analysis, 52% explicitly identified themselves as MMR articles, which represents a statistically non-significant difference from those not identifying as MMR (χ2 = 0.08, p = 0.78); that is, this proportion is not statistically significantly different from what we would have expected to see based on chance alone. “Mixed methods” were included in the titles of 14% of empirical articles, a small, statistically significant proportion compared to those titles not specifying “mixed methods” (χ2 = 25.92, p < 0.001; V = 0.72). The remaining articles incorporated qualitative and quantitative components and the authors spoke to some degree of integration, or used both methods in a manner suggestive of integration. This occurred most often in reference to inferences. Although it is understood that best practices in MMR are evolving, incorporating the words “mixed methods” in manuscript titles offer advantages to the researcher and the reader. Titles that include these terms rapidly alert readers to the research approach used while acting as a workable “placeholder” for a study (Creswell, 2015, p. 12). However, as recently as 2007, it was acknowledged that little guidance exists for authors seeking to publish MMR (Creswell & Tashakkori, 2007). Recent resources, such as Creswell’s (2015) Concise Guide to Mixed Methods Research and Dahlberg, Wittink, and Gallo’s (2010) chapter in the Sage Handbook of Mixed Methods in Social and Behavioural Research, offer guidance in this respect. In order to be included in this CA, MMR needed to be mentioned in theoretical articles. However, during our initial article screening, it became evident that authors varied in the extent to which they discussed MMR. Theoretical articles, therefore, were coded according to their relative focus on MMR. We wanted to capture whether articles had an explicit focus on MMR versus merely mentioning MMR within the context of a larger discussion or, finally, not referring explicitly to MMR at all. Of the MMR articles published in QI, 100% had an exclusive focus on MMR, followed by QHR at 71%, TQR at 64%, QR at 50%, IJQM at 33% and FQS at 13% (χ2 =12.31, p < 0.05; V = 0.31), indicating a statistically significant difference in whether journals focused 11


exclusively on MMR. Similarly, we found the empirical works published in FQS to be among the least likely to self-identify as MMR or be among the least likely to include “mixed methods” in the research title. Perhaps this is a function of journal editorialship. How is MMR defined in qualitative research journals? Defining MMR is a core component of a MMR manuscript (Creswell, 2015). Yet, only 16% of empirical articles provided an explicit MMR definition, despite approximately one half of these works having self-identified as MMR. Compared to the empirical articles, a statistically significantly greater proportion (39%) of theoretical articles provided definitions of MMR (χ2 = 6.90, p < 0.01; V = 0.27). In both empirical and theoretical samples, definitions of MMR most often referred to mixing at the level of methods. This was more common in theoretical articles, where 71% defined MMR at the methods level. Many definitions were simplistic, referring simply to the combination of qualitative and quantitative techniques. Relatively fewer empirical articles (40%) defined MMR at the methods level. Definitions at the methodological level were more commonly seen in empirical works (40%) compared with a 29% proportion in theoretical works. An example of a definition in this category is “research in which the researcher collects, analyzes, and integrates qualitative research and quantitative research to study a problem . . . a methodology that encompasses philosophical assumptions, designs, and methods” (Plano Clark, 2010, p. 428). Another 20% of empirical articles defined MMR by mixing at the level of data. A representation of these definitions is provided in Figure 3.

Methodology

Theoretical

Methods

Empirical

Data

0%

10%

20%

30%

40%

50%

60%

70%

80%

Figure 3. Comparative focus of mixed methods definitions. The discrepancy in how MMR is defined across empirical and theoretical manuscripts was interesting in so much as it mirrors longstanding debates around MMR definitions (Johnson et al., 2007). Similar to Creswell’s (2009) insight that probing of definitions seemed to have ceased in conference reports, MM definitions were not a dominant topic of inquiry in our review. Also, we considered that the longstanding history of MM, particularly within the discipline of sociology (Gilbert, 2006; Greene, Caracelli, & Graham, 1989; Plano Clark, 2010) might have contributed to a persistent regard for MMR as a combination of methods, techniques, or procedures, rather than as a methodology. Alternatively, a methods-based approach to MMR might be regarded as being particularly useful as an introduction to the field because it provides a more concrete and accessible entry-point than a methodological orientation (Creswell, 2015). How are research questions approached in qualitative research journals? Although all empirical articles included in this CA had purpose statements, authors were statistically significantly less likely to identify a qualitative (10%), qualitative and quantitative (6%), 12


quantitative alone (4%), or MM (0%) research question (χ2 = 81.04, p < 0.001; V = 0.74). Overall, research questions were not explicitly stated in 80% (n = 40) of empirical studies. The practice of including research questions in published research reports might vary across research traditions, disciplines, and over time, and might not be advocated uniformly by leading MM researchers. However, their inclusion in published reports would illuminate some of the intricacies of formulating questions in this complex domain. Leaders in the field have historically overlooked the procedures and implications of devising a well-formed MMR question (Onwuegbuzie & Leech, 2006; Plano Clark & Badiee, 2010). Recent attention to developing and identifying MMR questions (e.g., Creswell, 2015; Plano Clark & Badiee, 2010) has provided excellent guidance in this regard. Including how MMR questions in published reports would advantage the reader by acting as a signpost, providing important hints and direction regarding the likelihood of a particular MMR design, the associated sequence and timing of data collection, and probable types of data analysis and integration procedures utilized (Creswell, 2015; Onwuegbuzie & Leech, 2006). Researchers are encouraged to attend closely to MMR questions and to familiarize themselves with available resources. How are MMR designs approached in qualitative research journals? Of the empirical articles, approximately one half (48%) explicitly identified a MMR design typology, which is a statistically non-significant difference from those not identifying a MMR design typology (χ 2 = 0.08, p = 0.78). Sequential designs of an exploratory or explanatory nature dominated this sample, representing 66% of all designs. Of these, 36% were exploratory sequential and 30% were explanatory sequential. The majority (67%) of the 18 exploratory sequential designs were instrument development models. Additional typologies were identified and included convergent (14%), multi-phase (14%) and embedded (6%) designs. The proportions of each design were statistically significantly different (χ2 = 15.6, p < 0.01; V = 0.28). The overwhelming majority (92%) of the 26 studies that self-identified as MMR explicitly utilized a MMR design typology. This high occurrence might reflect the proliferation of such typologies in the MMR literature (e.g., Creswell & Plano Clark, 2011; Greene et al., 1989; Morse & Niehaus, 2009). Typologies, as tools for research design and pedagogy, might offer the additional advantages of a common language, structure, and legitimization to the field of MMR (Guest, 2013). Yet, typologies also might obscure temporal order and render the point of integration unclear to the reader, depending on the point of emphasis (e.g., data collection or analysis) (Guest, 2013). Although we did encounter a lack of clarity related to the point of integration and temporal order, we were also challenged at times to understand the context of particular manuscripts, because 46% of the articles were identified as being part of a larger study, or program of study. This typically meant that quantitative results from the study were reviewed briefly and then qualitative findings were explored more extensively. When MM studies are published separately, it is more difficult to assess, or even identify, the actual point of integration and, in turn, the study’s typology. Our impression of this sample of studies was that, generally, researchers emphasized a fixed approach to study design. Only 6% of the empirical works appeared emergent in nature and that, overwhelmingly, manuscripts conveyed a message of design fidelity or adherence to the initial research plan. We were at times unclear whether this tendency reflected a social desirability for cleanliness in research reporting, considering that the process and outcomes of MMR are not always predictable (Bryman, 2006a), and qualitatively driven MMR may be particularly constructivist and organic in nature. What is the function of integrating in qualitative research journals? As Bryman (2006b) identified, “dimensions of typologies draw attention to different aspects of multi-strategy 13


research” (p. 98), for instance, the function of integrating qualitative and quantitative components. To understand the function of integrating, we explored the rationales presented by study authors. Using Greene et al.’s (1989) approach, we identified primary and secondary rationales for MMR when applicable. The rationales possible through this approach comprised: (a) triangulation, (b) complementarity, (c) development, (d) initiation, and (e) expansion (Greene et al., 1989). Approximately one half (52%) of the empirical articles provided a rationale for using and integrating qualitative and quantitative research methods. However, these were often implicit or embedded within the study’s purpose. The rationales provided are displayed in order of occurrence by primary rationale, in Table 2. Table 2 Rationales for Integrating Qualitative and Quantitative Research Category

Primary Rationale (%)

Secondary Rationale (%)

Not stated

48%

81%

Development

21%

5%

Complementarity

14%

8%

Triangulation

8%

3%

Expansion

6%

3%

Initiation

3%

0%

Development was the most common rationale provided, evidenced in 21% of empirical studies, followed by complementarity at 14%. The high occurrence of development as a rationale corresponded with the large volume of instrument development models. Triangulation was more common than expansion and initiation. Because previous analyses have reported that initiation is uncommon, this was somewhat of an expected finding. We did find the low occurrence of expansion to be an unexpected result, particularly considering that 66% of our study sample reflected sequential designs using distinct methods for separate, but related, inquiry components. This finding prompted us to question whether there was a potential misalignment between the stated rationales and the actual conduct of mixing qualitative and quantitative research. We were less concerned with studies claiming development as the study rationale; indeed, there was some internal consistency between this finding and the occurrence of instrument development models. Further, as Bryman (2006b) previously illustrated, “when ‘instrument development’ and ‘sampling’ are the rationales, they are nearly always used in this way” (p. 110). Other rationales, particularly triangulation, complementarity, and expansion, might be less clear because overlap exists between their operational components. This might lead to ambiguity in interpreting these functions—one need only refer to the extensive literature on the ambiguity surrounding triangulation to substantiate this insight (e.g., Denzin, 2012; Wolf, 2010). Although it is beyond the scope of this study, future work that examines the correspondence between stated rationales, such as triangulation and expansion, with actual integration procedures, would contribute further insights to this domain. How does “mixing” occur in qualitative research journals? A dominant sub-question guiding our CA was “what is being mixed in these MMR studies, and how does this mixing occur?” All 14


empirical articles were included because they mixed at the level of methods; that is, all articles integrated qualitative and quantitative data in some capacity. However, the tendency for researchers to present qualitative and quantitative data in different manuscripts hindered our ability to assess the extent to which mixing and integration actually occurred. Although others may have elected to exclude such studies, as did Bryman (2006b) in his CA, we believe that this captured a pertinent trend and, therefore, a relevant finding in the publishing practices of MMR in qualitatively oriented journals. Given this, we included studies that demonstrated linking of qualitative or quantitative research components, or claimed integration, however superficially. We did so despite our diminished ability to ascertain the integrity of the researchers’ integration approaches. In other words, we took the researchers’ integration claims at face value. Because our CA focuses on how MMR is approached in qualitative research journals, it is perhaps not surprising that the included manuscripts favoured publication of the qualitative aspects of MMR. Indeed, drawing integrated inferences, or referring to how qualitative and quantitative components related, was the most common level of mixing, occurring in 98% of studies. Publishing practices might have influenced how authors approached integration because qualitative and quantitative components often were contained in separate publications, particularly because of the dominance of sequential designs. This is aligned with the historical focus on component designs and the relative lack of quality examples illustrating fully integrated designs in the MMR literature (Bazeley, 2002; Bazeley & Kemp, 2012). The emphasis on component designs has prompted some to question whether “these really do constitute a mixed methods study or rather, are two separate studies which happen to be about the same topic” (Bazeley, 2002, p. 3). We also noted at times, studies of other design typologies (e.g., embedded) that self-identified as MMR but demonstrated the same disconnected publishing practices. For example, in their study of burnout and coping of occupational therapists, Gupta, Paterson, von Zweck, and Lysaght (2012) utilized a “mixed methods approach to gain a comprehensive view of stressors” (p. 4). The authors used a concurrent embedded model with a predominantly quantitative focus and reported their quantitative and qualitative results in separate manuscripts. Focus group data were analysed using a hermeneutic approach; the analysis and findings of which were included in the present review. Although the authors use a comprehensiveness rationale for employing MMR, how qualitative and quantitative components contributed to this objective was insufficiently described. Quantitative findings are “presented elsewhere” (p. 6) and data are not presented in a manner that facilitates comparison (e.g., side-by-side). In fact, only one reference is made to the accompanying quantitative research manuscript. Mixing at the level of data, including crossover data transformation procedures was evident in 22% of studies. This most often occurred within convergent designs. Exploratory designs (28%) occasionally mixed their data, typically through transforming qualitative data into quantitative data. Integrating at the level of methodology also was fairly common, evident in 10 studies. An interesting example of this approach included the interface of phenomenology within a controlled clinical trial (Bishop, Jacobson, Shaw, & Kaptchuk. 2012). The weighting of qualitative and quantitative study components is another relevant dimension of MMR (Bryman, 2006b). Not surprisingly, the vast majority of studies (86%) prioritized qualitative data. This occurred across research designs. Only 10% of studies provided equal weighting to quantitative and qualitative data, as evidenced by equal credence being paid to each data strand in the article as well as both data strands influencing inferences. The remaining 4% of studies provided inadequate reporting details to ascertain weighting or, in one case, prioritized quantitative data. The proportions among these three weighting types were statistically significantly different (χ2 = 100.08, p < 0.001; V = 0.82), indicating a contrast between the expected proportions and what we observed.

15


Where integration occurred (i.e., the point of integration) varied extensively across studies. Almost one quarter of studies (22%) integrated by way of developing an instrument or data collection tool, meaning that the research findings of one strand contributed to creating a new measure or improving an existing one. The extent to which qualitative findings were discussed in relation to the tools development varied but generally was well described. For instance, Lutz, Kneipp, and Means (2009) reported on the first of three phases of a community-based participatory research study designed to develop a culturally appropriate health-screening questionnaire. They conducted three rounds of focus groups that iteratively informed the development of constructs and process components related to their screening measure, and described this process in detail. Subsequent stages of the study (i.e., assessing the questionnaire’s acceptability through a survey methodology and eventual use in a randomized controlled trial) were outlined as successive stages of study. Interestingly, the authors did not identify this as a MM program of study. Data collection was a common point of integration, present in 14% of articles, a finding consistent with previous reports (e.g., Hesse-Biber, 2010). For instance, Capezza (2003) examined the cultural-psychological foundations of violence and non-violence by collecting questionnaire, narrative, and observational data simultaneously. Integrating during data analysis was common and documented in 30% of studies. At this level, data were cross-compared (for example) in relation to convergent or divergent findings. Overall, convergent designs tended to demonstrate integration earlier in the research process and 71% of these designs began integrating at the point of data collection. Given the abundance of sequential designs and the presence of multiphase designs, it is not surprising that 86% of studies collected data in a sequential manner. As a result, most studies in this sample used a connected approach to data analysis wherein one data strand leads to the other. Concurrent data collection was less common and was noted in 14% of studies, all of which were convergent designs (χ2 = 49.48, p < 0.001; V = 0.70); these differences were statistically significant. Merged (12%) and embedded (4%) approaches to data analysis were infrequent in this sample, consistent with the high representation of sequential approaches to data analysis observed. Merged data analysis occurred within convergent designs only. In one article, the analysis procedures were impossible to decipher because of insufficient reporting of the quantitative aspect of the study. Which recommendations about MMR are made in qualitative research journals? The overwhelming majority (82%) of empirical articles did not provide explicit recommendations around MMR. Another 10% of empirical articles provided recommendations to some extent— through statements such as “considering using MMR; it is possible to integrate methods better.” However, such statements often were more from a MM advocacy perspective. Clear recommendations were provided in only 5% of empirical articles; Plano Clark (2010) provides a strong example when she proposes five suggestions for those “interested in proposing a mixed methods study and for funding representatives interested in facilitating mixed methods proposals” (p. 437). Other authors made recommendations explicitly aligned with previous literature. For instance, Medlinger and Crickel (2008) identify that “pragmatism and available resources should guide the choice of research strategy to fit the research question at hand” (p. 290) and relate this to the potential utility of their double helix model for MMR. In contrast to the empirical studies, recommendations were made in a greater proportion (45%) of theoretical articles and many authors forwarded multiple recommendations. Five themes were identified through open coding and comprised: (a) advocacy (i.e., advocating for the benefits of MMR or related approaches); (b) foundational principles (i.e., those aspects or beliefs necessary to or underlying MMR); (c) procedures (i.e., recommendations related to the “how-to” or technical aspects of MMR); (d) reflexivity (i.e., reflecting on some aspect of MMR or practice); 16


or (e) future directions (i.e., possibilities for enhancing MM research or practice). MMR procedures were the most common of the five themes identified through the analysis of recommendations in theoretical articles, forwarded by 11 authors. Four subcategories were identified under the MMR procedure theme and included: (a) MMR designs; (b) research problems or questions; (c) language or reporting; and (d) analysis procedures. Multiple recommendations were common within this theme. Of these, language and reporting were most common, present in five recommendations. Design related recommendations and those related to research problems or questions were present in four recommendations each. Three recommendations were specific to data analysis, only one of which pertained to data integration. For instance, Moran-Ellis et al. (2006) make a number of arguments and recommendations pertaining to the meaning of integration, terminology, and justifications. In total, an additional five recommendations were made speaking to sampling, team research, validity, utility, and clarity in understanding, respectively. In the theoretical articles, 10 recommendations were made related to foundational principles of MMR. Most often, these related to theoretical or methodological considerations (n = 6 recommendations). For example, Shaw (2003) identified that epistemological integrity is necessary for meaningful and well-conducted research. Similarly, Miller and Fredericks (2006) recommended that, “methodological, structural, and substantive concerns must be recognized and addressed before MMR can be integrated into evaluation research” (p. 574). Ensuring that researchers have adequate training or skills to conduct MMR were present in three recommendations. Awareness of research processes or current issues in the field of MMR and ethical considerations were present in three and two recommendations, respectively. A visual representation of these recommendations is provided in Figure 4.

Figure 4. Pie chart illustrating the recommendations in theoretical articles Note. n = 16 for smaller pie chart (Procedure Recommendations) as authors/articles (n = 11 for Procedures) made multiple recommendations in one article 17


To what extent are political dimensions of MMR discussed in qualitative research journals? Political dimensions, specifically related to funding or deconstructing MMR, were present in 25% of theoretical articles in varying degrees. Inclusivity guided our approach to coding these articles; that is, articles that mentioned ethics were coded as political (see, for e.g., Hesse-Biber, 2012). The overwhelming majority (91%) of these articles focused on deconstructing, which we defined as critically examining assumptions related to an aspect of MM research or practice. For example, the work of Moran-Ellis et al. (2006) was classified as deconstructing because they critically examined and differentiated the processes and outcomes of triangulation and integration procedures in a manner that challenges mainstream practices. Similarly, Morse (2009) deconstructed the very notion of MMR as consisting exclusively of quantitative and qualitative research approaches, purporting that multiple qualitative or multiple quantitative research methods also qualify as MMR. Ethical considerations of MMR were raised in 9% of theoretical articles. Ethics were discussed in diverse manners in these sources. For instance, Mertens (2010) refers to the ethical dimensions of social justice and human rights in MMR, discussing how the transformative paradigm has enabled her to clarify the ethical dimensions of her practice. Conversely, Christ (2014) discusses how ethical considerations invariably infiltrate traditional randomized controlled trials (RCT), thereby advocating for the inclusion of MMR approaches into the RCT design. When Creswell (2009) provided a map of the MMR terrain, he anticipated increased attention to the politicized dimensions of MMR. Our findings suggest that the politicized dimensions, particularly deconstructing MMR, are indeed a focal aspect of inquiry into politicization. Although subtle, we were able to detect an approximately 30% increase in the proportion of articles attending to deconstructing MMR (and to a lesser degree, ethics) beginning in 2009. Indeed, 2009/2010 seem to be a tipping point for theoretical work relating to deconstructing MMR and the ethics of MMR, with seven articles about these topics published over these 2 years; five were published between 2003 and 2008 and four were published between 2011 and 2014. However, quality articles attending to these concepts also were noted prior to this point. Useful examples of such articles include Onwuegbuzie and Leech’s (2004) exploration of the meanings of significance in relation to MMR, and Moran-Ellis et al.’s (2006) critique of integration and mixing in the MMR context. How Has MMR Influenced Qualitative Research, More Broadly, If At All? This CA has highlighted numerous ways in which MMR influences qualitative research, and also stimulated further hypotheses in this regard. We question whether the rise in popularity of the generic qualitative research method and the tendency to mix within and between qualitative research methodologies coincided with the re-emergence and increasing popularity of MMR as a methodological approach (e.g., Scott, Archibald, Pullishy, & Chambers, in press). Debates surrounding paradigmatic compatibility also exist within the qualitative domain, enabling alternatives to methodological muddling, methodological purity, and divisive methodological distinctions (Kahlke, 2014). How MMR has shaped the questions asked by qualitative researchers is also a point of reflection. For instance, questions about the participant experience during clinical trials; exploring the meanings of quantitative measures, and using qualitative data to inform the development of quantitative measures in instrument development models were common in this CA. Such questions might be inspired by MMR. We believe that the growing popularity and acceptance of MMR as a legitimate methodology has influenced researchers to re-examine the previously distinct divide between qualitative and quantitative research. This re-examining has foreseeably softened some of the claims previously made about research methodologies—for instance, in notions that the researcher can be bracketed out of the research process; claims related to absolute objectivity; or that numbers and qualitative

18


research cannot coexist in a qualitative research study. For instance, authors frequently discuss majorities and proportions, and often provide numerical representations of these through counts or percentages (Maxwell, 2010). Regardless of one’s involvement in MMR, the role of numbers and proportions, while contested at times, can be meaningful to the practice of qualitative research (Sandelowski, Voils, & Knaft, 2009). Concluding Remarks and Recommendations In this article, we explored how MMR has been approached in six qualitatively oriented journals. We identified predominant publishing and research practices and examined several pressing issues, such as the extent to which MMR studies are published in qualitative research journals; the types of studies published; how qualitative and quantitative data strands are used and mixed; and how MMR is discussed by researchers, editors, and theorists publishing in leading qualitatively oriented journals. We suggested that the publishing trends of empirical manuscripts favor publishing component pieces of designs as opposed to the design as a whole, and argued that this practice can obscure assessment of integration. The common practice of publishing qualitative and quantitative results as separate manuscripts presented some unique challenges to assessing the meaningfulness and extent of integration. The dominant tendency was to publish qualitative study components in qualitative research journals and quantitative findings in quantitative research journals, consequently making little use of sideby-side display or other approaches to facilitate mixing. We recognize there is not one formulaic path to integration in MMR (Bazeley & Kemp, 2012). The extent of integration can be influenced by the sequence of data collection, the overall MMR design, the type of data collected, and “can occur throughout or at different stages of the research process” (Wooley, 2009, p. 7). Regardless of these nuances, mutual illumination or additive benefit is a necessary outcome of bringing together qualitative and quantitative study components in a meaningful way (Wooley, 2009). Although integration takes different forms, we found that these publishing practices presented a challenge to assessing, and potentially to conducting, meaningful integration. A number of factors likely influence this practice. First, carry-over from the paradigm wars has, for some, led to a preference towards maintaining the epistemological integrity of seemingly separate research strands by keeping research components independent and integrating when drawing conclusions (Bazeley, 2009). Second, the space limitations of publishing are challenging for MMR; dual publications alleviate the need for absolute parsimony and enable a more thorough description of procedures and analysis for each strand than might be possible if published together. Third, the pressure to publish that is pervasive across academic circles should not be overlooked (Schraagen, 2012). Dividing findings for publication into journals representing each respective research tradition might be seen as desirable because it enables a higher number of research outputs. Finally, despite the emergence of excellent resources pertaining to integration, a skills barrier still might exist. Important lessons for research, practice, and teaching can be gleaned from these findings. Sequential designs were most frequently encountered and were often reported in separate publications. This was particularly true instrument development research; many of these manuscripts only presented the qualitative portion of their work while making reference to the quantitative work as opposed to including the final product (the instrument) or validation information with the qualitative phase in one manuscript. Publishing practices demonstrating a preference for conveying results in separate manuscripts and authors inadequately demonstrating methods of integration were both problematic. Reporting findings in separate manuscripts makes identifying the point of integration between strands difficult and additionally, might encourage less rigorous forms of integration than otherwise would be expected. In response to these challenges, researchers who integrate qualitative and quantitative data need 19


to be attentive to how integration is represented in their manuscripts, irrespective of whether the substantive qualitative and quantitative research components are published separately. Information about both qualitative and quantitative strands and strong evidence of integration need to be provided to qualitative research journals editors if they are to publish MMR in these arenas. Signposts of MMR, such as using mixed methods in article titles, including a MM purpose statement, and having clear rationale(s) for mixing should be included regardless of publication venue. Moreover, instructors of MMR and those mentoring trainees conducting MMR need to emphasize integration as a hallmark of MMR and find creative ways, such as the use of visual diagrams, to illustrate how and where integration is occurring. This is especially relevant because empirical articles are increasingly published in qualitatively oriented journals, and because sequential designs, especially those of an explanatory nature, are particularly popular with trainees and novice researchers (Creswell, 2015). Novice MM researchers are encouraged to work within specific, well-established designs, or typologies, which serve as road maps for data collection, analysis, integration, and interpretation. Becoming familiar with design typologies and learning to integrate qualitative and quantitative data at various points in the study increases one’s flexibility and opens the door to more advanced MMR designs. Furthermore, given the popularity of sequential designs, it is prudent that qualitative researchers become particularly familiar with exploratory types of designs and model variants. It is advisable that researchers at all levels aim for design coherence, particularly, how the research purpose and rationale for mixing influence design, and what extent of integration is enabled through the design and research procedures. This coherence should be represented through the appropriate use of metaphors to describe variations of mixing (Bazeley & Kemp, 2012). Adopting the common variations of merging, connecting, and embedding can accurately reflect the points of integration within a given study. There are a few limitations to this CA. First, we recognize that the extent to which a given study’s rated elements represent overarching categories matters greatly. In our study, we rated specific, well-defined MMR characteristics (e.g., study purpose, research questions) to answer the research questions and took a consensus-based rating approach, whereby we fully discussed and reconciled any discrepancies to address potential reliability and validity issues. Other informative methods to enhance content validity, such as using a Table of Specifications (ToS), may have been adopted (Newman, Lim & Pineda, 2013). Indeed, using a ToS might be a fruitful avenue for future research particularly when broader conceptual study categories are rated (e.g., methodological coherence or quality). Second, we faced challenges during article coding. These were compounded by different publishing practices of the journals that we reviewed. The tendency to publish qualitative and quantitative research separately, the often deficient reporting of related quantitative research, and the inconsistent presentation and location of important information across journals contributed to these challenges. Third, we included articles published between 2003 and 2014. As such, a similar CA of previously published articles might reveal important trends not captured in the present research. The six qualitative research journals provided a broad sample of nondisciplinary specific articles; however, an alternative sample of qualitative research journals might have yielded different results. Including leading quantitative journals also might have yielded different findings; would publishing practices be similar in these journals and would similar designs be utilized? Future research in this arena might provide fruitful data for cross comparison and further understandings of MMR practices.

20


Numerous areas for future research were identified through this work, enabling systematic study of emerging MMR practices over the past decade. For example, we have begun examining how stated rationales for mixing corresponded (or not) with actual integration procedures, and whether features of visual displays differ by research design. These works are in progress and contribute to mapping the field of MMR during this reflexive era of practice (Creswell, 2009; Creswell & Plano Clark, 2011). Overall, findings from this CA contribute to longstanding efforts to delineate data mixing procedures (Bryman, 2007; Tashakkori & Teddlie, 2010), to illuminate influential practices in qualitative research, quantitative research, and MMR traditions, and to assist in understanding MMR practices and perspectives across diverse communities.

21


References Archibald, M. (2015). Investigator triangulation: A collaborative strategy with potential for mixed methods research? Journal of Mixed Methods Research. Advance online publication. doi:10.1177/1558689815570092. Bazeley, P. (2002, April). Issues in mixing qualitative and quantitative approaches to research. Paper presented at the 1st international conference: qualitative research in Marketing and Management, University of Economics and Business Administration, Vienna, Austria. Bazeley, P. (2009). Editorial: Integrating data analyses in mixed methods research. Journal of Mixed Methods Research, 3, 203-207. doi:10.1177/1558689809334443 Bazeley, P., & Kemp, L. (2012). Mosaics, Triangles, and DNA: Metaphors for Integrated Analysis in Mixed Methods Research. Journal of Mixed Methods Research, 6, 55-72. doi:10.1177/1558689811419514 Bishop, F., Jacobson, E., Shaw, J., & Kaptchuk, T. (2012). Participant’s experiences of being debriefed to placebo allocation in a clinical trial. Qualitative Health Research, 22, 11381149. doi:10.1177/1049732312448544 Bryman, A. (2006a). Editorial. Qualitative Research, 6, 1-7. doi:10.1177/1468794106058865 Bryman, A. (2006b). Integrating quantitative and qualitative research: How is it done? Qualitative Research, 6, 97-113. doi:10.1177/1468794106058877 Bryman, A. (2006c). Paradigm peace and the implications for quality. International Journal of Social Research Methodology, 9, 111-126. doi:10.1080/13645570600595280 Bryman, A. (2007). Barriers to integrating quantitative and qualitative research. Journal of Mixed Methods Research, 1, 8-22. doi:10.1177/2345678906290531 Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105. doi:10.1037/h0046016 Capezza, N. (2003). The cultural-psychological foundations for violence and non-violence. An empirical study. Forum: Qualitative Social Research, 4(2), 1-24. Retrieved from http://www.qualitative-research.net/fqs/. Christ, T. (2014). Scientific-based research and randomized controlled trials, the “gold” standard? Alternative paradigms and mixed methodologies. Qualitative Inquiry, 20, 72-80. doi:10.1177/1077800413508523 Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum. Creswell, J. W. (2009). Editorial: Mapping the field of mixed methods research. Journal of Mixed Methods Research, 3, 95-108. doi:10.1177/1558689808330883 Creswell, J. W. (2015). A concise introduction to mixed methods research. Thousand Oaks, CA: Sage.

22


Creswell, J. W., & Plano Clark, V. L. (2011). Designing and conducting mixed methods research (2nd ed.). Thousand Oaks, CA: Sage. Creswell, J. W., & Tashakkori, A. (2007). Editorial: Developing publishable mixed methods manuscripts. Journal of Mixed Methods Research, 1, 107-111. doi:10.1177/1558689806298644 Dahlberg, B., Wittink, M., & Gallo, J. (2010). Funding and publishing integrated studies: Writing effective mixed methods manuscripts and grant proposals. In A. Tashakkori & C. Teddlie (2nd ed.), SAGE handbook of mixed methods in social & behavioral research (pp. 775802). Thousand Oaks, CA: Sage. Denzin, N. K. (2012). Triangulation 2.0. Journal of Mixed Methods Research, 6, 80-88. doi:10.1177/1558689812437186 Gilbert, T. (2006). Mixed methods and mixed methodologies: The practical, the technical, and the political. Journal of Research in Nursing, 11, 205-217. doi:10.1177/1744987106064634 Greene, J. C., Caracelli, V. J., & Graham, W. F. (1989). Toward a conceptual framework for mixed-method evaluation design. Educational Evaluation and Policy Analysis, 11, 255274. Retrieved from http://www.jstor.org/stable/1163620 Guest, G. (2013). Describing mixed methods research: An alternative to typologies. Journal of Mixed Methods Research, 7, 141-150. doi:10.1177/1558689812461179 Gupta, S., Paterson, M., von Zweck, C., & Lysaght, R. (2012). Using hermeneutics to understand burnout and coping strategies utilized by occupational therapists. The Qualitative Report, 17(105), 1-25. Retrieved from http://www.nova.edu/ssss/QR/QR17/sgupta.pdf Harden, A., & Thomas, J. (2010). Mixed methods and systematic reviews: Examples and emerging issues. In A. Tashakkori & C. Teddlie (2nd ed.), SAGE handbook of mixed methods in social and behavioral research (pp. 749-774). Thousand Oaks, CA: Sage. Hesse-Biber, S. (2010) Mixed methods research: Merging theory with practice. New York, NY: Guilford Press. Hesse-Biber, S. (2012). Weaving a multimethodology and mixed methods praxis into randomized control trials to enhance credibility. Qualitative Inquiry, 18, 876-889. doi:10.1177/1077800412456964 Johnson, R. B., Onwuegbuzie, A. J., & Turner, L. A. (2007) Toward a definition of mixed methods research. Journal of Mixed Methods Research, 1, 112-133. doi:10.1177/1558689806298224 Kahlke, R. (2014). Generic qualitative approaches: Pitfalls and benefits of methodological mixology. International Journal of Qualitative Methods, 13, 37-52. Retrieved from http://ejournals.library.ualberta.ca.login.ezproxy.library.ualberta.ca/index.php/IJQM/artic le/view/19590/16141 Lutz, B., Kneipp, S., & Means, P. (2009). Developing a health screening questionnaire for women in welfare transition programs in the United States. Qualitative Health Research, 19, 105-115. doi:10.1177/1049732308327347

23


Mason, J. (2006). Mixing methods in a qualitatively driven way. Qualitative Research, 6, 9-25. doi:10.1177/1468794106058866 Maxwell, J. A. (2010). Using numbers in qualitative research. Qualitative Inquiry, 16, 475-482. doi:10.1177/1077800410364740 Medlinger, S., & Cwikel, J. (2008). Spiraling between qualitative and quantitative data on women’s health behaviours: A double helix model for mixed methods. Qualitative Health Research, 18, 280-293. doi:10.1177/1049732307312392 Mertens, D. (2010). Transformative mixed methods research. Qualitative Inquiry, 16, 469-474. doi:10.1177/1077800410364612 Moran-Ellis, J., Alexander, V. D., Cronin, A., Dickinson, M., Fielding, J., Sleney, J., & Thomas, H. (2006). Triangulation and integration: Processes, claims and implications. Qualitative Research, 6, 45-59. doi:10.1177/1468794106058870 Morse, J. (2009). Editorial: Mixing qualitative methods. Qualitative Health Research. 19, 15231524. doi:10.1177/1049732309349360 Morse, J., & Niehaus, L. (2009). Mixed method design: Principles and procedures. WalnutCreek, CA: Left Coast Press. Newman, I., Lim, J., & Pineda, F. (2013). Content validity using a mixed methods approach: Its application and development through the use of a table of specifications methodology. Journal of Mixed Methods Research, 7, 243-260. doi:10.1177/1558689813476922 O’Cathain, A., Murphy, E., & Nicholl, J. (2008). Multidisciplinary, interdisciplinary, or dysfunctional? Team working in mixed-methods research. Qualitative Health Research, 18, 1574-1585. doi:10.1177/1049732308325535 Onwuegbuzie, A. J., & Leech, N. L. (2004). Enhancing the interpretation of “significant” findings: The role of mixed methods research. The Qualitative Report, 9, 770-792. Retrieved from http://www.nova.edu/ssss/QR/QR9-4/onwuegbuzie.pdf Onwuegbuzie, A. J., & Leech, N. L. (2006). Linking research questions to mixed methods data analysis procedures. The Qualitative Report, 11, 474-498. Retrieved from http//www.nova.edu/ssss/QR/QR11-3/onwuegbuzie.pdf Plano Clark, V. L. (2010). The adoption and practice of mixed methods: U.S. trends in federally funded health-related research. Qualitative Inquiry, 16, 428-440. doi:10.1177/1077800410364609 Plano Clark, V. L., & Badiee, M. (2010). Research questions in mixed methods research. In A. Tashakkori & C. Teddlie (2nd ed.), SAGE handbook of mixed methods in social and behavioral research (pp. 275-304). Thousand Oaks, CA: Sage. Sandelowski, M., Voils, C., & Knafl, G. (2009). On quantitizing. Journal of Mixed Method Research, 3, 208-222. doi:10.1177/1558689809334210 Schraagen, J. M. (2012). To publish or not to publish: A systems analysis of longitudinal trends in publishing strategies of a human factors research organization. Theoretical Issues in Ergonomics Science, 14, 499-530. doi:10.1080/1463922x.2012.656334 24


Scott, S. D., Archibald, M., Pullishy, L., & Chambers, T. (in press). Qualitative evidence in pediatrics. In K. Olson, R. Young, I. Z. Schultz (Eds.), Handbook of qualitative health research for evidence-based practice. New York, NY: Springer. Shaw, I. (2003). Qualitative research and outcomes in health, social work and education. Qualitative Research, 3, 57-77. doi:10.1177/146879410300300103 Tashakkori, A., & Teddlie, C. (1998). Mixed methodology: Combining qualitative and quantitative approaches. Thousand Oaks, CA; Sage. Tashakkori, A., & Teddlie, C. (2010). SAGE handbook of mixed methods in social and behavioral research (2nd ed.). Thousand Oaks, CA: Sage. Wolf, F. (2010). Enlightened eclecticism or hazardous hotchpotch? Mixed methods and triangulation strategies in comparative public policy research. Journal of Mixed Methods Research, 4, 144-167. doi:10.1177/1558689810364987 Wooley, C. (2009). Meeting the mixed methods challenge of integration in a sociological study of structure and agency. Journal of Mixed Methods Research, 3, 7-25. doi:10.1177/1558689808325774 Zhang, W., & Creswell, J. (2013). The use of “mixing” procedure of mixed methods in health services research. Medical Care, 51, 51-57. doi:10.1097/MLR.0b013e31824642fd

25


Appendix A References for studies included in this review. N = 94 Ali, A. (2005). Using the Delphi technique to search for empirical measures of local planning agency power. The Qualitative Report, 10, 718-744. Retrieved from http://www.nova.edu/ssss/QR/QR10-4/ali.pdf Alquati Bisol, C., Sperb, T. M., & Moreno-Black, G. (2008). Focus groups with deaf and hearing youths in Brazil: Improving a questionnaire on sexual behavior and HIV/AIDS. Qualitative Health Research, 18, 565-578. doi:10.1177/1049732307307868 Anaf, S., & Sheppard, L. (2007). Mixing research methods in health professional degrees: Thoughts for undergraduate students and supervisors. The Qualitative Report, 12, 184192. Retrieved from http://www.nova.edu/ssss/QR/QR12-2/anaf.pdf Aydin, S. (2010). A qualitative research on portfolio keeping in English as a foreign language writing. The Qualitative Report, 15, 475-488. Retrieved from http://www.nova.edu/ssss/QR/QR15-3/aydin.pdf Ballard, K., & Bates, A. (2008). Making a connection between student achievement, teacher accountability and quality classroom instruction. The Qualitative Report, 13, 560-580. Retrieved from http://www.nova.edu/ssss/QR/QR13-4/ballard.pdf Bhattacharyya, E., Patil, A., & Sargunan, R. A. (2010). Methodology in seeking stakeholder perceptions of effective technical oral presentations: An exploratory pilot study. The Qualitative Report, 15, 1549-1568. Retrieved from http://www.nova.edu/ssss/QR/QR156/bhattacharyya.pdf Bishop, F., Jacobson, E., Shaw, J., & Kaptchuk, T. (2012). Participant’s experiences of being debriefed to placebo allocation in a clinical trial. Qualitative Health Research, 22, 11381149. doi:10.1177/1049732312448544 Brander, R.A., Paterson, M., & Chang, Y.E. (2012). Fostering change in organizational culture using a critical ethnographic approach. The Qualitative Report, 17(90), 1-27. Retrieved from http://www.nova.edu/ssss/QR/QR17/brander.pdf Bryman, A. (2006a). Editorial. Qualitative Research, 6, 1-7. doi:10.1177/1468794106058865 Capezza, N. (2003). The cultural-psychological foundations for violence and non-violence. An empirical study. Forum: Qualitative Social Research, 4(2), 1-24. Retrieved from http://www.qualitative-research.net/fqs/. Chenail, R. J. (2011). Ten steps for conceptualizing and conducting qualitative research studies in a pragmatically curious manner. The Qualitative Report, 16, 1713-1730. Retrieved from http://www.nova.edu/ssss/QR/QR16-6/chenail.pdf Christ, T. (2014). Scientific-based research and randomized controlled trials, the “gold”standard? Alternative paradigms and mixed methodologies. Qualitative Inquiry, 20, 72-80. doi:10.1177/1077800413508523

26


Conrad, P., De Allegri, M., Moses, A., Larsson, E. C., Neuhann, F., Müller, O., & Sarker, M. (2012). Antenatal care services in rural Uganda: Missed opportunities for good-quality care. Qualitative Health Research, 22, 619-629. doi:10.1177/1049732311431897 Cooper, B., & Glaesser, J. (2012). Qualitative work and the testing and development of theory: Lessons from a study combining cross-case and within-case analysis via Ragin's QCA. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 13(2), Art.4. Retrieved from http://nbn-resolving.de/urn:nbn:de:0114-fqs120247 de Guzman, R., Leonard, N. R., Gwadz, M. V., Young, R., Ritchie, A. S., Arredondo, G., & Riedel, M. (2006). “I thought there was no hope for me”: A behavioral intervention for urban mothers with problem drinking. Qualitative Health Research, 16, 1252-1266. doi:10.1177/1049732306294080 Denzin, N. K. (2010). Moments, mixed methods, and paradigm dialogs. Qualitative inquiry, 16, 419-429. doi:10.1177/1077800410364608 Diriwächter, R., & Valsiner, J. (2005). Qualitative developmental research methods in their historical and epistemological contexts. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 7(1), Art. 8. Retrieved from http://www.qualitativeresearch.net/fqs-texte/1-06/06-1-8-e.htm Eberle, T. S., & Elliker, F. (2005). A cartography of qualitative research in Switzerland. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 6(3), Art. 24. Retrieved from http://nbn-resolving.de/urn:nbn:de:0114-fqs0503244 Elliott, J. (2008). The narrative potential of the British birth cohort studies. Qualitative Research, 8, 411-421. doi:10.1177/1468794106093637 Fechner, M. J. (2010). Culture and comorbidity in East and West Berliners. Qualitative Health Research, 20, 400-408. doi:10.1177/1049732309359172 Gaglio, B., Nelson, C. C., & King, D. (2006). The role of rapport: Lessons learned from conducting research in a primary care setting. Qualitative Health Research, 16, 723-734. doi:10.1177/1049732306286695 Gaupp, N. (2013). School-to-work transitions—Findings from quantitative and qualitative approaches in youth transition research. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 14(2), Art. 12. Retrieved from http://www.qualitativeresearch.net/index.php/fqs/article/view/1895 Grim, B. J., Harmon, A. H., & Gromis, J. C. (2006). Focused group interviews as an innovative quanti-qualitative methodology (QQM): Integrating quantitative elements into a qualitative methodology. The Qualitative Report, 11, 516-537. Retrieved from http://www.nova.edu/ssss/QR/QR11-3/grim.pdf Gupta, S., Paterson, M., von Zweck, C., & Lysaght, R. (2012). Using hermeneutics to understand burnout and coping strategies utilized by occupational therapists. The Qualitative Report, 17(105), 1-25. Retrieved from http://www.nova.edu/ssss/QR/QR17/sgupta.pdf

27


Güthlin, C., Anton, A., Kruse, J., & Walach, H. (2012). Subjective concepts of chronically ill patients using distant healing. Qualitative Health Research, 22, 320-331. doi:10.1177/1049732311421914 Hall, J. N., & Ryan, K. E. (2011). Educational accountability: A qualitatively driven mixedmethods approach. Qualitative Inquiry, 17, 105-115. doi:10.1177/1077800410389761 Hammerli, K., Znoj, H., & Berger, T. (2010). What are the issues confronting infertile women? A qualitative and quantitative approach. The Qualitative Report, 15, 766-782. Retrieved from http://www.nova.edu/ssss/QR/QR15-4/hammerli.pdf Hesse-Biber, S. (2010a). Emerging methodologies and methods practices in the field of mixed methods research. Qualitative Inquiry, 16, 415-420. doi:10.1177/1077800410364607 Hesse-Biber, S. (2010b). Qualitative approaches to mixed methods practice. Qualitative Inquiry, 16, 455-468. doi:10.1177/1077800410364611 Hesse-Biber, S. (2012). Weaving a multimethodology and mixed methods praxis into randomized control trials to enhance credibility. Qualitative Inquiry, 18, 876-889. doi:10.1177/1077800412456964 Howe, K. R. (2011). Mixed methods, mixed causes? Qualitative Inquiry, 17, 166-171. doi:10.1177/1077800410392524. Jeanty, G. C., & Hibel, J. (2011). Mixed methods research of adult family care home residents and informal caregivers. The Qualitative Report, 16, 635-656. Retrieved from http://www.nova.edu/ssss/QR/QR16-3/jeanty.pdf Johnstone, P. L. (2004). Mixed methods, mixed methodology health services research in practice. Qualitative Health Research, 14, 259-271. doi:10.1177/1049732303260610 Kaasalainen, S., Williams, J., Hadjistavropoulos, T., Thorpe, L., Whiting, S., Neville, S., & Tremeer, J. (2010). Creating bridges between researchers and long-term care homes to promote quality of life for residents. Qualitative Health Research, 20, 1689-1704. doi:10.1177/1049732310377456 Keptner, K. M. (2011). Mixed methods design: A beginner's guide. The Qualitative Report, 16, 593-595. Retrieved from http://www.nova.edu/ssss/QR/QR16-2/keptner.pdf Kneipp, S. M., Lutz, B. J., Levonian, C., Cook, C., Hamilton, J. B., & Roberson, D. (2013). Women’s experiences in a community-based participatory research randomized controlled trial. Qualitative Health Research, 23, 847-860. doi:10.1177/1049732313483924 Koenig, T. (2006). Compounding mixed-methods problems in frame analysis through comparative research. Qualitative Research, 6, 61-76. doi:10.1177/1468794106058874 Kura, S. Y. B. (2012). Qualitative and quantitative approaches to the study of poverty: Taming the tensions and appreciating the complementarities. The Qualitative Report, 17(34), 119. Retrieved from http://www.nova.edu/ssss/QR/QR17/kura.pdf

28


Kurz, A. E., Saint-Louis, N., Burke, J. P., & Stineman, M. G. (2008). Exploring the personal reality of disability and recovery: A tool for empowering the rehabilitation process. Qualitative Health Research, 18, 90-105. doi:10.1177/1049732307309006 Le, H., Perry, D. F., Genovez, M., & Cardeli, E. (2013). In their own voices: Latinas’ experiences with a randomized controlled trial. Qualitative Health Research, 23, 834-846. doi:10.1177/1049732313482591 Luxardo, N., Colombo, G., & Iglesias, G. (2011). Methodological and ethical dilemmas encountered during field research of family violence experienced by adolescent women in Buenos Aires. The Qualitative Report, 16, 984-1000. Retrieved from http://www.nova.edu/ssss/QR/QR16-4/luxardo.pdf Lutz, B., Kneipp, S., & Means, P. (2009). Developing a health screening questionnaire for men in welfare transition programs in the United States. Qualitative Health Research, 19, 105115. doi:10.1177/1049732308327347 Mason, J. (2006). Mixing methods in a qualitatively driven way. Qualitative Research, 6, 9-25. doi:10.1177/1468794106058866 Maxwell, J. A. (2010). Using numbers in qualitative research. Qualitative Inquiry, 16, 475482.doi:10.1177/1077800410364740 Medjedović, I. (2011). Secondary analysis of qualitative interview data: Objections and experiences. Results of a German feasibility study. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 12(3), Art. 10. Retrieved from http://www.qualitative-research.net/index.php/fqs/article/view/1742 Meetoo, D., & Temple, B. (2003). Issues in multi-method research: Constructing self-care. International Journal of Qualitative Methods, 2(3) Art. 1. Retrieved from http://www.ualberta.ca/~iiqm/backissues/2_3final/pdf/meetootemple.pdf Medlinger, S., & Cwikel, J. (2008). Spiraling between qualitative and quantitative data on women’s health behaviours: A double helix model for mixed methods. Qualitative Health Research, 18, 280-293. doi:10.1177/1049732307312392 Mehl-Madrona, L., Mainguy, B., & Valenti, M.P. (2013). Mixed methodology approaches to exploring spiritual transformation. The Qualitative Report, 18(5), 1-11. Retrieved from http://www.nova.edu/ssss/QR/QR18/mehl-madrona5.pdf Mertens, D. (2010). Transformative mixed methods research. Qualitative Inquiry, 16, 469-474. doi:10.1177/1077800410364612 Miller, S., & Fredericks, M. (2006). Mixed-methods and evaluation research: Trends and issues. Qualitative Health Research, 16, 567-579. doi:10.1177/1049732305285691 Miyata, H., & Kai, I. (2009). Reconsidering evaluation criteria for scientific adequacy in health care research: An integrative framework of quantitative and qualitative criteria. International Journal of Qualitative Methods, 8(1), 64-75. Retrieved from http://wigan-ojs.library.ualberta.ca/index.php/IJQM/article/view/1822

29


Molina, G. (2009). An integrity perspective on the decentralization of the health sector in Colombia. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 10(2) Art. 5. Retrieved from http://www.qualitativeresearch.net/index.php/fqs/article/view/1309 Moorefield-Lang, H. M. (2010). Arts voices: Middle school students and the relationships of the arts to their motivation and self-efficacy. The Qualitative Report, 15(1), 1-17. Retrieved from http://www.nova.edu/ssss/QR/QR15-1/moorefield-lang.pdf Moran-Ellis, J., Alexander, V. D., Cronin, A., Dickinson, M., Fielding, J., Sleney, J., & Thomas, H. (2006). Triangulation and integration: Processes, claims and implications. Qualitative Research, 6, 45-59. doi:10.1177/1468794106058870 Morrow, K., Rosen, R., Salomon, L., Woodsong, C., Severy, L., Fava, J., Barroso, C. (2011). Using integrated mixed methods to develop behavioral measures of factors associated with microbicide acceptability. Qualitative Health Research, 21, 987-999. doi:10.1177/1049732311404245 Morse, J. (2008). Editorial: Serving two masters: The qualitatively-driven, mixed-method proposal. Qualitative Health Research, 18, 1607-1608. doi:10.1177/1049732308326246 Morse, J. (2009). Editorial: Mixing qualitative methods. Qualitative Health Research. 19, 15231524. doi:10.1177/1049732309349360 Morse, J. M. (2010). Simultaneous and sequential qualitative mixed method designs. Qualitative Inquiry, 16. 483-491. doi:10.1177/1077800410364741 Naidoo, K. (2007). Researching reproduction: Reflections on qualitative methodology in a transforming society. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 9(1), Art. 12, Retrieved from http://nbn-resolving.de/urn:nbn:de:0114fqs0801121 Novotná, G., Urbanoski, K. A., & Rush, B. R. (2011). Client-centered design of residential addiction and mental health care facilities: Staff perceptions of their work environment. Qualitative Health Research, 21, 1527-1538. doi:10.1177/1049732311413782 O'Donnell, A. B., Lutfey, K. E., Marceau, L. D., & McKinlay, J. B. (2007). Using focus groups to improve the validity of cross-national survey research: A study of physician decision making. Qualitative Health Research, 17, 971-981. doi:10.1177/1049732307305257 O’Grady, C. P., & Skinner, W. J. W. (2012). Journey as destination: A recovery model for families affected by concurrent disorders. Qualitative Health Research, 22, 1047-1062. doi:10.1177/1049732312443736 Ohlen, J. (2010). Review: Janice M. Morse and Linda Niehaus (2009). Mixed method design: Principles and procedures. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 12(1), Art. 15. Retrieved from http://nbn-resolving.de/urn:nbn:de:0114fqs1101159 Onwuegbuzie, A. J., & Leech, N. L. (2004). Enhancing the interpretation of “significant” findings: The role of mixed methods research. The Qualitative Report, 9, 770-792. Retrieved from http://www.nova.edu/ssss/QR/QR9-4/onwuegbuzie.pdf

30


Onwuegbuzie, A. J., & Collins, K. M. T. (2007). A typology of mixed methods sampling designs in social science research. The Qualitative Report, 12, 281-316. Retrieved from http://www.nova.edu/ssss/QR/QR12-2/onwuegbuzie2.pdf Onwuegbuzie, A. J., & Dickinson, W. B. (2008). Mixed methods analysis and information visualization: Graphical display for effective communication of research results. The Qualitative Report, 13, 204-225. Retrieved from http://www.nova.edu/ssss/QR/QR132/onwuegbuzie.pdf Onwuegbuzie, A. J., & Leech, N. L. (2006). Linking research questions to mixed methods data analysis procedures. The Qualitative Report, 11, 474-498. Retrieved from http//www.nova.edu/ssss/QR/QR11-3/onwuegbuzie.pdf Orvik, A., Larun, L., Berland, A., & Ringsberg, K. (2013). Situational factors in focus group studies: A systematic review. International Journal of Qualitative Methods, 12, 338-359. Retrieved from http://ejournals.library.ualberta.ca/index.php/IJQM/ Pacheco, E., & Blanco, M. (2008). Work and family: An exercise in mixed methodology. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 9(1), Art. 28. Retrieved from http://www.qualitative-research.net/index.php/fqs/article/view/341 Phellas, C. (2005). Review: Keith F. Punch: Introduction to social research: Quantitative and qualitative approaches. Forum Qualitative Sozialforschung /Forum: Qualitative Social Research, 7(2), Art. 4. Retrieved from http://nbn-resolving.de/urn:nbn:de:0114fqs060249 Plano Clark, V. L. (2010). The adoption and practice of mixed methods: U.S. trends in federally funded health-related research. Qualitative Inquiry, 16, 428-440. doi:10.1177/1077800410364609 Propst, D. B., McDonough, M. H., Vogt, C. A., & Pynnonen, D. M. (2008). Roving focus groups: Collecting perceptual landscape data in situ. International Journal of Qualitative Methods, 7(3), 1-14. Retrieved from http://ejournals.library.ualberta.ca/index.php/IJQM/ Rapport, F., Jerzembek, G., Seagrove, A., Hutchings, H., Russell, I., Cheung, W., & Williams, J. G. (2010). Evaluating innovations in the delivery and organization of endoscopy services in England and Wales. Qualitative Health Research, 20, 922-930. doi:10.1177/1049732309354282 Rogers, A., Kennedy, A., Nelson, E., & Robinson, A. (2005). Uncovering the limits of patientcenteredness: Implementing a self-management trial for chronic illness. Qualitative Health Research, 15, 224-239. doi:10.1177/1049732304272048 Rowan, N., & Wulff, D. (2007). Using qualitative methods to inform scale development. The Qualitative Report, 12, 450-466. Retrieved from http://www.nova.edu/ssss/QR/QR123/rowan.pdf Saraceno, M. J., & Tambling, R. B. (2013). The sexy issue: Visual expressions of heteronormativity and gender identities in Cosmopolitan magazine. The Qualitative Report, 18(80), 1-18. Retrieved from http://www.nova.edu/ssss/QR/QR18/saraceno80.pdf

31


Shaw, I. (2003). Qualitative research and outcomes in health, social work and education. Qualitative Research, 3, 57-77. doi:10.1177/146879410300300103 Smyth, R. (2006). Exploring congruence between habermasian philosophy, mixed-method research, and managing data using NVivo. International Journal of Qualitative Methods, 5(2), 1-11. Retrieved from http://wiganojs.library.ualberta.ca/index.php/IJQM/article/view/4395. Stevens, K. J. (2010). Working with children to develop dimensions for a preference-based, generic, pediatric, health-related quality-of-life measure. Qualitative Health Research, 20, 340-351. doi:10.1177/1049732309358328 Stineman, M. G., Rist, P. M., & Burke, J. P. (2009). Through the clinician's lens: Objective and subjective views of disability. Qualitative Health Research, 19, 17-29. doi:10.1177/1049732308327853 Tarr, J., & Thomas, H. (2011). Mapping embodiment: Methodologies for representing pain and injury. Qualitative Research, 11, 141-154. doi:10.1177/1468794110394067. Tavernier, S. S., Totten, A. M., & Beck, S. L. (2011). Assessing content validity of the patient generated index using cognitive interviews. Qualitative Health Research, 21, 1729-1738. doi:10.1177/1049732311420169 Terrell, S. R. (2012). Mixed-methods research methodologies. The Qualitative Report, 17, 254280. Retrieved from http://www.nova.edu/ssss/QR/QR17-1/terrell.pdf Thomas, E. (2011). From qualitative data to instrument development: The Women’s Breast Conflict Scale. The Qualitative Report, 16, 908-932. Retrieved from http://www.nova.edu/ssss/QR/QR16-4/thomas.pdf Townsend, L., Ragnarsson, A., Mathews, C., Johnston, L. G., Ekström, A. M., Thorson, A., & Chopra, M. (2011). “Taking care of business”: Alcohol as currency in transactional sexual relationships among players in Cape Town, South Africa. Qualitative Health Research, 21, 41-50. doi:10.1177/1049732310378296 Verd, J. M., & López, M. (2011). The rewards of a qualitative approach to life-course research. The example of the effects of social protection policies on career paths. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 12(3), Art.15. Retrieved from http://nbn-resolving.de/urn:nbn:de:0114-fqs1103152 Von Zweck, C., Paterson, M., & Pentland, W. (2008). The use of hermeneutics in a mixed methods design. The Qualitative Report, 13, 116-134. Retrieved from http://www.nova.edu/ssss/QR/QR13-1/vonzweck.pdf Wagner, P. J., Warren, P. R., & Moseley, G. (2010). Patient and physician perceptions of dimensions of necessity of medical utilization. The Qualitative Report, 15, 301-317. Retrieved from http://www.nova.edu/ssss/QR/QR15-2/warren-wagner.pdf Westhues, A., Ochocka, J., Jacobson, N., Simich, L., Maiter, S., Janzen, R., & Fleras, A. (2008). Developing theory from complexity: Reflections on a collaborative mixed method participatory action research study. Qualitative Health Research, 18, 701-717. doi:10.1177/1049732308316531

32


Weston, J. M., Norris, E. V., & Clark, E. M. (2011). The invisible disease: Making sense of an osteoporosis diagnosis in older age. Qualitative Health Research, 21, 1692-1704. doi:10.1177/1049732311416825 Wiedemann, G. (2013). Opening up to big data: Computer-assisted analysis of textual data in social sciences. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 14(2), Art. 13. Retrieved from http://nbn-resolving.de/urn:nbn:de:0114fqs1302231 Wuest, J. (2011). Are we there yet? Positioning qualitative research differently. Qualitative health research, 21, 875-883. doi:1049732311401424 Yeager, J., & Sommer, L. (2005). How linguistic frames affect motivational profiles and the roles of quantitative versus qualitative research strategies. The Qualitative Report, 10, 463-511. Retrieved from http://www.nova.edu/ssss/QR/QR10-3/yeager.pdf Yehl, A. (2010). From “clueless” to “completed”: A review of The Essential Guide to doing your research project. The Qualitative Report, 15, 746-749. Retrieved from http://www.nova.edu/ssss/QR/QR15-3/oleary.pdf

33

Message from the Editor - Journal Hosting and Publishing - University

Message from the Editor - Journal Hosting and Publishing - University

Suggest Documents

Message from the Editor - Journal Hosting and Publishing - University ...

Message from the Editor - Journal Hosting and Publishing - University ...

Message from the Editor - Journal Hosting and Publishing

Message from the Editor - Journal Hosting and Publishing

Message from the Editor - Journal Hosting and Publishing

Message from the Editor - Journal Hosting and Publishing

Message from the Editor - Journal Hosting and Publishing

Message from the Editor - Journal Hosting and Publishing

Message from the Editor - Journal Hosting and Publishing

Message from the Editor - Journal Hosting and Publishing

Message from the Editor - Journal Hosting and Publishing

Message from the Editor - Journal Hosting and Publishing

Message from the Editor - Journal Hosting and Publishing

Message from the Editor - Journal Hosting and Publishing

Message from the Editor - Journal Hosting and Publishing

Message from the Editor - Journal Hosting and Publishing

Message from the Editor - Journal Hosting and Publishing

Message from the Editor - Journal Hosting and Publishing

Message from the Editor - Journal Hosting and Publishing

Message from the Editor - Journal Hosting and Publishing

Message from the Editor - Journal Hosting and Publishing

Message from the Editor - Journal Hosting and Publishing

Message from the Editor - Journal Hosting and Publishing

Message from the Editor - Journal Hosting and Publishing