Journal of Operations Management 16 Ž1998. 441–454
Building operations management theory through case and field research Jack Meredith
)
Babcock Graduate School of Management, Wake Forest UniÕersity, P.O. Box 7659, Winston-Salem, NC 27109, USA
Abstract Case and field research studies continue to be rarely published in operations management journals, in spite of increased interest in reporting such types of studies and results. This paper documents the advantages and rigor of caserfield research and argues that these methods are preferred to the more traditional rationalist methods of optimization, simulation, and statistical modeling for building new operations management theories. In the process of describing the constructs of inference and generalizability with reference to case research, we find the existing definitions inadequate and thus extend and refine them to better discriminate between alternate research methodologies. We also elaborate on methods for increasing the generalizability of both rationalist and caserfield research studies. A major conclusion is that these alternate research methods are not mutually exclusive and, if combined, can offer greater potential for enhancing new theories than either method alone. q 1998 Elsevier Science B.V. All rights reserved. Keywords: Building operations management theory; Case and field research; Rationalist methods
1. Introduction A number of recent papers and editorials Že.g., Wood and Britney, 1989; McCutcheon and Meredith, 1993; Ebert, 1989. have pointed out the relative paucity of case and field research in operations management. This form of empirical research continues to be poorly understood and infrequently published in our top journals. In part, this may be due to unfamiliarity with the nature of theory building using case and field study methods. As one example, a researcher some time back submitted a paper on steel mini-mill technology in the early days of mini-mills. The paper was rejected on the basis of a referee’s )
Tel.: q1-910-759-4467; e-mail:
[email protected]
criticism that a sample of nine was simply too small for statistical conclusions. The researcher’s rejoinder was that this was not a sample, it was the entire population! Both the referee and the editor were then at a loss, not knowing where to go from there. The intent of this paper is to clearly convey why the empirical methods of case and field research are preferred to the more traditional rationalist ŽMeredith et al., 1989. methods of optimization, simulation, and statistical modeling for building new operations management theories. In doing so, we review the major issues involving theory building such as the nature of the theory-development process and the requirements for establishing rigor in research. As we explore the constructs of inference and generalizability, in particular, with reference to case research, we find the existing definitions inadequate and thus
0272-6963r98r$19.00 q 1998 Elsevier Science B.V. All rights reserved. PII S 0 2 7 2 - 6 9 6 3 Ž 9 8 . 0 0 0 2 3 - 0
442
J. Meredithr Journal of Operations Management 16 (1998) 441–454
extend and refine them to allow better discrimination between alternate research methodologies. With regard to generalizability, we identify three primary ways of increasing this critical characteristic of rigorous research and offer specific methods to help with each category. We also find that the objectivity provided by quantification in the rationalist methods can be a hindrance in the attempt to build theory because a qualitative understanding of the quantified factors is still required for theories to be accepted by others in, and outside, the field. But a major conclusion here is that these alternate research methods are not mutually exclusive and, if combined, can offer even greater potential for enhancing new theories than either method alone. To facilitate our discussion of the theory-building process, we frequently contrast caserfield research with rationalism, the dominant research paradigm in operations management for the last 35 years. Rationalism, an epistemological paradigm that includes the beliefs of positivism and some forms of empiricism, generally employs quantitative methodologies to describe or explain phenomena and here specifically includes optimization models, simulation modeling, survey methodology, and Žless frequently in operations management. laboratory experiments. Rationalism is concerned with explaining what happens and how, so as to achieve some goal or end such as predicting production system characteristics, or perhaps the effect of some change in managerial policy on plant measures. Caserfield study is one example of an alternative research paradigm known as interpretivism and uses both quantitative and qualitative methodologies to help understand phenomena. It is more process- or means-oriented and helps the researcher comprehend why certain characteristics or effects occur, or do not occur. We begin our discussion by first defining rationalist and case research and identifying their advantages and disadvantages. We then describe theory development and the roles of these two approaches in the theory-building process of identification, explanation, prediction, and understanding. Since the basis for theory-building is drawing conclusions, we next discuss the issues of research design, inference, and deduction. Following this, we discuss the four requirements for achieving research rigor so as to
assure that our conclusions are valid, ending with an extensive discussion of the topic of generalizability to other populations. We conclude with a description of the trade-offs between alternate research methods and the applicability of each in the theory-development process.
2. Rationalist research and case research 2.1. Definition of rationalist research One major characteristic of rationalist research is the belief that the phenomenon being studied exists ‘out there’, independent of the research context or beliefs and assumptions of the researcher ŽKlein and Lyytinen, 1985, p. 136; Guba, 1990, p. 20–21.. Thus, the relationships and observations are considered to be independent of the theories used to explain them and can hence be studied, manipulated at will, and controlled as needed by the researcher. Another major characteristic of rationalist research is the goal of determining the distributions of a set of pre-specified variables in the population or verifying a set of pre-specified relationships. In this paper, we will primarily consider the rationalist methods of modeling by equations, laboratory experiments, and statistical survey analysis in making our comparisons to case research. Note that our division here does not parallel a frequently-used one of dividing operations management research between so-called ‘theoretical’ Ža misnomer, since all these paradigms can be theoretical. and ‘empirical’ methods. Typical equation-modeling research would include variants of the economic order quantity formulation as well as production-inventory system simulations. Statistical survey research is currently being used in quality management, supply chain, and technology management investigations, among many others. 2.2. Definition of case research For our purposes here, we base our definition of case research on Benbasat et al. Ž1987. Žp. 370., Bonoma Ž1985. Žp. 203–204., Eisenhardt Ž1989. Žp. 534., and Yin Ž1994. Žp. 13.. A case study typically uses multiple methods and tools for data collection
J. Meredithr Journal of Operations Management 16 (1998) 441–454
from a number of entities by a direct observerŽs. in a single, natural setting that considers temporal and contextual aspects of the contemporary phenomenon under study, but without experimental controls or manipulations. The methods and tools employed include both quantitative and qualitative approaches as well as obtrusive and unobtrusive methods. Example entities include financial data, interviews, memoranda, business plans, organization charts, tools and other physical artifacts, questionnaires, and observations of managerial or employee actions and interactions. The goal is to understand as fully as possible the phenomenon being studied through ‘perceptual triangulation’ ŽBonoma, 1985, p. 203., the accumulation of multiple entities as supporting sources of evidence to assure that the facts being collected are indeed correct. A subtle but important point here is that understanding can only be considered knowledge within the confines of someone’s, typically the researcher’s, perceptual framework. Thus, understanding is not ‘out there’ in the rationalist sense that the meaning of this work stands by itself and is obvious to anyone who looks at it. Rather, the understanding that is achieved is only meaningful within a framework of assumptions, beliefs, and perspectives specified by the researcher, usually his or her own. Hence, the understanding is not without bias and cultural taint. For example, if we speak of a stochastic simulation, an entire set of understandings, procedures, and expectations spring to mind, whereas if we speak of critical social theory, or cybernetic systems theory, or growth need strength, completely different paradigms come to mind Žor may not come to mind, depending on one’s background.. Note also the importance of direct observation wfirst source Žseeing it oneself. rather than second Žspeaking or writing to someone who saw or experienced it. or third, or sometimes no source at allx, the role of the context in which the phenomenon is occurring Žan important consideration in later attempts at generalization., and the dynamics of the temporal dimension through which the events of the phenomenon unfold Žthereby helping to understand the how and why elements of the phenomenon.. The case study is restricted to a single setting; when multiple settings are investigated to help extend the generalizability of the results, we call this a multiple
443
case study or, if many cases are involved and are selected with some definite research pattern in mind, a field study. However, a field study is not an attempt to ‘increase the sample size of the study’ but rather, like follow-on experiments or surveys, to extend the study to new populations. Typically, every site in the field study is examined just as thoroughly as a single case study with the same objective of perceptual triangulation. Perhaps a useful generic example here might be Margaret Mead’s famous studies of Pacific island societies, each culture being a separate case that needed to be fully understood all by itself. 2.3. AdÕantages of rationalist research Table 1 illustrates some of the major advantages and disadvantages of rationalist and case research methods. Some of the strengths of rationalist research are the precision it can achieve in its variables Že.g., set-up or holding costs. and thus, the testability and reliability this offers. That is, the measurable quantitative variables can be very carefully specified and then precisely tested, or checked by another researcher. Another major advantage of the rationalist approach is the knowledge and wide acceptance of its standard research procedures Žmodel formulation, variance reduction techniques, sample size., particularly in operations management. 2.4. AdÕantages of case research Benbasat et al. Ž1987. Žp. 370. identify three outstanding strengths of the case study approach: Ž1. Table 1 Advantages and disadvantages of rationalist and case research methods Advantages
Disadvantages
Rationalist
Precision Reliability Standard procedures Testability
Case
Relevance Understanding Exploratory depth
Sampling difficulties Trivial data Model-limited Low explained variance Variable restrictions Thin results Access and time Triangulation requirements Lack of controls Unfamiliarity of procedures
444
J. Meredithr Journal of Operations Management 16 (1998) 441–454
the phenomenon can be studied in its natural setting and meaningful, relevant theory generated from the understanding gained through observing actual practice; Ž2. the case method allows the much more meaningful question of why, rather than just what and how, to be answered with a relatively full understanding of the nature and complexity of the complete phenomenon; and Ž3. the case method lends itself to early, exploratory investigations where the variables are still unknown and the phenomenon not at all understood. An operations management example that well-illustrates all three of these strengths is the case study conducted by Gerwin Ž1981., on one of the first flexible manufacturing systems ŽFMSs.. Yin Ž1994., McCutcheon and Meredith Ž1993., and Eisenhardt Ž1989. identify other advantages of the case method such as the richness of its explanations and its potential for testing hypotheses in well-described, specific situations. 2.5. DisadÕantages of rationalist research Rationalist research methods also have their drawbacks. First, Aldag and Stearns Ž1988. point out that obtaining valid empirical generalizations depends to a large extent on the use of sampling procedures that are rigorous, representative of a well-specified population, and provide a source of information concerning the constructs to be measured. Yet, despite the importance of employing rigorous sampling criteria for generalizability of the findings, the great majority of quantitative studies based on sampling appear to use samples of convenience or opportunity: ‘‘The remainder w87% of the research studiesx identified samples based on the investigators’ convenience or opportunity’’ ŽAldag and Stearns, 1988, p. 259.. Additionally, Bailey Ž1992. Žp. 50. notes that using a rationalist model employing statistical analysis of objective measures of the variables results in data which, although perhaps measurable and precise, are quite likely also to be the most trivial. Other rationalist failings noted by Bailey are that the research cannot produce information that goes beyond the original model such as anomalies Ža point elaborated on in a later section., that because the data are collected out of context, they cannot account for possibly crucial variations related to context, and that the rationalist researcher risks producing reliable but unimportant ‘so what’ results.
Other scholars ŽBenbasat et al., 1987, p. 369; Bonoma, 1985, p. 203; Van Maanen, 1982, p. 13. identify other failings of rationalist studies: the abstract and remote character of key variables, the lack of comparability across studies, the failure to achieve much predictive validity, the causal complexity of multivariate analysis, the distribution restrictions of statistics such as normality, the often trivial amount of explained variance, the large sample sizes required, and finally, the difficulty of understanding, interpreting, and especially implementing the results of these studies Žtermed ‘thin results’ in Table 1.. There may be statistical methodology issues as well: Berger and Berry Ž1988. Žpp. 160, 163. describe with an example how ‘significant evidence at the 0.05 level can actually arise when the data provide very little or no evidence in favor of an effect’. They conclude that this ‘indicates a potentially serious flaw in the logic behind the use of P-values and other standard measures of evidence’. 2.6. DisadÕantages of case research Some of the difficulties of doing case research are the requirements of direct observation in the actual contemporary situation Žcost, time, access hurdles.; the need for multiple methods, tools, and entities for triangulation; the lack of controls; and the complications of context and temporal dynamics. And as noted in Section 1, another serious disadvantage of the case method is the lack of familiarity of its procedures and rigor by our colleagues. For example, Aldag and Stearns Ž1988. Žp. 260–261. point out that qualitative research in general is commonly perceived as exhibiting a tendency for construct error, poor validation, and questionable generalizability. 3. The theory development process 3.1. Theory building, testing, and modification Regardless of whether a caserfield study approach or a rationalist approach is employed, Whetten Ž1989. Žp. 491. points out that ‘‘During the theory-development process, logic replaces data as the basis for evaluation . . . This requires explaining the whys underlying the reconstituted whats and
J. Meredithr Journal of Operations Management 16 (1998) 441–454
445
hows’’. Sooner or later, therefore, if we wish to develop or extend theory, we must get back to why —that is, understanding. This was clearly the purpose of Gerwin Ž1981. in studying this new technology of FMSs. The rationalist methods, with their ability to tell us what the phenomenon entails and how it works, still need to make the final leap to understanding in order to generate or develop theory rather than just test it. That is why many scholars of research Že.g., Richardt and Cook, 1979, p. 17. tend to believe that the rationalist methods are most appropriate for testing or Õerifying existing theory while the interpretive methods, such as case studies, are best for generating or extending theory. Although researchers ŽMcCutcheon and Meredith, 1993, among others. frequently point out that both approaches can be used for both purposes, the natural emphasis of the case approach on understanding is clearly most directly focused on theory building. Table 2 arrays the stages of theory development against the research objectives of what, how, and why. Across the top is the progression in the development of theory starting with theory building, followed by testing and then modification of the theory to account for the test results. Although not shown, the process then repeats with more testing and modification, with this process continuing indefinitely. In the body of the table, the case and rationalist methods are positioned where they find greatest applicability. As noted earlier, the rationalist methods are primarily directed to the what and how rows but only the case method is positioned along the why row. In terms of the columns, the rationalist methods are most appropriate for testing theories that have been previously developed by other methods such as caserfield studies or other interpretive methods. In a field such as operations management, where rational-
ist methods have been the predominant and favored research paradigm for years, the theory-building stage has suffered. Hence, the theories that rationalists have been able to test have been limited to primarily mathematical ones, or else those that could be developed from intuition, supposition, and occasional reports of field work appearing in either academic or practitioner publications. As noted earlier, case methods can also be used for testing theories but, due to the effort involved, are primarily useful when developing new theory or testing particular issues or aspects of an existing theory. In terms of theory modification, both rationalist and case methods are applicable, but in different ways. Considering a rationalist example first, in survey methodology, a diagram of factor entities or variables may be depicted as boxes, along with their expected interactions with other variables, as indicated by lines or arrows. There may even be hypothesized directions of causation, or hypothesized positive and negative correlations. Following testing, some of these variables may be found not to be significant in the phenomenon after all Žhence, the minus signs in Table 2., or the correlations may be the opposite of what was expected, or there may even be relationships among some of the factors that were not anticipated Žthe plus signs in Table 2.. Thus, rationalism can alter both the what and how of the theory as originally posited. The case method can do this as well Žthrough observation and triangulation, as explained later. but, due to the nature of its methodology, can also identify new variables and relationships not conceived of in the original theory ŽMcCutcheon and Meredith, 1993.. That is, the rationalist methods are limited to the model as originally formulated, but caserfield methods can and often do go beyond the original model, particularly if there is a need to explain anomalies or unexpected results.
Table 2 Theory development under rationalist and case research methods
3.2. Identification, explanationr prediction, and understanding
Theory building
Theory testing
Theory modification
What
Case
How
Case
Why
Case
Rationalist ŽCase. Rationalist ŽCase. Žnot relevant.
Rationalist Ž". Case Ž". Rationalist Ž". Case Ž". Case
The issues of identification Ž what ., explanation Ž how ., and understanding Ž why . are worth exploring in more detail. Although Hudson and Ozanne Ž1988. Žp. 510. maintain that for rationalists, the goal of explanation is prediction, Bacharach Ž1989. Žp. 501.
J. Meredithr Journal of Operations Management 16 (1998) 441–454
446
sees explanation and prediction differently: ‘‘explanation establishes . . . meaning . . . while a prediction tests that . . . meaning’’. However, for our purposes here, we need not be concerned with this fine distinction. On the other hand, understanding entails considerably more than explanation Žor prediction.. Kaplan Ž1964. illustrates this with the example of the ancient astronomers who made excellent predictions of the future positions of the planets but were unable to say why Žunderstanding. the planets behaved this way. Hudson and Ozanne Ž1988. Žp. 510. point out another difference between explanation and understanding: understanding is a never-ending process rather than an end. One final distinction may also be made—with explanation or prediction, we are usually interested in similarities, whereas with understanding, we are equally, if not more, interested in differences Že.g., how managing an FMS is different from managing a robot, or a transfer line..
4. Research design, inference, deduction, and drawing conclusions 4.1. Three sets of factors It is common practice in research studies, both rationalist and interpretivist, to divide the factors of interest into three sets as shown in Table 3. One set of factors, called parameters by some researchers, define the population of interest and are those we attempt to hold constant during the study. That is, they vary among different populations but are held constant in the population under study. In rationalist studies, these parameters are simply fixed by the researcher in the equations of the model or simulation, are factored into the sample selected by the survey researcher, or are held fixed by the experiTable 3 Handling the sets of research factors
Rationalist Case
Parameters
Independent variables
Dependent variables
Fix Select for
Manipulate Monitor or controlr select for
Observe Observe
menter. In case studies, the researcher controls for these factors through the selection of the situation or site to be studied. For example, the researcher may wish to study the use of flexible manufacturing systems in fast-response automotive supply chains. Clearly, the researcher must limit the case to those in the automotive supply business who have FMSs and compete on the basis of fast-response. The other two sets of factors, called the independent Že.g., order size or frequency. and dependent Že.g., total cost. variables, we try to manipulate Žin rationalist studies. or monitor or controlrselect for Žin case studies. to observe how the independent variables influence the dependent variables. Because we are limited in our ability to monitor or control the number of independent variables, we typically try to put all the other factors into the set of parameters that are being held constant Ž‘all else being equal’. for our study. Our ability to draw accurate conclusions from the study often depends upon how well we correctly identify and allocate the factors among these two sets. Placing too many factors into the parameter set often results in significant Žstatistically, in rationalist studies. but uninteresting ŽDavis, 1971. results that are only trivial extensions to current theory. On the other hand, a problem with studies that find few or no significant conclusions is often that an insufficient number of factors Že.g., backorders or environmental factors. were identified for placement into the parameter set and instead became, by default, unrecognized independent variables. These unrecognized independent variables then gave rise to such variability in the dependent variables that we could discern no statistically Žin rationalist studies. significant effect of the variables we were monitoring or manipulating. The purpose of randomization in the sample selection process of survey studies ŽCalder et al., 1981. is to minimize the impact of these unrecognized independent variables. However, this randomization may also result in minimizing the range, and hence, effect size ŽVerma and Goodale, 1995. of the independent variables. Thus, a better procedure is to attempt to select a sample that exhibits a full range of the independent variables since these are the factors that we are most confident will impact the dependent variables.
J. Meredithr Journal of Operations Management 16 (1998) 441–454
4.2. Importance of qualitatiÕe knowing An extremely important philosophical consideration in drawing research conclusions brought out by Richardt and Cook Ž1979. Žp. 13, 22, 23. is that ‘quantitative understanding presupposes qualitative knowing’. In other words, researchers cannot benefit from their use of numbers if they cannot communicate, in common sense terms, what their numbers mean Žsuch as the ‘cost of quality’ or even something as apparently straightforward as ‘order size’.. To reiterate an earlier point, researchers operate from a specific perceptual framework but to communicate the common sense importance of their results, they must be able to communicate in terms of the perceptual frameworks of others. Since all facts are imbued, in part, with subjective understanding and using objective indicators to define outcomes and make conclusions is a subjective process, the mechanical assignment of numbers to measures in rationalist research does not guarantee objectivity. In fact, it tends to obscure the true subjectivity of the study and its findings through its apparent objectivity. For example, the variables in a research study concerning multi-stage, multi-echelon inventory planning may be well-understood among a narrow group of researchers in that field, but to other researchers not familiar with the research in that area, as well as to managers of such systems in corporations, the variables being discussed in the findings may well be unfamiliar or confusing, and may not even have a real-world equivalent for corporate managers. Thus, the objectivity provided by quantitative measures in rationalist research assumes that all parties understand, in a qualitatiÕe sense, both the perceptual framework of the researcher as well as the measures. Yet, in much of operations management research, this is often not the case. Thus, we find that quantitative measures, in and of themselves, are insufficient and qualitative understanding is required for drawing research conclusions and communicating the importance of the results. 4.3. Inference and deduction Rationalist studies commonly involve two types of inference but there is usually no distinction drawn
447
between their difference. One type, which we will term here ‘representational inference’, is an attempt to determine if the sample chosen is representative of the target population. An inference is made about the target population Že.g., automotive suppliers. on the basis of data collected from the sample. For example, the sample data may be used to test if a variable is different from zero and, if the sample results are significant, that conclusion is then extended to the population. Here, we worry primarily about sample size and sometimes find that we must increase the sample size to increase our confidence in the test. ‘Relational inference’, our term for the second type, is where we attempt to determine if one factor is related to Že.g., correlated with, caused by, modified by. another. Here we may encounter more complex issues concerning power Žthe ability to identify an effect when it in fact exists., significance level, sample size, and effect size Žsee Verma and Goodale, 1995; Baroudi and Orlikowski, 1989.. We may also have difficulties with internal validity, the correctness of our conclusion of a relationship. In a case study, we deal with only relational inference because the case is not intended to represent a sample from a population. For instance, we might select one mini-mill to study among the population of nine described earlier. But our intent in the case study is not to measure variables in the sample and statistically infer relationships because we can directly obserÕe the processes and use logic to deduce or infer relationships. Thus, in this situation, the case is akin to an experiment in a laboratory or a complete survey. If additional cases are undertaken, it is not equivalent to increasing the sample size but rather, to extending the experiment or survey to another population that probably has different parameters in some ways, but is similar in other ways, as with Margaret Mead’s Pacific island societies. With case studies, we use rigorous data collection, observation, triangulation, and logic rather than mathematics or statistics to make our deductions and inferences, a good example again being Gerwin Ž1981.. In the process of trying to understand, we strive to identify the variables Žthe parameters are already defined by the case chosen. that affect the phenomenon, estimate their variability, determine their effect size, and understand their workings and relationships. Power is determined in case studies by
448
J. Meredithr Journal of Operations Management 16 (1998) 441–454
exactly the same factors as in statistical studies ŽVerma and Goodale, 1995, p. 141; Baroudi and Orlikowski, 1989, p. 89.: the significance level, the effect size, and the precision of estimates of the variables. However, the significance levels and effect sizes are evaluated analytically in case studies rather than statistically. The variable estimates are determined in case studies in a manner similar to increasing the sample size in statistical studies: by triangulating on the variables and their effects. ŽYin, 1994 identifies other techniques as well such as patternmatching and the use of protocols..
5. Achieving rigor in research A difficulty researchers conducting case studies in operations management often face is the common misperception that case research is not ‘rigorous’ because many of the variables may not be mathematically quantified and the independent variables cannot be manipulated at will. But as Lee Ž1989., McCutcheon and Meredith Ž1993., Bonoma Ž1985., Richardt and Cook Ž1979., and Yin Ž1994., among many others, note, the case study method is guided by the same overall principles and follows as welldefined rules of evidence and proof as the rationalist methods. Lee Ž1989. Žp. 41., for example, points out that the scientific method does not actually require such elements as laboratory controls, statistical controls, mathematical propositions, or replicable observations, although many scholars incorrectly believe that it does. These are simply a means to obtaining rigor that case studies achieve through different means. How case studies attain each of the four requisites of rigor—controlled observations, controlled deductions, replicability, and generalizability—is described by Lee Ž1989. Žpp. 39–41., and shown in Table 4.
5.1. Controlled obserÕations First, controlled observations in case research are attained through natural Žrather than laboratory or statistical. controls, the same controls that astronomers and geologists use. Natural controls rely on the selection of the phenomena during the study’s experimental design stage for their control, thereby allowing particular factors Že.g., managerial policies, inventory systems. to be, in essence, ‘held constant’ while others Že.g., costs, defect rates. are left free to vary as they would naturally. In this way, case and field studies achieve the same ends through natural methods that experiments and statistical methods achieve through direct control of the independent variables. 5.2. Controlled deductions Second, since formal logic encompasses mathematics, the requirement of controlled deductions in Table 4 can be attained by applying the rules of formal logic to verbal propositions arising from the case study. Thus, although sometimes desirable for precision, it is not necessary to mathematically quantify all of the variables in the study. As Lee Ž1989. Žp. 40. points out: ‘‘ . . . mathematics is a subset of formal logic, not vice versa. Logical deductions in the general case do not require mathematics. wAx case study that performs its deductions with verbal propositions Ži.e., qualitative analysis. therefore only deprives itself of the convenience of the rules of algebra; it does not deprive itself of the rules of formal logic, to which it may therefore still turn when carrying out the task of making controlled deductions.’’ For example, the theory of evolution was logically deduced by Darwin through words and sentences, not numbers and mathematics. In operations, our theories concerning implementation Že.g., Pressman and Wildavsky, 1973; Meredith, 1981.,
Table 4 Methods to meet the requirements for rigor
Rationalism Case
Controlled observation
Controlled deduction
Replicability
Generalizability
Laboratory or statistics Natural
Mathematics Logic
Results Theory
Assumptive Theoretic
J. Meredithr Journal of Operations Management 16 (1998) 441–454
innovation ŽGerwin, 1988., and the product-process matrix ŽHayes and Wheelwright, 1979. were also logically, rather than mathematically, deduced. 5.3. Replicability The third requirement of Table 4, replicability, is attained in rationalist studies by achieving the same quantitative results when the study is precisely duplicated: same parameters, independent variables, dependent variables, and controls. Note that we are not trying to replicate the earlier development of the theory but rather the same study results based on the assumed theory. That is, precisely duplicate the study and the researcher should get precisely the same results. In case studies, because exactly the same case conditions can never be fully duplicated in another situation, replicability is attained by applying the resulting case study theory to a somewhat different set of conditions, which very well might result in a different prediction. Thus, even though the prediction is different, the same theory is being tested.
6. Generalizability The fourth and most difficult requirement in Table 4, generalizibility to new populations Žother industries, other suppliers in the value chain., also known as ‘external validity’, is as problematic for case studies as it is for rationalist studies. Hedrick et al. Ž1993. Žp. 40. define external validity as the ‘‘extent to which it is possible to generalize from the data and context of the research study to broader populations and settings’’. Cook and Campbell Ž1979. Žp. 37. and Yin Ž1994. Žp. 33. have a similar definition: ‘‘the domain to which a study’s findings or presumed causal relationships can be generalized’’. ŽBonoma Ž1985, p. 200., defines a similar concept, ‘currency’, as ‘‘the characteristics of research that affect the contextual relevance of findings across measures, methods, persons, settings, and time’’ Žitalics in original... When speaking of generalizability, an interesting and illustrative conundrum has developed in the operations field between those rationalists who do
449
algorithmic and simulation modeling research and those interpretivists who do case and field research. The former often maintain that their results are highly generalizable because they apply in any situation and time frame where the assumptions hold Žand for many robust findings, even when some of the assumptions do not hold., whereas the findings from case research have little generalizability because the results are only valid for that case’s situation. On the other side, the caserfield researchers often maintain that the theory developed from their studies is applicable to other similar Žin the sense of having the same population parameters. situations and even in situations that are not similar but where the theory would still apply and predict a different result. Likewise, they maintain that the algorithmic and simulation results have little generalizability because real situations are much more complex than the simplified reality assumed by the rationalists and no real situation ever satisfies all the assumptions on which the findings have been based. 6.1. Types of generalizability It would seem that both camps’ defense of the generalizability of their own approach has some merit but their criticism of the opposing camp’s approach has some flaws. Clearly, some algorithmic and simulation models such as queuing and inventory studies certainly well duplicate reality and have utility in the real world, while some of the theories developed from case and field studies such as those concerning implementation or technology management apply to other situations similar to, and sometimes even different from, the original situationŽs. studied. As with inference, the literature does not well-discriminate between these two approaches to generalizability. Therefore, as indicated in Table 4, we term the former ‘assumptive generalizability’ and the latter ‘theoretic generalizability’. Assumptive generalizability represents those rationalist studies, especially descriptive and normative models such as econometric analyses, optimization studies, and simulations, where the assumptions precisely identify the environment Žparameters and variables. being studied. The findings from these studies may then be
450
J. Meredithr Journal of Operations Management 16 (1998) 441–454
generalized to situations where other factors Žvariables. not considered in the study have minimal influence relative to the factors considered, or to situations that even violate the study assumptions but the findings are nevertheless found to be robust to those assumptions. Theoretic generalizability represents those interpretivist studies, especially case and field research, where the theory itself indicates that it would be applicable in a particular situation. That is, the parameters and variables in the theory give an indication as to its range of generalizability. As noted earlier, a well-known operations example of this is the product-process matrix of Hayes and Wheelwright Ž1979. and its many extensions. Yin Ž1994. Žp. 30. defines two other types of ‘generalization’ that occur in certain types of studies but do not particularly concern external validity. When discussing statistical survey studies, he defines ‘statistical generalization’ as that process of generalizing the findings from the sample to the population it purports to represent; it is critically dependent on the sample size and variation within the sample and population. In some ways however, Yin’s term may be a misnomer because the generalization is a purely logical rather than statistical one. Even if a confidence interval is supposedly determined for a population variable, it is completely determined by the sample taken and not by any of the characteristics of the actual population; that is, nothing ‘statistical’ is done in projecting the sample findings to the population. Note that there is no attempt here to generalize the theoryrfindings to other populations Žexternal validity.. Yin’s second type of generalization he terms ‘analytical generalization’: generalizing the findings of a study to create theory. Note that this is again similar to the concept of statistical generalization above in that the research study’s findings are used for another purpose, in this situation to create theory, typically concerning the population represented in the study. However if, say, a case study provided insight into other factors not typically present in this population, then the generalization may be extended to other situations or populations also. The process of actually making this extension would then represent external validity or, as we term it here, theoretic generalizability. If the theory holds for other similar
or dissimilar situations, replication or extension of the theory, respectively, may be claimed. And if two or more cases support the same findings but do not support rival findings, then even greater confidence in the theoretic generalizability of the theory has been established. 6.2. Methods to enhance generalizability Eisenhardt Ž1989. Žp. 537. and Glaser and Strauss Ž1967., like Yin, also speak of the difference between statistical and analytical, but in terms of sampling. They distinguish theoretical Žanalytical. sampling from statistical sampling by noting that the purpose of statistical sampling is simply to obtain accurate statistical evidence on the distributions of variables within the population. The purpose of theoretical sampling, however, is to replicate or extend the emergent theory by identifying extremes, polar types Žopposite situations along some dimension., or candidates for niche situations to help discover categories, properties, and interrelationships that will extend the theory. There are three primary methods for enhancing the generalizability of a study. One way is to include as many independent variables as possible in the study so other situations that include these factors will also thereby be included in the theory. In rationalist studies, this would mean more equations, more complex models, more laboratory experimental treatments, more factors to statistically query in the questionnaire. In case and field studies, this would mean greater depth of observation and more triangulation to identify additional independent variables that may be having a subtle influence on the results Žbut possibly less subtle in another, different population.. The researcher, of course, hopes that these additional variables will not require additional complexity in the theory being developed because theoretical simplicity is always a highly desirable characteristic of research Žnot to mention the perennial desire by editors to shorten manuscript pages.. Hence, the wise choice here for the researcher is to include as many factors as possible that are suspected of not significantly altering the developing theory and avoid factors that would probably require major modifications in the theory.
J. Meredithr Journal of Operations Management 16 (1998) 441–454
The second method consists of including multiple populations in the original study to develop a more comprehensive theory—different subjects in experiments, different sampling frames in surveys, polar types or extremes in multiple case studies. This situation is common in operations when two different industries are tested concerning the same phenomenon and the researcher suspects that industry type is irrelevant for this phenomenon. Again, the advice to the researcher here is to include populations that would not be expected to significantly confound the developing theory. For example, the polar types or extremes should be those that the researcher intuitively believes would dovetail with a compact, efficient theory rather than pose an anomaly that cannot be easily explained. The researcher usually has suspicions—an intuitive theory—about why some populations act the way they do while other populations act in other ways, but then yet other populations act in ways that are bewildering. The researcher should study the first two populations and leave the bewildering one for follow-on studies, perhaps by another researcher with other intuitive suspicions. The third way of increasing generalizability is by testing the original theory on alternate populations. If the theory passes the test, then its relevance is extended even further. If it does not pass the test, the researcher has an opportunity to extend or replace the theory. Here, the researcher may have a suspicion that a theory will not hold in a particular population for certain reasons—an intuitive new theory. If the researcher’s suspicions are confirmed, the new, more generalizable theory replaces the previous theory such as when Einstein’s theory of relativity replaced Newton’s more limited theory of gravity.
451
inherently more generalizable than qualitative results’’. This is because, as we have also argued here, generalization is a far more inductive process than simple statistical projection of a sample to the population. Although a large and diverse set of cases can aid in such generalization, ‘so can a depth of understanding of a single case’.
7. Conclusions 7.1. A methodological continuum As a result of the natural advantages and disadvantages of the rationalist statistical methodology and the interpretivist caserfield research method described earlier, there is a trade-off between them that changes with the number of units being investigated. This is shown in Fig. 1 which plots the number of units of analysis in the research study against the applicability of the statistical method and the caserfield study method. McCutcheon and Meredith Ž1993. present an extensive list of published operations management studies that fall among the categories of study types in Fig. 1. At the bottom of the number of units of analysis axis, the single case study with its extensive qualitative description and contextual and temporal analysis is the most applicable method and statistical methodology is inappropriate. The single case is particularly appropriate for completely new, exploratory investigations. Again, the study of Gerwin Ž1981. of FMSs is an excellent example of this type of study.
6.3. The role of understanding in generalizability Calder et al. Ž1982. Žp. 240., in discussing the importance of generalizability, note the argument that ‘the absence of external validity implies an attendant lack of construct validity’. The point is that research that is weak in generalizability cannot provide an adequate test of theory. Richardt and Cook Ž1979. Žp. 15. note also that generalizability depends on much more than simple sample size and thus, ‘‘there is no reason quantitative results should be
Fig. 1. Methodological applicability relative to number of units.
452
J. Meredithr Journal of Operations Management 16 (1998) 441–454
Moving up from this, we encounter the multiple case study of perhaps two to eight situations ŽEisenhardt, 1989, p. 545, recommends 4 to 10.. With considerably more effort, this will extend the generalizability of the single case findings. The multiple case study is appropriate when there is some knowledge about the phenomenon but much is still unknown. Including extreme cases and polar types is highly desirable here. Maffei and Meredith Ž1995., as well as Gerwin and Tarondeau Ž1982., offer examples of this approach. However, at eight units, the use of small sample statistics is becoming feasible also, particularly if the effect size of the phenomenon under study is quite large. If the research design is carefully constructed to identify polar types, a liberal significance criterion is used Žsuch as 0.10., and the effect size is large, the small number of sites may not reduce the statistical power so much that a statistical effect will not be detected ŽBaroudi and Orlikowski, 1989.. Maffei Ž1991., for example, illustrates this possibility with a simple regression in her multiple case study. Increasing the number of units further into the low teens brings us to a situation where both statistical and caserfield methods are equally applicable, yet neither is particularly appropriate or advantageous in this region. For multiple case studies, most of the interesting extremes and polar types can be captured with fewer cases, whereas field studies require a greater number of situations to array against the parameters of interest. And for statistical approaches, we are still working with very small samples. Further increasing the number of units into the high teens or lower twenties results in a full-fledged field study plus statistical analysis. In this situation, considerably less qualitative description may be involved, but more statistical measures of variables can be documented. Given the larger number of units, the power of our small sample statistical analyses is somewhat higher so we may be able to obtain significant results. The case descriptions may be relatively slight or perhaps some minimal level is provided for all sites, but a few sites are investigated more thoroughly. Dennis Ž1993. and Marsh Ž1993. conducted such field studies, collecting statistical data as well as fairly extensive qualitative documentation about each site and then running regressions and cluster
analyses with the statistical data to supplement their qualitative findings. Finally, at the upper twenties and thirties extensive description is nearly impossible other than listing basic characteristics and showing means and ranges of the sampling parameters Že.g., industry SIC code, size of firm, annual sales.. However, with this number of units considered as a sample, we are finally beginning to obtain adequate power in our statistical analyses of the relationships between variables and thus, make more confident statistical inferences. The curves in Fig. 1 are convex for the two types of studies for different reasons. For case and field studies, the mental confusion as more sites are added grows exponentially rather than linearly, hence, the non-linear relationship. For statistical studies, the use of small sample statistics and the acceptability of higher levels of the significance criterion for studies of ‘new’ phenomena provide what is generally considered to be ‘acceptable’ evidence. 7.2. Trade-offs among methodologies The result of these pros and cons of rationalist vs. interpretivist methods, as noted by Bonoma Ž1985. and others, is some natural trade-offs between the methods. The reliability, internal validity, and measurement precision available with rationalist approaches can only be obtained at the expense of the contextual and temporal richness that case and field studies offer. The explanatory power of rationalism is obtained by sacrificing the understanding gained through interpretivism. But, fortunately, this is not an all or nothing proposition. Both methods may be combined at times to yield more than either method by itself Že.g., see Kaplan and Duchon, 1988.. Although this is not seen very frequently, requiring either a researcher wellschooled in both methods or a team comprised of both skills, there is indeed a range of research studies as shown in Fig. 1 where this combined approach could be very profitable in deriving solid, relevant theory or enhancing existing theory. 7.3. Implications for researchers We have shown that case and field studies exhibit the same level of rigor and adhere to the same requirements of good research as rationalist studies,
J. Meredithr Journal of Operations Management 16 (1998) 441–454
but achieve these goals by different means. For example, requirements such as controlled observation, controlled deduction, and replicability are obtained in case research through natural controls, logic, and theory testingrprediction, respectively. Where statistical studies deal with two types of inferencerepresentational and relational—case and field studies deal only with the latter. For relational inference, case studies must address the same difficulties of power, significance level, sample size, and effect size as statistical studies. Similarly, for generalization of the findings to new populations Žexternal validity., rationalist studies rely on assumptive generalization, whereas case and field studies use theoretic generalization. Although there exist many trade-offs between rationalist and caserfield studies, the former tend to focus on explanationrprediction Žwhat and how. while the latter are more concerned with understanding Žwhy.. Thus, rationalist studies are preferred for testing and modifying new theories, where the variables are well known and the parameters have been clearly identified, or well-established theories in new situations with new parameters. Then the rationalist researcher can see whether the theory holds, as well as determine any changes that are required in the existing factors of the theory. The caserfield focus on understanding is preferable for new theory development in operations management Žfor some example topics, see McCutcheon and Meredith, 1993. because eventually, the explanation of quantitative findings and the construction of theory based on those findings will ultimately have to be based on qualitative understanding. Caserfield methods are also useful for selective testing of existing theories in particular situations or circumstances, such as a polar type or an extreme situation. They are also useful when existing theory must be extended to include new factors, or for situations that require a deeper understanding of what is happening to modify or extend current theory. It is useful to note, however, that there do exist research situations where both methods may be usefully applied if the research team includes the dual talent and skills required Žfor some examples, see Meredith et al., 1989.. Finally, just as with rationalist studies, caserfield research must be conducted with rigor and satisfy the
453
standard requirements of competent research. For case and field studies, this often means close and detailed observation, triangulation, careful determination of cause–effect chains, and logical inference. The same applies to generalization to other populations Žexternal validity.; moreover, just as with experiments, surveys, or other rationalist studies, additional cases may need to be studied in order to properly extend the theory to cover these populations.
Acknowledgements This research was supported by the Babcock Graduate School of Management, Wake Forest University, Research Fellowship Program.
References Aldag, R.J., Stearns, T.M., 1988. Issues in research methodology. J. Manage. 14 Ž2., 253–273. Bacharach, S.B., 1989. Organizational theories: some criteria for evaluation. Acad. Manage. Rev. 14 Ž4., 496–515. Bailey, M.T., 1992. Do physicists use case studies? Thoughts on public administration research. Public Administration Rev. 52 Ž1., 47–54. Baroudi, J.J., Orlikowski, W.J., 1989. The problem of statistical power in MIS research. MIS Q. 13 Ž1., 87–106. Benbasat, I., Goldstein, D.K., Mead, M., 1987. The case research strategy in studies of information systems. MIS Q. 11 Ž3., 369–386. Berger, J.O., Berry, D.A., 1988. Statistical analysis and the illusion of objectivity. Am. Scientist 76, 159–165. Bonoma, T.V., 1985. Case research in marketing: opportunities, problems, and a process. J. Marketing Res. 22, 199–208. Calder, B.J., Phillips, L.W., Tybout, A.M., 1981. Designing research for application. J. Consumer Res. 8, 197–207. Calder, B.J., Phillips, L.W., Tybout, A.M., 1982. The concept of external validity. J. Consumer Res. 9, 240–244. Cook, T.D., Campbell, D.T., 1979. Quasi-Experimentation: Design and Analysis Issues in Field Settings. Houghton Mifflin, Boston, MA. Davis, M.S., 1971. That’s interesting! Philos. Soc. Sci. 1, 309–344. Dennis, D., 1993. Defining Production and Inventory Control Systems for Process Industries. Unpublished doctoral dissertation, University of Cincinnati, Cincinnati, OH. Ebert, R.J., 1989. Announcement on empiricalrfield-based methodologies in JOM. J. Operations Manage. 8 Ž4., 294–296. Eisenhardt, K.M., 1989. Building theories from case study research. Acad. Manage. Rev. 14 Ž4., 532–550.
454
J. Meredithr Journal of Operations Management 16 (1998) 441–454
Gerwin, D., 1981. Control and evaluation in the innovative process: the case of flexible manufacturing systems. IEEE Tran. Eng. Manage. EM 28 Ž3., 62–70. Gerwin, D., 1988. A theory of innovation processes for computer-aided manufacturing technology. IEEE Trans. Eng. Manage. 35 Ž2., 90–100. Gerwin, D., Tarondeau, J.C., 1982. Case studies of computer integrated manufacturing systems: a view of uncertainty and innovation processes. J. Operations Manage. 2 Ž2., 87–99. Glaser, B., Strauss, A., 1967. The Discover of Grounded Theory: Strategies in Qualitative Research. Wiedenfeld and Nicolson, London. Guba, E.G., 1990. The Paradigm Dialog. Sage Publications, Newbury Park, CA. Hayes, R.H., Wheelwright, S.C., 1979. Link manufacturing process and product life cycles. Harvard Bus. Rev., January– February, pp. 133–140. Hedrick, T.E., Bickman, L., Rog, D.J., 1993. Applied Research Design: A Practical Guide. Sage Publications, Newbury Park, CA. Hudson, L.A., Ozanne, J.L., 1988. Alternative ways of seeking knowledge in consumer research. J. Consumer Res. 14, 508– 521. Kaplan, 1964. The Conduct of Inquiry. Crowell, New York. Kaplan, B., Duchon, D., 1988. Combining qualitative and quantitative methods in information systems research: a case study. MIS Q. 12 Ž4., 571–586. Klein, H.K., Lyytinen, K., 1985. The poverty of scientism in information systems. In: Mumford, E., Hirscheim, R. ŽEds.., Research Methods for Information Systems. North Holland, Amsterdam. Lee, A.S., 1989. A scientific methodology for MIS case studies. MIS Q. 13 Ž1., 33–50. Maffei, M.J., 1991. Infrastructure and the Routine Use of Flexible
Manufacturing Technology. Unpublished doctoral dissertation, University of Cincinnati, Cincinnati, OH. Maffei, M.J., Meredith, J.R., 1995. The impact of infrastructure on achieving benefits from flexible manufacturing technology. J. Operations Manage. 13 Ž4., 273–298. Marsh, R.F., 1993. The Life Cycle of Manufacturing Cells. Unpublished doctoral dissertation, University of Cincinnati, Cincinnati, OH. McCutcheon, D.M., Meredith, J.R., 1993. Conducting case study research in operations management. J. Operations Manage. 11 Ž3., 239–256. Meredith, J.R., 1981. The implementation of computer-based systems. J. Operations Manage. 2 Ž1., 11–21. Meredith, J.R., Raturi, A., Amoako-Gyampah, K., Kaplan, B., 1989. Alternative research paradigms in operations. J. Operations Manage. 8 Ž4., 297–326. Pressman, J.L., Wildavsky, A., 1973. Implementation. University of California Press, Berkeley, CA. Richardt, C.S., Cook, T.D., 1979. Beyond qualitative vs. quantitative methods. In: Richardt, C.S., Cook, T.D. ŽEds.., Qualitative and Quantitative Methods in Evaluation Research. Sage Publications, Newbury Park, CA, pp. 7–32. Van Maanen, J., 1982. Introduction. In: Van Maanen, J., Dabbs, Jr., J.M., Faulkner, R.R. ŽEds.., Varieties of Qualitative Research. Sage Publications, Newbury Park, CA, pp. 11–29. Verma, R., Goodale, J.C., 1995. Statistical power in operations management research. J. Operations Manage. 13 Ž2., 139–152. Whetten, D.A., 1989. What constitutes a theoretical contribution?. Acad. Manage. Rev. 14 Ž4., 490–495. Wood, A.R., Britney, R.R., 1989. Production and operations management: research and teaching opportunities in the 1990s. Operations Manage. Rev. 8 Ž3–4., 33–43. Yin, R.K., 1994. Case Study Research: Design and Methods, 2nd edn. Sage Publications, Newbury Park, CA.
