IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART A: SYSTEMS AND HUMANS, VOL. 39, NO. 1, JANUARY 2009
57
Understanding Decision-Support Effectiveness: A Computer Simulation Approach Jeffrey E. Kottemann, Kathleen M. Boyer-Wright, Joel F. Kincaid, and Fred D. Davis
Abstract—The interplay between decision-making and decisionsupport tools has proven puzzling for many years. One of the most popular decision-support tools, what-if analysis, is no exception. Decades of empirical studies have found positive, negative, and null effects. In this paper, we contrast the marginal-analysis decision-making strategy enabled by what-if with the anchoring and adjustment decision-making strategies prevalent among unaided decision makers. By using an aggregate production planning decision task, we develop a Monte Carlo simulation to model 1000 independent what-if decision-making episodes across a myriad of conditions. Results mirror and explain seemingly contradictory findings across multiple prior experiments. Thus, this paper formalizes a simulation approach that expands the scope of previous findings regarding unaided versus what-if analysis aided decision making and suggests that relative performance is quite sensitive to task conditions. In this light, then, performance effect differences in past research are to be expected. While our analysis involves a single task context, the larger and more important point is that, even within a single task context, performance differences between unaided and aided decision making are emergent. Index Terms—Decision making, decision-support systems (DSSs), simulation.
I. I NTRODUCTION
B
OUNDED rationality [1] has become a fundamental premise in human decision and decision-support research. Given that people have limited abilities to memorize, to calculate, and to construct mental models of complex phenomena, they resort to simplifications. People “satisfice” rather than “optimize.” People use heuristics. Indeed, people use heuristics even in situations where cognitive demands are not high [2]. The goal of computer-based decision-support systems (DSSs) is to help decision makers become more rational, more analytical, and less bounded. Decision-support tools augment human memory, perform sophisticated computations, and provide mechanisms for modeling complex phenomena [3]. Among the mental heuristics people use, anchoring and adjustment strategies are prevalent [4]. Consider a marketing manager who must determine future advertising budgets, product prices, production quantities, and sales force deployment with the objective of maximizing future profits. By using anchoring Manuscript received September 8, 2007; revised January 23, 2008 and May 1, 2008. Current version published December 17, 2008. This paper was recommended by Associate Editor J. B. Yang. J. E. Kottemann and K. M. Boyer-Wright are with the Perdue School of Business, Salisbury University, Salisbury, MD 21801 USA (e-mail:
[email protected]). J. F. Kincaid is with Winton-Salem State University, Winston-Salem, NC 27110 USA. F. D. Davis is with the University of Arkansas, Fayetteville, AR 72701 USA. Digital Object Identifier 10.1109/TSMCA.2008.2007992
and adjustment, the manager would begin with current values and adjust them up or down, perhaps too much, perhaps too little, perhaps just right. Anchoring and adjustment strategies are not categorically bad and can be effective in certain circumstances [5], [6]; however, improper anchoring and under- or overadjustment can lead to biases and errors in relatively simple judgment tasks [7] as well as in decision making in complex dynamic task environments [8]. Because decision outcomes are not observable until the adjustment is made and the resulting effects are manifest, the manager may learn appropriate strategies over time but at the price of costly lag effects and mistakes [8]. Now, consider the marketing manager making decisions while using a popular decision-support tool—what-if analysis. What-if analysis involves a formal computational model of a decision task, allowing decision makers to interactively evaluate alternate scenarios and explore the task/problem space in the spirit of marginal optimization. Upon entering various values for decision and environmental variables into a spreadsheet, they can immediately see estimates of the resulting values of outcome variables. By using what-if analysis, the same marketing manager would enter assorted values for individual variables, making marginal adjustments in search of preferred outcomes a priori to making a final decision. This approach ostensibly leads to improved decision quality and performance. Despite the conceptual and popular appeal of what-if analysis, empirical findings relating what-if analysis to decision quality have demonstrated mixed results over more than two decades [3], [9]–[19]. For example, Dos Santos and Bariff [12] demonstrated that coaching users to formalize the marginal analysis strategies enabled by what-if analysis increased task performance above that of users without such coaching. Similarly, Van Bruggen et al. [17] confirmed that the use of what-if analysis lessened the use of anchoring and adjustment heuristics and increased decision quality. Several studies, however, indicate contradictory evidence in which unaided decision makers fared as well or better than those assisted by what-if analysis tools. Goslar et al. [14] examined this problem in the context of a marketing case, finding no difference between the DSS group and the non-DSS group in terms of the number of alternatives considered, time spent, perceived confidence, amount of data considered, decision-making process, and overall performance. Kottemann et al. [10], [15], [16], [19] evaluated the production planning task across multiple contexts, finding considerable variation in performance between aided and unaided subjects. In several instances, the use of what-if analysis led to
1083-4427/$25.00 © 2008 IEEE
58
IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART A: SYSTEMS AND HUMANS, VOL. 39, NO. 1, JANUARY 2009
overconfidence without commensurate improvements in decision outcomes. The latter finding is particularly troubling considering the prevalent use of what-if analysis to solve decision problems. We argue that the primary explanation for the variety of results from prior studies is that decision-support effectiveness is contingent on environment characteristics—demand trend, randomness, and time horizon—of the task relative to the tool at hand. Given the complexity of and potential variations in experimental conditions, it is not surprising that studies contrasting aided and unaided decision makers produced inconsistent findings. Fortunately, advances in computing power enable modeling and simulation of decision makers under multiple conditions and with subtle variation in environment attributes. Thus, it is possible to identify decision domains suited to what-if analysis capabilities, as well as decision conditions for which mental heuristics yield superior results. Employing Monte Carlo methods—in a manner similar to that of Payne et al. [20]—we simulate both the anchoring and adjustment strategies typical of unaided decision makers and the marginal analysis strategy enabled by what-if analysis. The Monte Carlo simulations model 1000 independent what-if decision-making episodes across a myriad of conditions in the context of an aggregate production scheduling task. A comparison of the simulation results explains the mixed results in prior empirical studies. We show that the relative performance shifts across patterns and levels of environmental factors, as well as over time. Finally, we investigate the effects of multimethod (“hybrid”) decision-support tools and discuss implications for DSS design. We then discuss limitation of the present study and suggest avenues for future research. This paper formalizes a simulation approach that expands the scope of previous findings regarding unaided versus what-if analysis aided decision making [3], [9]–[19] by investigating a wider range of task conditions than previous laboratory studies and finds that relative decision-making performance is quite sensitive to task conditions. While our analysis involves a single task context, the larger and more important point is that, even within a single task context, performance differences between unaided and aided decision making are emergent. II. T ASK C ONTEXT U SED FOR A NALYSIS AND I LLUSTRATION For purposes of analysis and illustration, we examine the fairly general task of aggregate production planning, which has a long history in economic and behavioral research. Following the introduction of the formalized aggregate production planning task [21], [22], the task has since been used extensively to explore behavioral aspects of unaided decision making as well as the effects of decision-support methods such as bootstrapping, graphic displays, forecasting, formal heuristics, fuzzy methods, and what-if analysis. In this task, a decision maker faces product demand over a series of decision periods and must determine production and staffing levels with the objective of minimizing cumulative total
costs. The cost function for a period quadratic cost components
t+1
is the sum of three
Workforce Change = 64.3 (workforcet − workforcet+1 )2 Workforce Over-Time or Idle-Time = 0.8 (workforcet+1 ∗ k1 ) − (productiont+1 )2 Inventory Holding or Shortage = 0.02 (inventoryt +productiont+1−demandt+1 −k2 )2 where k1 = 5.47 is a productivity constant and k2 = 320 is an optimal safety stock constant. The functional forms and the specific values for coefficients and constants were derived by economists based upon an actual production facility [21], [22]. Notice that the cost components are interrelated in that one cost component can be avoided only by trading it for another. For example, if workforce is set unduly low relative to production, then worker overtime cost is driven higher. If, in turn, production is set unduly low, then the cost of inventory shortage is driven higher. As these are quadratic equations, increases and decreases in the equation parameters create equivalent effects. In addition, the marginal costs of the components differ by orders of magnitude. The decision maker’s task, then, is to determine how to allocate costs among the three cost components in order to minimize cumulative costs over time. This aggregate planning task is representative of the large class of tasks in which decision makers must address recurring decisions in a dynamic environment. Similar environments arise across many aspects and levels of human decision making, from investing corporate finances to managing a household budget. Recall that anchoring and adjustment strategies are prevalent mental heuristics across a broad range of task environments. The aggregate planning task is no exception: In studies involving unaided decision makers, the use of anchoring and adjustment strategies has been empirically demonstrated [23], [24]. When making production and workforce decisions over time, people attempt to track (see [2] and [25]) upcoming demand. Tracking strategies are quintessential anchoring and adjustment strategies [5]. Individuals using a Demand-tracking strategy will anchor on current levels and adjust future workforce and production levels in an attempt to track and satisfy anticipated changes in demand. They concentrate heavily on tracking demand by adjusting production and incurring workforce change costs rather than using overtime/idle time or inventory excess/backorder. Past laboratory research has found that unaided decision makers’ typical strategy differs fundamentally from decision makers using what-if analysis decision support [10], [11], [15]–[19]. Contrast demand tracking with the Marginal-costing strategy enabled by what-if analysis. By using what-if analysis, decision makers can focus on changes in production and workforce levels that lead to lower costs given the upcoming demand. In short, they will be driven primarily by the marginalcost implications of their decisions. Referring back to the cost
KOTTEMANN et al.: UNDERSTANDING DECISION-SUPPORT EFFECTIVENESS
equations, it can be seen that the marginal rates for workforce change are higher than those for inventory and overtime/idle time. It has been found that what-if users, in being driven by marginal-cost considerations, do not track demand by adjusting production and workforce as much as absorbing marginal costs through inventory and overtime/idle time [10], [26]. Take, for example, a situation in which an upcoming period’s demand increases. A demand-tracking decision maker will increase production and workforce levels upward to track demand. A whatif user focusing more on the marginal costs of the three cost components will instead absorb more of the increase through using inventory and worker overtime. As the present task involves marginal costs, the marginal analysis strategy is labeled as marginal costing. Again, the use of this strategy by what-if analysis users has been empirically demonstrated [10], [26]. A. Hypothesis Development To frame the research agenda, we adopt the common operations management typology of possible environmental conditions as delineated by Krajewski and Ritzman [28] to investigate the following four hypothetical decision-making scenarios: 1) decision making under conditions of escalating environmental uncertainty; 2) decision making under conditions of stable environmental trends; 3) decision making under conditions of cyclic demand; 4) decision making across varying time horizons. In essence, each of these four questions relates to a single issue of environmental uncertainty vis-à-vis environmental trend. Intuition suggests that anchoring and adjustment strategies, where the decision maker makes small adjustments to an observable pattern, would work well in conditions where there is a consistent trend in demand. Environmental uncertainty, manifesting as demand fluctuations, would make it more difficult for the decision maker to anticipate the correct approach; thus, the “what-if” or marginal-costing approach is more likely to yield the preferred results. These expectations are consonant with Hoch and Schkade’s [27] experimental findings that anchor and adjustment strategies performed poorly in less predictable environments. In scenario one, therefore, we would expect that a marginalcosting strategy, which accounts for changes in current conditions, would, on average, outperform demand tracking and that these differences would accentuate as conditions of environmental uncertainty escalate. Conversely, scenario two with a stable trend would enable a demand-tracking decision maker to make appropriate incremental adjustments based on an established pattern rather than on a margin analysis. These contrasting scenarios can be restated as the following two hypotheses. 1) H1 : Overall, marginal-costing decision strategy will outperform a demand-tracking strategy under conditions of accelerating environmental uncertainty. 2) H2 : Overall, a demand-tracking decision strategy will outperform a marginal-costing strategy under conditions of continued but stable trend.
59
Cyclical demand patterns contain elements of both uncertainty and trend through the intertwining of amplitude and frequency. We hypothesize an interaction effect in which cyclical demand environments characterized by high amplitude will favor marginal-costing decision strategies with an increase in the differential of marginal costing over demand tracking as cycle frequency increases. Low-cycle amplitude coupled with longcycle wavelengths should mirror stable environmental trend conditions, thus favoring a demand-tracking strategy. These can be restated as the following two hypotheses. 1) H3a : Overall, a marginal-costing decision strategy will outperform a demand-tracking strategy under the conditions of high-amplitude/high-frequency cyclical demand. 2) H3b : Overall, a demand-tracking decision strategy will outperform a marginal-costing strategy under the conditions of low-amplitude/low-frequency cyclical demand. The cyclical demand scenarios illustrate an important aspect of this paper. The hypothesized differences are based on relative ranges in which one decision-making approach strategy emerges as the dominant strategy rather than defined ranges. Thus, one can anticipate an intermediate range in which neither strategy offers a consistent cost benefit over the other. This last point is aptly demonstrated by the fourth scenario positing differences in decision-making performance across varying time horizons. Take, for example, two money managers using two different investment strategies. Their relative performance for one-year and five-year time frames could be quite different, indeed reversed. This relativity is manifest in the conflicting results of prior studies in which the number of decision periods that subjects were asked to complete varied across studies. Kottemann and Remus [15] explicitly investigated this issue in a lab setting and found that the relative average performance of aided (what-if analysis) and unaided decision makers changes—indeed, in that study, switches—over time. Furthermore, the study task had an underlying linear trend with moderate randomness. Results indicated that, while what-if users performed better in the short run, unaided subjects performed better in the longer run (after period 18). In contrast, in another study using the same basic task, Davis and Tong [11] found that what-if users outperformed unaided subjects after a total of 16 decision periods, with no trend and with moderate-to-pronounced randomness. Therefore, while it is possible to hypothesize that there will be differences in the performance ranges of each strategy across varying time horizons, the actual direction of those differences cannot be determined. Thus, the following hypothesis is stated. H4 : The range of performance differences between marginalcosting and demand-tracking decision strategies will vary across changing time horizons. B. Monte Carlo Simulation Methodology The computer simulation developed in this paper implements demand-tracking and marginal-costing strategies via cost allocation tradeoff weights that mimic the strategies described earlier and are shown hereinafter. To meet changes in demand, demand-tracking cost allocation weights absorb costs primarily through increases and decreases in workforce, secondarily
60
IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART A: SYSTEMS AND HUMANS, VOL. 39, NO. 1, JANUARY 2009
Fig. 1. Cumulative cost as a function of randomness level.
through worker overtime or idle-time and least through inventory holding costs or backorder costs. Marginal-costing cost allocation weights reflect more the marginal costs of component costs and absorb costs primarily through inventory holding costs or backorder costs, secondarily through worker overtime and idle-time costs and lastly through increases or decreases in workforce levels. These weights mirror the actual decisionmaking subjects in laboratory experiments [10], [15], [19].
This paper operationalizes the relative cost performance of demand tracking versus marginal costing using a computational Monte Carlo model developed in Visual Basic to simulate the opposing strategies.1 Each simulation run represents 1000 independent decision-making trials using both strategies under incremental variations of task/environmental conditions. That is, for each value of the variable(s) being manipulated (e.g., level of demand variability and rate of demand increase), we perform 1000 independent simulation runs (trials), each trial being of time length t, which is the number of decision periods per trial. We use values of 24, 12, and 6 for t because it has been determined that decision-making subjects complete task learning around period 12 [23]. To illustrate the relationship of decision performance to environmental conditions, the performance differences (demand tracking–marginal costing) for each simulation case are mapped into response surfaces using the graphic capabilities of the R programming tool. Thus, each data point represents the mean cost outcome of 1000 Monte Carlo decision-making trials. Moreover, unless otherwise noted, each trial is 24 periods long since we found in our simulations that the patterns of performance differences between marginal costing and demand tracking do not change meaningfully after period 24. The resulting graphs (see Figs. 1 and 2) and response surfaces (see Figs. 3–11) are shown and discussed hereinafter. III. S IMULATION R ESULTS A. Environmental Uncertainty Environmental uncertainty is operationalized as a random fluctuation in future demand. The range of random fluctuation 1 Simulation
code available upon request to the corresponding author.
Fig. 2.
Cumulative cost as a function of demand trend.
simulated is 0 to 500. That is, we methodically increase the demand fluctuation from plus-or-minus 0 units to plus-orminus 500 units per period. Fig. 1 shows the cumulative total costs for demand tracking and marginal costing as random fluctuation increases. As shown in Fig. 1, the performance of demand tracking and marginal costing is each affected by environmental uncertainty; however, marginal costing fares quite well, whereas the performance of demand tracking deteriorates. As the randomness of future demand increases, the relative performance difference increases substantially, supporting hypothesis 1. Again, this result is consistent with Hoch and Schkade’s [27] observation that anchor and adjustment strategies may perform poorly in less predictable environments. In the cases shown in Fig. 1, demand tracking, in its attempt to track anticipated patterns of demand, overreacts to random fluctuations. The larger the degree of random fluctuation, the more pronounced the cost implications become. In contrast, marginal costing, by focusing more on marginal-cost implications, performs quite gracefully. B. Environmental Trend Environmental trend is operationalized in Fig. 2 as increasing rates of increase in demand (with no random fluctuation). The range of rates of increase simulated is 0 to 500. Such trends should likely provide more hospitable task environments for demand tracking. Since demand tracking may react inappropriately in less predictable environments, it should fare better when there are trends to track. The performance of marginal costing should not necessarily be harmed by trends, but nonetheless, it does not track trends expressly since it focuses on total cost through the lens of marginal costs of each of the three cost components. Fig. 2 bears this out. Both demand-tracking and marginal-costing cumulative costs naturally increase as the rate of increase in demand increases, because it costs more to supply more. Yet, demand tracking is more successful, supporting hypothesis 2. Fig. 3 shows the interplay of environmental uncertainty and trend, confirming the results of tests in which each attribute is varied independently. Simultaneous variation of these two basic aspects of the task environment shows clear patterns of performance differences. Axes are as follows: “R” randomness, “T” trend, and “CD” cost difference (demand-tracking cost–marginal-costing cost). The light-gray regions show conditions where marginal costing has superior performance relative to demand tracking. The dark-gray regions show where marginal costing is inferior to demand tracking. The cost
KOTTEMANN et al.: UNDERSTANDING DECISION-SUPPORT EFFECTIVENESS
61
Fig. 3. Cost difference (demand tracking versus marginal costing) at 24 periods.
Fig. 4. Cost differences with cyclical demand (demand tracking versus marginal costing) at 24 periods.
difference axis (z-axis) maps from the minimum (negative) to the maximum (positive) cost difference. The point of zero cost difference is given by where the response surface depicted shifts in shade of gray. Relatively speaking, demand tracking benefits most with low random flux and high trend (hypothesis 2); marginal costing benefits most with high random flux and low trend (hypothesis 1).
and high amplitude, demand tracking should further be favored (compared to low frequency and low amplitude) much as it was in the simulation tests involving an amplified linear trend. In Fig. 4, at 24 periods, higher frequencies favor marginal costing as the preferable decision-making strategy, whereas lower frequencies favor demand tracking. High frequency with high amplitude magnifies the performance benefits offered by a marginal-costing decision-making strategy, while low frequency with high amplitude strengthens the relative performance of demand tracking. Thus, hypotheses 3a and 3b are supported.
C. Cyclic Demand Environments Cyclic trends, a common environmental condition, are now considered. The demand-tracking and marginal-costing decision strategies are simulated for sine-wave demand patterns with varying frequency and amplitude. The range of frequencies simulated is 1 to 24; the range of amplitude is 0 to 2400. The results of varying randomness and trends described earlier offer analogs to predict the effects of frequency and amplitude on relative performance. High-frequency cycles that swing wildly are, in effect, similar to highly random patterns. A demand-tracking decision maker faced with a wildly swinging demand pattern is very similar to the demand-tracking decision maker who attempts to track random demand. Analogously, a marginal-costing decision maker should fare relatively well. Recalling hypotheses 3a and 3b, we expect that cyclic demand patterns with high frequency will favor a marginalcosting strategy. Conversely, low frequency will give the tracker a more gradual up/down trend to follow, and we expect demand tracking to outperform marginal costing in such cases. Interactions between amplitude and frequency can also be predicted. With high frequency and high amplitude, wild swings become more pronounced, further favoring marginal costing (compared to high frequency and low amplitude). With low frequency
D. Variations in Time Horizons Figs. 5 and 6 show decision-making performance differences between demand-tracking and marginal-costing strategies for randomness and trend across decreasing time horizons. In Fig. 5, the 24-period time horizon in Fig. 3 is rolled back to 12 periods. The response surface flattens. In Fig. 6, the time horizon is 6 periods; marginal costing dominates completely. Relative cost differences emerge over time with cyclical demand as well (see Figs. 7 and 8), with marginal costing doing better overall in the earlier periods. Recall that hypothesis 4 simply states an expectation of differences in the relative performance ranges across varying time horizons without specifying directionality. These results suggest that marginal costing is the more beneficial decision strategy across both environmental conditions in the short term (6 periods). However, it appears that a demand-tracking strategy can emerge as the more efficacious decision approach in conditions of a long-term demand trend. These results were replicated with a straightforward manipulation of randomness
62
IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART A: SYSTEMS AND HUMANS, VOL. 39, NO. 1, JANUARY 2009
Fig. 5. Cost difference (demand tracking versus marginal costing) at 12 periods.
Fig. 7. Cost differences with cyclical demand (demand tracking versus marginal costing) at 12 periods.
Fig. 6. Cost difference (demand tracking versus marginal costing) at 6 periods.
Fig. 8. Cost differences with cyclical demand (demand tracking versus marginal costing) at 6 periods.
and trend and, in the case of cyclical demand, supporting hypothesis 4 across both simulation cases. While the demand-tracking and marginal-costing strategies operationalized here are distinct, they also represent points on a continuum. Any given decision maker could lie in between, be more extreme, or, in fact, be a “hybrid” in the sense of adopting a combined or multimethod strategy. Such an approach was evaluated by Davis and Kottemann [26] in an experiment where
both marginal-costing and demand-tracking decision-support tools were made available to decision makers. The marginalcosting tool was a what-if spreadsheet. The demand-tracking tool was a formal tracking heuristic that made decision recommendations. The task environment involved trend and moderate randomness, so subjects should have benefited from the added availability of the demand-tracking tool. The experimental results confirmed this expectation. Subjects that better utilized the
KOTTEMANN et al.: UNDERSTANDING DECISION-SUPPORT EFFECTIVENESS
Fig. 9.
Cost difference (hybrid versus marginal costing) at 24 periods.
demand-tracking part of the hybrid tool outperformed subjects who primarily used the marginal-costing component of the hybrid tool. Detailed analysis supported the findings that the former subjects performed the task more like trackers than did the latter subjects. To demonstrate this point and to evaluate the robustness of the simulation approach, marginal costing is contrasted with a hybrid strategy (analogous to the multimethod decision-support method described previously). The hybrid strategy balances marginal-costing and demand-tracking tactics by making the three component cost tradeoff allocation weights equal and by incorporating an exponential smoothing of three periods of future demand forecasts. Figs. 9–11 show the relative performance of these two strategies over 24, 12, and 6 decisionmaking periods, respectively. In these figures, light gray indicates that marginal costing is a more effective strategy, and dark gray indicates that the hybrid strategy is better. The hybrid strategy dominates completely across all time horizons. Although the hybrid strategy represents a superior decision approach given the constraints of the aggregate production planning problem described here, this may not be representative of all decision environments. In the Davis and Kottemann [26] study described earlier, the demand-tracking tool itself, when allowed to perform the task autonomously, outperformed all subjects. The emergent problem then became how to provide decision makers with information that would enable them to identify and use the appropriate strategy and corresponding decision-support tool. Novel feedback methods had to be applied before subjects would begin to properly orient their use of decision-support tool options—in particular, some subjects were given cumulative feedback regarding how they would be performing if they had been following the tracking heuristics recommendations exclusively. These subjects, with unambigu-
63
Fig. 10. Cost difference (hybrid versus marginal costing) at 12 periods.
Fig. 11. Cost difference (hybrid versus marginal costing) at 6 periods.
ous feedback regarding the value of incorporating some degree of tracking behavior into their strategies while still having use of a what-if analysis tool, performed significantly better than other subjects. IV. D ISCUSSION AND C ONCLUSION To summarize, the simulation results confirm each of the stated hypotheses, supporting the overarching argument that the efficacies of specific decision-making strategies and supporting
64
IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART A: SYSTEMS AND HUMANS, VOL. 39, NO. 1, JANUARY 2009
tools are dependent on both environmental characteristics and the tool at hand. Conflicting experimental findings of the past become intelligible when considered in light of such task contingencies. Some past empirical studies have found that whatif analysis helps decision performance. Others have found that what-if analysis harms decision performance. Still, others have found no performance difference. By using Monte Carlo methods, we have focused on a dynamic task environment and systematically manipulated a key variable—demand, which is among the most important exogenous variables that decision makers must face [28]. The resulting simulations provide numerous sample combinations of varying demand types beyond what studies involving human subjects could feasibly cover. As shown here, the relative performance effects of aided (marginal-costing) and unaided (demand-tracking) decision making are quite sensitive to task environmental factors providing an explanation for more than two decades of mixed results from prior empirical research [3], [9]–[19]. While it is important to remember that our analysis involves a single task context, the larger and more important point to remember is that, even with a single task context, performance differences between unaided and aided decision making are emergent. Providing multiple decision-support tools to enable decision makers to address varying demand environments is a basic premise of the DSS philosophy [29]. What-if analysis, while extremely popular, is only one tool, but as prior lab results and the current results show, it is not effective in all circumstances. Because of constraints in human subject research, it has been impossible to systematically vary conditions in such a way as to develop a response surface corresponding to the sensitivity and scope provided in this analysis. Our simulation of a hybrid tool is consistent with the notion that providing decision makers with a combination of decision-support tools is advantageous. Practically speaking, however, helping decision makers select the proper tools for any given circumstance remains a key area for future research. The approach used here expands the possibilities of DSS research. Numerous manipulations are possible and await future research in the lab, in the field, and with additional simulations. By using the current task context, different task parameter values and different decision-making models can be investigated. In general, and extending from the method suggested by Sterman [30], researchers can determine the decision strategies engendered by assorted tools in lab and field studies and then conduct simulations to explore performance implications across a wide array of situations. The results of a wide set of such studies should provide guidelines to help decision makers choose appropriate tools for assorted circumstances. However, numerous studies with human subjects were performed before the simulation study reported here could be meaningfully conducted. One encouraging direction is to use automated methods for capturing human decision-making strategies. Fernland et al. [31] present an interesting approach for capturing tactical strategies of human decision makers using genetic programming and context-based reasoning. Further in the future, it may become possible to construct DSSs that are able to assess relevant contingencies and suggest
the tool or set of tools for a decision maker to use in any given situation. This presupposes a full-fledged robust knowledge base, which, in turn, implies the substantial research that must occur before such a knowledge base can be formalized. In the interim, research to identify individual decision-support tools or simple hybrids that are effective across a broad range of circumstances is in itself extremely useful. Such an interim effort will not only contribute to the longer range goal of highly intelligent DSS but also will offer immediate benefits to decision makers facing increasingly complex demand environments. It is important to point out the limitations of the present research. First, while we did use a task representative of the large class of recurring decision making under uncertainty and did manipulate key aspects of such task environments, there are other important classes of tasks to be investigated as well as novel characteristics of task environments, such as “broken leg cues,” in which one-time unexpected events occur which may dramatically alter the task environment. Exploring other decision-support methods in aggregate production planning, such as fuzzy methods [32], as well as other task domains, is certainly called for. Second, even the domain of recurring decision making under uncertainty can have other fundamental differences that may affect results. For example, our simulated task did not involve competitive agents. Task domains may vary with respect to the time-horizon effect in domains involving head-on competition. In chess, for example, if White dramatically outplays Black early in the match, White will enjoy an advantage over Black for subsequent moves. Sharda et al. [33] conducted a lab study of what-if analysis in a competitive business game context. What-if users gained an advantage over unaided subject in the first few periods of the game. In addition, while the relative performance difference did not change noticeably thereafter, the what-if users maintained the lead that was established early on. Thus, the present research can be furthered through simulation research extended to competing [34], [35] or symbiotic agents [36] as well as to categorically different task domains such as multiattribute choice [20], [37]. R EFERENCES [1] H. A. Simon, “A behavioral model of rational choice,” Q. J. Econ., vol. 69, no. 1, pp. 99–118, Feb. 1955. [2] R. M. Hogarth, Educating Intuition. Chicago, IL: Univ. Chicago Press, 2001. [3] M. A. Eierman, F. Niederman, and C. Adams, “DSS theory: A model of constructs and relationships,” Decis. Support Syst., vol. 14, no. 1, pp. 1– 26, May 1995. [4] J. W. Payne, J. R. Bettman, and E. J. Johnson, “Behavioral decision research: A constructive processing perspective,” Annu. Rev. Psychol., vol. 43, pp. 87–131, 1992. [5] R. M. Hogarth, “Beyond discrete biases: Functional and dysfunctional aspects of judgmental heuristics,” Psychol. Bull., vol. 90, no. 2, pp. 197– 217, 1981. [6] O. Svenson, “Cognitive strategies in a complex judgment task: Analyses of concurrent verbal reports and judgments of cumulated risk over different exposure times,” Org. Behav. Hum. Decis. Process., vol. 36, no. 1, pp. 1–15, Aug. 1985. [7] A. Tversky and D. Kahneman, “Judgment under uncertainty: Heuristics and biases,” Science, vol. 185, no. 4157, pp. 1124–1131, Sep. 1974. [8] J. D. Sterman, “Misperceptions of feedback in dynamic decision making,” Org. Behav. Hum. Decis. Process., vol. 43, no. 3, pp. 301–335, Jun. 1989. [9] I. Benbasat and A. S. Dexter, “Individual differences in the use of decision support aids,” J. Acc. Res., vol. 20, no. 1, pp. 1–11, 1982.
KOTTEMANN et al.: UNDERSTANDING DECISION-SUPPORT EFFECTIVENESS
[10] F. D. Davis and J. E. Kottemann, “User perceptions of decision support effectiveness: Two production planning experiments,” Decis. Sci., vol. 25, no. 1, pp. 57–78, Jan. 1994. [11] J. G. Davis and K. K. W. Tong, “The effects of forecast accuracy and what-if analysis on decision making,” in Proc. ASAC Conf., 1997, pp. 80–90. [12] B. L. Dos Santos and M. L. Bariff, “A study of user interface aids for model-oriented decision support systems,” Manag. Sci., vol. 34, no. 4, pp. 461–468, Apr. 1988. [13] J. Fripp, “How effective are models?” Int. J. Manage. Sci.: Omega, vol. 13, no. 1, pp. 19–28, 1985. [14] M. D. Goslar, G. I. Green, and T. H. Hughes, “Decision support systems: An empirical investigation,” Decis. Sci., vol. 17, no. 1, pp. 79–91, 1986. [15] J. E. Kottemann and W. E. Remus, “The effects of planning horizon on the effectiveness of what-if analysis,” Int. J. Manage. Sci.: Omega, vol. 20, no. 3, pp. 295–301, 1992. [16] J. E. Kottemann, F. D. Davis, and W. E. Remus, “Computer assisted decision making: Performance, beliefs, and the illusion of control,” Org. Behav. Hum. Decis. Process., vol. 57, no. 1, pp. 26–37, Jan. 1994. [17] G. H. Van Bruggen, A. Smidts, and B. Wierenga, “Improving decision making by means of a marketing decision support system,” Manage. Sci., vol. 44, no. 5, pp. 645–658, May 1998. [18] G. L. Lilien, A. Rangaswamy, G. H. Van Bruggen, and K. Starke, “DSS effectiveness in marketing resource allocation decisions: Reality vs. perception,” Inf. Syst. Res., vol. 15, no. 3, pp. 216–235, Sep. 2004. [19] J. E. Kottemann and W. E. Remus, “What-if analysis, and performance,” in Environments for Supporting Decision Processes, H. G. Sol and J. Vecsenji, Eds. Amsterdam, The Netherlands: North Holland, 1991, pp. 203–214. [20] J. W. Payne, E. J. Johnson, J. R. Bettman, and E. Coupey, “Understanding contingent choice: A computer simulation approach,” IEEE Trans. Syst., Man, Cybern., vol. 20, no. 2, pp. 296–309, Mar./Apr. 1990. [21] C. C. Holt, F. Modigliani, and J. F. Muth, “Derivation of a linear decision rule for production and employment,” Manage. Sci., vol. 2, no. 2, pp. 159– 177, Jan. 1956. [22] C. C. Holt, F. Modigliani, J. F. Muth, and H. A. Simon, Planning—Production, Inventories, and Workforce. Englewood Cliffs, NJ: Prentice-Hall, 1960. [23] W. E. Remus and J. E. Kottemann, “Semi-structured recurring decisions: An experimental study of decision making models and some suggestions for DSS,” MIS Q., vol. 11, no. 2, pp. 233–243, 1987. [24] W. E. Remus and J. E. Kottemann, “Anchor-and-adjustment behavior in a dynamic decision environment,” Decis. Support Syst., vol. 15, no. 1, pp. 63–74, Sep. 1995. [25] E. C. Poulton, Tracking Skill and Manual Control. New York: Academic, 1974. [26] F. D. Davis and J. E. Kottemann, “Determinants of decision rule use in a production planning task,” Org. Behav. Hum. Decis. Process., vol. 63, no. 2, pp. 253–269, Aug. 1995. [27] S. J. Hoch and D. A. Schkade, “A psychological approach to decision support systems,” Manage. Sci., vol. 42, no. 1, pp. 51–64, Jan. 1996. [28] L. J. Krajewski and L. P. Ritzman, Operations Management: Strategy and Analysis. Reading, MA: Addison-Wesley, 1999.
65
[29] R. H. Sprague, Jr., “DSS in context,” Decis. Support Syst., vol. 3, no. 3, pp. 197–202, Sep. 1987. [30] J. D. Sterman, “Testing behavioral simulation models by direct experiment,” Manage. Sci., vol. 33, no. 12, pp. 1572–1592, Dec. 1987. [31] H. K. G. Fernland, A. J. Gonzalez, M. Georgiopoulos, and R. F. Demara, “Learning tactical human behavior through observation of human performance,” IEEE Trans. Syst., Man, Cybern. A, Syst., Humans, vol. 36, no. 1, pp. 128–140, Feb. 2006. [32] R. Y. K. Fung, J. Tang, and D. Wang, “Multiproduct aggregate production planning with fuzzy demands and fuzzy capacities,” IEEE Trans. Syst., Man, Cybern. A, Syst., Humans, vol. 33, no. 3, pp. 302–313, May 2003. [33] R. Sharda, S. H. Barr, and J. C. McDonnell, “Decision support system effectiveness: A review and an empirical test,” Manage. Sci., vol. 34, no. 2, pp. 139–159, Feb. 1988. [34] K. M. Sim and S. Y. Wang, “Flexible negotiation agent with relaxed decision rules,” IEEE Trans. Syst., Man, Cybern. A, Syst., Humans, vol. 34, no. 3, pp. 1602–1608, Jun. 2004. [35] N. Garg, D. Grosu, and V. Chaudry, “Antisocial behavior of agents in scheduling mechanisms,” IEEE Trans. Syst., Man, Cybern. A, Syst., Humans, vol. 37, no. 6, pp. 946–954, Nov. 2007. [36] E. Toru, K. Hirasawa, J. Hu, and N. Ota, “A study of evolutionary multiagent models based on symbiosis,” IEEE Trans. Syst., Man, Cybern. A, Syst., Humans, vol. 36, no. 1, pp. 179–193, Feb. 2006. [37] H. Deng and C.-H. Yeh, “Simulation-based evaluation of defuzzificationbased approaches to fuzzy multiattribute decision making,” IEEE Trans. Syst., Man, Cybern. A, Syst., Humans, vol. 365, no. 5, pp. 968–977, Sep. 2006.
Jeffrey E. Kottemann received the Ph.D. from the University of Arizona, Tucson. He is currently a Professor of information and decision sciences at Salisbury University, Salisbury, MD.
Kathleen M. Boyer-Wright received the Ph.D degree from George Washington University, Washington, DC. She is currently an Associate Professor of information and decision sciences at Salisbury University, Salisbury, MD.
Joel F. Kincaid received the Ph.D. degree from North Carolina State University, Raleigh. He is currently an Associate Professor of economics at Winston-Salem State University, Winston-Salem, NC.
Fred D. Davis received the Ph.D. degree from the Massachusetts Institute of Technology, Cambridge. He is currently a Professor of information systems at the University of Arkansas, Fayetteville.
