Capturing Non-linear Judgment Policies Using

Capturing Non-linear Judgment Policies Using Decision Tree Models of Classification Behavior Daniel Lafond1, Benoît R. Vallières2, François Vachon2, Marie-Ève St-Louis2, & Sébastien Tremblay2 1 Thales Research & Technology, Québec, Canada; 2Université Laval, Québec, Canada Policy capturing is a decision analysis method that typically uses linear statistical modeling to estimate the basis of expert judgments. Using more flexible data mining algorithms may yield more accurate models or instead result in poor functional estimations. The objective of this study is to test the effectiveness of a decision tree induction algorithm for policy capturing in comparison to the standard linear approach. We examined human classification behavior using a simulated naval air-defense task in order to empirically compare the C4.5 decision tree algorithm to linear regression on their ability to capture individual decision policies. The pattern of results shows that C4.5 outperformed linear regression in terms of goodness-of-fit and cross-validation accuracy. Results also show that the decision tree models of individuals’ judgment policies actually classified contacts more accurately than their human counterparts. We conclude that nonlinear policy capturing can yield useful models for training and decision support applications. INTRODUCTION In various decision making, surveillance and supervisory control contexts, understanding the psychological processes involved can help improve training procedures and cognitive systems engineering designs to better support the operator at work (Bryant, 2007; Gonzalez, 2005; Morrison, Kelly, Moore, & Hutchins, 1998). One approach shown to be particularly useful for decision analysis is policy capturing (Brehmer, 1994; Cooksey, 1996), which allows the functional estimation of an individual’s judgment policy from past decisions without the use of verbal protocols. Indeed, there is an inherent difficulty in understanding and explaining one’s own decision making process, partly because expertise can include procedural and implicit knowledge (Berry, 1987) and this constitutes a major obstacle to verbal protocol analysis (Ericsson & Simon, 1984). Policy capturing involves using statistical models or machine learning algorithms to approximate the judgments of a decision maker (Brehmer & Brehmer, 1988; Kim, Chung, & Paradice, 1997). Individuals must first make decisions on a series of cases that have different attributes. Judgment policies are then inferred from their decision behavior. For example, when using regression models for policy capturing, the judgments (i.e., the dependent variable) are regressed on the attributes. This procedure yields a weighted model describing a judgment policy in terms of cues, relative cue weights, and the specific mathematical function relating the cues to the judgment (Dhami & Harries, 2001; Smith, Scullen, & Barr, 2002). The key strengths of the policy capturing method are that it is not intrusive and does not presume that decision makers are fully aware of—or able to verbally express—the basis for their judgments. This approach has the advantage of functionally capturing procedural and implicit knowledge that could not be expressed otherwise, and constitutes a practical means to identify useful constraints for designing operator interfaces and decision support systems. While the vast majority of policy capturing studies has opted for a linear modeling approach, some studies have

shown that non-linear decision rules can provide psychologically realistic decision models, such as “fast-and-frugal” decision trees (Backlund, Bring, Skaner, Strender, & Montgomery, 2009; Dhami & Harries, 2001; Smith & Gilhooly, 2006) or logical rules selected using a genetic search algorithm (Rothrock & Kirlik, 2003). Capturing decision policies using non-linear modeling techniques has the potential to yield more accurate models, but also risks leading to poor functional estimations that simply overfit the training data (Marewski & Gigerenzer, 2012; Martignon, Katsikopoulos, & Woike, 2008). The purpose of the present study is thus to test if a nonlinear policy capturing methodology can achieve a better descriptive and predictive accuracy compared to the linear approach. This work focuses on naturalistic decision making in dynamic time-pressured and stressful situations such as maritime decision making. Indeed, this context provides a highly relevant use case for the application of policy capturing for the design of decision support or training solutions (see also Bryant, 2007; Rothrock & Kirlik, 2003). Naval Air-Defense Task and Microworld A radar contact classification task in naval air-defense was selected for the present policy capturing study. This task typically occurs within the broader context of a Threat Evaluation and Weapons Assignment process (TEWA; see Roux & van Vuuren, 2007). Liebhaber and Smith (2000) examined the cognitive aspects of the TEWA process of naval air-defense officers. They identified twenty-two relevant factors, the most important being signal emissions, course, speed, altitude, point of origin, Identification Friend or Foe (IFF) response, flight profile, intelligence information, and distance from the detector. Each attribute had a range of values that were associated with one of three categories: Friendly, Neutral, and Hostile. Ozkan, Rowe, Calfee and Hiles (2005) suggested that attributes fall into two categories: Those affecting the time a contact could take to reach the home ship and those affecting the intrinsic suspiciousness of the contact. Ozkan et al. (2005) developed agent-based simulations of naval air-defense. The

simulated Combat Information Center (CIC) watchstanders were implemented as agents who detected, reported, classified and prioritized contacts and selected appropriate actions. To model the cognitive aspects of contact classification, Ozkan et al. (2005) used linear models to categorize contacts as friendly, unknown neutral, suspect or hostile, based on a weighted sum of six numeric factors: closing course, speed, altitude, electronic signal, origin, and IFF mode. Chalmers, Easter and Potter (2000) suggested that threats can be assessed in terms of (1) their functional distance (i.e., time before hit), and (2) their functional state. The functional state of menace attributed to a contact depends on its classification (i.e., unresolved, suspect, hostile intention) and identification (i.e., missile/aircraft/torpedo). In particular, the allegiance of an entity, i.e., friendly/neutral/hostile, is of prime interest for threat analysis (Roy, Paradis, & Allouche, 2002). While this type of classification task could formally be reproduced using a minimalist static experiment structure as a set of isolated successive trials, a more naturalistic and ecologically valid approach was chosen using a microworld as a testbed to assess the method’s potential in a realistic context. Microworlds are simulated task environments designed for the study of decision making in complex and dynamic situations. They provide a good compromise between experimental control and realism (Brehmer & Dörner, 1993). A number of different microworlds have been previously used to study individual and team cognition in naval air-defense. Prominent examples include Argus (Schoelles & Gray, 2001), the DDD simulation (Miller, Young, Kleinman, & Serfaty, 1998), TANDEM (Dwyer, Hall, Volpe, Cannon-Bowers, & Salas, 1992) and TITAN (NTT Systems; see Bryant, 2007). Most of these microworlds focused on threat assessment and did not require taking defensive measures in real time or assessing the temporal immediacy of threatening contacts. The purpose-built microworld used in the present study is called the Simulated Combat Control System (S-CCS; see, e.g., Vachon, Vallières, Jones, & Tremblay, 2012). This microworld allows studying the cognitive aspects of decision support in a complex task requiring the management of multiple functions: classifying threats, assessing threat immediacy and engaging hostile contacts. S-CCS is a functional simulation of anti-air warfare command and control practiced aboard the Canadian Navy’s Halifax class frigates. The operator plays the role of tactical coordinator, who must monitor the operational space, be aware of significant changes, keep focus on critical threats and coordinate responses to threats. This microworld has been used to provide a comprehensive assessment of the cognitive support of different prototype decision support solutions (Lafond, Vachon, Rousseau, & Tremblay, 2010).

linear regression method. Although a linear approach may be more robust to noise, it may lead to underfitting due to its lack of flexibility. Of these two methods, only the decision tree approach can capture a non-exhaustive search for attributes for which there is increasing evidence in dynamic time-pressured task contexts (e.g., Bryant, 2007; Lafond et al., 2009; Marewski & Gigerenzer, 2012; Rothrock & Kirlik, 2003). METHOD Participants Sixty university students (mean age: 23.4 years, SD: 4.03) from a wide variety of backgrounds (mainly Social Sciences and Engineering), including 29 men and 31 women, received $20 compensation for their participation. Apparatus The naval air-defense task used in this study was presented using the S-CCS microworld on a standard personal computer with a flat screen monitor, a keyboard, and a mouse. S-CCS is a low-fidelity computer-controlled simulation of single ship naval above water warfare, designed for cognitive engineering research purposes. This microworld enables us to generate displays, develop scenarios and record the status of all objects and events. The display consists of a geospatial interface, similar to typical radar systems. Figure 1 shows the S-CCS interface. The spatial interface consisted of a black screen which included a central point corresponding to the ship and four concentric circles defining different kill probability regions. Contacts could appear anywhere on the radar as dots with a green squared contour. Contacts moved on the radar at various speeds, which was proportional to the length of the line attached to it. The ship was hit when a hostile contact reached the central point of the screen.

Figure 1. S-CCS interface. 1) Parameters, 2) Radar screen, 3) Action buttons.

Hypothesis We used the S-CCS microworld to collect operator judgments in order to test the effectiveness of a decision tree induction algorithm for policy capturing in comparison to a linear regression approach. Our hypothesis is that a decision tree method for policy capturing will describe and predict human decision making behavior more effectively than a

Design and Procedure Participants were required to perform three functions related to TEWA. First they had to evaluate the threat level of the contacts (or tracks) on the radar by considering five parameters: (1) origin (hostile territory or not), (2) altitude (low, high), (3) IFF (friendly, neutral or enemy), (4) detection

of weapon systems (positive, negative), (5) military electronic emission (positive, negative). Other parameters were present on the display but not relevant to the categorization task. A contact could be associated to one of three levels of threat depending on the number of threatening cues (0-1 = nonhostile, 2-3 = uncertain, 4-5 = hostile). Second, if the contact was evaluated as hostile, operators needed to assess the threat immediacy by calculating the temporal proximity of the contact (1 = under 15 s, 2 = between 15-30 s, 3 = above 30 s) based on two parameters (time to closest point of approach, or TCPA, and closest point of approach in units of time, or CPAUT). Third, the operators had to engage hostile contacts with antimissiles to prevent the ship from being hit. Participants executed these three tasks using a mouse to select contacts and to click on the classification, threat immediacy and engage buttons. Track attributes appeared in the upper left panel of the screen when a specific contact had been hooked. The firing device required waiting for the antimissile to intercept the target before being able to fire at the next target. Contacts were presented in groups, referred to as bursts. A burst is a set of 27 contacts (11 non hostile, 8 uncertain, 8 hostile) varying in speed and trajectory in the radar. A burst lasted four minutes. Participants took part in a practice session with two practice blocks of four bursts each, followed by a test session with three experimental blocks of four bursts. Feedback on response accuracy was provided after each burst. Instructions to participants specified that the classification task (non-hostile/uncertain/hostile) was mandatory and that accuracy was particularly critical to avoid neutral or friendly fire. Furthermore, participants were required to re-classify contacts if they observed a change in their behavior on the radar (change in speed or heading). Importantly, the three categories in this classification task were non-linearly separable (Medin & Schwanenflugel, 1981). This means that a linear regression cannot perfectly discriminate members of each category. While this context can potentially favor the non-linear approach in reason of its superior flexibility, linear regression may still provide a more accurate functional model of human decision policies if it is more in line with the operators’ bounded cognitive capabilities. Comparing two such methods may also allow identifying the presence of a mix of strategies across participants (e.g., Bryant, 2007). Policy Capturing Techniques In the context of the present study, we compared a linear regression approach to a non-linear decision tree modeling method. Decision trees are particularly relevant in naturalistic decision making contexts since they fall to some extent into the category of fast-and-frugal heuristics which are well-suited to describe human cognition under time pressure (Luan, Schooler, & Gigerenzer, 2011). Using the RapidMiner software (Rapid-I, GmbH), we created 120 distinct policy capturing analyses (i.e., 60 participants × 2 methods). Because threat assessment was categorical, we used logistic linear regression rather than multiple linear regression for policy capturing (Backlund et al., 2009; Dhami & Harries, 2001; Smith & Gilhooly, 2006). In practice, a single-layer feedforward neural network (equivalent to linear logistic

regression; Sharma, Sandooja, & Yadav, 2013) was used in RapidMiner to derive linear models of individual judgment policies (the W-MLP algorithm was used with the hidden layer parameter set to zero). Decision trees were derived using the C4.5 algorithm (Quinlan, 1993; labeled W-J45 in RapidMiner), with the M parameter (i.e., minimum instances per leaf) set to 2 in order to avoid including unique decisions in the model. Finally, 10-fold cross-validation tests were performed for each of the 120 models to estimate the models’ predictive accuracy and verify that higher model fits were not simply attributable to overfitting of noise in the data due to greater model flexibility. RESULTS The average classification accuracy of participants on the task was high, with a mean percent correct of 95.1% (SD = 4.42%). Table 1 shows how the 4.9% classification errors were distributed across error types. Table 1. Distribution of classification errors (in %) Perceived hostility Non-Hostile Uncertain Hostile Total

True hostility Non-Hostile Uncertain 34.8 26.2 1.7 24.3 27.9 59.1

Hostile 2.9 10.0 12.9

Total % 37.7 36.2 26.1 100

Linear Logistic Regression Models The goodness-of-fit of the 60 linear logistic regression models, defined as the concordance with the participants’ responses, was on average 89.10% (SD = 2.17%; see Table 2). When using these models to classify the radar contacts in the simulated task, model performance was on average 89.99% (SD = 4.46%), which was significantly inferior to human performance, t(118) = 154.6, p < .001. Nonetheless, a strong correlation was observed between the models’ classification accuracy and human accuracy, r(58) = 0.83, p < .001. Decision Tree Models Sixty decision trees were derived from participants’ responses. Although virtually all decision trees included the five parameters relevant to the classification task, many decision paths ended in “leaves” (a response category) that required fewer than five “feature tests”. Tree complexity, defined as the total number of decision nodes and leaves, varied between 15 and 41 (M = 29.13, SD = 4.79). In comparison, a decision tree representing an exhaustive search strategy would have a complexity of 63, while the simplest 100% correct decision tree would have a complexity of 43. Figure 2 shows a decision tree representing the typical result (with a complexity of 29) and Figure 3 illustrates the simplest decision tree derived. Below each feature test (round node), the branch on the left refers to a value of 0 (nonthreatening) and the branch on the right refers to a value of 1 (potentially threatening). The numbers within each decision leaf (square node) correspond to the number of correct/incorrect matches to the participant’s response.

Figure 2. Representative decision tree derived from participants’ classification behavior. N-H = Non-hostile, U = Uncertain, H = Hostile.

Figure 3. Simplest decision tree derived from a participant’s classification behavior with number of correct/incorrect classifications in each leaf.

A strong correlation was observed between the models’ classification accuracy and human accuracy, r(58) = 0.88, p < .001. We found a significant correlation between tree complexity and the decision trees’ classification accuracy on the task, r(58) = 0.28, p = .031. Furthermore, tree complexity was significantly correlated with human classification accuracy, r(58) = 0.26, p = .045. Table 2 compares linear regression modeling outcomes with decision tree modeling results in terms of goodness-of-fit with participants’ decision patterns, in terms of prediction accuracy on “held-out” data samples (cross-validation score) and in terms of performance of individual models against the “ground truth”. It shows that the decision tree models provided a significantly better descriptive and predictive accuracy in relation to human decisions and could be used to classify contacts more accurately than the linear models. This pattern actually holds true for all 60 individual participants.

concurrent tasks (monitoring critical changes and taking defensive measures). Findings supported our hypothesis, showing that the decision tree models derived were systematically superior to the linear models both in terms of their descriptive accuracy and predictive (cross-validation) accuracy. In general, simpler decision tree models tended to be less accurate when using them to classify the radar contacts, and decision tree complexity significantly correlated with human classification accuracy. Hence, in the present context, performance decreased with distance from the optimal tree of complexity 43 (i.e., all 60 participants were overly frugal to different extents). Interestingly, the decision tree models derived from the participants’ judgments actually yielded a higher proportion of correct classifications than the participants themselves. This result mirrors a classic finding in policy capturing research that linear models of decision makers’ judgments can actually outperform decision makers themselves (Dawes & Corrigan, 1974; Goldberg, 1971; Karelaia & Hogarth, 2008; Meehl, 1954), except that in the present case this result was achieved using a non-linear approach. This result demonstrates that non-linear models can be useful tools for “judgmental bootstrapping”, i.e. inferring robust decision rules based on experts’ decisions (Armstrong, 2001), which are not subject to the inconsistencies which can occur in humans decision patterns due to various factors such as fatigue or distraction. Bryant (2007) previously found that a portion of participants would tend to use fast-and-frugal heuristics in a naval air-defense task, while another portion would perform a more exhaustive information search. It seems that the task conditions used in the present study favored a single type of strategy, namely the non-linear (and non-exhaustive) approach. However, future work should compare different types of non-linear policy capturing methods, such as the matching heuristic (Martignon, Katsikopoulos & Woike, 2008) or decision rules inferred using a genetic algorithm (Rothrock & Kirlik, 2003). The policy capturing method presented herein is not restricted to empirically-controlled contexts such as microworlds and, hence, can lead to useful applications in the areas of training and decision support. Expert knowledge could be formally captured and used to provide specific guidance and insights to trainees in an intelligent tutoring context. In the same vein, decision tree models from the best experts in various domains could help extracting strategies

Table 2 Performance comparison of two policy capturing methods across three metrics Method Goodness-of-fit Cross-validation Model accuracy with participants (prediction) accuracy on task Linear M = 89.10%, M = 88.47%, M = 89.99%, Regression SD = 2.17% SD = 2.45% SD = 4.46% Decision trees t-test *p < .001

M = 96.00%, SD = 2.28% t(118) = 16.96*

M = 95.28%, SD = 2.66% t(118) = 14.60*

M = 98.51%, SD = 4.01 % t(118) = 11.01*

Finally, we also found that the decision tree models’ accuracy on the task (M = 98.51%) was significantly greater than human accuracy (M = 95.01%): t (118) = 188.42, p < .001. This was not the case for the linear models. DISCUSSION The goal of the present study was to test if a non-linear policy capturing technique could yield more accurate models of individual decision policies than the standard linear method in the context of a dynamic naval air-defense simulation involving high degrees of time pressure and task load. In this type of context, we hypothesized that a non-exhaustive and non-linear process exemplified by decision trees would be more effective for policy capturing approach. Participants generally mastered the classification task despite the high time-pressure and the presence of competing

that may in turn serve as the bases for new training programs. Furthermore, policy capturing could be used to assess operator expertise, understand errors, and predict potential mistakes. Inferring an operator’s decision model allows outlining the key information elements underlying each type of decision (e.g., leaf in a decision tree) for which the rate of correct/incorrect responses is known. That way, it is possible to localize error sources (i.e., problematic leaves) and then target the operator’s decision support needs for specific situations. Finally, judgmental bootstrapping could be quite valuable in a decision support context. This may be achieved by monitoring operator assessments in real-time, comparing them with the inferred decision policy, and generating alerts in cases of mismatch. This non-intrusive “shadowing” process, previously outlined in Lafond, Tremblay, and Banbury (2013) constitutes a promising future avenue for improving humanmachine system effectiveness without putting the user in the back seat and without imposing an additional burden on the operator. REFERENCES Armstrong, J. S. (2001). Judgmental bootstrapping: Inferring experts’ rules for forecasting. In J. S. Armstrong (Eds.) Principles of Forecasting: A Handbook for Researchers and Practitioners (pp. 171–192). Norwell, MA: Kluwer. Backlund, L.G., Bring, J., Skaner, Y., Strender, L.E., & Montgomery, H. (2009). Improving fast and frugal in relation to regression analysis: test of 3 models for medical decision making. Medical Decision Making, 29, 140-148. doi: 10.1177/0272989X08326091 Berry, D. C. (1987). The problem of implicit knowledge. Expert Systems, 4, 144–151. doi:10.1111/j.1468-0394.1987.tb00138.x Brehmer, B. (1994). The psychology of linear judgment models. Acta Psychologica, 87, 137–154. doi:10.1016/0001-6918(94)90048-5 Brehmer, A., & Brehmer, B. (1988). What have we learned about human judgment from thirty years of policy capturing?. Advances in psychology, 54, 75–114. doi:10.1016/S0166-4115(08)62171-8 Brehmer, B., & Dörner, D. (1993). Experiments with computer simulated microworlds: Escaping both the narrow straits of the laboratory and the deep blue sea of the field study. Computers in Human Behavior, 9, 171– 184. doi:10.1016/0747-5632(93)90005-D Bryant, D. (2007). Classifying simulated air threats with fast and frugal heuristics. Journal of Behavioural Decision Making, 20, 37–64. doi: 10.1002/bdm.540 Chalmers, B. A., Easter, J. R., & Potter, S. S. (2000). Decision-centred visualisations for tactical decision support on a modern frigate. Proceedings of the Command and Control Research and Technology Symposium, Monterey, CA. Cooksey, R. W. (1996). Judgment analysis: Theory, methods, and applications. San Diego, CA: Academic Press. Dawes, R. M., & Corrigan, B. (1974). Linear models in decision making. Psychological Bulletin, 81, pages 95–106. doi:10.1037/h0037613. Dhami, M. K., & Harries, C. (2001). Fast and frugal versus regression models of human judgement. Thinking and reasoning, 7, 5–27. doi:10.1080/13546780042000019 Dwyer, D., Hall, J., Volpe, C., Cannon-Bowers, J. A., & Salas, E. (1992). A performance assessment task for examining tactical decision-making under stress (Report No. 92-002). Orlando, FL: NTSC. Ericsson, K. A., & Simon, H. A. (1984). Protocol analysis: Verbal reports as data. Cambridge, MA: MIT Press. Goldberg, L. R. (1971). Five models of clinical judgment: An empirical comparison between linear and nonlinear representation of the human inference process. Organizational Behavior and Human Performance, 6, 458–479. doi:10.1016/0030-5073(71)90028-6 Gonzalez, C. (2005). Decision support for real-time, dynamic decisionmaking tasks. Organizational Behavior and Human Decision Processes, 96, 142–154. doi:10.1016/j.obhdp.2004.11.002

Karelaia, N., & Hogarth, R. (2008). Determinants of linear judgment: A metaanalysis of lens model studies. Psychological Bulletin, 134, 404–426. doi:10.1037/0033-2909.134.3.404 Lafond, D., Champagne, J., Hervet, G., Gagnon, J.-F., Tremblay, S., & Rousseau, R. (2009). Decision analysis using policy capturing and process tracing techniques in a simulated naval air-defense task. Proceedings of 53rd Meeting of the Human Factors and Ergonomics Society (pp. 1220-1224). Santa Monica, CA: Human Factors and Ergonomics Society. Lafond, D., Tremblay, S., & Banbury, S. (2013). Cognitive shadow: A policy capturing tool to support naturalistic decision making. Proceedings of the IEEE Conference on Cognitive Methods in Situation Awareness and Decision Support, San Diego, CA. Lafond, D., Vachon, F., Rousseau, R., & Tremblay, S. (2010). A cognitive and holistic approach to developing metrics for decision support in command and control. In D. Kaber & G. Boy (Eds.), Advances in Cognitive Ergonomics (pp. 65–75). Danvers, MA: CRC Press. Liebhaber, M. J., & Smith, P. (2000, June). Naval air defense threat assessment: Cognitive factors and model. Command and Control Research and Technology Symposium, Monterey, CA. Luan, S., Schooler, L. J., & Gigerenzer, G. (2011). A signal detection analysis of fast-and-frugal trees. Psychological Review, 118, 316–338. doi: 10.1037/a0022684 Marewski, J. N., & Gigerenzer, G. (2012). Heuristic decision making in medicine. Dialogues in Clinicial Neuroscience, 14, 77–89. Martignon, L., Katsikopoulos, K. V., & Woike, J. (2008). Categorization with limited resources: A family of simple heuristics. Journal of Mathematical Psychology, 52, 352-361. doi:10.1016/j.jmp.2008.04.003 Medin, D. L., & Schwanenflugel, P. J. (1981). Linear separability in classification learning. Journal of Experimental Psychology: Human Learning and Memory, 7, 355–368. doi: 10.1037/0278-7393.7.5.355 Meehl, P. E. (1954). Clinical versus statistical prediction: A theoretical analysis and a review of the evidence. Minneapolis, MN: University of Minnesota Press. Miller, D. L., Young, P., Kleinman, D., & Serfaty, D. (1998). Distributed dynamic decision-making simulation phase I: Release notes and user’s manual. Woburn, MA: Aptima. Morrison, J. G., Kelly, R. T., Moore, R. A., & Hutchins, S. G. (1998). Implications of decision-making research for decision support and displays. In J. A. Cannon-Bowers and E. Salas (Eds.), Making Decisions Under Stress: Implications for Individual and Team Training (pp. 91– 113). Washington, DC: APA. Ozkan, B., Rowe, N. C., Calfee, S. H., & Hiles, J. E. (2005). Three simulation models of naval air defense. Proceedings of the International Command and Control Research and Technology Symposium (Paper I-94). Quinlan, J. R. (1993). C4.5: Programs for Machine Learning. San Francisco, CA: Morgan Kaufmann Publishers. Rothrock, L., & Kirlik, A. (2003). Inferring rule-based strategies in dynamic judgment tasks: Toward a noncompensatory formulation of the lens model. IEEE Transactions on Systems, Man, and Cybernetics—Part A: Systems and Humans, 33, 58–72. doi:10.1109/TSMCA.2003.812601 Roux, J. N., & van Vuuren, J. H. (2007). Threat evaluation and weapon assignment decision support: A review of the state of the art, ORiON, 23, 151–186. doi:10.5784/23-2-54 Roy, J., Paradis, S., & Allouche, M. (2002). Threat evaluation for impact assessment in situation analysis systems. In I. Kadar (Eds.), Proceedings of SPIE, vol. 4729: Signal Processing, Sensor Fusion, and Target Recognition XI (pp. 329-341). Orlando, FL: SPIE. doi:10.1117/12.477618 Schoelles, M. J., & Gray, W. D. (2001). Argus: A suite of tools for research in complex cognition. Behavior Research Methods, Instruments, & Computers, 33, 130–140. doi:10.3758/BF03195358 Sharma, A., Sandooja, B., & Yadav, D. (2013). Extreme Machine Learning: Feed Forward Networks. International Journal of Advanced Research in Computer Science and Software Engineering, 3(8), 1366–1371. Smith, L., & Gilhooly, K. (2006). Fast and frugal versus regression models of decisionmaking: The case of prescribing for depression. Applied Cognitive Psychology, 20, 265–274. doi: 10.1002/acp.1189 Smith, S.E. Scullen, & Barr, S.H. (2002). Conducting studies of decision making in organizational contexts: A tutorial for policy-capturing and other regression-based techniques. Organizational Research Methods, 5, 388–414. doi:10.1177/109442802237117 Vachon, F., Vallières, B. R., Jones, D. M., & Tremblay, S. (2012). Nonexplicit change detection in complex dynamic settings: What eye movements reveal. Human Factors, 54, 996-1007.