Models of Human Expertise as Blueprints for Cognitive Engineering: Applications to ..... sufficiently mature to support engineering applications. This work offers a ...
Models of Human Expertise as Blueprints for Cognitive Engineering: Applications to Landmine Detection James J. Staszewski Department of Psychology Carnegie Mellon University Landmines pose a major military threat as well as a serious humanitarian problem. The difficulty of detecting buried landmines, especially modern landmines with minimal metallic content, contributes heavily to the problem. The two projects described here took a cognitive engineering approach to improve detection capability via development and implementation of scientifically principled operator training. Each analyzed and modeled the landmine detection skills of expert users of two different handheld detection systems, the PSS-12, the U.S. Army standard equipment, and the PSS-141, an advanced technology system under development at the time of this work. Model-based training programs were developed and tested. Both proved highly effective for developing detection skills, producing the greatest increases in detection rates against the most challenging targets. The US Army has adopted and now uses both programs. Results demonstrate the practical utility of information-processing models of expertise for designing instruction and developing important human skills. 1. INTRODUCTION Several basic research studies suggest that informationprocessing models of expertise are useful resources for designing instruction (Biederman & Schiffrar, 1987; Chase & Ericsson, 1982; Wenger & Payne, 1995; Staszewski, 1988, 1990). Questions remain about the practical utility of using expert models for this purpose, however (Alexander, 2003; Hatano & Oura, 2003; Patel, Kaufman, & Magder, 1996). This report addresses this issue by summarizing the results and impacts of two applied projects. Their common goal was to develop training for operators of specific handheld landmine detection systems that would improve upon existing training and increase landmine detection capability. Both projects employed a cognitive engineering approach, which used equipmentspecific models of expert skill to guide the design of the respective training programs. The first step of each project identified experts by empirical testing of operators who had extensive experience with specific handheld landmine detection equipment. Informationprocessing analyses of most proficient operators were performed and findings were used to develop cognitive process models of the experts' skills. The content and structure of the models then served as blueprints for designing equipmentspecific training programs for novices. Each training program was administered and trainees' performance was tested on multiple occasions. This presentation proceeds by first describing the approach to training development called Cognitive Engineering Based on Expert Skill (CEBES). The problem-domain in which its practical utility was tested is then discussed. Sketches of the goals, methods, and results of each of two projects follow, along with the practical impact of each. Conclusions about the efficacy of the CEBES approach and a discussion of their implications close the presentation. 1.1 Cognitive Engineering Based on Expert Skill Cognitive engineering applies the theory, principles, and methods of cognitive science to solve applied problems in which human performance depends upon the quality of
participants' thinking and skills. Cognitive engineering based on expert skill (CEBES) is a specialized case of cognitive engineering that uses domain-specific models of human expertise as the foundation for engineering. In the immediate context the problems involved operator training. This approach first develops cognitive models of expert skill and then uses these models as blueprints for training. Reverse-engineering the knowledge and skills used by experts in performing important tasks produces such models. Just as microbiology's models of the genome support cloning of organisms with desirable properties, the CEBES approach uses models of expertise as templates for replicating the knowledge and skills that are the foundation of exemplary human performance. The advantage of the CEBES approach over conventional training design has been demonstrated in two recent landmine detection training projects. These efforts used the CEBES approach to design training programs for operators of the currently fielded AN19/PSS-12 and recently developed and deployed PSS-14 handheld landmine detection systems. 1.2 The Landmine Problem Landmines are a major threat to both military and civilian personnel. A recent military assessment (Hambric and Schenck, 1996) stated “Mines are a major threat in all types of combat and will be the major threat in Operations Other than War…The widespread employment of landmines threatens to neutralize US advantages in firepower and mobility by severely limiting our ability to maneuver and disrupting our tactical synchronization.” Later reports on operations in Bosnia (Schneck & Green, 1997), Afghanistan (LaMoe & Read, 2002), and Iraq confirmed this assessment and suggested that the threat's severity may have been underestimated. Subsequent events have confirmed LaMoe & Reed's (2002) prediction that the countermine problem in contemporary operational environments would worsen. Landmines also jeopardize civilians whose lands have been mined during conflict as well as for years afterward -- unless the ordnance is either removed or neutralized -- due to this weapon's longevity. An estimated 45-50 million mines remain to be cleared in 90 countries. Mines claim an estimated 1520,000 victims per year, one third of whom die. Survivors
typically lose limbs, although blindness and shrapnel wounds occur as well. Beyond this physical toll, mine warfare imposes psychological and economic hardships on affected societies as well. Removing and/or neutralizing buried mines involves first detecting their locations. Detection typically relies on human operators of hand-held equipment working in close proximity to live ordnance. Few tasks punish human error more suddenly and brutally. After action reports from combat and peacekeeping operations as well as controlled testing showed that U.S. soldiers’ performance with the standard issue AN19/PSS-12 equipment was substandard and that detection rates for low metal mines were dangerously low. Two heavily experienced PSS-12 operators, however, produced 90%+ detection rates against high- and low-metal anti-tank (AT) and anti-personnel (AP) mines in field tests. Analysis of the experts’ detection activities also showed that their techniques and strategies differed from conventionally taught operating procedures (Staszewski, 1999). The most important differences were in the techniques experts used to enhance the sensitivity of their equipment and their strategies for information search and decision. Whereas soldiers made mine declarations on the basis of auditory outputs signaling the presence of conductive material, experts used the onset and offset of outputs that occurred during sweeping motions of the detector to synthesize spatial patterns that then they compared to the patterns they had learned were produced by mines. The differences between more- and less-experienced operators in performance and process motivated development of training that based instruction on experts' techniques and strategies. Studies then tested whether instruction based on a cognitive model of PSS-12 expertise could close the observed expert-soldier performance gap. 2. CEBES Training for the PSS-12 Detector2
component tasks, procedures, and training materials, readers are referred to Staszewski & Davison (2000), Davison & Staszewski (2002), or Davison, Staszewski, & Boxley (2001). In the initial test of the program, 22 U. S. Army combat engineers who had recently completed standard mine detection training participated as operators/trainees. This experiment used a pretest-posttest design. Mine simulants served as targets in testing and training. Targets included 5 different mine types and represented high- (M) and low-metal (LM) AT and AP mine types. Pretest performance failed to distinguish the treatment and control groups. Both achieved very low detection rates (15-17%) on low metal anti-personnel (AP-LM) mines. Treatment-group soldiers then received approximately 15 hours of experimental, hands-on training. Posttest results showed that the treatment group achieved an overall probability of detection of 0.94. This rate exceeded the control group’s by a factor of 3. The treatment group’s detection rate on the smallest, low metal targets (M14 simulants) was more than 6 times that of the control group. Treatment-control results by target type are shown in Figure 1. PSS-12 Initial Training Results: Posttest Performance 1.00 0.90 0.80 0.70 0.60 0.50 0.40 0.30 0.20 0.10 0.00 AT-M
AP-M
AT-LM
AP-LM
M14
Control
Four general principles guided the design of training. First, training content and organization reflected the expert model. Second, detection rate was made the focal measure of performance and learning, reasoning that improvements in discrimination and rate of area coverage would follow. Third, instruction and tasks were organized hierarchically, based on the expert's goal structure, to minimize information/working memory overload; instruction and drills started on part-tasks, and, when proficiency was achieved, instruction and drills focused on integration of the acquired subskills. The fourth principle involved providing individual trainees with ample practice and timely feedback on performance in each drill. Initial instruction and drills focused on three basic skills; (1) proper equipment preparation, (2) sweep/area coverage techniques that maximized the PSS-12's sensitivity and ensured thorough area coverage for identifying and localizing potential threats, and (3) development of pattern acquisition and target recognition skills. After the basics, soldiers proceeded to blind search drills, which called for integration of basic skills in a simulated mine detection activity. For more comprehensive descriptions of the training program, its
Mine Type
Treatment
Figure 1. Probability of detection as a function of target class. Error bars show 95% confidence intervals. M14 results are a subset of the displayed AP-LM data. A second study intended both to confirm the reliability of the initial study’s treatment effects and explore their generality retested a subset of the treated soldiers in the same setting, now wearing body armor. Results replicated those of Study 1. A third study tested treatment group soldiers on real, deactivated mine targets in a different location. Multiple mines from each mine type were used. The surface of the test lanes was expected to increase detection difficulty. Soldiers nonetheless achieved an aggregate detection rate of 0.97 and showed significant improvement in finding 100% of the M14 targets. Subsequent practical training efforts using (a) older noncommissioned officers as trainees and (b) military instructors have generalized these findings. Despite unanticipated reductions in training times per man to 3-4 hrs and 1 hr in
these studies, respectively, the pattern of robust pre-test posttest gains found in the initial study resulted, although detection rates dropped roughly 10%. In early 2002 a DTLOMS board composed of highranking officers at the U.S. Army Engineer School reviewed evaluated the CEBES training program and results and recommended its adoption. The USAES commandant subsequently ordered its adoption. A Safety of Use Message was broadcast to all U.S. Army units to cease use of the old techniques due to the hazard involved and receive training on the use of the techniques of the CEBES program. Troops so trained use the PSS-12 for countermine operations in Afghanistan and Iraq. The Colombian Army also uses the training program and the techniques it teaches. 3. CEBES Training for the New PSS-14 Detector A second project tested whether the CEBES approach could improve performance of an advanced, dual-sensor handheld detection technology (PSS-14) while it was in development. Like the PSS-12, the PSS-14 used electromagnetic induction (EMI) to sense conductive material, specifically the metallic components found in mines. Along with the EMI sensor, the PSS-14 incorporated groundpenetrating radar and related signal processing algorithms to "sense" buried objects. Different auditory tones for each sensor, which could occur simultaneously, alerted the operator to the presence and location of buried objects with the appropriate properties. After nine-years and $38M had been invested in system development, the prototype's performance in initial testing was disappointing. As shown by the striped bars in Figure 2, detection rates against small, low-metal AP mines were worse than those for the PSS-12, the system the prototype was designed to replace. A subsequent performance review conducted by a panel of military, government, academic, and private sector personnel concluded that weak operator training contributed substantially to the poor showing and made it difficult to assess the system's potential. The program was granted a reprieve and, in hope of improving performance, funding for an additional year of development was approved and a CEBES approach was applied to develop new operator training. An informationprocessing analysis was performed of the skill of the most experienced and highest-performing operator and a model of his skill was developed. Although skilled operation of the PSS-14 demanded greater precision in the movement of the sensor head over the ground surface than was required for the PSS-12 and the outputs of two independent sensor channels produced patterns of substantially greater complexity, the expert models for the two systems were surprisingly similar in terms of mechanisms and their organization. A training program for novice operators was designed using the expert model as a blueprint. The surprisingly strong similarities between the models of PSS-12 and PSS-14 expertise made it possible to employ many of the components of the PSS-12 training program in the PSS-14 training. Like the PSS-12 program, it incorporated drills designed facilitate acquisition of sweep/area coverage techniques that maximized
the equipment's sensitivity and supported development of pattern acquisition and recognition subskills. The greater complexity of the PSS-14 system, however, meant that achieving these goals required additional training steps. For example, maximizing the sensitivity of the metal detection and ground-penetrating radar sensors required maintaining the sensor head movements within narrow ranges of velocity and height above the ground surface. Because effective coverage also required that these movements parameters be maintained over cross-lane sweeps (perpendicular to the path of the operator) 1.5 m wide, with sufficient forward overlap on successive sweeps as the operator moved forward, while listening for the output of either sensor, just remembering all the requirements, let alone meeting them, challenged novices. To reduce complexity and information overload, the sweep/coverage task was divided into simpler components, with each component being trained and learned individually and progressively combined. To give trainers needed measures of sweep dynamics and trainees feedback, an automatic, a computer-based optical sensor head tracking system was developed (Herman, McMahill, & Kantor, 2000) that captured a trainee's movements of the sensor head in various drills and displayed performance on selected dimensions against the specified standard via a graphical interface. Also, because the PSS-14’s two sensors responded to different properties of mines and pattern/signature acquisition techniques differed for each sensor, these techniques were trained and executed separately, with trainees receiving instruction and practice on how to integrate the distinct metal and GPR signatures into a whole, complex pattern and interpret them. These factors, plus the requirement that operators be trained to detect mines in two environments (gravel roadway and fields with vegetation) increased training time beyond that for the PSS-12 training to 32-35 hours per operator. Two field tests of the PSS-14 training program were performed, the first using 2 U.S. Army NCOs and 3 civilians as trainees. The second used 2 enlisted U.S. Army paratroopers and 2 U.S. Marine Corps enlistees. Results from both CEBES training tests are shown with the results from the initial prototype test in Figure 2. CEBEStrained operators produced substantial detection gains overall relative to initial test results and achieved manifold improvement against the greatest threat, small, low-metal AP mines. The new results cast the potential of the PSS-14 in a considerably better light. In addition, consistencies in asymptotic performance among operators against AT-LMs and AP-LMs identified limitations in the system's design that guided re-engineering efforts. The observed performance improvements and the diagnostic information obtained justified continuation of funding for PSS-14 development. Subsequent refinements have raised detection rates into the 97-100% range (Santiago, Locke, & Reidy, 2004). Recognizing the critical contribution that operator training makes to the PSS-14's mine detection capability, the U. S. Army has adopted the policy of distributing the CEBES
training program with the PSS-14 equipment only as an integrated package. U.S. forces trained in this program now use the PSS-14 in countermine operations in Afghanistan and Iraq. Reports from the field suggest indicate that the PSS-14’s performance in the hands of a well-trained soldier represents a leap forward in mine detection capability. The U.S. Humanitarian Demining Research and Development Program is now developing the system further for humanitarian demining applications. Original Training/Test Results vs. Two CEBES Training Tests
1.00 0.90 0.80 0.70 0.60 0.50 0.40 0.30 0.20 0.10 0.00 AT-M
AP-M
AT-LM
AP-LM-L
AP-LM-S Original Test
Target Type
CEBES Test 1 CEBES Test 2
Figure 2. Probability of detection as a function of target class. Error bars show 95% confidence intervals. -L and -S suffixes in abscissa labels signify AP-LM targets with large (3.5-4.1”) and small (2.2-3.1”) diameters, respectively. 4. DISCUSSION The results of these two mine detection training projects illustrate the performance gains producible with properly designed operator training for both fielded and emerging technologies. Increasing the proficiency of operators of the PSS12 and PSS-14 has increased our capability to detect and subsequently neutralize hidden landmines. Hopefully, this progress has already contributed to reducing the carnage these weapons cause and will continue to do so. The results also demonstrate the practical utility of the cognitive models of human expertise beyond computer scientists’ development of expert systems. This demonstration holds implications for several related disciplines. For the fields of cognitive psychology and cognitive science, this work suggests that the principles, theory, and methods developed in their relatively brief lifetimes are sufficiently mature to support engineering applications. This work offers a clear and concrete answer to the recent question: “Once you have done a cognitive task analysis, how do you make use of the products?” (Chipman, Schraagen, & Shalin, 2000, p. 9). The potential value of basic research on human expertise for instructional theory and design has been recognized for some time (Glaser, 1984, 1989). More recently, however, several sources have pointed out that this potential has yet to be fulfilled (Alexander, 2003; Chipman, et al., 2000; Hatano & Oura, 2003;
Patel, et al, 1996). Perhaps the outcomes of the current projects can restore enthusiasm for investigating this approach and encourage analyses of expertise in other domains that will advance basic understanding of expert skill as well as support new instructional applications. Beyond the implications of this work for human factors professionals who deal with training, it is also relevant to a methodological concern expressed recently by Byrne & Gray (2003). Their concern focused on the decline they observed in the frequency with which formalisms from the mathematical and physical sciences guide human factors work, citing work in the area of cognitive engineering as an example. Such a decline should not be surprising. Technological developments in the workplace have cast operators more in the role of information-processors and less as physical controllers of tools/systems. Constructs related to human information processing, such as mental processes, knowledge, learning, and internal representation, are inherently abstract, unlike many of the phenomena with which the natural sciences deal, posing a greater measurement challenge. This has led to information processing models expressed in qualitative terms without sacrificing scientific standards or value. Examples include theories relating and explaining many cognitive phenomena (including expertise and its acquisition, e.g., Richman, Staszewski, & Simon, 1995) expressed formally as computer simulations (Anderson & Lebiere, 1998; Simon, 1979, 1989)3. Less formal, conceptual information-processing models, the form in which the models used to describe mine detection expertise in this work have been expressed, typically serve as blueprints for developing formal simulation models. The relative absence of the quantitative formalisms of the physical sciences in cognitive science does not equate to diminished scientific rigor. Interpreting their infrequent use in cognitive engineering as a reflection of eroding engineering discipline is questionable. Mathematical models, however precise and elegant, are not the only scientific foundations for rigorous and effective engineering. This work shows that cognitive models expressed in qualitative form can provide useful and practical knowledge bases for human factors engineering. Regarding the practical utility of CEBES, the outcomes of the training studies illustrate the effectiveness of this approach. Another important dimension of practicality in engineering involves economics. Considering that cognitive task analyses require non-trivial time and resources (Chipman, et al, 2000), it is reasonable to ask ‘Is the CEBES approach cost-effective?’ For the PSS-14 effort (for which cost information is available), the total cost of analyzing and modeling its expert's skill, developing the training program, and testing it represented 0.3% of the prototype system’s development cost. In closing, scientifically sound conclusions about the effectiveness of the CEBES approach or its generality will require considerable systematic investigation. The applied nature of the projects described here and the limitations of the test designs they employed will not support such conclusions. But the successes achieved with this approach and documented here should be sufficient to motivate appropriately designed evaluative studies of CEBES.
5. NOTES 1
The PSS-14 was originally known as the Handheld Standoff Mine Detection System and called the HSTAMIDS for most of its development. 2 Development and testing of the PSS-12 CEBES training program were performed in collaboration with Dr. Alan Davison, Chief, Army Research Laboratory, Human Research and Engineering Directorate U.S. Army Maneuver Support Center Field Element. 3 Computer simulations used to express cognitive theories are formally reducible to mathematical models. However, expressing them in such form for either programming or communication is about as practical as describing hurricanes at the level of particle physics. 6. REFERENCES 82nd Airborne Division Public Affairs Office. (2003). Media release #03-018 Fort Bragg, NC: 82nd Airborne Division. Alexander, P. A. (2003). Can we get there from here? Educational Researcher, 32(8), 3-4. Anderson, J. R., & Lebiere, C. (1998). The atomic components of thought. Hillsdale, NJ: Erlbaum. Biederman, I., & Shiffrar, M. M. (1987). Sexing day-old chicks: A case study and expert systems analysis of a difficult perceptuallearning task. Journal of Experimental Psychology Learning, Memory, and Cognition, 13, 640-645. Byrne, M. D., & Gray, W. D. (2003). Returning human factors to an engineering discipline: Expanding the science base through a new generation of quantitative methods - Preface to the special section. Human Factors, 45, 1-4. Chase, W. G., & Ericsson, K. A. (1982). Skill and working memory. In G. H. Bower (Ed.), The psychology of learning and motivation, Vol. 16, pp. 1-58). New York, NY: Academic Press. Schraagen, J.M., Chipman, S., & Shalin, V. L. (2000). Introduction to cognitive task analysis. In S. Chipman, J. M. Schraagen, and V. L. Shalin (Eds.), Cognitive task analysis (pp. 3-23). Mahwah, NJ: Erlbaum. Davison, A., & Staszewski, J. (2002). Handheld mine detection based on expert skill: Reference guide for training plan design and training site development. Aberdeen Proving Ground, MD: Army Research Laboratory Human Research and Engineering Directorate. Davison, A., Staszewski, J., & Boxley, G. (2001). Improving soldier performance with the AN/PSS12. Engineer, 31, 2001, 17-21. Glaser, R. (1984). Education and thinking: The role of knowledge. American Psychologist, 39, 93-104. Glaser, R. (1989). Expertise and learning: How do we think about instructional processes now that we have discovered knowledge structures? In D. Klahr and K. Kotovsky (Eds.), Complex information processing: The impact of Herbert A. Simon pp. 269-282). Hillsdale, NJ: Erlbaum. Hambric, H. H., & Schneck, W. C. (1996). The antipersonnel mine threat. Proceedings of the Technology and the Mine Problem Symposium, 1, Naval Postgraduate School, 18-21 November 1996, 3-11 – 3-45. Hatano, G., & Oura, Y. (2003). Reconceptualizing school learning using insight from expertise research. Educational Researcher, 32(8), 26-29. Herman, H., McMahill, J., & Kantor, G. (2000). Training and performance assessment of landmine detector operator using motion tracking and virtual mine lane. In A. C. Dubey & J. F. Harvey & J. T. Broach & R. E. Dugan (Eds.), Detection and remediation technologies for mines and minelike targets V, Proceedings of the Society for Photo-Optical Instrumentation Engineers 13th Annual Meeting, Vol. 4038. Orlando: SPIE.110-121.
LaMoe, J. P., & Read, T. (2002). Countermine operations in the contemporary operational environment. Engineeer, 32, April 2002, 21-23. Patel, V. L., Kaufman, D. R., & Magder, S. A. (1996). The acquisition of medical expertise in complex dynamic environments. In K. A. Ericsson (Ed.), The road to excellence. Hillsdale, NJ: Erlbaum. Richman, H. B., Staszewski, J., & Simon, H. A. (1995). Simulation of expert memory using EPAM IV. Psycho- logical Review, 102, 305-330. Santiago, A., Locke, M., & Reidy, D. (2004). Operational test results for the AN/PSS-14 (HSTAMIDS). Proceedings of the UXO/Countermine Forum, 9-12 March 2004, St. Louis, MO. Schneck, W. C., & Green, B. M. (1997). A road fraught with danger: Lessons in mine clearance. Janes’s International Defense Review, 11, 66-69. Simon, H. A. (1979) Models of thought, Volume 1. New Haven, CT: Yale University Press. Simon, H. A. (1989) Models of thought, Volume 2. New Haven, CT: Yale University Press. Staszewski, J. (1988). Skilled memory in expert mental calculation, In M. T. H. Chi, R. Glaser, & M. J. Farr (Eds.), The nature of expertise (pp. 71-128). Hillsdale, NJ: Erlbaum. Staszewski, J. (1990). Exceptional memory: The influence of practice and knowledge on the development of elaborative encoding strategies. In W. Schneider & F. E. Weinert (Eds.) Interactions among aptitudes, strategies, and knowledge in cognitive performance (pp. 252-285). New York: Springer-Verlag. Staszewski, J. (1999). Information processing analysis of human land mine detection skill. In T. Broach, A. C. Dubey, R. E. Dugan, and J. Harvey, (Eds.), Detection and Remediation Technologies for Mines and Minelike Targets IV, Proceedings of the Society for PhotoOptical Instrumentation Engineers 13th Annual Meeting, SPIE Vol. 3710, 766-777. Staszewski, J. (2001a). Expert analysis and training: Using cognitive engineering to develop mine detection skill. Demining Technology Information Forum Journal, 1, http://www.maic.jmu.edu/dtif/toc_dtif.html. Staszewski, J. (2001b). Training for Operators of Handheld Detection Equipment: A Cognitive Engineering Approach. Proceedings of the UXO/Countermine Forum, 9-11 April 2001, New Orleans, LA. Staszewski, J., & Davison, A. (2000). Mine detection training based on expert skill. In A. C. Dubey J. F. Harvey, J. T. Broach, and R. E. Dugan, (Eds.), Detection and Remediation Technologies for Mines and Mine-like Targets V, Proceedings of Society of Photo-Optical Instrumentation Engineers 14th Annual Meeting, SPIE Vol. 4038, 90-101. Wenger, M., & Payne, D. (1995). On the acquisition of mnemonic skill: Application of skilled memory theory. Journal of Experimental Psychology: Applied, 1, 195-215.
