4-ITS Paper Final 10-pgs - CiteSeerX

To be Presented at: ITS '98 Conference, San Antonio, TX 17-19 August, 1998

An Advanced Embedded Training System (AETS) for Tactical Team Training W. Zachary1, J. Cannon-Bowers2, J. Burns3, P. Bilazarian4, and D. Krecker4 1

CHI Systems, Inc., 716 N. Bethlehem Pike, Suite 300, Lower Gwynedd, PA [email protected]; http://www.chiinc.com 2 Naval Air Warfare Center-Training Systems Division, Orlando, FL [email protected]; http://www.ntsc.navy.mil 3 Sonalysts, Inc., Orlando, FL [email protected]; http://www.sonalysts.com 4 Lockheed Martin Advanced Technology Laboratories, Camden, N.J. {pbilazar, dkrecker} @atl.lmco.com; http://www.atl.lmco.com

Abstract. The Advanced Embedded Training System (AETS) applies intelligent tutoring systems technology to improve tactical training quality and reduce manpower needs in simulation-based shipboard team training. AETS provides layers of performance assessment, cognitive diagnosis, and instructor-support on top of the existing embedded mission simulation capability. Detailed cognitive models of trainee task performance are used to drive the assessment, diagnosis and instructional functions of the system.

1

Introduction and System Rationale

The development of automated instruction has proceeded through several stages, beginning with simple computer-aided instruction (CAI) systems in the late 1950s which provided strictly didactic material in rigid instructional sequences. Dominant themes in the last two decades have been the emphasis on providing dynamic environments for applying and practicing problem-solving knowledge (problem-based learning), focusing diagnosis on the underlying knowledge state of the student rather than observed behavior alone (student modeling and cognitive diagnosis), and adapting the instruction to the student's evolving knowledge state (adaptive tutoring). The dominant paradigm recently has been "cognitive apprenticeship"([1]), in which the computer acts as an adaptive coach to the student (the apprentice) who works through a series of problem-solving exercises. Current intelligent tutoring systems (ITS) mainly fall into this category. All of these research-oriented ITS were built in problem domains which share several features. The domains involved little indeterminacy (cases in which multiple knowledge paths can lead to the same action), the relevant knowledge was comparatively closed and could be represented by relatively few rules, and the problemsolving activities were individually-based (rather than team-based) and involved non real-time problems that could be easily stopped and restarted. Unfortunately, tactical


domains possess none of these features. There is great indeterminacy, the problems and required knowledge are complex and open, the problem solving is team-based, and the problem environment is fast-paced. Perhaps because of these reasons, classical intelligent tutoring systems technology has been slow to emerge into the tactical world. A major attempt to solve these problems, however, has been made in the Advanced Embedded Training System (AETS), a Navy Science & Technology Advanced Technology Demonstration project to apply ITS technology to shipboard team training (see [2]). 1 AETS used the concepts of intelligent tutoring—problem-based learning, cognitive diagnosis, student modeling, and focused, adaptive tutoring—but found that new technologies had to be developed to deal with the indeterminacy, open knowledge spaces, team problem solving, and real-time nature of tactical domains. This paper overviews AETS architecture and operation, with focus on these new technologies. 1.1

The Task Domain

The Navy’s Aegis command and control system has an embedded simulation mode, in which a mission simulation is 'worked' using the actual tactical workstations on the ship. When a tactical team currently receives team training via embedded simulation, however, all performance measurement, diagnosis, feedback, and instruction is performed by human instructors, often in a labor-intensive manner (up to one instructor per student), and with a great deal of inconsistency across instructors and ships. The AETS was undertaken to improve this process by using ITS technology to make simulation-based team training less instructor-intensive, more thorough, and more consistent, by supporting, but not removing human instructors. AETS focuses on the Air Defense Team (ADT), one of several (albeit one of the most important) teams functioning in an Aegis ship's Combat Information Center. The job of the ADT is to protect own-ship and other assets (e.g., an aircraft carrier) from air attack. This job can be very difficult under conditions of low-intensity conflict, such as the Persian Gulf, where the skies are filled with commercial aircraft, industrial aircraft (e.g., helicopters moving to/from oil drilling platforms), and military aircraft from multiple countries, many of which are hostile to one another and to US forces, and where the threat of terrorist attack is also omnipresent. The activities of the ADT revolve around representations of airborne objects (i.e., fixed-wing aircraft, rotorcraft, or missiles) in the space around the ship. These objects are generically called 'tracks', and are displayed as icons on a geographical display at each person's workstation. The major source of track data is the powerful radar on-board the Aegis ship, but there are other sources, including radars on surveillance aircraft, other shipboard sensors (e.g., one detecting electronic emissions from a track); sensors on other ships, ground forces, and other intelligence sources. A track may be known 1

This work was performed under contracts N00014-96-C-0051, N61339-97-C-0405, and N61339-96-C-0055. The views expressed are those of the authors and do not reflect the official positions of the organizations with which they are affiliated or of the sponsors. The authors wish to acknowledge the support and contributions of RADM George Huchting and Susan Chipman, and the many members of the project team.


from just one or from multiple data types. In some cases, different data sources may create different tracks that, in 'ground truth', represent the same object. At the same time, single tracks can represent multiple objects, such as close flying aircraft. The air defense team as a whole is responsible for: • disambiguating and interpreting information about each air track; • determining if a track represents a potential threat to own-ship or a defended asset; • taking a variety of actions to neutralize or minimize any potential threat. The space of actions that can be taken is large. The ADT may, for example, send verbal warnings calling for a track to change course. In some cases, the team may direct friendly aircraft to try to get visual identification on a track, to challenge the track, to escort it away from the own-ship, or, in some cases, launch a missile to attack and destroy the track. Internally, team members will verbally pass information they have inferred or received to other members of the team; these internal messages are important in correlating data across sources and helping the team share a common mental model of the airspace.

2

AETS Architecture

The architecture of the AETS was designed to fulfill four main goals: 1. automate the capture and analysis of data on trainee problem-solving; 2. conduct automated, intelligent performance assessment and diagnosis of those data; 3. provide on-line instruction and feedback based on assessment and diagnosis; and 4. support a reduced human training staff in providing improved post-exercise debriefs. Figure 1 depicts the overall functional architecture of the AETS. The center row of the figure highlights four generic components that work in parallel. Three fully automated components supplement and support a fourth interactive component (far right) that allows a human instructor to record observations and initiate feedback. The instructor receives information from the automated components, makes notes on his direct observations, and communicates with trainees using a hand-held device known as ShipMATE (Shipboard Mobile Aid to Training and Evaluation). The primary interactions with the team being trained are shown at the four corners of Figure 1. A training session begins with a pre-exercise briefing on the training scenario. After the briefing, the existing simulation capability of the Aegis-class ship plays out the scenario on the team’s workstation consoles. The team’s responses are observed both by the automated system and by the instructor. During the scenario, real-time automated and instructor-initiated feedback is sent to the trainees. Shortly after the end of the scenario, the instructor uses ShipMATE to present a post-exercise debriefing, using automatically generated reports and his own computer-assisted performance assessment. The automated data capture component observes and aggregates the actions of each trainee in multiple modalities. It captures all keystroke sequences and aggregates them automatically into higher level functional interactions with the workstation. In parallel, the system records all trainee speech communications, recognizing


and analyzing them into semantic components that define the source, destination, message type, and objects and relationships mentioned. Also, in parallel, each trainee's eyes are tracked, and the eye-movement data are analyzed to assess what the trainee viewed, when, and for how long. The output of the automated data capture component is three streams of observed high level actions (HLAs) for each trainee. The combination of keystroke, speech, and eye HLAs provides a coherent record of what the trainee is doing during the exercise. Team Briefing

Automated Data Capture Keystroke Analysis

Scenario

Automated Assessment & Diagnosis

Automated Instructional Analysis & Feedback

Instructor Data Capture & Feedback

COGNET Models

Speech Recognition

Performance Assessment

Eye Tracking

Cognitive Diagnosis

Team Debriefing

Team Responses

Training Management Module (TMM)

Student Model Automated Feedback

Instructor Hand-Held Device (ShipMATE)

Recording & Note Taking Instructor Feedback

On-Line Feedback to Team

Fig. 1. AETS Functional Architecture

The automated assessment and diagnosis component dynamically compares this picture of what the trainee is doing with a model-based specification of what the trainee should be doing. An executable cognitive model of each trainee position passively observes the information provided to the trainee and identifies HLAs that an experienced operator would be expected to take at that point in the exercise. The model also identifies the knowledge and skill elements needed to generate the expected behavior and training objectives associated with the expected HLAs. Performance assessment and cognitive diagnosis are two stages of automated analysis of the trainee’s behavior. Performance assessment compares observed HLAs of the trainee to expected HLAs generated by the model to determine if the recommended behavior occurred. Results are summarized by the calculation of overall numeric and qualitative scores. Cognitive diagnosis relates performance assessment results to knowledge and skill elements identified by the cognitive model and makes inferences to determine what knowledge and skills are (and are not) demonstrated in the observed behavior. The automated instructional analysis and feedback component has two interrelated functions. First, it maintains a dynamic student model for each trainee. The student model records inferences about the trainee’s mastery of training objectives, as evidenced by automated performance assessment results. The second function is the generation of real-time automated feedback. Feedback is triggered by automated performance assessment results and involves first selecting an instructional opportunity and then selecting a feedback template. Instructional selection depends on the


qualitative result of a current performance comparison, how recently the trainee received feedback on the associated training objective, and the priority of the training objective. The structure and modality of the feedback depend on a feedback template, which is selected according to the training objective involved and its mastery by the trainee. Feedback, which is provided through a limited display window and/or highlighting of console display elements, is judiciously limited to avoid intrusion onto task performance. This component also provides the instructor with operator performance summaries for use in debriefing. Instructor data capture and feedback is an interactive software component supporting the instructor and running on a hand-held computer called ShipMATE. ShipMATE provides a medium for the automated components to communicate information to the instructor, who can decide how (and if) to use it. ShipMATE also allows the instructor to save voice comments and digital ink notes about trainee performance and to generate real-time feedback and post-exercise debriefs. ShipMATE facilitates instructor-generated feedback to trainees through a database of feedback templates that the instructor can call up and transmit. For post-exercise debriefing, ShipMATE constructs a variety of reports from automated performance, diagnosis, and instructional information and allows the instructor to select, organize, and present or replay captured information. 2.1

Cognitive Modeling

AETS depends on embedded cognitive models of each of the trainees. These models must generate, dynamically and in real time, expert-level solutions to the simulated missions being worked by the team being trained. Creating such models proved in itself a major challenge. The models were developed using the COGNET representation [3]. This framework was highly appropriate to the need, having been designed specifically for modeling and analysis of real-time, multi-tasking domains. Moreover, partial COGNET models of the ADT already existed [4], along with a software system to execute the models as needed within AETS. Figure 2 shows the architecture of the COGNET model of each team member. There are three separate processes organized into two levels of processing, the 'reasoning kernel', which simulates the cognitive processes, and the 'shell' which simulates the perceptual and motor processes, and manages the sensory and motor interactions with the external environment. The perceptual process, which internalizes sensed information, is instantiated as a set of spontaneous-computation (i.e., selfactivating) rules that capture the relevant perceptual knowledge about how to internalize sense information and relate it to the evolving mental model of the situation. These rules are fired as relevant perceptual cues are received from the environment. In parallel, a cognitive process activates and executes complex chunks of procedural knowledge (called cognitive tasks). These tasks are activated according to the current context provided, as defined by an internal model of the external situation. This internal model, expressed as declarative knowledge, is represented as a multi-panel blackboard structure and is stored in an extended working memory. Each panel is subdivided into levels, or categories, on which hypotheses (representing individual knowledge elements) are dynamically posted and unposted (as the situation changes).


The cognitive tasks are activated in a context sensitive manner, based on the current blackboard contents. Often, tasks must compete for the attention, as one context (pattern of blackboard information) can cause multiple cognitive tasks to be activated. As the cognitive tasks execute, the simulated operator develops and executes plans for solving the problem and/or pieces of his ADT role. The cognitive tasks may also change the blackboard (e.g. make inferences), as well as activate specific action processes (typically speaking and/or workstation actions). These processes generate specific operator acts in the external world, through the (also parallel) operation of the motor processing system. The cognitive tasks are represented as GOMS-like goal hierarchies with a formally defined operator set. In AETS, the embedded cognitive models need to generate diagnostically useful predictions of trainee actions at both behavioral and cognitive levels. This predictive use of the models required adaptation of the models, in two ways. First, their action space had to be abstracted. As discussed above, the predicted ENVIRONMENT (external world or simulation) behaviors had to be defined at an abstract level, above that of SHELL simple keystrokes; these abPerceptual Processing Action Effectors stracted behaviors are the HLAs discussed above. These abstract HLAs were not a natural product Extended Working Memory of either the modeling effort or Task Execution the cognitive task analysis on which the models were based. Attention Management Instead, they required a separate Cognitive Processing Reasoning Kernel effort to define them conceptually, and to embed them into the Fig. 2. Executable COGNET model processing structure model structure. A second adaptation was required to allow a cognitive diagnosis of the trainee in cases where the trainee's behavior diverged from the recommended path. The models were augmented to create a trace of the knowledge elements involved in producing each behavior expected of the trainee, beginning with the relevant perceptual stimuli. The main problem in creating this knowledge trace was computational. Each perceptual stimulus received by the model could lead to an unknown number of possible future behaviors; the cognitive architecture was modified to build and maintain each of these possible threads in realtime, and in a way that provided a valid trace of the cognitive processes involved. To solve this problem, multiple possible means of recording and tracing the reasoning threads were developed, and assessed both for theoretical reasonableness (in human information processing terms) and computational efficiency. Because these cognitive models had to be able to perform complex tactical tasks at an expert level, they are relatively complex. The model of the team coordinator, called the Anti-Air Warfare Coordinator or AAWC, for example, includes more than 25 cognitive tasks, with more than 500 lower level goals and subgoals, and several thousand instances of individual cognitive or action operators. Its ‘mental model’ includes 15 blackboard panels, with more than 90 individual levels and more than 500


hypotheses active on this blackboard at a typical moment. Its perceptual system includes more then 100 perceptual rules, and its action system more then 300 domainspecific action types. In addition, to simplify the complex action space, a total of 65 HLAs were defined. Of these, three were eye-movement actions, 24 were keystroke/workstation-based actions, and the remainder were speech actions. 2.2

Performance Assessment

The AETS conducts automated performance assessment both at the action level and at the event level. This section describes some of the challenges and solutions involved. At the action level, the AETS compares observed actions taken by a trainee with expected actions generated by a cognitive model. A meaningful and efficient comparison depends on a common taxonomy of observed and expected actions. The granularity cannot be too fine, or the actions would lack independent significance. Nor can it be too coarse, or data capture would miss important actions. The AETS solution defines a list of high-level action (HLA) types with specific attributes that vary from one instance to another. Keystroke HLAs define keystroke sequences that accomplish meaningful functions, speech HLAs correspond to entire message communications, and eye HLAs represent dwells on certain parts of the console. In each case, the HLA type defines the basic action, while variable attributes hold specific information entered, spoken, or seen. There are a variety of challenges in the recognition and evaluation of HLAs. While keystrokes can be definitively captured, current technology does not support foolproof speech recognition or eye tracking. Consequently, a confidence measure is associated with observed speech actions so that low confidence recognition can be discounted in evaluation. Similarly, some eye HLAs relate to small regions of the console rather than to the precise data displayed there. Evaluation difficulties arise from the multiplicity of ways in which HLA attributes may be set (or left unset). Matches between observed and expected actions are scored based on a classification of attributes into key and non-key categories. The overall positive/negative evaluation depends on how key attributes match. Non-key attributes only affect a finer gradation of scoring. The AETS supports an Event-Based Approach to Training (EBAT) by collecting and evaluating sets of expected actions related to critical scenario events. For complex scenarios in which many different events simultaneously demand the trainee’s attention, a central difficulty for EBAT is distinguishing which actions relate to which events. The AETS solution selects from the scenario a number of instructionally critical events and specifies for each one a time window for responses and a response identifier (or condition). Following the occurrence of a critical scenario event, the AETS collects all the model-generated expected actions that fall within the time window and match the response identifier (typically a track number). When the time window expires, a composite evaluation aggregates the trainee scores over the collection of expected actions.


2.3

Cognitive Diagnosis: Recognition-Activated Model Assessment

Classical ITSs typically work in relatively deterministic knowledge spaces, and so are able to diagnose cognitive state at a very fine grain by processing each observable low-level action. AETS, in contrast, had to work in an indeterminate space. While the COGNET models generate the high level actions that the trainee needs to perform, there are typically many reasoning sequences for generating each of those actions from low-level knowledge elements in them. To solve this problem, the RAMA algorithm was developed. RAMA (Recognition-Activated Model Assessment) does not try to perform cognitive diagnosis continuously, but only periodically, when the behavioral assessment component recognizes that the trainee either has or has not taken some recommended HLA. This recognition activates a bayesian inference process [5] that assesses all the knowledge that could have been used to generate the current recognized event from the likely knowledge state of the last recognized event. RAMA works on the knowledge traces produced by COGNET, (see above), and operates as follows: from any given point in the problem-solving process where an operator action has been recognized to have been taken (or not taken): − Process the cognitive model forward to (the next expected) High Level Action (HLA); note the trace of the knowledge states used along the way and call it the current knowledge trace; − Wait for the performance monitoring system to observe the HLA or conclude that trainee did not perform it; − If HLA was taken correctly, update the system's belief that trainee understands and has used correctly the intermediate knowledge states in the current knowledge trace; − If HLA was not taken correctly, update the system's belief that one or more of the intermediate knowledge states in the current knowledge trace was not understood or used correctly by the trainee. Through this periodic bayesian processing approach, RAMA is able to build, over time, a coherent model of the trainee's state of acquisition of each element of knowledge in the COGNET model. 2.4

Feedback, Instruction, and Team Training Support

The AETS mixes both individual feedback/instruction and team training, and uses both automated and human delivery methods. The interface to the AETS is provided through two elements, a Training Management Module (TMM) and the Shipboard Mobile Aid for Training and Evaluation (ShipMATE). Viewing the scenario as the “curriculum”, the TMM (currently under development) is being designed to support semi-automated training materials preparation. When complete, the TMM will allow an instructor to review team performance profiles (based on historical data) in order to define training objectives. Once this has been done, the instructor will be presented with candidate events that support the training objectives and they can then place their scenario anywhere in the world. Once the context has been established, the TMM


will populate the scenario with the selected events complete with requisite contextual information and with linkages to performance measurement systems. Providing a tool such as the TMM enables automated individual feedback/instruction that is delivered at the trainee's workstation using objective-based templates that are dynamically tailored to the specific context. Therefore, the automated performance measurement and diagnosis elements of the AETS receive much of the TMM measurement specifications. In order to integrate the instructor into the AETS, ShipMATE is used to take advantage of their expertise and observational skills. This portable PC is “networked” with the AETS and allows a mobile instructor to observe and collect data on multiple trainees at the same time. ShipMATE also provides tools for the instructor to formulate and deliver training to the trainer, either verbally or through the trainee's workstation and it supports allocation of complex measurement tasks to the most sophisticated component of the AETS—the human. For example, team process assessment and diagnosis requires application of a measurement scheme that relies on human observation of team communications—ShipMATE hosts this tool. Following the exercise, instructors will use ShipMATE to prepare individual and team level feedback. ShipMATE enables this by allowing for rapid review and reduction of instructor-based and automated measurement system data. ShipMATE is also designed to support presentation of debrief materials both for individuals and for the team. Through the dynamic link to the automated components of AETS, performance measurement data collected and reduced by instructors can be fed back to the TMM to update performance records in anticipation of subsequent training evolutions.

3

Implementation Status of AETS

AETS is being implemented and evaluated in three phases. Phase one, completed in 1997, developed the system's communication and computing infrastructure and integrated an initial system prototype for only two watchstanders, using a mix of real and placeholder components. The automated data capture component and the instructor data capture component were fully functional in this initial prototype, while the automated assessment and diagnosis was only partially implemented, with no active cognitive diagnosis. The automated instructional analysis and feedback system was represented only by a placeholder component. Phase two expanded the system to include fully functional prototypes of all components of the architecture, and a team of four watchstanders. The integration of the Phase two AETS, still in a laboratory context, is followed by an initial assessment of the effectiveness of the system, at the time of the writing of this paper. Phase three will, in calendar 1999, upgrade and expand all components of the Phase two system and demonstrate the system's ability to communicate with operational equipment.


4

Conclusions

Although it is too early to assess the operational value, there are already several clear implications and lessons learned from this large and ambitious project: Hybrid approaches work. There is no 'pure' ITS architecture that can or should be directly applied to create an ITS in a complex, real-world environment such as Aegis . Rather, system details and architecture must be fit to the constraints and opportunities of the application environment. A flexible architecture and advanced system integration methods can integrate the diverse pieces into a hybrid whole. Use architectural redundancy. The fluid nature of this task domain, and the difficulties of observing trainee performance (digitally) forced AETS not to rely on any one source for all its data, and hence required multiple data paths in its architecture. These multiple paths (e.g., collecting data separately from eyes, hands, and voice) allowed the data sources to be used alone if necessary, yet add value to one other when all were available, and allowed AETS to perform just behavioral diagnosis if cognitive analysis could not be performed at any time. By anticipating these concessions to the complexity and uncertainty in real-time and real-world domains in its architecture, AETS was able to make a virtue of necessity. Embedded training is an ideal home for ITS. Complex real-time systems like Aegis are increasingly being built with embedded simulation capability. Using this capability provides a platform that allows the ITS to train in the actual work environment and eliminates the (often costly) need to create/simulate workstations, interfaces, and/or underlying systems. However, it also focuses attention on system integration issues, and requires the ITS to be designed to the software interfaces provided by the host environment. Although the embedded ITS designer ultimately has little control of the training environment and must compromise constantly, the payoff of the embedded training context is clear -- virtually instant fieldability. Embedded ITS like AETS can ultimately provide a continuous learning environment through a seamless web of training, practice, and performance, all using a common workstation and a common set of performance standards and measures. With reduced need for off-site training facilities and on-site instructors, embedded ITS can not only provide better training, but also better training at reduced cost. References [1] Collins, A., Brons, J., & Newman, S. (1989) Cognitive apprenticeship: teaching the craft of reading, writing, and mathematics. In Resnick, L. (Ed.), Knowing, learning and instruction: Essays in honor of Robert Glaser. Hillsdale, N.J.: Lawrence Erlbaum Assoc. [2] Zachary, W., & Cannon-Bowers, J. (1997) Guided Practice -- a New Vision for Intelligent Embedded Training. Proc. of Human Factors & Ergonomics Society 41st Annual Meeting. Santa Monica, CA: HFES, pp. 1111-1112. [3] Zachary, W., Ryder, J., Ross, L., & Weiland, M. (1992) Intelligent Human-Computer Interaction in Real Time, Multi-tasking Process Control and Monitoring Systems. In M. Helander & M. Nagamachi (Eds.). Human Factors in Design for Manufacturability. NY: Taylor & Francis, pp. 377-402. [4] Zachary, W., Ryder, J., & Hicinbothom, J. (in press) Cognitive Task Analysis & Modeling


of Decision Making in Complex Environments. In J. Cannon-Bower & E. Salas (Eds.), Decision Making Under Stress, Wash, DC: APA. [5] Martin, J. & VanLehn, K.. (1995) Student assessment using bayesian nets. International Journal of Human Computer Studies, 42:575-591.