NASA/TM–2005-213465
List Models of Procedure Learning Michael P. Matessa Ames Research Center, Moffett Field, California Peter G. Polson University of Colorado Boulder, Colorado
October 2005
The NASA STI Program Office . . . in Profile
Since its founding, NASA has been dedicated to the advancement of aeronautics and space science. The NASA Scientific and Technical Information (STI) Program Office plays a key part in helping NASA maintain this important role. The NASA STI Program Office is operated by Langley Research Center, the Lead Center for NASA’s scientific and technical information. The NASA STI Program Office provides access to the NASA STI Database, the largest collection of aeronautical and space science STI in the world. The Program Office is also NASA’s institutional mechanism for disseminating the results of its research and development activities. These results are published by NASA in the NASA STI Report Series, which includes the following report types: • TECHNICAL PUBLICATION. Reports of completed research or a major significant phase of research that present the results of NASA programs and include extensive data or theoretical analysis. Includes compilations of significant scientific and technical data and information deemed to be of continuing reference value. NASA’s counterpart of peer-reviewed formal professional papers but has less stringent limitations on manuscript length and extent of graphic presentations. • TECHNICAL MEMORANDUM. Scientific and technical findings that are preliminary or of specialized interest, e.g., quick release reports, working papers, and bibliographies that contain minimal annotation. Does not contain extensive analysis. • CONTRACTOR REPORT. Scientific and technical findings by NASA-sponsored contractors and grantees.
• CONFERENCE PUBLICATION. Collected papers from scientific and technical conferences, symposia, seminars, or other meetings sponsored or cosponsored by NASA. • SPECIAL PUBLICATION. Scientific, technical, or historical information from NASA programs, projects, and missions, often concerned with subjects having substantial public interest. • TECHNICAL TRANSLATION. Englishlanguage translations of foreign scientific and technical material pertinent to NASA’s mission. Specialized services that complement the STI Program Office’s diverse offerings include creating custom thesauri, building customized databases, organizing and publishing research results . . . even providing videos. For more information about the NASA STI Program Office, see the following: • Access the NASA STI Program Home Page at http://www.sti.nasa.gov • E-mail your question via the Internet to
[email protected] • Fax your question to the NASA Access Help Desk at (301) 621-0134 • Telephone the NASA Access Help Desk at (301) 621-0390 • Write to: NASA Access Help Desk NASA Center for AeroSpace Information 7121 Standard Drive Hanover, MD 21076-1320
NASA/TM–2005-213465
List Models of Procedure Learning Michael P. Matessa Ames Research Center, Moffett Field, California Peter G. Polson University of Colorado Boulder, Colorado
National Aeronautics and Space Administration Ames Research Center Moffett Field, California 94035-1000
October 2005
Available from: NASA Center for AeroSpace Information 7121 Standard Drive Hanover, MD 21076-1320 (301) 621-0390
National Technical Information Service 5285 Port Royal Road Springfield, VA 22161 (703) 487-4650
TABLE OF CONTENTS 1. INTRODUCTION ....................................................................................................................... 1 1.1 Problems and Solutions ........................................................................................................ 1 1.2 Outline of The Paper............................................................................................................. 2 2. THEORETICAL FOUNDATIONS ............................................................................................. 2 2.1 The Three Stages of Skill Acquisition................................................................................... 2 2.2 The Representation of Skills at the End of Formal Training .................................................. 3 2.3 Implications of the Human Memory Literature for Understanding the Associative Stage.................................................................................................................. 3 2.4 The Loci of Expertise Effects in Skill Acquisition ................................................................ 4 2.5 Current Airline Training Practices ........................................................................................ 4 3. THE LISTS OF ACTIONS MODEL ........................................................................................... 5 3.1 The Underlying Representation............................................................................................. 6 3.2 An Example List................................................................................................................... 7 3.3 Using the Serial List Learning Literature to Make Training Time Predictions ....................... 7 3.3.1
The Savings Paradigm............................................................................................... 8
3.3.2
The Relationship Between FMS Procedure Complexity (List Length) and Training Performance ......................................................................................... 8
3.4 Modeling a Training Program ............................................................................................... 9 3.4.1 Estimating the Time Required to Train a FMS Procedure.......................................... 9 3.4.2
Approximate and More Realistic Models of A Training Program ............................ 11
3.4.3
Using the Approximate Model to Analyze Training Program Modifications............ 11
3.5 Summary of Applications of the Lists of Actions Model..................................................... 13 4. SUMMARY .............................................................................................................................. 14 5. REFERENCES .......................................................................................................................... 15 6. BIBLIOGRAPHY...................................................................................................................... 17
iii
iv
LIST MODELS OF PROCEDURE LEARNING Michael P. Matessa and Peter G. Polson∗
Ames Research Center
1. INTRODUCTION
This paper presents a new theory of the initial stages of skill acquisition and then employs the theory to model current and future training programs for flight management systems (FMSs) in modern commercial airliners like the Boeing 777 and the Airbus A320. The theoretical foundations for the theory are a new synthesis of the literature on human memory and the latest version of the ACT-R theory of skill acquisition (Anderson, Bothell, Byrne, Douglass, Lebiere, and Qin, in press; Anderson and Schunn, 2000; Taatgen and Lee, 2003). The models derived from the theory describe training programs where to-be-learned procedures are formally trained, and trainees must demonstrate mastery before they can achieve journeyman status or go on to more advanced on-the-job training. Airline transition training programs are examples of this paradigm. The models generate estimates of the training time required to master the procedures specified in the training program syllabus. These estimates and the underlying theory are used to evaluate current and future training programs and support construction of business cases justifying investments in training-program modifications. 1.1 Problems and Solutions Numerous studies have documented operational and training problems with the modern autoflight systems, in particular the FMS and its pilot interface, the control display unit (CDU). During the last few years, more attention has been given to the limitations of current autoflight training methods. Many studies have concluded that current training programs are inadequate in both depth and breadth of coverage of FMS functions (Air Transport Association, 1997, 1998, 1999; BASI, 1998; FAA Human Factors Team, 1996). This paper argues that the inadequacies of the programs are due to airline training practices that encourage pilots to master FMS programming tasks by memorizing lists of actions, one list for each task. Treating FMS programming skills as lists of actions can interfere with acquisition of robust and flexible skills.
∗ University of Colorado at Boulder, Boulder, CO 80309
1
Blackmon and Polson (2002) argue that a solution to these training problems is to develop new training programs based on an intelligent computer-aided instruction technology, cognitive tutors (Anderson, Corbett, Koedinger, and Pelletier, 1995; Corbett, Koedinger, and Anderson, 1997). Blackmon and Polson claim that cognitive tutors can dramatically increase the depth and breadth of autoflight skills while maintaining the same training footprint. However, airlines will not even consider radical changes to current training practices until the research community provides sound business cases justifying large capital investments and radical modifications to the pilot training culture. A critical step in building a defensible business case is estimating the time required for both current and proposed training programs. This paper presents a new method for estimating training time based on a theory of the initial states of skill acquisition that is analogous to the Card, Moran, and Newell (1983) Keystroke Level Model for estimating skilled performance by characterizing procedures as sequences of actions and drastically simplifying descriptions of the underlying cognitive processes. The new method is referred to as the Lists of Actions Model. 1.2 Outline of The Paper The paper first gives the theoretical foundations for the new Lists of Actions Model and describes it in detail. Then the model is applied to the learning of FMS procedures. The model makes predictions about current training times that agree with times found in empirical research. The model is then used to analyze modifications to current training programs.
2. THEORETICAL FOUNDATIONS
The foundations of the theory are the Fitts (1964) three-stage model of skill acquisition that underlies almost all model analyses of skill acquisition and the latest ACT-R simulation models of skill-acquisition processes proposed by Anderson, et al. (2004) and Taatgen and Lee (2003). There is widespread agreement that the acquisition of a complex skill involves three stages, first described by Fitts (1964). The following description of these stages has been adapted from VanLehn (1996) to describe acquisition of the skills necessary to operate an autoflight system. 2.1 The Three Stages of Skill Acquisition The Cognitive Stage involves learning enough about the interface of a system to be able to generate descriptions of the steps in a procedure that can be successfully memorized. Initial training on such systems includes learning descriptions for elements of displays and controls. The Associative Stage involves memorizing a description of the actions required to perform a procedure. This stage ends with the successful storage in memory of representations that can be used to reliably generate the actions necessary to perform the procedure. The third or Autonomous Stage involves refining, strengthening, and transforming the mental representation of the skill to improve both speed and accuracy of performance. The endpoint of the process is a mental representation of the procedure that supports rapid and accurate performance even in high-workload situations—automaticity. 2
2.2 The Representation of Skills at the End of Formal Training The declarative representation of a procedure enables a newly trained operator to perform correctly but at some cost. The procedure is executed interpretively by retrieving description, decoding it, and generating the actions described by the retrieved information. These processes are error-prone, they take extra time, and they impose additional workload. In addition, declarative representations are easily forgotten. They can be maintained only by practicing the procedure at least every two or three weeks. The ACT-R cognitive architecture assumes that all these limitations can be overcome by acquiring a procedural representation of the procedure that enables very rapid retrieval of the actions necessary to perform it. The procedural representation is generated, refined, and strengthened during successful execution of the procedure mediated by the declarative representation. However, recent results (Anderson, et al., 2004) support the claim that achieving automaticity can take more than 100 repetitions of a procedure. The transformation from a declarative to a procedural representation of a skill is ACT-R’s explanation of the processes that describe Fitts’s Autonomous Stage. Thus trainees complete their formal training while they are still in the Associative Stage for all but the most frequently executed procedures. It follows that training time is in large part determined by the number of repetitions of a procedure necessary to acquire a useful declarative representation that is resistant to forgetting, i.e., supports accurate recall after retention intervals of as long as two weeks. Most of the additional experience necessary to convert the declarative representation into the much more durable and flexible procedural representation occurs during a trainee’s initial on-the-job experience. However, declarative representations enable a trainee to pass a qualifying exam and perform acceptably during on-the-job training. 2.3 Implications of the Human Memory Literature for Understanding the Associative Stage Research on human memory during the last 100 years has focused on the learning and retention of a wide variety of declarative representations from serial lists of nonsense syllables (Ebbinghaus, 1888/1913) to expository and native texts (Kintsch, 1998). The theory presented in this paper assumes that it is possible to model a training program by analogy to one or more experimental paradigms that have been employed to study human memory, e.g., serial or paired-associates list learning. The three steps of constructing such a model by analogy are described as follows. First, specify an experimental paradigm in the human-memory literature that has a structure analogous to that of the training program. Next, identify the features of tasks and training procedures that are analogous to variables that have been investigated in the context of the analogous experimental paradigm, e.g., mapping list length onto complexity of FMS procedures. Third, review the literature on the selected experimental paradigm to derive quantitative relationships between measures of learning and retention and the variables that are mapped onto important features of tasks and training methods, e.g., list length, distribution of practice, retention interval, and properties of list elements. 3
The list-learning literature contains extensive parametric studies of time and trials to mastery and retention for these and other variables. These quantitative relationships can be used to derive a model of a specific training program and make quantitative predictions of training time. It is also possible to propose change in existing training programs that are analogous to experimental manipulations in the memory literature that enhance both learning and retention, improving the efficiency of a training program. 2.4 The Loci of Expertise Effects in Skill Acquisition The theory developed in this paper hypothesizes that the Cognitive and Associative Stages of skill acquisition are the loci of expertise effects in skill acquisition. The characteristics of the learning processes for these two stages are determined by the strategies employed by a trainee. The learning strategies are determined by interactions between 1) the properties of the system user interface, 2) the trainee’s expertise acquired from experience with similar systems, and 3) the assumptions made by a trainee about how to learn and perform tasks using the system. These interactions also determine the experimental paradigm that best models the cognitive and associative stages of skill acquisition and the difficulties of these stages. The Cognitive Stage involves learning how to encode descriptions that when retrieved from memory enable them to generate actions necessary to perform tasks. The Associative Stage involves memorizing lists of these descriptions that enable trainees to perform tasks that will be on a proficiency examination at the completion of a training program. Catrambone (1995) has shown that novices in a domain—in the absence of explicit guidance on how to learn skills—assume that skill acquisition involves memorizing lists of actions required to perform tasks and that performing a task involves retrieving and performing each memorized action. It will be shown that for a novice pilot, learning a new FMS procedure is analogous to learning a serial list of nonsense syllables. Experts in a domain have much more powerful encoding strategies and are able to impose meaningful structures on representations that are well integrated with their domain knowledge. Pilots with extensive experience with several different FMS-equipped aircraft show all the attributes of domain experts acquiring new knowledge in their domain. For example, they can successfully perform a moderately complex procedure after a single demonstration. In addition, experts’ lists tend to be much shorter. Experts can generate descriptions that concisely describe a sequence of several actions. Finally, experts have specialized cognitive skills that enable them to generate actions when retrieval fails during execution of an infrequently performed task. 2.5 Current Airline Training Practices Flight manuals, FMS pilots’ guides, computer-based training (CBT) lessons, and other documents all contain very different descriptions of the steps in procedures, ranging from bare lists of key presses in flight manuals to detailed descriptions of each function invoked by a key press in a pilots’ guide. A demonstration of a procedure in a CBT lesson or by an instructor pilot in a fixed-base simulator presents the series of required actions and resulting display changes that occur during successful execution of a procedure. 4
None of these methods for training FMS procedures provides pilots with much—if any—clear guidance on how to generate memorable—or even meaningful—descriptions of a FMS procedure. Pilots using a glass cockpit for the first time tend to focus on actions required to perform a procedure because they will be tested on their ability to reproduce the action sequence to demonstrate mastery of the procedure during later simulator sessions or on a check ride. Learning FMS procedures involves both mastering a specific set of FMS programming tasks and acquiring specialized problem-solving skills. The ability to problem solve to use the FMS in an unfamiliar situation is based on knowledge of the CDU interface conventions, a retrieval structure that segments an action sequence into subsequences along with the knowledge of the purpose of each subsequence, and the skills necessary to use this knowledge and cues provided by the interface. None of these skills and knowledge is provided to pilots in today’s training programs. Pilots must induce them by making generalizations across procedures using the FMS. This induction problem is very difficult in the absence of explicit instructions. This is why clear evidence of reconstruction skills is seen only in pilots with experience with several different FMS-equipped aircraft. In addition, Fennell (2004)1 observed that pilots have strong default assumptions about learning cockpit procedures. Typical procedures are sequences of actions such as button presses, switch positionings, and observations of instrument readings and indicators. Memorizing lists is not an unreasonable approach to learning required short standard operating procedures (SOPs). However, it is a very poor model to apply to the acquisition of FMS programming skills. Instructors’ descriptions of their experiences with such trainees are certainly consistent with the hypothesis that these pilots try to learn FMS procedures as serial lists of actions. They require several repetitions of a procedure before they can perform it without errors. Trainees rapidly forgot newly learned procedures, and they must be retrained after a one- or two-day retention interval. Furthermore, these trainees’ knowledge of these painfully acquired procedures is very brittle. They have trouble generalizing their skills to new procedures that are closely related to a trained procedure.
3. THE LISTS OF ACTIONS MODEL
A representation of a task must encode both item (actions and parameters) and order information. Anderson, Bothell, Lebiere, and Matessa (1998) assumed that item and order information is encoded in a hierarchical retrieval structure incorporated in their ACT-R model of serial list learning shown in figure 1. The order information is encoded in a hierarchically organized collection of chunks. The terminal nodes of this retrieval structure represent the item information.
1 Personal communication, K. Fennell, United Airlines, Oct. 1, 2004.
5
3.1 The Underlying Representation The model assumes that pilots transitioning to their first FMS-equipped aircraft master a cockpit procedure by memorizing a serial list of declarative representations of individual actions or summaries of subsequences of actions. Learning this list takes place during briefings and demonstrations of the procedure, study of the procedure as described in a flight manual or other training materials, and attempts to execute the procedure as part of a CBT lesson or in a simulator session. It is assumed that each of these attempts to learn the list is analogous to a test-study trial in a serial recall experiment. An interpretive process uses the list to perform the procedure. This process incorporates the knowledge necessary to understand each step description and to execute actions necessary to perform each step. Thus, an item such as “Press the LEGS key” would generate the actions required to locate the Legs key on the CDU keyboard and press it. A parameter such as a waypoint identifier would be represented in working memory as a sequence of letters. The interpretative process would generate the keystrokes necessary to enter the identifier into the scratch pad.
Figure 1. List Model Representation. The retrieval structure for a list of nine digits of Anderson et al. (1998).2 The list actions representation is a consequence of pilots’ decisions to treat the task of mastering FMS procedures as learning serial lists of actions. The retrieval structure shown in figure 1 is generated by processes that adults use to memorize any arbitrary serial list of items. It is assumed that a novice representation of a FMS procedure with nine actions would be represented by replacing the terminal-node chunks with chunks representing individual actions in the procedure. The retrieval structure only encodes order information and supports access to the chunks representing individual actions. The groupings of the actions imposed by this structure have no
2 Figure 1 is figure 2 copied from Anderson, et al. (1998, p. 348).
6
relationship to the underlying task structure. Because these retrieval structures are unique to each task, they block transfer of training. 3.2 An Example List The following is a possible list describing an FMS procedure for the Boeing 777 for responding to the following hold clearance that would be generated by a pilot with limited glass-cockpit experience. NASA 1: Hold west of Haden on the 270O radial. Right turns. 10 mile legs. Expect further clearance at 2130. The list shown in figure 2 has undesirable properties and would be difficult to memorize. It is long—14 items—and it is organized as a linear sequence of actions that cannot be directly stored in memory (Anderson, et al., 1998). Some kind of idiosyncratic organization would have to be imposed on it to break it up into sublists before it could be successfully memorized. Furthermore, the representation of the procedure for programming a hold shown in figure 2 is specific to a particular clearance. It would be relatively easy to generalize this representation to clearances with identical parameters but with different values. However, generalizing this procedure to cover the entry of any hold clearance requires numerous nontrivial inferences.
1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14.
Press HOLD Function/Mode Key. Press LSK 6L, if a holding pattern is already in the route. Line select waypoint identifier for Haden to scratchpad. Press LKS 6L. Enter the quadrant and the radial into the scratchpad, W/270. Press LSK 2L. Enter the turn direction into the scratchpad, R. Press LSK 3L. Enter the leg distance into the scratchpad, 10. Press LSK 5L. Enter expect further clearance time into the scratchpad, 2130. Press LSK 3R. Verify the resulting holding pattern on the ND. Press EXECUTE.
Figure 2. A possible novice representation of a FMS procedure for responding to a hold clearance. 3.3 Using the Serial List Learning Literature to Make Training Time Predictions The Lists of Actions Model assumes that learning a FMS procedure is analogous to memorizing serial lists of nonsense syllables for a pilot with limited FMS experience. Training times can be estimated using results of an experimental paradigm initially developed by Ebbinghaus (1888/1913, Chapter 8). 7
3.3.1 The Savings Paradigm On the first day of the experiment, participants learn a serial list of items to a criterion of mastery of one perfect recitation of the list. Items are presented in a fixed order. During the presentation of an item in a list, participants must study it and attempt to recall the next item to be presented. Participants return to the laboratory 24 hours later and relearn the list to the same criterion of mastery. Training stops on the first day that participants perform perfectly on the first presentation of the list after a 24-hour retention interval. 3.3.2 The Relationship Between FMS Procedure Complexity (List Length) and Training Performance Table 1 presents the number of retentions on each successive day and the number of days of training required to be able recall a list perfectly after 24 hours for lists of 4, 8, 12, 16, and 24 items. The numbers in the table were derived by synthesizing the results of several experiments from the listlearning literature starting with the data from Ebbinghaus (1885/1913, Chapter 8). The numbers are extrapolations generated by fitting power functions to Ebbinghaus’s results and then adjusting them to account for the fact that he used a very rapid presentation rate. Table 1. Average number of repetitions of a list required to reach a criterion of one perfect recall as a function of list length and days. List Length 4 8 12 16 24
Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 7 Day 8 1.5 4.5 8.5 15.5 23
1.2 2.4 3.9 6.6 9.5
1.1 1.8 2.7 4.2 5.9
1 1.5 2.1 3.2 4.3
1.4 1.8 2.6 3.4
1 1.6 2.2 2.9
1 2 2
1.8 1.8
Total 4.7 12.5 21.7 38.2 52.8
Table 1 makes clear that procedures described by long lists with 12 or more items are very difficult to master. The 16- and 24-item lists do not reach criterion even after 8 days of training. Furthermore, examination of the column for Day 2 shows that much is forgotten after initial mastery of lists on Day 1. Underwood (1957) showed that interference caused the large amounts of forgetting shown in table 1 and in the original results of the studies from which it was derived. Participants in these early studies of list learning participated in many different experiments. The forgetting is caused by interference between the representations of a current list and previously learned lists. These interference effects are also present in FMS training. A transition training program is described by the List Model as involving concurrent mastery of from 20 to 30 lists describing various FMS procedures. Many days of practice are required to overcome the effects of interference between the memory representations of various procedures.
8
Recall that pilot instructors find that even moderately complex procedures are difficult to train and are rapidly forgotten. The impact of list length shown in table 1 and the very powerful interference effects found in this list-learning literature account for these problems. 3.4 Modeling a Training Program This section illustrates the methods involved in using table 1 to construct a simplified quantitative model of a training program. In this training program, procedures for performing FMS programming tasks are introduced during CBT lessons and training sessions with nonpilot instructors. Then, these skills are further trained and practiced in briefings and simulator sessions conducted by pilot instructors. All actual practice of these skills takes place in fixed-base and full-motion simulators. 3.4.1 Estimating the Time Required to Train a FMS Procedure The model assumes that the training of a specific FMS procedure is analogous to the serial-listlearning experiment described in Section 3.3.1. During any given simulator session, different examples of a procedure (e.g., a direction to clearance with different waypoints) are performed repeatedly until a pilot can execute the procedure without errors or hesitations. The number of required repetitions for a given simulator session and the number of days necessary to master the procedure are taken from table 1. Training time is estimated by calculating the amount of time it would take to administer N repetitions of a procedure of length L during one session in a fixed-base or full-motion simulator. The model’s description of the training processes has three time parameters: session setup time (SST), repetition setup time (RST), and step time (ST). SST is the time required to set up a simulator to begin training a specific procedure. RST is the time required to set up the simulator for the next repetition, and ST is the time required for a trainee to perform a step and receive feedback from the instructor if necessary. These values are then summed over days to generate a training- time prediction for a given procedure. The time devoted to training a procedure on one day = SST + N*RST + N*L*ST. The values for N, the number of repetitions on a day, are taken from table 1. Values for SST and RST were set to 120 seconds, and ST was set to 5 seconds. Current fixed-based and full-motion simulators were found to be ill-suited to this kind of training; they are designed to simulate the execution of complete missions. Figure 3 shows the total time necessary to train a procedure plotted as a function of length, L. The function turns out to be linear with a slope of approximately 11 minutes per step.
9
Figure 3. The time in minutes to train a procedure of length L. Figure 4 shows the percentage of the total training time that was devoted to actually practicing a procedure and receiving feedback from an instructor. The figures plotted are the actual training times (N*L*ST) summed across days divided by the total training times plotted in figure 2.
Figure 4. Percentage of training time devoted to actually practicing the procedure.
10
Figure 4 makes very clear the inefficiencies involved in using a mission-oriented simulator to train FMS procedures. 3.4.2 Approximate and More Realistic Models of a Training Program A complete FMS training program is modeled by identifying the FMS procedures specified in the syllabus and describing each procedure as a list of items to be memorized. The complexity of a FMS procedure is represented in the model as the length of a serial list of items describing how to perform it. The lists are generated from examples of FMS procedure descriptions in airline and manufacture training and reference documentation. An example is shown in figure 1. Pressing a page/mode key (e.g., HOLD) or line select key is described as a single list item. Entries into the scratch pad, which may involve several keystrokes, are each described as single items. It is similar to descriptions of FMS procedures in an airborne flight manual for the Boeing 777. An approximate model of a training program can be generated by computing the distribution of list lengths and then computing the total time required to train the complete collection of tasks. The function fitted to the data in figure 2 can be used to estimate training time for list lengths not included in table 1. A more realistic model would be based on a description of an actual Boeing 777 FMS training program of a major airline, including a realistic syllabus that specifies the tasks introduced and/or practiced on any given day. This model would specify the tasks that are introduced or practiced on any day during a multiday training program during a reasonable length simulator session, e.g., four hours. 3.4.3 Using the Approximate Model to Analyze Training Program Modifications The following examples assume that a FMS training program can be described as a collection of 30 lists. Based on a very preliminary study of an actual training program, a hypothetical distribution of list lengths was generated and is shown in table 2. In addition, no attempt was made to model a realistic syllabus specifying what tasks were introduced or practiced on a given day of the training program. Table 2. Frequency distribution of list lengths for a hypothetical FMS training program. List Length 4 8 12 16 24
Frequency 4 4 10 10 2
11
3.4.3.1 A model of a current airline training program Using the list-length distribution of table 2, a model of a current training program was constructed using the same values, 120 seconds, for SST and RST used to compute the predictions shown in figures 2 and 3. The total training time was approximately 48 hours. Polson, Irving, and Irving (1994) reported that pilot instructors for the Boeing 737-300 fleet of a major airline estimated that pilots spent between 15 and 50 hours practicing FMS skills in a simulator. This large value for total training time for FMS procedures—48 hours—is cause by two main factors. First, table 1 shows that successful training of complex FMS procedures—lists of 12 or more items—requires a very large number of repetitions distributed over at least 8 days. Second, figure 3 shows that a full mission simulator is a very inefficient—as well as very expensive—device for administering the necessary repetitions. 3.4.3.2 Introducing part-task trainers Numerous studies have shown that PC-based, part-task simulators can be used successfully to train skills such as performing FMS procedures (e.g., Salas, Bowers, and Prince, 1998; Salas, Bowers, and Rhodenizer, 1998; and Polson, Irving, and Irving, 1994). The lesson planners incorporated into commercially developed simulators can be programmed to deliver the necessary repetitions while minimizing the SST and RST (Aerosim Technologies, www.aerosim.com; Tricom Technologies, www.tricom-tech.com/products.htm; CAE, www.Cae.com; and Wicat, www.wicat.com). Use of such a trainer was modeled by reducing the values of SST or RST to 5 seconds. Total training time was reduced to 20 hours. This demonstration indicates that these trainers can very efficiently deliver the large number of repetitions necessary to master complex FMS procedures, leading to a large reduction in total training time—20 versus 48 hours. 3.4.3.3 Training pilots to generate shorter lists Pilots do not receive an explicit instruction on how to encode FMS procedures in memory early in training. Furthermore, descriptions of procedures in airline training and reference materials are inconsistent. The example in figure 2 is consistent with some of the more detailed descriptions of FMS procedures in the Boeing 777 Flight Manual of a major airline. Catrambone (1995) has shown that novices tend to describe problem solutions in terms of actions used to solve the problem. In the case of FMS programming skills, this process leads to long lists that are very difficult to memorize. Many studies in memory literature show that subjects can be easily trained to use more effective encoding strategies to enhance learning and retention of materials to be memorized. This model explores the consequences to teaching pilots methods for generating more concise descriptions of FMS procedures, leading to short lists. This example assumes that a hypothetical encoding strategy would have little or no effect on description list lengths for 4- and 8-item lists, that 12-item lists would be reduced to 8-item lists, that 16-item lists would be reduced to 12-item lists, and that 24-item lists would be reduced to 16-item lists. This encoding scheme would change the distribution of list lengths shown in table 2 to the distribution presented in table 3. The column labeled Novice List Length is from table 2. The column labeled Trained List Length shows the results of the encoding strategy described previously. 12
Table 3. Frequency distribution of list lengths for a hypothetical FMS training program for two different method of encoding lists. Novice List Length 4 8 12 16 24
Trained List Length 4 8 8 12 16
Frequency 4 4 10 10 2
The result of training these encoding skills is to reduce total training time from 48 hours to 29 hours, assuming the use of fixed-base and full-motion simulators. If we combine the use of part-task trainers with use of more efficient encoding strategies, total training time is reduced to 8.1 hours. The encoding and decoding skills assumed in this example are not trivial, and there is no way at this point of estimating the amount of time required to train such skills or if they can be taught to pilots during the initial stages of transition training to their first FMS-equipped aircraft. However, these skills—if mastered early in training—would be useful during later stages of training and line operations in supporting more effective acquisition of new skills on the line. 3.5 Summary of Applications of the Lists of Actions Model The examples presented in the preceding sections are illustrations of using results of the serial-listlearning literature to model existing airline training programs and suggest enhancements. Table 4 summarizes these results. Mastering FMS procedures takes a large number of repetitions, especially for procedures longer than 12 steps (see table 1.) A training program should be set up to deliver the necessary repetitions as efficiently as possible by employing part-task trainers. Additional large gains in training efficiency can be achieved by teaching pilots to generate shorter descriptions of FMS procedures. Table 4. Summary of four modeling experiments. Approximate Model of FMS Training All Practice in a Full-Mission Simulator All Practice Using a Part-Task Trainer Efficient Encoding Skills, Practice in FullMission Simulator Efficient Encoding Skills, Practice with Part-Task Trainer
Total Training Time for 30 FMS Tasks (hours) 48 20 29 8.1
13
The serial-list-learning literature describes numerous robust phenomena that have important implications for airline training. These phenomena can guide the development of incremental changes to existing training programs that can improve pilot performance and reduce training time. The examples presented in the preceding section, introducing part-task trainers and more efficient encoding strategies, are good illustrations. Many training programs introduce FMS procedures early in a three- to six-week transition training course. Focus of the program then switches to emergency procedures and other topics. Trainees can have very limited or no opportunity to practice flight plan modifications and other FMS procedures during the remainder of the program. Results shown in table 1 indicate that forgetting new learned procedures is a major problem. Rapid forgetting can be overcome only by distributing opportunities for practicing FMS procedures throughout a training program. Airline training and reference documentation do not contain complete sets of detailed and consistently described FMS procedure descriptions. Thus, pilots new to FMS-equipped aircraft must invent their own encoding methods. The list-learning literature shows that providing subjects with efficient encoding strategies can facilitate learning. Providing pilots with consistent descriptions of procedures and strategies for memorizing them should reduce training time and improve pilot performance. However, this literature also contains an important lesson. There is no “silver bullet” (Anderson and Schunn, 2000). Mastering a complex FMS procedure will require a large number of repetitions that must be distributed over many days and then practiced occasionally throughout the rest of a training program. Developers of training programs must recognize this fact and then deliver the necessary repetitions as efficiently as possible.
4. SUMMARY
This paper hypothesizes that initial mastery of an FMS or any other procedure is analogous to memorizing a serial list of items. Starting in 1888 with the publication of Ebbinghaus’s monograph on memory to the present, a huge collection of empirical and theoretical results has been published describing the learning and forgetting of such lists. This literature provides us with a rich collection of results that have enabled us to identify problems with current airline training practices and to propose many different solutions to these problems. The goals of this research program are to validate these insights and develop useful solutions to known problems with existing training programs.
14
5. REFERENCES
Air Transport Association: Towards an Operational Philosophy and Model Training Program for FMC-Generation Aircraft (first report by the Human Factors Committee Automation Subcommittee), 1997. Request a copy of this report by e-mail to Tom Chidester (
[email protected]). Air Transport Association: Potential Knowledge, Policy or Training Gaps Regarding Operation of FMS-Generation Aircraft (second report by the Human Factors Committee Automation Subcommittee), 1998. Request a copy of this report by e-mail to Tom Chidester (
[email protected]). Air Transport Association: Performance of Standard Navigation Tasks by FMS-Generation Aircraft (third report by the Human Factors Committee Automation Subcommittee), 1999. Request a copy of this report by e-mail to Tom Chidester (
[email protected]). Anderson, J. R.; and Schunn, C. D.: Implications of the ACT-R Learning Theory: No Magic Bullets. In Advances in instructional psychology: Educational design and cognitive science, R. Glasser, ed., Lawrence Erlbaum Associates, Malwah, N.J., vol. 5, 2000, pp. 1–34. Anderson, J. R.; Bothell, D.; Byrne, M. D.; Douglass, S.; Lebiere, C.; and Qin, Y.: Integrated Theory of the Mind. Psychological Review, vol. 3, no. 4, 2004, pp. 1036–1060. Anderson, J. R.; Bothell, D.; Lebiere, C.; and Matessa, M.: An Integrated Theory of List Memory. J. Memory and Language, vol. 38, 1998, pp. 341–380. Anderson, J. R.; Corbett, A. T.; Koedinger, K.; and Pelletier, R.: Cognitive Tutors: Lessons Learned. J. Learning Sciences, vol. 4, 1995, pp. 167–207. BASI: Advanced Technology Safety Survey Report. Flight Safety Digest Special Issue, Flight Safety Foundation, June–Aug. 1998, pp. 137–216. Available at www.basi.gov.au/sprog/ advtek.htm (9/00). Blackmon, M. B.; and Polson, P. G.: Key Advances for Improving Aviation Training Design. Proc. HCI-Aero-2002, International Conference on Human Computer Interaction in Aeronautics, AAAI Press, 2002. Card, S. K.; Moran, T. P.; and Newell, A.: The Psychology of Human-Computer Interaction. Lawrence Erlbaum, Hillsdale, N.J., 1983. Catrambone, R.: Aiding Subgoal Learning: Effects on Transfer. J. Educational Psychology, vol. 87, 1995, pp. 5–17.
15
Corbett, A. T.; Koedinger, K. R.; and Anderson, J. R.: Intelligent Tutoring Systems (Chapter 37). In Handbook of Human-Computer Interaction, 2nd edition, M. G. Helander, T. K. Landauer, and P. Prabhu, eds., Elsevier Science, Amsterdam, The Netherlands, 1997. Ebbinghaus, H.: Memory: A Contribution to Experimental Psychology. Henry A. Ruger and Clara E. Bussenius, trans., Columbia University, N.Y., 1885/1913. FAA Human Factors Team: Federal Aviation Administration Human Factors Report on the Interfaces Between Flight Crews and Modern Flight Deck Systems. Federal Aviation Administration, Washington, D.C., 1996. Fitts, P. M.: Perceptual-Motor Skill Learning. In Categories of Human Learning, A. W. Melton, ed., Academic Press, New York, 1964, pp. 243–285. Kintsch, W.: Comprehension: A Paradigm for Cognition. Cambridge University Press, 1998. Polson, P. G.; Irving, S.; and Irving, J. E.: Applications of Formal Methods of Human Computer Interaction to Training and the Use of the Control and Display Unit (final report). System Technology Division, ARD 200, Department of Transportation, FAA, Washington, D.C., 1994. Salas, E.; Bowers, C. A.; and Prince, C., eds.: Special Issue: Simulation and Training in Aviation. International J. Aviation Psychology, vol. 8, no. 3, 1998. Salas, E.; Bowers, C. A.; and Rhodenizer, L.: It Is Not How Much You Have but How You Use It: Toward a Rational Use of Simulation to Support Aviation Training. International J. Aviation Psychology, vol. 8, no. 3, 1998, pp. 197–208. Taatgen, N. A.; and Lee, F. J.: Production Compilation: A Simple Mechanism to Model Complex Skill Acquisition. Human Factors, vol. 45, no. 1, 2003, pp. 61–76. Underwood, B.: Interference and Forgetting. Psychological Rev., vol. 54, 1957, pp. 49–59. VanLehn, K.: Cognitive Skill Acquisition. Ann. Rev. of Psychology, vol. 47, 1996, pp. 513–539.
16
6. BIBLIOGRAPHY
Bovair, S.; Kieras, D. E.; and Polson, P. G.: The Acquisition and Performance of Text Editing Skill: A Production System Analysis. Human Computer Interaction, vol. 5, 1990, pp. 1–48. Catrambone, R.: The Subgoal Learning Model: Creating Better Examples so that Students Can Solve Novel Problems. J. Experimental Psychology: General, vol. 127, 1998, pp. 355–376. Ericsson, K. A.; and Lehmann, A. C.: Expert and Exceptional Performance: Evidence of Maximal Adaptation to Task Constraints. Annual Rev. Psychology, vol. 4, 1996, pp. 273–305. Ericsson, K. A.; and Kintsch, W.: Long-Term Working Memory. Psychological Rev., vol. 102, no. 2, 1995, pp. 211–245. FAA Human Factors Team: Report on the Interfaces Between Flightcrews and Modern Flight Deck Systems, U.S. Department of Transportation, Federal Aviation Administration, Washington, D.C., June 18, 1996. Fennell, K.; Polson, P. G.; and Sherry, L.: An Analysis of FMS Training Difficulty as a Function of Crew Experience. To be submitted for publication as a NASA Ames TM. Franzke, M.: Turning Research into Practice: Characteristics of Display-Based Interaction. Proc. Human Factors in Computing Systems CHI ’95, ACM, New York, 1995, pp. 421–428. Honeywell Boeing 777 Flight Management System Pilot's Guide. Honeywell Publication #C283641-22-00. Honeywell International, Air Transport Systems, Product Support Distribution. P.O. Box 21111, Phoenix, AZ, 85036-1111, 1995. Hovland, C. I.: Human Learning and Retention. In Handbook of Experimental Psychology, S. S. Stevens, ed., 1960, pp. 613–689. Hutchins, E. L.; Hollan, J. D.; and Norman, D. A.: Direct Manipulation Interfaces. In User Centered System Design, D. A. Norman and S. W. Draper, eds., Lawrence Erlbaum Associates, Hillsdale, New Jersey, 1986, pp. 87–124. Kitajima, M.; and Polson, P. G.: A Comprehension-Based Model of Correct Performance and Errors in Skilled, Display-Based Human-Computer Interaction. International J. Human-Computer Studies, vol. 43, 1995, pp. 65–99. Kitajima, M.; and Polson, P. G.: A Comprehension-Based Model of Exploration. Human–Computer Interaction, vol. 12, 1997, pp. 345–389. Kitajima, M.; Soto, R.; and Polson, P. G.: LICAI+: A Comprehension-Based Model of the Recall of Action Sequences. In Proc. Second European Conference on Cognitive Modeling, F. Ritter and R. M. Young, eds., Nottingham University Press, Nottingham, 1998, pp. 82–89. 17
Mangold, S. J.; and Eldredge, D.: Flight Management Systems Information Requirements. In Proc. Eighth International Aviation Psychology Symposium Conference, R. S. Jensen, ed., Ohio State University Press, Columbus, Ohio, 1995. McCrobie, D.; Alkin, M.; Sherry, L.; Feary, M.; Polson, P.; and Palmer, E.: Enhancing Vertical Navigation Performance in Glass-Cockpit Aircraft. In Proc. Ninth International Symposium on Aviation Psychology, vol. 1 (A98-28868 07-53), Ohio State University, Columbus, Ohio, 1997, pp. 434–439. McLennan, S.; Irving, J. E.; Polson, P. G.; and Blackmon, M. H.: Experimental Evidence Favoring Adoption of Cognitive Tutors for Flightcrew-Automation Training. Submitted to International J. Aviation Psychology. Nielsen, J.; and Mack, R.: Usability Inspection Methods. John Wiley & Sons, New York, 1994. Norman, D. A.: The Psychology of Everyday Things. Basic Books, New York, 1988. Polson, P.; and Smith, N.: The Cockpit Cognitive Walkthrough. Proc. Tenth Symposium on Aviation Psychology. Ohio State University, Columbus, Ohio, 1999, pp. 427–432. Rieman, J.: A Field Study of Exploratory Learning Strategies. ACM Transactions on ComputerHuman Interaction, vol. 3, 1996, pp. 189–218. Rieman, J.; Young, R. M.; and Howes, A.: A Dual-Space Model of Iteratively Deepening Exploratory Learning. International J. Human-Computer Studies, vol. 44, 1996, pp. 743–775. Sherry, L.: A Formalism for the Specification of Operationally Embedded Reactive Systems. In Proc. International Council on System Engineering, St. Louis, Missouri, 1995. Sherry, L.; Polson, P.; Feary, M; and Palmer, E.: When Does the MCDU Interface Work Well? Lessons Learned for the Design of New Flightdeck User-Interfaces. Proc. HCI-Aero-2002, International Conference on Human Computer Interaction in Aeronautics, S. Chatty, J. Hansman, and G. Boy, eds., AAAI Press, Menlo Park, Calif., 2002. Sherry, L.; Feary, M.; Polson, P.; and Fennell, K.: Drinking from a Fire Hose: Why the Flight Management System can be Hard to Train and Difficult to Use. NASA TM-2003-212274, 2003. Sherry, L.; Polson, P.; Fennell, K.; and Feary, M.: Drinking from the Fire Hose: The Over-Reliance on Memorized Action Sequences for MCDU Operation. Honeywell Publication #C69-5370022, 2002. Smith, D.; Irby, C.; Kimball, R.; Verplank, B.; and Harlem, E.: Designing the Star User Interface. Byte, vol. 7, no. 4, 1982, pp. 242–282.
18
Wharton, C.; Rieman, J.; Lewis, C.; and Polson, P.: The Cognitive Walkthrough Method: A Practitioner's Guide. In J. Usability Inspection Methods, J. Nielsen and R. Mack, eds., John Wiley & Sons, New York, 1994.
19
Form Approved OMB No. 0704-0188
REPORT DOCUMENTATION PAGE
The public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Department of Defense, Washington Headquarters Services, Directorate for information Operations and Reports (0704-0188), 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to any penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number.
PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS.
1. REPORT DATE (DD-MM-YYYY)
3. DATES COVERED (From - To)
2. REPORT TYPE
10/2005
Technical Memorandum
4. TITLE AND SUBTITLE
5a. CONTRACT NUMBER
xx
List Models of Procedure Learning
5b. GRANT NUMBER
5c. PROGRAM ELEMENT NUMBER
5d. PROJECT NUMBER
6. AUTHOR(S)
Michael P. Matessa and Peter G. Polson1 5e. TASK NUMBER
5f. WORK UNIT NUMBER
457280.02.07.01 8. PERFORMING ORGANIZATION REPORT NUMBER
7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)
Ames Research Center Moffett Field, CA 94035-1000 and 1 Department of Psychology University of Colorado, Boulder, CO 80309
A-0514345
10. SPONSORING/MONITOR’S ACRONYM(S)
9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES)
NASA
National Aeronautics and Space Administration Washington, D.C. 20546
11. SPONSORING/MONITORING REPORT NUMBER
NASA/TM-2005-213465 12. DISTRIBUTION/AVAILABILITY STATEMENT
Unclassified — Unlimited Distribution: Nonstandard Subject Category 54 Availability: NASA CASI (301) 621-0390 13. SUPPLEMENTARY NOTES
Point of Contact: Michael P. Matessa, Ames Research Center, MS 262-4, Moffett Field, CA 94035-1000 (650) 604-6291 14. ABSTRACT
This paper presents a new theory of the initial stages of skill acquisition and then employs the theory to model current and future training programs for fight management systems (FMSs) in modern commercial airliners like the Boeing 777 and the Airbus A320. The theoretical foundations for the theory are a new synthesis of the literature on human memory and the latest version of the ACT-R theory of skill acquisition (Anderson, Bothell, Byrne, Douglass, Lebiere, and Qin, in press; Anderson and Schunn, 2000; Taatgen and Lee, 2003).
15. SUBJECT TERMS
Learning, Prediction, List length
16. SECURITY CLASSIFICATION OF: a. REPORT
17. LIMITATION OF ABSTRACT
b. ABSTRACT c. THIS PAGE
Unclassified Unclassified Unclassified
Unclassified
18. NUMBER 19a. NAME OF RESPONSIBLE PERSON OF Michael P. Matessa PAGES
24
19b. TELEPHONE (Include area code)
(650) 604-6291 Standard Form 298 (Rev. 8-98) Prescribed by ANSI Std. Z39-18
