While these new performance evaluation tools offer a more powerful means of .... and crew communications, situational awareness and decision-making, automation .... be seen to form a sub-set of existing CRM behavioural markers.
Enhancing Instructional Systems
Enhancing Instructional Systems: The Development of a Tool for Evaluating Instructor and Student Performance. Matthew J. W. Thomas
The evaluation of flight crew performance has been an important area of focus throughout the history of the aviation industry. In response to the current need for an organisational approach to performance evaluation, along with the need to focus evaluation on a complex array of nontechnical skills, a number of new approaches to the evaluation and assessment have been developed in recent times. While these new performance evaluation tools offer a more powerful means of evaluating operational performance, there currently does not exist an equivalent evaluation tool for the aviation training environment. This paper describes the development of a new tool for performance evaluation in relation to flight training operations. Building on the existing framework developed for the operationally-oriented tools, a new set of behavioural markers have been developed specifically for the analysis of instruction. This new tool provides a new systematic method for the evaluation of instructor and student performance, and provides a variety of information which can be used to enhance an organisation’s instructional systems.
Introduction An essential predicate of high-quality operational performance is high-quality training. Training, by its very nature, is concerned with the development of competencies for adequate operational performance and indeed training forms the very foundation of operational performance. Although training has always been an important focus for all aspects of the aviation industry, there is always a danger of a mismatch developing between operational needs and training practices. As highlighted in the recent report on the Boeing 747-438 accident in Bangkok, latent failures in relation to deficient training were found to be significant causal factors contributing to the accident. In this case, the operational need for flight crew training in relation to wet runway landings were not realised (ATSB, 2001). To ensure that aviation training can adequately fulfil operational requirements it is necessary to employ appropriate tools for the on-going evaluation of training systems. Furthermore, as Hunt (2000) suggests, a key variable towards the improvement of operational performance and safety involves the upgrading of 41
Enhancing Professionalism in Aviation
flight-instructor knowledge and skills, and the development of more professional “aviation-teacher” competencies. The research presented in this paper is concerned with the development of a tool for the evaluation of training systems, both from the perspective of evaluating training effectiveness in light of operational requirements, as well as from the perspective of the surveillance-based assessment of instructor and student performance. Recent Developments in Performance Evaluation The on-going evaluation of performance has become an essential element of the aviation industry. From the early stages of ab initio training, through to the most advanced levels of air transport operations, pilots become accustomed to regular performance evaluation through the traditional sequence of testing and checking. Indeed, the strict regulatory environment across all aspects of the aviation industry ensures that personnel licensing is achieved with the direct assistance of performance evaluation and the rigorous assessment of proficiency. Although the various schemes of testing and assessment still form the backbone of licensing and certification, in recent times the focus of performance evaluation has been the subject of significant change. The first of these changes relates to the focus and scope of assessments and evaluation, and involves a broadening of focus to include the evaluation of crews and even whole operational divisions rather than just individuals. This shift in focus has been precipitated by an emergent understanding of the organisational aspects of performance and the renewed interest in systems rather than individual performance. In particular, research exploring latent organisational failures as important contributing factors to aviation incidents and accidents has led to the realisation that the ultimate safety of an operation comes from more than just the performance of individuals. This new focus has been termed the organisational approach to aviation safety (Maurino, 1995). This organisational approach to performance evaluation involves not only a new focus on larger operational units, but also includes a change in the way in which evaluation is undertaken. Instead of a sequence of tests or checks which are performed on a strict temporal basis, the new forms of performance evaluation involve a more surveillance-based approach. This means that performance evaluation is undertaken as a continuous process designed to identify the strengths and weaknesses of an operational system, both with respect to the individual components and the larger operational units. A second change in the area of performance evaluation involves an increased awareness of the complex array of non-technical competencies that are essential to performance in today’s operational environment. New forms of flight-crew performance evaluation have incorporated a significant shift away from the 42
Enhancing Instructional Systems
simplistic assessments of pilots’ technical flying skills towards a more balanced assessment of technical and non-technical skills with a particular emphasis on crew resource management. The evaluation of flight-crew performance has traditionally been based on a pass-fail grading which does not allow for a detailed analysis of crew competency across a variety of accomplishments (Helmreich, Taggart, and Wilhelm, 1994). Not only is the simple binary evaluation of competency inadequate for performance evaluation, it also fails to provide any information that can be fed back to the organisation in order to improve training and therefore improve overall crew performance. While other systems designed to evaluate pilot performance exist, such as Besco’s (1992) Professional Performance Analysis (PPA) system, most of these systems involve the post-hoc analysis of incident and accident data rather than the real-time evaluation of operational performance. Similarly, initial efforts to assess the value and impact of crew resource management focussed on the pilot’s opinions on the training or measuring attitudinal change. Recently, the need to establish more immediate and meaningful measures of crew performance has been identified and observational tools have been developed for the assessment of crew’s non-technical skills (Gregorich & Wilhelm, 1993). While these systems have been designed specifically for the complex environment of modern air-transport operations, many lessons can be drawn from these existing systems and they serve to inform the development of an evaluation tool for the flight-training environment. New Tools for the Evaluation of Flight Crew Performance During the last decade the need for a new approach to performance evaluation has given rise to a number of different systems. A common aspect of these new systems of performance evaluation is the use of “behavioural markers” to access what would otherwise be hidden cognitive processes. Although the term “behavioural markers” seems somewhat at odds to the developments cognitive psychology has provided to the field of aviation psychology, overt behaviour remains perhaps the best objective measure of performance. A behavioural marker system seeks to identify and rate competencies that are manifested in discreet behaviours (Law & Sherman, 1995). One seminal system to be developed for the evaluation of flight-crew performance was the Line/Line-Oriented-Simulation Checklist (LLC), which was designed by the Human Factors Research Project at the University of Texas (Helmreich, Butler, Taggart, & Wilhelm, 1994). This system has now become a key tool for many international airlines’ Crew Resource Management evaluation (Flin & Martin, 2001). The LLC focuses on six categories of performance, namely team management and crew communications, situational awareness and decision-making, automation management, special situations, technical proficiency and overall evaluation measures. The 28 43
Enhancing Professionalism in Aviation
behavioural markers embedded within these six categories are rated by observers on a four point scale ranging from poor to outstanding performance. After nearly a decade of development, the LLC now forms an essential element of the Line Operations Safety Audit (LOSA) system. Developed by the University of Texas to compliment the traditional Check and Training approach, the LOSA system has been designed to provide a wide range of data about the overall performance of an airline’s flight operations. Not only does the LOSA system focus on key non-technical performances, but also collects data on safety threats and crew error. Particular emphasis is placed on evaluation the crew’s actions taken in relation to the management of these external safety threats and errors (Klinect, Wilhelm, & Helmreich, 1999). A second system for flight-crew performance evaluation has been developed in response to certain requirements under the European Joint Aviation Requirements. Within the European regulatory environment, both the Joint Aviation Requirements Operations (JAR-OPS) and the Joint Aviation Requirements Flight Crew Licensing (JAR-FCL) require the training and assessment of flight-crew’s s non-technical skills in multi-crew operations. While this regulatory requirement was incorporated into both the areas of licensing and operations, there was no existing system capable of providing the required assessment of individual flight-crew performance in relation to nontechnical skills. Accordingly, a consortium of industry and academic organisations, after detailed review of existing evaluation methodologies, developed an original methodology called the NOTECHS system (Flin, Goeters, Hormann, & Martin, 1998). The NOTECHS system identifies four categories of non-technical skills, namely Cooperation, Leadership and Managerial Skills, Situation Awareness and DecisionMaking. These categories can be divided into two interpersonal or “social-skills” categories and two “cognitive-skills” categories. This differentiation between interpersonal and cognitive skills is in line with previous research undertaken to identify the crucial elements flight-crew performance. While these systems offer a new means of evaluating flight-crew performance in the normal or simulated line operations environment, there has only been limited development of a comparable observational methodology for the training environment. Dismukes, Jobe and McDonnell (1997) have developed the Debriefing Assessment Battery which is used to assess instructor and student participation in LOFT debriefings. This methodology involves 28 items rated on a seven point scale. Each of the items focuses on instructor and crew activity during a LOFT debriefing session. The Debriefing Assessment Battery provides a clear example of the desired approach to the evaluation of performance in the training environment. However, an instrument which is able to provide information at a more broader level, and that can be transferred across a variety of training situations, is required. As high quality operational performance is necessarily 44
Enhancing Instructional Systems
predicated upon good training, it is essential that the type of performance analysis now being embedded within operations is also adopted to assess the overall performance of instructional systems. Method As the generic methodological framework for observational performance analysis can be easily transferred to the training environment, the major task towards the development of an instructional systems analysis methodology was the development of suitable behavioural markers. Accordingly, the identification of the key aspects of flight instructor and student performance is a crucial aspect of this research project. In line with the goal of evaluating the training system a set of instructional markers was developed to rate performances essential to effective instruction and learning rather than any other non-technical skills used in operations. The methodology used for the development of the instructional markers firstly involved a thorough literature review of the field of flight instruction and student learning. As Telfer (1993) suggests, while there is only a limited body of research that pertains directly to aviation instruction and training, there is a wide body of research on instruction and learning in general which can be drawn upon in our quest for understanding aviation training. Accordingly, the focus of this literature review extended beyond current research specifically focussing on aviation training and examined the broader concepts of instruction and adult learning in general. Secondly, a group of Subject Matter Experts were employed in order to finalise and refine the set of instructional markers derived from the literature review. The expertise of this group was drawn upon in order to create an observational tool which would have maximum utility and usability as a means of evaluating instructor and student performance. Results In order to maintain an explicit link with the existing tools for flight crew performance evaluation, it was decided to utilise existing core CRM categories as a common framework for the development of the instructional markers. By utilising these already well established categories of non-technical skill the observational tool remains operationally relevant and the instructional behavioural markers can be seen to form a sub-set of existing CRM behavioural markers. Accordingly, the categories of Communication, Situation Awareness, Task Management and Instructional Processes were utilised. Under each of these four categories key non-technical skills for both instructors and students were identified and behavioural markers developed.
45
Enhancing Professionalism in Aviation
Identification of Instructor Non-Technical skills Ten key instructional markers were identified within the four categories of instructor non-technical skills. The first category of instructor non-technical skills relate to the Communication processes used in the training situation. Four key skills were identified as of particular relevance to both ground and simulator training. Briefing, performance feedback, clear explanation and debriefing are all critical aspects of instructor - student communication, and have significant bearing on the quality of training. The set of instructor non-technical skills identified in this process includes generic markers relating to Task-Management and Situation Awareness categories. These skills appear to be as relevant to the process of instruction as they are to the operational environment proper. The skills of maintaining a task focus, managing fatigue and stress, and monitoring and cross-checking both systems and crew members were identified as crucial aspects of the instructional process. The final category of instructor non-technical skills pertains to the Instructional Processes used by the instructor. Two key aspects of instructor performance were identified by the Subject Matter Experts in this area. Firstly, it was evident that a good instructor would be receptive to students' particular characteristics. For learning to be effective, it is necessary for the instructor to understand the existing knowledge of the student, and to shape the instruction around areas in which students require particular emphasis. Secondly, a more general marker was developed that pertained to how well the instructor Once identified, the set of markers describing instructor non-technical skills were developed as "word-pictures" to describe in some detail what constituted effective performance. These descriptions can be found in Table One. Identification of Student Non-Technical Skills In line with the set of instructional markers, ten key markers of student performance were identified within the four categories of non-technical skills. The markers for student performance focus on the role the student plays in the instructional. With particular reference to the processes involved in the training of flight crew, it has been demonstrated that the way in which students approached their studies is reflected in the quality of their learning outcome (Moore, 1995). The first category of student non-technical skill is that of Communication. As the instructional process is one of constant interaction between instructor and student, the communication behaviours of students can be seen to have considerable bearing on the quality of learning outcome achieved. Three key communication skills were identified as of particular relevance to flight crew training. Inquiry, receiving feedback, and participation in debriefing were identified as critical aspects of instructor - student communication.
46
Enhancing Instructional Systems
Table 1. Description of Instructor Non-Technical Skills Communication Briefing - The briefing provides students with a clear outline of the training session. It should cover the major points to be covered in the session, and highlight the main learning objectives. The instructor should assess student preparation and provide clear indications of expected performance during the session. A standard briefing should review previous training and place the current session in the context of previous training sessions. The briefing should also be interactive, and provide students with the opportunity to ask questions and shape the learning process. Performance Feedback - It is also essential that the instructor provides the students with adequate feedback during, and after, the training session. By providing a critique of performance, accompanied by clear indications of how performance can be improved, students stand a much better chance of meeting instructional objectives. Feedback should not be overly negative, and should be sympathetic to the crews' performance. Appropriate positive reinforcement should be made where appropriate. Clear Explanation - To promote learning, at all times the instructor should provide a clear explanation of the task. Where appropriate, the explanation should be accompanied by demonstration. Debriefing - The debriefing session provides a forum for performance feedback and also allows the training session to be placed in the context of the overall training programme. The debriefing should also be interactive, and provide students with the opportunity to ask questions and shape their learning process.
Situation Awareness Vigilance - The instructor should remain vigilant throughout the training session. The instructor should remain alert to the aircraft status as well as student performance at all times. Monitor and Cross-Check - As an ancillary member of the crew, the instructor should monitor and cross check aircraft systems to maintain situational awareness and ensure the suitable progression of the training session.
Task Management Adaptability - The instructor should remain adaptable and flexible throughout the training session. Changes to the training session should be made where appropriate, to allow for unexpected events, or to take advantage of areas of poor or outstanding performance. This adaptability is necessary to achieve the most learning outcome from training session. Management of Fatigue and Stress - It is necessary that the instructor should manage fatigue and stress throughout the training exercise. In particular, standard performance requires that the instructor should never become frustrated or angry with the crew, as this is a clear detriment to learning. Also, the instructor should be responsible for managing the fatigue and stress of the crew who are undertaking training. Allowances should be made for the crews' fatigue, stress or anxiety, where appropriate.
Instructional Processes Receptive to Students' Characteristics - It is important that instruction should be shaped in a manner that is sympathetic to the crews' needs and current status. Effective learning is not achieved if the lesson proceeds at a rate either too slow or too fast for the crews' current status. Similarly, an instructor should adopt and instructional technique that the student responds to in a positive manner. Promotion of Student Learning - All activities throughout the training session should be undertaken to promote enhanced learning from the crew. A standard performance in this area would involve maintaining a task focus throughout the session and adopting an effective and efficient instructional technique.
47
Enhancing Professionalism in Aviation Table 2. Description of Student Non-Technical Skills Communication Inquiry - The crew should be assertive in their inquiry throughout the training process. They should state any questions they might have where appropriate, and take responsibility for identifying and rectifying any deficiencies in knowledge that are important. Standard performance in this regard also entails Receiving Feedback - The crew should be open to performance feedback at all time during training. They should incorporate this feedback into their learning process and remain amenable to constructive critique. Participation in Debriefing - An essential prerequisite to enhanced learning is the crews' active participation in the debriefing session. The debriefing session is a time for performance evaluation and critique, as well as a forum for establishing guidelines for future training and performance.
Situation Awareness Vigilance - The crew must remain alert and attentive throughout the training process. Standard performance requires that crews maintain their concentration and task-focus. Monitor and Cross Check - The crew must maintain situation awareness of the aircraft status and systems at all times during training. Understanding of their influence on the aircraft status and systems is essential part of the learning process.
Task Management Workload Management - It is important that the crew effectively manage their workload. Work overload or underload is detrimental to effective learning. Management of Fatigue and Stress - Fatigue, frustration, anger and anxiety are detrimental to the learning process. The crew should be able to recognise the symptoms of these problems and effectively manage them to reduce their negative affects.
Instructional Processes Preparation - A key to effective learning in the training environment is good preparation. Standard performance should indicate that a crew has undertaken the necessary preparation for each training session. Attention - Standard performance in this are involves the crew paying attention to the instructor and to the training session throughout. Self-evaluation and critique - A crew should demonstrate their evaluation of their own performance. Reflective analysis of their performance is a very important part in the learning process. At appropriate points in the training, and especially in the debriefing session, the crew should demonstrate self-evaluation and critique.
Again, a generic set of markers for performance under the Task-Management and Situation Awareness categories were utilised, reflecting the nature of these categories. Vigilance, monitoring and cross-checking, workload management and the management of fatigue and stress are all important factors in determining adequate student engagement in the learning process. The final category of student non-technical performance relate to the Instructional Processes student adopt in their training. In this regard, preparation, 48
Enhancing Instructional Systems
attention and self-evaluation and critique were identified as factors which had the potential to drastically affect the quality of learning outcome. Once these key performances had been identified, the set of markers describing student non-technical skills were developed as "word-pictures" to describe in some detail what constituted effective performance. These descriptions can be found in Table Two. Discussion Together, the ten instructor performances and ten student performances identified as key behavioural markers of competent performance have the potential to provide a broad understanding of the efficacy of a training session. The methodology is currently being evaluated through a research with a major international airline and a large flight-training organisation. While the methodology can be used on its own, to analyse instructor and student performance, it can also be used alongside the existing tools for evaluating crew performance. In the LOFT situation, the use of the instructional markers alongside the University of Texas' LOSA methodology can be seen to provide a wealth of information, both in relation to how the training session was conducted, and in relation the crews' performance in simulated line operations. The analysis of crews' Threat and Error Management, which forms a major focus of the LOSA methodology, has the potential to assist considerably in our developing understanding of how error is used by instructors and students to facilitate learning. Furthermore, by the comparative analysis of data relating to crew performance during simulated line operations with data from actual line operations there is the potential to understand both the transfer of training as well as the differences in crew performance in the two quite different environments. Conclusion The current shift in focus towards an organisational approach to evaluation and assessment, along with a greater emphasis on the evaluation of flight-crews’ nontechnical skills has led to new systems of performance evaluation being developed throughout the aviation industry. This paper has discussed the development of a tool for the evaluation of flight instructor and student performance in the flighttraining environment. This tool, which is currently subject to further development and evaluation, offers a new systematic approach to gathering information on the strengths and weaknesses of flight training practices.
References ATSB. (2001). Boeing 747-438 VH-OJH, Bangkok, Thailand (Investigation Report 199904538). Canberra, Australia: Australian Transport Safety Bureau. 49
Enhancing Professionalism in Aviation
Besco, R. O. (1992). Analyzing Knowledge Deficiencies in Pilot Performance. International Journal of Aviation Psychology, 2(1), 53-74. Dismukes, R. K., Jobe, K. K., & McDonnell, L. K. (1997). LOFT Debriefings: An analysis of instructor techniques and crew participation (NASA Technical Memorandum 110442). Moffett Field, USA: NASA Ames Research Center. Flin, R., Goeters, K. M., Hormann, H. J., & Martin, L. (1998). A Generic Structure of Non-Technical Skills for Training and Assessment: Paper presented at the 23rd Conference of the European Association for Aviation Psychology, Vienna, September 1998. Flin, R., & Martin, L. (2001). Behavioural Markers for Crew Resource Management: A Review of Current Practice. International Journal of Aviation Psychology, 11(1), 95-118. Gregorich, S. E., & Wilhelm, J. A. (1993). Crew Resource Management Training Assessment. In E. L. Wiener, B. G. Kanki, & R. L. Helmreich (Eds.), Cockpit Resource Management (pp. 173-198). San Diego, USA: Academic Press, Inc. Helmreich, R. L., Butler, R. E., Taggart, W. R., & Wilhelm, J. A. (1994). The NASA/University of Texas/FAA Line/LOS Checklist: A behavioural markerbased checklist for CRM skills assessment (Technical Paper 94-02). Austin, Texas: University of Texas Aerospace Crew Research Project. Hunt, G. J. F. (2000). Different Approaches to Flight Crew Licencing: Nationally-Based Inspectorial Systems versus Globally-Based Surveillance Systems. Paper presented at the Eighth Global Trainair Conference and Training Symposium, Madrid, Spain. Klinect, J. R., Wilhelm, J. A., & Helmreich, R. L. (1999). Threat and Error Management: Data from Line Operations Safety Audits. In R. S. Jensen (Ed.), Proceedings of the Tenth International Symposium on Aviation Psychology (pp. 683-688). Columbus, Ohio: Ohio State University. Law, J., & Sherman, P. (1995). Do Raters Agree? Assessing inter-rater agreement in the evaluation of air crew resource management skills. In R. Jensen (Ed.), Proceedings of the 8th Symposium of Aviation Psychology (pp. 608-612). Columbus, USA: Ohio State University. Maurino, D. (1995). The future of Human Factors and psychology in aviation from ICAO's perspective. In N. McDonald, N. Johnston, & R. Fuller (Eds.), Applications of Psychology to the Aviation System . Aldershot, UK: Ashgate. Moore, P. J. (1995). Across Airline Differences in Pilot Learning: The Roles of Experience and Qualification. In N. Johnston, R. Fuller, & N. McDonald (Eds.), Aviation Psychology - Training and Selection . Aldershot, UK: Ashgate. Reason, J. (1990). Human Error. Cambridge, UK: Cambridge University Press. 50
Enhancing Instructional Systems
Reason, J. (1997). Managing the risks of organisational accidents. Aldershot, UK: Ashgate. Telfer, R. A. (1993). Effective Aviation Instruction. In R. A. Telfer (Ed.), Aviation Instruction and Training (pp. 219-236). Aldershot, UK: Ashgate.
51
