Sustaining and advancing performance improvements ... - CiteSeerX

SUSTAINING AND ADVANCING PERFORMANCE IMPROVEMENTS ACHIEVED BY CREW RESOURCE MANAGEMENT TRAINING Gary Grubb, John C. Morey and Robert Simon Crew Performance Group Dynamics Research Corporation Andover, Massachusetts 01810 The U.S. Army implemented its version of Aircrew Coordination Training (ACT) in 1994 (Department of the Army, 1992). During fiscal years 1984-1989, crew coordination failures identified by the Army Safety Center directly contributed to the loss of 147 lives and $292 million in aviation accident costs. In 1990, Army leadership directed the Army Aviation Center to focus its aviation training and evaluation on crew performance. We worked closely with Army aviation training, evaluation, and safety personnel to develop, validate, and field an ACT Exportable Training Package. Just before initial fielding, the Army estimated savings of nearly $30 million and 15 lives per year as a result of implementing the ACT training and evaluation program.

ABSTRACT The results of experience in applying team coordination training reflect the differences in areas of emphasis between organizations. For example, the US Army’s Aircrew Coordination Training (ACT) program emphasized mission performance measures and changes in behaviors while the US Air Force Air Combat Command/US Air Forces Europe (ACC/USAFE) Crew Resource Management (CRM) program relied heavily on attitudinal measures and student end of course critiques. Both program evaluation methods and measurement approaches revealed near-term and significant improvement in mission effectiveness and flying safety due to initial team training. Results of the Army ACT validation test showed significant improvement in aircrew’s overall mission effectiveness between pre- and post-training evaluation scenarios. The results of Air Force CRM attitudinal pre- and post-training surveys revealed statistically significant positive shifts in student attitudes towards CRM behaviors and skills. Posttraining trend analysis indicates that CRM attitudes and skills, like technical flying skills, show significant decay over time without refresher training, day-to-day emphasis and evaluation.

A joint NASA/Military Airlift Command workshop in May 1986 provided the impetus to begin CRM training in the U.S. Air Force (USAF), beginning in the transport aircraft community. By the early 1990’s, CRM had made inroads into the fighter community. The first fighter CRM programs were oriented more towards research than operations and were not widely embraced by experienced fighter pilots. There was a sense among them that CRM was “just another safety course” of little value to experienced, risk taking fighter pilots. In 1995, this cultural reality caused the USAF to introduce new CRM ideas and behavioral-based training approaches.

INTRODUCTION Crew or Cockpit Resource Management (CRM) programs were instituted in the 1980’s, first in commercial aviation and later in military aviation, to address adverse mishap rate trends that showed the inability of many aviators to work well together in periods of high stress or workload (Helmreich, Merritt, & Wilhelm, 1999). Minor aircraft malfunctions were resulting in fatal accidents with alarming regularity. While aviators generally displayed excellent knowledge and understanding of aircraft systems, operating procedures, rules and regulations and other technical information, they often displayed a glaring inability to communicate effectively, distribute workload, maintain or regain situational awareness and make sound decisions. Military aviation took note of the success of CRM in the civilian sector and instituted similar training programs (Orlady & Foushee, 1987).

CRM programs have been structured in various ways and continue to evolve as the perspective of what makes effective CRM training changes. Most programs seem to include at least these basic elements: !"A discussion of the core behaviors or basic skill sets that make up CRM. Each program structures these core behaviors differently, but all contain common elements. !"An examination of the applicability of CRM behaviors in the “real world.” This typically takes the form of one or more case studies of real-world incidents or accidents and includes an analysis of where or when proper CRM behaviors could have been employed.

1

!"Some type of role-playing or practice of CRM behaviors in a simulated mission setting, i.e., line-oriented flight training (LOFT) or its equivalent. !"Some form of assessment of the learning or changes in attitudes and behaviors that have taken place as a result of the training, and the evaluation of the training by the students.

CRM TRAINING RESULTS We worked closely with the Army Aviation Center and Air Force Air Combat Command to implement ACT/CRM training and evaluation methods and measures. Behavioral and mission performance results Results of the Army ACT validation test showed significant improvement in aircrew’s overall mission effectiveness between pre- and post-training evaluation scenarios.

DIFFERENCES IN ORGANIZATINAL EMPHASIS Methods of obtaining feedback from CRM training include student responses to discussions, case study questions, test questions, attitudinal surveys, course critiques and evaluations, and instructor observations and assessments. Nearly all of these were used in the ACC/USAFE and Army CRM/ACT programs. Our experience in applying team coordination training reflects the differences in areas of emphasis between the Army and Air Force programs.

!"Overall mission effectiveness !"Navigation accuracy !"Mission objectives

+20% +38% +44%

During the Army ACT validation test, all aircrews made errors in completing the complex, tactical simulator missions (Simon & Grubb, 1993). Pre- to post-training results improved markedly for navigation, instrument flight recovery, and missionthreatening error performance measures.

For example, the Army’s ACT program emphasized mission performance measures and changes in ACT behaviors while the Air Force CRM program relied heavily on attitudinal measures and student’s end of course critiques.

!"Course deviations !"Arrive at correct landing zone !"No early descent below minimums !"Manage mission threatening error

Behavioral and mission performance measures Army ACT program performance methods and measures included:

–45% +38% +32% +27%

Like the mission effectiveness increases, results of the Army ACT validation test showed significant improvement in aircrew flying safety between pre- and post-training evaluation scenarios.

!"ACT behaviors or Basic Qualities evaluated with supporting behaviorally anchored rating scales !"Aircrew Training Manual task performance !"Mission performance of two flight simulator scenarios similar in difficulty in terms of time stress, navigational demands, quantity and capabilities of simulated threat.

!"Unexpected weather recovery !"Emergency detection !"Number of aircraft crashes

+25% +38% –43%

Instructor-evaluators rated crew performance of the set of thirteen Army ACT behaviors (called Basic Qualities) during each testbed mission. There was improvement between the pre-training and posttraining evaluations in every Basic Quality. Statistically significant improvements were reached on 12 of 13 Basic Qualities.

Attitudinal measures Air Force CRM effectiveness measures focused on changes in student attitudes towards CRM. Grubb, Morey, and Simon (1999) showed that an attitude survey instrument based on the Theory of Reasoned Action provided assessment capabilities to detect student attitude changes as a result of training. Student attitudes provided diagnostic information and insights into specific objectives of course curriculum, command climate influences, and intent to perform CRM behaviors in daily operations. Attitude measure results were used to determine areas of emphasis for program and courseware improvements such as training program themes, course content, and training frequency.

As a result of ACT training, Army aircrews learned behavioral skills and team coordination techniques to keep them focused on what is important and to deal with emergencies and unforeseen problems in such a way that they do not lose sight of mission objectives.

2

2.5

1.97

1.9 1.64 1.34

ALL ACFT

HELO

ALL

.74

ALL ACFT

1.3

ALL ACFT HELICOPTER

1.26

0.9 HELO

ALL ACFT

1.2 0.83 ALL

0.5

HELICOPTER

1.5 1.0

2.03

1.6 HELICOPTER

1.77

HELO

2.0

ACT Started

2.0

ALL ACFT

During every Air Force CRM training class, students were presented with 24 attitude items divided into pre- and post-training instruments of eleven and thirteen items each. The Air Force results revealed statistically significant positive shifts in student attitudes towards CRM behaviors and skills. Figure 1 shows the mean pre-training and post-training attitude scores from 3,600 students combined for all platforms (showing the eight core behaviors of the original USAF program).

HELICOPTER

ACCIDENT RATE/100,000 FLYING HOURS

Attitudinal Results

0.0 FY 93

FY 94

FY 95

FY 96

FY 97

FY 98

FY 99

Figure 2. Army aviation Class A accident trend. Unlike the ACC CRM program, Army ACT was presented as “one-time training” without annual continuation or sustainment training. Not unexpectedly, the aviation accident rate increased when ACT was no longer emphasized.

44

42 Pretraining Posttraining

Mean Attitude Score

40

Temporary measures such as awareness videos, assistance visits, safety newsletter articles, and a webbased training support package have been ineffective substitutes for focused ACT training. The director of Army Safety recently stated, “In fact, FY99 produced Army aviation’s worst safety performance since Desert Shield/Desert Storm.”

38

36

34

32

30 M on Pl an ng

ic at

t

io

s

en

es

em

un

ag

m

ni

om

an

t

s

en

es

em

M

C

d

e

oa

ag

en

en

ce an rm rfo Pe

an

ar

s

ar

ic

w

am

lA

yn

na

Aw

iv

kl

ct

si

fe

or

M

ss

k

is

Ef

W

is

re

io

D

an um H

R

St

at

up

tu

ro

Si

G

n

To see whether the Air Force CRM program had any demonstrable effect on mishaps, we asked the Air Force Safety Agency to provide the fighter/attack operations factor mishap rates in ACC and USAFE for the period before and since CRM implementation. The results are shown in Figure 3.

Figure 1. Mean pre- and post-training attitude scores for CRM behaviors. Program improvement was managed through quarterly program reviews, which focused on summary class results, and annual adjustments were made to the program based on these reviews. Analyses of attitude data over multiple quarters of training indicated that the group dynamics (GD) and situational awareness (SA) CRM behaviors required special attention. Instructors received feedback after each class from computerized post-training reports, and were able to make rapid changes to their facilitation techniques.

ACC and USAFE Fighter/Attack Mishap Rates 5.0 4.5 4.0 3.5 3.0 2.5 2.0 1.5 1.0 0.5 0.0

TREND ANALYIS FINDINGS

ACC CRM Training

USAFE CRM Training

FY93

Army ACT implementation had an immediate positive impact on operational safety. The aviation Class A accident rate dropped from 1.75 per 100,000 flying hours (1993) to .75 (1996), a reduction of more than 50 percent (Figure 2).

FY94

FY95

FY96

FY97

FY98

FY99

Figure 3. ACC/USAFE Fighter/Attack Mishap Rates Both commands had been experiencing gradual reductions in their human factors-related mishap rates, even before they instituted CRM training. Although ACC’s Ops Factor rate became somewhat erratic after the CRM program was introduced, so was its overall Class A rate, with 1997 being a particularly bad year. Human factors related mishaps have traditionally accounted for nearly 80 percent of all mishaps (Flying Safety, 2000; GAO, 1996). After introducing CRM training in ACC and USAFE, human factors related

3

REFERENCES

mishaps accounted for 20 percent less than the GAO estimate: approximately 60 percent of the total Class A mishaps each year. Surveys administered during annual continuation training revealed an increased emphasis on the importance of CRM by unit leaders and aircrews alike. Aircrew members consistently reported the positive effect of CRM training on flying safety and mission effectiveness.

Department of the Air Force. (1998). Air force instruction 11-290: Cockpit/crew resource management training program. Washington, DC: Author. Department of the Army. (1992). Aircrew coordination exportable training package (Vol. 1-3). Fort Rucker, AL: U.S. Army Aviation Center.

Initial implementation of crew resource management training had a positive effect on US Army and US Air Force flying operations. Like technical flying skills, CRM attitudes and skills decay over time without command emphasis and evaluation. The Army’s increase in Class A accident rate when ACT was not emphasized is a good example. Trend analysis of Air Force aircrew member attitudes toward CRM revealed a 25 percent decline in attitude towards CRM after 12-14 months (Figure 4).

Grubb, G., Morey, J., & Simon, R. (1999). Applications of the theory of reasoned action model of attitude assessment in the air force CRM program. Proceedings of the Tenth International Symposium on Aviation Psychology, 298-301. Helmreich, R., Merritt, A., & Wilhelm, J. (1999). The evolution of crew resource management training in commercial aviation. The International Journal of Aviation Psychology, 9 (1), 19-32. Lacoste, Gene (1999, December). Improving Aviation Safety Performance. Flightfax27 (12).1-3.

Attitude Trend Between Training Events 96 FT Post Training to 97 CT PreTraining n = 352

% Attitude Change

125% 100% 75%

Orlady, H. W., & Foushee, H. C. (Eds.) (1987). Cockpit resource management training: Proceedings of NASA/MAC workshop (NASA CP 2455). Moffett Field, CA: NASA Ames Research Center.

50% 25% 0% -25% -50% -75% -100% 0

Mean (96 FT Post) = 24.4

100

200

300

400

500

600

Days Between Courses Student Data Point

Best Fit Curve

Simon, R., & Grubb, G. (1993). Validation of crew coordination training and evaluation methods for army aviation (Tech. Rep. No. E-785U). Wilmington, MA: Dynamics Research Corporation.

Figure 4. Attitude trend between training events. INITIATIVES TO SUSTAIN PERFORMANCE IMPROVEMENTS

USAF Class A Mishaps. US Air Force Flying Safety Magazine, Jan/Feb 2000, 47.

The challenge is how to sustain and advance the cultural and team performance improvements achieved by initial CRM training. To this end, we recommend:

U.S. General Accounting Office. (1996). Human Factors: Status of efforts to integrate research on human factors into FAA’s activities (GAO/RCED96-151, June 27,1996). Washington, D.C.: Author

!"Fully integrate CRM into the organizational structure, command climate, rules, and regulations that set the stage for daily flying operations. !"Institute ongoing CRM program evaluation and sustainment activities to keep training realistic, and relevant to operational missions and conditions. !"Correlate mishap investigation and mishap data analysis to the CRM program structure (core behaviors) so that mishap investigation data can be used to target specific areas of the CRM training program for increased emphasis. !"Include instruction on strategies, tools, and techniques that apply CRM behaviors and skills to avoid, trap, or mitigate aircrew error.

Research for this report was supported by contract F44650-95-0002. The views, opinions, or findings expressed are those of the authors and should not be construed as an official position of the U.S. Department of Defense or its agencies.

4