Human Factors and Ergonomics in Healthcare

Proceedings of the International Symposium of Human Factors and Ergonomics in Healthcare http://hcs.sagepub.com/

Is Crew Resource Management Training Effective in Healthcare Settings? Paul O'Connor, Angela O'Dea and Ivan Keogh Proceedings of the International Symposium of Human Factors and Ergonomics in Healthcare 2013 2: 105 DOI: 10.1177/2327857913021020 The online version of this article can be found at: http://hcs.sagepub.com/content/2/1/105

Published by: http://www.sagepublications.com

On behalf of:

Human Factors and Ergonomics Society

Additional services and information for Proceedings of the International Symposium of Human Factors and Ergonomics in Healthcare can be found at: Email Alerts: http://hcs.sagepub.com/cgi/alerts Subscriptions: http://hcs.sagepub.com/subscriptions Reprints: http://www.sagepub.com/journalsReprints.nav Permissions: http://www.sagepub.com/journalsPermissions.nav Citations: http://hcs.sagepub.com/content/2/1/105.refs.html

>> Version of Record - Jun 1, 2013 What is This?

Downloaded from hcs.sagepub.com at Library Services HSE West on July 31, 2013

2013 International Symposium on Human Factors and Ergonomics in Health Care: Advancing the Cause

105

Is Crew Resource Management Training Effective in Healthcare Settings? Paul O’Connor1, Angela O’Dea1, and Ivan Keogh2 1

Department of General Practice, National University of Ireland, Galway, Ireland.

2

Academic Department of Otolaryngology, National University of Ireland, Galway, Ireland.

The healthcare industry has seen a recent proliferation in the adoption of team training, such as crew resource management (CRM). Empirical studies of the effectiveness of CRM-type training in healthcare were subjected to meta-analysis. Fourteen CRM evaluation studies were found to fulfill the a priori criteria for inclusion in the meta-analysis. The metrics of CRM training effectiveness analyzed were: reactions, attitudes, knowledge, behaviors, and organization. CRM trained participants responded positively to CRM, the training had large effects on the participants’ knowledge and behaviors, and a medium effect on their attitudes. The training was also found to have a medium effect on the number of errors/adverse events and a small effect on the time to complete clinical tasks. Although there are methodological limitations that mean the results from this study should be treated with a little caution, the findings are nevertheless encouraging.

general conclusion regarding the effectiveness. They found

Introduction Since Crew Resource Management (CRM) training was first adopted in civilian aviation in the early 1980s it has become Copyright 2013 Human Factors and Ergonomics Society. All rights reserved. DOI 10.1177/2327857913021020

the most widely used team training strategy in the world. CRM training is designed to target knowledge, skills, and abilities as well as mental attitudes and motives related to cognitive processes and interpersonal relationships (Flin et al.,

evidence of an improvement in attitudes and teamworking after training and some evidence of improved technical performance, efficiency and reduced errors, in the short and possibly medium term. The purpose of this paper is to use meta-analyses techniques to evaluate the effectiveness of CRM type training in healthcare settings.

2008). Method There have been a number of literature reviews and a metaanalysis of the effectiveness of CRM across a range of highrisk industries. The conclusion from these cross-industry reviews is that reactions to the training are generally positive, and CRM training has positive effects on the teamwork attitudes, knowledge, and behaviors of participants. However, there is a lack of evidence regarding the effect on safety and/or performance (O’Connor et al. 2008; Salas et al 2006; Salas et

An electronic literature search of PUBMED, Embase and PsychInfo was carried out using appropriate keywords. A total 1,283 studies were identified. For a study to be included in the meta-analysis, the training had to utilize a CRM training model and an evaluation had to be reported from at least one of the levels of Kirkpatrick’s (1976) training evaluation hierarchy

(reactions,

attitudes,

knowledge,

behavior,

organization and meet the same set of inclusion criteria as

al 2001).

used by O’Connor et al. (2008). In total 14 studies were There have also been three literature reviews specifically focused on CRM training in healthcare (Buljac-Samardzic et al, 2010; McCulloch, Rathbone & Catchpole, 2011; Zeltser & Nash, 2010). However, only McCulloch et al. (2009) drew any

appropriate for the meta-analysis (one study was under review, but given the small number studies, and the fact that it met the inclusion criteria it was included in the meta-analysis). The MIX 2.0 software was used to carry out the meta-analyses.



106

Results

Discussion

Six studies included measurement of reactions to CRM

This study provides evidence to support the utility of CRM

training that met the criteria for reactions level data in the

training in healthcare settings. This type of evaluation data is

meta-analysis. Overall, participants were positive about the

necessary in order to provide an argument to support the use

various training courses they attended. On average participants

of limited resources (money, time, and people) required to

rated training four out of five possible points (mean= 4.19).

implement a successful CRM training program.

Three evaluations of knowledge change were included in the

Overall, participants responded positively to CRM training,

meta-analysis. The three different comparison of knowledge

the training had large effects on the participants’ knowledge

2

and behaviors, and a medium effect on their attitudes. The

resulted in a large mean effect size (d= 1.52). However, the I 2

test indicated high levels of heterogeneity (I = 78.8%).

training was also found to have a medium effect on the number of errors/adverse events and a small effect on the time

Six evaluations of attitude change were included in the meta-

to complete clinical tasks. However, these results should be

analysis. The results show a medium mean effect size of

considered within the context of three methodological

training on attitudes (d= 0.51). However, the level of

limitations.

heterogeneity, or consistency, across the studies was found to be large, as shown by the high I2 values (I2= 75.1%).

1.

The small number of studies included in the metaanalyses represents a constraint to the present effort.

Eight evaluations were included in the meta-analysis that

The vast majority of the studies examined could not be

analyzed the effects of CRM training on behavior. A large

included in the meta-analysis due to a lack of, or

mean effect size resulted (d= 1.18), although the level of

insufficient, data reported to allow effect sizes to be

2

calculated.

heterogeneity was also large (I = 82.5%). 2.

The evaluations included in each meta-analysis were

Three organizational level effects of training were examined

not necessarily independent of each other. To illustrate,

using meta-analysis techniques: number of errors/adverse

in some cases it was possible to extract inter-group and

events (three evaluations), time to complete clinical tasks (two

repeated measures design comparisons of the impact of

evaluations), length of stay in hospital (two evaluations). A

the training on behaviors from the same study.

medium effect size of the training was found for number of errors/adverse events (d= 0.54) a small effect size was found

3.

There were high levels of heterogeneity (or lack of consistency), of effects across the studies.

for time to complete clinical tasks (d= 0.22) and no effect size of length of stay of patient (d= 0.13). There was a high

Conclusion

heterogeneity for the number of errors/adverse events (I2= 75.1%), but the Q statistics for time to complete (Q= 0.76) and

It is important that researchers and reviewers be more rigorous

length of stay (Q= 1.72) were not significant.

about the quality of the reporting of CRM training evaluation studies. Despite the methodological limitations, the results of the meta-analysis are encouraging in terms of the utility of CRM-type training in medicine. The findings provide



evidence to support the inclusion of CRM-type training in the

.

training curriculum of all healthcare professionals.

Bibliography

Buljac-Samardzic, Wijngaarden,

M., J.D.,

Dekker-van &

van

Doorn, Wijk,

C.M., K.P.

van

(2010).

Interventions to improve team effectiveness: a systematic review. Health Policy, 94,183-195. Flin R, O’Connor P, & Crichton M. (2008). Safety at the Sharp End: Training Non-technical Skills. Aldershot, England Ashgate Publishing Ltd. Kirkpatrick DL. (1976). Evaluation of training. In: Craig RL, Bittel LR, editors. Training and Development Handbook. New York: McGraw Hill. McCulloch,

P.,

Rathbone,

J.,

Catchpole,

K.

(2011).

Interventions to improve teamwork and communications among healthcare staff. British Journal of Surgery, 98, 469-479. O’Connor, P., Campbell, J., Newon, J., Melton, J., Salas, E., &Wilson, K.A. (2008). Crew Resource Management Training Effectiveness: A Meta-Analysis and Some Critical Needs. International Journal of Aviation Psychology,18, 353-368. O’Connor, P., Flin, R., & Fletcher, G. (2002). Methods used to evaluate the effectiveness of CRM training: A literature review. Journal of Human Factors and Aerospace Safety;2, 217-234. Salas, E., Burke, C.S., Bowers, C.A., & Wilson K.A. (2001). Team training in the skies: Does crew resource management (CRM) training work? Human Factors, 43, 641-674. Salas, E., Wilson, K.A., Burke, C.S., & Wightman, D.C. (2006). Does crew resource management training work? An update, an extension, and some critical needs. Human Factors, 48, 392-412. Zeltser, M.V. & Nash DB. (2010). Approaching the evidence basis for aviation-derived teamwork training in medicine. American Journal of Medical Quality, 25, 13-23.


107