a Delphi study - Semantic Scholar

International Journal for Quality in Health Care 2009; Volume 21, Number 6: pp. 433 –440

10.1093/intqhc/mzp042

Determination of health-care teamwork training competencies: a Delphi study ROBYN CLAY-WILLIAMS AND JEFFREY BRAITHWAITE Centre for Clinical Governance Research, Australian Institute of Health Innovation, University of New South Wales, Sydney, NSW 2052, Australia

Abstract Objective. The purpose of this study was to determine the optimum content of a 1-day classroom-based crew resource management (CRM) course for health-care personnel working in ad hoc teams in complex, time-critical hospital departments such as surgery, intensive care or emergency. Design. A two-round modified Delphi panel. Participants selected teamwork competency components suitable for inclusion in 1 day of training from a list developed via literature review. Participants. Fifteen experts in health care, CRM and training. Main Outcome Measure. Knowledge, skill and attitude competency components for a 1-day CRM course. Results. Of the 110 knowledge, skill and attitude CRM competency components, 40 components were selected by greater than 70% of respondents, whereas the remaining 62 components were selected by fewer than 55% of respondents. These 40 competency components ranged across five competency domains: communication, task management, situational awareness, decision-making and leadership, and provided a consensus on the most critical areas for inclusion in training for healthcare personnel. Conclusions. This new competency model is now available for use. Although the sample size was limited, a high degree of consensus was reached after only two rounds. A modified Delphi technique within the context of competencies first refined from the literature was a useful and cost-effective method for determining the content of a 1-day CRM training course for health-care workers. Keywords: crew resource management, teamwork, competency-based education, needs assessment

Introduction Increasingly, health services are delivering aviation crew resource management (CRM) style training to medical staff, with the aim of improving teamwork behaviours and reducing errors. Aviation has become safer over time, and this has been attributed amongst other things to education, so the argument is there is much value in applying the methods to health care [1] in the quest to improve patient safety. A growing body of published literature supports this approach [2–11]. CRM training consists of imparting teamwork knowledge, skills and attitudes in a facilitated learning environment [12], and can be delivered in the classroom, or using a simulator. The majority of present CRM programs are funded commercially or by governments, are usually implemented in large medical centres [10, 13, 14] and are often inaccessible. Many other health-care organizations find it difficult to fund

this type of training, and there is a need for structured information on which mix of competencies should be taught, to allow in-house programs to be developed. Although information on general and specific content of CRM training is available in the public domain, there is little in the way of guidelines on how to select from a growing body of material when designing a training program. Publication of CRM competency components selected by experts for the healthcare environment will facilitate discussion and help work towards standardization of this type of training.

Method Research question The purpose of this study was to determine the optimum content of a 1-day classroom-based CRM course for

Address reprint requests to: Robyn Clay-Williams, Centre for Clinical Governance Research, Australian Institute of Health Innovation, University of New South Wales, Sydney, NSW 2052, Australia. Tel: þ61 421 582 638; Fax: þ61 2 9663 4926; E-mail: [email protected] International Journal for Quality in Health Care vol. 21 no. 6 # The Author 2009. Published by Oxford University Press in association with the International Society for Quality in Health Care; all rights reserved

433

Clay-Williams and Braithwaite

health-care personnel working in complex, time-critical hospital departments such as surgery, intensive care or emergency. Because team composition in these environments is constantly changing, students should ideally be trained in portable, individual team skills. This is the first phase of a larger study investigating the effectiveness of classroom- and simulator-based CRM courses for health-care workers to improve the teamwork attitudes and behaviours of the participants. A number of methods are available for selecting course content. Possibilities included conducting a training needs analysis via interview or focus group, engaging an expert panel to develop the curriculum or using a Delphi procedure to survey experts in the field. While alternative methods have strengths and weaknesses, combining approaches can lead to stronger designs. The Delphi technique within the context of a refined, structured, literature-based competency list was chosen as its main advantage is the ability to obtain timely advice and expertise from a wide range of geographically dispersed participants with an economy of effort. Other healthcare studies have used this approach; it is a useful method that might be labelled an ‘Evidence-Initiated Delphi Process’. In addition to obtaining consensus on the optimum composition of the CRM course, the panel would also provide avenues for examining alternative content elements, thereby militating against inadvertently omitting important competency components from consideration. Delphi process The Delphi process was originally developed by the RAND corporation [15], and has been commonly adopted in healthcare research as a method of obtaining agreement among a group of experts [16]. It is a multistage process that collates and synthesizes opinion of individual panellists to form group consensus. Three defining characteristics of the Delphi method are [17]: anonymity of participants, iterative polling rounds interspersed with feedback and statistical analysis of group results. Qualitative and quantitative data are collected and fed back to participants over a number of rounds until consensus is obtained. The Delphi method has been criticized, most notably by Sackman [18], as not meeting scientific standards in terms of selection of participants, formulation of questions or analysis and interpretation of results. Much of the criticism relates to using Delphi in its original manifestation as a forecasting tool, where broad opinion is solicited about the probability of occurrence of some future event. In this study, rather than formulating a response on a broad topic de novo, seeking initial responses via a series of open-ended questions, participants selected from a large list of competencies distilled from the literature, and seen as valid and applicable to teamwork training for health-care workers. In effect, the respondents contributed to a training needs analysis, whereby they chose the most relevant CRM competency components to prospective trainees. Should the technique succumb to pitfalls, such as group tendency to over-conformity or snap judgements by time pressed respondents, the worst case would

434

be a course that was less efficient, but likely to be still useful and recognizable as CRM training. Although the recommended number of Delphi iterations is often three to four [15], time constraints and the benefit of starting with an extant competency list, meant the process was designed in two rounds following the streamlined RAND-UCLA protocol [19]. Selection of participants Specialist participants were selected based on their knowledge of CRM principles, familiarity with the health-care environment and experience in education or training. This background would not only give panellists the ability to prioritize the most important CRM components for the target trainees, but also to select a group of components that were both teachable in 1 day and coherent as a course of training. As CRM is relatively new to health care, there were few candidates who met this requirement. Thus it was not possible to sample participants randomly. The majority of candidates were anaesthetists running medical simulation training programs. In order to broaden the sample, participants with expertise in developing CRM training for other industries were included. All except one of these industry experts had experience in translating CRM to health care. Candidates were also asked to recommend others that they thought would be able to contribute, and those nominated with a suitable background were contacted. A final group of 21 experts was enrolled. Determination of Round 1 knowledge, skills and attitude components Following an extensive review of the literature on teams and team training in high-reliability industries and in health care, five basic CRM competencies that may be applicable to medical personnel working in surgery, intensive care or emergency departments were determined: communication, task management, situational awareness, decision-making and leadership. These competencies were divided into knowledge, skills and attitudes (KSAs) that were supported by the literature as possible components of a CRM course. Randomizing the statements was seen as counterproductive, as part of the aim was to determine the composition of a cohesive course in addition to prioritizing individual competency components. Competency component statements were formulated as a compromise between optimum length to yield consensus (20 – 25 words) [20], and minimization of ambiguity. As all participants were familiar with CRM principles and techniques, keywords and phrases were able to be used in many cases to convey complex concepts economically. The list of KSAs provided sufficient material for an estimated 3 to 4 days of intensive classroom facilitated training. Delphi panel process In the first Delphi round, participants were given a table listing 102 derived CRM competencies, and asked to select

Health-care CRM Delphi study

those components that they would include if designing a 1-day course. Space was provided at the bottom of each group to add any extra competencies that participants believed should be included, and at the end of the list provision was made for additional comments. Following Round 1, eight KSAs nominated by panel members were added to the competency component list, giving a total of 110 competencies for consideration in Round 2. To minimize the possible questionnaire fatigue effect [21], the order of the subsections was reversed, and presented in the order of leadership, decision-making, situational awareness, task management and communication. Within each subsection, the order of the individual competency components was also reversed. In the second round, participants were provided with collated statistical results from the first round, a summary of the written comments raised by Delphi panel members in Round 1, and investigator responses to any questions. KSAs selected by more than 75% of respondents were highlighted to give participants an idea of the likely composition of the course after the first Delphi round. Respondents were given an opportunity to revise their previous selections based on those results. Initial individual judgements were therefore modified by group information. Data analysis Data collected in Rounds 1 and 2 were entered into an Excel spreadsheet. The percentage of respondents who selected each competency component and the percentage of selections made in each of the five competency categories were determined and graphed. Ethical considerations Ethics approval for the research was granted by the Hunter New England Area Health Service Human Research Ethics Committee (HNEHREC reference no 08/08/20/5.19) and the University of New South Wales Human Research Ethics Committee (HREC reference nos: 08/HNE/262 and 08274). Although the attempt was made to preserve anonymity of candidates, some of the participants were colleagues and, through correspondence on other matters, became aware of each others’ participation. There were no indications that this distorted results. Strict anonymity was maintained for participant responses.

Results Characteristics of respondents Fifteen of 21 invitees returned questionnaires for Round 1, giving a response rate of 71%. The average age of the participants was 47 years (range 35 to .60), and two-thirds were male. The average experiences of respondents were 19.3 years in health care (range 0– 30), 10.9 years in CRM (range 2– 25), 6.9 years in CRM in health care (range 0– 15) and

13.3 years in education or training (range 5– 37). Eleven of 21 invitees returned questionnaires for Round 2, giving a response rate of 52%. Round 1 results The percentage of respondents that selected each competency component is shown at Annex A. Many of the panellists raised concerns regarding the adequacy of 1 day of training to impart the material. Some panellists believed it was not possible to reduce the training to 1 day and still have a meaningful educational product. These respondents tended to select only knowledge components, and advocated training delivery via a lecture. The remainder elected to preserve the experiential adult learning format and attempted to select across a range of competencies most applicable to the needs of the target group. On average, respondents tended to select more than could be reasonably accomplished in 1 day. Round 2 results Competency components selected by participants for inclusion in the course are shown in Table 1. A total of 40 competency components were selected by more than 70% of respondents (Table 1, in boldface). Figure 1 shows a comparison between Rounds 1 and 2 of the percentage of competency components selected for inclusion in the training, and illustrates that fewer selections overall were made in Round 2.

Discussion The Delphi technique contextualized by a refined literature review and competency listing was a useful method for determining the content of a 1-day CRM training course for health-care workers. It provided a rapid, low-cost alternative to a complex training needs analysis conducted by multiple interviews, focus groups or expert panels. Identification of competency components within each of the five competency domains such as communication, task management, situational awareness, decision-making and leadership provides confirmation of a broad correlation between the outcomes of this study and the content of the major CRM programs implemented in health-care today, where each of these competencies is normally included. By providing a list of ranked competency components, the study will also allow trainers the flexibility to tailor training to the circumstances of their team. In addition to providing recommendations for content of a 1-day course, the competency component list may provide a greater degree of transparency of the underlying structure of health-care CRM programs in general. The individual competency components can often be inferred by reviewing program material, but they are normally not explicitly listed. The percentage of participants who selected each component may also provide a guide to prioritization for trainers who have less than a full day available for training, or who have

435


Table 1 Delphi panel results: CRM’s KSAs % of panel selecting competency

...............................................

Round 1 panel

Round 2 panel

.............................................................................................................................................................................

Competency: communication (communicates efficiently and effectively) Knowledge Discusses the importance of open communications in medicine. Describes call-out and check-back procedures. Describes strategies to deal with conflict. Describes a method for asserting a corrective action. Describes a method for articulating concern about a course of action. Lists the elements of a brief. Describes situations where a brief would be appropriate to facilitate group performance. Lists the elements of a debrief. Describes situations where a debrief would be appropriate to facilitate group performance. Lists the steps in SBAR. Lists the elements of an effective handover brief. Discusses the importance of clear written communication in medicine. Skills Establishes an atmosphere to encourage open communications. Listens critically and provides feedback to clarify information. Implements strategies to deal with conflict. Asserts a corrective action in a firm and respectful manner. Articulates when concerned about a course of action. Conducts briefs where appropriate to facilitate group performance. Conducts debriefs where appropriate to facilitate group performance. Communicates critical information in a succinct manner. Conducts effective handover briefs. Attitudes Speaks up when concerned. Treats others with respect. Values the importance of listening. Values the importance of resolving conflict in the workplace. Displays a positive attitude towards briefing. Displays a positive attitude towards debriefing. Values the importance of clear written communication. Voices uncertainty, ‘I don’t know’. Requests others opinions Competency: task management (manages tasks efficiently and effectively) Knowledge Describes contemporary approaches to human error (systems approach). Describes reason’s taxonomy of errors (slips, lapses, mistakes, violations). Describes a method of minimizing error (avoid, trap, mitigate). Describes how high and low workload can contribute to error. Describes how checklists and standardization of procedures can minimize error. Describes the advantages and disadvantages of automation in regard to error. Describes the incident and adverse event reporting procedure. Lists the types of incidents or adverse events that should be reported. Skills Manages changing priorities and if necessary, re-focus team members to accommodate the changed priorities. Prioritizes critical tasks. Maintains vigilance during routine tasks or periods of low workload.

100 92 69 85 100 54 62

91 91 36 91 100 9 55

38 38

0 18

69 77 46

36 91 18

85 85 69 85 85 38 15 85 62

100 91 73 91 91 18 9 100 27

92 77 69 54 46 54 38 – –

100 82 45 36 9 9 0 18 27

92 54 54 85 100 46 38 23

100 18 45 100 100 18 9 9

77

91

38

18 (continued )

436


Continued % of panel selecting competency

...............................................

Round 1 panel

Round 2 panel

.............................................................................................................................................................................

Monitors systems, environment and team members, collects and analyses information to identify potential or actual errors. Implements strategies and procedures to prevent errors or takes action in the time available to correct errors. Applies checklists and standard operating procedures to prevent clinical, procedural or communication errors; and identifies committed errors before patient safety is affected. Allocates sufficient resources and time to complete workload. Distributes and assigns work thoughtfully; re-allocates functions where required. Reports incidents and adverse events. Steps back/hands off. Attitudes Displays a respect for safe practices. Displays a positive attitude towards open reporting and a blame-free culture. Appreciates the ubiquitous nature of error and acknowledges the potential for error in self and others. Acknowledges that safety is everyone’s responsibility. Applies accepted clinical knowledge. Competency: situational awareness (maintains situational awareness) Knowledge Describes the concept of ‘situational awareness’, and how to identify when it has been lost. Describes the concept of a team ‘shared mental model’. Lists the steps in identifying work environmental or operational threats that could affect safety of the patient. Describes a support process to facilitate maintenance of team situational awareness. Describes a method of providing constructive feedback. Describes strategies that may be used to maintain individual situational awareness. Lists the elements of SITREPS and RECAPS. Describes the concept of a ‘mental rehearsal’, and where it may be used to improve performance. Skills Ensures that all team members have a clear picture of the objective. Ensures that all team members have role clarity and relevant information to achieve goals. Identifies work environmental or operational threats that could affect safety of the patient. Uses support process to facilitate maintenance of team situational awareness. Provides constructive feedback to other team members for improving performance. Asks questions to facilitate understanding of the situation. Uses strategies to maintain individual situational awareness. Maintains shared mental models via SITREPS and RECAPS. Employs mental rehearsals, where appropriate, to facilitate improved performance. Attitudes Appreciates the importance of maintaining situational awareness. Appreciates the importance of a team ‘shared mental model’. Displays a positive attitude towards asking and answering questions. Displays a positive attitude towards providing and receiving feedback.

69

55

62

36

54

45

46 69 54 –

18 55 27 18

85 77 92

73 91 100

92 –

100 9

85

100

77 46

73 18

54

27

31 100 15 54

18 100 9 27

85 77

91 91

31

18

8 38

0 9

77 85 23 15

73 100 18 0

77 54 92 77

91 45 91 82 (continued )

437


Continued % of panel selecting competency

...............................................

Round 1 panel

Round 2 panel

.............................................................................................................................................................................

Appreciates the importance of SITREPS and RECAPS in maintaining a team ‘shared mental model’. Appreciates the importance of mental rehearsals in improving performance. Competency: decision-making (makes appropriate decisions) Knowledge Describes the process of naturalistic decision-making. Describes the process of analytical decision-making. Lists the steps in ‘OODA loop’ monitoring. Defines risk in terms of consequence and likelihood. Describes reason’s ‘three bucket method’ of real-time risk assessment. Skills Employs naturalistic decision-making techniques in time-critical situations. Employs analytical decision-making techniques when time is not a factor. Assesses solutions and risks with other team members. Readily anticipates and shares the information needs of other team members. Decides on a course of action. Communicates plans of action and directs team members to clearly specified tasks. Provides ‘if, then. . . ’ direction to team members to facilitate future action. Monitors and assesses progress to ensure a safe outcome; or modifies action when a safe outcome is not assured. Adopts real-time risk management strategies. Attitudes Appreciates the importance of making a decision. Is willing to review decisions in light of new information. Displays a positive attitude to managing risk. Understands the concept of ‘Global Review’. Competency: leadership (displays effective team skills in position of leader or follower) Knowledge Lists common characteristics of effective leadership. Lists common characteristics of effective followership. Lists and describes individual’s attitudes that are hazardous to safety. Lists and describes group attitudes and behaviours that may adversely impact performance. Lists individual’s physiological factors that may adversely impact performance. Understands [describes] the importance of hands-off role of leader. Skills Leader considers the condition (capability) of other team members to perform duties. Leader monitors and appraises team members’ performance. Leader assists other team members in demanding situations. Backs-up and fills in for other team members as required. Leader motivates and encourages other team members. Self-assesses physiological ability to perform on the job. Cross-monitors actions of other team members. Steps back/hands off leadership role. Appropriate allocation of roles and responsibilities. Attitudes Is an active member of the team. Displays a positive attitude to the importance of giving and receiving support. Maintains appropriate interpersonal skills. Displays a positive attitude to the importance of individual and group attitudes in the work environment.

438

15

9

15

9

69 62 23 54 23

36 36 0 18 0

38 23 69

18 9 55

77 77 69 62

91 100 45 18

23

0

62 92 62 –

55 100 27 9

100 92 62 85

100 100 36 100

85 –

91 18

77

91

62 38

55 9

62 46 85 – –

36 9 100 27 36

62 69 69 69

45 45 55 55


Figure 1 Percentage of competency components selected for inclusion in the training. identified a specific problem in their team that needs to be addressed. The stated competency component could be used to access public domain training material (e.g. DoD TeamSTEPPS) and provide a flexible and cost-effective path to developing customized training and effective subsequent evaluation. In addition to fewer selections overall in Round 2, the final selections for each competency component appeared to be more polarized ‘for’ or ‘against’ in Round 2 than in Round 1. Possible explanations for these results include that participants considered the training holistically and pruned their selections to a realistic number of components for 1 day of training; participants were influenced by the 75% line drawn separating components for inclusion or deletion after the first round and responded to that rather than the individual components; or participants succumbed to group conformity whereby the first round effectively summarized the ‘expert correct answer’ to which everyone then agreed. A likely explanation is a combination of the three: most of the panel were conversant with the Delphi process, and, being aware that a decision had to be made after the second round, was helpful in trying to reach maximum consensus as a group. Polarization of final responses will facilitate selection of competency components for inclusion in the training. Forty competency components were selected by more than 70% of respondents, whereas the remaining components were selected by fewer than 55%. The 40 competency components provided an indication of the most critical areas for inclusion in training for communication, task management, situational awareness, decision-making and leadership competencies for health-care personnel. To minimize the number of required Delphi rounds, it is important to give panel members as much information about the research question as possible; however, provision of too much guidance, such as the 75% cut-off criteria for selection at the end of the first round, may introduce bias against alternate views. Concerns regarding generalizability of the data from a small participant sample size are somewhat mitigated by the high level of agreement reached after only two rounds. This study showed that the recommended three to four rounds may not

be necessary to obtain consensus when the presented baseline subject matter is evidence based and panellists are well acquainted with the Delphi method. A limitation of this study is the large number of variables available for selection. Whilst it was imperative to include all of the major competency components that emerged from the literature review, this has potential to compromise the accuracy of this method. However, fears of questionnaire fatigue appear to be baseless. There were more decisionmaking and leadership components selected in Round 1, where they were at the end of the survey, than in Round 2 where they were placed at the beginning.

Conclusions Although health-care CRM training has been widely implemented over the last decade, there is not a large body of evidence linking the training to improved outcomes in patient care. It is requested that trainers who decide to utilize this material consider the importance of training evaluation and its contribution to the evidence base supporting the efficacy of CRM in health care.

Acknowledgements We would like to acknowledge the assistance of the members of the Delphi panel, who provided their unrecompensed expertise in the form of timely responses to the questionnaires and helpful suggestions concerning this study and the research design.

Funding This research was partially supported under a University of New South Wales scholarship and by National Health and Medical Research Council (NHMRC) Programme grant 568612.

439


References 1. Helmreich RL. On error management: lessons from aviation. Br Med J 2000;320:781 –5.

11. Boddington R, Arthur H, Cummings D et al. Team Resource Management and patient safety: a team focused approach to clinical governance. Clin Governance 2006;11:58 –68.

2. Kohn LT, Corrigan JM, Donaldson MS. To err is human: building a safer health system. A report of the Committee on Quality of Health Care in America, Institute of Medicine. Washington, DC: National Academy Press, 2000.

12. Helmreich RL, Foushee HC. Why crew resource management? Empirical and theoretical bases of human factors training in aviation. In Wiener EL, Kanki, Barbara G, Helmreich, Robert L et al. (eds). Cockpit Resource Management. San Diego, CA: Academic Press, 1993.

3. Helmreich RL, Sexton B. Managing threat and error to increase safety in medicine. Teaming Up: Components of Safety Under High Risk. Aldershot: Ashgate, 2004.

13. Dunn EJ, Mills PD, Neily J et al. Medical team training: applying crew resource management in the veterans health administration. Jt Comm J Qual Patient Saf 2007;33:317 –25.

4. Musson DM, Helmreich RL. Team training and resource management in healthcare: Current issues and future directions. Harv Health Policy Rev 2004;5:25 –35.

14. Bleakley A, Boyden J, Hobbs A et al. Improving teamwork climate in operating theatres: the shift from multiprofessionalism to interprofessionalism. J Interprof Care 2006;20:461 –70.

5. Morey JC, Simon R, Jay GD et al. Error reduction and performance improvement in the emergency department through formal teamwork training: evaluation results of the MedTeams project. Health Serv Res 2002;37:1553 –81.

15. Dalkey N, Helmer O. An experimental application of the Delphi method to the use of experts. Manag Sci 1963;9:458–67.

6. Baker DP, Gustafson S, Beaubien JM et al. Medical Teamwork and Patient Safety: The Evidence-Based Relation. Washington, DC: American Institute for Research, 2003.

17. Dalkey NC. The Delphi Method: An Experimental Study of Group Opinion. Santa Monica, CA: United States Air Force Project RAND, 1969.

7. Salas E, Bowers CA, Edens E. Improving Teamwork in Organizations: Applications of Resource Management Training. Mahwah, NJ: Lawrence Erlbaum Associates Publishers, 2001.

18. Sackman H. Delphi Critique; Expert Opinion, Forecasting, and Group Process. Lexington, Mass: Lexington Books, 1974.

8. Gaba DM, Howard SK, Fish KJ et al. Simulation-based training in anesthesia crisis resource management (ACRM): a decade of experience. Simul Gaming 2001;32:175 –93. 9. Flin R, O’Connor P, Mearns K. Crew resource management: improving team work in high reliability industries. Team Perform Manage 2002;8:68–78. 10. Alonso A, Baker DP, Holtzman A et al. Reducing medical error in the Military Health System: how can team training help? Hum Resour Manage Rev 2006;16:396–415.

440

16. Jones J, Hunter D. Qualitative research: consensus methods for medical health services research. Br Med J 1995;311:376 –80.

19. Brook R. The RAND-UCLA appropriateness method. In McCormick KA, Siegel R (eds). Clinical Practice Guideline Development: Methodology Perspectives. Rockville, MD: U.S. Department of Health and Human Services: Agency for Health Care Policy and Research, 1994. 20. Salancik JR, Wenger W, Helfer E. The construction of Delphi event statements. Technol Forecast Soc Change 1971;3:65 –73. 21. Converse JM, Presser S. Survey Questions: Handcrafting the Standardized Questionnaire. Beverly Hills: Sage Publications, 1986. Accepted for publication 13 September 2009