A Comprehensive Methodology for Examining the ...

Proceedings of the Human Factors and Ergonomics Society 58th Annual Meeting - 2014

703

A Comprehensive Methodology for Examining the Impact of Surgical Team Briefings and Debriefings on Teamwork Katherine E. Law, M.S.1, Emily Hildebrand, Ph.D.2, Joao Oliveira-Gomes, M.S.3, Susan Hallbeck, Ph.D.3, Renaldo C. Blocker, Ph.D.3 1 University of Wisconsin-Madison, Industrial and Systems Engineering Department, Madison, WI 2 Aroniza State University, College of Technology & Innovation, Mesa, AZ 3 Mayo Clinic Robert D. and Patricia E. Kern Center for the Science of Health Care Delivery, Rochester, MN The adoptions of briefing and debriefing protocols have evolved from the Joint Commission’s initiative to improve communication and safety in the operating room. Briefing normally occurs prior to incision and is used to discuss and confirm critical information, while debriefing occurs during or after surgery. Debriefing provides a unique opportunity for individuals and teams to immediately reflect on their performance, allowing them to more easily identify errors and develop plans to improve their next performance. Studies have shown that using briefings and debriefings improve communication and teamwork. However, there is still much to learn about the value of both for surgical teams. This paper presents a robust methodology for examining and measuring the impacts of surgical team briefings and debriefings on teamwork. The methodology includes (1) audio/video recording the surgical care process, (2) prospective observations using a validated electronic data collection tool, (3) pre- and post-surgery surveys, and (4) individual surgical team member interviews. The current paper describes the methodology to obtain a robust and comprehensive data set for analyzing the impacts of briefing and debriefing on teamwork; the results of the surgeries recorded using this methodology will be presented in subsequent papers.

Copyright 2014 Human Factors and Ergonomics Society. DOI 10.1177/1541931214581164

INTRODUCTION In the interest of patient safety, medical teams responsible for complex tasks must perform at optimum levels. However, due to a fragmented health care system, occurrences of avoidable medical errors are very common in medical practice, with 44% of adverse events in Medicare patients considered likely preventable (Levinson, 2010). These errors are more likely to have serious consequences for patients if they occur in critical units such as the operating room (OR), where the stakes are very high (Christian et al., 2006). Failures in team communication and cohesiveness in the OR are quite common and have been deemed as a main cause of human errors during surgeries (Dwyer, 2004; Lingard et al., 2004; Gawande, Sinner, Studdert, & Brenan, 2003). External pressures to reduce costs, time constraints, and team member rotations during surgeries are possible explanations for communication failures during surgical procedures. The Joint Commission has proposed initiatives to combat these communication and safety issues in the OR through the adoption of briefing and debriefing protocols. Briefing typically occurs prior to incision and is used to discuss and confirm critical information: surgical procedure, safety concerns, or patient information. Debriefing occurs during or after surgery and provides a unique opportunity for individuals and teams to reflect on their practice, allowing them to more easily identify errors and develop plans to improve their next surgical performance. While briefing has become a standard of care in the OR, debriefing is still not a regular practice in U.S. hospitals (Ahmed et al., 2012). Previous studies have successfully related the implementation of structured debriefings protocols in the OR

with improvements in performance, team communication and error identification. Surgical debriefing has been proven to reduce the incidence of adverse events (Hamad, Brown, & Clavijo-Alvarez, 2007; Mahmood & Darzi, 2004), improve technical performance (Hamad et al., 2007; Rogers, Regehr, Howdieshell, Yeh, & Palm, 2000) and maximize clinical learning (Hamad et al., 2007; Mahmood & Darzi, 2004; McGaghie, Issenberg, Petrusa, & Scalese, 2010). Recent literature has shown that debriefing could also improve communication and teamwork in the OR. Berenholtz and colleagues (2009) developed and implemented a debriefing process implemented in the OR that took 2.5 minutes to complete. The authors followed up with surveys to evaluate caregiver perceptions on teamwork and the tool. The majority of caregivers reported that their tool improved interdisciplinary communication and teamwork. Another study reported that implementation of a briefing and debriefing protocol resulted in decreased delays and improved cases scores (Wolf, Way, & Stewart, 2010). Additionally, Papaspyros , Javangula, Adluri, and O’Regan (2010) were able to establish a relationship between the existence of structured debriefing protocols and improvements in patient safety in the cardiac OR. After implementing a briefing and debriefing protocol, team members were able to identify and resolve multiple errors and failures, such as faulty instruments or excessive OR traffic. Several authors have also proposed key components and methodologies for an effective debriefing. While some were mostly based on personal experience (Rudolph, Simon, Raemer, & Eppich, 2008; Dieckmann, Molin, Lippert, & Ostergaard, 2009), an evidence-based tool, Objective Structured Assessment of Debriefing (OSAD), was developed

Downloaded from pro.sagepub.com at University of Wisconsin-Madison on June 23, 2015


and tested for debriefing in surgeries (Arora et al., 2012). This tool was reported to be feasible, reliable and valid for use in debriefing scenarios. In another study aimed to evaluate if an education intervention named “SHARP: 5-step Feedback Tool for Surgery” could possibly improve the quality and quantity of debriefing in surgery, the inter-rater reliability of OSAD was evaluated as excellent (0.994) (Ahmed et al., 2013). During the study, Ahmed and colleagues (2013) found that using SHARP significantly increased the quality of debriefing, promoted patient safety, and was well accepted by surgical trainees and trainers. As evidenced by the aforementioned tools, interest in the importance and impact of debriefings on teamwork has increased; however, there still exists a need for an extensive direct measurement of debriefing effectiveness on teamwork. Therefore, this paper presents a methodology for investigating the current practice of surgical team briefings and debriefings, regardless of specialty, in order to understand and accurately measure its impact on team functioning. METHODOLOGY The development of this methodology was based on an institutional and national (Zuckerman et al., 2012) need to demonstrate the benefits of debriefing, increase compliance and commitment through clinicians’ buy-in, and discover the most appropriate time to debrief. The methodology includes (1) audio/video recording the surgical care process, (2) prospective observations using a validated electronic data collection tool, (3) pre- and postsurgery surveys, and (4) individual surgical team member interviews in order to obtain a robust and comprehensive data set for analyzing the impacts of briefing and debriefing on teamwork; the results of the surgeries recorded using this methodology will be presented in subsequent papers. The following section will elaborate on the methodological process, as shown in Figure 1. Study setting and participants The methodology was created by a research team of human factors experts and one clinical expert at a 794-bed non-profit academic tertiary medical center. The framework was validated for the institution’s gynecological surgery

704

department. Over a six month period, two human factors researchers used this methodology to assess the impact of surgical team debriefings on teamwork and team performance in the OR. The researchers observed and video-recorded surgical procedures and administered surveys to multiple gynecological surgery teams. For the study, a surgical day was defined as the scheduled cases for a surgical team in the same OR for one day. The researchers observed and video-recorded the process from the time of patient arrival in the operating room to the time of patient departure from the operating room. In addition, the researchers recorded the morning team briefing and debriefing for each case. Using questions derived from the Applied Cognitive Task Analysis methodology, researchers also conducted semistructured interviews with individual surgical team members (Millitello & Hutton, 1998). Interviews were held in an empty OR during operating hours and usually lasted 20 minutes. The interviews were intentionally kept short as time is valuable for surgical team members. The surgical team included the following roles: surgeon, resident, circulating nurse (RN), certified surgical technologist (CST), certified surgical assistant (CSA), certified registered nurse anesthetist (CRNA), and anesthesiologist. Because the surgical team could include others not mentioned above, the surgical team was defined by those present at the team briefing or at the start of the case. Study Tools This methodology combines: (1) prospective observations using a Tablet-PC data collection tool for identifying non-routine events (Blocker, Eggman, Zemple, Wu, & Wiegmann, 2010), (2) audio and video recordings using a GoPro® Hero 3+ Camera, (3) pre- and post-surgery surveys, and (4) individual surgical team member interviews. Obtaining buy-in The researchers went about identifying champions and obtaining buy-in from surgical staff and clinicians three months prior to data collection. Literature indicates there are many benefits in identifying champions for change in an organization (Greenhalgh, Robert, Macfarlane, Bate, &

Figure 1. Timeline of the comprehensive methodology



705

Table 1. Event categorization by type and impact (Blocker et al., 2010; Blocker et al., 2012; Parker et al., 2010) Type Communication Coordination

Environment Equipment External interruptions

Patient factors Technical skills Training Other Impact No impact Acknowledge/no delay Momentary delay Moderate delay Full case cessation

Definition Any breach or lapse in team cooperativeness, cohesion, and/or familiarity as a result of verbal information exchanged (e.g. misunderstanding, communication unheard, case-related communication, extraneous conversation) Any breach or lapse in team cooperativeness, cohesion, and/or familiarity as a result of poor organization function (e.g. personnel exchanges, improperly configured equipment, not adhering to surgeon or team preferences, requesting or providing assistance to fellow team members) Any disruption affecting the auditory or visual status of the operating room setting based on the physical characteristics of the room (e.g. noise, temperature, lighting). Any equipment problem hindering the smooth progression of the surgical procedure (e.g. malfunctions, improper use, unfamiliar equipment, maintenance). Any disruption affecting the auditory or visual status of the operating room that is not directly relevant to the treatment of the patient (e.g. extraneous people, phone calls, or intercom messages that did not relate directly to the procedure at hand). Disruptions related to the patient’s unique anatomy (e.g. excessive amount of unanticipated adhesions or scar tissue). Any skill-based or decision (thinking) error (e.g. poorly executed tasks, misinterpretation of relevant information, omitted steps). Teaching a new skill, correcting an improper action, posing questions to test the knowledge of the team, student, or trainee. Any disruption not falling into one of the above categories Definition No acknowledgement of the event occurring despite the disruption. A surgical team member is aware of the disruption, but there is no pause in the ﬂow of the operation. Pauses in surgical flow of the operation for 10 seconds as a result of the surgical ﬂow disruption. One or more surgical team members pause their current task and engage in a secondary activity that impeded the progress of the original task and significantly disrupts surgical flow of the operation.

Kyrakidou, 2004; Rogers, 2010), and it has been shown that some clinical staff members are more likely to buy-in to change when champions identify with their professional group (Gardner, Dowden, Togni, & Bailie, 2010), so the researchers met with each role individually. The researchers also confirmed with the institution’s legal department that procedure recordings could not be used for performance evaluation. Department surgeons were the first group approached to foster a positive relationship between clinicians and the researchers. The researchers discussed their interests in understanding teamwork issues in the OR setting and addressed surgeons’ concerns. Following, the researchers met with the departments for each surgical role and presented on research objectives during staff meetings. The researchers were transparent in data collection plans, and discussed developing a video policy to allay staff fears that the videos could be used to evaluate performance. During this meeting, the researchers also explained that the surgical staff members are the experts at their jobs. The purpose of the research is to learn from staff, not staff from researchers. The researchers confirmed with the institution’s legal department that procedure recordings could not be used for performance evaluation. From the meetings and presentations, the researchers were able to obtain buy-in and recruit surgical team members in each role to act as champions for the project.

Rationale for electronic data collection tool Non-routine events (NREs), or surgical flow disruptions, have previously been linked to operative errors that occur during cardiac surgery (Wiegmann, ElBardissi, Dearani, Daly, & Sundt III, 2007). A validated, electronic Tablet-PC data collection tool was developed to identify NREs and has been generalized for use beyond the cardiac OR (Blocker et al., 2010). The tool has been used in prior studies to identify NREs and their impact on the natural progression of surgical cases and team functioning in the cardiac OR and emergency department (Blocker et al., 2012, Gangi et al., 2014). The data collection tool as described by Blocker and colleagues (2010), prospectively records information about NREs that disrupted the flow of the surgical care process. NREs are time-stamped and categorized by observers in real time according to (1) type of NREs (for definitions, see Table 1); (2) the potential and/or actual impact of the NREs using a scale (1=no impact; 2=acknowledge/no delay; 3= momentary delay; 4=moderate delay; 5=full case cessation) (for definitions, see Table 1); (3) the surgical team member affected by the NREs; and (4) the description of the NREs. Through capturing NREs, it will be possible to identify if a relationship between debriefings and operative error exists. Rationale for audio and video recordings Using audio and video recording for the entire surgical case that includes team briefings and debriefings



provided an opportunity for the researchers to (1) revisit the surgery procedure, (2) clarify NREs that occurred during the procedure, and (3) capture events that were missed or could not be captured prospectively. Video recording allowed researchers to capture much of the complexity of the surgical care process, supplemented the prospective observation data, and provided an opportunity to increase quality and reliability of data capture. As a result, the researchers were able to link the impact of debriefing on teamwork with respect to NREs. Tools have previously been developed to measure teamwork in the operating room (Yule et al., 2008; Mishra, Catchpole, & McCulloch, 2009). The Oxford Non-Technical Skills (NOTECHS) scale is often used to measure the quality of teamwork; in addition, the Non-technical Skills for Surgeons (NOTSS) behavior system has been used to provide feedback pertaining to observable non-technical skills. Because NOTSS was developed to allow feedback to novice and consultant surgeons, the researchers surmised it could also measure the impact of debriefings on surgical performance. This study used both NOTECHS and NOTSS to assess the impacts of debriefing on surgical performance. The results will be presented in a subsequent paper. Rationale for pre- and post-surgery surveys As mentioned earlier, the gynecological surgery service performs one team briefing in the morning for all procedures for a case in the same OR. Conducting a presurgery survey after the morning team briefing, just prior to the start of the first scheduled surgery procedure, provided a baseline for measuring the clinician perspective on teamwork. A post-surgery survey was conducted at the end of each surgical procedure for the case. The pre- and post-surgery surveys assessed the clinicians’ opinions about the morning team briefing and the procedure debriefing(s), respectively. These surveys captured clinician perceptions of what information discussed in the morning team briefing and debriefings was essential to teamwork. In addition, the preand post-surgery surveys captured the clinicians’ opinions regarding what information should have been included in the team briefing and debriefings to enhance teamwork and to circumvent NREs that occurred during the surgical procedures. The pre- and post-surgery surveys results were able to be cross-checked for accuracy using the video recording and data collection tool. Rationale for interviews Given the complex nature of the surgery care processes, some information and details are unobservable and cannot be captured or known through the data collection tool, video recording, and surveys; therefore, interviewing surgical team members in various roles provides a more in-depth understanding of processes, perceptions, culture, and tacit knowledge related to teamwork and briefings and debriefings. Interviewing team members allows for further discussion of what information should be included in a debriefing and when

706

that information should be presented in the briefing and debriefing process. Over a one week period, the researchers interviewed at least two members of the surgical team, including: surgeon, resident, circulating nurse (RN), certified surgical technologist (CST), certified surgical assistant (CSA), certified registered nurse anesthetist (CRNA), and anesthesiologist. The interviews captured in-depth data on why each role believed certain information was vital for team briefings and debriefings, the underlining reasons why compliance was steady, and the reasoning for why buy-in was not consistent across all roles. DISCUSSION & CONCLUSION While previous studies have used each of the aforementioned data collection methods independently or have combined two methods, the current study integrated four methods to create a robust scientific methodology for understanding and measuring the briefing and debriefing impact on teamwork. It is widely known the limitations to performing observations; however, through multiple methods, this methodology allowed for triangulation to reduce the Hawthorne effect and better understand the multifaceted issues in healthcare (Holden, 2001). As a result, the researchers were able to create a briefing and debriefing protocol specific to gynecological surgery. Based on this validation study, this methodology could be applied across multiple surgical specialties to understand and examine various aspects of the complex surgical and emergency environments. There are challenges to implementing such a robust methodology in the surgical suites. In this study, obtaining buy-in with surgical staff was time intensive and more difficult to earn due to the use of video recording; however, it is a common issue many researchers face in the clinical environment (Mackenzie, Martin & Xiao, 1996; Mackensie & Xiao, 2003; Oakley et al., 2006; Shouhed et al., 2014). The researchers had to ensure that videos were neither accessed nor watched by those in administrative positions. As a result, the videos could not be viewed legally from an administrative standpoint, and the videos could not be subpoenaed in court. For the added protection of the clinicians, the researchers established a policy where all videos were analyzed and automatically destroyed within seven days of the surgical procedure recording. With this policy, in conjunction with laws surrounding quality improvement research, the researchers were able to gain buy-in from the clinicians and administration to implement the methodology. While this methodology is undoubtedly valuable, it is personnel intensive due to the many aspects involved in data collection and analysis. Additionally, the researchers noted the importance of including a clinical expert on the research team. The clinical expert used for this methodology worked at the same institution as the surgical teams observed, which could influence their perception of performance; as a result, only the human factors experts were able to observe surgeries and



perform data analysis. In the future, the researchers recommend including a clinical expert from outside the institution to provide a diverse perspective from buy-in to data analysis and synthesis. Furthermore, identifying NREs were used as a surrogate measure to predict adverse events and surgical error. The researchers hope future iterations will include direct measurement of clinical outcomes to aid in understanding the impacts of team communication and cohesion and informing the development of tailored team briefings and debriefs. As indicated, this paper presented a methodology for measuring briefing and debriefing effects on teamwork. The results from using this methodology will be presented in subsequent papers. REFERENCES Ahmed, M., Arora, S., Russ, S., Darzi, A., Vincent, C., Sevdalis, N. (2013). Operation debrief: a SHARP improvement in performance feedback in the operating room. Annals of surgery, 258(6),958-963. Ahmed, M., Sevdalis, N., Paige, J., Paraqi-Gururaja, R., Nestel, D., Arora, S. (2012). Identifying best practice guidelines for debriefing in surgery: a tri-continental study. The American Journal of Surgery, 203(4), 523-529. Arora, S., Ahmed, M., Paige, J., Nestel, D., Runnacles, J., Hull, L., Darzi, A., Sevdalis, N. (2012). Objective structured assessment of debriefing: bringing science to the art of debriefing in surgery. Annals of surgery, 256(6), 982-988. Berenholtz, S.M., Schumacher, K., Hayanga, A.J., Simon, M., Goeschel, C., Pronovost, P.J., Shanley, C.J., Welsh, R.J. (2009). Implementing standardized operating room briefings and debriefings at a large regional medical center. Joint Commission Journal on Quality and Patient Safety, 35(8), 391-397. Blocker, R., Duff, S., Wiegmann, D., Catchpole, K., Blaha, J., Shouhed, D., Ley, E., Karl, C., Karl, R., and Gewertz, B. (2012). Flow disruptions in trauma surgery: type, impact, and affect. Human Factors and Ergonomics Society Annual Meeting Proceedings, 56(1), 811-815. Blocker, R.C., Eggman, A., Zemple, R., Wu, C.E., & Wiegmann, D.A. (2010). Developing an observational tool for reliably identifying work system factors in the operating room that impact cardiac surgical care. Human Factors and Ergonomics Society Annual Meeting Proceedings, Health Care, 5, 879-883. Christian, C. K., Gustafson, M. L., Roth, E. M., Sheridan, T. B., Gandhi, T. K., Dwyer, K., ... & Dierks, M. M. (2006). A prospective study of patient safety in the operating room. Surgery, 139(2), 159-173. Dieckmann, P., Molin, F.S., Lippert, A., Ostergaard, D. (2009). The art and science of debriefing in simulation: Ideal and practice. Medical teacher, 31(7), e287-e294. Dwyer, K. (2002) Surgery-related claims and the systems involved. The Journal of medical practice management: MPM, 18(6), 332-336. Gangi, A., Blaha, J., Law, K. E., Shouhed, D., Gewertz, B., Ley, E. J., ... & Catchpole, K. (2014). Standardized Teamwork Training Improves Trauma Workflow. Journal of Surgical Research, 186(2), 657-658. Gardner, K. L., Dowden, M., Togni, S., & Bailie, R. (2010). Research article Understanding uptake of continuous quality improvement in Indigenous primary health care: lessons from a multi-site case study of the Audit and Best Practice for Chronic Disease project. Gawande, A.A., Sinner, M.J., Studdert, D.M., Brenan, T.A. (2003). Analysis of errors reported by surgeons at three teaching hospitals. Surgery, 133(6), 614-621. Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., & Kyriakidou, O. (2004). Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Quarterly, 82(4), 581-629. Hamad, G.G., Brown, M.T., and Clavijo-Alvarez, J.A. (2007). Postoperative video debriefing reduces technical errors in laparoscopic surgery. The American Journal of Surgery, 194(1), 110-114.

707

Holden, J. D. (2001). Hawthorne effects and research into professional practice. Journal of evaluation in clinical practice, 7(1), 65-70. Levinson, D. R. (2010). Adverse events in hospitals: national incidence among Medicare beneficiaries. Department of Health & Human Services. Lingard, L., Espin, S., Whyte, S., Regehr, G., Baker, G., Reznick, R., Bohnen, J., Orser, B., Grober, E. (2004) Communication failures in the operating room: an observational classification of recurrent types and effects. Quality and Safety in Health Care, 13(5), 330-334. Mackenzie, C. F., Martin, P., & Xiao, Y. (1996). Video analysis of prolonged uncorrected esophageal intubation. Anesthesiology, 84(6), 1494-1503. Mackenzie, C. F., & Xiao, Y. (2003). Video techniques and data compared with observation in emergency trauma care. Quality and Safety in Health Care, 12(suppl 2), ii51-ii57. Mahmood, T. and Darzi, A. (2004). The learning curve for a colonoscopy simulator in the absence of any feedback: no feedback, no learning. Surgical Endoscopy And Other Interventional Techniques, 18(8), 1224-1230. McGaghie, W.C., Issenberg, S.B., Petrusa, E.R., Scalese, R.J. (2010). A critical review of simulation‐based medical education research: 2003–2009. Medical education, 44(1), 50-63. Millitello, L.G., & Hutton, R.J.B. (1998). Applied cognitive task analysis (ACTA): a practitioner’s toolkit for understanding cognitive task demands. Ergonomics, 41 (11), 1618-1641. Mishra.A., Catchpole, K., McCulloch, P. (2009). The oxford NOTECHS system: Reliability and validity of a tool for measuring teamwork behaviour in the operating theatre. Quality and Safety in Health Care, 18, 104-108. Oakley, E., Stocker, S., Staubli, G., & Young, S. (2006). Using video recording to identify management errors in pediatric trauma resuscitation. Pediatrics, 117(3), 658-664. Papaspyros, S.C., Javangula, K.C., Adluri, R.K., O’Regan, D.J. (2010). Briefing and debriefing in the cardiac operating room. Analysis of impact on theatre team attitude and patient safety. Interactive cardiovascular and thoracic surgery,10(1), 43-47. Rogers, D.A., Regehr, G., Howdieshell, T.R., Yeh, K.A., Palm, E. (2000). The impact of external feedback on computer-assisted learning for surgical technical skill training. The American Journal of Surgery, 179(4), 341-343. Rogers, E. M. (2010). Diffusion of innovations. Simon and Schuster. Rudolph, J.W., Simon, R., Raemer, D.B., Eppich, W.J. (2008). Debriefing as formative assessment: closing performance gaps in medical education. Academic Emergency Medicine, 15(11), 1010-1016. Shouhed, D., Blocker, R., Gangi, A., Ley, E., Blaha, J., Margulies, D., ... & Catchpole, K. R. (2014). Flow Disruptions During Trauma Care. World journal of surgery, 38(2), 314-321. Wiegmann, D. A., ElBardissi, A. W., Dearani, J. A., Daly, R. C., & Sundt III, T. M. (2007). Disruptions in surgical flow and their relationship to surgical errors: an exploratory investigation. Surgery, 142(5), 658665. Wolf, F.A., Way, L.W. and Stewart, L. (2010). The efficacy of medical team training: improved team performance and decreased operating room delays: a detailed analysis of 4863 cases. Annals of surgery, 252(3), 477-485. Yule, S., Flin, R., Maran, N., Rowley, D., Youngson, G., and Paterson-Brown, S. (2008). Surgeons’ non-technical skills in the operating room: realiability testing of the NOTSS behavior rating system. World Journal of Surgery, 32: 548-556. Zuckerman, S., France, D., Green, C., Leming-Lee, S., Anders, S., and Mocco, J. (2012). Surgical debriefing: a reliable roadmap to completing the patient safety cycle. Neurosurg Focus, 33 (5):E4.
