Surg Endosc (2010) 24:45–50 DOI 10.1007/s00464-009-0522-3
Measuring mental workload during the performance of advanced laparoscopic tasks Bin Zheng Æ Maria A. Cassera Æ Danny V. Martinec Æ Georg O. Spaun Æ Lee L. Swanstro¨m
Received: 19 February 2009 / Accepted: 20 April 2009 / Published online: 23 May 2009 Ó Springer Science+Business Media, LLC 2009
Abstract Introduction Mental workload is a finite resource and is increased while learning new tasks and performing complex tasks. Measurement of a surgeon’s mental workload may therefore be an indication of expertise. We hypothesized that surgeons who were expert at laparoscopic suturing would have more spare mental resources to perform a secondary task, compared with surgeons who had just started to learn suturing. Methods Standardized suturing tasks were performed on a bench-top model. Twelve junior residents (novices) and nine fellows and attending surgeons (experts) were instructed to perform as many sutures as possible in 6 min. An adjacent monitor was placed 15° off axis to the first and randomly displayed 30 true visual signals among 90 false ones. Participants were required to identify the true signals while continuing to suture. Laparoscopic sutures were evaluated using the Fundamentals of Laparoscopic Surgery (FLS) scoring system. The secondary (visual detection) task was evaluated by calculating the rate of missed true signals or detection of false signals. Results Experts completed significantly more secure sutures (6 ± 2) than novices (3 ± 1; p = 0.001). The suture performance score was 50 ± 20 for experts,
B. Zheng (&) Centre of Excellence for Surgical Education and Innovation, University of British Columbia, 3602-910 W.10th Ave, Vancouver, BC V5Z 4E3, Canada e-mail:
[email protected];
[email protected] M. A. Cassera D. V. Martinec G. O. Spaun L. L. Swanstro¨m Minimally Invasive Surgery Program and Legacy Institute for Surgical Education and Innovation (LISEI), Legacy Health System, Portland, OR, USA
significantly higher than for novices (29 ± 10; p = 0.005). The rate for detecting visual signals was higher for experts (98%) compared with for novices (93%; p = 0.041). Conclusion Practice develops automaticity, which reduces the mental workload and allows surgeons to have sufficient spare mental resources to attend to a secondary task. Visual detection provides a simple and reliable way to assess mental workload and situation awareness abilities of surgeons during skills training, and may be an indirect measure of expertise. Keywords Mental workload Surgical competence Assessment Visual detection Secondary task Surgical education
For most surgeons, performing a laparoscopic procedure is more physically and mentally demanding than performing an open procedure [1–5]. A benchmark skill set for laparoscopic surgery is making a suture and tying a secure knot at a designated surgical location [6]. To achieve this surgical goal, a surgeon needs to coordinate his/her eyes, hands, and a long-shaft needle driver in a skillful manner as well as mentally transfer a two-dimensional (2D) environment to a three-dimensional (3D) one [5]. Learning such skills has been shown to be stressful for novices [7–9]. Measurement of the amount of mental work to perform complex tasks can be challenging. In an earlier study on laparoscopic mental workload, surgeons were required to perform a knot-tying task in a laparoscopic training box. Electrodes were placed on the forehead and ear of the surgeon to measure electro-ocular activity, and on the right palm to measure skin conductance. These two physiological responses, controlled by the sympathetic nervous system of a human being, are used to interpret the level of
123
46
mental stress of an operator [10, 11]. Using this system, surgeons who were tying knots with laparoscopic instruments were reported to have higher mental workloads than when performing the similar task with open instruments [10]. Other studies have used a subjective survey form to assess mental workload in laparoscopic surgeons, such as the National Aeronautics and Space Administration (NASA) Task Load Index and the Dundee Stress State Inventory [12–15]. These subjective measures require an operator to self-evaluate the mental stress level associated with a task. However, the answers to survey questions are affected by respondent’s working memory and ability to judge the task difficulty [13]. In addition to the above direct measures of mental workload, there is a psychological approach developed on the basis of the human information processing (HIP) model in cognitive science [16]. The HIP model argues that the mental system has limited capacity. When performing a tough task that requires more mental resources, an operator will be left with a lower amount of spare mental resources to perform a secondary task. As mental resources are stretched thin, performance starts to deteriorate. Performance degradation is initially directed to the task designated as secondary. Taking this into consideration, the performance of the secondary task helps us to estimate the workload required by a primary task. This study used a secondary task to assess the mental workload of a surgeon during a laparoscopic suture task. In the process of selecting the type of secondary task, we took the multiple resource theory (MRT) into consideration [17, 18]. According to the MRT model, a human operator has different sensorimotor modalities that can process multiple types of information simultaneously; for example, a skillful driver can listen to music while manipulating his/ her car appropriately through a busy city centre without violating traffic lights. Each sensorimotor modality has distinctive properties and can process information up to a certain quantity. Excess workload placed on one sensorimotor modality can cause problems and result in slower task performance or movement errors. According to this notion, the effect of a secondary task will be more meaningful when it is loaded onto the same modality as the primary task. Following this theory, the secondary task that we selected for this study was a visual detection task. It is designed based on the fact that laparoscopy is image-guided surgery and the performance of surgical tasks is mainly guided through visuomotor pathway. In a review paper published in 1995, Dr. Cuschieri addressed that the quality of visual presentation is the most important component for the safe performance of a laparoscopic procedure [5]. Studies have found that the location of display, quality of image, depth cue, and alignment of images all have an
123
Surg Endosc (2010) 24:45–50
effect on surgeon performance [19–21]. These results support our statement that the performance of laparoscopic surgery is primarily dependent on the visual feedback. In a laparoscopic procedure, tactile and kinesthetic feedback also plays a role for guiding laparoscopic performance [22]. However, surgeons also believe that the role of tactile feedback in laparoscopic procedures is reduced when compared with open surgery [22]. Based on the above facts, we designed our study to assess the surgeon’s performance of a secondary task while using the visual channel. Explicitly in this study, we asked the surgeon to detect true visual signals from a list of false signals presented to the peripheral visual field while the surgeon was performing a laparoscopic suturing task. Both the suture and the visual detection task performances would be compared between experienced and novice laparoscopic surgeons. The spare mental resources of surgeons would be assessed by checking their performances on the secondary task. We hypothesized that the experienced surgeons who were confident in making sutures would have superior mental resources to detect distractive visual signals, compared with surgeons who were still building laparoscopic suturing skills.
Methods Participants Participants were recruited from within the Legacy Health System, a nonprofit healthcare provider located in Portland, OR, and affiliated with Oregon Health and Science University. Twelve junior surgical residents (PGY1 and PGY2) were recruited as the novice group. Nine surgeons formed the expert group, including senior residents (PGY 4, PGY5), laparoscopic fellows, and attending laparoscopic surgeons. Apparatus The laparoscopic suture task was performed in a commercially available laparoscopic training box (TRLCD 03 Laparoscopic Trainer, 3-D Technical Services, Franklin, Ohio). The trainer measures 17.500 long, 12.500 wide, and 8.500 high, including a 1000 liquid-crystal display (LCD) color monitor and a camera which is mounted inside the training box. A piece of synthetic soft tissue (Soft Tissue Pad, 3-D Technical Services, Franklin, OH) was placed on the bottom of the trainer. A pair of Jarit needle drivers (Jarit Carb-Bite 600-250, Jarit Surgical Instruments, Inc., Tuttlingen, Germany) were used for making a suture and tying a knot. The needle drivers were placed through two top ports on the trainer box (Fig. 1).
Surg Endosc (2010) 24:45–50
Fig. 1 Experiment setup
A laptop computer (HP Pavilion ZE 2000, HewlettPackard Company, Palo Alto, CA) was placed next to the laparoscopic trainer, 15° off axis to the laparoscopic view. The computer’s 15.200 monitor randomly displayed a total of 30 true visual signals (a rectangular square displayed on the right side of the laptop screen) and 90 false signals (a rounded-corner square or a rectangular square displayed at the left side of the screen) during the 6-min exam. Each visual signal appeared on the screen for 0.5 s. Procedure and measures Prior to taking the test, participants were required to complete a consent form and a pre-test questionnaire to provide demographic data, training level, and laparoscopic experience. Laparoscopic experience was measured by having participants estimate the number of basic and advanced laparoscopic procedures they had performed as surgeon or as assistant. An overall experience score was obtained by summing the number of listed surgical procedures. Following FLS protocols, surgeons were required to make sutures on predetermined spots marked by an experimenter. The laparoscopic camera was adjusted to display the suture site in the center of the monitor. Participants were instructed to perform as many sutures as possible in 6 min. Within this period of time, participants were also instructed to respond to the true visual signal by saying ‘‘check’’ verbally. The missing of a true signal and the incorrect verbal response to a false signal were collected as ‘‘detecting errors.’’ The detecting errors were used
47
to calculate the correct rate for visual detection using the following equation: visual detection rate (%) = [(120 detection errors)/120] 9 100. At the beginning of the test all participants were instructed to concentrate on the suture task and complete as many sutures as possible. The visual detection task was the secondary task, which they were encouraged to perform only when they felt comfortable to do so. The FLS scoring system was used to evaluate suture quality. FLS scoring has undergone strict validation and reliability measurements. For the suture task, task performance was scored by movement speed and accuracy of the movements. The penalty score for the suture tasks was applied when the suturing was conducted inaccurately (measured by deviation from predetermined dot positions) or the knot quality was low (measured by knot tightness and gap between tissue). Performance was reported as a normalized score on a scale of 100 points. Briefly, faster movement speed (shorter movement time), higher movement accuracy (less movement mistakes), and more sutures completed yielded higher performance scores. Statistics Prior to data collection, power analysis was carried out based on the assumed minimal difference of visual detection that might be presented between residents and experience surgeons. When using a similar method to assess the mental workload of surgeons during suturing tasks, Stefanidis et al. recorded a 33-point difference in the visual detection score between trained residents and laparoscopic experts [23]. Stefanidis et al. reported ranges with the mean instead of standard deviations. Judging by the large range around the means, a conservative estimation of 30 points of standard deviation was used to calculate the effect size. When applied to a one-way analysis of variance (ANOVA) model with an alpha of 0.05 (two tailed) and a beta of 0.15 (power of 85), the estimated sample size was eight surgeons per group. Our study, which used 12 residents and 9 laparoscopic experts, should provide a confident level of power to test the hypothesis. To test our hypothesis, a single-factor analysis of variance (one-way ANOVA) was employed to compare the number of sutures made, the suture score, and the visual detection rate between novice and experienced surgeon groups. Pearson’s tests were conducted between the surgical experiences and suture performance as well as the surgical experiences and visual detection performance to investigate which of these factors correlated better with surgical training. p \ 0.05 was considered significant. Results are reported as mean ± standard deviation unless otherwise stated.
123
48
Surg Endosc (2010) 24:45–50
Results
Discussion
A total of 12 junior residents and 9 experienced laparoscopic surgeons completed the test. The mean age of junior residents was 29 years, significantly younger than the experienced surgeons (38 years, p = 0.032). Four residents had not been involved with any laparoscopic procedures; others had 1–3 years of experience (mean 2 years). Experienced surgeons had over 5 years of experience with performing laparoscopic procedures (p = 0.006). The experienced surgeons performed a larger number of laparoscopic procedures and achieved a significantly higher experience score (65/100) compared with the junior residents (26/100; p \ 0.001). Table 1 summarizes the demographic differences between the two surgeon groups. Experienced surgeons completed more sutures and knot tying (five sutures) on average than the junior residents (three sutures) within 6 min (p = 0.001). Sutures completed by the experienced surgeon had higher quality (50/ 100) than those performed by junior residents (29/100; p = 0.006). Concurrently, experienced surgeons performed better in the secondary task; they achieved a rate of 98% in detecting visual signal correctly, compared with a 93% detection rate in junior residents (p = 0.041). Table 2 summarizes the differences in task performances between the two surgeon groups. Figure 2 presents a scatter plot between surgical experience scores and results of primary and secondary task performance. Pearson correlation coefficients (r) was moderated between surgical experience score and visual detection rate (r = 0.37, p = 0.095). This correlation was weaker than the correlations between experience score and the number of suture completed (r = 0.81; p \ 0.001) as well as the suture score (r = 0.76, p \ 0.001).
The evidence presented in this paper supports our original hypothesis. Experienced surgeons who have developed skills for performing complex surgical tasks are able to allocate more spare mental resources to perform secondary tasks. This phenomenon can be explained by the automaticity theory in the field of motor learning [24]. After substantial practice, the coordination of a movement will become mentally embedded. When automaticity occurs, movements are performed consistently and efficiently without requiring much mental resource [24]. Mental resources are thus released for new and unfamiliar tasks. This explains why experienced surgeons are able to notice abnormal events in the operating room (OR) , respond to events faster, and initiate preemptive maneuvers better than novice surgeons. The spare mental resource provides the foundation for situation awareness and perhaps would lead to better outcomes in the complex environment of a surgery. There was a linear relationship between surgeons’ laparoscopic experiences and their secondary task performance (Fig. 2). With this linear relationship, we can use the performance of secondary tasks to estimate surgeons’ skill level of the primary task. Some authors believe that the secondary task approach is reliable and sometimes even more sensitive than other conventional measures used to measure a surgeon’s skill level [23]; for example, Stefanidis et al. showed that experienced surgeons achieved higher score in a visual discriminating task than novice surgeons before they exhibited any significant difference in suture tasks [23]. We did, however, find that experienced surgeons performed significantly better in both primary and
Table 1 General description of the two surgeon groups Junior residents
Experienced surgeons
p value
Number of participants
12
9
–
Age (years)
28.8 ± 4.2
38.2 ± 7.4
0.032
Year of lap training
1.7 ± 1.1
5.4 ± 3.5
0.006
Experience score
25.7 ± 4.9
65.2 ± 23.7
0.000
Table 2 Task performance on the primary and secondary tasks Junior residents Number of sutures completed
Experienced surgeons
p value
3±1
5±3
0.001
Suture score
29 ± 10
50 ± 20
0.006
Visual detection rate (%)
93 ± 7
98 ± 2
0.041
123
Fig. 2 Scatter plots. Performance of the suture task (calculated by number of suture and suture score) and visual detection task (evaluated by detection rate) correlated positively with the surgical experience score. Correlations between suture performance and surgical experience were stronger than the correlation between visual detection performance and surgical experience
Surg Endosc (2010) 24:45–50
secondary tasks than the novices. In addition, the correlation between a surgeon’s suture performance and their surgical experience score was stronger than the correlation between the visual detection performance and surgical experience (Fig. 2). We believe that the performances of both the primary and secondary tasks are affected by the mental resources of an operator. It is logical to assess skills status by checking subjects’ performances in both the primary and secondary tasks. The current study has some limitations. First, the design of the visual signal was not subjected to a validity check. Future studies are needed to verify that the visual signal being displayed is suitable for surgeons in the laparoscopic context, in terms of its size, color, location, and frequency. Second, we did not incorporate a second measure of mental workload in this study, such as the NASA Task Load Index (TLX) form at the end of trials. Future studies will include a subjective measure in addition to the objective measure obtained by the secondary task. A positive correlation between these two measures will support the validity of using secondary tasks for measuring mental workload of surgeons. Third, the tasks included in the study are highly simulated, and therefore may not represent clinical situations where surgical tasks are more complicated and distractions are presented in multiple ways. A good solution to this problem will be to repeat our tasks in the OR while real laparoscopic procedures are carried out. Fourth, at the beginning of each trial, participants went through a short orientation directing them to concentrate on the primary suturing task instead of the visual detection task. This short orientation may not have been sufficient to guide participants through the entire test course. Some participants slowed down their suturing in order to watch visual signals. The tendency to shift attention to the visual detection task might introduce bias into the results; therefore, one should be cautious in interpreting the result from the secondary task performance.
Conclusion Results support our hypothesis that surgeons who have mastered complex surgical skills are able to allocate more mental resources to perform secondary tasks. In this situation, the performance on the secondary task can then reflect the amount of mental workload used in performing the primary task and be used to infer the operator’s laparoscopic skill level. This may point the way to a quick and easy way to measure competence and expertise in laparoscopic surgery. Acknowledgement This project was supported by NOSCARÒ Research Grants, a joint initiative supported by the American Society
49 for Gastrointestinal Endoscopy (ASGE) and the Society of American Gastrointestinal and Endoscopic Surgeons (SAGES).
References 1. Berguer R, Chen J, Smith WD (2003) A comparison of the physical effort required for laparoscopic and open surgical techniques. Arch Surg 138:967–970 2. Berguer R, Forkey DL, Smith WD (2001) The effect of laparoscopic instrument working angle on surgeons’ upper extremity workload. Surg Endosc 15:1027–1029 3. Berguer R, Gerber S, Kilpatrick G, Remler M, Beckley D (1999) A comparison of forearm and thumb muscle electromyographic responses to the use of laparoscopic instruments with either a finger grasp or a palm grasp. Ergonomics 42:1634–1645 4. Keehner MM, Tendick F, Meng MV et al (2004) Spatial ability, experience, and skill in laparoscopic surgery. Am J Surg 188:71–75 5. Cuschieri A (1995) Visual displays and visual perception in minimal access surgery. Semin Laparosc Surg 13:209–214 6. Aggarwal R, Hance J, Undre S et al (2006) Training junior operative residents in laparoscopic suturing skills is feasible and efficacious. Surgery 139:729–734 7. Dubrowski A, Park J, Moulton CA, Larmer J, MacRae H (2007) A comparison of single- and multiple-stage approaches to teaching laparoscopic suturing. Am J Surg 193:269–273 8. Korndorffer JR Jr, Dunne JB, Sierra R, Stefanidis D, Touchard CL, Scott DJ (2005) Simulator training for laparoscopic suturing using performance goals translates to the operating room. J Am Coll Surg 201:23–29 9. Moorthy K, Munz Y, Dosis A, Bello F, Chang A, Darzi A (2004) Bimodal assessment of laparoscopic suturing skills: construct and concurrent validity. Surg Endosc 18:1608–1612 10. Berguer R, Smith WD, Chung YH (2001) Performing laparoscopic surgery is significantly more stressful for the surgeon than open surgery. Surg Endosc 15:1204–1207 11. Weinger MB, Reddy SB, Slagle JM (2004) Multiple measures of anesthesia workload during teaching and nonteaching cases. Anesth Analg 98:1419–1425 (table of contents) 12. Klein MI, Warm JS, Riley MA, Matthews G, Gaitonde K, Donovan JF (2008) Perceptual distortions produce multidimensional stress profiles in novice users of an endoscopic surgery simulator. Hum Factors 50:291–300 13. Carswell CM, Clarke D, Seales WB (2005) Assessing mental workload during laparoscopic surgery. Surg Innov 12:80–90 14. Cao CG (2007) Guiding navigation in colonoscopy. Surg Endosc 21:480–484 15. O’Connor A, Schwaitzberg SD, Cao CG (2007) How much feedback is necessary for learning to suture? Surg Endosc 22:1614–1619 16. Norman DA, Bobrow DG (1975) On data-limited and resourcelimited processes. Cogn Psychol 7:44–64 17. Wickens CD (1984) Processing resources in attention. In: Parasuraman R, Davies DR (eds) Varieties of attention. Academic Press, New York, pp 63–102 18. Wickens CD (2002) Multiple resources and performance prediction. Theor Issues Ergon Sci 3:159–177 19. Emam TA, Hanna G, Cuschieri A (2002) Ergonomic principles of task alignment, visual display, and direction of execution of laparoscopic bowel suturing. Surg Endosc 16:267–271 20. Hanna GB, Cuschieri A (1999) Influence of the optical axis-totarget view angle on endoscopic task performance. Surg Endosc 13:371–375
123
50 21. Hanna GB, Cuschieri A (2000) Influence of two-dimensional and three-dimensional imaging on endoscopic bowel suturing. World J Surg 24:444–448 discussion 448-9 22. Bholat OS, Haluck RS, Murray WB, Gorman PJ, Krummel TM (1999) Tactile feedback is present during minimally invasive surgery. J Am Coll Surg 189:349–355
123
Surg Endosc (2010) 24:45–50 23. Stefanidis D, Scerbo MW, Korndorffer JR Jr, Scott DJ (2007) Redefining simulator proficiency using automaticity theory. Am J Surg 193:502–506 24. Marteniuk RG (1976) Information processing in motor skills. Holt Rinehart and Winston, New York
