ORIGINAL REPORTS
Construct Validity, Assessment of the Learning Curve, and Experience of Using a Low-Cost Arthroscopic Surgical Simulator Henry B. Colaco, Colaço, MSc, FRCS (Tr&Orth) MFSTEd, Katie Hughes, BSc (Hons), Eyiyemi Pearse, MA, FRCS (Orth), Magnus Arnander, MSc, FRCS (Orth), and Duncan Tennent, FRCS (Orth), PgCert (MedEd), MAcadMEd, FFSTEd St George’s University of London, London, UK OBJECTIVE: We have developed a low-cost, portable shoulder simulator designed to train basic arthroscopic skills. This study aimed to establish the construct validity of the simulator by determining which parameters discriminated between experience levels and to assess the experience of using the simulator.
hands 0.7 times on average during the task compared with 2.8 and 3.4 times for trainees and students, respectively. More than 95% of participants found the exercise interesting and agreed or strongly agreed that the simulator was easy to use, easily portable, and well designed and constructed.
DESIGN: Participants were given an introductory presen-
DISCUSSION: This study has established construct validity
tation and an untimed practice run of a 6-step triangulation task using hooks and rubber bands. A total of 6 consecutive attempts at the task were timed, and the number of times the participant looked at their hands during the task was recorded. Participants then completed a questionnaire on their experience of using the simulator.
of the simulator by demonstrating the ability to distinguish between surgical experience levels. The learning curve shows improvement in individuals with or without arthroscopic or surgical experience. Simulation is becoming increasingly important in the training of medical students and surgical trainees; this study has established that low-cost portable arthroscopic C box trainers may play a significant role. ( J Surg Ed ]:]]]-]]]. J 2016 Association of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.)
SETTING: St George’s Hospital, London and the South
West London Elective Orthopaedic Centre, Surrey. PARTICIPANTS: Medical students, trainee doctors and
surgeons, and consultant surgeons were approached to use the simulator. Participation was voluntary and nonincentivized. In total, 7 orthopedic consultants, 12 trainee doctors (ranging from foundation year 1 to clinical fellow post-Certificate of Completion of Training), and 9 medical students were recruited. RESULTS: The average time for medical students to complete
the task was 161 seconds, compared to 118 seconds for trainees, and 84 seconds for consultants. The average fastest time for medical students was 105 seconds, 73 seconds for trainees, and 52 seconds for consultants. Students were significantly slower than trainees (p ¼ 0.026) and consultants (p ¼ 0.001). However, times did not differ significantly between trainees and consultants. Consultants looked at their
Correspondence: Inquiries to Katie Hughes, BSc, St George’s University of London, Cranmer Terrace, London, UK; E-mail:
[email protected], m0900432@ sgul.ac.uk
KEY WORDS: simulation, arthroscopy, surgery, training COMPETENCY: Practice-Based Learning and Improvement
INTRODUCTION Arthroscopy has become an integral part of the surgical workload within trauma and orthopedics. However, it is a challenging skill to learn which requires manual dexterity, spatial awareness, and accurate manipulation of arthroscopic instruments. Restrictions in working hours in Europe and the United States limit the time trainees are able to spend observing and undertaking arthroscopic procedures in the operating theater.1-3 Subsequently, development of these skills may come at a cost of increased operating times, higher complication rates, and inferior surgical outcomes for patients.4,5 Given this, there is clearly a need for trainees to develop their arthroscopic skills outside the theater environment. Simulation in arthroscopic surgical training has been shown to shorten the learning curve, reduce errors in live surgery, and improve patient outcomes.6 Simulators also
Journal of Surgical Education " & 2016 Association of Program Directors in Surgery. Published by 1931-7204/$30.00 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.jsurg.2016.07.006
1
facilitate the safe practice of visualization, triangulation, and feedback skills for trainees.7 Simulation is becoming established as an integral part of modern surgical training in orthopedics and across other surgical specialties.8-11 There are numerous surgical simulation systems currently available, ranging from simple box trainers to highly advanced virtual reality systems. However, their availability is limited and they can be expensive to purchase and maintain with prices ranging from £350 to £140,000 GBP ($500-$200,000 USD).12 Although basic box trainers remain an effective, low-cost training tool—they often require specific equipment or operating displays that limit their application and availability.12 Given this, there appears to be a gap in the market for an inexpensive, widely available, portable arthroscopic simulator. Aslam et al.13 have successfully constructed a homemade laparoscopic simulator, but a similar arthroscopic-specific trainer is yet to have been described in the literature. We have designed and constructed a novel box trainer designed to learn basic arthroscopic triangulation skills. It can be assembled from widely available inexpensive materials. It is light, easily portable, and can be taken apart and reassembled in a matter of seconds. Before a surgical simulator can be used to train surgeons or to assess skills, it must be assessed for construct validity. Construct validity is the degree to which a training tool is able to distinguish between experience levels of the user. More experienced individuals would be expected to score higher than novices—reflecting their true surgical ability when performing an actual procedure.14-17 We hypothesized that the triangulation task would be able to distinguish between experience levels. We also aimed to assess the learning curve of the task and the experience of using the simulator.
MATERIALS AND METHODS Apparatus The simulator was constructed from a 0.7-L translucent blue polypropylene box (155 mm long # 100 mm wide x 80 mm high), with a removable lid (Really Useful Products Ltd., W. Yorks, UK). A total of 7 access portals were drilled using a 20-mm spade-tipped drill bit, and sealed with 23.7mm flexible silicone sealing grommets (Maplin Electronics, S. Yorks, UK). The portals were located to replicate posterior (P), lateral (L), superolateral (SL), high anterior (HA), low anterior (LA), and Neviaser (N) portals commonly used in shoulder arthroscopy. By rotating the lid, the box can be used to represent both right and left shoulders; and adhesive-backed hook and loop tape strips in the base allow for different inserts to be secured. The box itself is secured to a rubber-backed medium-density fiberboard base with adhesive hook and loop tape, which allows fixation to a tabletop using an adjustable clamp. The inside of the box is 2
TABLE 1. Simulator Component Cost. Currency Conversions Made in May 2016 2-Pack 0.7-L really useful box USB-powered 01 “pencil” scope 10-Pack 23.7-mm cable grommets Command kitchen utensil hooks MDF board Clamp Colored elastic bands Total
£2.99/$4.30 £34.69/$50.50 £3.59/$5.00 £2.50/$3.60 £1.95/$2.80 £1.78/$2.50 £1.62/$2.30 £49.12/$70.88
MDF, medium-density fiberboard.
viewed using a handheld Supereyes 2.0 megapixel Y002, 7mm diameter 01 USB-powered “pencil” scope with 4 LED light source (Shenzhen D&F Co., Shenzhen, China) and free trial access “miXscope” digital microscope software (EdH Software LLC, Boise, ID) on an Apple MacBook Pro (Apple, Cupertino, CA). A donated DePuy Mitek suture manipulator (DePuy Synthes, West Chester, PA), 3-hinged Command kitchen utensil hooks (3M, St Paul, MN), and colored elastic bands were used to complete the triangulation task. The total cost of assembly was less than £50 ($72 USD) (Table 1). Photographs of the exterior, interior, and fully assembled box simulator are shown in Figures 1 to 4.
Task and Assessment Participants were first given an introductory presentation on the box simulator and given a “trial run” to familarize themselves with the box and the scope field of view. They had the opportunity to ask any questions about the box or the task. They were then guided through a 6-step abstract triangulation task using the suture manipulator, elastic bands, and hinged utensil hooks. Participants were allowed an untimed practice attempt before being asked to complete 6 consecutive timed attempts. An observer timed the task using a stopwatch from the moment the suture manipulator was first introduced to the box to the moment the last hook was flipped into its final position. The task involved first flipping the 3 utensil hooks toward the center of the box using the suture manipulator via the SL portal, introducing a rubber band to the box via the LA portal, using the L portal to hook the rubber band around the 3 hooks, unhook and remove the band using the LA portal, and finally return the hooks to their starting position again via the SL portal. As the box simulator is semiopaque, it is possible to view the inside of the box and watch the training task under direct vision. Participants were instructed to not look at their hands or the box during the task apart from when introducing the suture manipulator through a portal, but were not informed that this was recorded as a secondary outcome measure. The total time of each task attempt and Journal of Surgical Education " Volume ]/Number ] " ] 2016
FIGURE 1. The exterior of the box simulator.
the number of times they looked at their hands during the task was then recorded. Participants were then asked to complete a posttask questionnaire about how interesting they found the task, the ease of use of the box, its portability, design, and construction (Appendix I). They were also asked to give feedback on if basic arthroscopic triangulation skills would be a worthwhile addition to basic surgical skills course, if they would use an arthroscopic training kit if given one, and if practice with the box simulator could improve early attempts at arthroscopic surgery. Participants were also able
FIGURE 3. The interior of the box simulator with the hooks in their starting position flipped away from the center of the box.
to give qualitative free text feedback on their experience of using the simulator. Participants Surgical consultants, trainee doctors, trainee surgeons, and medical students working or on clinical placements at St George’s Hospital and the South West London Elective Orthopaedic Centre were recruited to complete the task over a 2-week period. The task was commonly completed during natural breaks between operating cases and teaching activities. Participation was voluntary and nonincentivized. The participants provided information on their training level, gender, age, total number of arthroscopies performed, number of shoulder arthroscopies performed, and hours of video games played per week (Table 2). Statistical Analysis Comparison between the 3 groups was performed using the software package SPSS, version 22.0 (IBM, Armonk, NY). Observed differences were considered significant if p o 0.05.
FIGURE 2. The box simulator fully assembled with triangulation task in progress. Journal of Surgical Education " Volume ]/Number ] " ] 2016
FIGURE 4. The interior of the box simulator during the triangulation task. The utensil hooks have been flipped toward the center of the box and a rubber band hooked around them. 3
TABLE 2. Demographic Details of Participants
Training Level Consultants (n ¼ 7)
Gender Male Male Male Male Male Male Male
Trainees (n ¼ 12) Junior trainees (n ¼ 7) FY1 Male GP SHO Male Surgical CT1 Female Surgical CT1 Male Surgical CT1 Male Surgical CT1 Female Surgical CT2 Male Senior trainees (n ¼ 5) Orthopedic ST3 Female Orthopedic ST3 Male Orthopedic ST4 Male Orthopedic ST5 Male Clinical fellow Male Medical students (n ¼ 9) Third year Male Third year Male Fourth year Male Fourth year Female Fourth year Male Fourth year Female Fifth year Male Sixth year Female Sixth year Male
Total Number of Arthroscopies Performed
Number of Shoulder Arthroscopies Performed
100 100þ 500þ 500þ 1000þ 300 7000þ
10 0 0 30 80 200 7000þ
2 0 0 0 1 0 0
45 46 39 52 40 45 49
0 0 0 0 0 0 0
0 0 0 0 0 0 0
2 0 0 0 0 0 0
23 33 26 26 26 27 33
20 0 36 — 200
0 0 2 — 10
0 0 1 — 5
31 29 32 — 37
— — — — — — — — —
— — — — — — — — —
4 0 0 3 14 0 4 0 1
21 22 22 24 22 21 23 24 24
Video Games/ Week (h)
Age (y)
FY1, foundation year 1; GP SHO, general practitioner senior house officer.
Data were analyzed using a repeated measures analysis of variance and an one-way analysis of variance.
RESULTS In total, 28 individuals were recruited. They included 7 orthopedic consultants, 12 trainee doctors, and 9 medical students. Of the trainees, 7 were considered “junior” and 5 were considered “senior.” The junior trainees were either enrolled on a nonsurgical training program or in their first 2 years of core surgical training, whereas the senior trainees were enrolled on an orthopedic-specific training program (Table 2). The 9 medical students ranged from third to sixth year of study. The consultants’ arthroscopic experience ranged from approximately 100 cases to more than 7000 cases, most trainees had comparatively little arthroscopic experience and all of the students were arthroscopy naive. None of the participants had any prior experience with the box trainer or with other similar simple arthroscopic 4
simulator. Among them, 85% were male (n ¼ 24) and 15% were female (n ¼4) with a mean age of 31.2 (!9.6) years. The percentage of participants who completed the postparticipation questionnaire was 96% (n ¼ 27). Time Taken to Complete the Task Construct validity was evaluated by comparing the time taken to complete the task across the 3 groups. A general trend was observed that every subgroup improved their average time taken to complete the task from their first attempt to their last attempt. It was also clear that the higher the training level of a participant, the faster they were at completing the task. As training level increased, the average number of times looked at hands also decreased. The average attempt time, average fastest attempt time, the average first and last attempt times, and the number of times each subgroup looked at their hands on average during the task is shown in Table 3. Journal of Surgical Education " Volume ]/Number ] " ] 2016
TABLE 3. Results of Triangulation Task Using Box Simulator
Training Level Consultants (n ¼ 7) All trainees (n ¼ 12) Senior trainees (n ¼ 5) Junior trainees (n ¼ 7) Medical students (n ¼ 9)
Average Attempt Time (s) 84 118 110 126 161
(SD (SD (SD (SD (SD
! ! ! ! !
13) 20) 20) 20) 61)
Average Fastest Attempt Time (s)
Average First Attempt Time (s)
Average Last Attempt Time (s)
Difference in Average First and Last Attempts (s)
Average Number of Times Looked at Hands
52 73 70 76 105
103 161 129 187 234
58 99 132 80 129
45 62 %3 107 105
0.7 2.8 1.4 3.8 3.4
The more experienced participants were faster at completing the task. The medical student average time of 161 seconds (standard deviation [SD] ! 61) was 27% slower than the average trainee time of 118 seconds (SD ! 20) and 48% slower than consultant time of 84 seconds (SD ! 13). The junior trainees average time of 126 seconds (SD ! 25) was 13% slower than senior trainee time of 110 seconds (SD ! 6). Trainees were 29% slower than consultants on average, with junior trainees being 33% slower and senior trainees 24% slower than consultants. Comparing all attempts across the 3 groups, medical students were significantly slower than both trainees (p ¼ 0.026) and consultants (p ¼ 0.001). However, times did not differ significantly between consultants and trainees as a whole (p ¼ 0.067). When separated by their experience level, neither the senior (p ¼ 0.317) nor the junior trainees (p ¼ 0.056) were significantly slower than consultants, although the junior trainees were close to achieving statistical significance. Senior trainees were significantly faster than medical students (p ¼ 0.036) but the junior trainees were not (p ¼ 0.086). The fastest attempt time also was significantly different across the 3 groups (p ¼ 0.003). As shown in Figure 5, there was an improvement in average fastest attempt time as training level increased. The medical student average fastest attempt took 105 seconds, 30% slower than the trainee fastest attempt time of 73 seconds and 50% slower than the consultant time of 52 seconds for consultants. The trainees’ fastest
Across all the 3 subgroups, a significant difference in time between the first attempt and the last attempt was observed (p o 0.001). A similar reduction in task time was observed across all groups. As shown in Figure 6, from first attempt to last, the average time consultants took to complete the task decreased by 44% from 103 seconds to 58 seconds. Trainees decreased their time by 39% from 161 seconds to 99 seconds overall, with junior trainees decreasing by 57% from 187 seconds to 80 seconds. Medical students decreased by 45% from 234 seconds to 129 seconds. The notable exception to this trend was the senior trainee group, whose average time actually increased by 3 seconds from first attempt to last. There was no significant interaction between task completion time and training level—showing that regardless of experience, all individuals significantly improved their times with successive attempts. Figures 7 to 9 show individual attempt times across the 3 subgroups. This gives a graphical representation of the learning curve for each participant. Although students and trainees generally improved their times with each attempt, there was greater variation in their times compared with consultants. They also rarely achieved the same consistent
FIGURE 5. Average fastest attempt time and average time related to training level (with standard error bars).
FIGURE 6. Improvement in average attempt time from first to last attempt across training levels (with standard error bars).
Journal of Surgical Education " Volume ]/Number ] " ] 2016
5
attempt time was 29% slower than the consultants’ fastest attempt time. Improvement in Successive Attempt Times
FIGURE 7. Consultant's individual attempt times.
fast times consultants achieved from their very first attempts (Fig. 10). Looking at Hands Although participants were instructed to not look at their hands or the box during the task, apart from when switching the manipulator between portals, many could not resist. Consultants looked at their hands at an average of 0.7 times across their 6 attempts, compared to 2.8 times for trainees (3.8 times for juniors and 1.4 times for seniors) and 3.4 times for medical students. Although all individuals looked at their hands less with each successive attempt (p o 0.001), there was no significant interaction between looking at hands and training level (p ¼ 0.919). Owing to the relatively low level of video game experience, it was not possible to observe a relationship between hours played a week and attempt times. Feedback on the Box Simulator Feedback from the postparticipation questionnaire was generally positive, with 78% agreeing or strongly agreeing that they would use an arthroscopic training kit if given one. The percentage of participants who strongly agreed or agreed that basic arthroscopic triangulation skills would be a worthwhile addition to basic surgical skills course was
FIGURE 8. Trainee's individual attempt times and training levels. 6
FIGURE 9. Medical student's individual attempt times.
92% and the percentage of participants who strongly agreed that practice with the box simulator could improve early attempts at arthroscopic surgery was 76%. More than 95% of participants found the exercise interesting and agreed or strongly agreed that the simulator was easy to use, easily portable, and well designed and constructed. It was also empirically observed during data collection that participants found the task engaging, fun, and competitive, despite a lack of a material incentive. Participants often expressed a desire to improve on their time with each successive attempt and to be faster than colleagues of a similar training grade.
DISCUSSION The aim of this study was to evaluate the construct validity, learning curve, and experience of using a low-cost arthroscopic simulator. Previous studies have established construct validity for arthroscopic simulators by showing that experienced surgeons performed better than novices in their respective simulation tasks.13,16 The box simulator task was capable of discriminating between different levels of expertise. As would be expected, the consultant group performed the task faster, more consistently and with fewer erroneous glances at their hands than the trainees and the medical students, although this did not reach significance. Trainees were in turn significantly faster than the medical students. Although the difference in the completion time between consultants and medical students was statistically significant
FIGURE 10. Summary of average attempt times. Journal of Surgical Education " Volume ]/Number ] " ] 2016
(p ¼ 0.001), the observed difference between consultants and trainees did not reach significance (p ¼ 0.067). Even when the trainee group was separated by their experience level, neither senior (p ¼ 0.317) nor junior trainees (p ¼ 0.056) were significantly slower than consultants. The lack of significant difference between trainees and consultants may be because of the relatively small sample size, the heterogeneous nature of the “trainee” group, and the simple nature of the abstract task. The trainee subgroup analysis was limited by the fact that the “senior” and “junior” groups only contained 7 and 5 individuals, respectively. Unfortunately, this sample size did not afford sufficient statistical power to allow for comparison. Previous studies have also observed poor performances in intermediate “trainee” groups in construct validity studies of arthroscopic simulators.14,18 Srivastava et al.14 speculate that “experts knew what to expect and novices came with an open mind and became engaged as learners; the intermediate group, in contrast, proceeded in a self-exploratory manner with less attention to performance goals.” All groups showed an improvement in performance from first to last attempt and all individuals improved regardless of their training level. Consultants decreased their average time by 44%, trainees by 39%, and medical students by 45%. We acknowledge that the simulation task does not reproduce a complex compound task representative of arthroscopic surgery, perhaps reflected in the fact that the box trainer task could not differentiate between trainees and consultants. However by creating a novel task, we were able to validate the box trainer as a concept, rather than conduct an assessment of individuals’ surgical performance. If the simulator required the same level of skill as used in an actual surgical procedure, the more experienced consultant group would be expected to show the most rapid improvement in performance.14 In addition to completing the task in less time, more experienced individuals also looked at their hands fewer times during the task. Consultants looked at their hands at an average of 0.7 times across their 6 attempts, compared to 2.8 times for trainees (3.8 times for juniors and 1.4 times for seniors) and 3.4 times for medical students. It is important to acknowledge that looking at hands would possibly confer an advantage in completing the task—meaning that the junior trainees and students who looked at their hands more often may have benefited by completing the task faster. This may further explain why the results between trainees and consultants did not reach statistical significance. Feedback from the postparticipation questionnaire was positive, with 78% strongly agreeing or agreeing they would use an arthroscopic training kit if given one. The 22% who were neutral or disagreed were senior trainees and consultants who did not feel the need to practice basic arthroscopic skills, or medical students or trainees not interested in pursuing a surgical career. More than 95% of participants found the exercise interesting and agreed or
strongly agreed that the simulator was easy to use, easily portable, and well designed and constructed. We acknowledge that the use of a 01 scope was a limitation of this study. This was used because the creation of a more realistic arthroscopic environment would have required a USB-powered camera, mount, 301 arthroscope and battery-powered light source. This would have increased the cost of the simulator by more than £500. Although the users did not gain experience of simultaneously manipulating camera and arthroscope, the experience was felt to be valuable. The high cost of arthroscopic instruments may be a barrier to distributing the simulator on a wider scale, but industry donation schemes, such as the one used for this study, are available and cost-neutral. We did not assess economy of movement or accuracy during the task. These are among a number of additional parameters that can be measured using more expensive systems.11 Feedback revealed that some participants would prefer an opaque box, which could be used with a more powerful LED light source. The translucent box provided an additional assessment domain by allowing the number of times participants glanced at their hands during the task to be counted. As reflected in the results, this provided an additional domain to be to establish construct validly as it inversely correlated with number of attempts (p o 0.001) and training level, without reaching statistical significance.
Journal of Surgical Education " Volume ]/Number ] " ] 2016
7
CONCLUSIONS This box simulator is a valid, low-cost, accessible, portable training tool for trainee surgeons to learn basic arthroscopic skills. The abstract triangulation task used in this study demonstrated construct validity, by discriminating between surgical experience levels and has been met with positive feedback. The level of complexity and face validity could be increased by development of a replica glenohumeral joint inside the box. The use of simulation in modern surgical training is increasing and becoming integrated in curricula. Simulation allows trainees to improve their skills in a low-stakes but highly focused environment. However, the high cost and poor availability of simulators must be addressed to optimize training opportunities. Despite the low-tech nature of the assessment domains, this box trainer task was able to distinguish between different surgical experience levels, and both arthroscopic-naive and experienced individuals were shown to improve their performance. The positive feedback is encouraging, and further supports the use of simple arthroscopic box simulators in the future of training for medical students and trainee surgeons.
ACKNOWLEDGMENTS Duncan Tennent received royalties from Arthrex. St George’s Hospital Shoulder Unit receives funding for fellowship from Arthrex.
REFERENCES 1. Canter R. Impact of reduced working time on surgical
training in the United Kingdom and Ireland. Surgeon. 2011;9(suppl 1):S6-S7. 2. Chikwe J, de Souza AC, Pepper JR. No time to train
the surgeons. Br Med J. 2004;328(7437):418-419. 3. Philibert I. New requirements for resident duty hours.
J Am Med Assoc. 2002;288(9):1112. 4. Farnworth LR, Lemay DE, Wooldridge T, Mabrey
10. Preece R. The current role of simulation in urological
training. Cent Eur J Urol. 2015;68(2):207-211. 11. Shaharan S, Neary P. Evaluation of surgical training in
the era of simulation. World J Gastrointest Endosc. 2014;6(9):436-447. 12. Xiao D, Jakimowicz JJ, Albayrak A, Buzink SN,
Botden SM, Goossens RH. Face, content, and construct validity of a novel portable ergonomic simulator for basic laparoscopic skills. J Surg Educ. 2014;71 (1):65-72.
JD, Blaschak MJ, DeCoster TA, et al. A comparison of operative times in arthroscopic ACL reconstruction between orthopaedic faculty and residents: the financial impact of orthopaedic surgical training in the operating room. Iowa Orthop J. 2001;21:31-35.
13. Aslam A, Nason GJ, Giri SK. Homemade laparoscopic
5. Bridges M, Diamond DL. The financial impact of
Heinrichs W, Ladd AL. Initial evaluation of a shoulder arthroscopy simulator: establishing construct validity. J Shoulder Elb Surg. 2004;13(2):196-205.
teaching surgical residents in the operating room. Am J Surg. 1999;177(1):28-32. 6. Akhtar KS, Chen A, Standfield NJ, Gupte CM. The
role of simulation in developing surgical skills. Curr Rev Musculoskelet Med. 2014;7(2):155-160. 7. Howells NR, Brinsden MD, Gill RS, Carr AJ, Rees JL.
Motion analysis: a validated method for showing skill levels in arthroscopy. Arthroscopy. 2008;24(3):335-342. 8. Atesok K, Mabrey JD, Jazrawi LM, Egol KA. Surgical
surgical simulator: a cost-effective solution to the challenge of acquiring laparoscopic skills? Ir J Med Sci. 2015. 14. Srivastava S, Youngblood PL, Rawn C, Hariri S,
15. McDougall EM. Validation of surgical simulators.
J Endourol. 2007;21(3):244-247. 16. Hung AJ, Zehnder P, Patil MB, Cai J, Ng CK, Aron
M, et al. Face, content and construct validity of a novel robotic surgery simulator. J Urol. 2011;186(3): 1019-1024. 17. McCarthy A, Harley P, Smallwood R. Virtual arthro-
simulation in orthopaedic skills training. J Am Acad Orthop Surg. 2012;20(7):410-422.
scopy training: do the virtual skills developed match the real skills required? Stud Health Technol Inform. 1999;62:221-227.
9. Thomas GW, Johns BD, Marsh JL, Anderson DD. A
18. Friedman CP, Dev P, Dafoe B, Murphy G, Felciano
review of the role of simulation in developing and assessing orthopaedic surgical skills. Iowa Orthop J. 2014;34:181-189.
R. Initial validation of a test of spatial knowledge in anatomy. Proc Annu Symp Comput Appl Med Care. 1993:791-795.
SUPPLEMENTARY MATERIAL Supplementary data associated with this article can be found in the online version at http://dx.doi.org/10.1016/j.jsurg. 2016.07.006.
8
Journal of Surgical Education " Volume ]/Number ] " ] 2016
