Review Article
Role of Virtual Reality Simulation in Teaching and Assessing Technical Skills in Endovascular Intervention Kamran Ahmed, MRCS, Aoife N. Keeling, FFR, RCSI, MRCPI, MSc, Morkos Fakhry, MBBCh, MSc, Hutan Ashrafian, BSc, MRCS, Rajesh Aggarwal, MRCS, PhD, Peter A. Naughton, FRCSI, MD, Ara Darzi, FRCS, MD, KBE, Nicholas Cheshire, MD, FRCS, Thanos Athanasiou, PhD, FETCS, and Mohammed Hamady, FRCR
Training in endovascular intervention ultimately aims to produce interventionalists who demonstrate competence in technical skills. Herein, the authors investigate the rationale for simulation-based training by providing an overview of the psychological theories underpinning acquisition of technical skills, training and assessment history, recent advances in simulation technology, and a critical appraisal of their role in training and assessment in endovascular intervention. Simulators have potential for training and assessment and promise solution to many shortcomings of traditional ‘apprenticeship’ training models. Before inclusion into the curriculum, further work is needed regarding fidelity, validity, reliability, and design of simulators to ensure accurate transfer of acquired endovascular skills from simulator to patient. J Vasc Interv Radiol 2010; 21:55– 66 Abbreviation:
VR ⫽ virtual reality
ENDOVASCULAR intervention, in the wake of continuously developing technology and interdisciplinary collaboration, has entered into an era where technical skills training and competence assessment are equally important. Training ultimately aims to produce interventionalists who demonstrate competence in both technical and nontechnical skills (decision making, communication, and teamwork). At present, training is carried out From the Department of Biosurgery & Surgical Technology, Imperial College London, 10th Fl, QEQM Bldg, St. Mary’s Hospital Campus, Praed St., London, W2 1NY, England (K.A., M.F., H.A., R.A., P.A.N., A.D., N.C., T.A.); the Department of Radiology, St Mary’s Hospital, London, England (M.H.); and the Department of Radiology, Northwestern Memorial Hospital, Chicago, Illinois (A.N.K.). Received May 13, 2009; final revision received September 13, 2009; accepted September 16, 2009. Address correspondence to K.A.; E-mail:
[email protected] None of the authors have identified a conflict of interest. © SIR, 2010 DOI: 10.1016/j.jvir.2009.09.019
mainly on patients, sometimes at the expense of patient safety (1), which raises several legal and ethical concerns. In addition, there is an increased cost and intricacy of endovascular procedures, a decrease in simpler diagnostic angiography due to replacement by noninvasive imaging, and residents with capped working hours accompanied by an increasing population of patients with serious and complex problems needing expert input from the beginning (1,2). Moreover, subjective assessment by mentors in the current system, although valuable, remains unreliable and does not fulfill the criteria for the process of summative assessment (3). Therefore the existing master-apprentice model is unlikely to be acceptable. Continuing advances in virtual reality (VR) simulator technology seem to be promising regarding their utility in medical training and assessment. The challenge is, however, to judiciously employ the contemporary advances in technology in the context of a structured and efficient training system with objective competence evaluation
underpinned with evidence-based components and a sound theoretical basis. The implementation of training programs that employ simulators at a certain level must be justified with a corresponding level of evidence. Herein, we present an overview of the history of training and assessment, the psychological theories behind the acquisition of technical skills, and recent advances in simulation technology for training and assessment. In addition, we perform a critical appraisal of the tools in terms of feasibility, validity, and utility of simulation in endovascular intervention.
PSYCHOLOGICAL BASIS OF SKILL ACQUISITION Technical proficiency has been considered to be one of the most important skills of an interventionalist (4). Various theories of cognitive motor learning to enable skill acquisition have been described in the literature, including Kopta’s theory, Schmidt’s schema theory, the cognitive appren-
55
56
•
Simulation for Teaching and Assessing Endovascular Skills
ticeship model, and Ericsson’s rationale for the acquisition of expertise (5). Kopta’s theory highlights the importance of observation followed by practice. It involves three phases of acquisition of motor skills. The cognitive phase is observing new procedures. This is followed by the integrative phase, which involves feedback during practice. This information is put together toward appropriate motor responses, resulting in less erratic movements. Finally, there is the autonomous phase, where continued practice results in efficient performance of the task, without cognitive input resulting in automatic movement (6). Schmidt’s theory looks more into how a motor skill develops. As trainees practice a maneuver, they learn the relationship between the movement parameters of the maneuver and the outcome (7). They then improve their understanding of the relationship between a maneuver outcome and their control of the movement’s parameters. An important prediction of the theory is that people will learn more quickly the relationship between manipulating parameters and achieving a desired outcome if they practice a task in a wide variety of situations and experience errors in the process. Practice that lacks variety, but is instead precise or repetitious, will not (from Schmidt’s perspective) provide enough information for a learner to fathom the rules that underlie the generalized motor program. This is very important when considering VR training. The cognitive apprenticeship model decontextualizes the knowledge so that a trainee can apply new skills in more than one setting. The cognitive apprenticeship model has three components performed by the mentor (modeling, coaching, and scaffolding) and three components performed by the student (articulating, reflecting, and exploring). This model of cognitive motor learning emphasizes the early stages of learning (8). Ericsson’s model of skill acquisition of expertise emphasizes the importance of focused attention and practice in acquiring expert skill. Ericsson found that time of day was an important factor in success, with the morning being the best time to practice because this was when the ability to
perform complex cognitive activities was the highest (9). The core concepts of these models of cognitive learning must all be considered when designing and implementing an effective simulator training curriculum. Trainees require focused attention, which translates into protected uninterrupted time to observe, practice, obtain trainer feedback, experience procedure variety, and reflect on the acquisition of a new endovascular skill.
HISTORY OF TRAINING AND ASSESSMENT OF TECHNICAL SKILLS IN CRAFT SPECIALITIES When considering the history of methods of medical training and assessment of technical skills in craft specialties such as endovascular intervention, authors begin with Sushutra from ancient India and Halstead from the early 19th century (10,11), who described the concept of clinical training based on the practice of inanimate objects—the so-called “see one, do one, teach one approach.” Today, Sacks (12) eloquently describes the vision for simulators as “practice a whole bunch on a simulator and get good, then see a few, do a few, teach a few.” The modern history of medical simulation began in the 1950s with the pioneering work in cardiopulmonary resuscitation (13), followed by the introduction of computer-controlled simulation by Denson and Abrahamson (14). Simulators have been developed recently that are capable of training with catheter and guide wire techniques with real-time simulated fluoroscopic imaging (15–17). Evaluation of technical skills was an essential component of the examination for the fellowship to the Royal College of Surgeons of England (18), but cadaver availability led to its discontinuation in the 1940s (19). Aptitude testing dates back to 1949, when all American dental schools implemented the Chalk Carving Test to assess the psychomotor ability of the candidates. This test was so simple and basic that a student who performed very poorly on the test did not usually improve with repeated testing. However, the test proved to be expensive to administer and was not as discriminatory as compared to the PaperPencil Examination, which replaced it
January 2010
JVIR
in 1973 (20). Other mechanical aptitude and space relations tests correlated significantly with technical skills of dental students (21,22). In 1871, Kopta (6) outlined the scheme for technical skills development featuring three major steps that included perception, integration, and automatization. Lippert et al (23) described acquisition of skills as to be multidimensional and explained that the process of achieving automatization requires theoretical knowledge and appropriate skills in a specific domain as well as psychomotor distinction. One of the first skills workshops for practitioners was held at the Royal College of Surgeons of England in 1977 (24). Since then, the intercollegiate basic surgical skills course has become a mandatory part of surgical training in the United Kingdom. Within the United Kingdom, the need for personality assessment techniques and aptitude testing in the selection of surgical trainees was raised during an Anglo-Dutch symposium held in 1987 (25), although no practical developments have ensued since. Recent interest in the selection of surgical trainees has been directed toward the use of aptitude tests (26 –28). In the United States, the need for psychomotor skills training in orthopedic surgery was recognized in the mid-1960s, and in Canada courses in surgical techniques were held as early as 1962 (23,29). In 1975, these were followed up with the first 18-hour course on motor skills for residents held at the University of Washington’s School of Medicine (23). The structured approach to assess clinical examinations (Objective Structured Clinical Examination) was first introduced by Harden et al (30) in 1979 and followed the Objective Structured Practical Examination as described by the same authors (30). The main features include (a) separate assessment of process and product through observation of performance and assessment of end result, (b) adequate sampling of skills and content to be tested, (c) an analytical approach to the assessment, (d) objectivity, and (e) feedback to the teacher and student (31). In the 1990s, an increase in the need for assessing the technical skills of doctors in craft specialties led researchers in Canada to develop a new tool named Objective Structured Assessment of Technical Skill. Objective
Volume 21
Number 1
Ahmed et al
Table 1 Categories of Available Simulations Simulation Synthetic models Animal models
Human cadaver (48)
VR simulation
Advantages
Disadvantages
Cheap Reusable portable Minimal risk High fidelity Full procedure simulation Assessment potential
Low face validity Basic generic tasks Low fidelity Cost Ethical issues Special facilities required Anatomic difference Availability Cost
High fidelity High face validity Full procedure simulation Assessment potential Reusable Data capture Minimal setup time Assessment potential High degree of realism
57
simulation is likely to shift training away from patients to a low-risk environment (41). Currently, there is an increasing trend toward adoption of VR endovascular interventional simulations for training. The types of simulation models include cadaver, animal, bench, and computer software– based simulators. We have looked into the utility, feasibility, and limitations of each modality of simulation for endovascular intervention (Table 1). Synthetic Models
Cost Maintenance Acceptability Non-availability of threedimensional models
Note.—From reference 41.
Structured Assessment of Technical Skill, a derivative of the Objective Structured Clinical Examination introduced in 1997, is a performance-based assessment exercise designed to assess the technical skills of surgical trainees (32). Trainees rotate through different stations within a predetermined time. Six-station examination of technical skills includes excision of skin lesion, hand-sewn bowel anastomosis, stapled bowel anastomosis, insertion of a T-tube, abdominal wall closure, and control of inferior vena caval hemorrhage. The assessment of technical proficiency is based on a global rating scale and a checklist specific to the operation or task, thus making this process more objective, reliable, and valid. The other assessment method introduced during the 1990s was the modified version of Human Reliability Assessment techniques (33). Human Reliability Assessment was adapted for medical practice to categorize and record errors during laparoscopic surgery. Subsequently, the applicability, feasibility, and validity of this observational methodology in the assessment of technical skills in otolaryngology and ophthalmology were confirmed by other groups (34–36). These tools have not been validated for workplace for assessment of interventionalists. Recently, a number of endovascular simulators have been developed that allow practice with the manipulation
•
of catheters and guide wires, contrast media injection, and real-time fluoroscopy (37). Simulators also can provide feedback as to operator performance regarding correct order of procedure steps, procedure time, fluoroscopy time, and specific procedure outcome measures. The field of interventional radiologic procedural simulation is still in the early stages of development. There is a dearth of evidence to support recognition for simulation of endovascular interventions, although there is a role for training any steps where content is shown to be valid. There are more limited roles where a simulation may be suitable for training tasks such as procedure sequencing and use of devices (38). Also, in combination with a dedicated clinical and cognitive curriculum, simulators may have a role in the training, continuous assessment, and re-certification of interventional radiologists (39).
Synthetic models are relatively simple and cost effective. These models can replicate the beating heart and dynamic behavior of human arterial circulation to some extent (42,43). The only problem that limits advanced simulation models is the effect of friction during passage of a real device through the simulated blood vessels (44). Guide wire insertion and balloon inflation can be taught by using the real devices and instruments on synthetic models (45). The feasibility and validity of these training models in the context of endovascular training remains unproved. Animal Models Animal models have a relatively higher degree of face validity, with an added advantage being that both open and minimally invasive procedures can be performed (46,47). The use of animal models is limited by cost, ethics, animal licenses, and lack of pathology. However, many atherosclerotic models can now be created. Moreover, animals require a great amount of logistics such as a dedicated animal facility, trained personnel (including an anesthetist), and an operating room. Human Cadavers
ROLE OF SIMULATION IN SKILLS TRAINING Simulators are instruments that reproduce, under artificial conditions, components of clinical tasks that are likely to occur under normal circumstances (40). Simulation, originally developed in aviation, allows training and practice of complex procedures to a proved proficiency before performance at the workplace. With further advances in simulation technology,
The human cadaveric circulation model offers the most lifelike conditions possible for training endovascular skills consisting of many elements of the normal and abnormal human circulation. The benefits include a better-quality model for research and development of endovascular technology and a widely applicable training tool for endovascular techniques (48). Catheter manipulation and balloon angioplasty can be performed repeat-
58
•
Simulation for Teaching and Assessing Endovascular Skills
Figure 1. Miller’s pyramid for assessing clinical competence. Adapted from reference 60 (Available in color online at www.jvir.org.).
edly; however, once stents or stentgrafts are deployed within the arteries, they cannot easily be retrieved, thus limiting that arterial segment for further device implantation. Cadavers can be an effective way of teaching technical procedural skills because the procedure can be performed repeatedly to some degree, but the effectiveness in endovascular intervention is not established. Moreover, limited availability and high cost related to transfer and preservation limits the use of human cadavers for endovascular training (49). Virtual Reality VR is a communication interface based on interactive three-dimensional visualization that allows the user to interact with and integrate different sensory inputs that simulate important aspects of real-world experience (50). In the current settings, computer-based simulation seems to be the most practical option for endovascular training (51,52). Despite reduced tactile feedback (haptics), lack of case-by-case variety, and lack of exact replication of fine motor skills with VR, an early endovascular training experience is possible with the advantage of objective assessment of performance. With the added benefit of both audio and video recordings of the trainee’s VR performance, expert reviewers can assess each step of the simulation at a remote time and location (53,54). In addition, VR provides a
potential opportunity to enable continuous assessment of trained endovascular specialists and a method to examine individual suitability for specialist recertification. Available endovascular simulators include Procedicus VIST (Mentice, Gothenburg, Sweden), Angiomentor (Simbionix, Cleveland, Ohio), Simsuite (Medical Simulation, Denver, Colorado), and Endovascular Accutouch (Immersion Medical, Gaithersburg, Maryland). These simulators include guide wire and catheter insertion; angioplasty and stent placement modules of carotid, renal, iliac, and coronary vessels; neuro-interventions; and closure of patent foramen ovale. VR simulation has been shown to be an effective way of teaching minimally invasive procedural skills in certain craft specialties (55). Improvement of endovascular technical skills of trainees following simulator-based training has been demonstrated by various authors (15–17,56).
TRANSFERABILITY OF SKILLS FROM VR MODELS TO REAL PATIENTS Transferability of endovascular skills acquired from VR to animals and human patients has been demonstrated by various authors. Transfer of skills to a pig model from VIST has been shown by Berry et al (51) using the observational parameters (global assessment by an assessor) of assess-
January 2010
JVIR
ment. Chaer et al (37) performed a randomized trial examining transfer of VR endovascular training (iliofemoral angioplasty) to the human model. In that study, training (VR and didactic) compared with the control group (didactic only) significantly improved performance that was rated with use of a procedure-specific checklist and a global rating scale (37). Simulator-derived performance reporting can also be used to track a trainee’s learning curve in endovascular interventions (57). Furthermore, VR can potentially be useful for the development of a proficiency-based training curriculum, with progress determined by the demonstration of skills performance to a predetermined benchmark level (52). Attainment of this benchmark could possibly then serve as a prerequisite to supervised procedure performance in the real patient setting, with failing to acquire this benchmark necessitating further VR training (58). A few limitations of these studies include validation through a straightforward iliac task along with an unvalidated, observer-based assessment tool that failed to measure dexterity— which is crucial to the safe performance of some key procedural steps in patients. Moreover, the simulator-trained group received additional cognitive training that was not delivered to the apprenticeship-trained group (37). Further research is needed to establish the transferability of skills acquired from the VR models to real human patients.
ASSESSMENT OF TECHNICAL SKILLS Assessment can be defined as the process of documenting, usually in measurable terms, knowledge, skills, attitudes, and beliefs (59). In 1990, psychologist George Miller proposed a pyramidal framework for the assessment of clinical competence. At the bottom of the pyramid is knowledge (“knows”), followed by competence (“knows how”), performance (“shows how”), and action (“does”). “Action” in this pyramid focuses on what happens in real life (60). The target of assessment in healthcare is this top level of the Miller pyramid to obtain evidence about doctor’s performance (Fig 1). The assessment tools can broadly be classified into observational and
Volume 21
Number 1
Ahmed et al
•
59
Table 2 Requirement for Optimal Assessment Tool Feasibility: Measure of whether something is capable of being done or carried out. Validity: Face validity is the extent to which the examination resembles the situation in the real world. Content validity is the extent to which the intended content domain is being measured by the assessment exercise—for example, while trying to assess technical skills we may actually be testing knowledge. Construct validity is the extent to which a test measures the trait that it purports to measure. One inference of construct validity is the extent to which a test discriminates between various levels of expertise. Concurrent validity is the extent to which the results of the test correlate with the gold standard tests known to measure the same domain. Predictive validity is the extent to which this assessment will predict future performance. Reliability: Reliability is a measure of a test to generate similar results when applied at two different points (test-retest). When assessments are performed by more than one observer, another type of reliability test is applicable that is referred to as inter-rater reliability, which measures the extent of agreement between two or more observers. Fidelity: Fidelity refers to the degree to which a simulation reproduces the state and behavior of a real world object, feature, or condition, in this case an endovascular intervention. Fidelity is therefore a measure of the realism of a model or simulation. Simulation fidelity has also been described in the past as the degree of similarity. Note.—From references 41, 75, and 89.
nonobservational tools. For assessment tools to be effective and acceptable, they should be feasible, reliable, and valid (Table 2). Observational Tools for Assessment Recently, a number of authors have successfully applied observational tools (rating scales) to assess the performance on VR simulators. Observational rating scales have been used in parallel to objective measures of VR simulators (47,61). Global assessment of technical skills is based on either procedure-specific or generic rating scales. Generic global rating scales have established validity and reliability in surgery and allied specialties. The components of generic rating scales include respect for tissue, time and motion, instrument handling, knowledge of instruments, use of assistants, flow of operation and forward planning, and knowledge of the specific procedure steps (32). In addition to global scales, procedure-specific checklists based on accurate performance of procedural steps and scoring based on error execution have also been used by various authors (62). Global and task-specific checklists have been validated on models such as vascular anastomosis (arterial anastomosis, arteriovenous anastomosis, insertion of vein patch and graft to arterial anastomosis), saphenofemoral disconnection, femoral triangle dissection, carotid endarterectomy, and
creation of a forearm arteriovenous bridge graft (62– 66). An observational scale based on global rating has been shown to demonstrate a significant level of construct validity with transfer of endovascular skills from VR to an animal model when scored by independent assessors (47). Nonobservational Tools Nonobservational tools—VR simulators and motion analysis devices—are based on an automated assessment process using computer-based measurements of specific procedural events. Performance scoring is not carried out by individual assessors. The VR simulator provides objective evidence of performance at the end of each task, whereas motion analysis performance is determined by graphical representation of movements and time. VR simulators instantly provide a nonobservational objective report of the performance. Scoring is based mainly on errors enacted, economy of movements, and time taken to complete a task. Observational rating scales have also been validated in combination with nonobservational VR scoring (67,68). VR simulators have been shown to differentiate between level of experience (17,69,70). A recent study by Kundhal and Grantcharov (71) demonstrated significant correlations between performance in
the operating room (assessed with a well-validated rating scale) and that in a virtual environment (assessed with a computer simulator), thus validating the simulator system as an objective tool for assessing minimally invasive psychomotor skills. The VIST simulator has been used widely by various researchers. VIST VR simulator produces an objective metrics at the end of a procedure that includes static (procedure time, fluoroscopy time, contrast medium volume, cine-loops) and dynamic simulator metrics (stent or balloon/vessel ratio, coverage percentage, placement accuracy, residual stenosis, and lesion coverage). Efficiency and economy of movements are known to be the key discriminators of technical skill in craft specialties (72). The Imperial College Surgical Assessment Device tracks hand movement in three dimensions by using electromagnetic sensors, with a composite score based on economy of movement and qualitative analysis (73). The Imperial College Surgical Assessment Device has been used in conjunction with checklists to assess technical skills of vascular and cardiac surgical trainees, both in laboratory and real settings (72,74). As yet, there is no motion analysis– based assessment study available for endovascular skill evaluation.
60
•
January 2010
Simulation for Teaching and Assessing Endovascular Skills
JVIR
Table 3 Simulator-based Studies in Endovascular/Interventional Radiology Study
Level Participants
Specialty
Task*
Procedicus VIST Berry et al, 2008 (76)
Interventional radiologists and medical students Radiology registrars General surgery
CAS
Klass et al, 2008 (57) Tedesco et al, 2008 (61)
16 interventional radiology experts and 16 medical students 12 new intake 17 4th- and 5th-year residents
Glaiberman et al, 2008 (77)
Interventional radiology fellows
Radiology fellowship
RAS (3 levels of difficulty)
Van Herzeele et al, 2007 (78)
45 experts with varying experience
Cardiologists, radiologists, vascular surgeons
CAS
Aggarwal et al, 2006 (70)
20 (⬎50 endovascular procedures as primary operator [n ⫽ 8], ⬍10 procedures [n ⫽ 12]) 20 experienced interventionalists without experience in carotid artery angiography
Vascular surgeons
RAS
Interventional cardiologists
Carotid angiograms; 5 test-trials
Patel et al, 2006 (68)
5 left RAS in 6 mo RAS
Berry et al, 2006 (91)
8 expert interventional radiologists, 8 medical students
Interventional radiologists and medical students
RAS, 6 repetitions
Dayal et al, 2004 (17)
21 physicians (5 of them were experts)
Physicians
CAS
20 novice endovascular experience in 2 groups (groups A1, A2) and 27 expert endovascular physicians (group B)
Surgical trainees/physicians
IAS
12 vascular surgeons and interventional radiologists
Vascular surgeons and interventional radiologists
IAS
5 experienced angiographers
Angiographers
RAS
29 subjects (16 untrained, 13 advanced)
Neurosurgeons
CAS
Procedicus-VIST and cognitive training Van Herzeele et al, 2008 (56)
Procedicus VIST and cognitive, compared with pigs Berry et al, 2007 (47) AngioMentor angiographic simulator Duncan et al, 2007 (53)
Simulator model Hsu et al, 2004 (69)
Volume 21
Number 1
Assessment Method
SM, DM, VAS SM, DM, MP SM, DM, GRS (2-expert), VAS SM, DM, GRS (2-expert) SM,DM, VAS SM SM, DM
SM, DM, VAS
SM, GRS, TSC
Ahmed et al
•
61
Results/Conclusion
Procedure and fluoroscopy times showed construct validity of the Procedicus VIST, other metrics did not. VR was valued more by novices. Procedure and fluoroscopy times improved, DM fluctuated. Procedure time as an endpoint. Procedure time, fluoroscopy time, contrast medium volume, coverage percentage, placement accuracy, residual stenosis, and cine-loops were similar between the two groups. Structured endovascular skills assessment based on a checklist carried out by a blinded assessor correlated well with prior procedural experience within a high-fidelity simulation environment. SM and DM can help differentiate levels of skills. Procedure and fluoroscopy times helped differentiate between levels of CAS experience in experienced interventionalists. DM scoring is currently not a valid mode of assessment and needs refinement. SM can differentiate expert interventional radiologists. Procedure and fluoroscopy times improved. SM at previous study showed that novices can be better than experts based on procedure and fluoroscopy times (88). Procedure time, fluoroscopy time, and CE improved. Internal consistency is high for CE. CE has highest test-retest reliability. Contrast medium volume showed high enough test-retest reliability. This measure was low for procedure time, fluoroscopy time, and number of cine-loops. DM is a more reliable metric than procedure time, fluoroscopy time, and contrast use. Procedure time, fluoroscopy time and contrast medium volume are a crude assessment of the technical performance of the operator. No difference in residual stenosis, placement accuracy, procedure time, lesion coverage, or TVR. The fluoroscopy time was greater for the novice group (P ⬍ .01). Experts rated 6 of the 8 subjective parameters favorably, whereas the novice group approved of 7. With the exception of fluoroscopy time, SM failed to stratify performances based on experience level. Procedure and fluoroscopy times improved significantly for novices. No statistically significant difference in score, procedure time, or fluoroscopy time was noted for experts. Improvement was noted in guide wire and catheter manipulation skills in novices. Novices derived the greatest benefit from simulator training in a mentored program, whereas experienced interventionalists did not seem to derive significant benefit.
SM, DM
With SM: A1 worse than A2, A1 worse than B, A2 better than B. With DM: A1 better than A2, A1 equal to B, A2 worse than B. Cognitive skills training significantly improved the quality of the end product on a VR endovascular simulator and is fundamental before the assessment of inexperienced subjects. Error module was not used in the study.
TSC, GRS, taskspecific checklist
Endovascular skills learned in the virtual environment may be transferable to the real catheterization laboratory as modelled in the Porcine Laboratory. Skills acquired by using pigs were not transferable when using VR. SM and DM were not used.
SM, DM (from videos), and aggregated together
Task analysis facilitated both protocol development and data analysis. Efficiency metrics were readily extracted from recordings of simulated procedures. Aggregating the metrics and dividing the procedure into segments revealed potential insights that could be easily overlooked because the simulator currently does not attempt to aggregate the metrics and only provides data derived from the entire procedure. The data indicate that analysis of simulated angiographic procedures will be a powerful method of assessing performance in interventional radiology.
Procedure time
Performance on the carotid stent placement simulator correlated with previous endovascular experience. Although both novice and advanced groups improved their procedure time after a 30–60-minute proctored training session, improvement in the novice group was greater than that in the advanced group, which suggests that novices may benefit disproportionately from this type of training. (continued)
62
•
January 2010
Simulation for Teaching and Assessing Endovascular Skills
JVIR
Table 3 Simulator-based Studies in Endovascular/Interventional Radiology (continued) Study Dawson et al, 2007 (15)
Level Participants Vascular surgery residents; type of simulator: SimSuite
Specialty Vascular surgery trainees
Task* Arteriography and intervention for treatment of aortoiliac, renal, and carotid artery disease
Note.—Assessment methods were as follows: objective nonobservational static simulator metrics (SM), which included procedure time, fluoroscopy time, contrast medium volume, and cine-loops; objective nonobservational dynamic simulator metrics (DM), which included tool/vessel ratio (TVR), coverage percentage, placement accuracy, residual stenosis, and lesion coverage; subjective self assessment, which included observation with visual analog scale (VAS); subjective observational method, which included experts using a mistakes profile (MP), global rating scale (GRS), task-specific checklist (TSC), and video recording assessment (VRA). * CAS ⫽ carotid artery stenosis angioplasty procedure, IAS ⫽ iliac artery stenosis angioplasty procedure, RAS ⫽ renal artery stenosis angioplasty procedure.
DISCUSSION Training effectiveness of simulators requires validity of the content for all possible procedural steps and that these are replicated with suitable fidelity (realism) and in an environment with appropriate face validity. For simulation-based assessment, steps that are considered crucial for the safe completion of the target procedure must be present (a part of face and content validation) and must also be correctly assessed with a weighting as to their critical nature. To date, no ideal tool for training and assessment is available (75). Herein, we presented an overview of the advances in knowledge within the constantly evolving field of training and assessment of endovascular technical skills (Table 3). Furthermore, we have pointed out the deficiencies in the literature along with a brief note about fundamental inconsistencies between the results of the relevant studies. Most of the available studies with regard to endovascular intervention are based on VR simulation. Wide variability exists in results from the studies carried out to date in terms of establishing the validity of static and dynamic parameters (Table 4) of VR simulators. Although performance and fluoroscopy times are used by some investigators to establish construct validity of the VIST simulator (17,70,57), others considered these parameters as not showing construct validity (76). According to Van Herzeele et al (56), cognitive training improved the end product assessment but slowed the
performance and fluoroscopy times. In addition, the performance and fluoroscopy times of novices was better than those of experts (56). Unlike others, who used performance and fluoroscopy times to assess performance improvement, this study assesses performance improvement on the end product assessment parameters (70). Another study (77) concludes that both static and dynamic parameters can be used to help differentiate between middle grade level participants. Van Herzeele et al (78) showed that performance and fluoroscopy times, components of static metrics, can be reliable in differentiating between experts and that dynamic metrics are unreliable, whereas Patel and Gould (79) have drawn the opposite results from their study. In another study (17), performance and fluoroscopy times were unable to differentiate the level of experienced performers; opposite results were demonstrated by another group (69). Error scoring has been shown to demonstrate significantly the level of experience in studies based on the Minimally Invasive Surgical Trainer– Virtual Reality (MIST-VR; Mentice Medical Simulation, Gothenburg, Sweden) simulator (80). However, error scorings based on assessment through VIST simulators are unable to demonstrate significant trends (57). Error analysis is a wellestablished method of scoring the performance, and the reasons why errors are unable to demonstrate the difference could be due to the fact that task analysis is not adequate. Due to this wide level of discrepancy among the results, individual tasks in a procedure
must be analyzed by the experts to define metrics and critical performance indicators. Test validation should include content, construct, concurrent, and predictive validation with the objective of demonstrating transfer of VR-trained skills to procedures on patients. Feedback must be analyzed in the context of the educational effect and acceptability to both the trainers and the trainees, as a system not acceptable to either will not last for long (81,82). Furthermore, many studies on training and transfer have focused on time to completion as the major endpoint (70,83,84). Experts can perform a procedure in a given time, but for a trainee or novice time to completion should not be considered as an indicator of technical skill. Quantitative measurements such as procedure and fluoroscopic times may measure efficiency but are inadequate for assessing quality. The important parameters must be analysis of errors and the quality of the end product (33,85). Advances in simulator technology have presented a great opportunity for endovascular training and assessment. However, key questions about the exact role of the existing simulators in training programs must still be resolved. It is evident that the current generation of simulators, regardless of the degree of fidelity, will only function as an adjunct to, not a replacement for, traditional teaching in the clinical setting. In existing form, simulators are unable to provide a substitute for the skills and experience gained by a trainee while under the close supervision of a mentor in real
Volume 21
Number 1
Ahmed et al
Assessment Method Procedure time, Contrast medium volume
•
63
Results/Conclusion Procedure time and contrast medium volume improved. Selection of angioplasty balloon catheters and stents was improved.
Table 4 Objective Parameters for the VIST-VR Simulator Parameter
Description
Procedure time
Minutes needed to complete the entire procedure
Fluoroscopy time
Duration of fluoroscopy during the procedure
Contrast medium
Total amount of contrast medium used, in milliliters No. recorded during procedure Percentage of lesion covered by selected tool Inflated tool’s diameter to the lesion diameter Longitudinal distance in millimeters between the tool’s center and the lesion’s center Percentage stenosis after dilation or stent deployment
Cine-loops Lesion coverage Tool:lesion ratio Placement accuracy Residual stenosis
Construct Validity (Significant Results)
Intertest Reliability
Berry et al 2008 (51), Klass et al (57), Dawson 2007 (15), Van Herzeele et al 2007 (78), Aggarwal et al 2006 (70), Patel et al 2006 (68) Berry et al 2006 (91), Berry et al 2008 (51), Van Herzeele et al 2007 (78), Aggarwal et al 2006 (70), Patel et al (68) Dawson et al 2007 (15), Patel et al 2006 (68) ... ... ...
Patel et al 2006 (68) ... ... ...
...
...
...
...
...
...
Note.—From reference 88.
settings. Simulators are more approved by novices than by experts. This shows that simulators have an evident role in familiarizing the novices with the technique of a certain procedure: the tools, the procedure steps, the basic target, the possible errors, and the basic movements. Once these components of a procedure are already known to an operator—as in the case of experts—the value of available simulators becomes limited. This might be explained by Schmidt’s theory, as noted above, that the limited variety of situations provided by the simulator for a specific procedure is in fact a limitation in the training capability of these tools. In addition, the
Figure 2. Simulated operating suite for training and assessment of technical and nontechnical skills at Imperial College London. (Available in color online at www.jvir.org.)
64
•
Simulation for Teaching and Assessing Endovascular Skills
reduced tactile feedback and lack of exact replication of fine motor skills are limitations. A great deal of work is already in progress to further develop and design high-fidelity endovascular simulators to more closely mirror the “real” interventional procedural experience and to improve VR validation and the ability for skill transfer. Depending on continued development, improved design, and further validation, it is likely that simulation-based training and assessment will be included in the curriculum in the near future. Moreover, progress is being made toward the approximation of educational and professional experts with a common goal on the horizon (ie, structured training with objective assessment). The Society of Interventional Radiology, the Cardiovascular and Interventional Radiological Society of Europe, and the Radiological Society of North America have established a medical simulation task force that is also supported by the British Society of Interventional Radiologists (86). The common objective is to establish recommendations for the validation of simulator models, use of simulators for training and certification in interventional radiology (38,87). Simulation models (Fig 2) may be suitable for certain aspects of procedural experience, such as learning the sequence of procedural steps and appropriate tool selection, thereby reducing the procedure learning curve. Patient-specific procedural simulation is another promising utility of simulators that can help interventional radiologists maintain their proficiency and skills (88,90). There is no counterpart to the quality of clinical workplace-based training and assessment. Simulators cannot be the replacement for training in a real environment because of the level of decision making, complexity, and interactions involved there. Therefore, training on simulators can only serve as an augmentation to the quality of training in the real setting. In view of continuing advancement in technology, simulation training may become a prerequisite for certification or credentialing; in its present form, however, it is insufficient for either (38). In conclusion, simulation training provides a safe platform for training outside the clinical workplace. Before widespread inclusion into practice,
performance assessment parameters of the VR simulators must be rationalized with subsequent validation. Further research regarding human factors– based simulator development and design is required. The issues with transferability of various training models, particularly if they are to become a part of training and assessment for certification or licensure, must be addressed. Development of a model is needed to simulate an entire interventional radiologic procedure. Furthermore, trainer observational assessment tool validation for marking trainees and specialists is still pending. References 1. Bridges M, Diamond DL. The financial impact of teaching surgical residents in the operating room. Am J Surg 1999; 177:28 –32. 2. Gates EA. New surgical procedures: can our patients benefit while we learn? Am J Obstet Gynecol 1997; 176: 1293–1298, discussion 1298 –1299. 3. Warf BC, Donnelly MB, Schwartz RW, Sloan DA. Interpreting the judgment of surgical faculty regarding resident competence. J Surg Res 1999; 86:29 –35. 4. Dawson S, Gould DA. Procedural simulation’s developing role in medicine. Lancet 2007; 369:1671–1673. 5. Wong JA, Matsumoto ED. Primer: cognitive motor learning for teaching surgical skill— how are surgical skills taught and assessed? Nat Clin Pract Urol 2008; 5:47–54. 6. Kopta JA. The development of motor skills in orthopaedic education. Clin Orthop Relat Res 1971; 75:80 – 85. 7. Schmidt RA. A schema theory of discrete motor skill learning. Psychol Rev 1975; 82:225–260. 8. Collins A, Brown JS, Newman SE. Cognitive apprenticeship: teaching the crafts of reading, writing, and mathematics. In: Resnick LB, ed. Knowing, learning, and instruction: essays in honor of Robert Glaser. Hillsdale, NJ: Lawrence Erlbaum Associates, 1989; 453– 494. 9. Ericsson KA. The acquisition of expert performance: an introduction to some of the issues. In: Ericsson KA, ed. The road to excellence: the acquisition of expert performance in the arts and sciences, sports, and games. Mahwah, NJ: Lawrence Erlbaum, 1996; 1–50. 10. Halsted WS. The training of the surgeon. Bull Johns Hopkins Hosp 1904; 15:267–275. 11. Iram R. The Far East. In: Iram R, ed. Surgery. an illustrated history. Philadelphia: Mosby, 1993; 65– 69.
January 2010
JVIR
12. Sacks D. President’s address at the 2008 SIR annual members’ business meeting. J Vasc Interv Radiol 2009; 20: 303–305. 13. Safar P. Ventilatory efficacy of mouth-tomouth artificial respiration; airway obstruction during manual and mouthto-mouth artificial respiration. J Am Med Assoc 1958; 167:335–341. 14. Denson JS, Abrahamson S. A computercontrolled patient simulator. JAMA 1969; 208:504 –508. 15. Dawson DL, Meyer J, Lee ES, Pevec WC. Training with simulation improves residents’ endovascular procedure skills. J Vasc Surg 2007; 45:149 – 154. 16. Wang T, Darzi A, Foale R, Schilling RJ. Virtual reality permanent pacing: validation of a novel computerized permanent pacemaker implantation simulator. J Am Coll Cardiol 2001; 37:493A– 494A. 17. Dayal R, Faries PL, Lin SC, et al. Computer simulation as a component of catheter-based training. J Vasc Surg 2004; 40:1112–1117. 18. Regulations relating to the examinations for the diploma of fellow. Minutes of Council 1941–1943. London, England: Royal College of Surgeons of England, 1943; 413– 423. 19. Quarterly council, 1946. Minutes of Council 1945–1947. London, England: Royal College of Surgeons of England, 1947; 447– 461. 20. Peterson S. The ADA chalk carving test. J Dent Educ 1974; 38:11–15. 21. Deubert LW, Smith MC, Jenkins CB, Berry DC. The selection of dental students. a pilot study of an assessment of potential manula ability by psychometric tests. Br Dent J 1975; 139:167–170. 22. Wong AY, Watson JF, Thye RP. Evaluation of predictor variables for a self-instructional preclinical course. J Dent Educ 1979; 43:637– 640. 23. Lippert FG III, Spolek GA, Kirkpatrick GS, Briggs KA, Clawson DK. A psychomotor skills course for orthopaedic residents. J Med Educ 1975; 50:982–983. 24. Apley AG. Fixation of fractures: organising a course. Ann R Coll Surg Engl 1980; 62:219 –222. 25. Gough MH, Holdsworth R, Bell JA, et al. Personality assessment techniques and aptitude testing as aids to the selection of surgical trainees. Symposium. England, 18 November 1987. Ann R Coll Surg Engl 1988; 70:265–279. 26. Gough M, Bell J. Introducing aptitude testing into medicine. Br Med J 1989; 298:975–976. 27. Grace DM. Aptitude testing in surgery. Can J Surg 1989; 32:396 –397. 28. Graham KS, Deary IJ. A role for aptitude testing in surgery? J R Coll Surg Edinb 1991; 36:70 –74.
Volume 21
Number 1
29. Couves CM. A course in surgical technique for medical students. Can J Surg 1970; 13:31–32. 30. Harden RM, Stevenson M, Downie WW, Wilson GM. Assessment of clinical competence using objective structured examination. Br Med J 1975; 1: 447– 451. 31. Harden RM, Gleeson FA. Assessment of clinical competence using an objective structured clinical examination (OSCE). Med Educ 1979; 13:41–54. 32. Martin JA, Regehr G, Reznick R, et al. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg 1997; 84:273–278. 33. Joice P, Hanna GB, Cuschieri A. Errors enacted during endoscopic surgery: –a human reliability analysis. Appl Ergon 1998; 29:409 – 414. 34. Tang B, Hanna GB, Joice P, Cuschieri A. Identification and categorization of technical errors by observational clinical human reliability assessment (OCHRA) during laparoscopic cholecystectomy. Arch Surg 2004; 139:1215–1220. 35. Malik R, White PS, Macewen CJ. Using human reliability analysis to detect surgical error in endoscopic DCR surgery. Clin Otolaryngol Allied Sci 2003; 28:456 – 460. 36. Cox A, Dolan L, Macewen CJ. Human reliability analysis: a new method to quantify errors in cataract surgery. Eye 2008; 22:394 –397. 37. Chaer RA, Derubertis BG, Lin SC, et al. Simulation improves resident performance in catheter-based intervention: results of a randomized, controlled study. Ann Surg 2006; 244:343–352. 38. Gould DA, Reekers JA, Kessel DO, et al. Simulation devices in interventional radiology: caveat emptor. Cardiovasc Intervent Radiol 2006; 29:4 – 6. 39. Gould DA. Interventional radiology simulation: prepare for a virtual revolution in training. J Vasc Interv Radiol 2007; 18:483– 490. 40. Krummel TM. Surgical simulation and virtual reality: the coming revolution. Ann Surg 1998; 228:635– 637. 41. Reznick RK, MacRae H. Teaching surgical skills: – changes in the wind. N Engl J Med 2006; 355:2664 –2669. 42. Tokuda Y, Song MH. A simple training model for coronary artery anastomoses. Heart Surg Forum 2006; 9: E880 –E882. 43. Stanbridge Rde L, O’Regan D, Cherian A, Ramanan R. Use of a pulsatile beating heart model for training surgeons in beating heart surgery. Heart Surg Forum 1999; 2:300 –304. 44. Suzuki Y, Fujitsuka M, Chaloupka JC. Simulation of endovascular neurointervention using silicone models: imaging and manipulation. Neurol Med
Ahmed et al
45.
46.
47.
48. 49. 50. 51.
52.
53.
54.
55.
56.
57.
58. 59.
Chir (Tokyo) 2005; 45:567–572, discussion 572–563. Schachner T, Bonaros N, Ruttmann E, et al. Training models for coronary surgery. Heart Surg Forum 2007; 10: E248 –E250. Donias HW, Schwartz T, Tang DG, et al. A porcine beating heart model for robotic coronary artery surgery. Heart Surg Forum 2003; 6:249 –253. Berry M, Lystig T, Beard J, Klingestierna H, Reznick R, Lonn L. Porcine transfer study: virtual reality simulator training compared with porcine training in endovascular novices. Cardiovasc Intervent Radiol 2007; 30:455– 461. Garrett HE Jr. A human cadaveric circulation model. J Vasc Surg 2001; 33:1128 –1130. Satava RM. Virtual reality surgical simulator: the first steps. Surg Endosc 1993; 7:203–205. Riva G. Applications of virtual environments in medicine. Methods Inf Med 2003; 42:524 –534. Berry M, Hellstrom M, Gothlin J, Reznick R, Lonn L. Endovascular training with animals versus virtual reality systems: an economic analysis. J Vasc Interv Radiol 2008; 19:233–238. Neequaye SK, Aggarwal R, Van Herzeele I, Darzi A, Cheshire NJ. Endovascular skills training and assessment. J Vasc Surg 2007; 46:1055– 1064. Duncan JR, Kline B, Glaiberman CB. Analysis of simulated angiographic procedures. II. Extracting efficiency data from audio and video recordings. J Vasc Interv Radiol 2007; 18:535–544. Duncan JR, Glaiberman CB. Analysis of simulated angiographic procedures. I.– Capture and presentation of audio and video recordings. J Vasc Interv Radiol 2006; 17:1979 –1989. Sutherland LM, Middleton PF, Anthony A, et al. Surgical simulation: a systematic review. Ann Surg 2006; 243: 291–300. Van Herzeele I, Aggarwal R, Neequaye S, et al. Experienced endovascular interventionalists objectively improve their skills by attending carotid artery stent training courses. Eur J Vasc Endovasc Surg 2008; 35:541–550. Klass D, Tam MD, Cockburn J, Williams S, Toms AP. Training on a vascular interventional simulator: an observational study. Eur Radiol 2008; 18: 2874 –2878. Grantcharov TP, Reznick RK. Teaching procedural skills. Br Med J 2008; 336: 1129 –1131. Epstein RM, Hundert EM. Defining and assessing professional competence. JAMA 2002; 287:226 –235.
•
65
60. Miller GE. The assessment of clinical skills/competence/performance. Acad Med 1990; 65:S63–S67. 61. Tedesco MM, Pak JJ, Harris EJ Jr, Krummel TM, Dalman RL, Lee JT. Simulation-based endovascular skills assessment: the future of credentialing? J Vasc Surg 2008; 47:1008 –1001, discussion 1014. 62. Pandey VA, Wolfe JH, Liapis CD, Bergqvist D. The examination assessment of technical competence in vascular surgery. Br J Surg 2006; 93:1132– 1138. 63. Sidhu RS, Park J, Brydges R, MacRae HM, Dubrowski A. Laboratory-based vascular anastomosis training: a randomized controlled trial evaluating the effects of bench model fidelity and level of training on skill acquisition. J Vasc Surg 2007; 45:343–349. 64. Datta V, Bann S, Aggarwal R, Mandalia M, Hance J, Darzi A. Technical skills examination for general surgical trainees. Br J Surg 2006; 93:1139 –1146. 65. Beard JD, Choksy S, Khan S. Assessment of operative competence during carotid endarterectomy. Br J Surg 2007; 94:726 –730. 66. Brydges R, Sidhu R, Park J, Dubrowski A. Construct validity of computer-assisted assessment: quantification of movement processes during a vascular anastomosis on a live porcine model. Am J Surg 2007; 193:523–529. 67. Hislop SJ, Hsu JH, Narins CR, et al. Simulator assessment of innate endovascular aptitude versus empirically correct performance. J Vasc Surg 2006; 43:47–55. 68. Patel AD, Gallagher AG, Nicholson WJ, Cates CU. Learning curves and reliability measures for virtual reality simulation in the performance assessment of carotid angiography. J Am Coll Cardiol 2006; 47:1796 –1802. 69. Hsu JH, Younan D, Pandalai S, et al. Use of computer simulation for determining endovascular skill levels in a carotid stenting model. J Vasc Surg 2004; 40:1118 –1125. 70. Aggarwal R, Black SA, Hance JR, Darzi A, Cheshire NJ. Virtual reality simulation training can improve inexperienced surgeons’ endovascular skills. Eur J Vasc Endovasc Surg 2006; 31: 588 –593. 71. Kundhal PS, Grantcharov TP. Psychomotor performance measured in a virtual environment correlates with technical skills in the operating room. Surg Endosc 2008. 72. Bann SD, Khan MS, Darzi AW. Measurement of surgical dexterity using motion analysis of simple bench tasks. World J Surg 2003; 27:390 –394.
66
•
Simulation for Teaching and Assessing Endovascular Skills
73. Datta V, Mackay S, Mandalia M, Darzi A. The use of electromagnetic motion tracking analysis to objectively measure open surgical skill in the laboratory-based model. J Am Coll Surg 2001; 193:479 – 485. 74. Hance J, Aggarwal R, Stanbridge R, et al. Objective assessment of technical skills in cardiac surgery. Eur J Cardiothorac Surg 2005; 28:157–162. 75. Vleuten CPM. The assessment of professional competence: developments, research and practical implications. Adv Health Sci Education 1996; 1:41. 76. Berry M, Reznick R, Lystig T, Lonn L. The use of virtual reality for training in carotid artery stenting: a construct validation study. Acta Radiol 2008; 49: 801– 805. 77. Glaiberman CB, Jacobs B, Street M, Duncan JR, Scerbo MW, Pilgrim TK. Simulation in training: one-year experience using an efficiency index to assess interventional radiology fellow training status. J Vasc Interv Radiol 2008; 19:1366 –1371. 78. Van Herzeele I, Aggarwal R, Choong A, Brightwell R, Vermassen FE, Cheshire NJ. Virtual reality simulation objectively differentiates level of carotid stent experience in experienced inter-
79.
80.
81.
82. 83.
84.
ventionalists. J Vasc Surg 2007; 46: 855– 863. Patel AA, Gould DA. Simulators in interventional radiology training and evaluation: a paradigm shift is on the horizon. J Vasc Interv Radiol 2006; 17: S163–S173. Chaudhry A, Sutton C, Wood J, Stone R, McCloy R. Learning rate for laparoscopic surgical skills on MIST VR, a virtual reality simulator: quality of human-computer interface. Ann R Coll Surg Engl 1999; 81:281–286. Ahmed K, Ashrafian H, Hanna GB, Darzi A, Athanasiou T. Assessment of specialists in cardiovascular practice. Nat Rev Cardiol 2009; 659 – 667. Ahmed K, Nagpal K, Ashrafian H. Do we need to train assessors? Med Educ 2009; 43:389. Lehmann KS, Ritz JP, Maass H, et al. A prospective randomized study to test the transfer of basic psychomotor skills from virtual reality to physical reality in a comparable training setting. Ann Surg 2005; 241:442– 449. Hamilton EC, Scott DJ, Fleming JB, et al. Comparison of video trainer and virtual reality training systems on acquisition of laparoscopic skills. Surg Endosc 2002; 16:406 – 411.
January 2010
JVIR
85. Szalay D, MacRae H, Regehr G, Reznick R. Using operative outcome to assess technical skill. Am J Surg 2000; 180:234 –237. 86. Gould D, Patel A, Becker G, et al. SIR/RSNA/CIRSE Joint Medical Simulation Task Force strategic plan executive summary. J Vasc Interv Radiol 2007; 18:953–955. 87. Gould DA, Reekers JA, Kessel DO, et al. Simulation devices in interventional radiology: validation pending. J Vasc Interv Radiol 2006; 17:215– 216. 88. Dawson S. Procedural simulation: a primer. J Vasc Interv Radiol 2006; 17: 205–213. 89. Holdsworth RF. Objective assessment: the state of the art. Ann Royal Coll Surg Engl 1988; 70:266. 90. Van Herzeele I, Aggarwal R, Neequaye S, Darzi A, Vermassen F, Cheshire NJ. Cognitive training improves clinically relevant outcomes during simulated endovascular procedures. J Vasc Surg 2008; 1223–1230. 91. Berry M, Lystig T, Reznick R, Lonn L. Assessment of a virtual interventional simulator trainer. J Endovasc Ther 2006; 13:237–243.
