Interactive Learning Environments 2004, Vol. 12, No. 3, pp. 167-178
Learning, Testing, and the Evaluation of Learning Environments in Medicine: Global Perfonnance Assessment in Medical Education David Williamson Shaffer', James A. Gordon'^, and Nancy L. Bennett'' 'Department of Educational Psychology, University of Wisconsin-Madison, Madison, WI, USA, ^G.S. Beckwith Gilbert and Katharine S. Gilbert Medical Education, Program in Medical Simulation, Harvard Medical School, Department of Emergency Medicine, Massachusetts General Hospital, USA, and ^Department of Continuing Education, Harvard Medical School, USA
ABSTRACT Changes in the profession of medicine are creating the demand for a substantive reexamination of current practices in medical education. Many ofthe major issues in addressing this challenge are structural and political rather than scientific; here we address one critical scientific issue that will be important to (though not by itself adequate for) guiding such changes. The factor we address is the development of a global perfonnance assessment: a standardized mechanism to assess individual skills and abilities that can be used to evaluate altemative educational interventions. We discuss the issues involved in developing such a system, and describe a set of principles for defining desired outcomes and developing assessment tools, including (a) wide clinical scope, (b) direct relationship to actual clinical perfonnance, (c) reliability and repeatability, and (d) non-longitudinal measurement. We discuss the role of simulation, reflective practice, and portfolios of work in developing such a system, and argue that whatever form such metrics take, a system of global perfonnance assessment will develop from close collaboration between clinicians and educators with innovative ways of thinking about performance and expertise.
THE BEST WAY TO TRAIN A DOCTOR? In the last century, medicine has created a well-developed sequence of training activities designed to produce physicians who are technically competent and
Address correspondence to: David Williamson Shaffer, Ph.D., Educational Sciences Room 1065, 1025 West Johnson Street, University of Wisconsin-Madison, Madison, WI 53706, USA. E-mail:
[email protected] 1049-4820/04/1203-167$16.00 © Taylor & Francis Ltd.
168
DAVID WILLIAMSON SHAFFER ET AL.
compassionate caregivers. The system begins with science education in college, followed by preclinical and clinical training in medical school, continuing with postgraduate training, and merging into a lifetime of continuing medical education. Is this the best way to train a doctor? Surprisingly, we don't know. Perhaps because of the remarkable progress of medicine in the modem era, altemative educational paradigms have not been well-studied. We have not agreed on a standardized level of clinical perfonnance that describes a baseline from which we can compare the differences between parts of the educational continuum, nor do we know how to measure those differences. Tools exist that can be used to assess specific aspects of performance for individual chnicians (Cuschieri, 2000; Dupras & Li, 1995; S. Smith, Torkington, & Darzi, 1999; Sutnick et al., 1994; Swartz & Colliver, 1996), but as we discuss in more detail below, extant tools focus on specific skills sets, or on an individual's book knowledge rather than clinical practice and outcomes. The medical education system has not yet developed global assessment tools that can be used to evaluate the efficacy of elements of the general sequence of medical education (such as the utility of required premedical curricula, mandated intemships, or 8-year residencies) and their impact on the quality of practice over a career. Demands on the medical education system, which have been accelerating in recent years, create an opportunity to ask substantive questions about current practices of medical education. The process of educational reform suggests that we begin by articulating principles that will guide our efforts: principles by which we will define a set of desired outcomes and develop tools to determine the extent to which those outcomes have been achieved (Iglehart, 1998).
MEDICAL EDUCATION IN CRISIS? Medical education in the United States has faced significant cuts in funding in the last decade, most notably from the Balanced Budget Act of 1997, which lowered Medicare reimbursements and cut subsidies for training (Krakower, Williams, & Jones, 1999). Measures to contain rising health care costs by managed care organizations and insurance companies have put physicians under pressure to see more patients, leaving less time for teaching medical students, intems, and residents (Simon et al., 1999; Van Hom, Bums, &
GLOBAL PERFORMANCE ASSESSMENT
169
Wholey, 1997). The same pressures on cost and efficiency also increase the number of outpatient procedures and decrease the length of hospital stays. In some areas of clinical medicine, particularly in subspecialties, one of the most critical factors is that there simply are not enough contact time with patients to give adequate clinical experience to trainees at many hospitals (Collins & Harden, 1998; McGraw & O'Connor, 1999). These budgetary problems are exacerbated by the pace of innovation in medicine and by increasing concems about patient safety (Kohn, Corrigan, & Donaldson, 2000). Innovation takes place in nearly all fields, of course, but in recent years the pace of innovation in medicine has been particularly high. Literally, there is more to know than ever before, even as there are fewer contacts with patients to leam from and less time to teach.
PRIMUS INTER PARES Without minimizing the problems of funding, patient safety, or increasing specialization in medicine, from the point of view of educational reform another fundamental problem is that we have only a limited understanding of how to measure the performance that we are hoping to achieve through training. Specialty and subspecialty certification boards, working from the ACGME core competencies, assess candidates using a combination of written and oral exams (Schubert et al., 1999; Veloski, Rabinowitz, Robeson, & Young, 1999). Medical schools have developed comprehensive exams such as the Objective Standardized Clinical Examinations (Bradley & Humphris, 1999; Kaufman, Mann, Muijtjens, & van der Vleuten, 2000; Schwartz et aL, 1998; Sloan, Donnelly, Schwartz, & Strodel, 1995; Sloan et al., 1996). However, these tests are inherently limited in their ability to assess the full range of clinical performance, particularly in procedural medicine (Hodges, Regehr, McNaughton, Tiberius, & Hanson, 1999; Mavis, Henry, Ogle, & Hoppe, 1996; Searie, 2000). To ensure overall clinical competence we currently depend on subjective assessments by attending physicians and training mentors (Noel et al., 1992; Williams, Klamen, & McGaghie, 2003) - a process that makes sense in light of research that suggests that expertise does not consist of isolated skills and knowledge guided by abstract mles but rather is a holistic process whereby practitioners draw directly from their prior experiences in making clinical judgements (Dreyfus & Dreyfus, 1986; Shaffer et al., 2000; Shaffer, Kigin, Kaput, & Gazelle, 2002). These practices have
170
DAVID WILLIAMSON SHAFFER ET AL.
made it possible over the last century to produce good doctors by observing their performance during training. They are more problematic, however, as a means for ongoing assessment of clinical competence for physicians in practice - an issue that is receiving increased attention in reports that highlight concems about patient safety and the prevalence of morbidity and mortality due to clinical mistakes (Kohn et aL, 2000). However, certifying individual clinicians through all stages of their careers is not the only reason for constructing a comprehensive performance assessment. Another critical reason for evaluating a broad set of clinical skills is that without a standardized way to measure overall clinical proficiency both within and across individuals, there is no way to evaluate the current or altemative educational interventions (Ali et al., 1999; Braun, Koss, & Loeb, 1999; Schweinhart, 1999; F. Smith, Singleton, & Hilton, 1998). Medical education and the field of medicine more generally - lacks a standard, replicable set of metrics for determining how different individuals perform across the range of abilities needed for clinical practice: knowledge of medicine, analytic and diagnostic skills, perceptual and motor skills, interpersonal skills, and perhaps most important, clinical wisdom and judgment (Albanese, 1999; Shaffer, Meglan, Ferrell, & Dawson, 1999; Weingarten, Polliack, Tabenkin, & Kahan, 2000). Unless the medical education system develops a standardized means of evaluating clinical perfonnance, it will be impossible to compare the cost and efficacy of one type of training against another.
I MAY NOT KNOW ART... What would a comprehensive perfonnance assessment look like? The idea of assessing overall clinical perfonnance is hardly new, and it seems likely that comprehensive performance assessments would share many features in common with Objective Standard Clinical Examinations and other existing assessment tools that attempt to bridge knowledge with demonstrated skill (Bradley & Humphris, 1999; Kaufman et al., 2000; Schwartz et al., 1998; Sloan et al., 1995, 1996). In particular, comprehensive perfonnance assessments would almost certainly involve interaction with some kind of mock or simulated patients. To satisfy the goal of comparing altemative educational strategies, however, comprehensive performance assessments will have to have some additional features that will distinguish them from existing assessments of clinical competence.
GLOBAL PERFORMANCE ASSESSMENT
171
Features of a Comprehensive Performance Assessment Wide Scope
v
Assessments of perfonnance in narrow dimensions or for limited domains of medical practice have their uses, for example, in subspecialty certification, or in determining proficiency in a new diagnostic or therapeutic procedure (Morgan & Cleave-Hogg, 2000; Phillips & John, 2000; S. Smith et al., 1999). To evaluate altemative approaches to general medical education, however, assessments need to cover a broad array of skills and content areas. Clearly, it is possible to develop a training program to maximize performance on a small set of tasks, but an assessment that looks only at part of a clinician's skill set might favor educational interventions that ignore other important aspects of clinical ability. For example, an assessment that focuses on the technical aspects of surgical skill might favor educational environments that produce clinicians who are skilled technicians, but who do not relate well to patients. If we assume that a core set of objectives are essential, we would demand that a curriculum for medical education would help students and physicians make progress in all areas of expertise. Direct Relationship to Actual Clinical Performance It is possible to collect information on specific perceptual or motor skills, such as deviation from ideal path length of an instmment during a procedure, and to look for correlations to overall measures of expertise (Chaudhry, Sutton, Wood, Stone, & McCloy, 1999). However, such skills-based assessments mistake correlation for causality. Problems of constmct validity aside, use of proxy measures for perfonnance may have a place in evaluating how well an individual clinician performs in a well-defined training regime (Cohen, Colliver, Marcy, Fried, & Swartz, 1996; Martin et al., 1997). But the use of proxy measures seems likely to lead to inaccurate evaluations of new educational systems that use different teaching techniques, emphasize different aspects of clinical performance, or that organize the curriculum differently. An effective performance assessment for evaluating training regimes will have to be based on observable behaviors that clearly impact clinical expertise. Reliability and Repeatability Subjective assessments of perfonnance play an important role in the evaluation of a clinician's skills (Scheuneman, Carley, & Baker, 1994; Warf,
172
DAVID WILLIAMSON SHAFFER ET AL.
Donnelly, Schwartz, & Sloan, 1999), and it seems likely that some form of observation and expert rating will play a role in the development of any meaningful assessment of perfonnance in medicine (Kwolek et al., 1997). But in order to compare educational interventions that differ in significant ways, results from measures taken at different times, in different places, and under different circumstances will have to be comparable (Brennan, Braslow, Batcheller, & Kaye, 1996; Sykes, Ito, Fitzpatrick, & Ercikan, 1997). Clearly articulated mbrics will need to be developed to guide subjective assessments of performance. Non-Longitudinal Longitudinal studies of the cost, safety, and efficacy of care will likely be the gold standard for metrics of clinical perfonnance. But perfonnance metrics must assess performance in a matter of days or weeks rather than months or years. It is difficult to compare educational interventions when assessment of the outcome is the rate-limiting step in the process of innovation. Evaluation of Individuals Versus Programs Clearly actual clinical practice is the only tme measure of performance, and therefore the only tme metric on which training practices can be judged. However, in order to cover the range of competencies that we would need to assess in order to evaluate altemative approaches to general medical education, direct observation of clinical practice would be impractical. There will thus be a need for an intermediate step between low-fidelity, large-scale instmments such as currently exist, and the gold standard of longitudinal observation of physicians in practice: an instmment that is high-fidelity, but replicable and can assess differences in performance on a wide scope of clinical proficiency in a reasonable period of time. For obvious reasons, metrics designed to assess the perfonnance of every practicing clinician or trainee must be cost effective. But cost may be a secondary goal in developing a performance metric to assess the efficacy of training systems. Assessing altemative educational interventions would require evaluating a smaller number of clinicians who leam in experimental educational interventions, and to use the perfonnance of those clinicians to judge the relative merits of the intervention. While it would clearly be preferable to have inexpensive experiments to meet that goal, it may be that only relatively expensive assessments would fulfill the criteria outlined above.
GLOBAL PERFORMANCE ASSESSMENT
173
While performance metrics for assessing educational interventions may differ in significant ways from tools for clinician certification, they will share important components. Thus the development of comprehensive metrics for assessing altemative approaches to medical education would be a useful step towards developing tools for certifying clinicians through the stages of a clinical career. Such comprehensive assessment techniques may become, in effect, a second gold standard: less time-consuming and less costly than longitudinal studies, still too expensive to use for evaluating clinical ability on an ongoing basis, but useful as a means to develop and validate more streamlined instmments that can be used to evaluate clinical skill more widely. Some Usual Suspects Tools currently exist in the field of medical education that could be mobilized in the service of creating comprehensive performance assessments that would be effective in the evaluation of altemative approaches to medical education. Simulation Medicine is a field where the knowledge that counts is the knowledge that can be used in action. Simulations, both technological simulations and "simulated patients" played by actors, make it possible to produce standardized patients that clinicians interact with in a variety of ways, including performing complex diagnostic and therapeutic procedures (Bradley & Humphris, 1999; Kaufman et al., 2000; Schwartz et al., 1998; Shaffer et al., 1999; Sloan et al., 1995, 1996). Such simulations make it possible to design a wide range of repeatable clinical situations, and thus to design assessments based on direct observation of clinical performance. Reflection In medical practice, a first-year resident and an attending physician do similar things with similar patients, but their similar approaches reflect very different thinking as they assess the situation and decide on a course of action. We judge a clinician not just by what he or she does, but by how he or she understands a clinical situation, which is why oral boards are included as part of the certification process in many specialties and attending physicians ask probing questions of intems and residents (Clyman, Melnick, & Clauser, 1999). Any comprehensive assessment of performance certainly needs to ask clinicians to explain their thinking, perhaps through the presentation of patients, real or simulated (Jolly & Ho-Ping-Kong, 1991).
174
DAVID WILLIAMSON SHAFFER ET AL.
Portfolios of Work Practicing clinicians require a broad range of skills and abilities, developed through extended study and long apprenticeship. This is part of the reason that candidates for certification document their casework and present their logbooks for examination (Everitt, 1997; Sambrook, Smith, Hewitt, & Goss, 1996). It seems likely that any comprehensive assessment of clinical ability will not be based on a single performance of expertise, but rather on a portfolio of work on collection of different tasks done over time (Ram et al., 1999; White, Tuckey, & Crane, 1999). Image of a System Medical expertise requires the ability to mobilize knowledge in action, the ability to understand complex clinical situations, and a significant depth of knowledge and skill over a wide range of clinical and scientific areas. Tools already exist, including simulation, reflection, and portfolios of work, that address these issues. But unless we integrate these concepts in innovative ways, it seems unlikely that we will develop a system of comprehensive performance assessment that can guide medical education reform. There is not yet empirical evidence to support this claim. At this point in time, any discussion of how to fulfill the criteria outlined above for comprehensive performance assessment is necessarily speculative. However, it might be useful to describe what such a system could look like as an object for further discussion and experimentation. Extending the work of Shaffer et al. (2000), we might imagine that a system of comprehensive performance assessment could ask trainees to diagnose and treat a range of simulated patients (Gordon, Oriol, & Cooper, 2004). The simulator records a trainee's actions, which he or she annotates with reflections at the end of the case to create a logbook. Trainees present these cases, including their logbooks, to a panel of experts who observe performance from the case recordings, and further probe trainees' thinking with directed questions. Evaluations based on standardized guidelines for performance allow experts to interpret a trainee's understanding, but insure overall reliability of the assessment. While such a system uses current methods for evaluation of clinical skill, where attending physicians observe intems and residents treating and presenting patients on the wards over time, the advantage of using standardized, simulated patients and explicit mbrics for performance evaluation would be to produce repeatable, reliable assessments that could be used to compare the
GLOBAL PERFORMANCE ASSESSMENT
175
results of training done at different times and in different places using different teaching methods.
ONE SMALL STEP... One critical factor in guiding changes along the continuum of medical education will be the development of new metrics to evaluate clinical performance, designed to assess individual skills and abilities in a way that generalizes to the evaluation of educational programs. Whatever form such metrics take, to be useful in guiding the process of educational reform, they must cover a wide range of medical skills, have a direct relationship to clinical perfonnance, be reliable and repeatable, and produce results in a reasonable period of time. Such comprehensive performance metrics will develop from close collaboration between clinicians and educators with innovative ways of thinking about performance and expertise. If this collaborative work is going to succeed, we must acknowledge that the development of new performance metrics will be an important component of a constmctive response to the challenges facing medical education. We must set up design criteria such as those outlined above to guide our approach. And we must begin solid empirical work to develop innovative approaches for performance assessment, extending existing techniques in new and powerful ways.
REFERENCES Albanese, M. (1999). Rating educational quality: Factors in the erosion of professional standards. Academic Medicine, 74(6), 652-658. Ali, J., Adam, R.U., Josa, D., Pierre, L, Bedaysie, H., West, U., et al. (1999). Comparison of performance of intems completing the old (1993) and new interactive (1997) Advanced trauma life support courses. Joumai of Trauma, 4(5(1), 80-86. Bradley, P., & Humphris, G. (1999). Assessing the ability of medical students to apply evidence in practice: The potential ofthe OSCE. Medical Education, 55(11), 815-817. Braun, B.L, Koss, R.G., & Loeb, J.M. (1999). Integrating Perfonnance Measure Data into the Joint Commission Accreditation Process. Evaluation and the Health Professions, 22(3), 283-297. Brennan, R., Braslow, A., Batcheller, A., & Kaye, W. (1996). A reliable and valid method for evaluating cardiopulmonary resuscitation training outcomes. Resuscitation, 52(2), 85-93.
176
DAVID WILLIAMSON SHAFFER ET AL.
Chaudhry, A., Sutton, C, Wood, J., Stone, R., & McCloy, R. (1999). Learning rate for laparoscopic surgical skills on MIST VR, a virtual reality simulator: Quality of humancomputer interface. Annals of the Royal College of Surgeons of England, 8I{4), 281-286. Clyman, S.G., Melnick, D.E., & Clauser, B.E. (1999). Computer-based case simulations from medicine: Assessing skills in patient management. In W.C. McGahie (Ed.), Innovative simulations for assessing professional competence (pp. 29—41). Chicago: Department of Medical Education, University of Illinois at Chicago. Cohen, D., Colliver, J., Marcy, M., Fried, E., & Swartz, M. (1996). Psychometric properties of a standardized-patient checklist and rating-scale form used to assess interpersonal and communications skills. Academic Medicine, 71, S87-S89. Collins, J., & Harden, R. (1998). AMEE Medical Education Guide No. 13: Real patients, simulated patients and simulators in clinical examinations. Medical Teacher, 20, 508-521. Cuschieri, A. (2000). Human reliability assessment in surgery - a new approach for improving surgical performance and clinical outcome. Annals ofthe Royal College of Surgeons of England, 82(2), 83-87. Dreyfus, H.L., & Dreyfus, S.E. (1986). Mind over machine: The power of human intuition and expertise in the era ofthe computer. New York: Free Press. Dupras, D., & Li, J. (1995). Use of an objective structured clinical examination to determine clinical competence./^cflrfem/c Merf/cme, 70(11), 1029-1034. Everitt, N. (1997). An altemative database for the computerized surgical logbook. Annals ofthe Royal College of Surgeons of England, 79(6 Suppl.), 249-253. Gordon, J.A., Oriol, N.E., & Cooper, J.B. (2004). Brining good teaching cases "to life": A simulator-based medical education service. Academic Medicine, 79, 23-27. Hodges, B., Regehr, G., McNaughton, N., Tiberius, R., & Hanson, M. (1999). OSCE checklists do not capture increasing levels of expertise. Academic Medicine, 10(14), 1129-1134. Iglehart, J. (1998). Forum on the future of academic medicine: Session III - getting from here to there. Academic Medicine, 73(2), 146-151. Jolly, B., & Ho-Ping-Kong, H. (1991). Independent learning: An exploration of student grand rounds. Medical Education, 25(4), 334-342. Kaufman, D.M., Mann, K.V., Muijtjens, A.M., & van der Vleuten, C.R (2000). A comparison of standard-setting procedures for an OSCE in undergraduate medical education. Academic Medicine, 75(3), 267-211. Kohn, L.T., Corrigan, J.M., & Donaldson, M.S. (Eds.). (2000). To err is human: Building a safer health system. Washington, DC: National Academy Press. Krakower, J.Y., Williams, D.J., & Jones, R.F (1999). Review of US medical school finances, 1997-1998. JAMA, 282, 847-854. Kwolek, C, Donnelly, M., Sloan, D., Birreli, S., Strodel, W., & Schwartz, R. (1997). Ward evaluations: Should they be abandoned? Journal of Surgical Research, 69(1), 1-6. Martin, J.A., Regehr, G., Reznick, R., MaeRae, H., Murnaghan, J., Hutchison, C, et al. (1997). Objective structured assessment of technical skill (OS ATS) for surgical residents. British Joumai of Surgery, 84(2), 273-278. Mavis, B., Henry, R., Ogle, K., & Hoppe, R. (1996). The emperor's new clothes: The OSCE reassessed. Academic Medicine, 71(5), 447^53. McGraw, R., & O'Connor, H. (1999). Standardized patients in the early acquisition of clinical skills. Medical Education, 55(8), 572-578.
GLOBAL PERFORMANCE ASSESSMENT
177
Morgan, P., & Cleave-Hogg, D. (2000). Evaluation of medical students' performance using the anesthesia simulator. Medical Education, 34(\). Noel, G., Herbers, J., Caplow, M., Cooper, G., Pangaro, L., & Harvey, J. (1992). How well do internal medicine faculty members evaluate the clinical skills of residents? Annals of Internal Medicine, 117, 757-765. Phillips, N., & John, N. (2000). Web-based surgical simulation for ventricular catheterization. Neurosurgery, 46(A), 933-936; discussion 936-937. Ram, P., Grol, R., Rethans, J., Schouten, B., Vleuten, C , & Kester, A. (1999). Assessment of general practitioners by video observation of communicative and medical performance in daily practice: Issues of validity, reliability and feasibility. Medical Education, 55(6), 447-454. Sambrook, P., Smith, A., Hewitt, G., & Goss, A.N. (1996). Development of a national clinical logbook system for oral and maxillofacial surgery trainees - the RACDS, OMS system. International Joumai of Maxillofacial Surgery, 25(3), 242-244. Scheuneman, A., Carley, J., & Baker, W. (1994). Residency evaluations: Are they worth the tmm Archives of Surgery, 729(10), 1067-1073. Schubert, A., Tetzlaff, J., Tan, M., Barnes, A., Ryckman, J., & Mascha, E. (1999). Consistency, inter-rater reliability, and validity of 441 consecutive mock oral examinations in anesthesiology: Implications for use as a tool for assessment of residents. Anesthesiology, 91 (\),2S&-29S. Schwartz, R., Witzke, D., Donnelly, M., Stratton, T, Blue, A., & Sloan, D. (1998). Assessing residents' clinical performance: Cumulative results of a four-year study with the Objective Structured Clinical Examination. Surgery, 124(2), 307-312. Schweinhart, L. (1999). Evaluation of early childhood programs. Child and Adolescent Psychiatric Clinics ofNorth America, 8(2), 395^07, viii-ix. Searie, J. (2000). Defining competency - the role of standard setting. Medical Education, 34(5), 363-366. Shaffer, D.W., Dawson, S., Meglan, D., Cotin, S., Ferrell, M., Norbash, A., et al. (2000). Design principles for the use of simulation as an aid in interventional cardiology training. Minimally Invasive Therapy and Applied Technologies, 10(2). Shaffer, D.W., Kigin, C , Kaput, J., & Gazelle, G. (2002). What is digital medicine? In R. Bushko (Ed.), Future of health technology: IOS Press. Shaffer, D.W., Meglan, D., Ferrell, M., & Dawson, S. (1999). Virtual rounds: Simulation-based education in procedural medicine. In H. Pien (Ed.), Proceedings of SPIE Vol. 3712: Battlefield Biomedical Technologies. Simon, S., Sullivan, A., Pan, R., Connelly, M., Peters, A., Clark-Chiarelli, N., et al. (1999). Views of managed care - a survey of students, residents, faculty, and deans at medical schools in the United States. New England Joumai of Medicine, 540(12), 928-936. Sloan, D., Donnelly, M., Schwartz, R., Felts, J., Blue, A., & Strodel, W. (1996). The use of objective structured clinical examination (OSCE) for evaluation and instruction in graduate medical education. Joumai of Surgical Research, (55(1), 225-230. Sloan, D., Donnelly, M., Schwartz, R., & Strodel, W. (1995). The Objective Structured Clinical Examination: The new gold standard for evaluating postgraduate clinical performance. Annals of Surgery, 222(6), 735-742. Smith, F., Singleton, A., & Hilton, S. (1998). General practitioners' continuing education: A review of policies, strategies and effectiveness, and their implications for the future. British Joumai of General Practice, 48(435), 1689-1695.
17 8
DAVID WILLIAMSON SHAFFER ET AL.
Smith, S., Torkington, J., & Darzi, A. (1999). Objective assessment of surgical dexterity using simulators. Hospital Medicine, 60(9), 672-675. Sutnick, A., Stillman, R, Norcini, J., Friedman, M., Williams, R., Trace, S.M., et al. (1994). Pilot study of the use of the ECFMG clinical competence assessment to provide profiles of clinical competencies of graduates of foreign medical schools for residency directors. Academic Medicine, 69(1), 65-67. Swartz, M., & Colliver, J. (1996). Using standardized patients for assessing clinical performance: An overview. Mount Sinai Joumai of Medicine, 55(3-4), 241-249. Sykes, R.S., Ito, K., Fitzpatrick, A.R., & Ercikan, K. (1997). Review of technical issues in large-scale perfonnance assessment. Joumai of Educational Measurement, 4, 379-384. Van Hom, R.L., Bums, L.R., & Wholey, D.R. (1997). The impact of physician involvement in managed care on efficient use of hospital resources. Medical Care, 55(9), 873-889. Veloski, J., Rabinowitz, H., Robeson, M., & Young, P. (1999). Patients don't present with five choices: An altemative to multiple choice tests in assessing physicians' competence. Academic Medicine, 74(5), 539. Warf, B., Donnelly, M., Schwartz, R., & Sloan, D. (1999). Interpreting the judgment of surgical faculty regarding resident competence. Joumai of Surgical Research, 86(\), 29-35. Weingarten, M., Polliack, M., Tabenkin, H., & Kahan, E. (2000). Variations among examiners in family medicine residency board oral examinations. Medical Education, 34(1), 13-17. White, A., Tuckey, J., & Crane, S. (1999). Education and training module for a specialist registrar: A move forward in specialist registrar education. British Joumai of Family Planning, 25(2), 77-80. Williams, R.G., Klamen, D.A., & McGaghie, W.C. (2003). Cognitive, social and environmental sources of bias in clinical perfonnance ratings. Teaching and Leaming in Medicine, 15(4), 270-292.
