at raising the general level of performance. Practitioners deemed to be at ... Madrid, Spain, 4Evaluation Bureau, Medical Council of Canada,. Ottawa, Canada ...
Papers from the 10th Cambridge Conference
The relationship between competence and performance: implications for assessing practice performance J-J Rethans,1 J J Norcini,2 M Baro´n-Maldonado,3 D Blackmore,4 B C Jolly,5 T LaDuca,6 S Lew,7 G G Page8 & L H Southgate9
Objective This paper aims to describe current views of the relationship between competence and performance and to delineate some of the implications of the distinctions between the two areas for the purpose of assessing doctors in practice.
at raising the general level of performance. Practitioners deemed to be at risk would undergo a more detailed assessment process focused on rigorous testing, with poor performers targeted for remediation or removal from practice.
Methods During a 2-day closed session, the authors, using their wide experiences in this domain, defined the problem and the context, discussed the content and set up a new model. This was developed further by e-mail correspondence over a 6-month period.
Conclusion We propose a new model, designated the Cambridge Model, which extends and refines Miller’s pyramid. It inverts his pyramid, focuses exclusively on the top two tiers, and identifies performance as a product of competence, the influences of the individual (e.g. health, relationships), and the influences of the system (e.g. facilities, practice time). The model provides a basis for understanding and designing assessments of practice performance.
Results Competency-based assessments were defined as measures of what doctors do in testing situations, while performance-based assessments were defined as measures of what doctors do in practice. The distinction between competency-based and performance-based methods leads to a three-stage model for assessing doctors in practice. The first component of the model proposed is a screening test that would identify doctors at risk. Practitioners who ‘pass’ the screen would move on to a continuous quality improvement process aimed
Introduction Most medical students start their careers as qualified doctors after successfully completing the final high stakes examinations. Traditionally, doctors have been 1
Skillslab, Maastricht University, The Netherlands, 2Psychometrics and Research, American Board of Internal Medicine, Philadelphia, Pennsylvania, USA, 3Faculty of Medicine, University of Alcala´, Madrid, Spain, 4Evaluation Bureau, Medical Council of Canada, Ottawa, Canada, 5Centre for Medical and Health Sciences Education, Monash University, Victoria, Australia, 6National Board of Medical Examiners, Philadelphia, Pennsylvania, USA, 7Royal Australian College of General Practitioners, Melbourne, Australia, 8Faculty of Medicine, Division of Educational Support and Development, College of Health Disciplines, University of British Columbia, Vancouver, Canada, 9Centre for Health Informatics and MultiProfessional Education, Medical School, University College London, UK Correspondence: Jan-Joost Rethans, MD PhD, Skillslab, Maastricht University, PO Box 616, 6200 MD Maastricht, The Netherlands. Tel.: 00 31Æ43Æ388Æ1790; Fax: 00 31Æ43Æ388 4127; E-mail: j.rethans@ sk.unimaas.nl
Keywords clinical competence ⁄ *standards; physicians, family ⁄ *standards; education, medical, continuing ⁄ *standards; quality of health care ⁄ standards. Medical Education 2002;36:901–909
regarded as competent enough to start working with patients immediately. Furthermore, it has been assumed that they would remain competent throughout their professional careers by taking postgraduate courses and that this, together with working in actual practice, would provide sufficient opportunities and evidence to ensure that they remained fit to practise. However, there are or at least two reasons why this view does not reflect the reality for medical practitioners today. Firstly, several studies have shown that there are differences between what doctors can do in controlled high stakes situations and what they really do in actual practice.1–5 Moreover, the relationship between demonstrating competency in examinations and behaviour in actual practice appears at the least to be problematic. Studies have shown both high and moderate as well as very low () 0Æ04) correlations between results of doctors performing examinations during tests and in actual practice.1–5 One of the problems with these
Blackwell Science Ltd ME D I C A L ED U C A T I ON 2002;36:901–909
901
902
Assessing practice performance
•
J-J Rethans et al.
• contain clear definitions; • take into account current knowledge about assessment; • be an international model, more or less independent of any national setting, and • promote the assessment of an individual medical professional.
Key learning points Competency-based assessment measures what doctors can do in controlled representations of professional practice. Performance-based assessment measures what doctors do in their professional practice. Assessing doctors in practice should best be done using a three-stage model. The staged model consists of: • a screening phase for all doctors, focusing on real practice; • a continuous quality improvement phase for those who pass the screen, and
The approach to the model, how to use it for screening, the issue of poor performers, the implications of this model and its strengths and weaknesses form the basis for the rest of the paper. Implementation is not part of this paper.
Definitions In this paper we will use the following definitions:
• a detailed assessment process for those ‘at risk’.
conflicting results is that authors do not make use of the same implicit or explicit definitions of both test conditions and methods used. The situations under which candidates are investigated and the descriptions of these situations are not uniform: competence (behaviour), competency, test behaviour, professional behaviour (actual) performance and test performance are only some of them. In addition, some studies refer to direct observations of doctors while others use opinions of peers and compare these. Yet these studies claim to describe the same results. We urgently need to establish a common terminology before we can make it possible to determine the implications which these results have for quality assurance or licensure programmes for medical professionals. Secondly, it is now known that merely undertaking postgraduate courses throughout a professional career, even if done from personal initiative, is not enough to remain working as a ‘competent’ doctor.6 This is not only because of the rapid changes in medical information technology. In addition, the growth of government and patient influences (satisfaction rates, legal rights, etc.) reflected in the quality improvement literature shows that additional factors direct the behaviour of a medical professional. This is reflected in the content of different national and international medical standards or guidelines programmes. The requirements of patients, society and peers make a professional career, and its assessment, much more complex now than it has been in the past. In this paper we propose a model for the assessment of performance and competence in actual practice with the following characteristics. It will:
• competency-based assessment measures what doctors can do in controlled representations of professional practice; • performance-based assessment measures what doctors do in actual professional practice. Although at first glance these definitions may seem wide open, their implications are not. Many research papers on assessment refer to ‘performance-based assessment’ and then describe candidates taking objective structured clinical examinations (OSCEs) and clinical physical examinations (CPXs). We propose that all assessments under examination-like settings should now be referred to as competence-based assessments, whereas assessments in actual practice should be referred to as performance-based assessments. Each setting has its own dynamics. Current knowledge about the test practice in both settings is shown in Table 1. In general, competence assessment is easier to administer. However, the perspectives of patients and society demand that doctors should meet the assessment standards in their working conditions in any given situation. In future, the emphasis should lie on the assessment of performance. A model for assessing practising doctors
Figure 1 depicts a model for assessing practising doctors. The model is founded upon an up-to-date understanding of what is known about assessing both clinical competence and clinical performance. The model is intended to be applicable across medical specialties and across international boundaries. It is sensitive to the need to be fair to the profession and its members, yet
Blackwell Science Ltd ME D I C AL ED U C AT I ON 2002;36:901–909
Assessing practice performance
•
J-J Rethans et al.
903
Table 1 Current knowledge of practices for competence and performance assessment Competence
Performance
Available technology Reasonable costs Traditional methods Measurement qualities (theory) known ‘Experimental’ context Logistically feasible Quantitative Professionally-based What can we assess?
Technology evolving High costs Traditional methods rejected Subjectivity versus objectivity Naturalistic context Logistically difficult Descriptive ⁄ qualitative Involves patients What else should we assess that we can’t learn from competency assessment?
Figure 1 Outline of the authors’ approach to performance assessment.
responsive to the public’s need to be assured that doctors’ performances are effective. We propose to apply it in a 5-year cycle, preferably dovetailing it with current national standards and using contemporary knowledge on psychometrics.
The model for assessing practising doctors has three components. These are: 1 a general screening component in which all practitioners will participate, followed by either
Blackwell Science Ltd ME D I C A L ED U C A T I ON 2002;36:901–909
904
Assessing practice performance
•
J-J Rethans et al.
2 a continuous quality improvement component for practitioners who perform well on the screening component, or 3 a rigorous diagnostic investigation and follow-up component for practitioners who perform poorly on the general screen. The context of the model
Most existing arrangements for the monitoring of individual practitioners involve either or both of two contrasting approaches.7 In each case, formal mechanisms are created for inspection of the practice performance of doctors. But there are numerous and significant differences. For example, the approach favoured in the US, arising from its tradition of entrepreneurial medicine, focuses on identification of the (allegedly few) poor performers and seeks to apply sanctions in the form of restrictions to the license to practice, including revocation.8 Elsewhere, notably in Canada, equally formal mechanisms are imposed, but the intention is described as continuous promotion of quality among all practitioners.9 In the UK, mainly because of recent health care alarms and public concern over the quality of practising doctors, both systems will be in operation from 2003.10 Whatever the context, there are indications of increasing support for implementation of systems of monitoring doctors’ practice performance.11 Such systems are now operational.12 Accordingly, this present climate affords an opportunity to consider the most desirable features of any system of practice performance assessment. The screening component
The arguments made here reflect several guiding principles. In a complex social and professional
environment, efficient use of resources is imperative. This must translate into a mechanism that meets the real challenge of balancing cost with effective measurement of performance. Hence the adoption of a screening philosophy that rigorously meets these constraints. An appropriate approach is to consider the task as a screening plus further investigation strategy. If relatively quick and efficient screening procedures can be developed, then only a limited number of potentially ‘at risk’ doctors need be subject to further, more detailed scrutiny. One way of achieving this might be by screening and scrutinising routinely collected data on items such as those outlined in Table 2 (e.g. patient reviews for communication skills, health checks for fitness to practise, prescribing behaviour and referral letters for appropriate management).13 These screens could be spread over 3–5 years so that a snapshot of each area is taken, thereby reducing the need for large scale screens every 5 years. Some of this might even be managed on a self-assessment or local peer assessment basis, with appropriate quality assurance devices (sampling) included. Data on the predictive validity of these screens could then be collected. Of course, it may be that quick and effective screening for behaviour as complex as practice performance just does not exist. Hence, screening should assess broadly, but with priority on practice performance. In epidemiological terms, the sensitivity of the first phase must be high. It would be more acceptable to have doctors wrongly classified as being at risk in the first phase, and then reclassified, after the next assessment stages, as good performing colleagues, than to have them classified as good performers who, in the end, are found to be deficient. Finally, the screening assessment must be consistent with, and possess sufficient rigour to correctly classify doctors for, components 2 and 3 described above.
Table 2 Potential screening methods and elements of performance assessment Methods
Elements
Peer review
Technical skills; team relationships professional behaviour; ethics; cognitive skills; patient management Communications; professional behaviour; impairment; satisfaction Technical competence; resource use; status of privileges Active participation in CME; professional behaviour
Patient review Performance appraisal Professional development ⁄ continuing medical education (CME) Compulsory health checks Chart audits or competence assessments Aggregate measures (e.g. patient immunisation rates) Prescribing behaviour Referral information
Physical fitness; cognitive function; psychological impairment Use of computerized records; decision-making; prescribing patterns; insight Standing in context of practice standards Knowledge currency; clinical decisions; patient management; impairment Patient management; decision-making
Blackwell Science Ltd ME D I C AL ED U C AT I ON 2002;36:901–909
Assessing practice performance
•
J-J Rethans et al.
Assumptions of screening assessment
Before we describe the specific components of an effective screening assessment, we should also define four complementary resources that we believe must be made available. These include: 1 a doctors’ health programme; 2 definitions of scope or standards of practice; 3 guidelines and standards for the profession, and 4 baseline information. Potential screening methods
While the premise of any screening is proactive in orientation, the process must be acceptable to the medical community and credible to the public. We acknowledge the importance of conventional indications of possible practice problems, such as patient complaints and actions following hospital review. Reactive measures In most countries, patient complaints constitute a very large volume of information and they characterise much of the communication between ordinary citizens and licensing bodies. A small fraction of these complaints are deemed worthy of intensive investigation. Although they have risen in number in recent years, it remains true that only a very small number of patient complaints (including malpractice suits) result in adverse judgements or disciplinary action.14 Performance reviews by hospital oversight committees may result in modification of privileges, or (less often) their termination. In US jurisdictions such actions are reportable to the licensing authority, which may undertake independent investigation as well. However, the credibility of a complaints system must remain low because it is neither truly peerreferenced nor self-regulating. Proactive measures A portfolio of proactive methods should represent the centrepiece of any proposed performance screening assessment. This is because of the professional value that imposes the obligation on doctors to be knowledgeable and proficient throughout their careers. There is a large array of possible screening methods and they vary in feasibility, utility and cost. In the interests of completeness we have included all of them here, irrespective of their potential for contributing meaningfully to the screening process. Care should be taken, however, that ‘statements of competence’ are applicable to real practice. The major elements of the initial screening process should become a routine part of a doctor’s professional life, consisting
905
of regular monitoring of performance (e.g. via patient satisfaction reviews, referral rate comparison groups) and more formal periodic assessments (e.g. via chart reviews, peer reviews). While the screening component is intended to focus on the assessment of performance, it could also include some low cost, easily administered competency assessments (e.g. written examinations). Similarly, while Table 2 includes important, possibly generic elements of professional performance, other elements of performance should not be excluded. For example, the important domain of clinical judgement may not be assessed adequately by those methods listed. Given the greater challenge of obtaining reliable performance assessments, competency assessments would add ‘reliability’ to the assessment process. However, this screening phase should ultimately emphasize performance assessments. To ensure rigour, competency assessments should be administered under controlled conditions. The screening component assumes that programmes to assess doctors’ health are already in place. If they are not, they should be encompassed in the screening process prior to periodic performance or competency assessments so as to rule out health-related causes of poor performance. Continuous quality improvement
The second component of the model focuses on learning, as its thrust is one of continuous quality improvement (CQI) for practitioners who perform well on the screening component. It is expected that over 90% of all practitioners would be directed to this component of the model from the screening component. This CQI component would build upon the results of the assessment strategies comprising the screening component of the model, and would be largely self-guided by the practitioner. This adds another argument in support of the proposal that the first screening phase should primarily be based in and on real practice: doctors will view the results of their performance data as ‘of their own’. This sense of personal identification with the subject can make an important contribution towards stimulating change. In this second component, practitioners could employ a reflective learning model to construct their learning around their screening assessment results and their individual practices. Any assessment activity built into this component need only be of minimal rigour, its purpose being to guide the practitioners to areas of strength and weakness in their own performances. Although CQI depends largely on the doctor’s selfguidance, it will be apparent if doctors remain in this
Blackwell Science Ltd ME D I C A L ED U C A T I ON 2002;36:901–909
906
Assessing practice performance
•
J-J Rethans et al.
component or transfer to other components because the initial screening phase is to be repeated periodically. Rigorous performance assessment
The purpose of the third component of the model is to obtain a detailed assessment of a practitioner who performed poorly on the screening as a basis for decisions on remediation, rehabilitation or exit from the profession. It is expected that less than 10% of practitioners will be directed to this component. Given the high stakes involved for both the practitioner and society, the assessment process in this component must be of the highest rigour. The standards applied in this component should preferably be at least national ones and be approved by bodies and ⁄ or boards of the profession(s) involved. This third component also involves the major task of seeking out doctors who are still remediable. Once again, the focus here is on assessing performance. However, this will be complemented by an equally rigorous assessment of competence, which will serve as a mechanism for understanding poor performance, and for developing remedial strategies for correcting poor performance. Whereas component 2 of the model was largely self-directed by the practitioner, component 3 will be directed by a professional or government body responsible for the quality of practitioners’ performance. We propose a staged assessment also in this phase, to be first performance-based and then competencybased. In order to understand why doctors perform poorly, we need to ask ‘why’ and ‘how come’ questions so that we can identify factors or reasons behind poor performance such as impairment, behavioural and health-related issues. These questions should be based on the outcomes of the first screening phase. The answers should direct the examiners firstly to explanations for poor performance (such as current ⁄ temporary ⁄ difficult personal circumstances) and secondly to specific assessment areas where candidates show inadequate results. At the end of this phase, more traditional competency-based assessment strategies may be used, such as detailed knowledge tests, standardised patient examinations of clinical and practical skills, structured oral tests, tests of cognitive abilities, direct observation of doctor–patient contacts and so on. This approach has been adopted by the UK General Medical Council (GMC) and has been implemented for the entire profession.10 The introduction of the GMC’s Performance Procedures in July 1997 has enabled the identification of doctors whose performance is seriously deficient, calling their registration into
question. This completes the GMC’s facilities for dealing with dysfunctional doctors, which were previously limited to conduct or health-related matters. The performance of doctors who enter the procedures is assessed in two parts, comprising peer review of practice to assess performance in the workplace and tests of competence to assess knowledge and skills basic to that performance. Equivalent assessments for peer review for all registered doctors have been devised and piloted. The competence assessment always includes a written examination and at least one practical examination, but the written examination may be an objective (multiple-choice) test, a short-answer test, or some form of essay test. The practical examination(s) may involve interactions with standardised patients, clinical models, or other forms of simulations. Trained nonmedical assessors recruited by open advertisement in the national press take part in both phases alongside the medical assessors. By spring 2001, 230 doctors had entered the programme. General practitioners are represented in proportion to their numbers in the profession, and other disciplines that have been assessed include surgeons, emergency doctors, anaesthetists, psychiatrists, paediatricians and radiologists.
The Cambridge Model for Performance and Competence Miller proposed an assessment model with four stages or levels, designated ‘knows’, ‘knows how’, ‘shows how’ and ‘does’, as shown in Fig. 2.15 Miller refers to ‘shows how’ as being an assessment of performance. However, as argued earlier in this paper ‘shows how’ nowadays should be referred to as competency-based assessment rather than as performancebased assessment. Miller’s triangle implicitly assumes that competence predicts performance. However, the exact relationship between competence and performance is complicated.1–5,10 Factors such as time pressure, day of the week, mood of the patient and doctor and impact of the fore-going examination influence clinical performance just as individual deficient competence on specific domain-related knowledge areas does. Miller’s figure is very useful in educational settings, especially for setting up medical curricula or courses for students. The four stages can easily be used to build an educational programme that begins with the assimilation of pure knowledge (reading books and articles) and progresses through the acquiral of clinical skills to development of real performance in practice. However, the model is no longer very helpful in terms of assessment in real practice.
Blackwell Science Ltd ME D I C AL ED U C AT I ON 2002;36:901–909
Assessing practice performance
•
J-J Rethans et al.
907
Figure 2 Miller’s Triangle15.
Another drawback of Miller’s figure is that it fails to account for the influences of other factors on clinical performance. While these influences are numerous, they may be arbitrarily classified as either systemrelated or individual-related. System-related influences include government programmes and initiatives, patient expectations, guidelines or policies developed by the practice facility, time, and accessibility to other health professionals. Individual-related influences include the physical and mental health of the doctor, their state of mind at the time of the performance assessment, and their relationships with others, including patients, peers (including other health professionals) and their own family. Miller’s triangle is a static figure, whereas contemporary assessment demands a more flexible figure. It is vitally important that our staged assessment programme allows us to view the performance of an individual doctor from different angles. These different angles will allow, for example, fuller answering of the ‘why’ questions submitted to the assessee.
To illustrate this and the interaction between these systemic and individual influences and competence and performance, we propose a modification to Miller’s triangle (Fig. 3). This effectively inverts Miller’s model. Clearly, competence is an important prerequisite for performance and is therefore represented in the main triangle in the centre. Alternatively, this may be conceptualised as the beam of light which competence sheds on performance, but which does not of itself illuminate the whole picture. We propose at least two further triangles, or shafts of light, to illuminate the influences of the system-related and individual-related factors that should also considered when assessing the performance of a doctor. In fact, it is very likely that more shafts of light are necessary to wholly illumine the performance of an individual doctor. The corollary is that not all the problems related to doctor performance will be explained by competence alone. Because of this, it may be more cost-effective in the first instance to examine other systemic or individual influences before conducting rigorous assessment of doctor competence.
Figure 3 The Cambridge Model for delineating performance and competence.
Blackwell Science Ltd ME D I C A L ED U C A T I ON 2002;36:901–909
908
Assessing practice performance
•
J-J Rethans et al.
Conclusion
Acknowledgements
The purpose of this paper was to describe current views of the relationship between competence and performance and to delineate some of the implications of this relationship for assessing doctors in practice. Competency-based assessments were defined as measures of what doctors do in testing situations, while performance-based assessments were defined as measures of what doctors do in practice. We argued that tests of competence are feasible and efficient, and produce results with known, valid properties. In contrast, methods of assessing performance are just evolving but hold greater promise of meeting the needs of patients and society by describing the actual quality of care delivered by doctors. The distinction between competency-based and performance-based methods leads naturally to a model for assessing doctors in practice. The first component of the model we propose is a screening test, a relatively quick and efficient procedure that would identify doctors at risk. Practitioners who ‘pass’ the screen would move on to a continuous quality improvement process aimed at raising general levels of performance. Practitioners at risk of poor performance would undergo a more detailed assessment process focused on rigorous testing, with poor performers targeted for remediation or removal from practice. Finally, the competence ⁄ performance distinction traces at least part of its heritage to Miller’s model, which divided methods according to whether the examinee is required to know, know how, show how or do. We propose the Cambridge Model, which extends and refines Miller’s work. It inverts his pyramid, focuses exclusively on the top two tiers, and identifies performance as a product of competence combined with the influences of factors related to the individual (e.g. health, relationships) and factors related to the system (e.g. facilities, practice time). The model provides a basis for understanding and designing assessments of practice performance.
Grateful acknowledgement is made to the sponsors of the 10th Cambridge Conference, the Medical Council of Canada, the Smith & Nephew Foundation, the American Board of Internal Medicine, the National Board of Medical Examiners and the Royal College of Physicians.
Contributors All authors contributed equally to the discussions leading to this paper, which were undertaken during the 10th Cambridge Conference. J-JR and JJN undertook main responsibility for preparing the draft, co-ordinating input from the other authors and writing the final version of the paper.
Funding There was no external funding for this study.
References 1 Rethans JJ, Sturmans F, Drop R, van der Vleuten C, Hobus P. Does competence of general practitioners predict their performance? Comparison between examination setting and actual practice. BMJ 1991;303:1377–80. 2 Ramsey PG, Wenrich M, Carline JD, Inui TS, Larson EB, Logerto JP. Use of peer ratings to evaluate physician performance. JAMA 1993;269:1655–60. 3 Ram P, van der Vleuten C, Rethans JJ, Schouten B, Hobma S, Grol R. Assessment in general practice: the predictive value of written-knowledge tests and a multiple-station examination for actual medical performance in daily practice. Med Educ 1999;33:197–203. 4 Southgate L, Campbell M, Cox J, Foulkes J, Jolly B, McCrorie P et al. The General Medical Council’s performance procedures: the development and implementation of tests of competence with examples from general practice. Med Educ 2001;35:20–8. 5 Page GG, Fielding DW. Performance on PMPs and performance in practice: are they related? J Med Educ 1980;55:529–37. 6 Davis DA, Thompson MA, Oxman AD, Haynes RB. Changing physician performance. A systematic review of the effects of continuing medical education strategies. JAMA 1995;274:700–5. 7 Dauphinee D, Fabb W, Jolly B, Lansley D, Wealthall S, Procopis P. Determining the content of certifying examinations. In. Newble DI, Jolly BC, Wakeford RE, eds. Certification and Recertification in Medicine: Issues in the Assessment of Clinical Competence. Cambridge: Cambridge University Press; 1994. 8 Norcini JJ. Recertification in the United States. BMJ 1999;319:1183–5. 9 Dauphinee WD. Revalidation of doctors in Canada. BMJ 1999;319:1188–90. 10 Southgate LH, Pringle M. Revalidation in the United Kingdom: general principles based on experience in general practice. BMJ 1999;319:1180–3. 11 Southgate LH, Cox J, David T, Hatch D, Howes A, Johnson N et al. The assessment of poorly performing doctors: the development of the assessment programmes for the General
Blackwell Science Ltd ME D I C AL ED U C AT I ON 2002;36:901–909
Assessing practice performance
•
J-J Rethans et al.
Medical Council’s Performance Procedures. Med Educ 2001;35 (Suppl.):2–8. 12 Cunningham JPW, Hanna E, Turnbull J, Kaigas T, Norman GR. Defensible assessment of the competency of the practising physician. Acad Med 1997;72:9–12. 13 Davies HA, Crossley J, Howe A, Jolly BC, Newble DI. Performance assessment using outpatient letters: the Sheffield assessment instrument for letters. Med Educ 2001;35:1115– 24.
909
14 Page GG, Bates J, Dyer SM et al. Physician-assessment and physician enhancement programmes in Canada. J Can Med Assoc 1995;153:1723–8. 15 Miller GE. The assessment of clinical skills ⁄ competence ⁄ performance. Acad Med 1990;65 (Suppl.):S63–S67. Received 20 March 2002; editorial comments to authors 13 June 2002; accepted for publication 17 June 2002
Blackwell Science Ltd ME D I C A L ED U C A T I ON 2002;36:901–909
