examiner: what you need to know and do. Richard Hays and Lindsay Bashford, School of Medicine, Keele University,. Newcastle-under-Lyme, Staffordshire, UK.
Assessment
Being an external examiner: what you need to know and do Richard Hays and Lindsay Bashford, School of Medicine, Keele University, Newcastle-under-Lyme, Staffordshire, UK
It is important to minimise ‘preventable’ error
BACKGROUND
T
he assessment of learners at all levels of medical education takes place in a complex context. There are many variables: learners, teachers, curriculum standards, assessment practices, all of which must be addressed. With respect to assessment practices, quality can be enhanced by following readily available assessment guidelines for controlling item quality, sampling, standard setting and managing numerical data.1,2 It is important to minimise ‘preventable’ error, particularly at the pass ⁄ fail borderline, as learners increasingly choose to challenge ‘adverse’ decisions. However, not
all institutions use identical assessment approaches, thereby challenging the reasonable expectation that the graduates of equivalent courses should have similar abilities, particularly when there is a common curriculum framework for either individual nations or multi-national collectives. There are many examples of such frameworks.3,4 One response to this challenge is the introduction of centrally managed and quality controlled national licensing examinations: the subject of live debate in many countries. Another response is that of quality assurance by external accrediting agencies, such as the Quality
160 Blackwell Publishing Ltd 2009. THE CLINICAL TEACHER 2009; 6: 160–163
Assurance of Basic Medical Education (QABME) process of the General Medical Council (GMC) in the UK;5 similar agencies exist in North America and Australia.6,7 Still another response is to employ external examiners to contribute external perspectives and expertise to assessment practices within individual institutions. External examiners are particularly important in the UK, where a heavy reliance is placed on observing clinical examinations and probing borderline candidates in viva voce examinations. A code of practice has been developed by the Quality Assurance Agency (QAA) following the publication of the Dearing report in 1997.8
With the establishment of more institutional education offices, with the availability of assessment guidelines, and the increasing use of external accreditation and national assessment processes, one could question the current relevance of external examiners. The answer may not be clear-cut, but if external examiners are to continue, there is a strong case for them to adopt a broader role in evaluating and providing feedback on how well education institutions apply assessment guidelines, rather than focusing on their individual subject or specialty content expertise. This paper describes the broader external examiner role, and suggests the necessary knowledge and skills required for this broader role.
WHAT COULD/SHOULD AN EXTERNAL EXAMINER DO (I.E. WHAT IS THE JOB DESCRIPTION)? The QAA code of practice states that external examiners have a set of core functions to report on: whether the academic standards are appropriate; the rigour of the assessment; the equity of treatment for students; the standards of student performance; the comparability of the standards and students between different institutions offering, in the case of medicine, a primary medical qualification; and any good practice that they have identified.9 While these are clearly core roles, recent developments in assessment technology allow for the further development of their role, to include an overview of all of the activities listed in Box 1. The external examiner role is evolving into one that is much broader than checking examination papers, observing clinical examinations and interrogating borderline candidates from a subject or specialty perspective. A particular advantage of having a broad external perspective on
Box 1. The scope of assessment practices that may be viewed by external examiners • The alignment of learning and assessment in the curriculum • Item sourcing, writing and quality assurance processes • Item banking and blueprinting • Methods and results of standard setting • Content of set papers from the blueprint
The external examiner role is evolving into one that is much broader than checking examination papers
• Observation of clinical and practical examinations • The safety and equivalence of examination-board decisions • Reporting of assessment results • Identifying good practice • Provision of feedback to candidates
assessment is that this can embrace the whole assessment process. This begins with the relationship of assessment to the curriculum, through to the decisions based on the derived assessment scores for both individual candidates and candidate cohorts. Inclusion of the whole list of activities in Box 1 is a time-consuming task that requires early engagement, frequent contact and at least two annual visits, as well as substantial work from a distance. Ideally, external examiners should, in addition to proof reading, attempt (rapidly) each written examination prior to approving them, as this provides a ‘feel’ for the standards and the degree of difficulty. The total task may be easier if the external examiners are engaged for 3–5 years, and devote considerable time in the first year to understanding the curriculum and assessment processes of the institution that they are visiting. This familiarity may be facilitated if the external examiner comes from an institution with similar approaches, such as the degree of integration and range of assessment methods used, although deliberately seeking a more external view may be appropriate at times. The additional requirement for external examiners to offer a broad overview of assessment practice, as well as their particular subject
expertise, opens up the opportunity to develop small teams of external examiners with differing backgrounds, who can provide high-quality oversight for fully integrated curricula.
WHO SHOULD BE AN EXTERNAL EXAMINER (I.E. WHAT IS THE PERSON SPECIFICATION)? External examiners appear to be in short supply, and recruitment of new examiners is a continuous process, often achieved via personal networks. It may be more appropriate to recruit through more formal processes when there is a sufficient pool of potential applicants. A list of ideal characteristics is provided in Box 2, and they apply to both clinicians and pre-clinical scientists, and are not necessarily restricted to senior staff. At first glance, Box 2 provides a daunting list of qualities, but it is not difficult for individuals interested in medical education to gain the knowledge and skills to support an interest and enhance their experience in medical education. Involvement as an examiner, and membership of examination boards and progress committees, is an ideal training ground, as the issues and bases for judgment will become familiar. The availability of formal postgraduate training in medical education is increasing, through
Blackwell Publishing Ltd 2009. THE CLINICAL TEACHER 2009; 6: 160–163 161
It is essential that external examiners have credibility
Box 2. Characteristics of external examiners • Experience in medical education, ideally both in curriculum design and in assessment • Experience as an examiner, ideally in comparable courses and levels • Experience in assessment item writing, banking, blueprinting and selection • Experience in standard setting • Experience as a member of an examination board • Experience of working in teams of assessors • Familiarity with common exam statistics (e.g. SD, SEM, Cronbach’s alpha, generalisability coefficients, z scores, compensation methods) • Familiarity with the rules of progression • Capacity for hard work • Sense of humour
postgraduate Diploma and Masters courses, and through engagement with the Higher Education Academy and the Academy of Medical Educators.10–12 Expertise in the sciences underpinning medicine is important for integrated undergraduate curricula, not only in the earlier years where these subjects are dominant, but also in the later years, when the application of basic science knowledge should be practised and assessed within a clinical context. At least some working knowledge and understanding of common examination statistics is important, because means, standard deviations (SD), standard errors of measurement (SEM) and reliability measures (e.g. Cronbach’s alpha) should be reported, as should how they inform any permitted rules of compensation. Inappropriate use of statistics can profoundly affect data interpretation, and even the final results. Regardless of the experience and qualifications, it is essential that external examiners have credibility, because their judgments may be used to help institutions make difficult decisions about the progress of failing students and any subsequent appeals. More than one external examiner may be necessary to provide the whole list of desirable characteristics,
and so the ability to work effectively in a dedicated team is another desirable attribute that should be expected of external examiners.
THE REWARDS OF EXTERNAL EXAMINING What rewards are there for external examiners? The financial reward is generally small, although a case is often made for professionalising the role and
162 Blackwell Publishing Ltd 2009. THE CLINICAL TEACHER 2009; 6: 160–163
paying appropriately. This was one of the key recommendations of the Dearing report,8 but little progress has been made. On the other hand, many external examiners really enjoy the role. They often learn from seeing how others tackle tasks that are similar to those in their own institution, and return with enhanced expertise and value to their employers. Professional networks are created and expanded, something that can be very useful when difficult assessment dilemmas arise and advice is needed. Furthermore, the experience can help career advancement, as there are more senior roles within the education office at their own institution, recruitment opportunities at other institutions, and roles with external quality assurance organisations. Because most of the reward is currently intangible, just how attractive these career paths are is an individual decision.
GETTING INVOLVED Because the recruitment of external examiners is often via medical education networks, getting involved requires joining the local network. This should not be
difficult, as most medical schools and colleges are keen to recruit assessment item writers, standard setters and examiners. The best way is to become involved in curriculum delivery, engage in assessment development and implementation, enroll in assessment courses, seek a more experienced examiner as a mentor and express interest in sitting on exam boards. This can be facilitated by engagement with item banking consortia such as the Universities Medical Assessment Partnership (UMAP) in the UK or the International Database for Enhanced Assessments and Learning (IDEAL) consortium.13 Another opportunity for networking in the UK is via the Medical, Dentistry and Veterinary Medicine subject centre of the Higher Education Academy. Demonstrating a sound understanding of the issues and sensible decision making will go a long way towards developing a profile as an experienced examiner.
CONCLUSION The role of the external examiner can still have relevance to modern medical education, provided it involves participation in the complete range of assessment practices. This broader role requires individuals with training
and experience, but both are now readily available. Despite the shortage of short-term, tangible benefits, being an external examiner can be career enhancing in the longer term, through the development of breadth and depth of assessment knowledge, skills and experience. Education institutions should encourage faculty members to become external examiners, as they almost always return with enhanced value. REFERENCES 1. Newble D, Dauphinee D, DawsonSaunders B, MacDonald M, Mulholland H, Page G, Swanson D, Thomson A, Van Der Vleuten C. (1994) Guidelines for the development of effective and efficient procedures for the assessment of clinical competence. In: Newble D, Jolly B, Wakeford R (eds) The certification and recertification of doctors. Issues in the assessment of clinical competence. Cambridge, Cambridge University Press. 2. van der Vleuten CPM. The assessment of professional competence: developments, research and practical implications. Advs in Health Sci Educ 1996;1:41–67. 3. General Medical Council. Tomorrows doctors, 2003. Online at http:// www.gmc-uk.org/education/undergraduate/undergraduate_policy/ tomorrows_doctors.asp (accessed March 2009).
4. Frank JR, Jabbour M, Tugwell P, et al. Skills for the new millenium: report of the societal needs working group, CanMEDS 2000 Project. Annals Royal College of Physicians and Surgeons of Canada 1996; 29:206–216. 5. General Medical Council. Quality assurance of basic medical education. At: http://www.gmc-uk.org/ education/undergraduate/ undergraduate_qa.asp (accessed March 2009).
This broader role requires individuals with training and experience, but both are now readily available
6. Australian Medical Council. See http://www.amc.org.au/index. php/ar (accessed March 2009). 7. Liaison Committee for Medical Education. See http://www.lcme.org (accessed March 2009). 8. Dearing Report. The Dearing Report (10.93-96). See http://www.leeds. ac.uk/educol/ncihe/ (accessed March 2009). 9. QAA code of practice for external examiners. See http://www.qaa. ac.uk/academicinfrastructure/ codeOfPractice/section4/COP_ external.pdf (accessed March 2009). 10. Hays RB. Assessment in medical education: roles for clinical teachers. Clin Teach 2008;5:23–27. 11. Academy of Medical Educators. See http://www.medicaleducators.org/ (accessed March 2009). 12. Higher Education Academy. See http://www.heacademy.ac.uk/ (accessed March 2009). 13. IDEAL Assessment Consortium. See http://www.hkwebmed.org/ idealweb/homeindex.html (accessed March 2009).
Blackwell Publishing Ltd 2009. THE CLINICAL TEACHER 2009; 6: 160–163 163
