5. Methods: Process. â¢. NEATS' medical education team, comprising medical educators and an education officer, used brain storming to develop this project.
Evaluation of a Quantitative Formative Assessment Tool for use in GP Registrar External Clinical Teaching Visits (ECTs).
Adjunct Professor John Fraser, Director, Hunter New England Area Rural Training Unit, Senior Medical Educator, New England Area Training Services
0
Background
•
•
•
The Australian General Practice Training has emphasized external clinical teaching visits as a form of formative assessment during training for many years. During these visits, a senior GP observes the consultations of registrars and provides structured feedback and makes recommendations to improve performance. Formative assessment can assist in guiding learning. Opportunity to observe what registrars do at work.
1
Background •
•
•
NEATS has used a qualitative proforma to report on all ECT visits. (Black.)
This proforma reports on individual registrar’s consultations and globally describes their present strengths and weaknesses. Both the content and process of registrars consultations are explored in NEATS’ existing qualitative ECT form.
2
Background •
• •
In 2005, the number of medical educators and registrars expanded in the region (prior to this one educator undertook most ECTs and could monitor registrar progress).
NEATS’ strategic plan & RACGP standards endorse assessing the baseline skills for our present cohort of registrars Plan and allocate appropriate resources and monitor progress during training.
3
Objective •
•
This paper describes the development of a quantitative criteria based formative assessment tool for use in external clinical teaching visits with Australian general practice registrars in 2005-2006. Furthermore, this paper evaluates acceptability of this tool to GP registrars for use during ECTs.
4
Methods: Process •
• •
NEATS’ medical education team, comprising medical educators and an education officer, used brain storming to develop this project.
NEATS medical education team expansion had resulted in difficulties in comparing qualitative assessments conducted by different educators. Additionally, some registrars had limited insight about their present level of performance compared to the standard required for RACGP fellowship.
5
Methods •
• •
A quantitative assessment giving a visual representation of performance was considered to be a useful aide to assist learning by some educators.
Some educators expressed concerns about the implementation of this tool as the quantitative assessment may be counterproductive to improving performance. Perceived risk if the tool was used summatively to compare registrars with their peers rather than formatively.
6
Methods • • • •
Based of these different viewpoints, a trial was developed prior to wide spread adoption of the new tool. Standardization. The medical education team received instruction that the aim of the new tool was formative. All members of the team received training in providing structured feedback using Pendleton rules prior to undertaking any quantitative formative assessment of a registrar. Feedback was always given in person between registrar and educators. Registrar focused positives then negatives,
7
Methods •
•
We used a standardized validated consultation feedback proforma (Hays) which uses a Likert scale to assess criteria based components of a consult.
We modified the proforma to include more scope for comments and also to set the 5/5 rating as being on parallel with the FRACGP exam standard.
-
Proforma comments on subsections of introduction, history, examination, diagnosis, management, closing, general comments and global rating
8
Proforma ACN 099 568 939 PILOT ECT VISIT FEEDBACK FORM Registrar: ________________________________________
This form is designed to provide information to New England Area Training Services Ltd and your responses will be treated as confidential. 1=strongly disagree 2=disagree 3=neither agree or disagree 4=agree 5=strongly agree INTRODUCTORY PHASE The introduction to the patient was appropriate The patient was placed at ease
1
2
3
4
5
Comments: ____________________________________________________________________________________
HISTORY-TAKING PHASE The patient laws listened to attentively Non-verbal clues were appropriately followed up Appropriate question style was used Medical jargon was avoided Appropriate eye contact was made Psycho-social factors were considered An examination history was obtained
1
2
3
4
5
Comments: ____________________________________________________________________________________
EXAMINATION PHASE
1
2
3
4
5
9
Methods •
• •
Individual quantitative formative assessment ratings were completed with qualitative assessments and returned by all five medical educators.
These assessments were discussed with the registrar during visits. Registrar strengths and areas for improvement were outlined. A separate evaluation questionnaire was given to the registrar and returned.
10
Methods •
• •
The evaluation proforma questionnaire asked registrars
useful feedback on their consultation skills, useful prompts for discussions with a medical educator
if the form should be used in future . ratings were fair and if they could discuss any disagreements with NEATS (medical educator or GP supervisor).
The evaluation form rated responses on a 5 point likert scale. (1= strongly disagree to 5 = strongly agree) This was condensed to agree or not agreeing in analysis. Associations were explored using Chi square test and Fisher’s exact tests in EPIINFO. (Version 3.3, USA)
11
Results • • •
•
From September 2005 to March 2006, 26 external clinical teaching visits were conducted. A quantitative formative assessment form was completed at all visits by medical educators. A total of 21 registrar feedback forms were received from these visits. Response rate (80.1%). Of respondents, in 2005, there were 2 registrars in basic terms and 5 registrars in advanced terms. In 2006, 5 basic, 2 advanced and 7 subsequent term registrars replied.
12
Results •
• • •
20 (95%) of responding registrars agreed the form was useful feedback and a useful prompt for discussion with their medical educator.
19 (90%) of responding registrars stated they would like to use the form with all ECT visits. 18 (86%) of responding registrars felt their ratings were fair. 17 of 19 registrars (89%) felt they could discuss disagreements in ratings with a medical educator or GP supervisor. (2 missing)
13
Results •
Medical educators’ mean global rating of registrar performance was 3.7 out of 5 (Range 1 –5).
14
Results • • • •
A sub-analysis of registrar performance ratings and evaluation forms was conducted for 2005 and 2006. There was no statistically significant differences in registrar performance across all stages of the consult between the two years. In 2006, more registrars strongly agreed, that the form gave them useful discussion points for their medical educator than the 2005 cohort (Fisher exact, 2 tailed, p=0.04). In 2006, more registrars also strongly agreed that they would like to use this form with all their ECT visits than the 2005 cohort (Fisher exact, 2 tailed, p=0.04).
15
Results Strengths of Cohort. • • • • • •
Appropriate follow-up arrangements were made 4.3 The introduction to the patient was appropriate 4.2
Empathy and understanding was exhibited 4.2 The patient was placed at ease 4.1 Explanation to the patient was adequate 4.1
The timing of closure was appropriate 4.1
16
Results: Cohort Area for Improvement • • • • •
Psycho-social factors were considered 3.4
Illness prevention/health promotion was provided 3.4 Correct use of time and resources was made 3.5 Other relevant problems were defined 3.5
Appropriate question style was used 3.6
17
Discussion • • •
• •
A quantitative formative assessment was acceptable to the majority of GP registrars. This tool complements other formative assessment tools used by NEATS in GP training To facilitate learning a formative assessment needs to be mapped against the curriculum. Furthermore it needs to be valid, reliable, fair and the required standard needs to be explicitly stated. Setting the standard as the FRACGP rather than their present state of training facilitated this process.
18
Discussion • • •
Both the registrar and educator need to be clear on the standards and the intended purpose of an assessment. Assessments can be used to guide learning and provide feedback (i.e. formative) or be used to assess whether a end point or standard has been reached (summative). In training assessment can be formative (coach) or summative in nature
19
Discussion • •
A major limitation of this study is that we are unable to link registrar evaluations with their quantitative assessments. We are unable to assess whether registrars who scored low in their quantitative assessment differed in their evaluations of this tool compared to higher rating registrars.
20
Discussion • • • •
Some caution is required in extending this tool to summative intraining assessment. medical educators’ training and mentoring role may not always be conducive to implementing a summative assessment or regulatory role. formative assessment with structured feedback needs some frankness and openness between registrars and educator. Unlikely to occur if assessment may incur penalty
21
Discussion •
• •
More registrars strongly agreed with the quantitative ECT tool being a useful discussion point and being used in all ECTs in 2006
May reflect that medical education teams need to familarise with a new tool and time use it effectively (Alternatively, this may reflect difference in the two cohorts of registrars). There is insufficient data to explore this further.
22
Discussion •
Medical educators commented on the value of the tool during the pilot period and have used the tool to:
Identify cohort learning priorities and to assist educational program planning., to identify trends in the ratings of registrars between educators and discuss whether this reflects inter-observer reliability or a decline in performance requiring further observation to target registrars requiring more intensive educational assistance. This tool will continue to be used in conjunction with the existing qualitative tool. Both measure different things.
23
Global or Criteria Based •
•
•
A critieria based instrument can only measure specific components of a situation
What do you think of my tail?
Based on all criteria this is a useful elegant tail
24
Global or Criteria Based
•
•
Bigger Picture can be more valid and reliable
Using an Elephant to catch a grasshopper is not always the most effective method despite its handsome tail
25
