Universe... or the last ofthe great romantics... and some- timesyou'rea hero-doing extraordinary thingsand making people's lives better. My very favourite movieĀ ...
I, The Science of 9
I
SUMMARY In-training evolution requires a rigorows approach and high academic standards. Qualitative research prnciples can be applied to meet student, preceptor, ond progrm needs; methods that enhane trust are helpful. Ideally the student and preceptor should collaborate to explore the student's strengths and weokNesses. While stu t Oan preceptor my not agree, the qualitfotive process dolries reosons for drfferences of opinion. A properly performed in-training evoluotion not only documents skills attained and laing needs, but olso wE facilitate learning during the rotation. RESUME Lt'.voleaion en cours de formatin n6cessite une epproche rigoureuse et des criteres acomiques visont l'excellence. 11 est possile d'appliquer les prmcipes de Ia recherche quaitotive et do reponre a hi fois -x besoiNs de l'etudiant, do precepteur et du programme; les methodes qui diveloppent hi conflance sont utiles. ldialement, 1'etudiint et le prncepteur devraient joindre lurs efforts afin d'explorer les forces et les failalsses do l'etudiont. Lorsque pr6cepteur et etudiant ne sont pos necessairement d'accord, le processus qualtoati darifle les raisons de ces iferences d'opnions. Une hnNn evaluation en cours de formation permettro non sevlement de documeater les habletes ocquises et les besoms d'ap prntissage, mais elle facilitero assi l'opprntissage pendant les stoges rotatoires. Cm kmn
A 1990 3620022006.
In-T-raining Evaluation M.G. DONOFF,
E
MD, CCFP
MPROVING
I
IN-TRAINING
evaluation is high on the agenda of many medical educators. But many faculty are uncomfortable with evaluation and do not know how to do it. What will make them more capable and enthusiastic? Changing the reporting form may help but will not in itself make the task intellectually rewarding for either student or preceptor. Clarification of goals and objectives would be useful for many rotations but does not in itself measure whether the student has mastered those objectives. What is needed is an academically rigorous method to determine a student's achievement of clearly identified objectives that will facilitate learning during the rotation. The method should lend itself to providing a report at the end of the rotation that both guides the student and helps the program directors plan future educational activities. I see in-training evaluation as, fundamentally, a process of inquiry; and I believe that qualitative research methods can make evaluation credible and valid. In this article I explore applying principles of qualitative research to develop rigorous, trustworthy in-training evaluation. Dr Donoff is an Assistant Pmfessor, Department of Family Medicine, at the Universi_ ofAlberta, Edmonton.
2002 Canadian Family Physician VOL 36: November 1990
IS THIS REALLY SCIENCE? Certainly one would expect scientific process to reveal previously unknown knowledge. Properly performed, in-training evaluation allows the student and preceptor to collaborate to understand the student's attainment of knowledge, skills, and attitudes in the discipline being explored. At the onset of the rotation, the student's strengths and deficiencies are unknown or known to a limited extent. By the end, both have learned something of these matters: this is new knowledge. The fact that the new knowledge is not generalizable to all circumstances does not detract from the scientific nature ofthis enterprise. Generalizability is not the sine qua non ofscience. The new knowledge is vitally important to the student, the institution, and the student's future patients and colleagues. The understanding ofhow the student performed during this rotation is tentatively applicable to future performance in certain other contexts. The concept of "tentative application" is well described in scientific literature using qualitative methodology. How a student performs during this rotation can only tentatively indicate future performance in other settings. To attempt to generalize the student's competence to all other settings is to ignore the multitude of factors that influence performance. Educators are well aware of the difference between competence (can the student do it?) and perform-
ance (does the student do it?). The analogy in conventional medical science is the difference between efficacy (can it work?) and effectiveness (does it work?). Another way to express the scientific nature of in-training evaluation is that the evaluation process helps students make sense of their experience in grappling with the principles of a discipline. The preceptor's feedback and the student's developing self-awareness clarify when knowledge skills and attitudes are adequately understood and applied and when they are not. Research "emphasizes the rigorous and systematic collection and analysis of data" '; it has been defined as "the persistent, disciplined effort to make sense and order out of the phenomena of subjective experience"2; this is the spirit of what the preceptor and student must do to evaluate the student's performance.
WHERE ARE THE REIHABILITY COEFFICIENCY TABLES? Most readers ofevaluation literature will be familiar with studies that use mathematical analysis to gauge the reliability and validity of instruments to measure knowledge and competence. Yet it makes little sense to subject in-training evaluation methods to similar analysis. The clinical rotation is not a test. To develop rating forms as if the rotation were a very lengthy Objective Structural Clinical Examination (OSCE) is neither feasible nor scientifically defensible. Evaluation methods like OSCEs are samples chosen carefully to test specific hypotheses of clinical competence. A rotation, on the other hand, is a sequence of clinical experiences that cannot be predetermined or standardized. It demands that a student analyze, synthesize, and evaluate at a higher level than on a structured test. Therefore the correlation of in-training evaluation scores with other measurement tests may not be high because different attributes are being measured. More importantly, the reason for measurement is not primarily to classify students or rank them but to identify learning needs and document acquired skills.
HYPOTH1ESIS TESTING It is poor scientific method to use conventional hypothesis testing as the model for
evaluating students on a rotation. First, the number of hypotheses to be tested is unmanageably high. To be valid, research usually must limit the number of hypotheses being tested. With a student, we would need to test hypotheses about data acquisition, management, and professional attitudes for multiple illnesses and diseases, in different age groups, at different stages of disease presentation, in different patients, in different settings. Most of these variables enter any single rotation. Second, the actual cases with which the student will be involved cannot be predetermined. There is little standardization of one student's experience to another's. Third, and most importantly, we do not know which hypotheses are the most relevant for a given student. We don't spend identical times testing each student's knowledge, skills, and attitudes because it is more efficient to individualize attention to the student's needs so that the rotation can address them. The challenge becomes to have a scientifically valid process to understand an individual's strengths and weaknesses. Such paradigms do exist. Canadian Faminy PhySiCia
VOL 36: Nomber
1990 2003
APPROPRIATE PARADIGMS Most medical scientists have a predisposition to quantitative, positivistic paradigms of inquiry. We are all familiar with the design characteristics ofclinical trials to study therapeutic interventions, diagnostic tests, prognosis, and etiology. The limitations of such conventional methods in some areas of medicine and medical education have rekindled interest in methods from other disciplines, such as education and the behavioral sciences. The techniques of an anthropologist or an educator studying human behavior may be more applicable to working with a student than methods used for drug trials. Which of the following scenarios best describes in-training evaluation? 1. I know precisely what I need to find out about this student, and I can control variables to test hypotheses. 2. I don't know this particular student's weaknesses, and I cannot control all the variables as I observe. If the second scenario more accurately describes our in-training evaluation situation, the characteristics of a qualitative investigation would be appropriate. Certain characteristics will enhance trustworthiness of the evaluation. These will be described in the context of in-training evaluation.
General characteristics of a qualitative inquiry Direct observation. Data about a student's performance is acquired through direct observation of the performance or by observing the results. The latter may include talking to patients or staff and chart review. The student's case history, whether written or oral, cannot be used exclusively to assess performance. The preceptor learns most about the student's skills by directly observing the student in action. Direct observation also allows for specific, timely feedback, which we all know is better for learning. As preceptors, we must determine for ourselves how much time need be devoted to direct observation, as opposed to gathering second-hand data from the student, charts, or other observers.
Respectfor context. Context has a major effect on performance. The setting in which the student is observed is representa2004 Canadian Family Physieia
VOL 36: Novembr 1990
tive of the settings in which the student will apply his or her skills unsupervised. A preceptor observing a student in an environment like that in which the student will later be expected to perform can gather data more valid than any test. Humans as instruments of inquiry. The student and the preceptor themselves are better instruments of the inquiry than any test instrument because "only the human instrument has the characteristics necessary to cope with an indeterminate situation."3 The human as an instrument of scientific inquiry is a powerful concept for in-training evaluation. Humans can sense and respond to all personal and environmental cues; they are highly adaptable and can collect information about multiple factors simultaneously. Any scientific instrument can be improved upon through refinement of sensitivity and accuracy as well as calibration. Preceptors can improve themselves as instruments of inquiry through faculty development. Students involved with this type of evaluation can improve their self-assessment skills, which is one of the identified goals of modern medical education.
Purposive sampling. The preceptor identifies and focuses on areas of greater need or interest for a particular student. Conventional scientists would frown on an investigator who changes the focus of the inquiry in midstream; rigid adherence to a predetermined protocol is required in conventional tests of one or a few hypotheses. Anthropologists and behavioral scientists, on the other hand, would criticize an investigator who does not respond to observed cues and pursue them diligently. The student's unique strengths and weaknesses are not predictable; the preceptor must be prepared to change the approach as they become apparent. Subject participation. The process of evaluation involves the active participation of both the student and the preceptor in evaluating both the student's and the preceptor's strengths and weaknesses. This can only occur with clearly understood and agreed-upon instructional objectives. These must be written in a fashion that de-
scribes observable changes in behavior. Contrary to some opinion, this does not exclude objectives pertaining to attitudes and values.4 Some may argue that our instructional objectives for each rotation, if properly written, would provide hypotheses for testing. As appealing as this would be, it is more valid to see objectives as terms of reference for the interaction between learner and teacher than as hypotheses to be tested in the complex setting of a clinical rotation. Qualitative researchers usually identify the underlying theory (or theories) with which they enter their study. As preceptors, our educational objectives correspond to our theories ofwhat is important in our discipline. Qualitative researchers must be open and clear about these beliefs to avoid hidden agendas as they proceed with their study. For students to work openly with us, they need to know what we believe to be important.
Open acknowledgment of the investigator's values. In-training evaluation is value laden, not value free. The preceptor's and the student's values weigh heavily in the process, and these values must be made explicit in reaching conclusions. Clear educational objectives do not eliminate the influence of values in interpreting another's behavior. Denying the effect of personal values or attempting to circumvent them is neither feasible nor honest when human beings act as instruments of inquiry.
Descriptive reporting. A report summarizing what has been learned about the student's strengths and weaknesses should be descriptive so that it can usefully indicate both to the student and to the program which learning needs will need to be addressed as the student proceeds through the program. Qualitative research terminology values "thick description" where appropriate. This evaluation differs from the more usual rating scale, which attempts only to classify the student's performance in comparison with others. Criterion-referenced rating scales are a move in the right direction, but the behavior being judged must be described completely enough to be clear. A standard rating scale does not reflect the
specific behaviors the student and preceptor actually agreed to address during the rotation. Purposive sampling may cause actual teaching objectives to vary, to some extent, from student to student.
Characteristics that enhance trustworthiness Prolonged engagement. Time is required for the student to develop a relationship with the preceptor that allows an honest disclosure of skills. But the student's initial defensiveness will melt only if the preceptor's attitude helps. Obviously, prolonging a rotation in the absence of an open relationship will not add more information. Persistent observation. The degree to which the preceptor directly observes the student - using a judicious sampling - also has a major bearing on the trustworthiness of the evaluation. This is not good news for those who wish to be responsible preceptors but do not wish to schedule some time to directly observe their charges. Finding time is a challenge to busy practitioners who receive little recognition for teaching efforts.
Triangulation. It is a principle of investigation that evidence should be corroborated. No important opinion about a student's abilities should be considered credible unless the data can be reaffirmed from another source. This may mean the preceptor observing the same behavior with another patient or another procedure; or it may involve observation by a colleague. Member checking. In behavioral science the investigators commonly check their interpretations with a member of the study group for congruence with the views of the subjects. The preceptor should find out whether his or her understanding of what is going on corresponds to the student's understanding. Naturally, preceptor and student will not always agree on the assessment of the student's ability (although the degree of agreement is greater than most expect if the principles of qualitative inquiry are being applied). Documented agreement enhances credibility. When preceptor and student disagree, the very act of checking often brings forward possible reasons for the disagreement. Canadian Famnily Physian VOL 36: November 1990 2005
Field notes. Field notes, a term obviously taken from anthropology, are also relevant to medical education. Students should keep a logbook of their experiences on rotations, and preceptors should keep some notes about their student's interests, strengths, and difficulties. A review of both improves the summary report. Field notes document observations as well as methods used to increase trustworthiness. Interpretations are also recorded. Personal statements, hunches, feelings, and unanswered questions are all helpful in creating an "audit trail" that shows how the investigator was reasoning and how the conclusions in the summary report were reached.
CONCLUSION Some of the terminology used by qualitative investigators may be unfamiliar to medical educators, but the concepts, when understood, are common sense. This paper retains some
of the language used by behavioral scientists to remind medical readers that the validation of concepts presented here comes from disciplines not frequently read by physicians. By applying these principles, in-training evaluation can gain credibility among medical faculty. Making the implicit methods of evaluation more explicit increases accountability. More importantly, the task of preceptorship can become a source of academic satisfaction for interested medical faculty. Students will benefit by having interested teachers and becoming active participants in developing self-awareness as clinicians. Testing to establish competence for licensure, certification, and recertification is currently receiving a great deal of attention. Medical educators should remember that testing measures competence, not performance, and does not always facilitate learning. In fact, frequent tests can adversely affect learning. Our prime
purpose as educators is to facilitate learning, not merely to determine whether it has occurred. In-training evaluation can contribute to both U tasks.
Requests for reprints to: Dr M.G.
Donoff Royal Alexandra Hospital Family Clinic, 10240 Kingsway Ave, Edmonton, AB T5H 3 V9
References 1. Bogdon RC, Biklen SK. Qualitative research for education: an introduction to theory and methods. Boston, Mass: Allyn and Bacon, 1982:207-10. 2. Kirschenbaum H, Henderson VL, eds. The Carl Rogers reader. Boston, Mass: Houghton Mifflin, 1989:26. 3. Lincoln YS, Guba EG. Naturalistic inquiry. Beverly Hills, Calif: Sage, 1985:193. 4. Anonymous. Instructional objectives module. In: Teaching improvement project systems for health care educators. Lexington, Ky: Center For Learning Resources, College of Allied Health Professions, University of Kentucky. 40536-0218.
~~Remember that dream youhad last night?
4
i! It's me, your conscience. Living here in your dreams is like having passes to the world's most exciting movies ... with you and me in the starring roles! Sometimes you're the monster that swallowed the Universe ... or the last of the great romantics ... and sometimes you're a hero -doing extraordinary things and making people's lives better. My very favourite movie is the one where you reach deep within us and change the world, simply by giving time s and money to help the people around us. And like great stars, you make helping causes you care about look easy. Could we watch that one tonight?
206 andanFaiy
hyica
6 VL _}
jNoebr19
2006 Canadzan Family 0 PhysicianVOL 3: November
199
to ~ ~ Imagine is a national program
encourage giving and volunteering.