With respect to OSCEs, the authors call for collaborative efforts among consortia schools in concert with the Center for the Advancement of ..... manager. ⢠Charge the âPerformance Committeeâ with the responsibility of acting upon evaluations ...
Evaluating Student Performance in the Experiential Setting with Confidence Diane E. Beck School of Pharmacy, Auburn University AL 36849-5502
Larry E. Boh School of Pharmacy, University of Wisconsin, Pharmacy Practice Division, 425 North Charter Street, Madison WI53706
Patricia S. O’Sullivan College of Nursing, University of Arkansas for Medical Sciences, 4301 West Markham Street, Little Rock AR 72205-7199 This paper proposes that pharmacy schools should have a functional experiential evaluation system (EES). An EES consists of multiple assessment methods, guidelines for their optimal use, and procedures that promote faculty-student communication about performance. With respect to selection of assessment methods, pharmacy educators are encouraged to adopt the combination of observation-based ratings, Objective Structured Clinical Exams (OSCEs), and examinations consisting of Extended Matching Items (EMIs). Practical strategies an individual instructor may use to enhance the accuracy of observation-based ratings and examinations are described. With respect to OSCEs, the authors call for collaborative efforts among consortia schools in concert with the Center for the Advancement of Pharmaceutical Education (CAPE). At the school level, the entire faculty must be accountable for maintaining a quality EES that promotes faculty-student communication about performance. Only after implementation of a comprehensive quality EES can a school’s faculty feel confident in decisions about students’ preparedness to deliver pharmaceutical care.
INTRODUCTION
As pharmacy schools develop and expand their entry-level degree program’s experiential component, faculty confidence in decisions about student performance in the practice setting has taken on renewed importance. To achieve this confidence, first the faculty must have data of appropriate quality and quantity from which to base an assessment. Second, the assessment methods must be free of biases that potentially threaten the process. Third, the assessment methods collectively must provide a level of accountability. These qualities ensure validity and reliability of the assessment process. Decisions made under these circumstances assure the pharmacy faculty and other stakeholders that the school only graduates those who can successfully provide pharmaceutical care. An earlier paper explained why instructors should use several methods for assessing student performance during experiential rotations and why observation-based performance ratings should be the primary component(1). It was cautioned that observation-based performance ratings often suffer from rating inaccuracy and frequently are based on too few observations. Because rating inaccuracy has been attributed to the rater’s cognitive processing skills, Beck et al., described a model designed to make raters more aware of how they acquire, store, recall, and integrate information into ratings. Pharmacy educators were encouraged to provide their experiential faculty1 with a rater training program based on this model. However, because there are so many facets influencing performance assessment, rater training alone will not re1
A pharmacy school’s full-time, part-time, and volunteer faculty.
236
solve all assessment problems. One way to address these multiple facets is to establish an experiential evaluation system (EES). An EES consists of a combination of assessment methods, guidelines for their optimal use, and the procedures established for faculty-student communication about student performance(2). The goal of this paper is to describe how to implement an EES that promotes faculty confidence about performance decisions. First, the reader will be provided recommendations for selecting assessment methods for the EES. Second, practical strategies for overcoming the limitations of observation-based ratings, simulations, and written examinations will be delineated. Finally, suggestions for communicating and handling issues related to performance assessment will be discussed. Many of the recommended strategies can be implemented by an experiential faculty1 or individual instructor. However, others require either involvement of the entire school faculty or pharmacy leaders at the national level. SELECTING ASSESSMENT METHODS
Consider Goals and Objectives. When selecting the most appropriate assessment methods, an experiential faculty must first consider the learning goals and objectives for the rotation(3). These goals and objectives should be explicit statements that communicate the activity and level of performance the student is expected to demonstrate. Since the primary aim of pharmacy practice experiences is to have students successfully provide patient care in the actual clinical setting, the goals and objectives should reflect this endpoint. The learning pyramid offered by Miller(4) em-
American Journal of Pharmaceutical Education Vol. 59, Fall 1995
Fig. 1. Framework for clinical assessment(4). Published with permission from Academic Medicine.
phasizes attainment of such goals requires achievement of the highest levels of learning (Figure 1). Miller’s model has a role in experiential education much like that which Bloom’s Taxonomy for Cognitive Learning Objectives has fulfilled in classroom learning(5). According to Miller’s model, there are four levels of ability the learner must achieve in order to provide patient care and they successively build on each other. Miller posits at the lowest level, the practitioner must have a basic fund of knowledge or “know” information needed to accomplish the higher tiers of performance. Having a fund of knowledge however, does not assure that the individual can apply the information and solve problems. Therefore, the second tier indicates that the learner must “know how” to use the knowledge and reflects the usual definition of competence(4). However, competence does not assure that the learner can actually provide patient care. Hence, the third tier indicates the individual must be able to “show how” or perform the function. At the highest tier, the learner successfully provides patient care even when not in an artificial testing situation. Specifically, the learner “does” which means one accomplishes complex functions and incorporates them into routine practice. The provision of pharmaceutical care requires a fund of knowledge as a base, but the majority of the rotation goals should rest at the two highest levels. Therefore, the primary assessment methods adopted for an EES should infer whether the student “does” and the student can “show how.” Desired Attributes. In addition to inferring performance in routine clinical practice (i.e., the student “does”), the ideal assessment method should manifest two other major attributes(6). First, the method should accurately separate the good student from the bad one. Second, because pharmacists encounter an array of drug-related problems and issues in the clinical setting, the assessment method should accumulate enough data to predict how effectively the student solves patient problems of varying type and difficulty. Other characteristics of an ideal assessment method include low cost of development and administration, practicality, and ease of use(3).
Possible Assessment Methods. Table I provides a listing of assessment methods frequently used in experiential education and their strengths based on the attributes just delineated. Unfortunately, there is no single assessment method for which these characteristics are all strong(6). Therefore, the instructor must use a combination of assessment methods whose strengths complement each other(7,8). A combination of observation-based ratings, simulations, and written or computer-based examinations achieves this aim and therefore, should be the primary assessment methods of an EES(1). The instructors challenge is to minimize the limitations each exhibits. Patient or case presentations are frequently used to assess experiential student performance, but they should not serve as a primary assessment method(1). During a patient presentation, the student presents a patient case to a group of peers and faculty and describes relevant literature. Although patient or case presentations involve direct observation of the student, good interrater reliability can be achieved(9). However, the major limitations of this method are the inability to infer the student’s performance in the actual practice situation and management of a variety of patients with problems of varying type and difficulty. Faculty members should also be cognizant that the patient presentation primarily assesses the student’s ability to communicate patient information and often contains information elicited by faculty and peers during prior discussions about the patient(10,11). Also, it is possible for students to present a polished case study without having actively participated in providing patient care. If patient presentations are used as a secondary mechanism of assessment, instructors can minimize these limitations. For example, one could require that the content of the presentation elicit what the student did to resolve an actual drugrelated problem of the patient(9). In addition, the instrument used to evaluate this content should weight items assessing the student’s management of drug-related problems more than those related to delivery of presentation. Table I outlines other methods for evaluating clinical performance and their attributes concerning the characteristics of the ideal assessment method. Some of the methods, such as peer assessment, have not been tested in pharmacy education and primarily have a role in formative evaluation. Chart audits are advocated as an assessment method which infers how the practitioner performs in routine practice. However, until most pharmacy practitioners and students routinely document their recommendations in the patient’s chart this assessment method will have a limited role in evaluating student performance. The other methods cited in Table I do not have as many qualities as do observationbased ratings, simulations, and written or computer-based examinations. Therefore, the other methods listed in Table I should only supplement this combination. Weighting of Assessment Methods. Since pharmacy student rotations are experiential, the assessment methods that best measure the student’s performance in providing routine patient care should be weighted the most when assigning the final course grade(1). This relative weighting can be visualized by simply inverting the triangle in Miller’s model (Figure 2). As depicted in Figure 2, observation-based ratings should be weighted the most, followed by simulations. Written or computer-based examinations, which assess problem-solving skills rather than factual knowledge may also be
American Journal of Pharmaceutical Education Vol. 59, Fall 1995
237
Fig. 2. 1 he relative weighting of the evaluation methods applicable to experiential learning.
used, but they should be weighted the least. Based on these weightings, we will now focus our discussion on these three primary assessment methods. For each method, a description of its attributes and deficiencies will be highlighted. In addition, the reader will be provided with practical strategies for implementing and using each method. OBSERVATION-BASED RATINGS
Of the methods listed in Table I, observation-based ratings best infer how the student performs in the actual patient care setting(6). Therefore, they are best able to assess whether the student “does,” Although observation-based ratings have this strength and this method is widely used to evaluate pharmacy student performance(12), pharmacy educators have not elucidated ways to enhance their accuracy. The following strategies give special attention to ways an experiential instructor can strengthen rater accuracy and accumulate a sufficient number of observations. First, experiential faculty members can enhance rater accuracy by participating in Rater Accuracy Training (RAT) workshops(13). A RAT session usually begins by discussing the following points about the rating instrument: (i) the dimensions of practice comprising the instrument; (ii) its scale anchors; and (iii) types of behavior indicative of these anchors. Those participating in the workshop then view a video vignette depicting the performance of one or more students. After this viewing, each RAT participant individually rates the student’s performance. The trainer then follows up with a discussion about the accuracy of the participants’ ratings and the student’s effective and ineffective behaviors. Group discussions are an essential component of this training process and encourage program faculty to all use the instrument in the same way. Although such training can enhance rater accuracy, RAT alone will not resolve all causes of rater error. To further reduce rating error, experiential instructors must improve their cognitive processing skills(14-16). Specifically, experiential instructors must attend to how they acquire, encode, retrieve, and integrate performance information into a rating. Figure 3 describes these four processes and depicts how each builds on the others. For example, since acquisition of information provides the foundation of information that is recorded, stored, and eventually inte238
Fig. 3. The cognitive processing model and instructor strategies.
grated into performance ratings, the effects of biased observations can be profound. This figure also summarizes practical strategies the experiential instructor can implement to enhance his/her cognitive processing of performance information. Since these strategies can enhance the reliability of an important assessment method, we will now explain them in greater detail. Directly Observe the Student. Based on the Cognitive Processing Model described in Figure 3, experiential faculty members must make accurate observations and accumulate them in sufficient number and variety(1,6). Biased observations can occur if the experiential instructor has preconceived impressions about the student. Therefore, experiential instructors must observe the student with special care if they have had earlier interactions with the student or talked with another instructor who has taught the student. At the very least, experiential instructors must directly observe their student performing skills such as conducting medication histories, providing patient education, preparing intravenous medications, and communicating with health professionals. This premise is drawn from findings of medical educators who have identified the importance of directly observing medical students and residents as they perform physical examinations(17-22). They have concluded one should not solely rely on the student’s reported findings during rounds or patient presentations. Medical school faculty members emphasize that without observation, errors of physical exam procedures may go undetected, lead to inaccurate assessments of student performance, and instill incorrect practice skills. Data depicting the student managing a variety of patients will give faculty greater confidence in decisions about experiential performance(6). Therefore, to accumulate enough data, the experiential instructor needs to make a sufficient number of observations. Experiential instructors frequently ask, “What is a sufficient number of observations?” Unfortunately, a single quantifiable number does, not exist. The number of recorded observations should be sufficient such that upon review by another faculty member,
American Journal of Pharmaceutical Education Vol. 59, Fall 1995
Fig. 5. Ratings should be assigned when the student s performance has reached steady-state.
Fig. 4. Example of an anecdotal record.
this individual would reach the same conclusion. If the setting provides limited opportunities to. observe certain practice dimensions2, the options are to either supplement the site’s learning activities or reconsider the site as one for experiential training. Frequent but unobtrusive observations by the instructor minimize student awareness of being assessed and therefore, will better reflect how the student performs in routine practice. Experts in observational research indicate that the person being observed may initially perform differently than they routinely do. but after this initial awareness they revert to their usual performance(23). Therefore, the experiential instructor should observe the student several times daily. The experiential instructor is more likely to be unobtrusive if one observes the student from a distance and does not maintain vision on only the student. Record Observations. If experiential instructors store their observations in memory, they are likely to exhibit poor recall or less organization of information at the time of assessment or both(1). When the student asks why the instructor assigned a certain rating, the instructor has difficulty providing concrete evidence. For example, an instructor may tell the student. “I gave you a rating of marginal because you communicate poorly with others,” When the student asks what is meant by “poor” the instructor is not able to recall specific incidents and has difficulty explaining this rating. This issue can quickly escalate into a situation where the student challenges a grade and the instructor lacks documentation to support the decision. The experiential instructor may minimize this by re2
The independent qualities of the overall performance. For example, several dimensions of competence in pharmacy practice are communication skills, patient monitoring skills, and professionalism.
cording observations of the student(1). In the clinical setting, anecdotal records serve as one way to record observations of student behavior(24-26). The experiential instructor makes an anecdotal record by observing a single event and writing a description in specific terms. One should document facts and not opinions about the observed behaviors. Brevity and clarity are more important than grammatically correct sentences. Therefore, the instructor may use abbreviations, codes, and symbols to facilitate the recording step. The instructor should record the event as soon as possible after the observation to attain an accurate description. To enhance data retrieval, the instructor should include on each anecdotal record, the student’s name, and the task or dimension it pertains to. If the instructor wants to make an interpretation or assessment during the recording step, the instructor should document this separately from the described behavior (i.e., on the back of the anecdotal record). Figure 4 provides an example of an anecdotal record. Rater compliance is the major factor limiting the success of anecdotal records. Instructors should individualize the recording technique to meet their particular teaching and practice situations. For example, some instructors may prefer recording their observations on a pocket dictaphone. During an inpatient clerkship, the instructor may make notations directly on a card or monitoring form used to track patient-student assignments. Retrieving Performance Data. At the end of the assessment period, the experiential instructor should use a structured approach when retrieving the recorded observations(1). Anecdotal records should be sorted by dimension so that the instructor can rate each student on the first dimension before rating the next dimension. If an anecdotal record depicts several dimensions of providing patient care, it should be reviewed again when rating each applicable dimension. Integrate Data into Scores. At the time of assigning ratings, the instructor must integrate the accumulated data into scores1. When retrieving and reviewing the recorded observations, the experiential instructor should be mindful that, when under stress one is more likely to depend on first impressions and cannot accurately differentiate performance of the component dimensions being evaluated(27). Therefore, the assignment of ratings should be done at a predetermined time when one does not have other competing demands. Experiential instructors frequently encounter uneven
American Journal of Pharmaceutical Education Vol. 59, Fall 1995
239
student performances during a rotation. How the instructor addresses this variation when assigning ratings is a potential source of error. Reliability and generalizability theories assume that when instructors assign ratings, the student’s performance is at “steady-state”(28-29)(Figure 5). Given this premise, an experiential instructor may want to establish a priori, the number of days students typically need to become acclimated to the practice setting. Following this learning period, the instructor should weight observations equally since the student’s performance should be at a plateau. Designating the steady-state period, during which the experiential instructor will collect performance data, also separates the time of learning from assessment(30). This is important since students usually face a learning curve at the beginning of a rotation. During this learning phase, the student should feel comfortable in seeking feedback from the instructor about how to improve and perform various professional skills. They should not have concern about being penalized for what they do not know. Although this concept is sound, it can compromise the instructor’s opportunity to make a sufficient number of observations during the steady-state phase if the rotation is too short in duration. Frequently, instructors struggle with how to rate a student who performed “poorly” during the first portion of the “steady-state period” but improved significantly during the remainder of the rotation. To resolve this dilemma, the instrument could include a dimension that assesses the student’s ability to perform consistently. This is an appropriate dimension to evaluate since a competent practitioner must exhibit consistent performance. By having this as a separate dimension, the instructor is able to evaluate both the student’s current level of performance and consistency. Negative incidents, which occur either at the beginning or the end of the steady-state period, may bias the selection of a score more than if these events took place between these two times(31-32). The instructor can minimize such a problem by rating the student’s performance several times during the steady-state period and then using these intermediate ratings to guide the final assessment. Other Raters. Some instructors are just “better raters.” These individuals often have more experience in rating students. Research has found that these raters are superior in attending to the dimensions of multidimensional behaviors(33). In rating performance, these individuals can also more accurately differentiate among a group of students(34). Therefore, when an experiential instructor is perplexed about the most appropriate score, one may wish to seek the assistance of a colleague who has such abilities. Littlefield(35) has advocated a similar concept by involving “gold-standard raters” when evaluating the marginal student. In medical and dental education, academicians have proposed that other health professionals and patients who interact with the student may more appropriately observe and evaluate humanistic and professional-patient relationship skills(3638). Physicians and nurses can provide valuable insights about a pharmacy student’s performance in these areas. However, experiential instructors should include ratings assigned by patients with caution since they often over rate a student’s performance(39-40). These strategies should assist the experiential instructor in making more accurate ratings. However, the instructor must be willing to assign the most appropriate scores(41240
43). This conflict between the rating deserved and what the rater is willing to assign often becomes evident when the instructor must convert the ratings into a grade. For example, an instructor may conclude that a student deserves a grade of “A” based on the overall performance. However, when a mean percentage score is computed using the rating instrument the result is 80 percent or “B” level work (i.e., average rating of 4 using a 5 point rating scale). To minimize this, the EES should incorporate mechanisms that separate the process of assessment and assignment of a grade(42-43). For example, the experiential instructor could assess student performance using the assessment instrument but not assign a grade. The experiential program director would then undertake the responsibility of converting the ratings into a grade. The grade could be determined using a conversion table, established by faculty a priori, which interpolates the overall mean score to a grade (i.e., a mean rating of 4 on a 5 point scale corresponds to a grade of “A”). The experiential director would then present the grade to the instructor for final approval. SIMULATIONS
The desire for an assessment method that not only predicts performance in a variety of clinical situations but is reliable has led medical and nursing educators to use simulations (4457). Those described in the literature range from Objective Structured Clinical Examinations (OSCE’s) to computerbased simulations. An OSCE involves subjecting students to multiple standardized patient encounters as they rotate through a series of work stations(46). Standardized patients are individuals trained to serve as both a patient and evaluator(44-45), (5254). They may or may not be actual patients. Although OSCEs have proven reliable and can test a variety of problems, there are little data inferring they truly predict how the student will perform in routine practice(6,44). OSCEs conducted in a laboratory setting provide an artificial environment whereby students only have to perform well for a brief period. To compare medical resident performance in routine practice with that achieved during an OSCE, researchers have sent undetected standardized patients to residents’ clinics and compared the management of such patients to the resident’s performance during an OSCE(49,58). Tamblyn et al. found the OCSE data predicted the medical residents’ performance with the standardized clinic patients; however, they were not accurate for those residents performing in the lowest quartile(49,58). Several pharmacy educators have described use of OSCE’s to assess experiential student performance(59-61). The educators reporting these experiences, have provided limited psychometric data inferring validity as cited by Kane(6). Guidry et al.(61) reported the development, reliability, and validity of an OSCE designed to evaluate students completing a pharmacy externship program. Although good interrater reliability was achieved with most of the work stations requiring direct-observation by the instructor, the one involving patient communication skills did not. This OSCE correlated better with NAPBLEX scores than did written examinations and preceptor ratings. However, we do not know the validity and reliability of the written examinations and preceptor ratings. The implementation of an OSCE is a monumental task and cost is the primary factor limiting its widespread use(62,63). Effective OSCEs require time of not only health
American Journal of Pharmaceutical Education Vol. 59, Fall 1995
professionals, but standardized patients, and psychometricians. Given the extensive resources needed to implement valid and reliable OSCEs. we encourage a pharmacy school to make this assessment method an effort of the entire faculty and collaborate with regional schools(64). At the national level, pharmacy leaders are encouraged to develop consortia programs composed of nearby pharmacy schools. A consortia of pharmacy schools can share personnel, assessment instruments, and other supporting materials(64). Medical and nursing educators have described use of computerized simulations to assess whether the student can demonstrate successful management of a case and problemsolving skills (i.e., “shows how” and “knows how”). Although pharmacy schools have described use of computerized simulations, they have not widely used such methods in the actual assessment of student performance during experiential rotations(65-70). Since these often require fewer resources than an OSCE, they hold a promising future in the assessment of pharmacy student performance. However, computerized simulations are less able to assess performance in areas such as interpersonal communication skills unless expensive interactive programs are developed(69). WRITTEN/COMPUTER-BASED EXAMINATIONS
Written examinations have been frequently described in pharmacy education literature as a means of evaluating experiential student performance(10,71,72). Although this method is usually reliable and can evaluate the student’s knowledge about a broad sample of clinical problems, it is only an indirect measure of how the student would perform when providing routine patient care. This assessment method should be weighted the least since written examinations also reward reading and studying for the tests over active participation in the patient care unit(73). To assess whether the student “knows how,” the written test for an experiential rotation should test problem-solving and clinical judgment skills rather than just recall of factual information(74-77). This is more likely to be achieved by using Extended-Matching Items (EMIs) as an alternative to traditional test questions(77-81). In contrast to typical multiple choice questions, with EMIs there are 6-25 possible answers and any number of them may be correct. The National Board of Medical Examiners is currently using this format and believes it is: (i) easier to score than freeresponse questions; (ii) more frequently tests problemsolving abilities; and (iii) decreases the likelihood of the student guessing the correct answer(79). These examiners also advocate computer-based tests consisting of EMIs and patient simulations to better assess whether the student can “show how” and “knows how”. OTHER EES COMPONENTS
Self-assessment procedures are examples of an assessment method that is valuable but, because of the limitations noted in Table I, should only supplement the primary methods. During a rotation, the student and instructor must work together so that deficiencies can be identified and corrected during the learning period. Student self-assessments can serve as catalysts for such discussion and are therefore, particularly valuable in formative evaluations. Self-assessment also promotes development of lifelong learning skills(82-91). Although self-assessment procedures are valuable components of the EES. pharmacy educators should use these data carefully since medical students’ self-assess-
ments have correlated weakly with expert judgments(84,87,91). As one means of self-assessment, students can document their contributions to patient care and assess the quality and significance of their interventions(92-95). Such information provides the instructor and student with data that may be used to address whether the student is being exposed to good learning experiences and the extent to which the student is actively involved in patient care. This information also provides evidence to pharmacy educators and other stakeholders that a student has received an appropriate learning experience. In addition, these data emphasize to the student the outcomes expected of a practitioner. The student’s interventions would be valuable data to include in a portfolio. Portfolios have recently received recognition by educators as a mechanism for promoting reflective thinking and self-assessment and documenting learning achievements(96-99). Assembly of a portfolio requires the student to select examples of their work and reflect on what learning these examples represent. Since these skills are essential for lifelong learning, we believe they should be assessed during pharmacy practice experiences. Educators who incorporate portfolios as a component of their EES should provide students with structured guidelines about how to assemble one. These guidelines ensure the assembled materials provide insightful documentation for both the student and the program. However, further research is needed to establish the psychometric properties of portfolios and how to assign a grade for this work. In addition to assessment methods and guidelines for their optimal use, an EES should provide mechanisms for documenting and communicating student performance. These mechanisms can minimize any consequences that may result from performance decisions. The following student evaluation issues particularly require effective documentation and communication procedures. STUDENT EVALUATION ISSUES
Nothing illuminates the quality of a school’s EES more than the “problem student”(100-105). Performance deficiencies may present as either cognitive or noncognitive problems. Noncognitive problems such as poor interpersonal skills and non-assertive/shy behavior have been noted by medical educators to be the most difficult to act upon(2,100). Faculty members are sometimes reluctant to make negative decisions about a student’s rotation performance due to fear of legal consequences(38,106-111). However, instructor decisions about student performance have been upheld by the courts as long as the student has received due process. Based on an assessment of court cases, this is true for both tenure-track and affiliate faculty members(111). To assure due process, the experiential instructor must inform the student about rotation expectations at the beginning of the experience and communicate performance deficiencies early in the rotation(2,38.39). Due process also requires that the school’s EES includes a mechanism by which students can appeal the faculty member’s decision. Table II outlines the responsibilities of various faculty members in communicating performance assessments with students. For example, faculty members should provide students with early warning about any deficiencies(2). An important component of this process is to provide the student with feedback about their performance. Therefore.
American Journal of Pharmaceutical Education Vol. 59, Fall 1995
241
Table I. Evaluation methods used to evaluation clinical performance and the level of ability they most effectively measure Levels of ability based on Miller’s Model(3) Good interrater Low Good Feasability/ Knows Shows Method reliability generalizability ease of use Knows Does how cost how Written + ± 0 0 + + + + Exams, MCQs, EMIs, 75-80 Oral Exams + + 0 0 ± + + + 11, 126, 132 Triplejump + ± 0 ± 0 + ± 133 Chart/ Written + + ± 0 ± + + + simulations/ latent image Developer® A.B.Dick Co. 125, 128, 129 Computer based Exam with EMIs 75,134 Clinical Evaluation Exercise 135, 136 Videotape Reviews 137, 138 Standardized Patients 44, 65 Observations 71, 72, 121, 127, 138-149 Patient Conference/ Presentation 9, 10, 72, 150 Consultation letters/medical history write-ups 151, 152 Chart audits 153, 154 Peer Assessments 36-40, 155 Self- assessment/ portfolios 82-92, 97, 156, 157
+
+
±
0
+
+
±
+
+
0
0
0
+
+
+
0
±
±
+
±
+
?
+
+
0
0
+
±
±
±
0
0
±
0
+
+
+
±
±
+
±
+
±
±
+
±
+
±
±
+
±
±
±
±
+
±
0
0 = Weak; ± = Sometimes Weak; + = Strong a Predicts how effective the student solves patient problems of varying type and difficulty.
242
American Journal of Pharmaceutical Education Vol. 59, Fall 1995
Table II. Recommendations for achieving a functional EES School Faculty Responsibilities * Establish the school’s EES (Figure 6) • Establish overall goals and objectives for rotations. • Implement valid evaluation methods and adopt them across all program rotations. • Conduct rater training programs that emphasize accuracy (RAT) and effective use of cognitive processing skills. • Provide feedback to faculty about their rating skills. • Appoint the experiential program director as EES manager. • Charge the “Performance Committee” with the responsibility of acting upon evaluations of marginal/ deficient students. * Establish policies for: • Identifying and managing students with poor performance • Due process • Dismissal of failing students. Experiential Program Director Responsibilities • Ensure that experiential faculty have completed the school’s rater training program. • Communicate evaluation policies and procedures to faculty and students. • Ensure that performance evaluations are completed at the appropriate times and in a timely manner. • Coordinate determination of the final grade based on the instructor’s observation-based ratings and other assessment results. • Monitor the quality of the EES and communicate assessments to the school faculty. • Develop a course/experiential program portfolio for accountability of program learning(96). • Arrange for/serve as a “gold-standard” rater. Individual Experiential Instructor Responsibilities • Communicate rotation objectives, responsibilities, and evaluation methods both verbally and in writing at the beginning of the rotation. • Plan time for directly observing the student in the practice setting. • Provide feedback about performance throughout the rotation. • If the student exhibits deficiencies, communicate this to the student before the midpoint of the rotation. • Involve the student in making self-assessments during the rotation, discussing these assessments and comparing them to instructor assessments. • Document communications to the student about deficiencies. Include their self-assessments in this documentation. • Involve a peer faculty member as a “Gold-Standard Rater” when a student has questionable performance.
Table III outlines recommendations for providing feedback effectively(112-118). Faculty members often debate whether they should share past performance assessments with those who teach the student in current or upcoming rotations. Supporters suggest that sharing of this information promotes the elucidation of strengths and weaknesses early in the rotation. However, opponents point out that such information may bias the instructor’s assessment of student performance and breach student confidentiality. Medical educators have recommended that faculty share only problems related to mastery of program competencies with subsequent faculty(119). Furthermore, faculty should
Table III. Strategies for providing effective feedback General Recommendations on Providing Feedback • Feedback should be provided by the preceptor directly involved in supervising and rating the student. • Feedback should be given when alone with the student and in a setting conducive to discussing the performance. Recommended Procedures for Conducting a Feedback Session with the Student 1. Feedback should be undertaken with the preceptor and student working as allies-with common goals: • Begin the session by establishing a mutual goal of assessing the student’s performance and identifying ways to enhance the desired skills. 2. Then, seek the student’s input about their perceived performance. 3. Start the session by emphasizing positive attributes; Then, discuss negative attributes. • Alternatively, sandwich negative feedback between positive feedback. 4. Provide specific examples of incidents (anecdotal records) that lead you to assign a given score. 5. Feedback should be: • Based on first-hand data • Regulated in quantity • Limited to behaviors that are remediable • Phrased in descriptive non-evaiuable language • Focused on the performance ... not the performer (i.e., student) • Sensitive to the student’s self-esteem 6. Encourage the student to respond and discuss his/her feelings about your assessment. 7. In collaboration, develop a “prescription” that will help the student perform appropriately. 8. Feedback should include examples of correct performance/how the skills may be correctly performed. 9. End the session by jointly agreeing when another feedback session will be held. The major goal of this session will be to assess whether the prescription has been effective.
encourage the student to communicate this information. The student also must have the right to appeal the process. Medical educators further recommend that the student’s advisor should coordinate communicating this information. To minimize bias due to knowledge of previous assessment data, we recommend dissemination of this information after the new instructor has had time to observe the student and form one’s own initial judgments. The faculty should also encourage the student to share one’s self-assessment documents with the rotation instructor. Another rating issue concerns individual faculty members who resist using the school’s standardized assessment instrument. Although such a faculty member may have “their own” assessment instruments and desire to use them, this presents significant problems. First, although the method may seem very lucid to an individual instructor, it may not be similarly interpreted by other program faculty members. Furthermore, when a variety of rating forms are used during the rotation sequence, it is difficult to track students on a longitudinal basis, assess growth across rotations, and evaluate achievement of specific educational outcomes. Use of a standardized instrument across rotations provides data for longitudinally tracking student performance. In some specialized rotations, such as a drug information clerkship, unique assessment methods are needed. However, if the school’s standardized instrument is well designed, some of
American Journal of Pharmaceutical Education Vol. 59, Fall 1995
243
Fig. 6. An example of an experiential evaluation system(2). Adapted with permission from Academic Medicine.
its items should have applicability to such a rotation. If some items are not applicable to a given rotation, the instructor should have an option to indicate such when completing the instrument. DISCUSSION
Pharmaceutical care requires the pharmacist to perform complex functions that involve the integration of knowledge, skills, and clinical judgment(120). As evidenced by this review of literature, it is not an easy task to assess whether the pharmacy student “shows how” and “does” pharmaceutical care. Furthermore, the methods most appropriate for measuring these levels (i.e., observation-based ratings and simulations) are costly in terms of faculty time and financial resources. All assessment methods have limitations. Therefore, a combination of observation-based ratings, simulations, and written exams or computer-based tests with EMIs are recommended. To date, most experiential assessment methods described by pharmacy educators either lack psychometric data inferring accuracy or assess lower levels of learning such as what a student “knows” or “knows how” (69,71,72,121-127). Pharmacy educators have described use of written simulations or OSCEs but few schools have adopted these assessment techniques as part of their EES(10,70,128-130). A major factor contributing to this lack of data is limited resources at individual schools. This paper has identified how experiential instructors can select the best combination of assessment methods and enhance rating accuracy. However, an effective EES, will require collaboration of schools at the national level, and cooperation of faculty members, at the school level. Pharmacy leaders at the national level have accomplished an important step toward improving student/practitioner assessment by establishing the Center for the Advancement of Pharmaceutical Education (CAPE)(130). CAPE is encouraged to promote consortia programs since regional sharing of resources, such as training programs, will minimize any logistical problems. Based on the experience of medical educators, sharing of resources is necessary in order to develop psychometrically sound assessment methods. This strategy will also promote greater consistency in perfor244
mance outcomes among our pharmacy graduates. Significant strides are needed at the school level to ensure accountability. A school’s faculty, as a whole, should be responsible for the effectiveness of their EES. Table II and Figure 6 delineate a model EES(2,131). The individual experiential instructor is accountable for rating with accuracy and making a sufficient number of observations. However, several actions should be taken by the collective faculty at a school to make this responsibility possible. For example, rater training should be provided and required of all faculty members (both full-time and affiliate) who teach experiential rotations. To promote a sufficient number of observations, the school faculty must also evaluate whether the length of rotations is of appropriate duration. This paper has established the importance of making observations during the “steady-state” period of a student’s rotation and accumulating a sufficient number of them. Increasing this window of time should allow more opportunities for making “steadystate” observations. Hunt et al.(2), has cautioned medical educators that rotations, which are four weeks or less in duration, may compromise the effectiveness of their school’s evaluation system and the students’ learning. There must be a “gate-keeper” who monitors the effectiveness and identifies weaknesses of the school’s EES. The authors recommend that the faculty should appoint this responsibility to the school’s experiential program director. The experiential director is positioned to promote early recognition of performance problems and ensure due process. This individual should be accountable for the tracking of student information as the learner progresses through the rotation sequence and bringing problem-students to the attention of program faculty and/or a committee established to address performance deficiencies. Such information could be communicated to faculty in an “Annual Report of Experiential Evaluation.” CONCLUSIONS
Pharmacy educators can no longer ignore the need for assessment methods that assess whether the student “shows how” and “does” pharmaceutical care. Observation-based ratings most appropriately assess whether the student “does” and simulations can assess whether the student “shows how.” Experiential instructors must enhance their observation-based rating skills to accurately evaluate whether the student “does.” At the national level, pharmacy educators must collaborate and pool resources to develop psychometrically sound observation-based assessment methods and simulations, such as OSCEs. Finally, at the school level, a faculty must recognize its role and assume responsibility for assuring that the EES is effective. Only then will pharmacy educators feel confident about their graduates’ performance in the practice setting. Am. J. Pharm. Educ., 59, 236-247(1995); received 4/18/94, accepted 6/8/95. References (1) Beck, D.E., O’Sullivan, P.S. and Boh, L.E.,” Increasing the accuracy of observer ratings by enhancing cognitive processing skills,” Am. J. Pharm. Educ., 59, 228-235(1995). (2) Hunt, D.D., “Functional and dysfunctional characteristics of the prevailing model of clinical evaluation systems in North American medical schools,” Acad. Med., 67, 254-259(1992). (3) Berner, E.S. and Bender, K.J., “Determining how to begin,” In Evaluating Clinical Competence in the Health Professions, (edit. Morgan, M.K., and Irby, D.M.) C.V. Mosby Company, St. Louis
American Journal of Pharmaceutical Education Vol. 59, Fall 1995
MO (1978) pp. 3-10. Miller, G.E., “The assessment of clinical skills/competence/performance,” Acad. Med., 9(Suppl), S63-S67(1990). (5) Bloom, B.S., Taxonomy of Educational Objectives. Handbook I: Cognitive Domain, David McKay Co., Inc., New York NY (1956). (6) Kane, M.T., “The assessment of professional competence,” Eval. Health Prof., 15, 163-181(1992). (7) Risucci, D.A. and Tortolani, A.J., “A methodological framework for the design of research on the evaluation of residents,” Acad. Med., 65, 36-41(1990). (8) Frye, A.W., Hoban, J.D. and Richards, B.F.,” Capturing the complexity of clinical learning environments with multiple qualitative methods,” Eval. Health Prof., 16, 44-60(1993). (9) Beck, D.E. and Clayton, A.G., “Validity and reliability of an instrument that evaluates PharmD student performance during a patient presentation,” Am. J. Pharm. Educ., 54, 268-274(1990). (10) Engel, G.L., “The deficiencies of the case presentation as a method of clinical teaching,” N. Engl. J. Med., 284, 20-24(1971). (11) Rowland, P. A., Burchard, K.W., Grab, J.L. and Coe, N.P., “Influence of effective communication by surgery students on their oral examination scores,” Acad. Med., 66, 169-171(1991). (12) Boh, L.E., Pitterle, M.E., Schneider, F. and Collins, C.L., “Survey of experiential programs: Course competencies, student performance and preceptor/site characteristics,” Am. J. Pharm. Educ., 55, 105— 113(1991). (13) Pulakos, E.D., “A comparison of rater training programs: Error training and accuracy training,” J. Appl.Psychol, 69, 581-588(1984). (14) Wexley, K.N. and Klimoski, R. “Performance appraisal: An update,” in Performance Evaluation, Goal setting, and Feedback, (edits. Ferris, G.R. and Rowland, K.M.) JAI Press, Inc., Greenwich CT (1990) pp. 1-45. (15) DeNisi, A.S. and Williams, K.J., “Cognitive approaches to performance appraisal,” in Performance Evaluation, Goal Setting, and Feedback, (edits. Ferris, G.R. and Rowland, K.M.) JAI Press, Inc., Greenwich CT (1990) pp. 47-93. (16) Landy, F.J. and Farr, J.L., The Measurement of Work Performance —Methods, Theory, and Applications. Academic Press, New York, NY (1983) pp. 22-23. (17) Butterworth, J.S. and Reppert, E.H., “Auscultatory acumen in the general medical population,” JAMA, 174, 114-116(1960). (18) Seegal, D. and Wertheim, A.R., “On the failure to supervise students’ performance of complete physical examinations,” ibid., 180, 476-477(1962). (19) Hinz, C.F., “Direct observation as a means of teaching and evaluating clinical skills,” J. Med. Educ., 41, 150-161(1966). (20) Engel, G.L.,” Are medical schools neglecting clinical skills, “JAMA, 236, 861-63(1976). (21) Stillman, P.L., May, J.R. and Meyer, D.M., “A collaborative effort to studymethodsofteachingphysicalexaminationskills,”J. Med. Educ., 56, 301-306(1981). (22) Wray, N.P. and Friedland, J. A., “Detection and correction of house staff error in physical diagnosis,” JAMA, 249, 1035-1037(1983). (23) Barker, K.N., “Data collection techniques: observation,” Am. J. Hosp. Pharm., 37, 1235-1243(1980). (24) Thorndike R.L. and Hagen, E.P., Measurement and Evaluation in Psychology and Education, 4th ed., John Wiley & Sons. New York NY (1977) pg. 506-534. (25) DeMers, J.L., “Observational assessment of performance,” inEvaluating Clinical Competence in the Health Professions, (edits. Morgan, M.K. and Irby, D.M.) C.V. Mosby, St. Louis MO (1978) pp. 89-115. (26) Sax, G., Principles of Educational and Psychological Measurement and Evaluation, 2nd ed., Wadsworth Publishing Co., Belmont CA (1980) pp. 145-168. (27) Srinivas, S. and Motowidlo, S.J., “Effects of raters’ stress on the dispersion and favorability of performance ratings,” J. Appl. Psychol., 72, 247-251(1987). (28) Friedman, M. and Mennin, S.P., “Rethinking critical issues in performance assessment,” Acad. Med., 66, 390-395(1991). (29) Shavelson, R. J. and Webb, N.M., Generalizability Theory: A Primer, Sage Publications, Newbury Park CA (1991) pp. 1-16. (30) Anon., “Professional judgment in evaluation,” J. Nurs. Educ., 33, 243(1994). (31) Green, R.L., “Sources of recency effects in free recall,” Psychol. Bull, 99, 221-228(1986). (32) Kaplan, M.F., “Stimulus inconsistency and response dispositions in forming judgments of other persons,” J. Personality Soc. Psychol. 22, 58-64(1973). (33) Borman, W.C., “Individual differences correlates of accuracy in (4)
(34)
(35)
(36)
(37) (38) (39) (40) (41) (42) (43) (44) (45) (46) (47)
(48) (49) (50) (51) (52) (53) (54) (55)
(56) (57) (58)
evaluating others’ performance effectiveness,” Appl. Psychol. Meas., 3, 103-115(1979). Cardy, R.L. and Kehoe, J.F., “Rater selective attention ability and appraisal effectiveness: The effect of a cognitive style on the accuracy of differentiation among ratees,” J. Appl. Psychol., 69, 589594(1984). Littlefield, J.H., “Developing and maintaining a resident rating system,” in How to Evaluate Residents, (edits. Lloyd, J.S. and Langlsey, D.G.) American Board of Medical Specialties, Chicago IL (1986) pp. 117-130. Woolliscroft, J.O., Howell, J.D., Patel, B.P. and Swanson, D.B., “Resident-patient interactions: The humanistic qualities of internal medicine residents assessed by patients, attending physicians, program supervisors, and nurses,” Acad. Med., 69, 216-224(1994). Seefeldt, M. and Blumberg, P., “Rating dental students-a comparison of faculty and patient perspectives,” Eval. Health Prof, 7, 365374(1984). Butterfield, P.S., Mazzaferri, E.L. and Sachs, L.A., “Nurses as evaluators of the humanistic behavior of internal medicine residents,” J. Med. Educ., 62, 842-849(1987). Cope, D.W., Linn, L.S., Leahe, B.D. and Barrett, P.A., “Modification of residents’ behavior by preceptor feedback of patient satisfaction,”J. Gen. Int. Med., 1, 394-8(1986). Linn, L.S., DiMatteo, M.R., Cope, D.W. and Robbins, A., “Measuring physicians’ humanistic attitudes, values and beliefs,” Med. Care, 25, 504-515(1987). Banks, C.G. and Murphy, K.R., “Toward narrowing the researchpractice gap in performance appraisal,” Personnel Psychol., 38, 335345(1985). Cohen, G.S., Blumberg, P., Ryan, N.C. and Sullivan, P., “Do final grades reflect written qualitative evaluations of student performance?,” Teach. Learn. Med., 5, 10-15(1993). DaRosa, D.A., “Commentary on “Do final grades reflect written qualitative evaluations of student performance?,” ibid., 5, 1617(1993). Miller, G.M., “Commentary on Assessment of clinical skills with standardized patients: State of the art,” ibid., 2, 77-78(1990). van der Vleuten, C.P.M. and Swanson, D.B., “Assessment of clinical skills with standardized patients: State of the art,” ibid., 2, 5876(1990). Stillman, P.L., Swanson, D.B. and Smee, S., et al, “Assessing clinical skills of residents with standardized patients,” Ann. Intern. Med., 105, 762-771(1986). Vu, N.V., Barrows, H.S., Marcy, M.L., et. al., “Six years of comprehensive, clinical, performance-based assessment using standardized patients at the Southern Illinois University School of Medicine,” Acad. Med., 67, 42-50(1992). Anderson, M.B., Stillman, P.L. and Wang, Y., “Growing use of standardized patients in teaching and evaluation in medical education,” Teach. Learn. Med., 6, 15-22(1994). Tamblyn, R., Abrahamowicz, M., Schnarch, B., et. al, “Can standardized patients predict real-patient satisfaction with the doctorpatient relationship?,” ibid., 6, 36-44(1994). Colliver, J. A., Marcy, M.L., Vu, N.V., et al., “Effect of using multiple standardized patients to rate interpersonal and communication skills on intercase reliability,” ibid., 6, 45-8(1994). Shatzer, J.H., Wardrop, J.L., Williams, R.G. and Hatch, T.F., “Generalizability of performance on different-station-length standardized patient cases,” ibid., 6, 54-58(1994). Barrows, H.S., An overview of the uses of standardized patients for teaching and evaluating clinical skills,” Acad. Med., 68, 443451(1993). Colliver, J.A. and Williams, R.G., “Technical issues: Test application,” ibid., 68, 454-460(1993). Miller, G.E., “Conference summary,” ibid., 68, 471-474(1993). Finke, L., Messmer, P., Spruck, M., et al., “Measuring clinical skills via computer for RNs in a BSN-MSN educational mobility program,” in Measurement of Nursing Outcomes. Volume Three: Measuring clinical skills and Professional Development in Education and Practice, (edits. Waltz, C.F. and Strickland, O.L.) Springer Publishing Co., New York NY (1990) pp. 44-53. Ross, M., Carroll, G., Knight, J., et al., “Using the OSCE to measure clinical skills performance in nursing,” J. Adv. Nurs., 13, 45-56(1988). Bramble, K., “Nurse practitioner education: Enhancing performance through the use of the objective structured clinical assessment,” J. Nurs. Educ., 33, 59-65(1994). Tamblyn, R.M., Abrahamowicz, M., Berkson, L. et al., “Assessment of performance in the office setting with standardized patients,”
American Journal of Pharmaceutical Education Vol. 59, Fall 1995
245
Acad. Med., 10(Suppl), S22-S24(1992). (59) Noguchi, J. and Hill-Besinque, K.,” The use of standardized patients in an objective structured clinical examination,” Am. J. Pharm. Educ., 56(Suppl), 83S(1992). (60) Fielding, D.W., Page, G.G., Schulzer, M., et al., “Developing objective structured clinical examinations for performance evaluation in pharmacy,” ibid., 57(Suppl), 101S(1993). (61) Guidry, T.D. and Cohen, P. A.,” A practical examination for student assessment in an externship program,” ibid., 51, 280-284(1987). (62) King, A.M., Perkowski-Rogers, L.C. and Pohl, H.S., “Planning standardized patient programs: Case Development, patient training, and costs,” Teach. Learn. Med., 6, 6-14(1994). (63) Reznick, R.K., Smee, S., Baumber, J.S., et al., “Guidelines for estimating the real cost of an objective structured clinical examination,” Acad. Med., 68, 513-517(1993). (64) Morrison, L. J. and Barrows, H.S., “Developing consortia for clinical practice examinations: The Macy Project,” Teach. Learn. Med., 6, 23-27(1994). (65) Clem, J.R., Murry, D.J., Perry, P.J., et al, “Performance in a clinical pharmacy clerkship: computer-aided instruction versus traditional lectures,” Am. J. Pharm. Educ., 56, 259-263(1992). (66) Greer, M.L., Kirk, K.W. and Crismon, M.L., “The development and pilot testing of a community-practice, clinical pharmacy decisionmaking assessment instrument,” Eval. Health Prof., 12, 207231(1989). (67) Boh, L.E., Pitterle, M.E., Wiederholt, J.B. and Tyler, L.S., “Development and application of a computer simulation program to enhance the problem-solving skills of students,” Am. J. Pharm. Educ., 51, 253-261(1987). (68) MacKinnon, G.E., Pitterle, M.E., Boh, L.E. and DeMuth, J.E., “Computer-based patient simulations: Hospital pharmacists’ performance and opinions,” Am. J. Hosp. Pharm., 49, 2740-2745(1992). (69) McKenzie, M.W., Kimberlin, C.L., Philips, E.W., et al, “Content, preparation, and formative evaluation of an interactive videodisc system to enhance communication skills in pharmacy students,” Am. J. Pharm. Educ., 57, 230-238(1993). (70) Tobias, D.E., Speedie, S.M. and Kerr, R. A., “Evaluation of clinical competence through written simulations,” ibid., 42, 320-323(1978). (71) Nelson, A.A., Bober, K.F. and Bashook, P.G., “Evaluation of a college-structured practical experience program,” ibid., 40, 232236(1976). (72) Elenbaas, R.M., “Evaluation of students in the clinical setting,” ibid., 40, 410-417(1976). (73) Barrows, H.S., “The scope of clinical education,”J. Med. Educ., 61, 23-33(1986). (74) Elstein, A.S., “Beyond multiple-choice questions and essays: The need for a new way to assess clinical competence,” Acad. Med., 68, 244-249(1993). (75) Dillon, G.F. and Clyman, S.G.,”The computerization of clinical science examinations and its effect on the performances of thirdyear medical students,” ibid., 67, S66-S68(1992). (76) Richards, B.F., Philp, E.B. and Philp, J.R., “Scoring the objective structured clinical examination using a microcomputer,” Med. Educ., 23, 376-380(1989). (77) Case, S.M. and Swanson, D.B.,”Extended-matching items: A practical alternative to free-response questions,” Teach. Learn. Med., 5, 107-115(1993). (78) Van Susteren, T.J., Cohen, E.B., Simpson, D.E., “Alternate-choice test items: Implications for measuring clinical judgment,” ibid., 3, 33-37(1991). (79) Blackwell, T. A., Ainsworth, M.A., Dorsey, N.K., et al.,” A comparison of short-answer and extended-matching question scores in an objective structured clinical exam,” Acad. Med., 66(Suppl 9). S40S42(1991). (80) Ainsworth, M.A., Turner, H.E., Solomon, D.J., Callaway, M.R., “Innovation in conducting and scoring a clerkship objective structured clinical examination,” Teach. Learn. Med., 6, 64—67(1994). (81) Abrahamson, S., “Assessment of student clinical performance,” Eval. Health Prof., 8, 413-427(1985). (82) Gordon, M.J., “Self-assessment programs and their implications for health professions training,” Acad. Med., 67, 672-679(1992). (83) Gordon, M.J., “A review of the validity and accuracy of selfassessments in health professions training,” ibid., 66, 762-769(1991). (84) Arnold, L., Willoughby, T.L. and Calkins, E.V., “Self-evaluation in undergraduate medical education: A longitudinal perspective,” J. Med. Educ., 60, 21-28(1985). (85) O’Sullivan, P.S., Pinsker, J. and Landau, C, “Evaluation of strategies selected by residents: The roles of self-assessment, training 246
level, and sex,” Teach. Learn. Med., 3, 101-107(1991). (86) Abbott, S.D., Carswell, R., McGuire, M. and Best, M., “Selfevaluation and its relationship to clinical evaluation,” J. Nurs. Educ., 27, 219-224(1988). (87) Calhoun, J.G., Woolliscroft, J.O., Ten Haken, J.D., et al, “Evaluating medical student clinical skill performance — Relationships among self, peer, and expert ratings,” Eval. Health Prof, 11, 201212(1988). (88) Calhoun, J.G., Ten Haken J.D. and Woolliscroft, J.O., “Medical students’ development of self- and peer-assessment skills: A longitudinal study,” Teach. Learn. Med., 2, 25-29(1990). (89) Herbert, W.N., McGaghie, W.C., Droegemueller, W., et at, “Student evaluation in obstetrics and gynecology self- versus departmental assessment,” Obstet. Gynecol, 76, 458-461(1990). (90) Fincher, R.M. and Lewis, L.A., “Learning, experience, and selfassessment of competence of third-year medical students in performing bedside procedures,” Acad. Med., 69, 291-295(1994). (91) Woolliscroft, J.O., TenHaken, J., Smith, J. and Calhoun, J.G., “Medical students’ clinical self-assessments: comparisons with external measures of performance and the students’ self-assessments of overall performance and effort,” ibid., 68, 285-294(1993). (92) Briceland, L.L., Kane, M.P. and Hamilton, R.A., “Evaluation of patient-care interventions by PharmD clerkship students,” Am. J. Hosp. Pharm., 49, 1130-1132(1992). (93) Mason, R.N., Pugh, C.B., Boyer, S.B. and Stiening, K.K., “Computerized documentation of pharmacists’ interventions,” ibid., 51, 21312138(1994). (94) Brown, G., “Assessing the clinical impact of pharmacists’ interventions,” ibid., 48, 2644-2647(1991). (95) Kane, M.P., Briceland, L.L. and Hamilton, R.A., “Solving drugrelated problems in the professional experience program,” Am. J. Pharm. Educ., 57, 347-351(1993). (96) Arter, J. and Spandel, V., “NCME instructional module: Using portfolios of student work in instruction and assessment,” Educational Meas.: Issues and Practice, 11, 36-44(1992). (97) Jensen, G.M. and Saylor, C. “Portfolios and professional development in the health professions,” Eval. Health Prof., 17, 344-357(1994). (98) Schon, D., “The theory of inquiry: Dewey’s legacy to education,” Curriculum Inquiry, 22, 119-140(1992). (99) Wolf, K., Bixby, J., Glen J. and Gardner, H., “To use their minds well: Investigating new forms of student assessment,” Rev. Res. Educ., 17, 31-73(1991). (100) Metheny, W.P. and Carline, J.D., “The problem student: A comparison of student and physician views,” Teach. Learn. Med., 3, 133137(1991). (101) Tonesk, X., “Clinical judgment of faculties in the evaluation of clerks,” J. Med. Educ., 58, 213-214(1983). (102) Tonesk, X. and Buchanan, R.G., “Faculty perceptions of current evaluation systems,” ibid., 60, 573-576(1985). (103) Tonesk, X. and Buchanan, R.G., “An AAMC pilot study by 10 medical schools of clinical evaluation of students,” ibid., 62, 707718(1987). (104) Edwards, J.C., Plauche, W.C., Vial, R.H., et al., “Two ways to assess problems of evaluating third-year and fourth-year medical students: A case study,” Acad. Med., 64, 463-467(1989). (105) Hunt, D.D., Carline, J., Tonesk, X., et al, “Types of problem students encountered by clinical teachers of clerkships,” Med. Educ., 23, 14-18(1989). (106) Orchard, C.O., “The nurse educator and the nursing student: A review of the issue of clinical evaluation procedures,” J. Nurs. Educ., 33, 245-255(1994). (107) Irby, D.M. and Milam, S., “The legal context for evaluating and dismissing medical students and residents,” Acad. Med., 64, 639643(1989). (108) Helms, L.B. and Helms, C.M., “Forty years of litigation involving medical students and their education: I. General educational issues,” ibid., 66, 1-7(1991). (109) Helms, L.B. and Helms, C.M., “Forty years of litigation involving residents and their training: I. General programmatic issues,” ibid., 66, 649-655(1991). (110) Helms, L.B. and Helms, C.M., “Forty years of litigation involving residents and their training: II. Malpractice issues,” ibid., 66, 718725(1991). (111) Fassett, W.E. and Olswang, S.G., “Affiliate faculty and student evaluation, discipline and dismissals,” Am. J. Pharm. Educ., 55, 211218(1991). (112) Klein, R.H. and Babineau, R., “Evaluating the competence of trainees: It’s nothing personal,” Am. J. Psychiatry., 131, 788-
American Journal of Pharmaceutical Education Vol. 59, Fall 1995
791(1974). (113) Brinko, K.T., “The practice of giving feedback to improve teaching. What is effective?” J. Higher Educ., 64, 574-593(1993). (114) Baron, R.A., “Negative effects of destructive criticism: Impact on conflict, self-efficacy, and task performance,” J. Applied Psychol., 73, 199-207(1988). (115) Ilgen, D.R., Fisher, C.D. and Taylor, M.S., “Consequences of individual feedback on behavior in organizations,” ibid., 64, 349371(1979). (116) Rezler, A.G., Anderson, A.S., “Focused and unfocused feedback and self-perception,” J Educational Res., 65, 61-64(1971). (117) Wigton, R.S., Patil, K.D. and Hoellerich, V.L., “The effect of feedback in learning clinical diagnosis,” J. Med.Educ., 61, 816-822(1986). (118) Davies, D. and Jacobs, A., “Sandwiching complex interpersonal feedback,” Small Group Behavior, 16, 387-396(1985). (119) Cohen, G.S. and Blumberg, P., “Investigating whether teachers should be given assessments of students made by previous teachers,” Acad. Med., 66, 288-289(1991). (120) Commission to Implement Change in Pharmaceutical Education, “Background paper II: Entry-level, curricular outcomes, curricular content and educational process,” Am. J. Pharm. Educ., 57, 377385(1993). (121) Andrew, B.J., “A technique for assessment of pharmacy students skills in patient interviewing,” ibid., 37, 290-299(1973). (122) Warner, D., “Evaluation of a pilot pharmacy clerkship,” ibid., 34, 256-264(1970). (123) Speranza, K.A., “A diary study of the socialization process in a hospital clinical experience course,” ibid., 39, 24-30(1975). (124) McKenzie, M. W., Johnson, S.M. and Bender, K.J., “A competencybased, serf-instructional module on medication history interviewing for pharmacy students: Rationale, description and formative evaluation,” ibid., 41, 133-142(1977). (125) Tobias, D.E., Michocki, R.J. and Edmondson, W.H., “Evaluation of students in a competency-based undergraduate clinical clerkship,” ibid., 42, 31-34(1978). (126) Pancorbo, S., Holloway, R.L., McNeiley, E. and McCoy, H.G., “Development of an evaluative procedure for clinical clerkships,” ibid., 44, 12-16(1980). (127) Friend, J.R., Wertz, J.N., Hicks, C. and Billups, N.F., “A multifacetted approach to externship evaluation,” ibid., 50, 111-126(1986). (128) Roffman, D.S., Tobias, D.E. and Speedie, S.M., “Validation of written simulations as measures of problem solving for pharmacy students,” ibid., 44, 16-24(1980). (129) Carter, R.A., Bennett, R.W., Black, C.D. and Blank, J.W., “Use of simulations as performance standards for external degree doctor of pharmacy students,” ibid., 49, 119-123(1985). (130) Trinca, C.E., “The center for the advancement of pharmaceutical care education: Translating commitment and concern to the community,” ibid., 57(Suppl),7S-9S(1993). (131) Short, J.P., “The importance of strong evaluation standards and procedures in training residents,” Acad. Med., 68, 522-525(1993). (132) Zelenock, G.B., Calhoun, J.G., Ffockman, E.M. and Youmans, L.C., “Oral examinations: Acutal and perceived contributions to surgery clerkship performance,” Surgery, 97, 737-743(1985). (133) Smith, R.M., “The triple-jump examination as an assessment tool in the problem-based medical curriculum at the University of Hawaii,” Acad. Med., 68, 366-372(1993). (134) Wong, J., Wong, S. and Richard, J., “Implementing computer simulations as a strategy for evaluating decision-making skills of nursing students,” Comput. Nurs., 10, 264-269(1992). (135) Kroboth, F.J., Hanusa, B.H. and Parker, S., “The inter-rater reliability and internal consistency of a clinical evaluation exercise,” J. Gen. Intern. Med., 7, 174-179(1992).
(136) Dunnington, G., Reisner, L., Witzke, D. and Fulginiti, J., “Structured single-observer methods of evaluation for the assessment of ward performance on the surgical clerkship,” Am. J. Surg., 159, 423426(1990). (137) Premi, J., “An assessment of 15 years’ experience in using videotape review in a family practice residency,” Acad. Med., 66, 56-57(1991). (138) Rassin, G.M. and McCormick, D.P., “Use of video review to evaluate house staff performance of well-baby examinations: A preliminary investigation,” Teach. Learn. Med.,4, 168-172(1992). (139) Anderson, E., “Co-assessment as a unique approach to measuring students’ clinical abilities,” J. Nurs. Educ., 29, 42-43(1990). (140) Littlefield, J.H., Harrington, J.T., Anthracite, N.E. and Garman, R.E., “A description and four-year analysis of a clinical clerkship evaluation system,” J. Med. Educ., 56, 334-340(1981). (141) Noel, G.L., Herbers, J.E., Caplow, M.P., et al., “How well do internal medicine faculty members evaluate the clinical skills of residents?,” Ann. Intern. Med., 117, 757-765(1992). (142) Kaiet, A., Earp, J.A. and Kowlowitz, V., “How well do faculty evaluate the interviewing skills of medical students?,” J. Gen. Intern. Med., 7, 499-505(1992). (143) Cariine, J.D., Paauw, D.S., Thiede, K.W., and Ramsey, P.G., “Factors affecting the reliability of ratings of students’ clinical skills in a medicine clerkship,” ibid., 7, 506-510(1992). (144) Dielman, T.E., Hull, A.L. and Davis, W.K., “Psychometric properties of clinical performance ratings,” Eval. Health Prof., 3, 103117(1980). (145) Boodoo, G.M. and O’Sullivan, P.S., “Assessing pediatric clerkship evaluations using generalizability theory,” ibid, 9, 467-486(1986). (146) Boodoo, G.M. and O’Sullivan, P., “Obtaining generalizability coefficients for clinical evaluations,” ibid., 5, 345-358(1982). (147) O’Donohue, W.J. and Wergin J.F., “Evaluation of medical students during a clinical clerkship in internal medicine,” J. Med. Educ., 53, 55-58(1978). (148) Wood, V., “Evaluation of student nurse clinical performance — A continuing problem,” Int. Nurs. Rev., 29, 11-18(1982). (149) Thompson, W.G., Lipkin, M., Gilbert, D.A., et al., “Evaluating evaluation: Assessment of the American Board of Internal Medicine resident evaluation form,” J. Gen. Int. Med., 5, 214-217(1990). (150) McLeod, P.J., “The student case presentation: An integral component of the undergraduate curriculum,” Teach. Learn. Med., 3, 113116(1991). (151) McCain, G.A., Molineux, J.E., Pederson, L. and Stuart, R.K., “Consultation letters as a method for assessing in-training performance in a department of medicine,” Eval. Health Prof., 11, 2142(1988). (152) Woolliscroft, J.O., Calhoun, J.G., Beauchamp, C, et al., “Evaluating the medical history: Observation versus write-up review,” J. Med. Educ., 59, 19-23(1984). (153) Kane, R.L., Leigh, E.H., Feigal, D.W. et al., “A method for evaluating patient care and auditing skills of family practice residents,” J. Family Pract., 2, 205-210(1975). (154) Ognibene, A.J., Jarioura D.G., Illera, V.A., et al., “Using chart reviews to assess residents’ performances of components of physical examinations: A pilot study,” Acad. Med., 69, 583-587(1994). (155) Goldman, R.L., “The reliability of peer assessments—A metaanalysis,” Eval. Health Prof., 17, 3-21(1994). (156) Gerrish, K., “An evaluation of a portfolio as an assessment tool for teaching practice placements,” Nurse Educ. Today. 13, 172179(1993). (157) Callister, L.C., “The use of student journals in nursing education: Making meaning out of clinical experience,” J. Nurs. Educ., 32, 185186(1993).
American Journal of Pharmaceutical Education Vol. 59, Fall 1995
247