Introduction Results Conclusion Discussion Methods

Quality assessment techniques require the use of both formative as well as summative assessments. The earlier that a student at risk may be identified, the more likely that remediation will be effective. The Wingate University School of Pharmacy sought to develop a program that provides formative assessment feedback to every student every year. In order to provide a comprehensive evaluation of the student’s progression towards educational outcomes the Annual Skills Mastery Assessment (ASMA) Exam was developed. This exam assesses student performance on the mastery of terminal ability outcome statements as they progress through the curriculum.

A critical component of this exam is setting the appropriate cut score. Wingate University School of Pharmacy utilizes a modified Angoff Process, where panels of 10 faculty members determine cut scores for individual test items included in the exam. The combined item cut scores are used to calculate the actual cut score used on annual exam administrations.

Dr. Greg L. Alston and Dr. Wesley Haltom Wingate University School of Pharmacy Cut Score Averages Between Faculty Panel Types

If the exam is to be found reliable then the cut scores created by these faculty panels should not vary significantly between panels rating identical test items.

56.22

57 – Avg. for all groups

55.39

56

53

Clinical

Non-clinical

Mixed Faculty

The results strongly suggest that there is no significant difference between yearly cut scores generated by panels of clinical vs. non-clinical faculty when compared to a mixed panel used in previous years. Clinical faculty appear to be slightly more accurate. The range of scores between panels if projected onto a 100 item test would result in the cut score moving by less than 2 correct items or (2%). Since each exam’s actual cut score is calculated from the weighted average of item scores, this result is important in establishing the reliability of the annual ASMA exam. The results also strongly suggest that the faculty are adept at correctly identifying which items will be easy, average difficulty, or hard. Prior to the items appearing on the ASMA exam, panels correctly rated 60 items without a single error.

Panel Prediction of Item Difficulty vs. Actual Student Performance Faculty Predicted Cut Score

Actual test item performance history determined whether questions were rated as easy (p>0.93), medium (p>0.68 and p