Using assessment to support higher level learning: the multiple choice ...

Assessment & Evaluation in Higher Education Vol. 29, No. 6, December 2004

Using assessment to support higher level learning: the multiple choice item development assignment Martin R. Fellenz* Trinity College Dublin, Ireland

This paper describes the multiple choice item development assignment (MCIDA) that was developed to support both content and higher level learning. The MCIDA involves students in higher level learning by requiring them to develop multiple choice items, write justifications for both correct and incorrect answer options and determine the highest cognitive level that the item is testing. The article discusses the benefits and limitations of the scheme and presents data on the largely positive student reactions to the scheme. The development of the MCIDA also serves as an example for how traditional summatively oriented assessment procedures can be developed into tools that directly support student learning.

Introduction Many educators hold ambivalent attitudes towards the use of multiple choice examinations. Many theoretical arguments exist both for and against their use. In practice, they are, like other objective tests using closed answer formats, seen and used as an effective and efficient method of assessing students’ content learning (Hampton, 1993; Scouller & Prosser, 1994). For some time I have successfully used multiple choice tests for this very purpose in undergraduate management, organizational behaviour (OB) and organization theory classes. More recently, however, I have started to critically review my use of traditional assessment approaches in the light of their potential for direct contributions to student learning. Specifically, I have focused on finding ways in which the assessment methods I employ can also contribute more directly to student learning, particularly higher level learning. The purpose of this article is to present and discuss one of the outcomes of my efforts, the multiple choice item development assignment (MCIDA). After briefly reviewing the advantages and disadvantages of multiple choice tests, I describe the MCIDA, which combines the use of multiple choice items as both assessment and learning tools and generally extends the role of assessment into more directly supporting student learning. *School of Business Studies, Trinity College Dublin, Dublin 2, Ireland. Email: [email protected] ISSN 0260-2938 (print)/ISSN 1469-297X (online)/04/060703-17  2004 Taylor & Francis Ltd DOI: 10.1080/0260293042000227245

704

M. R. Fellenz

Advantages and disadvantages of multiple choice question (MCQ) assessment Multiple choice tests, like other objective tests, focus on the selection instead of the creation of answers. Such tests have a number of advantages (see Airasian, 1994, for a comprehensive discussion). They are generally very easy to mark, which saves time, particularly for large classes. Because of the question design, double grading is not necessary and there is little potential for conflict or debate about the assignment of grades. Many items can be answered in a short time period, so a broad selection of instructional topics can be assessed in a single exam. The question format controls for differences in writing ability, so that test performance is not a function of writing ability which increases the construct validity of multiple choice tests. In addition to these benefits, multiple choice tests also have a number of disadvantages and problems associated with them. Many items need to be constructed for a single exam and good item development is time consuming (Airasian, 1994). The difficulty of designing good multiple choice items is often underestimated (Airasian, 1994; Miller et al., 1998) and existing test banks frequently contain choice bias (Clute et al., 1987), consist of items of often questionable quality (Bluedorn, 1986) and test almost exclusively lower level learning (Hampton, 1993; Airasian, 1994; Scouller & Prosser, 1994). While it is relatively easy to design items that test learning at lower cognitive levels, it is very difficult to develop items that test higher cognitive levels (Miller et al., 1998, p. 143). Even with considerable development time, it is hard to use multiple choice items to assess creativity and synthesis and other complex, integrative and constructive skills (Miller et al., 1998). The closed nature of multiple choice tests also limits their ability to test student abilities regarding the recognition, identification and formulation of ill-structured problems. This is particularly limiting in topic areas that deal with complex or equivocal subject matters, as is common in organizational behaviour and many other behaviourally oriented fields. The closed nature of MCQ answer options limits student interaction with the test because it ‘does not allow pupils to construct, organize and present their own answers’ (Airasian, 1994, p. 174). In addition to these limitations inherent in the structure and the use of multiple choice exams, a more fundamental criticism that can be aimed at the format is that in traditional usage they give primacy to instructor perspectives and fail to acknowledge plurality, do not allow students to constructively interact with the tested knowledge, do not deal with relativities and thus generally reflect neither partnership-based (see for example Ramsey & Couch, 1994) and learning-centred educational philosophies (see for example Biliamoria & Wheeler, 1995) nor postmodern conceptions of assessment in higher education (see for example Bilimoria, 1995). This criticism is supported by evidence that suggests the overwhelming use of multiple choice tests for the narrow role of assessing low level content learning (see for example Hampton, 1993; Scouller & Prosser, 1994). As such, in the absence of additional features like immediate feedback (for an example see Michaelsen et al., 1994), multiple choice questions provide a very limited direct contribution to student learning. Furthermore, students may answer

Assessment to support higher level learning

705

correctly by guessing (Ebel & Frisbie, 1991; Airasian, 1994), which limits the construct validity of multiple choice tests. Similarly, as experience with the test format, test taking strategies and the type of preparation used also influence test performance (see for example Schermerhorn et al., 1992), the fairness of multiple choice tests, especially for heterogeneous student populations, is debatable. Little relevant research exists on the link between demographic diversity and performance on multiple choice tests used in college classrooms, but initial findings indicate that objective tests do not appear to be discriminating on the basis of sex or ethnic background (Ghorpade & Lackritz, 1998). Finally, the acceptance of multiple choice tests by students, teaching staff and administrators can differ, as can their fit with the educational traditions and cultures of different institutions. Many of the potential problems can be overcome through conscientious use of the format and by employing it alongside other examination methods as part of a carefully designed assessment protocol. Overcoming the lack of direct contributions to student learning, however, requires the use of multiple choice items as part of more comprehensive educational arrangements. Previous attempts at this have focused on the provision of timely feedback on multiple choice exam performance, the involvement of students in developing multiple choice items and the successive use and discussion of the same items in individual, group and then class-wide settings (see for example Michaelsen et al., 1994). The MCIDA described in this paper includes some of these features. However, it substantially departs from previous schemes on a number of dimensions. It focuses not on performance on actual multiple choice tests, but on the development and assessment of multiple choice items. This includes explicit reflection and active contemplation of the cognitive level of the material learned and assessed. Thus, it supports both lower level and higher level learning. It also provides a rich opportunity for meta-learning through considering the nature of the knowledge required to answer particular items and through examining how this knowledge is assessed through these items. The multiple-choice item development assignment In brief, the MCIDA requires students to work individually, in pairs or in small groups to develop and submit multiple choice items that can be used to test course-relevant knowledge. Initially, students are introduced in class to Bloom’s taxonomy of different cognitive levels of learning (Bloom et al., 1956) and are briefed on the procedures, requirements and grading conventions of the MCIDA. In the more interactive setting of tutorial groups, students are invited to actively discuss, question and critique both multiple choice exams in general and the MCIDA in particular. They also receive guidance regarding the design of multiple choice items (see Appendix A) and supporting materials on Bloom’s taxonomy. Throughout these tutorials, the links between multiple choice items and the cognitive level of knowledge required to answer them correctly is repeatedly clarified through both examples and multiple choice development exercises. Along with the assignment material, students receive many examples of items testing different levels of knowledge as well as information on relevant web-based resources. Through the

706

M. R. Fellenz

ensuing discussions, issues such as multiple choice exam preparation and test-taking strategies, potential for bias and limitations of objective tests can be comprehensively considered. These discussions are part of the introduction to the assignment, but I instigate similar discussions repeatedly in class and in tutorials throughout the course to identify and discuss emerging issues and problems. The procedural details of the MCIDA are as follows. Each student, student pair or small group is required to submit a specified number of multiple choice items covering different parts of the course content. I have required nine items for individuals and pairs in semester long OB courses with 20% of the course grade determined by the MCIDA. Student feedback indicates that the work required for good performance on the MCIDA is seen as quite significant (in one OB course, the average self-reported time spent on developing the first set of three items was 5.3 h, n ⫽ 52, SD ⫽ 2.3, range ⫽ 2–13 h), thus the assignment needs to receive an appropriate weighting in the overall assessment scheme of the course. The items must be submitted in three sets, thus leaving enough time for all students to receive and utilize feedback on earlier submissions before the next set of items is due. In small classes the frequency of submissions and feedback can be further increased to improve the value of such formative feedback through additional feedback cycles. Each multiple choice item submitted must consist of a stem with one correct and three incorrect answer options. Students need to identify the correct answer option and provide an explanation as to why this is accurate. They also need to provide specific explanations as to why each of the other options is incorrect. Finally, the students need to identify the highest cognitive knowledge level (Bloom et al., 1956) that is required to answer each submitted item correctly (see Table 1 for examples of item submissions; see also Hampton, 1993, for additional examples of multiple choice items in this context). After each submission, students receive feedback on their submitted items with comments about the formal aspects of their submission, the quality of item design, the accuracy and justification of the correct and incorrect answer options, the identification of the tested cognitive level and the overall suitability of the developed items for assessing course-relevant learning. Details of grading conventions are given in the next section. One aspect of the MCIDA that is particularly favoured by students is the fact that many submitted items are later used in a term exam. My explicit target is to have more than half of the multiple choice items on the exam (given about 3 weeks before the end of the course) made up of student submissions. This increases student ownership of the assessment procedures used and motivates students to participate and submit high quality multiple choice items. A potential problem of this is that students have the opportunity to share their submitted items with each other as a preparation for the exam. Even though I have not had any evidence that this has occurred, I change the order of answer options of student items. Grading of the MCIDA Several distinct aspects of MCIDA submissions can be directly related to the


707

Table 1. Examples of item submission for the MCIDA Example 1) The extent to which a target stands out among a group of other targets is called the target’s A) identity. B) novelty. C) salinity. D) salience. The only correct answer choice for item 1 is answer D. This option is correct because salience is the term used to describe what makes a target more likely to be selected from the available data by perceptual processes and thus makes it more likely to be perceived among other potential targets. A is incorrect because the identity of a target can only be known by a perceiver after perceptual processes have been completed; B is incorrect because it is one of the factors that can contribute to the salience of a target and does not in itself describe the degree to which the target stands out among other potential perceptual targets in a situation; and C is incorrect because salinity describes the degree of salt in a solution and has nothing to do with perception theory. This item tests Bloom’s basic cognitive level (Level 1: KNOWLEDGE) because it only requires recall of information (i.e. of common terms/basic concepts – here: salience) for the item to be answered correctly. Example 2) Peter’s performance-outcome perception that a good class performance will lead to a high participation grade is 0.8. His effort-performance expectation that spending much time on preparing for class will lead to a good in-class performance is 0.6. The value of a high participation grade to him is 0.9. Which of the following is true? A) Peter’s motivational score is 2.3. B) Peter’s expectancy is larger than his instrumentality. C) Peter will spend much time preparing for class. D) None of the above. Answer A is wrong because the motivational score is computed by multiplication, not by adding expectancy, instrumentality and valence. B is wrong because expectancy is 0.6 while instrumentality is 0.8. C is wrong because without knowledge of expectancies, instrumentalities and valences of other options we cannot predict Peter’s behaviour. Thus, based on expectancy theory, D is the only correct answer. This item tests a number of Bloom’s cognitive levels. It tests level 1 (KNOWLEDGE) because it requires recall of information (i.e. of common terms/basic concepts of Expectancy Theory); it tests level 2 (COMPREHENSION) because the terms ‘performance-outcome perception’ and ‘effort-performance expectation’ need to be understood and translated into ‘instrumentality’ and ‘expectancy’, respectively. The item also tests level 3 (APPLICATION) because it requires the formula for determining the total motivational force from Expectancy Theory to be applied.

learning goals underlying the scheme and can be graded according to the degree to which they demonstrate the attainment of these learning goals. These graded properties include the formal aspects of the submission, the adherence of the items to good design practice, the accurate identification of correct and incorrect answer options, the accuracy, comprehensiveness and persuasiveness of the justifications for correct and incorrect answer options, the cognitive level tested by the submitted

708

M. R. Fellenz

item and the suitability of the item for testing relevant course learning. The link of these submission characteristics to different learning goals and relevant grading conventions are discussed below.

Formal aspects of the submission This includes such issues as meeting deadlines and adhering to the designated submission procedures, along with spell-checking and proof-reading material prior to submission. It also relates to fundamental learning goals such as acquiring professional attitudes, time management, business communication skills, etc. I deduct a percentage of the grade depending on the severity of the problem for items that fail to meet professional standards. As timely feedback is an important part of the scheme, I have used a cumulative late submission penalty of 5% per day. This has largely eliminated submission tardiness without generating the often intense reactions of absolute submission cut-offs.

Adherence of the items to good design practice Part of the learning goals of the MCIDA is increasing students’ understanding about how multiple choice items work and how they test knowledge. This learning is reflected in the degree to which the student-designed items facilitate the effective and efficient assessment of knowledge. Thus, adherence to good design practice is a central aspect of the assessment of student contributions. I usually award about 25% of the grade on the basis of structure and design of the submitted items (see Appendix A for a brief checklist for writing multiple choice items).

Accurate identification of correct answer options and distractor items The accurate identification of correct and incorrect answer options is central to the use of multiple choice items. It is also a clear indicator of the students’ understanding of the content that their item is designed to test. I award about 10% of the grade for accurate identification of the correct and the three incorrect answer options.

Justifications for correct and incorrect answer options I place great value on the quality of the explanations as to why the different options are either correct or incorrect. These explanations must be comprehensive and convincing justifications for the use of the individual options. This part of the formal MCIDA requirements is a key test of students’ higher level functioning vis-a`-vis the material their item is supposed to test. A well-argued justification for all options is the best evidence that students have fully understood the material. This reflects the old adage that one hasn’t fully understood something until one has taught it. It is also the reason why I insist that students provide justifications that specify why each


709

answer option is either correct or incorrect. These justifications must be complete and grounded in the topic, theory or example of the item stem. For recall items of the type ‘Who are the authors of the Job Characteristics Model?’, such justifications obviously have less value. Justifications for answer options to items testing higher level knowledge, however, provide a challenging and revealing opportunity for a student to show that they have truly understood the material tested. As an example, explanations that an option is correct because the textbook specifies it as correct are not acceptable. I usually assign about 25% of the grade for this aspect of the submissions. Cognitive level tested by the submitted item A key part of the MCIDA is the requirement for students to identify the cognitive level of the educational objective each submitted item is testing. This is a crucial ingredient of the MCIDA because it engages students in meta-learning by requiring substantial reflection on both the nature of the relevant learning and on the cognitive skills and knowledge drawn on to answer a particular item. As discussed above, it is difficult to design multiple choice items that test higher level knowledge. Furthermore, the higher the cognitive level of the knowledge required to answer an item, the more difficult it is to determine and describe the nature of that required knowledge because higher level cognitive performance usually includes many or all of the relatively lower levels as well. Although the literature suggests that the cognitive levels tested can be assessed with relative ease and high inter-rater reliability by experienced teachers (Hampton, 1993), in my experience many undergraduate students have significant problems with accomplishing this task. This may be due to inexperience, to the inherent difficulty of the task or to bias due to ego involvement (Hampton, 1993). For this reason I am employing increasingly demanding expectations that are informed by the progress in performance across participating students. This use of relative rather than absolute standards in grading (see for example Glaser, 1963) is chosen to support the abilities students have and develop over the course of the MCIDA. It is intended to keep students on board and motivated to participate and learn and to develop an appreciation of both the challenges and the value of this aspect of the assignment, which is often not immediately clear to many students. In practice, I inform students that the development of items testing higher cognitive skill levels will bring higher grades. However, not all items (particularly those in the first set submitted) need to aim at high level skills in order for very high grades to be achieved in the MCIDA. The overall value of this aspect for the assigned grades is about 20%. Suitability of the item for testing relevant course learning The final aspect of the submitted items that I explicitly include in my grading is the overall suitability of an item for testing relevant course content (worth about 20% of the grade). While this can be seen as duplicating the assessment of the individual aspects discussed above, it is intended to provide a comprehensive evaluation of how

710

M. R. Fellenz

these distinct elements combine to provide a valid test of student learning. For example, the correct and incorrect answer options may be accurately specified, but one or more of them may be written in an ambiguous or misleading way, making the item unsuitable for actual use without rewriting these options. That said, an item may be written with an exceptional degree of creativity, using content that is especially well suited to test course knowledge in a novel and interesting way. Many intangible aspects combine with the technical ones described above to render some items particularly suitable or unsuitable for a specific course with its distinct student population. Overall, aspects such as creativity, relevance, elegance, pragmatism, low levels of ambiguity and equivocality and appropriate use of humour can be considered and explicitly included in the grading of the assignment and the feedback for students.

Mechanisms for providing feedback to students Students receive feedback through the grades awarded for each submitted item as well as through feedback forms (see Appendix B). This form is an efficient means of providing rich feedback to students on their submissions that can guide their further item development and enhance their learning from the MCIDA. It also structures the grading of the items, makes assessment decisions explicit for all stakeholders and reduces the time required for graders to provide meaningful, individualized feedback.

Ungraded learning goals In addition to these assessed learning goals, the MCIDA aims to achieve a number of other educational objectives. These include preparation for the multiple choice exam, understanding of the structure and logic of multiple choice exams, reinforcement of course learning goals, critical consideration of assessment procedures used in this and other courses and reflection on students’ own approaches to test preparation and test taking and, more fundamental and much more important, to learning in general. These learning goals are not explicitly assessed as part of the MCIDA, but I have tried to investigate them using the action-learning component of my own teaching practice and through the frequent solicitation of written and verbal feedback from students individually and in classroom and small group settings. The benefits and limitations of the MCIDA as revealed through student feedback and through my own experience and reflection are discussed in the next two sections, respectively.

Benefits of the MCIDA The MCIDA has been shown to provide a number of benefits. Feedback elicited in class and tutorial group discussions in several OB courses and through a questionnaire administered after the completion of one of the courses indicates that students


711

overwhelmingly see the MCIDA as valuable in aiding understanding and content learning (see Table 2 for details). Based on this formal feedback and the classroom and individual discussions with students it appears that the MCIDA is effective in increasing the time students spend on learning about the course content, both individually and collaboratively. Students better understand how multiple choice items are constructed, what kind of knowledge they test and how best to prepare to take multiple choice exams. Students feel more ownership of the assessment of their learning in the course and report that they have become more critically aware of the assessment approaches in this and other courses. They also develop a better understanding of the different levels of knowledge that can be gained in a college course like this and how they can develop these different types of knowledge. The students also report increased reflection regarding their own learning approaches. In addition to these student-reported benefits, the MCIDA provides a number of additional advantages. The assignment requires students to engage with the subject matter of the course at high cognitive levels. The justifications for why answer options are correct or incorrect, for example, demand evaluation and the construction of compelling arguments. Also, students are explicitly rewarded for creativity in the item development task. The iterative nature of the assignment provides formative feedback that leads to noticeable improvement in the quality of the submitted items over the course of the assignment. If used for pairs or small groups of students, the MCIDA strongly promotes collaborative learning due to the increase in constructive interactions about course content among students. In addition, even students with little previous experience of or low self-efficacy regarding multiple choice exams have extensive opportunities to develop a sound understanding of this assessment method and prepare adequately for such exams. The development and design tasks involved in the MCIDA can have additional learning benefits for students. Especially in topic areas where ill-structured problems are common, multiple choice item development will challenge students to recognize the complexity or equivocality inherent in certain questions. This recognition must then be used to guide the design of new items. A common mistake that novice item designers make is failing to recognize the different possible interpretations of seemingly straightforward item stems or answer options. In particular, the MCIDA requirement for students to provide justifications for the correct answer option as well as for the incorrect ones requires students to make explicit their understanding of the complexities of the subject matter. As part of the normal MCIDA grading procedure, students receive feedback on their ability to recognize and appropriately deal with such ill-structured issues and problems. Thus, their selection and design activities, along with the feedback they receive, provide students with excellent opportunities to creatively and critically come to terms with the intricacies inherent in many topic areas. All of this does not take away from the potential difficulties and limitations inherent in using multiple choice exams in areas rife with ill-structured problems, but the MCIDA can aid students in gaining relevant abilities such as problem recognition, identification and formulation skills that can help enable them to constructively deal with such problems.

8.8% 20.6% 17.7% 17.7% 20.6% 17.7% 26.5%

0.0% 0.0% 0.0% 2.9% 0.0% 2.9% 0.0%

D

26.5% 23.5% 20.6% 26.5% 17.7% 17.7%

26.5%

N

52.9% 52.9% 50.0% 44.1% 58.8% 41.2%

58.8%

A

Answer anchors used were: strongly disagree (SD); disagree (D); neither agree nor disagree (N); agree (A); strongly agree (SA).

a

The MCIDA assignment helped me to improve my understanding of course concepts (theories, models, etc.) The MCIDA assignment improved my overall knowledge of course material The MCIDA assignment was a valuable learning experience The MCIDA assignment made me spend more time learning about the content of the course The MCIDA assignment has challenged me to think more about the course content The MCIDA assignment has helped me to prepare for the MC mid-term exam The MCIDA assignment has helped me to understand better the different types of knowledge I can gain from this course

SDa

Table 2. Quantitative student feedback on the MCIDA (n ⫽ 34)

0.0% 5.9% 8.8% 8.8% 2.9% 14.7%

5.9%

SA

712 M. R. Fellenz


713

Benefits that the MCIDA has over similar schemes that attempt to use multiple choice assessment as a means for supporting learning (e.g. pre-instructional minitests; see Michaelsen et al., 1994) are the relatively small amount of class time required on a regular basis, the extensive involvement of students in higher level cognitive tasks, critical reflection on links between learning and assessment and meta-learning. In briefing and debriefing students in the context of the MCIDA, I have benefited from discussions about learning and assessment that indicated a level of depth and reflexivity rarely found among undergraduate students. I have found the MCIDA to be an excellent means of engaging students in reflecting on their own learning as well as on the approaches to teaching, learning and assessment used both in my course and others. An initially unplanned but welcome result of this engagement was the increasing internal pressures from students (Balla & Boyle, 1994) on assessment quality in my course and beyond. Critical and empowered students that understand, reflect on and comment on the role of assessment in their learning are an important and effective means of assuring continuous improvement in teaching, learning and assessment practices. The close link between the learning objectives and assessment practices in the MCIDA can help to focus students on the benefits of learning, with good grades as a direct result of such learning, rather than the often prevalent order of student focus on good grades with learning an intended but often incidental second order outcome. After using the MCIDA I am more convinced of the value that using multiple choice exams can have because when used in conjunction with the MCIDA their value goes beyond simple assessment of learning outcomes. Multiple choice questions can be an active and integral part of an assessment system that directly supports both content and higher level learning and simultaneously invites and supports student reflection and meta-learning. Limitations, challenges and opportunities for extension and development of the MCIDA Like any assessment and learning approach, the MCIDA has a number of potential limitations and challenges, some of which can be overcome by extending the scheme or linking it with other educational approaches and techniques. Time requirements Because it is a departure from traditional assessment approaches and students generally have little experience of the assignment tasks, the MCIDA requires designated class time for explanation, discussion and organization. Based on feedback received, good performance on the MCIDA requires a substantial commitment of time and effort on the part of the students. Similarly, it also requires a substantial amount of time and commitment from instructors to provide the support and feedback necessary for students to gain the full learning value from participating. Ultimately, in my experience, the time obligation for the instructor is not larger than

714

M. R. Fellenz

that required for many other forms of continuous assessment that support higher level learning. It is necessary, however, for instructors to have a good understanding of the issues in multiple choice item development, especially as they relate to using multiple choice items to test higher level learning. Student skill requirements Unlike multiple choice and other objective tests, the MCIDA favours students with good verbal and writing skills because of the central role of these skills in item design, in justifying the classification of correct and incorrect answer options and in explanation of the cognitive level tested. Extensive use of examples and good support for students can help ameliorate but not fully solve this potential problem. As I am using both the MCIDA and multiple choice exams, however, students with a preference for written assignments and those who prefer assessment through objective tests will find part of their assessment reflecting their particular inclinations. Also, I have recently started to require students to submit items in pairs or small groups of three, which increases the writing quality because of additional editing usually involved in such collaborations and because such pairs can utilize the skills of the better writer. Perceived difficulty and student self-efficacy Students generally experience the MCIDA as very challenging. They highlight the task of identifying the highest cognitive level that an item tests as particularly difficult. To ensure that all students, even those with low self-efficacy regarding the tasks central to the MCIDA, are motivated to engage with the assignment tasks it is important to provide substantial amounts of support, often in the form of individualized attention. This can be done by the instructor as well as in working groups in class or outside. Here is a promising opportunity to link the MCIDA with peer review approaches that provide learning opportunities for those receiving feedback and assistance and for those that provide such input (learning through teaching and learning through assessing; see for example Topping, 1998). This could be done by following the nested review sequence of individual item development, small group review and plenary review in the classroom, similar to the procedure described by Michaelsen et al. (1994). Another possible extension of the scheme would be the use of the MCIDA in successive courses so that higher level students that have gained substantial insight into multiple choice item development and the way in which such items test different cognitive skills can act as formal tutors or reviewers for lower level students. Such tutoring may even be a useful exercise for graduate students or others new to college level teaching as it would provide valuable opportunities to practice small group teaching as well as learning more about testing different levels of knowledge and cognitive skills. Exclusion of affective educational objectives The MCIDA, as described here, focuses specifically on higher order cognitive skills


715

[also called higher order thinking skills (HOTS); see Airasian, 1994] and does not explicitly deal with the attitudinal or affective domain of behaviour (see for example Krathwohl et al., 1964) that often forms a core part of learning objectives in higher education. To the degree that this affective domain includes characteristics such as attitudes, motivation, self-control, self-leadership or commitment to a specified task, the grading of the formal requirements of the MCIDA (timeliness, format, etc.) can be seen as implicitly covering this domain. If affective educational objectives are central to a particular course, then the limitation due to the educational goals fostered by the MCIDA needs to be recognized and the scheme needs to be augmented with other measures. Student rejection of objective tests Despite the effort to engage students in higher level learning and the extensive support available for students as part of the MCIDA, some students continue to dislike multiple choice question examinations and, by association, the MCIDA. I have found that many such critics can be won over by reinforcing the fact that the MCIDA focuses on a very different cognitive skill set than most traditional multiple choice exams students may have experienced in the past. Institutional and cultural barriers Finally, as with any pedagogical technology that places student learning at its centre, the typical challenges for instructors do not stop in the classroom. In many institutions there may be significant potential for conflict with established educational policies or with aspects of the educational culture. An example of the former may be the use of grading curves or other relative grading conventions that force instructors to adhere to predetermined grade distributions. This approach, even though it is compatible with the procedures of the MCIDA, is incongruent with the teaching and learning philosophy underlying the scheme (see also Hiller & Hietapelto, 2001, for a discussion of relevant philosophical issues and of possible career implications). Similarly, there may be institutions, parts of institutions or individuals that reject objective tests out of hand without considering how such tests can play a part in the comprehensive assessment of student learning or how they can in fact support and enhance student learning as the MCIDA attempts to do. In either case, it would be in the best interest of students to employ the MCIDA in a way that safeguards its key contributions to student learning and in the best interests of instructors to find ways to create acceptance for this innovation. My own experience with the MCIDA and other teaching and learning innovations has shown that clear, explicit and well-publicized links between learning goals and assessment procedures are key to addressing and overcoming such potential difficulties. Conclusion The central idea of the MCIDA of asking students to develop multiple choice or

716

M. R. Fellenz

other objective test items is not new (see for example Hampton, 1993; Michaelsen et al., 1994). The added advantage of the MCIDA compared with these examples is that it focuses to a much greater degree on enabling and supporting higher order student learning. The logic underlying the development and use of the MCIDA is well expressed by Miller and his colleagues in their discussions on the evaluation of different assessment methods: ‘If our observations and assumptions are correct, then it is conceivable that a change in assessment techniques could, after all, bring about changes in students’ approaches to learning’ (Miller et al., 1998, p. 228). Based on the available evidence the scheme achieves this objective. The MCIDA was designed to be, and has proven to be, much more than an assessment tool; it is first and foremost a teaching and learning tool. Its development and use has led me to re-evaluate my own approaches to teaching, learning and assessment. Based on my evaluation of student learning and the formal (teaching evaluations and the MCIDA survey) and informal feedback received on the scheme, it has led to demonstrably superior learning outcomes for students. The MCIDA is an example of how assessment can directly support learning. It is also a good example of one of the principles advanced by the American Association for Higher Education Assessment Forum on good practice for assessing student learning which states that ‘Assessment is not an end in itself but a vehicle for educational improvement’ (Astin et al., 1996). If we as educators view assessment ‘not as a tiresome chore incidental to our real work, but as a central part of teaching’ (Hampton, 1993, p. 470), we should be able to find ways to use all assessment approaches to create more interesting, more challenging, more supportive and more valuable learning opportunities for our students.

Acknowledgements I am grateful to Jim Quinn and Stuart Smith of Trinity College Dublin for valuable comments on earlier versions of this paper.

Notes on contributor Martin Fellenz lectures in the School of Business Studies, Trinity College Dublin. He received his PhD from the Kenan-Flagler Business School at the University of North Carolina at Chapel Hill. His research focuses on the areas of organizational justice, organizational change and transformation, and management education and development. His current research interests in the area of teaching and learning are in peer evaluation of student group work, assessment for learning, student-centred education and conceptions of management education and development. At Trinity College Dublin he teaches in the areas of organizational behaviour, organization theory and research methodology and the philosophy of science, and is Academic Director of the MSc (Mgmt) in organizational behaviour. In 2001 he was an inaugural winner of the universitywide Provost’s Teaching Award. He actively consults and conducts manage


717

ment development workshops with leading Irish and international public, private and not-for-profit organisations. References Airasian, P. W. (1994) Classroom assessment (2nd edn) (New York, McGraw-Hill). Astin, A. W., Banta, T. W., Cross, P., El-Khawas, E., Ewell, P. T., Hutchings, P., Marchese, T. J., McClenney, K. M., Mentkowski, M., Miller, M. A., Moran, E. T. & Wright, B. D. (1996) 9 Principles of good practice for assessing student learning. American Association for Higher Education Assessment Forum. Available online at: http://www.aahe.org/principl.htm (accessed 10 October 2002). Balla, J. & Boyle, P. (1994) Assessment of student performance: a framework for improving practice, Assessment and Evaluation in Higher Education, 19, 17–28. Biliamoria, D. (1995) Modernism, postmodernism, and contemporary grading practices, Journal of Management Education, 19(4), 440–457. Biliamoria, D. & Wheeler, J. V. (1995) Learning-centered education: a guide to resources and implementation, Journal of Management Education, 19(3), 409–428. Bloom, B. S. Engelhard, M. D., Furst, E. J., Hill, W. H. & Krathwohl, D. R. (Eds) (1956) Taxonomy of educational objectives. Handbook I: Cognitive domain (New York, Longmans, Green & Co). Bluedorn, A. (1986) Resources for the introductory management course, Academy of Management Review, 11, 684–691. Clute, R. C., Jackson, J. H. & McGrail, G. R. (1987) Choice bias in multiple choice questions from organizational behavior test banks, Journal of Management Education, 12(4), 124–127. Ebel, R. L. & Frisbie, D. A. (1991) Essentials of educational measurement (4th edn) (Englewood Cliffs, NJ, Prentice Hall). Ghorpade, J. & Lackritz, J. R. (1998) Equal opportunity in the classroom: test construction in a diversity-sensitive environment, Journal of Management Education, 22(4), 452–471. Glaser, R. (1963) Instructional technology and the measurement of learning outcomes: some questions, American Psychologist, 18, 519–521. Hampton, D. R. (1993) Textbook test file multiple-choice questions can measure (a) knowledge, (b) intellectual ability, (c) neither, (d) both, Journal of Management Education, 17(4), 454–471. Hiller, T. B. & Hietapelto, A. B. (2001) Contract grading: encouraging commitment to the learning process through voice in the evaluation process, Journal of Management Education, 25(6), 660–684. Krathwohl, D. R., Bloom, B. S. & Masia, B. B. (1964) Taxonomy of educational objectives. Handbook II: Affective domain (New York, David McKay). Michaelsen, L. K., Fink, L. D. & Watson, W. E. (1994) Pre-instructional minitests: an efficient solution to the problem of covering content, Journal of Management Education, 18(1), 32–44. Miller, A. H., Imrie, B. W. & Cox, K. (1998) Student assessment in higher education: a handbook for assessing performance (London, Kogan Page). Ramsey, V. J. & Couch, P. D. (1994) Beyond self-directed learning: a partnership model of teaching and learning, Journal of Management Education, 18(2), 139–161. Schermerhorn, J. R., Jr, Gardner, W. L., III & Dresdow, S. A. (1992) Success profiles for student examination performance in a large-lecture management course: an empirical study, Journal of Management Education, 16(4), 430–443. Scouller, K. M. & Prosser, M. (1994) Students’ experiences in studying for multiple-choice questions examinations, Studies in Higher Education, 19, 267–279.

718

M. R. Fellenz

Topping, K. (1998) Peer assessment between students in colleges and universities, Review of Educational Research, 68(3), 249–276.

Appendix A. General checklist for writing multiple choice items Overall: 1. Is the wording clear? 2. Are stem and answers written to reduce required reading time? Writing the stem: 3. Is all extraneous information excluded from the stem? 4. Are all words that are repeated in all answer choices included in the stem? 5. Are all critical words in the stem highlighted (e.g., NOT, EXCLUDED, …)? Writing the answers: 6. Is there only one correct answer? 7. Are the answer choices unambiguous? 8. Are answer options of similar length and parallel in grammatical structure? 9. Are phrases lifted directly from assigned readings? 10. Are all distractor items incorrect but plausible? 11. Do the distractor items contain familiar words? 12. Do recognizable keywords that appear in the correct answer option also appear in some or all of the distractors? 13. Are extreme words that often indicate incorrect answer options avoided (e.g., all, always, never)? 14. Are vague phrases avoided (usually, typically, may be, …)? 15. Is ‘All of the above’ avoided as answer choice as eliminating one option eliminates this one too? 16. Is ‘None of the above’ used if an additional distractor is needed?

Assessment to support higher level learning Appendix B. MCIDA submission feedback form

719