Medical E ducation
Basic principles of constructing Multiple Choice Questions 1
Sitanshu Sekhar Kar1, Subitha Lakshminarayanan2, Mahalakshmy T2
Associate Professor, 2Assistant Professor, Department of Preventive and Social Medicine, JIPMER, Puducherry, India
Abstract
Multiple Choice Questions (MCQs) are one of the most important well established tools used for assessment and selection processes in medical education. Designing good assessment tools requires time and effort and ultimately determines the learning outcome. The quality of MCQ depends on the quality of items on the whole and presence of efficient distractors. Flawed MCQs interfere with assessment process and therefore it is vital to develop reliable and valid items that are free of such flaws. Basic criteria for constructing good quality MCQs include item content that is tested and well-structured, flawless items. Three areas that are addressed in this article include general guidelines of using MCQ inevaluation, principles of constructing effective MCQs, and constructing higher order MCQs. Characteristics of effective MCQs are determined by the overall item, the stem and distractors. Interpretation and problem-solving items like MCQs with data, diagrams and images, use of clinical or lab vignettes, formats like Extended Matching items, Key feature questions and assertion reason questions require higher order thinking skills like analysis, synthesis and evaluation. Significant commitment and technical knowhow is required to prepare MCQs that are of good quality, reliable, able to test higher order thinking skills and consistent with curriculum objectives. Constructing MCQs is an art which can be perfected with practice and it is essential for medical educators to be skilled in effective test item writing to assess the learner’s knowledge. Key words: Assessment, effective MCQs, higher order MCQs
Introduction
The last decade witnessed a change in assessment strategies amongst medical teachers in India to accommodate some degree of objectivity in written assessment.1 Multiple choice questions (MCQs) have come to occupy a prominent place in the student’s assessment process.2 It represents one of the most important well established examination tools widely used in assessment at the undergraduate and postgraduate levels of medical examinations.3
over a broad coverage of concepts in a short span of time. Scoring is objective, reliable and quick using computers. Though MCQs are useful in measuring factual recall they can test higher order of thinking skills such as application, analysis, synthesis and evaluation of knowledge.1,3 A few other issues related to use of MCQs include guessing and cueing effect by the student and lack of feedback. Faulty items interfere with accurate and meaningful interpretation of test scores and have an adverse impact on the assessment process.1
Similar to other methods of assessment, they have their advantages and limitations. A distinct advantage of using MCQ is its ability to evaluate a large number of candidates
The basic MCQ or ‘item’ is one in which the candidate chooses one answer from a set of options provided (Single best response type). An item consists of a ‘stem’ followed by a number of options. Sometimes the stem is followed by the ‘lead in question’. The correct answer in the list of options is called a ‘key’ and the incorrect options are called ‘distractors’.3
MCQs are used most commonly for formative assessment and selection processes. Appropriately constructed MCQ based methods are efficient, objective, capable of discrimination and can be combined with other assessment strategies to contribute to a comprehensive student assessment strategy.4
Basic structure of MCQs
Address for Correspondence: Dr. Sitanshu Sekhar Kar, Associate Professor, Dept. of Preventive and Social Medicine, JIPMER, Puducherry,India- 605006. Email:
[email protected] Received: 07.06.2015, Accepted: 22.07.2015
Indian Journal of Community & Family Medicine | Vol. 1 | Issue 02 | Jul-Dec, 2015
65
Kar S S: Basic principles of constructing multiple choice questions ITEM Single best indicator of social development and well-being is a) Infant mortality rate b) Maternal mortality rate c) 1-4 year mortality rate d) Under 5 mortality rate
STEM DISTRACTORS KEY
Figure 1. Basic structure of an Item
Examiners constructing MCQs need to be aware of the different formats in order to decide the cognitive level of performance required by the candidates. The MCQs are an extensively used and time tested method of assessment of knowledge innational and international examination. These are of different types as classified by Hubbard and Clemans (1971).
Table 1. Types of Multiple Choice Questions (Hubbard and Clemans - 1971) Type of MCQ Example Single or best response type (Type A) Multiple completion type (Type K)
Writing effective MCQs
Designing good assessment tools requires time and effort and ultimately determines the learning outcome. Merely applying a particular tool is not enough; rather how well that tool has been designed makes all the difference. A poorly written MCQ is likely to distort learning.5
Constructing MCQs is an art which can be perfected with practice. Characteristics of effective MCQs aredetermined by the overall item, the stem and distractors.Flaws related to test-wiseness make it easier for some students to answer the question correctly, based on their test-taking skills alone; these flaws commonly occur in items that are unfocused. Flaws related to irrelevant difficulty make the question difficult for reasons unrelated to the trait that is the focus of assessment.6 Flawed MCQs interfere with accurate and meaningful interpretation of test scores and negatively affect student pass rates.7 Therefore, to develop reliable and valid tests, items must be constructed that are free of such flaws. Guidelines that are often referred in the context of writing MCQs include Haladyna’s guidelines8 and National Board of Medical Examiners (NBME) series.6
Relationship analysis type (Type E)
Extended Matching type
Two basic criteria for constructing good quality questions include item content (covering critical areas) and wellstructured, flawless items.6 This article provides guidelines that can be used in writing effective MCQs for selection examinations, continuing medical education materials, as well as routine assessment for medical student.Three areas that are addressed in this article include: (a) general guidelines of using MCQ in evaluation, (b) principles of constructing effective MCQs, and (c) constructing higher order MCQs.
Indian Journal of Community & Family Medicine | Vol. 1 | Issue 02 | Jul-Dec, 2015
Bengal gram is deficient in a) Lysine b) Tryptophan c) Methionine d) Leucine
True about vehicle transmission is: 1. Common source of infection is often traceable 2. Not always possible to isolate organism from vehicle 3. Many cases may occur simultaneously 4. Higher the dose, higher the number of cases a. 1, 2 b. 1, 2, 3, 4 c. 1, 3, 4 d. 1, 2, 3 Assertion: Measles vaccine should never be given before the age of 9 months BECAUSE Reason: The maternal antibodies present in the child till 9 months of age may neutralize the vaccine a. Both Assertion and Reasoning are true, reason is correct explanation b. Both Assertion and Reasoning are true, reason is not the correct explanation c. Assertion is true, Reason is false d. Assertion is false, reason is true e. Both Assertion and reason are false A. Calcium B. Fluoride C. Folic acid D. Iron E. Vitamin A F. Vitamin B1 (thiamine) G. Vitamin B6 H. Vitamin B12 (cyanocobalamin) I. Vitamin C J. Vitamin D K. Vitamin E
For each child, select the appropriate vitamin or mineral supplements. 1. A 1-month-old infant is brought to the physician for a well-child examination. He has been exclusively breast-fed,and examination shows normal findings. (select 2 supplements). 2. A 6-year-old girl has cystic fibrosis. She has been taking no medications. (select 3 supplements).
66
Kar S S: Basic principles of constructing multiple choice questions
A. GENERAL GUIDELINES FOR USING MCQs IN EVALUATION9 :
1. Scope of the examination and learning outcomes being tested should be clear before formulating an assessment strategy.
2. Cognitive levels being tested should be decided as the format of MCQ may depend on this fact.
3. Sufficient number of items isrequired for a valid MCQ examination that covers a specific topic; probably 60-100 items are optimal for an examination of 60-90 minutes duration.
4. Time allotment for examination needs due consideration. Different item formats require different time duration (single response - 40 seconds, whereas case history type – 60 to 90 seconds).
5. Negative marking and guessing correction are important issues that need to be addressed keeping the objective of assessment in mind. 6. Fixation of pass level in MCQ examination is more difficult than other formats. Though several formats are available, MCQs function much better for the purpose of ranking.
7. Validation process is an important step in formulation
Table 2. Checklist for constructing effective Multiple Choice Question items Area
DO’s
Content issues
Base each item on important content area / learning Avoid opinion-based items. outcome Avoid trick items. Ensure each item is wholly independent of the others Frame items that are specific. Choose items of appropriate level of difficulty
Writing the stem
Writing the choices/ distractors
Formatting and style issues
Ensure that the directions in the stemare very clear. Include the central idea and common elements in the stem Keep it clear, concise and unambiguous. If using a lead-in, it should clearly indicate how to answer the MCQ. Keep vocabulary simple.
Keep choices independent/ mutually exclusive Make sure that only one of these choices is the right answer. Keep choices homogeneous in content (distractors are in the same category as the correctanswer). Make all distractors plausible. Vary the location of the right answer according to the number of choices. Arrange choices in logical or numerical order, wherever appropriate. Keep the length of choices about equal.
Format the item vertically instead of horizontally. All parts of the item should be in the same page. Mix of optimum number of items of different levels of difficulty. Group similar formats together. Edit and proof items. Use correct grammar, punctuation, capitalization, and spelling. Keep it brief and minimize the amount of reading in each item.
DON’Ts
Avoid negatives such as NOT or EXCEPT; if used ensure that the word appears capitalized and boldface. Avoid double negatives. Avoid vague expressions like fairly high, considerably greater, etc. Avoid clues suggestive of the right answer
Avoid All-of-the-above. None-of-the-above should be used carefully. Avoid negatives such as NOT. Avoid the use of specific determiners such asalways, never, completely, and absolutely. Avoid grammatical inconsistencies
Avoid abbreviations
Indian Journal of Community & Family Medicine | Vol. 1 | Issue 02 | Jul-Dec, 2015
67
Kar S S: Basic principles of constructing multiple choice questions of MCQs. Prevalidation (before the item appears in an examination) and post validation (item analysis after the test) are essential to ensure validity of the item.
8. Instructions to candidates on how to respond to individual items, time allotment and negative marking, if any, should be clearly outlined in the beginning of the paper.
B. CONSTRUCTING EFFECTIVE MCQs: 1,8,6,10
Constructing multiple choice questions is a laboriously acquired art, and it is much easier to advise how not towrite them than to give a simple way of writing them. The questionsshould be relevant, short, understandable and discriminant.11 It is worth following a checklist whileformulating MCQs (Table 2).
Prevalidation of the items can be done by discussion with three or four experts on the relevance to learning outcome; clarity, brevity and appropriateness of stem; plausibility of all distracters and decision of the correct choice. It is also important to estimate the level of cognition tested by the item and the expected difficulty level. Construction of the item can be verified by checking if it can be answered by only reading the stem & lead‐in.
C. CONSTRUCTING HIGHER ORDER MCQs MCQ shouldtest the appropriate level of learning and ensure assessment of intended learner competence. Bloom’s taxonomy ofcognitive learning,described as a hierarchy of knowledge, comprehension, application,analysis, synthesis, and evaluation, has been adopted for test development.7
Traditionally, items are classified by the cognitive processes required to answer the question (eg, recall, interpretation, problemsolving, comprehension, or reasoning). Recall items are thought to test examinees’ knowledge of isolated facts.Interpretation items require examinees to review some information (often in tabular or graphical form) and reach some conclusion(eg, a diagnosis). Problem-solving items present a situation and require examinees to take some action (eg, the next step in patient management). Interpretation and problem-solving items are thought to involve “higher order” skills, rather thanjust rote memory of factual information.6 Clinical vignettes provide a good basis for questions at both comprehension and application levels.10 Each item would begin with the presenting problem of a patient, followed by the history, physical findings, results of diagnostic studies, initial treatment, subsequent findings, etc. Brief patient
Table 3. Constructing Higher order Multiple Choice Questions Cognitive level
Lower order
Higher order
Bloom’s Taxonomical domains
Knowledge Comprehension Application
Analysis Synthesis Evaluation
Examples of MCQs
If P indicates prevalence, I: incidence and D: duration of illness, then the relationship between prevalence and incidence is expressed as a) P = I X D b) I = P X D c) P = I/D d) P = D/I Prevalence of a disease a) Can only be determined by a cohort study b) Is the number of new cases in a defined population c) Describes the balance between incidence, mortality and recovery d) Is the best measure of disease frequency in etiological studies
If drug prevents mortality but does not affect cure, then which of the following will be true a) Incidence will decrease b) Incidence will increase c) Prevalence will decrease d) Prevalence will increase
If the prevalence is very low as compared to the incidence for a disease, it implies a) Disease is very fatal and/or easily curable b) Disease is nonfatal c) Calculation of prevalence and incidence is wrong d) Nothing can be said, as they are independent
Indian Journal of Community & Family Medicine | Vol. 1 | Issue 02 | Jul-Dec, 2015
68
Kar S S: Basic principles of constructing multiple choice questions
vignettes or laboratory vignettes may be suitable for Basic sciences. The overall structure of an item for such vignettes can be depicted by an item template.6 Use of patient and lab vignettes to assess application of knowledge has several benefits like enhancing “face validity” of the exam and focussing on important information rather than trivia. It also helps to identify those examinees who have memorized facts, but are unable to use that information effectively.7,6
Solving MCQs with data, diagrams and images require higher order thinking skills like analysis and synthesis level. Extended Matching items are multiple-choice items organized into sets that use one list of options for all items in the set. This reduces the chances of recognition effect (as more than one combination of vignette and option exists) and also helps in testing application and problem solving ability. Key feature questions and assertion reason questions are examples of Evaluation type of questions.10
Item analysis
Item analysis is a means to ensure quality of MCQs
and facilitate question banking. It provide a numerical assessmentof item difficulty and item discrimination. Itemdifficulty is determined from the percentage ofstudents who answered each item correctly.Item discrimination refers to the percentage differencein correct responses between two groups of students (generally referring to students in the top 25% and the lower 25%).7
CONCLUSION
MCQs can be used as a meaningful and effective tool of evaluation in medical education. The quality of MCQ depends on the quality of items on the whole and presence of efficient distractors. Flawed MCQs interfere with assessment process and therefore it is vital to develop reliable and valid items must be constructed that are free of such flaws. Significant time, effort and commitment is required to prepare MCQs that are of good quality, reliable, able to test higher order thinking skills and consistent with curriculum objectives. It is essential for medical educators to be skilled in effective test item writing to assess the learner’s knowledge.
REFERENCES
1. Singh T, Gupta P, Singh D. Principles of Medical Education. First. New Delhi: Jaypee Brothers. 2. Srinivasa DK, Adkoli BV. Multiple choice questions: how to construct and how to evaluate? Indian J Pediatr. 1989 Feb;56(1):69–74. 3. Al-Rukban MO. Guidelines for the construction of multiple choice questions tests. J Fam Community Med. 2006;13(3):125– 33. 4. Brady A-M. Assessment of learning with multiple-choice questions. Nurse Educ Pract. 2005 Jul;5(4):238–42. 5. Singh T, Natu MV. Examination reform at the grassroots: teacher as the change agent. Indian Pediatr. 1997 Nov;34(11):1015–9. 6. Case S, Swanson D. Constructing Written Test Questions For the Basic and Clinical Sciences [Internet]. National Board of Medical Examiners, Philadelphia; Available from: http://www. nbme.org/pdf/itemwriting_2003/2003iwgwhole.pdf
7. Collins J. Education techniques for lifelong learning: writing multiple-choice questions for continuing medical education activities and self-assessment modules. Radiogr Rev Publ Radiol Soc N Am Inc. 2006 Apr;26(2):543–51. 8. Haladyna TM, Downing SM, Rodriguez MC. A Review of MultipleChoice Item-Writing Guidelines for Classroom Assessment. Appl Meas Educ. 2002 Jul 1;15(3):309–33. 9. Ananthakrishnan N, Sethuraman K, Kumar S. Medical Education: Principles and practice. Second. Pondicherry: Alumni association of National Teacher Training Centre, JIPMER, Pondicherry; 10. Singh T, Anshu. Principles of assessment in Medical Education. First. New Delhi: Jaypee Brothers. 11. Lowe D. Set a multiple choice question (MCQ) examination. BMJ. 1991 Mar 30;302(6779):780–2.
Indian Journal of Community & Family Medicine | Vol. 1 | Issue 02 | Jul-Dec, 2015
69
