Blueprints In Health Profession Education Series |Constructing MCQs. 2. Blank Page ... Common Technical Item Flaws. ... BS., MHPE. Medical Education Unit.
BLUEPRINTS IN HEALTH PROFESSION EDUCATION SERIES
CONSTRUCTING A-TYPE MULTIPLE CHOICE QUESTIONS (MCQS): STEP BY STEP MANUAL EDITED BY MOHAMMED ELHASSAN ABDALLA ABDELRAHIM MUTWAKEL GAFFAR RASHA ALI SULAIMAN
2011
M
O H A M E D
E
L H A SS A N
A
B D A L L A
, A
B D E LR A H I M
M
U T W A K E L
G
A F F A R
& R
A S H A
A
L I
S
U L A I M A N
Blank Page
Blueprints In Health Profession Education Series |Constructing MCQs
2
(CC): Constructing AType Multiple Choice Questions (MCQs): Step By Step Manual by Mohammed Elhassan Abdalla, Abdelrahim Mutwakel Gaffar and Rasha Ali Suliman is licensed under a Creative Commons AttributionNonCommercial ShareAlike 3.0 Unported License.
Acknowledgment We would like to express our gratitude to Dr Fathalrahman Dafa Allah English Center, Jazan University for his assistance in reviewing and proof reading of the first draft of this work.
Blueprints In Health Profession Education Series |Constructing MCQs
3
Contents Introduction.......................................................................................................................................... 5 What can MCQs Assess? ....................................................................................................................... 7 Types of MCQs ...................................................................................................................................... 8 The anatomy of Atype MCQs ............................................................................................................... 9 Pathology in MCQs (threats to their validity)........................................................................................13 The management of MCQ Pathologies .................................................................................................14 Common Technical Item Flaws. ............................................................................................................16 Itemwriting guidelines ........................................................................................................................19 Writing highcognitivelevel MCQs .......................................................................................................21 Bibliography.........................................................................................................................................25
Blueprints In Health Profession Education Series |Constructing MCQs
4
0
Introduction
Assessment is one of the important components of the teaching and learning cycle, as it is the major driver of students’ learning, and thus it is said that the assessment tail wags the curriculum dog. A very important rule has been stated by Shwarts, namely, that for every evaluative action, there is an equal, greater or sometimes opposite educational reaction. This reaction mainly affects the expected impact from the educational process. In the field of medical education, many assessment tools are used to evaluate the achievement of students; those tools are intended to assess the three domains in Bloom's taxonomy of educational objectives. Among those tools, Multiple Choice Questions (MCQs) are the most widely used assessment method in medical schools. MCQs have been used in assessment methods for more than 50 years. Many factors have contributed to the popularity of this method, including the facts that MCQs are easy to administer, easy to mark for even a large number of students, efficient, and objective. We are pleased to introduce this manual as part of the Blueprints in Health Profession Education series. We hope that this effort will help health profession educators and students utilise appropriate teaching, learning,and assessment techniques. The aim of "Constructing AType Multiple Choice Questions (MCQs): a step by step manual" is to provide a simple, stepbystep guide for health professional educators to assist them in constructing highlevel MCQs to enhance their students’ assessment and learning processes. Dr Mohamed Elhassan Abdall, MB. BS., MHPE.
Medical Education Unit Faculty of Medicine Jazan University Jazan KSA
Dr Rasha Ali Suliman MB.BS, MD
Ophthalmology Department Medical Education Unit Faculty of Medicine Jazan University Jazan – KSA
Blueprints In Health Profession Education Series |Constructing MCQs
Dr Abdelrahim Mutwakel Gaffar, MB. BS., DPH., FCM.
Family and community Medicine Department Medical Education Unit Faculty of Medicine Jazan University Jazan – KSA
5
Sec�on 1 Background
Blueprints In Health Profession Education Series |Constructing MCQs
6
1
What can MCQs Assess?
MCQS can test the cognitive domain well at all levels, although there are considerable criticisms of their use in professional education, as they tend to assess only the recall of knowledge. However, they can test any higher level of the cognitive domain and can also discriminate well between students, if the test is sufficiently well constructed. There is a great mass of research concluding that the testing of knowledge is the best way to determine expertise; accordingly, much effort is expended to ensure the good construction of questions. In Miller's Pyramid (shown below), MCQs, along with other types of written tests, can facilitate the assessment of the first two levels, which are the “Know” (knowledge) level and the “Knows How” (competence) level. The degree to which this assessment is achieved depends mainly on the level of competency exhibited in the construction of the MCQ items.
Blueprints In Health Profession Education Series |Constructing MCQs
7
2
Types of MCQs
There are many types of MCQs described in the literature. Of these, the most commonly used is the type that asks the students to choose the best answer, i.e., the Atype, which typically provides 45 options from which the student can choose. There is no psychometrical law behind the number of options; thus, one cannot make a judgment on the characteristic qualities of MCQs based upon the number of options available. There is great evidence suggesting that the technique of providing more options does not improve the psychometric value of MCQs; instead, the threeoption technique produces more tractability and discrimination than does the technique of providing more options. This effect occurs because the threeoption technique saves the time of teachers and gives them the opportunity to utilise that time to add more questions to an assessment, thereby improving the validity of tests. The second most popular MCQ format is that of Extended Matching Questions, called the RType, in which a theme is given for each question, students should match the options given (usually between 326 in number) with the scenarios (usually between 510 in number), and the matching process is introduced by a leadin question. Based on the aforementioned background regarding types of MCQs, we will discuss in detail several constructions of “Atype questions”. For further information on the other types of MCQs, please visit: Constructing Written Test Questions For the Basic and Clinical Sciences, by contributing Authors Susan M. Case, PhD and David B. Swanson, PhD, available at http://www.nbme.org/PDF/ItemWriting_2003/2003IWGwhole.pdf
Blueprints In Health Profession Education Series |Constructing MCQs
8
3
The anatomy of Atype MCQs
The “Atype” question is composed of three parts: the Stem, the Leadin question and the options. This scheme is illustrated below. 1. Stem (A) Should include as much information as possible regarding the point of interest. (B) Should be relatively long and should not contain any unwanted information. 2. Leadin question 3. Options Should be relatively short with no additional data than the stem. A. Distracter B. *(Key answer) C. Distracter D. Distracter E. Distracter Example MCQ: Stem: A 40yearold female has a 3day history of progressive weakness in her extremities. She has been healthy except for an upper respiratory tract infection 10 days prior to hospital admission. Her temperature is 37.8C (100 F), blood pressure is 130/80 mm Hg, pulse is 94/min, and respirations are 42/min and shallow. She has symmetric weakness of both sides of the face and in the proximal and distal muscles of the extremities. Sensation is intact. No deep tendon reflexes can be elicited; the plantar responses are flexor. Leadin: Which of the following is the most likely diagnosis for this case? Options: A. Acute disseminated encephalomyelitis B. GuillainBarré syndrome* Blueprints In Health Profession Education Series |Constructing MCQs
9
C. Myasthenia gravis D. Poliomyelitis E. Polymyositis The questions can be contextfree or contextrich stimulants. Contextrich questions can involve a case scenario, and the question to be answered may be dependent upon that scenario, whereas contextfree questions usually test factual knowledge, which is an important part of problem solving. However, in professional life, factual knowledge is typically needed only in a limited way; thus, it is recommended that to achieve the purpose of using the contextfree questions, one must increase their professional authenticity. Contextfree questions are easy to construct and are often preferred by teachers for this reason. By contrast, the contextrich question attempts to test higher levels of knowledge and reflects relevance to professional life, and thus it is more highly recommended for use in medical education contexts. The following are two examples of contextfree and contextrich stimulant questions, to elucidate the difference between the two types: 1 What is the most important mechanism of heart failure in children of one year old? A. Ventricular weakness B. Volume overload C. Pressure overload D. Ventricular dilatation 2 A 1yearold baby who is a known case of VSD was admitted to the ER by his mother, who complained of coughing and shortness of breath that had disturbed his feeding for the last 3 days. On examination, the baby was found to be ill, as his temperature was 38.7C and his pulse was a regular 110/min. He was dyspneic with a respiratory rate of 70/min and intercostal recessions, and he had a liver that was 4 cm below the costal margin. What is the most likely mechanism of his symptoms? A. Ventricular weakness B. Volume overload C. Pressure overload D. Ventricular dilatation Blueprints In Health Profession Education Series |Constructing MCQs
10
Although the two questions appear similar, the first is entirely an assessment of information recall, whereas the second stimulates students to think and resembles a real life situation that students may face in the future.
Blueprints In Health Profession Education Series |Constructing MCQs
11
Section 2 Pathology in MCQs
Blueprints In Health Profession Education Series |Constructing MCQs
12
4
Pathology in MCQs (threats to their validity)
There are many threats to the validity of any test. These threats can be categorised into two primary classifications: content underrepresentation (CU), which refers to an inadequate sampling of the content; and domain or Construct Irrelevance variance (CIV), which refers to the presence of variables that systematically interfere with the ability to meaningfully interpret scores or ratings (flawed items, inappropriate reading levels, statistically biased items, test wiseness, teaching to the test and cheating ). The role of teachers is to make sure that these threats and their related factors are minimised to ensure a maximally valid and fair test. Content underrepresentation can primarily be avoided by the use of a test blueprint, a term that has been borrowed from architecture. The underlying notion here is that an assessment process needs to be conducted according to a replicable plan, and the use of the blueprint will ensure that the test has been developed and mapped carefully against the educational objectives of the course, ensuring a fair representation of the assessment objectives. The other primary threat to validity is CIV, which can be summarised as the presence of illstructured items. Fairness and validity require that the questions should not only cover the important content but also be wellconstructed and easy to understand. One of the major problems in question construction is that very few staff members in medical schools have formal training in question construction. Moreover, problems in exam construction may also stem from the rushed process of exam construction. In the USA, for instance, it is common practice for a medical student to be exposed to highstakes exams for at least 3 hours every 46 weeks during his schooling, but paradoxically, most of the questions are prepared at the last minute, exams are often generated by many different individuals who taught the course, and little time is available to review the questions for overall quality before exams are to be provided to students. Perhaps the most important factor in creating CIV is that there may be no agreement between exam creators regarding the standardisation of the question formats for an exam.
Blueprints In Health Profession Education Series |Constructing MCQs
13
5
The management of MCQ Pathologies
The use of a blueprint can contribute to the elimination of not only CU but a portion of the CIV, as well, as it will ensure that, the test is using the appropriate methods to achieve the desired educational objective and that the students obtain unbiased results and scores from a wellrepresented test. Despite their importance as discussed above, blueprints are underutilised by teachers in medical schools; in a study published in 2003, the authors have indicated that very few medical schools in the USA are using blueprints for the validation of tests. Teachers must be trained in the construction and use of blueprints, although many publications have elucidated the steps involved in developing an exam blueprint. The Association of Medical Education in Europe (AMEE) has published a simple guide for blueprint development as part of its Twelve Tips series. This guide can be used for training and discussion. The following is an example of a blueprint for the written test in Paediatrics.
Objective 1
% of module content 30
Objective 2
Module content (objectives)
Knowledge Comprehension
Application/ Analysis
Total N (%)
4
4
12
20 (33.3)
25
4
2
7
13 (21.7)
Objective 3
15
2
4
4
10 (16.7)
Objective 4
20
2
3
5
10 (16.7)
Objective 5
10
2
2
3
7 (11.7)
Total
100
14
15
31
60
To help improve the validity of the inhouse MCQs, the following must be adopted by the authorities in medical schools: �
Development of faculty programmes that concentrate on assessment validity.
�
Adoption of the blueprint process for exam planning.
�
Development or adoption of guidelines for item construction.
Blueprints In Health Profession Education Series |Constructing MCQs
14
Section 3 Construction of MCQs
Blueprints In Health Profession Education Series |Constructing MCQs
15
6
Common Technical Item Flaws.
There is no guarantee that MCQ items written in textbooks and test banks are good questions. Ellsworth et al. (1990) found that approximately 60% of MCQs in instructor guides in introductory psychology textbooks contained itemwriting flaws (IWFs). In accounting test banks, Hansen (1997) found that 75% of MCQs violated at least one itemwriting guideline. StagnaroGreen & Downing (2006) found that 100% of the items used in the NEJM for CME were flawed—each item had at least 3 �laws. In another study, Terrant and Ware have indicated a very important conclusion, which is that many inhouse questions are flawed because the majority of teachers do not have formal training in item construction. The impact of technical flaws on students' performance Downing (2005) assessed the impact of �lawed items on pass rates of 5 tests in a US medical school and reported that 10–25% of the examinees who were classi�ied as failures would have passed if the flawed items had been removed from the tests. In this study, technical flaws only impacted the examination pass rates and not the higher achieving students. However, Tarrant & Ware (2008) examined 10 examinations given to HK nursing students and found that fewer examinees would have passed the test if the flawed items were removed (90.6% vs. 94.3%) and more examinees would have obtained scores ≥ 80% if the �lawed items were removed (20.9% vs. 14.5%). Therefore, the �lawed items benefited borderline students but penalised higher achieving students in this instance, challenging the assumption that higher achieving students are more test wise.
Terrant et al. has determined the most frequent violations to item construction, namely, either making the item too difficult or too easy. Technical item flaws are classified into two groups: Test wiseness and Irrelevant Difficulty.
Blueprints In Health Profession Education Series |Constructing MCQs
16
Table 1: Common Technical Item Flaws Testwiseness
Irrelevant Difficulty
1. Grammatical cues
1. Vague terms
2. Logical cues
2. Poor sequence in numeric data
3. Word repeats
3. Unnecessary information
4. Longest option is correct
4. Unclear or ambiguous
5. Absolute terms
information
6. Implausible distracters
5. >1 or no correct answer
7. Convergence (Options in C are all in 3
6. Fails the “Hand Cover Test”
options each, rest are only in 2 options)
The question should be answered without
8. All of the above
looking into the options.
9. None of the above
7. Negative(s) (not, except) in the stem
Examples of MCQs with technical item flaws follow: 1. Which of the following strategies is the best for the prevention of type 2 Diabetes Mellitus? A. A populationbased strategy. B. A highrisk strategy. C. Screening strategy. D. Population, highrisk and screening strategy.* (This demonstrates convergence, and the longest option is the correct option.) 2. What is the target population for primordial prevention activities? A. Highrisk groups. B. Total population.* C. Patients. D. Patients with complications. (A word is repeated in the stem and correct option. It is fine to repeat the word in the stem and distracter.) Blueprints In Health Profession Education Series |Constructing MCQs
17
3. Mr. Khalid is a patient admitted to the A & E in shock. After 10 minutes, you observe an ECG abnormality and subsequently a ventricular fibrillation (VF). Mr. Khalid is unconscious and not breathing. What is the most reliable sign of cardiac arrest? A. Absence of pulse.* B. Absence of breathing. C. Unconsciousness. D. All of the above. (Unnecessary information, unclear or ambiguous information (ECG abnormality) and the use of "all of the above".) 4. Which one of the following factors is NOT associated with Type 2 diabetes? A. Obesity. B. Physical inactivity. C. IGT and IFG. D. Viral infections.* (Negative (not, except) in the stem.) 5. A medical doctor is assessing a patient with a brainstem injury. After performing the Glasgow Coma Scale, what is his next step in the assessment plan? A. Conduct lumbar puncture. B. Perform arterial blood gas measurement. C. Check cranial nerve functioning and respiratory rate.* D. Perform a pulmonary wedge pressure measurement. (The logical cue is that the word "cranial" used in option C is logically homogeneous with the word "brainstem" in stem) 6. Which one of the following is true about malaria? A. Common in children under five. B. Caused by Plasmodium vivax. C. Fansidar is the drug of choice. D. Use of impregnated bed nets is advisable. (Fails the “Hand Cover Test”, and the options are heterogeneous.)
Blueprints In Health Profession Education Series |Constructing MCQs
18
7
Item-writing guidelines
To produce effective MCQs, one must ensure that the items are free from flaws. Item writing flaws in MCQs can affect student performance by resulting in items that are either more or less difficult to be answered. It is found by certain authors that flaws typically make the item less difficult. Additionally, flawed items may yield a certain ambiguity, without which 1025% of failed students would have passed the exam. In another study, the authors found that flawed item writing may reach 33% within one test and that those flawed items tend to be more difficult and lead more students to unfairly fail that exam. MCQs written at a lower cognitive level are significantly more likely to contain item writing flaws than those tests written for higher levels, which may be due to the attention and time spent by teachers in constructing the latter types of items. The table below illustrates evidencebased MultipleChoice (MC) ItemWriting Guidelines prepared by Haladyna et al. The table is adopted from Thomas M. Haladyna, Steven M. Downing, and Michael C. Rodriguez, A Review of MultipleChoice ItemWriting Guidelines for Classroom Assessment. Applied Measurement in Education. 2002, Vol. 15, 3, pp. 309–334.
Blueprints In Health Profession Education Series |Constructing MCQs
19
Table 2: A Revised Taxonomy of MultipleChoice (MC) ItemWriting Guidelines Content concerns 1. Every item should re�lect speci�ic content and a single speci�ic mental behaviour, as called for in test specifications (twoway grid, test blueprint). 2. Base each item on content that is important to learn; avoid trivial content. 3. Use novel material to test higherlevel learning. Paraphrase textbook language or language used during instruction when used in a test item to avoid testing for simple recall. 4. Keep the content of each item independent from the content of other items on the test. 5. Avoid overly speci�ic and overly general content when writing MC items. 6. Avoid opinionbased items. 7. Avoid trick items. 8. Keep vocabulary simple for the group of students being tested. Formatting concerns 9. Use the question, completion, and best answer versions of the conventional MC, the alternate choice, true false (TF), multiple truefalse (MTF), matching, and the contextdependent item and item set formats, but AVOID the complex MC (Type K) format. 10. Format the item vertically instead of horizontally. Style concerns 11. Edit and proof items. 12. Use correct grammar, punctuation, capitalisation, and spelling. 13. Minimise the amount of reading in each item. Writing the stem 14. Ensure that the directions in the stem are very clear. 15. Include the central idea in the stem instead of the choices. 16. Avoid window dressing (excessive verbiage). 17. Word the stem positively, avoid negatives such as NOT or EXCEPT. If negative words are used, use the word cautiously and always ensure that the word appears capitalised and boldface. Writing the choices 18. Develop as many effective choices as you can, but research suggests that three choices are adequate. 19. Make sure that only one of these choices is the right answer. 20. Vary the location of the right answer according to the number of choices. 21. Place the choices in logical or numerical order. 22. Keep the choices independent; the choices should not be overlapping. 23. Keep the choices homogeneous in content and grammatical structure. 24. Keep the length of choices approximately equal. 25. Noneoftheabove should be used carefully. 26. Avoid Alloftheabove. 27. Phrase choices positively; avoid negatives such as NOT. 28. Avoid giving clues to the right answer, such as a. Specific determiners including always, never, completely, and absolutely. b. Clang associations, choices identical to or resembling words in the stem. c. Grammatical inconsistencies that cue the testtaker to the correct choice. d. Conspicuous correct choice. e. Pairs or triplets of options that clue the testtaker to the correct choice. f. Blatantly absurd, ridiculous options. 29. Make all distracters plausible. 30. Use typical errors of students to write your distracters. 31. Use humour if it is compatible with the teacher and the learning environment.
Blueprints In Health Profession Education Series |Constructing MCQs
20
8
Writing high-cognitive-level MCQs
Bloom's taxonomy of educational objectives has described six cognitive levels: Knowledge, Comprehension, Application of knowledge, Analysis, Synthesis and Evaluation. As it has been illustrated by the figure below:
Most MCQs are written to assess lowerlevel thinking (cognitive) skills (recall, basic comprehension). Ideally, MCQs should be written to assess higherlevel thinking (cognitive) skills (comprehension, application and analysis). Students always learn what we assess, and not what we teach, and if we only examine them on facts and how to recall knowledge, they will only learn how to recall knowledge. However, if we change our assessment to focus on assessing upper cognitive functions (comprehension, application of knowledge and analysis) they will learn these skills instead. The good test is the test that assesses what we want our students to learn. To assess higher levels of thinking in MCQs, students need to be provided with information or a scenario to solve a problem, not merely asked to recall the facts. Blueprints In Health Profession Education Series |Constructing MCQs
21
Examples: Thrombocytopenia results from the deficiency of which blood component? A. Prothrombin B. Fibrinogen C. Thrombin D. Platelets * E. Iron (Adapted from the Institute of Medical and Health Sciences Education, University of Hong Kong) This question assesses recall of knowledge, but if we change the form of the question by adding some information for the students, it can assess higher cognitive functions rather than lower ones. The same question can be changed to be: Mrs. Chan, a 32yearold woman is admitted to the Accident and Emergency Department complaining of fatigue and generalised weakness. She reports that her gums have been bleeding excessively when brushing her teeth. An assessment reveals small red dots over her trunk and gums and bruising on her upper arms. A blood chemical analysis would most likely reveal a deficiency of which blood component in this patient? A. Prothrombin B. Fibrinogen C. Thrombin D. Platelets * E. Iron (Adapted from the Institute of Medical and Health Sciences Education, University of Hong Kong) The new question can now assess the application of knowledge rather than the recall of the knowledge. This new format is called scenario or vignette, and includes information about the patient and his/her health condition as well as other information needed by the student to solve the problem or to answer the question.
Blueprints In Health Profession Education Series |Constructing MCQs
22
How to write a vignette A good vignette should include some or all of the following parts: General information about the patient: �
Age,
�
Gender,
�
Socioeconomic status,
�
Education level, etc.
(e.g., A 40yearold female of low socioeconomic status) Site of Care: �
Hospital
�
Health centre
�
Department
�
Clinic
(e.g., comes to the outpatient clinic) Presenting Complaint (e.g., complaining of abdominal pain) Duration of symptoms (e.g., that has continued for 2 days). Patient health history �
Family History
�
chronic illness
�
previous attacks
�
Current treatments
Physical Findings �
Examination findings
�
+/ Results of Diagnostic Studies
�
+/ Initial Treatment, Subsequent Findings, etc.
The first three items are essential, and the others depend on the scenario and the question you want to ask. How should you develop your lead-in question? After developing your vignette, you must develop your question from that vignette. Below are some examples of questions that you can use. (You can ask more than one question from same vignette.)
Blueprints In Health Profession Education Series |Constructing MCQs
23
Suggested lead-in questions: �
A (patient scenario) with (symptoms and signs). Which of the following is the most common diagnosis?
�
A (patient scenario) has (symptoms and signs). Which [additional] finding would suggest a diagnosis of (disease 1) rather than (disease 2)?
�
A (patient scenario) has (sign). What diagnostic finding is indicative of [health problem]?
�
A (patient scenario) has (abnormal symptoms and signs). Which of the following ( lab. results) would be expected to be positive?
�
A (patient scenario) has (abnormal findings) and is taking (medications). Which of the following medications is the most likely cause of his (abnormal finding)?
�
A (patient scenario) has (abnormal findings). These observations suggest that the disease is a result of the (absence or presence) of which of the following (enzymes, hormones)?
�
A (patient description) has a (type of injury and location). Which of the following structures is most likely to be affected?
�
(time period) after a (event such as meal with certain foods), a (patient or group description) became ill with (symptoms and signs). Which of the following (organisms, agents) is most likely to be found upon analysis of (food)?
�
A (patient scenario) and (symptoms and signs, lab findings). What is the most appropriate (intervention), (at this time)?
�
A (patient scenario) with (certain symptoms and signs, type of injury). Which (expected outcome) would be most appropriate for this patient?
�
A (patient scenario) with (abnormal findings, family history). What type of (educational/counselling) intervention is most appropriate for this patient?
Development of the distracters First, you must write the key answer of the question; this answer should be agreed upon, and it is advisable to avoid answers that do not meet a consensus. Write the distracters afterward, as you must be sure that they are plausible distracters. These choices should be directly related to the leadin question (e.g., all are investigations, diagnoses, etc.) The golden rule in developing the MCQs is that these questions should be revised by a panel before being delivered to the students. Blueprints In Health Profession Education Series |Constructing MCQs
24
Bibliography 1. Mccoubrie, Paul. Improving the fairness of multiplechoice questions:a literature review. Medical Teacher. 2004, Vol. 26, 8, pp. 709–712. 2. Anderson, John. For Multiple Choice Questions. Medical Teacher. 1979, Vol. 1, 1. 3. Lau, Man Pang. A theory of multiplechoice examination. British Journal of Mcdical Education. 1972, 6. 4. Case, Susan M. Constructing Written Test Questions For the Basic and Clinical Sciences. Philadelphia : Na�onal Board of Medical Examiners, 2002. 5. Vleuten, Lambert W T Schuwirth & Cees P M van der. Different written assessment methods: what can be said about their strengths and weaknesses? MEDICAL EDUCATION. 2004, 38, pp. 974–979. 6. Marie Tarrant, James Ware. A comparison of the psychometric properties of three and fouroption multiplechoice questions in nursing assessments. Nurse Education Today. August 2010, Vol. 30, 6, pp. 539543. 7. Steven M Downing, Thomas M Haladyna. Validity threats: overcoming interference with proposed interpretations of assessment data. MEDICAL EDUCATION. 2004, 38, pp. 327333. 8. Downing, Steven M. Constructirrelevant Variance and Flawed Test Questions: Do Multiplechoice Itemwriting Principles Make Any Difference? AC AD E M I C ME D I C I N E. October 2002, Vol. 77, 10 Supplement, pp. S103S104. 9. Downing, Steven M. Validity: on the meaningful interpretation of assessment data. Medical Education . 2003, 37, pp. 830837. 10. Hamdy, Hossam. Blueprinting for the assessment of health care professionals. THE CLINICAL TEACHER. 2006, 3, pp. 175179. 11. Patrick D. Bridge, Joseph Musail,Robert Frank,Thomas Roe,Shalomo Sawilowsky. Measurement practices: methods for developing contentvalid student examinations. Medical Teacher. July 2003, Vol. 25, 4, pp. 414–421. 12. Ralph F. Jozefowicz, MD, Bruce MRalph F. Jozefowicz,Bruce M. Koeppen,Susan Case,Robert Galbraith,David Swanson, Robert H. Glew. The Quality of Inhouse Medical School Examinations. ACADEMIC MEDICIN E. February 2002, Vol. 77, 2, pp. 156161. 13. Sylvain Coderre, Wayne Woloschuk, Kevin Mclaughlin. Twelve tips for blueprinting. Medical Teacher. 2009, 31, pp. 322324. 14. Hansen, JD. Quality multiplechoice test questions: itemwriting guidelines and an analysis of auditing testbanks. J Educ Bus. 1997, Vol. 73, pp. 94–7. 15. Marie Tarrant, James Ware. Impact of itemwriting flaws in multiplechoice questions on student achievement in highstakes nursing assessments. Medical Education. 2008, 42, pp. 198–206.
Blueprints In Health Profession Education Series |Constructing MCQs
25
16. Downing, SM. The effects of violating standard itemwriting principles on tests and students: the consequences of using flawed test items on achievement examinations in medical education. Adv Health Sci Educ. 2005, Vol. 10, pp. 133–43. 17. Tarrant M, Knierim A, Hayes SK, Ware J. The frequency of itemwriting flaws in multiplechoice questions used in highstakes nursing assessments. Nurse Educ Today. 2006, Vol. 26, pp. 662–71. 18. Marie Tarrant, Aimee Knierim,Sasha K. Hayes,James Ware. The frequency of item writing flaws in multiplechoice questions used in high stakes nursing assessments. Nurse Education Today. 2006, 26, pp. 662671. 19. Thomas M. Haladyna, Steven M. Downing, Michael C. Rodriguez. A Review of MultipleChoice ItemWriting Guidelines for Classroom Assessment. Applied Measurement in Education . 2002, Vol. 15, 3, pp. 309–334.
Blueprints In Health Profession Education Series |Constructing MCQs
26
��������������������������������������������������������������������������� ��������������������������������������������������������������������������������� �����������������������������������������������������
