Development and Evaluation of a Multiple-Choice Question (MCQ) Summative Assessment Using Moodle SMARAK MISHRA, NARAYAN PRABHU & STEVEN WILLIAMS HEALTH PROFESSIONS AND TEL
Project Scope Assessment
Learning & teaching
Technology
Background Who are the students? What subjects? Why MCQ for assessment? ◦ ◦ ◦ ◦
Learning outcomes Recall/memory Wide area for coverage Efficiency -team teaching
The Process 1.Planning of summative MCQ questions 2.Develop Moodle based test 3.Administer the test 4.Evaluate (Feedback & Analysis) 5.What did we learn?
1.Planning & Developing MCQ Questions Understanding the spread of content Complex planning ◦ Subject areas ◦ Weighting from each area ◦ Level of difficulty of questions ◦ Number of batches of students ◦ Randomised questions ◦ No of questions required in the pool
Subjects
Level of Difficulty
Musculoskeletal Category 1 (Easy) Neuro
Category 2 ( Intermediate)
Cardiorespiratory
Category 3 (Hard)
Biomechanics
1. Planning & Developing MCQ Questions (Contd.) Area/ Subject Biomechani cs CSp/ TSp L spine Shoulder Elbow Wrist& Hand Pelvis & Hip Knee Ankle & Foot Neuro Anatomy Neuro Physiology CR Total
Teams GD/SM/JM/R MS Jhi/KMH Jhi/KMH LM/ST NP/JN
No of Qs in the exam
Category Pool 1
Pool 5
28
1 1 2 1
8 9 12 21
NP/JN
2
11
RM/JCr SH/MT/JB
1 2
9 12
GY/SH/SJH
2
15
DOC/ID/DE
4
DOC/ID/DE JH/ JC/JA/JR
Category Pool 2
Category Pool 3
2
13
2
10
1
5
23
1
8
2
11
1
4
3
27
1
13
1
10
1
4
6 30
40 215
2
18
3
15
1
7
1.Planning & Developing MCQ Questions (Contd.)
Staff development regarding MCQ
Dividing staff in to smaller teams & subject areas
Collate questions from the team
Verification of questions by a different team
2. Development of Moodle Test Input into the system
Proofreading
Sort into question paper
Pilot testing
Formative assessment
Administration of the Moodle test
3. Administration of MCQ Test ICT suite Team role-
◦ TELA ◦ Invigilator, ◦ Tutor
Run the test in batches Results release
4. Feedback and analysis Student feedback
Staff feedback
MCQ statistics
4. Feedback and Analysis (Continued) Issues Identified
Time consuming process A lots of questions were not fit for purpose Co-ordinating different teams within the unit team was hard work Uploading was time intensive Formatting in moodle drove us crazy! Some terms in Moodle analytics needed further understanding Students wanted the same set of questions ( Fairness) A couple of questions were out of unit content
4. Feedback and Analysis (Continued) Statistical Terms Coefficient of internal consistency (sometimes called Cronbach Alpha) - This is a measure of whether all the items in the quiz are testing basically the same thing. Thus it measures the consistency of the text, which is a lower bound for the validity. Higher numbers here are better. Error ratio - the variation in the grades comes from two sources. First some students are better than others at what is being tested, and second there is some random variation. We hope that the quiz grades will largely be determined by the student's ability, and that random variation will be minimised. The error ratio estimates how much of the variation is random, and so lower is better. Standard error - this is derived from the error ratio, and is a measure of how much random variation there is in each test grade. So, if the Standard error is 10%, and a student scored 60%, then their real ability probably lies somewhere between 50% and 70%.
4. Feedback and Analysis (Continued) Statistical Terms Facility Index - the percentage of students that answered the question correctly. Discrimination index - this is the correlation between the score for this question and the score for the whole quiz. That is, for a good question, you hope that the students who score highly on this question are the same students who score highly on the whole quiz. Higher numbers are better.
Quiz Structure Analysis
4. Feedback and Analysis (Continued) Individual Question analysis
5. What Did We Learn? Upfront time investment
Proofread & pilot
Elaborate planning
TELA advice & support - crucial
Importance of quality MCQs
Marking was instantaneous
Proof reading & rechecks
Specific feedback
Getting acquainted to system
Useful analytics
Smaller groups work better with input
Easy to rerun and update Need to update as per feedback
Any Question?
Our Contact Smarak Mishra
Narayan Prabhu
Steven Williams
Senior Lecturer in Physiotherapy
Senior Lecturer in Physiotherapy
TELA
Email:
[email protected]
Email:
[email protected]
Email:
[email protected]
Tel: 0161-247-2902
Tel: 0161-247-2954
Tel: 0161-247-2954
