Arthritis & Rheumatism (Arthritis Care & Research) Vol. 53, No. 5, October 15, 2005, pp 764 –771 DOI 10.1002/art.21438 © 2005, American College of Rheumatology
ORIGINAL ARTICLE
Cluster Randomized Controlled Trial of the Impact of a Computer-Assisted Learning Package on the Learning of Musculoskeletal Examination Skills By Undergraduate Medical Students PIRASHANTHIE VIVEKANANDA-SCHMIDT, MARTYN LEWIS, ANDREW B. HASSELL, THE ARC VIRTUAL RHEUMATOLOGY CAL RESEARCH GROUP
AND
Objective. To identify whether there was measurable impact of a specific computer-assisted learning (CAL) package, “Virtual Rheumatology,” on the learning of musculoskeletal examination skills by medical students. Methods. We conducted 2 parallel, cluster-randomized controlled trials using undergraduate curricula at 2 locations: Newcastle and London, UK. Medical students attending a musculoskeletal rotation were allocated to the intervention (Virtual Rheumatology CD) or the control arm of the study by placement group. A formative 14-item objective structured clinical examination (OSCE) assessment on the examination of shoulder and/or knee joints was the main outcome measure at Newcastle. At London, a 17-item knee station formed part of the summative OSCE. We also used a questionnaire including a 15-item confidence log (C-Log) for self assessment of musculoskeletal examination skills and knowledge. Analysis was by intention to teach. Results. At Newcastle, there were 112 students in the CD allocated group and 129 in the non-CD group. The CD allocated group performed significantly better on the OSCE (P ⴝ 0.002) and C-Log (P ⴝ 0.005) than the non-CD group. At London, there were 48 students in the CD allocated group and 65 in the non-CD group. The CD allocated group performed better on the knee OSCE than the non-CD group (adjusted P ⴝ 0.040), but there was little difference in the change in C-Log scores from baseline to followup between the 2 groups (P ⴝ 0.582). Conclusion. The Virtual Rheumatology CD has a positive impact on the acquisition of musculoskeletal examination skills in medical students. Further study is needed to see if similar advantages could be gained in other clinical specialities and how CAL resources could be effectively integrated into the medical curriculum. KEY WORDS. Randomized; Computer-assisted learning; Musculoskeletal skills.
INTRODUCTION Musculoskeletal problems are a common reason for consultation in both primary (1) and secondary (2) care. There is, however, evidence of deficiencies in musculoskeletal examination skills among health professionals (2) and undergraduate medical students (3). Although the last decade Supported by an educational project grant from the Arthritis Research Campaign. Pirashanthie Vivekananda-Schmidt, D.Phil, Martyn Lewis, PhD, Andrew B. Hassell, MD: Keele University, Keele, North Staffordshire, United Kingdom. The members of the Arc Virtual Rheumatology CAL Research Group are listed in Appendix A. Address correspondence to Pirashanthie VivekanandaSchmidt, D.Phil, Keele University, Keele, Staffordshire, UK ST5 5BG. E-mail:
[email protected]. Submitted for publication January 25, 2005; accepted in revised form May 4, 2005.
764
has seen considerable changes in many medical schools, with more structured teaching of clinical skills, there remains considerable variation in teaching between and even within medical schools (4). Because of the pressure on the medical curriculum to cover all medical specialties, improvement in the learning of musculoskeletal examination skills is unlikely to result from an increase in allocated time, but rather from improved quality of teaching and/or an improvement in the curriculum. Computer-assisted learning (CAL) programs offer a number of advantages. They allow self-paced autonomous learning and improve access to learning materials for students who are geographically dispersed. The learning content can be presented through multiple media. Within a musculoskeletal curriculum, this can help students learn when presenting conceptually difficult materials, such as 3-dimensional relationships in anatomy, by simplifying information with the use of animated visual images. Al-
Impact of a CAL Package on the Learning of Musculoskeletal Examination Skills though students prefer and learn best with “real-life” examples (4,5), issues such as the move to ambulatory care and concern over patient confidentiality have made patient-based teaching more challenging to achieve. Also, CAL resources can offer the students the advantage of acquiring the core skills and knowledge prior to contact with patients, thereby allowing the students to best utilize patient contact time. Research investigating the effectiveness of CAL on student learning within medical education is limited (6). The increasingly popular role of CAL in undergraduate medical education, as well as the cost and labor involved in its design, emphasize the importance of evaluating its usefulness in learning. Current evidence is equivocal and not methodologically strong (6 –9). Our aim was to evaluate the impact of CAL on undergraduate clinical skills learning by using a model, the CD-ROM Virtual Rheumatology (10).
MATERIALS AND METHODS We conducted 2 randomized controlled trials in 2 medical schools in the United Kingdom: University of Newcastle upon Tyne and Royal Free, and University College Medical School London. The curriculum structure at the 2 medical schools differed, so the trials were conducted separately, but addressed the same research objective. There were 2 outcome measures: an objective structured clinical examination (OSCE) and a confidence log (C-Log). OSCE is a valid and widely used measure for the assessment of clinical examination skills (11–13), and confidence is an important factor in educational achievement (14). The CD-ROM Virtual Rheumatology (10). The CD-ROM was produced with the support of an Arc funded educational project grant. It consists of 3 interactive modules: Introduction to Terms in Anatomy; Clinical Anatomy and Examination of the Shoulder Joint; and Clinical Anatomy and Examination of the Knee Joint. These modules include illustrations of anatomic terms and living anatomy, photographs of the skeleton, and digital video sequences to demonstrate examination techniques. A self-assessment component is included in each module. Participants. At Newcastle, third-year undergraduate medical students attending a musculoskeletal week during a 12-week clinical skills module were included in the study. Prior to the start of the placement, students had attended 1 week of clinical skills teaching. In London, third-year undergraduates rotating through a 5-week locomotor module were included. Unlike Newcastle, there was no preplacement teaching of clinical skills, although musculoskeletal examination had been taught as part of the introductory course at the beginning of the third year. Cluster randomization. The cluster unit was the clinical placement group; each group was randomized to the intervention or control arms. Block randomization was
765
used (block size ⫽ 2) to ensure that approximately equal numbers of students were distributed between the 2 arms of the study. An independent researcher not involved in outcome assessment was responsible for group allocation, using a computer-generated random number table. Outcome measures. For the initial Newcastle groups, the OSCE consisted of a station on knee examination and one on shoulder examination. For logistic reasons, students in later groups were examined on one station only. Each station was 6 minutes long. Assessors were blind to group allocation. There was a 14-item checklist for scoring the OSCE. For each item a score of 0, 1, or 2 was given for “not done,” “done,” and “done well,” respectively. In addition, we also added a global rating scale (GRS) as a supplementary measure (a 10-cm visual analog scale ranging from “poor” [score 0] to “excellent” [score 100]) to the OSCE score sheets. GRSs have been shown to be valid measures for the assessment of clinical skills (15). Stations were video taped to ascertain the consistency in scoring the OSCE. In the London trial, as part of the end of year summative OSCE, one station assessed students’ examination of the knee joint. The station was 5 minutes long and assessors were blind to group allocation. The OSCE checklist consisted of 17 items, and each item was scored 0 or 1 for “not done” and “done,” respectively. A prevalidated, self-complete, 15-item confidence log was used, including 5 items assessing clinical examination of the shoulder and 4 for the knee, 2 addressing general musculoskeletal knowledge, and 4 on clinical anatomy of the knee and shoulder. Each item was rated on an 11-point numerical rating scale ranging from “not at all confident” (score 0) to “very confident” (score 10). Procedure. Figure 1 illustrates the trial procedures. Participants completed a baseline questionnaire on the first day of the musculoskeletal rotation. This included the perceived C-Log and questions on age, sex, access to computers, experience in using computer-based learning material, and use of other learning resources. On day 1 of the musculoskeletal module in the Newcastle trial, students allocated to the intervention arm were each given a CD followed by 1 hour access time to a computer laboratory during lunchtime (arranged in response to findings of the pilot, which suggested that CD
Figure 1. Flowchart of trial procedures. OSCE ⫽ objective structured clinical examination.
766 use would be higher if access to computers was better). On day 1 of the module in the London trial, students allocated to the intervention were given a verbal introduction to the content of the CD-ROM, and each student was given a CD. Students in the Newcastle trial took part in a formative OSCE on shoulder and/or knee examination on day 4. Students were also asked to complete a followup questionnaire, which included the perceived C-Log, on day 4 of the week’s placement. Within the CD allocated group, the questionnaire also asked whether the individual had used the CD-ROM. Students in the London trial completed a knee OSCE as part of the end of year assessment 1–10 months after completion of the module. The followup questionnaire was completed on day 24 of the placement. Statistical analysis. Cluster sizes were expected to be ⬃20 on average, and between-cluster variances were expected to be small relative to within-cluster variances (expected intraclass correlation coefficient ⬍0.05) (16). On this premise, a sample size of between 120 –240 participants was required to detect a moderate to large effect size (measured as the ratio of the difference in mean scores between groups to the pooled SD of scores) of 0.5– 0.8 (17) with 80% power and 5% significance level (2-tailed). Item scores on the OSCE were summed: the scale range was 0 –28 for both the shoulder and knee stations in Newcastle and 0 –17 for the knee station in London. If both stations were completed in Newcastle, an overall score was calculated by taking the average of the shoulder and knee scores. The primary measure for the OSCE was the overall score based on scores for the shoulder and/or knee (OSCE total). Subanalyses were carried out for the separate scores of the shoulder and knee stations. Secondary analysis was carried out using the OSCE-GRS (scale: 0 –100). For the perceived C-Log, C-Logtotal was derived by averaging the 15 individual scores (scale: 0 –10). Subanalyses were also carried out for average scores of the 5 clinical shoulder items and the 4 clinical knee items. Outcome data relating to the C-Log summary scores at followup are given both in terms of absolute scores and change scores compared with baseline. The primary outcome was the OSCE. The principal analysis was performed using the intention-to-teach approach, i.e., comparing the outcomes of all students followed up in the CD allocated group with those of all students followed up in the non-CD group. A secondary (per protocol) analysis was also carried out on the main data. This approach compared CD users in the CD allocated group (excluding CD non-users in the CD allocated group) with all the participants in the non-CD group. Exploratory subgroup analyses of the main data were carried out in relation to other baseline factors. Analysis was carried out using a hierarchical linear regression with 2 levels (cluster level and student level). “Cluster” was entered as a random variable to account for the variance between individual clusters. The primary analysis was unadjusted but a sensitivity analysis was also carried out adjusting for the baseline covariates age, sex, computer access, experience with CAL, and C-Log scores
Vivekananda-Schmidt et al of participants. MlwiN (18) and SPSS (19) were used to perform the statistical analyses. Statistical significance is given at the 5% probability level (2-tailed). Consent and ethics approval. Approval was obtained from respective chairs of the University Ethics committees. Verbal consent was obtained from all students for the completion of questionnaires. At Newcastle, written consent was obtained from the students for video recording of the OSCE and use of the data for research. At London, the OSCE station was one station in a set of summative final examination stations. It was agreed that this score should be excluded from the aggregate OSCE score, if the difference between the 2 groups on the knee station was sufficiently large to significantly affect the students’ aggregate score on the OSCE.
RESULTS Recruitment and participant flow. Figure 2 illustrates the flow of subjects through the trial. Baseline characteristics of those recruited are shown in Table 1. At Newcastle, 241 students were recruited. Thirteen clusters were randomized between May 2002 and July 2003: 6 to the CD allocated group and 7 to the non-CD group. The median cluster size was 18 (range 18 –20). A total of 112 subjects were assigned to the CD allocated group, and 129 to the non-CD group. At London, 113 students were recruited. Seven clusters were randomized between October 2002 and July 2003: 3 to the CD allocated group and 4 to the non-CD group. The median cluster size was 16 (range 14 –18). A total of 48 subjects were assigned to the CD allocated group, and 65 to the non-CD group. OSCE scores. In the Newcastle trial, 156 students underwent OSCE assessment: 34 completed both shoulder and knee stations, and 122 completed either one of the stations. Baseline characteristics of the students who completed the OSCE were similar to those who did not, e.g., baseline mean C-Log scores were 6.5 and 6.4, respectively. OSCE scores for the 2 study groups and their differences are summarized in Table 2. The primary outcome was significantly higher in the CD allocated group: mean difference 1.6 (95% confidence interval [95% CI] 0.5, 2.6; P ⫽ 0.002); effect size ⫽ 0.5. Observed mean differences were greater for the assessment of the shoulder than the knee. In the London trial, 105 students underwent the OSCE assessing knee examination. The mean score of the CD allocated group was significantly higher than that of the non-CD group (Table 2) after adjusting for baseline covariates (unadjusted P ⫽ 0.064; adjusted P ⫽ 0.040). C-Log scores. At Newcastle, ⌬C-Logtotal scores were available for 178 students at the primary end point. The baseline profile of these participants was similar to those 70 students whose scores were not available, e.g., baseline mean C-Log scores were 6.6 and 6.3, respectively. C-Log scores for the 2 study groups and their differences are
Impact of a CAL Package on the Learning of Musculoskeletal Examination Skills
767
Figure 2. Trial profile, flow of participants through trial. * There were no specific exclusion criteria. All students approached at baseline were both eligible and consented to participate in the study. OSCE ⫽ objective structured clinical examination; ⌬C-Log ⫽ change in the perceived confidence log from baseline to followup.
summarized in Table 3. The CD allocated group had a significantly higher mean ⌬C-Logtotal score than the non-CD group: mean difference 0.3 (95% CI 0.1, 0.6; P ⫽ 0.005); effect size ⫽ 0.4. Mean scores were significantly different between the CD allocated group and non-CD group for the ⌬C-Log shoulder assessment, but not for the ⌬C-Log knee assessment. At London, change scores were available for 80 students at the primary end point. There were no statistically significant differences between the 2 study groups (Table 3). Further analyses. On subgroup analysis, statistical interaction in relation to the effect size of the intervention (for both OSCE and ⌬C-Logtotal) separately with respect to sex and baseline C-Logtotal was not statistically significant (P ⬎ 0.05). On-CD analysis, mean scores were significantly higher for CD users than for participants in the non-CD group (per protocol). The mean difference for the OSCE was 2.3 (95% CI 1.2, 3.4), and for the ⌬C-Logtotal was 0.3 (95% CI 0.1, 0.6). In the CD allocated group, CD users were more likely than CD nonusers to access a computer at home and have a higher baseline confidence (Table 4).
DISCUSSION Our findings provide evidence that the provision of a CAL package has measurable educational value for the learning of musculoskeletal examination skills. At both Newcastle and London, the CD allocated group performed significantly better in the musculoskeletal OSCE. Also, the increase in students’ perceived confidence in performing joint examination was significantly larger in the CD group than the non-CD group in Newcastle, although this was not the case in London.
Newcastle students received 1 week of clinical skills training immediately prior to the musculoskeletal rotation, whereas London students received their musculoskeletal teaching at the beginning of the academic year (i.e., 1– 8 months prior to undertaking the current rotation). This contrast in curriculum structure was reflected in the different baseline scores of the 2 schools: Newcastle students had higher baseline perceived confidence than London students. However, it is encouraging that scores were comparably high in both medical schools at followup. As an additional learning resource, the CD appears to have made a significant impact on self confidence in Newcastle. However, for students who start off with lower self confidence as in London, any difference made by the CD on perceived confidence may be small relative to the overall increase in confidence gained from a full teaching program. Although perceived confidence is not an accurate indicator of the level of learning of the student (20,21), it is an important predictor of student success in learning (14). The higher baseline confidence of CD users in comparison with CD nonusers in London and Newcastle suggests that students’ confidence is a factor that determines the likelihood of using novel resources such as CAL tools. In 2001, Greenhalgh (6) identified 12 randomized controlled trials (RCTs) of CAL programs in undergraduate medical education, 8 of which involved ⬍100 participants. Rogers et al (22) found that the use of a CAL program as the sole teaching method resulted in an inferior performance in tying a surgical knot compared with a seminar approach. However, the seminar method included personalized feedback with 1 tutor per 6 students. A number of studies compared classroom teaching or printed text methods with a CAL program (23–25). However, it seems likely that CAL would be available in addition to printed
768
Vivekananda-Schmidt et al
Table 1. Baseline characteristics of the CD group and non-CD group*
Newcastle Age, years 19–25 ⬎25 Sex Female Male Most regular access to computers Home University Other Experience using CAL material No Yes C-Log scores, mean ⫾ SD‡ C-Logtotal C-Logshoulder C-Logknee London Age, years 19–25 ⬎25 Sex Female Male Most regular access to computers Home University Other Experience using CAL material No Yes C-Log scores, mean ⫾ SD C-Logtotal C-Logshoulder C-Logknee
CD group
Non-CD group
n ⫽ 112†
n ⫽ 129†
103 (92) 9 (8)
122 (96) 5 (4)
77 (69) 35 (31)
87 (67) 42 (33)
36 (33) 71 (64) 3 (3)
27 (22) 92 (75) 3 (3)
7 (6) 105 (94)
16 (12) 113 (88)
6.6 ⫾ 1.1 6.4 ⫾ 1.4 7.2 ⫾ 1.2 n ⫽ 48†
6.4 ⫾ 1.3 6.2 ⫾ 1.5 6.8 ⫾ 1.4 n ⫽ 65†
45 (93.8) 3 (6.2)
64 (98.5) 1 (1.5)
32 (66.7) 16 (33.3)
36 (55.4) 29 (44.6)
18 (37.5) 30 (62.5) 0 (0)
20 (31.3) 44 (68.8) 0 (0)
8 (16.7) 40 (83.3)
8 (12.3) 57 (87.7)
38 ⫾ 1.5 2.9 ⫾ 1.8 3.6 ⫾ 1.6
3.7 ⫾ 1.3 3.0 ⫾ 1.5 3.4 ⫾ 1.5
* Values are number (percentages) unless stated otherwise. CAL ⫽ computer-assisted learning; C-Log ⫽ confidence log; C-Logtotal ⫽ average score across all 15 items of the confidence log; C-Logshoulder ⫽ average score for the 5 items assessing clinical examination of the shoulder; C-Logknee ⫽ average score for the 4 items assessing clinical examination of the knee. † Numbers do not always add to the total due to some missing data. In Newcastle, 111 in the CD group and 127 in the non-CD group completed the C-Log. In London, 48 in the CD group and 64 in the non-CD group completed the C-Log. ‡ Confidence log scales range from 0 (not at all confident) to 10 (very confident).
materials and often alongside face to face teaching; hence, the importance of our study, which was aimed at identifying any additional benefit from a CAL program. Summers et al (23) found that students taught via the CAL program performed better in terms of tying surgical knots than those receiving instruction from a teacher or by videotape. However, the study was somewhat artificial in that the tutor was not permitted to provide performance feedback to students in any of the groups, including the didactic group. An extension of the same search strategy by Greenhalgh (6) to present day revealed 24 studies, none of which were RCTs and none used skills learning as an outcome measure. Designing a valid, educational RCT poses a number of methodologic challenges. Statistical power issues were ad-
dressed by determining sample size a priori. Currently the gold standard for assessing clinical competence is the OSCE (26). The sensitivity of the OSCE may be restricted by the small range of scoring points on the checklist. Moreover, not all students underwent the OSCE in Newcastle, but there was little reason to suspect, when reviewing the baseline data, that these students were a select sample of the study population. Furthermore, the true effect size of CD use might have been underestimated by the intention-to-teach analysis, because only 50% of the participants in the intervention group actually used the CD. Embedding the resource in the curriculum would have improved uptake (9,27,28). The fact that OSCE results were statistically significant in favor of the CD-allocated group in both centers, despite the different time intervals
Impact of a CAL Package on the Learning of Musculoskeletal Examination Skills
769
Table 2. Summary of OSCE scores for the comparison of CD and non-CD groups (intention-to-teach approach)* CD group OSCE Newcastle Shoulder/knee# Shoulder/knee–GRS†† Shoulder# Shoulder–GRS (range 0–100) Knee# Knee–GRS (range 0–100) London Knee (range 0–17)
Non-CD group
Difference† (95% CI)
n‡
Mean ⴞ SD
n‡
Mean ⴞ SD
Unadjusted§
75 59 43 29 46 30
20.5 ⫾ 3.2 75.0 ⫾ 12.2 20.2 ⫾ 4.0 77.0 ⫾ 12.4 21.2 ⫾ 2.7 73.1 ⫾ 11.9
81 48 50 25 51 23
18.8 ⫾ 3.2 67.9 ⫾ 12.4 17.6 ⫾ 2.9 70.1 ⫾ 12.5 20.5 ⫾ 3.4 65.5 ⫾ 12.2
1.6 (0.5, 2.6)** 7.0 (2.4, 11.7) 2.4 (0.9, 3.9) 6.8 (0.3, 13.4) 0.7 (⫺0.5, 1.9) 7.5 (1.1, 14.0)
1.9 (0.8, 3.0) 7.5 (2.4, 12.7) 2.7 (1.1, 4.3) 7.3 (0.0, 14.6) 1.0 (⫺0.3, 2.3) 7.1 (0.8, 13.5)
47
15.1 ⫾ 1.8
58
14.1 ⫾ 2.4
0.8 (0.0, 1.7)††
0.9 (0.0, 1.8)††
Adjusted¶
* OSCE ⫽ objective structured clinical examination; 95% CI ⫽ 95% confidence interval; GRS ⫽ global rating scale. † Mean difference in scores for CD group – non-CD group using MLwiN fitting 2-level linear models with random intercept and slope (taking into account the hierarchical structure with random cluster effects). ‡ Number of OSCE scores analyzed. § Crude analysis not adjusted for baseline factors. ¶ Analysis adjusted for baseline characteristics of participants (age, sex, computer access, experience of CD-ROM, C-Log score). # OSCE scale range: 0 –28. ** Primary outcome. †† Precise confidence intervals (rounding to 3 decimal places): unadjusted analysis (⫺0.046, 1.682); adjusted analysis (0.040, 1.828).
between intervention and assessment, lends weight to the argument that the CD has an impact on improving student performance. A number of questions, however, remain unanswered. What would be the cost implications of providing CAL resources to undergraduates? The cost of designing the Virtual Rheumatology CD was £11,740 ($22,045). A webbased resource can be more cost effective because it will reduce the cost of producing CD-ROMs and the cost of
updating the resource. However, the disadvantage is that students need Internet access of sufficient speed for interactive visual and animated material. Further work also needs to be carried out to determine if CAL resources are of equal educational value in all specialties or whether they are of more interest to only certain specialties. Most importantly, the educational impact of a CAL resource will probably depend on its quality. In conclusion, we have shown in 2 well-established
Table 3. Summary of confidence log (C-Log) scores for the comparison of CD and non-CD groups (intention-to-teach approach)* CD group
Newcastle C-Logtotal ⌬C-Logtotal C-Logshoulder ⌬C-Logshoulder C-Logknee ⌬C-Logknee London C-Logtotal ⌬C-Logtotal C-Logshoulder ⌬C-Logshoulder C-Logknee ⌬C-Logknee
Non-CD group
Difference† (95% CI)
n‡
Mean ⴞ SD
n‡
Mean ⴞ SD
82 81 82 81 83 83
7.4 ⫾ 1.0 0.7 ⫾ 0.8 7.1 ⫾ 1.2 0.6 ⫾ 1.2 7.8 ⫾ 1.2 0.4 ⫾ 1.0
98 97 99 98 101 100
6.7 ⫾ 1.3 0.3 ⫾ 0.8 6.3 ⫾ 1.5 0.2 ⫾ 1.1 7.3 ⫾ 1.3 0.4 ⫾ 1.0
0.7 (0.3, 1.0) 0.3 (0.1, 0.6) 0.8 (0.4, 1.2) 0.4 (0.0, 0.7) 0.5 (0.2, 0.9) 0.1 (⫺0.2, 0.4)
35 35 35 35 35 35
7.2 ⫾ 1.3 3.5 ⫾ 1.7 6.9 ⫾ 1.4 4.0 ⫾ 1.9 7.5 ⫾ 1.4 4.0 ⫾ 1.8
46 45 46 46 46 45
7.4 ⫾ 1.1 3.7 ⫾ 1.4 7.0 ⫾ 1.5 4.0 ⫾ 1.8 7.7 ⫾ 1.0 4.2 ⫾ 1.6
⫺0.2 (⫺0.7, 0.3) ⫺0.2 (⫺0.9, 0.5) ⫺0.2 (⫺0.8, 0.5) 0.0 (⫺0.8, 0.8) ⫺0.2 (⫺0.8, 0.3) ⫺0.3 (⫺1.1, 0.5)
Unadjusted§
Adjusted¶
0.4 (0.1, 0.6) 0.4 (0.1, 0.8) 0.1 (⫺0.1, 0.4)
⫺0.2 (⫺0.7, 0.3) ⫺0.1 (⫺0.8, 0.6) ⫺0.3 (⫺0.8, 0.2)
* Confidence log absolute scales range from 0 (not at all confident) to 10 (very confident); change scales (⌬C-Log) represent scores at followup minus those at baseline and have a possible range of –20 to ⫹20. C-Logtotal ⫽ average score across all 15 items of the confidence log; C-Logshoulder ⫽ average score for the 5 items assessing clinical examination of the shoulder; C-Logknee ⫽ average score for the 4 items assessing clinical examination of the knee. † Mean difference in scores for the CD group and non-CD group using MLwiN fitting 2-level linear models with random intercept and slope (taking into account the hierarchical structure with random cluster effects). ‡ Number of C-Log scores analyzed. § Crude analysis not adjusted for baseline factors. ¶ Analysis adjusted for baseline characteristics of participants (age, sex, computer access, experience of CD-ROM, C-Log score).
770
Vivekananda-Schmidt et al
Table 4. Baseline characteristics of CD users and CD non-users among participants in the CD group*
Newcastle Age, years 19–25 ⬎25 Sex Female Male Most regular access to computers Home University Teaching hospital Experience using CAL material No Yes C-Logtotal score, mean ⫾ SD‡ London Age, years 19–25 ⬎25 Sex Female Male Most regular access to computers Home University Experience using CAL material No Yes C-Logtotal score, mean ⫾ SD‡
CD users
CD non-users
n ⫽ 54†
n ⫽ 35†
51 (94) 3 (6)
33 (94) 2 (6)
35 (65) 19 (35)
24 (69) 11 (31)
20 (37) 33 (61) 1 (2)
9 (27) 23 (70) 1 (3)
2 (4) 52 (96) 7.0 ⫾ 0.9 n ⫽ 20†
3 (9) 32 (91) 6.2 ⫾ 1.3 n ⫽ 16†
19 (95) 1 (5)
15 (94) 1 (6)
13 (65) 7 (35)
13 (81) 3 (19)
10 (50) 10 (50)
5 (31) 11 (69)
1 (5) 19 (95) 3.9 ⫾ 1.7
4 (25) 12 (75) 3.6 ⫾ 1.3
* Values are numbers (percentages) unless stated otherwise. CAL ⫽ computer-assisted learning; C-Log ⫽ confidence log. † Sample sizes reflect the numbers of participants who responded to the question asking about use of the CD; 23 of the 112 participants in the CD group in Newcastle and 12 out of the 48 participants in the CD group in London did not complete this question. ‡ Confidence log scales range from 0 (not at all confident) to 10 (very confident). C-Logtotal is the average score across all 15 items of the confidence log.
musculoskeletal curricula that a CAL resource (Virtual Rheumatology CD) has a measurable educational impact on the learning of musculoskeletal examination skills. The features of CAL offer potentially more than the traditional learning resources by, for example, enabling the students to better use the clinical experience and limited contact time with clinicians and patients. Further studies should explore the limitations to the effective embedding of CAL resources into the medical curriculum and to investigating whether the same advantages could be achieved across other clinical areas. Finally, this study demonstrates that RCTs may have a role in assessing educational interventions, and can be used in conjunction with other nonquantitative methodologies for informing medical educators regarding issues of best practice.
ACKNOWLEDGMENTS The authors would like to thank professor Peter Croft for his support throughout this research project. Dr. Louisa Badcock was involved in the preliminary study design.
REFERENCES 1. McCormick A, Fleming D, Charlton J. Morbidity statistics from general practice: fourth national study 1991–1992. London: HMSO; 1995. OPCS Series MB5 No.3. 2. Doherty M, Abawi J, Pattrick M. Audit of medical inpatient examination: a cry from the joint. J R Coll Physicians Lond 1990;24:115– 8. 3. Coady D, Kay L, Walker D. Regional musculoskeletal examination: what the students say. J Clin Rheumatol 2003; 9:67–71. 4. Kay L, Walker D. Improving musculoskeletal clinical skills teaching: a regionwide audit and intervention study. Ann Rheum Dis 1998;57:656 –9. 5. Jolly BC, Jones A, Dacre JE, Elzubeir M, Kopelman P, Hitman G. Relationships between students’ clinical experiences in introductory clinical courses and their performances on an objective structured clinical examination (OSCE). Acad Med 1996;71:909 –16. 6. Greenhalgh T. Computer assisted learning in undergraduate medical education. BMJ 2001;322:40 – 4. 7. Carr MM, Reznick RK, Brown DH. Comparison of computerassisted instruction and seminar instruction to acquire psychomotor and cognitive knowledge of epistaxis management. Otolaryngol Head Neck Surg 1999;121:430 – 4. 8. Elves AW, Ahmed M, Abrams P. Computer-assisted learning:
Impact of a CAL Package on the Learning of Musculoskeletal Examination Skills
9. 10. 11. 12. 13. 14. 15. 16.
17. 18. 19. 20. 21. 22.
experience at the Bristol Urological Institute in the teaching of urology. Br J Urol 1997;80 Suppl 3:59 – 62. Bresnitz EA, Gracely EJ, Rubenstein HL. A randomized trial to evaluate a computer-based learning program in occupational lung disease. J Occup Med 1992;34:422–7. Virtual rheumatology [CD-ROM]. London, England: Arthritis Research Campaign; 1999. Harden RM. Twelve tips for organizing an objective structured clinical examination (OSCE). Med Teach 1990;12:259 – 64. Simon SR, Volkan K, Hamann C, Duffey C, Fletcher SW. The relationship between second-year medical students’ OSCE scores and USMLE Step 1 scores. Med Teach 2002;24:535–9. Woodburn J, Sutcliffe N. The reliability, validity and evaluation of the objective structured clinical examination in podiatry. Assess Eval Higher Educ 1996;21:131– 46. Bong M, Hocevar D. Measuring self-efficacy: multitrait-multimethod comparison of scaling procedures. Appl Meas Educ 2002;15:143–71. Hodges B, McIlroy JH. Analytic global OSCE ratings are sensitive to level of training. Med Educ 2003;37:1012– 6. Smeeth L, Ng ES. Intraclass correlation coefficients for cluster randomized trials in primary care: data from the MRC Trial of the Assessment and Management of Older People in the Community. Control Clin Trials 2002;23:409 –21. Cohen J. Statistical power analysis for the behavioural sciences. 2nd ed. Hilsdale (NJ): Lawrence Earlbaum Associates; 1998. Rasbash J, Browne W, Goldstein H, Yang M, Plewis I, Healy M, et al. A user’s guide to MLwinN Version 1.10. London: Institute of Education, University of London; 2001. SPSS for Windows. Chicago: SPSS; 2003. Liddell MJ, Davidson SK. Student attitudes and their academic performance: is there any relationship? Med Teach 2004;26:52– 6. Morgan PJ, Cleave-Hogg D. Comparison between medical students’ experience, confidence and competence. Med Educ 2002;36:534 –9. Rogers DA, Regehr G, Yeh KA, Howdieshell TR. Computerassisted learning versus a lecture and feedback seminar for
23. 24.
25.
26.
27. 28.
771
teaching a basic surgical technical skill. Am J Surg 1998;175: 508 –10. Summers AN, Rinehart GC, Simpson D, Redlich PN. Acquisition of surgical skills: a randomized trial of didactic, videotape, and computer-based training. Surgery 1999;126:330 – 6. D’Alessandro DM, Kreiter CD, Erkonen WE, Winter RJ, Knapp HR. Longitudinal follow-up comparison of educational interventions: multimedia textbook, traditional lecture, and printed textbook. Acad Radiol 1997;4:719 –23. Bell DS, Fonarow GC, Hays RD, Mangione CM. Self-study from web-based and printed guideline materials: a randomized, controlled trial among resident physicians. Ann Intern Med 2000;132:938 – 46. Probert CS, Cahill DJ, McCann GL, Ben-Shlomo Y. Traditional finals and OSCEs in predicting consultant and self-reported clinical skills of PRHOs: a pilot study [published erratum appears in Med Educ 2003;37:1058]. Med Educ 2003;37:597– 602. Mehta MP, Sinha P, Kanwar K, Inman A, Albanese M, Fahl W. Evaluation of Internet-based oncologic teaching for medical students. J Cancer Educ 1998;13:197–202. Issenberg SB, Pringle S, Harden RM, Khogali S, Gordon MS. Adoption and integration of simulation-based learning technologies into the curriculum of a UK undergraduate education programme. Med Educ 2003;37 Suppl 1:42–9.
APPENDIX A: MEMBERS OF THE ARC VIRTUAL RHEUMATOLOGY COMPUTERASSISTED LEARNING (CAL) RESEARCH GROUP Pirashanthie Vivekananda-Schmidt, D.Phil, Martyn Lewis, PhD, Andrew B. Hassell, MD: Keele University; David Coady, MRCP, David Walker, MD, Lesley Kay, FRCP: University of Newcastle upon Tyne; Monica J. McLean, PhD: University of Oxford; Inam Haq, MRCP, Anisur Rahman, MRCP, Jane Dacre, MD: Royal Free and University College Medical School.
