The Journal of Continuing Education in the Health Professions, Volume 26, pp. 137-144. Printed in the U.S.A. Copyright (c) 2006 The Alliance for Continuing Medical Education, the Society for Medical Education, the Society for Academic Continuing Medical Education, and the Council on CME, Association for Hospital Medical Education. All rights reserved.
Original Article
Evaluation of an Online Bioterrorism Continuing Medical Education Course Linda Casebeer, PhD; Kathryn Andolsek, MD, MPH; Maziar Abdolrasulnia, MBA, MPH; Joseph Green, PhD; Norman Weissman, PhD; Erica Pryor, RN, PhD; Shimin Zheng, PhD; and Thomas Terndrup, MD Abstract Introduction: Much of the international community has an increased awareness of potential biologic, chemical, and nuclear threats and the need for physicians to rapidly acquire new knowledge and skills in order to protect the public’s health. The present study evaluated the educational effectiveness of an online bioterrorism continuing medical education (CME) activity designed to address clinical issues involving suspected bioterrorism and reporting procedures in the United States. Methods: This was a retrospective survey of physicians who had completed an online CME activity on bioterrorism compared with a nonparticipant group who had completed at least 1 unrelated online CME course from the same medical school Web site and were matched on similar characteristics. An online survey instrument was developed to assess clinical and systems knowledge and confidence in recognition of illnesses associated with a potential bioterrorism attack. A power calculation indicated that a sample size of 100 (50 in each group) would achieve 90% power to detect a 10% to 15% difference in test scores between the two groups. Results: Compared with nonparticipant physicians, participants correctly diagnosed anthrax (p = .01) and viral exanthem (p = .01), but not smallpox, more frequently than nonparticipants. Participants knew more frequently than nonparticipants who to contact regarding a potential bioterrorism event (p = .03) Participants were more confident than nonparticipants about finding information to guide diagnoses of patients with biologic exposure (p = .01), chemical exposure (p = .02), and radiation exposure (p = .04). Discussion: An online bioterrorism course shows promise as an educational intervention in preparing physicians to better diagnose emerging rare infections, including those that may be associated with a bioterrorist event, in increasing confidence in diagnosing these infections, and in reporting of such infections for practicing physicians. Key Words: Bioterrorism, continuing medical education, online, evaluation
Dr. Casebeer: Outcomes, Inc., Birmingham, Alabama; Dr. Andolsek: Duke University Medical Center, Durham, North Carolina; Mr. Abdolrasulnia: Outcomes, Inc., Birmingham, Alabama; Dr. Green: Duke University Medical Center, Durham, North Carolina; Dr. Weissman: School of Health Related Professions, University of Alabama at Birmingham, Birmingham; Dr. Pryor: School of Nursing, University of Alabama at Birmingham, Birmingham; Dr. Zheng: Division of Continuing Medical Education, University of Alabama School of Medicine, Birmingham; Dr. Terndrup: Department of Emergency Medicine, University of Alabama School of Medicine, Birmingham. This study was funded in part by contract No. 290–00–0022 with the Agency for Healthcare Research and Quality. Conflicts of interest: none. Correspondence: Linda Casebeer, PhD, Outcomes, Inc., 1827 1st Avenue North, Suite 304, Birmingham, AL 35203; e-mail:
[email protected]. The Journal of Continuing Education in the Health Professions, Vol. 26 No. 2, Spring 2006 Published online in Wiley InterScience (www.interscience.wiley.com) • DOI: 10.1002/chp.62
137
Evaluation of an Online Bioterrorism Continuing Medical Education Course
received training, knowledge of how to obtain information in the event of a bioterrorist attack significantly increased confidence in their ability to respond appropriately.3 To address events that rarely occur but may have critical consequences, educational theorists suggest strategies such as frequent training and decision support tools, including online information systems and screen savers.4 Multiple curricular initiatives and continuing education activities have been developed to enhance health care provider preparedness for potential bioterrorism, although their design and development have not been well documented. An evidence report released by the Agency for Healthcare Research and Quality reviewed 60 programs and reported that few bioterrorism preparedness training programs have been rigorously evaluated.5 A recent randomized educational trial of online bioterrorism education for emergency physicians failed to show evidence of benefit; actual utilization levels in this trial, however, were low.6 In the present study, we evaluated the educational effectiveness of a bioterrorism online continuing medical education (CME) activity. The hypotheses were that physicians who participated in the activity, compared with nonparticipant physicians, would score significantly higher than nonparticipants in their knowledge of bioterrorist threats, their confidence in accessing information, and their awareness of surveillance systems.
Introduction Only a few isolated cases of bioterrorism had occurred in the United States before September 11, 2001, and the anthrax outbreak that followed.1 These events increased awareness of potential threats of biologic, chemical, and nuclear terrorism. They also highlighted the need for physicians to rapidly acquire new knowledge and skills in order to effectively protect the public’s health. Even clinicians who may never face catastrophic events caused by bioterrorism agents could encounter mass casualties associated with natural disasters or emerging infectious agents, such as severe acute respiratory syndrome and West Nile virus. All of these conditions pose similar, special challenges that require prompt knowledge acquisition of emerging rare infections and an expeditious, accurate understanding of the interface between the private and public health infrastructure. These challenges are especially prevalent in emergency medicine and primary care, including family physicians, general internists, and pediatricians, who are most likely to be the first responders. A needs assessment of emergency department physicians identified the development and implementation of comprehensive bioterrorism preparedness plans as critical issues, including physician education and training.2 Greenberg et al. noted that limited knowledge and low confidence among physicians and nurses on bioterrorism topics could be addressed by continuing education and staff development.2 A recent survey by Chen et al.3 reported that 95% of family physicians believed a bioterrorist attack is a real threat, but only 27% felt the U.S. public health system could adequately respond to such an attack, and only 26% percent were confident that they personally knew how to respond. Despite feeling unprepared, only 18% of family physicians surveyed had received bioterrorism preparedness training. Among those who had
Methods Study Design and Population This was a retrospective study of physicians in practice designed to evaluate the effectiveness of an online bioterrorism course, described in detail in a separate publication.7 Two university CME offices—at Duke University Medical Center, Durham, North Carolina, and University of Alabama at Birmingham—collaborated in designing the instrument, identifying outcome
The Journal of Continuing Education in the Health Professions, Vol. 26 No. 2, Spring 2006 • DOI: 10.1002/chp
138
Casebeer et al.
measures, developing the research protocol, and analyzing the data. The protocol and survey instrument were reviewed and approved by the institutional review boards at both universities. The setting and population for this study was online educational participation from U.S. physicians of any specialty. Eligible physicians had completed an online CME activity between October 2001 and June 2003, hosted on a university medical school CME Web site. The participant group included physicians who had completed the bioterrorism CME activity and applied online for a CME certificate for category 1 credit. The nonparticipant group consisted of physicians who had completed one or more of the other online CME courses offered on the university Web site on a range of topics related to chronic disease management in primary care and applied for CME credit but who had not completed the bioterrorism course. The amount of bioterrorism education accessed in a year was considered a key variable in influencing study outcomes between the two groups. Physicians in each group were eligible for inclusion in the study if they self-reported accessing 6 hours or more annually of bioterrorism education.
revised. The first 5 survey items assessed physician knowledge of diagnosis and management strategies for illnesses associated with emerging infections. These items consisted of brief patient vignettes, followed by 5 case management choices in a multiple-choice format. Using a Likert scale, 4 items assessed self-confidence in finding information to guide the management of biologic, chemical, radiation, and mental health related to terrorism. Three items were used to assess whether the physician knew of a preparedness plan in his or her organization and whether the physician knew nationally and locally where to report a potential bioterrorist incident; these items are reported in Table 1. Additional items assessed preferred methods for keeping up-to-date on the recognition and management of potential bioterrorist threats, number of hours of bioterrorism education accessed annually, and general demographic characteristics such as medical specialty, number of years in practice, and geographic location. A power calculation determined that a sample size of 100, with 50 per group, would achieve 90% power to detect a 10% to 15% difference in test scores between the control and intervention groups at the .05 level. In addition, to determine whether the sample was representative of U.S. physicians in practice, the sample was tested for demographic differences between the online participants and overall U.S. physician demographics, as defined by profiles published by the American Medical Association, as well as between the two populations.9 On September 5, 2003, the survey was sent by e-mail broadcast in random blocks of 100 each, drawn from 2 populations (bioterrorism course, U.S. physician participants [N = 505]; and online course participants, except for bioterrorism [N = 1,604]) until 50 usable responses were received from each group. It was estimated that it would take subjects about 15 to 30 minutes to complete the questionnaire; participation in the study was voluntary. All data were
Survey Content and Administration Case vignettes were used to evaluate physician knowledge of biologic and chemical threats. Requiring physicians to react to clinical vignettes has been shown to be a valid measure of clinical quality and competence.8 The study was designed to assess the ability to diagnose and report emerging infections and confidence levels in seeking information to guide diagnosis. The primary outcome measures were performance on case vignettes requiring the diagnosis and management of emerging rare infections, confidence in finding information needed to diagnose and manage emerging rare infections, and recognition of appropriate reporting mechanisms for emerging rare infections. A survey instrument was developed, field tested, and
The Journal of Continuing Education in the Health Professions, Vol. 26 No. 1, Winter 2006 • DOI: 10.1002/chp.
139
Evaluation of an Online Bioterrorism Continuing Medical Education Course
Table 1
System Responses to Bioterrorism Participant (N = 61)
System Responses
Nonparticipant (N = 64)
Frequency
Percentage
Frequency
Percentage
p
Does your organization have an emergency preparedness plan that includes a protocol for bioterrorism events?
46
79.3
38
59.4
.02*
Do you know what department of your local public health agency handles bioterrorism events?
42
67.7
31
48.4
.03*
Do you know what national agency has the charge of overseeing bioterrorism preparedness?
50
80.6
53
81.5
.90
*Significant at p < .05.
reported in aggregates to protect the identity of the subjects.
included in the results, with the total participant group responses of 61 and nonparticipant group of 64. Demographic characteristics of years in practice, gender, and practice location from the samples were compared with those published by the American Medical Association for the overall population of U.S. physicians, and no significant differences were found.9 Responses to clinical vignettes were compared between the 2 groups (Table 2). Physician participants correctly diagnosed anthrax and viral exanthem more frequently than nonparticipants. Twenty-two participants (36%) correctly diagnosed smallpox compared with 19 nonparticipants (30%). However, nonparticipants misdiagnosed smallpox as hantavirus infection more frequently than did course participants (Table 3). There were no significant differences between the 2 groups in the choice of ciprofloxacin hydrochloride (Cipro) to treat anthrax. The responses from the nonparticipant group and the participant group were then sorted by the physicians’ years in practice and self-designated geographic practice settings (urban, suburban, or
Data Analysis The responses from physicians who completed the online bioterrorism course (the participant group) were compared with responses from a sample of physicians who completed one or more other nonbioterrorism CME courses (the nonparticipant group) during the same time frame. Two-sided t tests were used to test the mean differences between groups and χ2 for dichotomous variables. Significance was tested at the .05 level. Data were collected electronically in MS Office Access 2000 (Microsoft, Redmond, Washington) and analyzed using SAS version 8.0. Results Within the 30 days of September 2003, the required sample size of 50 eligible physicians in each group was achieved; additional eligible surveys were returned following the conclusion of the e-mail broadcast solicitation period and were
The Journal of Continuing Education in the Health Professions, Vol. 26 No. 2, Spring 2006 • DOI: 10.1002/chp
140
Casebeer et al.
Table 2
Comparison of Correct Case Responses Participant (N = 61)
Cases
Nonparticipant (N = 64)
Frequency
Percentage
Frequency
Percentage
p
Diagnosis: viral exanthema
47
75.8
35
54.7
.01*
Diagnosis: disseminated varicella
56
90.3
60
92.3
.69
Diagnosis: smallpox
22
36.1
19
29.7
.45
Diagnosis: anthrax
58
93.6
49
77.8
.01*
Treatment: ciprofloxacin (anthrax)
57
91.9
59
90.8
.82
*Significant at p < .05.
rural). There were no significant differences in the percentage of correct responses depending on the location of the practice setting. However, statistically significant differences did exist in responses associated with the number of years physicians indicated they had practiced. There were no differences between participant and nonparticipant groups in the recognition of the patient with anthrax among physicians who had practiced 20 years or less. However, among those in practice more than 20 years, physician participants accurately diagnosed a patient with anthrax more frequently than nonparticipants (57 [94%] participant group versus 50 [78%] non-
Table 3
participant group). The participant group was more confident than the nonparticipant group about finding information to guide the diagnosis of a patient with biologic, chemical, toxic, or radiation exposures but not in addressing mental health issues related to terrorism (Table 4). The participant group was more likely to be aware of health systems approaches to bioterrorism (Table 1). They were more likely to know if their own organization had a protocol for bioterrorism events and to identify which department of their local public health agency would handle bioterrorism events. Both groups identified which national agency had the responsibility for
Patient with Smallpox: Participant and Nonparticipant Responses Participant (N = 61)
Answers
Nonparticipant (N = 64)
Absolute Value
Percentage
Absolute Value
Percentage
p
West Nile virus
9
14.8
6
9.4
.35
Smallpox virus
22
36.1
19
29.7
.45
Hantavirus
7
11.5
17
26.6
.03*
Influenza virus
6
9.8
9
14.1
.47
Varicella virus
17
27.8
16
25.0
.72
*Significant at p ≤ .05. The Journal of Continuing Education in the Health Professions, Vol. 26 No. 2, Spring 2006 • DOI: 10.1002/chp
141
Evaluation of an Online Bioterrorism Continuing Medical Education Course
Table 4
Confidence in Finding Information to Guide Diagnosis Participant
Type of Information
Nonparticipant
Mean
SE
Mean
SE
p-
Patient with biologic exposure
8.21
0.24
7.17
0.26
.01*
Patient with chemical or toxic exposure
8.03
0.29
7.05
0.29
.02*
Patient with radiation exposure
7.64
0.32
6.71
0.31
.04*
6.44
0.35
5.89
0.34
.26
Patient with mental health Issues around terrorism *Significant at p ≤ .05.
overseeing bioterrorism preparedness. There were no significant differences based on practice location, except for the question regarding awareness of their organization’s emergency preparedness plan. Urban physicians more frequently reported that their organization had an emergency preparedness plan. Comparison of preferred learning formats revealed no significant differences between participants and nonparticipants (Table 5). Nearly 80% of participating physicians (n = 48) and 84% of nonparticipants (n = 54) preferred Internet modules as a learning format. CD-ROM
Table 5
and printed literature followed as preferred learning formats. Discussion As a distribution system for education, the Internet offers unique capabilities to teach those with access, anytime, anywhere. In examining practicing physicians’ Internet information-seeking patterns, previous studies have concluded that electronic media are viewed increasingly as credible sources of clinical information.10,11 The Internet is the first
Education Format Preferences
Participant (N = 61) Formats
Nonparticipant (N = 64)
Frequency
Percentage
Frequency
Percentage
p
Internet module
48
80.3
54
84.8
.49
Videotape
16
24.2
18
27.3
.69
CD-ROM
32
48.5
34
51.5
.73
Audiotape
8
12.1
6
9.1
.57
Printed literature
30
45.4
31
47.0
.86
Note: Participants could choose more than one response. *Significant at p ≤ .05. The Journal of Continuing Education in the Health Professions, Vol. 26 No. 2, Spring 2006 • DOI: 10.1002/chp
142
Casebeer et al.
educational distribution system to reach a geographically disparate audience rapidly and simultaneously. In addition, Internet courses can be immediately updated when new information becomes available. These features would be critical in the face of a large-scale bioterrorism incident or an emerging infection such as avian influenza. Although the Internet has been recognized as a source of clinical information for potential bioterrorism events, little emphasis has been placed on evaluation of the effectiveness of such information and education applied to clinical practice. The present study demonstrates differences in diagnosis, confidence in finding information to guide diagnosis of emerging rare infections, and awareness of surveillance systems between a participating group of physicians who had voluntarily completed an online bioterrorism CME activity compared with nonparticipating physicians who had completed online CME courses unrelated to bioterrorism. These findings are consistent with those of Chen et al.,3 with the intervention group reporting greater confidence in their ability to recognize and manage most conditions or illnesses associated with a bioterrorist attack. Compared with a trial conducted by Chung et al.,6 this study demonstrated greater knowledge differences. It was not clear in Chung’s study how many of their participants were emergency medicine residents, who may have been immersed in many other learning opportunities on this topic.6 Nor was it clear whether the content from the initial lecture (which all participants received) contained sufficient key knowledge to have diluted the effect of the subsequent Internet modules in the participant group. A large number of the physicians in the participant group failed to access the online material.6 The current study has several limitations. The study was a retrospective examination of participation in a bioterrorism educational intervention. There is no comparison of baseline diagnosis and management skills of those who participated in
Lessons for Practice • Online bioterrorism courses can enhance physician awareness of potential bioterrorist threats within the differential diagnoses of emerging rare infections. • Online bioterrorism courses can increase physicians’confidence in being able to find information to guide the diagnosis of biological, chemical, or radiation exposure in patients.
the educational intervention. The physicians in the participant group voluntarily chose to complete the course and may have had a greater interest in this topic. The sample represents physicians who already use online CME, which according to other studies may be only about a third of practicing physicians.11 Prospective trials would allow randomization of participants to a control and a treatment arm. Additional questions concerning format and best methods for communicating time-sensitive information could be addressed. As one of the first evaluation studies of online bioterrorism preparedness education for health care professionals, however, this study demonstrates the potential of online courses to prepare health care professionals to recognize potential bioterrorist threats and to diagnose emerging rare infections. Additional prospective studies are needed to confirm these results. Conclusion Bioterrorism has become an increasingly important issue. Bioterrorism preparedness among physicians on the front lines of the health care system is now a necessity. The Internet offers a
The Journal of Continuing Education in the Health Professions, Vol. 26 No. 2, Spring 2006 • DOI: 10.1002/chp
143
Evaluation of an Online Bioterrorism Continuing Medical Education Course
5. Agency for Healthcare Quality and Research. Training of clinicians for public health events relevant to bioterrorism preparedness. Summary, Evidence Report/Technology Assessment: Number 51. AHRQ publication No. 02-E007. Rockville, Md: Agency for Healthcare Quality and Research, 2001.
distribution system with unique capabilities to teach anytime and anywhere the learner is available and to reach a geographically disparate audience rapidly and simultaneously. Providing online bioterrorism updates and CME courses can increase accurate diagnosis, awareness, and information access and may contribute to the health of the public.
6. Chung S, Mandl KD, Shannon M, Fleisher GR. Efficacy of an educational Web site for educating physicians about bioterrorism. Acad Emerg Med 2004; 11(2):143–148.
Acknowledgment
7. Terndrup T, Nafziger S, Weissman N, Casebeer L, Pryor E. Online bioterrorism continuing medical education: development and preliminary testing. Acad Emerg Med 2005; 12(1):45–50.
Lori Andrade, an Alliance for CME fellow, contributed to the development of the evaluation instrument. References
8. Peabody JW, Luck J, Glassman P, Dresselhaus TR, Lee M. Comparison of vignettes, standardized patients, and chart abstraction: a prospective validation study of 3 methods for measuring quality. JAMA 2000; 283(13):1715–1722.
1. Crupi RS, Asnis DS, Lee CC, Santucci T, Manno MJ, Flanz BJ. Meeting the challenge of bioterrorism: lessons learned from West Nile virus and anthrax. Am J Emerg Med 2003; 21(1):77–79.
9. American Medical Association. Physician characteristics and distribution in US. Chicago: American Medical Association Press, 2004.
2. Greenberg MI, Jurgens SM, Gracely EJ. Emergency department preparedness for the evaluation and treatment of victims of biological or chemical terrorist attack. J Emerg Med 2002; 22(3):273–278. 3. Chen FM, Hickner J, Fink KS, Galliher JM, Burstin H. On the front lines: family physicians’ preparedness for bioterrorism. J Fam Pract 2002; 51(9):745–750.
10. Casebeer L, Bennett N, Kristofco R, Carillo A, Centor R. Physician Internet medical information seeking and on-line continuing education use patterns. J Contin Educ Health Prof 2002; 22(1):33–42.
4. Romiszowski AJ. Designing instructional systems: Decision making in course planning and curriculum design. New York: Nichols Publishing, 1999.
11. Bennett NL, Casebeer LL, Kristofco RE, Strasser SM. Physicians’ Internet informationseeking behaviors. J Contin Educ Health Prof 2004; 24(1):31–38.
The Journal of Continuing Education in the Health Professions, Vol. 26 No. 2, Spring 2006 • DOI: 10.1002/chp
144
