RESEARCH NOTES
▲ Online vs. Classroom Instruction: Student Satisfaction and Learning Outcomes in an Undergraduate Allied Health Pharmacology Course LaDonna S. Hale, PharmD Emily A Mirakian, MPA, PA-C David B. Day, EdS, MPAS, PA-C Online instruction is frequently utilized in allied health education yet only a small number of controlled comparative studies specifically in healthcare education have been published. Purpose: The purpose of this study was to compare student satisfaction and objective learning outcomes of an undergraduate allied health online pharmacology course using streaming media lectures to traditional classroom instruction. Methods: The online (n=47) and classroom (n=177) courses for fall 2005 and spring 2006 used the same instructor, notes, text, learning objectives, and exams. Student characteristics, learning style preferences, and aptitude for distance learning were measured. A statistically reliable, valid survey measured student satisfaction with elements of the course, instructor, and self-perceived knowledge gains. Learning outcomes were evaluated using withdrawal rates and exam scores. Results: Mean satisfaction scores for both courses were high, generally > 4.0/5.0. Mean scores from the classroom students were significantly higher than online students regarding students’ ability to share ideas, instructor’s ability to establish rapport, self-perceived knowledge gains, fundamental principles and application of material, and overall excellence of course. There were no significant differences in objective exam scores or withdrawal rates. Conclusion: The online and classroom pharmacology courses had similar withdrawal rates and exam scores, indicating similar learning gains. Overall, both courses had high student satisfaction ratings in all 43 criteria measured. However, students in the online course were less satisfied with 8 criteria related to student satisfaction with instructor rapport, course excellence, peer interaction, and self-perceived knowledge gains. J Allied Health. 2009; 38:e36–e42.
ONLINE EDUCATION has become increasingly utilized as a means of instruction. In 2002, 56% of two-year and fouryear institutions offered online courses and the number offer-
ing degree or certificate programs, which can be completed entirely through distance instruction, has increased from 8% in 2000 to 19% in 2001.1, 2 Likewise, an increasing number of allied health programs augment classroom instruction with online modalities and/or offer mostly or completely online degree completion programs. In the allied health professions, students must not only meet certain knowledge competencies, they must effectively integrate and apply that knowledge.3 Experts have presented arguments both for and against certain aspects of online education. An evidence-based literature review published in 2002 evaluated 35 research articles measuring various aspects of online instruction specifically related to healthcare education. 4 No controlled studies evaluating student satisfaction with online instruction were identified. One controlled study evaluated self-perceived knowledge gains and did show higher ratings with online instruction. Four other controlled studies evaluating objective knowledge gains found no significant difference. However, the multitude of variables which influence learning and cognition may complicate comparative research, making definitive conclusions difficult.5 Research into online education is still in its infancy and many questions remain unanswered.4 This study is unique in that it compares two semesters of allied health pharmacology courses taught by the same instructor, using the same text, notes, learning objectives, and exams and measures both actual and self-perceived knowledge gains as well as satisfaction with the instructor and course. Thus, the similarity of the pharmacology courses studied here may help limit the large number of variables usually encountered in comparative research.
Study Purpose LaDonna S. Hale is Associate Professor and David B. Day is Assistant Professor, Department of Physician Assistant, Wichita State University, Wichita, KS 67260; Emily A. Mirakian, formerly a student at Wichita State University, is currently a Physician Assistant, Wesley Medical Center, Emergency Department, Wichita, KS 67214. RN779: Received December 4, 2007; accepted February 28, 2008. Address correspondence and reprint requests to: LaDonna Hale, PharmD, Wichita State University, Department of Physician Assistant, 1845 Fairmount, Box 43, Wichita, KS 67260-0043. Phone (316) 978-5713; fax (316) 978-3669; e-mail
[email protected].
e-36
The purpose of this study is to compare online instruction using streaming media lectures to traditional classroom instruction in an undergraduate allied health pharmacology course taught by the same instructor as measured by: 1) percentage of student withdrawals, 2) mean exam scores, 3) letter grade distribution, and 4) student satisfaction with the instructor, course, and selfperceived knowledge gains. HALE, MIRAKIAN, AND DAY, Online v Classroom Pharmacology Course
Methods
SIMILARITIES AND DIFFERENCES BETWEEN COURSES
SETTING AND STUDY POPULATION This study was conducted in a three-credit hour 16-week undergraduate allied health pharmacology course at _____________ University. The University’s 15,000-student population is varied with a large number of part-time, nontraditional students. The average age of an undergraduate student is 24 years with over 85% being from the state of ________.6 The potential study population included all students enrolled in the online courses, 22 in fall 2005 and 25 in spring 2006 (total n = 47) and all students enrolled in the traditional classroom courses, 95 in fall 2005 and 82 in spring 2006 (total n = 177).
This pharmacology course is required for entry into the dental hygiene and nursing programs and serves as an elective for other allied health students. The course discusses most major classes of drugs regarding mechanism of action, pharmacokinetics, side effects, drug interactions, contraindications, therapeutic use, appropriate drug monitoring, and clinical application. Both courses used the same instructor, text, course notes, and learning objectives, similar grading criteria, and the same or similar paper-pencil exams. Each course had three exams, the last being comprehensive. Exams were 95% multiple choice and 5% short answer. Generally 20% of questions required a higher level of synthesis and application of knowledge and 10% of questions came from text readings. The classroom students took the exam in the classroom with the
TABLE 1. Instructional Elements of the Online and Classroom Pharmacology Courses. Online
Classroom
Similarities/Differences
Exams
3 Exams, 100 pts each (last exam being comprehensive)
Same
98% of exam questions are identical between the courses.
Quizzes
None
Weekly quizzes, 10 pts each
Weekly quizzes are used in the classroom course to reinforce weekly content.
Homework
Weekly homework assigned, including 3 asynchronous discussion board activities. 2 – 3 pts each
None
Due to online testing security concerns, weekly homework were used to reinforce course content. Since these are open-book, the point values are minimal.
Learning Objectives
Same
Same
All exam questions are derived from the learning objectives.
Online Practice Questions
Available 2 weeks before each exam.
Same
Online practice questions were available to both courses.
Course Text
Same text used for both courses.
Same
10% of exam questions come from the text rather than the notes.
Course Notes
110 page note packet required for the course.
Same
Same notes packet used for both courses.
Lecture Material
Approximately 2.5 hrs of video lecture viewing required each week.
Students in the TD course also had Approximately 2.7 hrs spent each access to the same video lectures week in the class room. used in the online class.
Enrollment Cap
25 student maximum
No maximum
Clinical Group Patient Case Assignments
Students complete 2 of 4 cases individually. Groups of 4 – 6 students meet online for a synchronous discussion of answers. Each group posts one set of answers. Students review and comment on each other’s work.
Students complete 4 cases either individually or in groups as homeCorrectness of answers is not work. Students form teams of 5 – directly graded. Full participation 7. Teams use an audience response is worth 5 pts. system to answer questions as each case is discussed.
e-37
HALE, MIRAKIAN, AND DAY, Online v Classroom Pharmacology Course
instructor present. The online students were required to use a proctored testing facility for exams. Most local students used the testing facilities on campus. Students living farther away had the option of using a proctored testing facility near them. The classroom students received lectures with the instructor present using PowerPoint while the online students viewed previously videotaped lectures through streamed media (either TegrityTM or MS ProducerTM) accessed either online or TABLE 2. Student Characteristics Characteristic Percent female Age Categories 18 – 25 26 – 30 31 – 40 ≥ 40 Student Rank Freshman Sophomore Junior Senior Graduate Student Other Self-Reported Grade Point Average £ 2.0 2.1 – 2.5 2.6 – 3.0 3.1 – 3.5 3.6 – 4.0 I do not wish to answer Computer Experience £ 1 year 2 – 4 years 5 – 10 years ≥ 11 years Previous Online Courses Yes No Distance courses, but not online No. of Previous Online Courses 0 courses 1 course 2 – 3 courses 4 – 5 courses ≥ 6 courses Previous Pharmacology Courses Yes No No, but have clinical pharm exp
Online n=26 84.6%
Classroom n=107 86.1%
34.6% 26.9% 26.9% 11.5%
75.0% 12.0% 8.3% 4.6%
0.001
3.8% 15.4% 38.5% 30.8% 2.8% 7.7%
0.9% 27.8% 45.4% 13.0% 4.6% 8.3%
0.238
0.0% 3.8% 3.8% 46.2% 46.2% 0.0%
0.0% 1.9% 4.6% 43.5% 24.1% 25.9%
0.592
3.8% 11.5% 42.3% 42.3%
0.9% 20.4% 48.1% 30.6%
0.375
80.8% 19.2%
79.6% 19.4%
0.885
0.0%
0.9%
19.2% 34.6% 15.4% 23.1% 7.7%
18.5% 41.7% 32.4% 3.7% 3.7%
0.009
7.7% 80.8%
14.8% 82.4%
0.111
11.5%
2.8%
p-Value 0.764
through CDs. This streaming video consists of audio and video of the instructor and the rolling PowerPoint slides. The course is offered through the online course management system, BlackboardTM. The streaming video lectures were also accessible by students in the classroom course as a means of repetition or to make up missed class lectures. Some group work was required in each course. In the online course, the only required face-to-face time with the instructor was a two hour orientation session at the beginning of the course. An estimated 95% of the instructor/student interaction in the online course was asynchronous as compared to less than 20% for the classroom course. There was an enrollment cap of 25 students in each online course; there was no cap in the classroom course. Table 1 shows other similarities and differences between instructional elements of the online and classroom courses.
COURSE INSTRUCTOR’S BACKGROUND AND TRAINING The course instructor is a registered pharmacist with a Doctor of Pharmacy and 10 years clinical experience and is a tenured Associate Professor with six years teaching experience. The fall 2005 semester was her twenty-first semester teaching the classroom course and fifth semester teaching the online course. Prior to teaching the online course she received specialized training in BlackboardTM management and online teaching methodologies.
MEASUREMENTS Because students were not randomly selected for the online course, but rather self-selected it, a distance learning aptitude survey was used to measure aptitude for online instruction and the Canfield’s Learning Styles Inventory (CLSI) was used to measure learning style preferences. These tests were used to detect potential differences between the two groups of students. Other measured student characteristics included: age, gender, years in school, prior online courses, and prior pharmacology experience. Student satisfaction with elements of the course, instructor, and self-perceived gains in learning was measured using the Individual Development and Educational Assessment (IDEA) Diagnostic Form Report Survey. Learning outcomes were objectively evaluated using withdrawal rates, mean exam scores and letter grade distribution. All students were included in the analysis of course withdrawal rates. Exam scores and grade distribution were analyzed only for students who completed all three exams. All survey data were included, regardless of the student’s course completion status; however, because the IDEA survey was completed following the last exam, it only represents those students who remained enrolled in the course.
DESCRIPTIONS OF SURVEY INSTRUMENTS
Data were compared using the Chi square test or Fisher’s exact test as appropriate.
Journal of Allied Health, Summer 2009, Volume 38, Number 2
The distance learning aptitude survey is a 12-question survey used by various colleges as a student self assessment. It was originally adapted from “Are Telecourses for Me?” from PBS-
e-38
Fostering Student Collaboration
Stimulating Student Interest
TABLE 3. Results for IDEA Student Perceptions and Satisfaction Survey Question Q4: Demonstrated importance and significance of the subject matter Q8: Stimulated students to intellectual effort beyond that required by most courses Q13: Introduced stimulating ideas about the subject Q15: Inspired students to set and achieve goals which really challenged them Q5: Formed “teams” or “discussion groups” to facilitated learning Q16: Asked students to share ideas with others whose backgrounds / viewpoints differed Q18: Asked students to help each other understand ideas or concepts
Establishing Rapport
Q1: Displayed personal interest in students and their knowledge Q2: Found ways to help students answer their own questions Q7: Explained reasons for criticism of academic performance
Q9: Encouraged students to use multiple recourses to improve understanding Q11. Related course material to real life situations Q14: Involved students in “hands on” projects eg research, case studies, or “real life” activities Q19. Gave projects, tests, or assignments that required original or creative thinking Q3: Scheduled course work in ways encouraging students to stay up-to-date with work Q6: Made it clear how each topic fit into the course Q10: Explained course material clearly and concisely Q12: Gave tests, projects, etc. that covered the most important points of the course Q17: Provided timely and frequent feedback on tests, projects, etc to help students improve Q21: Gaining factual knowledge (terminology, classifications, methods, trends) Q22: Learning fundamental principles, generalizations, or theories Q23: Learning to apply course material to improve thinking, problem solving, and decisions Q29: Learning how to find and use resources for answering questions or solving problems Q33: Amount of reading Q34: Amount of work in other (non-reading) assignments Q35: Difficulty of subject matter
Overall Ratings
Q37: I worked harder on this course than most courses I have taken Q39: I really wanted to take this course regardless of who taught it Q43: As a rule, I put forth more effort than other students on academic work Q41: Overall, I rate this instructor an excellent teacher
Misc Questions
Student Description
Course Difficulty
Structuring Learning on Relevant Objectives Classroom Experiences
Encouraging Student Involvement
Q20. Encouraged student-faculty interaction outside of class
Q36: I had a strong desire to take this course
Q42: Overall, I rate this course as excellent
Q38: I really wanted to take this course from this instructor Q40: As a result of taking this course, I have more positive feelings toward this field of study
Course Online Classroom Online Classroom Online Classroom Online Classroom Online Classroom Online Classroom Online Classroom Online Classroom Online Classroom Online Classroom Online Classroom Online Classroom Online Classroom Online Classroom Online Classroom Online Classroom Online Classroom Online Classroom Online Classroom Online Classroom Online Classroom Online Classroom Online Classroom Online Classroom Online Classroom Online Classroom Online Classroom Online Classroom Online Classroom Online Classroom Online Classroom Online Classroom Online Classroom Online Classroom Online Classroom
N 43 143 43 138 43 142 43 140 43 143 42 140 42 140 43 146 43 145 42 139 42 139 42 142 43 142 43 141 42 140 43 144 43 143 43 142 43 142 42 140 42 137 42 136 42 136 42 134 43 135 43 135 43 135 43 134 42 134 43 133 43 134 43 133 43 135 42 133 43 134
Mean 4.74 4.78 4.44 4.59 4.51 4.67 4.21 4.35 4.51 4.41 3.31 4.03 3.95 4.29 4.14 4.50 4.12 4.48 3.90 4.01 4.02 4.33 4.57 4.37 4.84 4.83 4.37 4.39 3.93 4.19 4.63 4.63 4.72 4.69 4.70 4.72 4.63 4.66 4.36 4.49 4.21 4.57 4.10 4.40 4.05 4.51 3.90 4.19 3.49 3.55 4.09 3.43 4.05 4.19 3.79 4.25 3.31 3.68 3.91 3.75 4.42 4.74 4.33 4.65 3.77 4.16 3.29 3.80 4.07 4.50
SD 0.581 0.574 0.854 0.761 0.768 0.580 1.036 0.913 0.768 0.914 1.473 1.156 1.081 0.971 1.014 0.824 1.028 0.809 1.165 1.148 1.179 0.981 0.831 0.956 0.531 0.519 0.926 0.860 1.237 1.036 0.691 0.677 0.549 0.644 0.741 0.718 0.874 0.713 0.906 0.791 0.871 0.755 1.031 0.772 1.125 0.740 1.246 0.997 0.768 0.817 0.868 0.989 0.754 0.725 1.013 0.836 1.405 1.135 0.781 0.933 1.006 0.648 0.993 0.686 1.377 0.908 1.154 0.996 1.142 0.763
p-Value 0.749 0.267 0.219 0.393 0.520 0.005 0.054 0.018 0.017 0.590 0.092 0.210 0.945 0.906 0.180 0.973 0.743 0.870 0.795 0.347 0.011 0.043 0.015 0.135 0.672 4.0/5.0 and found to be above the national and University averages for courses of similar size and type. Statistically significant differences were noted on 11 of the 43 criteria. Eight of these criteria were related to student satisfaction with the course, instructor, and self-perceived knowledge gains, two criteria were related to course difficulty and student self-description, and the relevance of one is unknown. Mean scores from the classroom students were significantly higher than the online students regarding students’ ability to share ideas; helping students answer their own questions; instructor’s level of interest in students; students’ self-perceived knowledge gains, fundamental principles, and application of knowledge; students positive feelings towards this course of study; and the overall excellence of the course. Online students had a higher mean score for the question related to the amount of work in the course, but the classroom students had a higher mean in response to how hard they
e-40
worked in the course. No significant differences were observed in objective measurements of learning outcomes regarding withdraw rates and mean exam scores, but there were differences noted in the letter grade distribution; see Table 4.
DISCUSSION Significant differences were found between the online and classroom courses in 11 of the 43 IDEA survey criteria. Online students were less satisfied with the instructor’s ability to establish rapport as indicated by Q1) displayed personal interest in students and their knowledge and Q2) found ways to help students answer their own questions; and fostering student interaction as indicated by Q16) asked students to share ideas and experiences with others whose backgrounds and viewpoints differed. The classroom had typical impromptu questions and discussion during lecture, while instructor/student interaction in the online class was generally asynchronous via e-mail. Students’ e-mail questions were answered within 24 to 48 hours for both courses; however, the instructor’s e-mail interactions were typically “down to business,” brief, and likely lacked the kind of personality expressed in face-to-face interaction. Due to the structure of this particular online course, opportunities for peer interaction and instructor interaction were limited. Online students were also less satisfied with self-perceived knowledge gains, fundamental principles, and application of course material. Yet there were no significant differences in mean exam scores, which did require application of knowledge and fundamental principles. Both courses had similar structured opportunities to practice knowledge application and participate in peer interaction through three sets of case studies. All instructor feedback related to the case studies was immediate for the classroom students (occurring as class discussion) but asynchronous for the online students. The online students completed two cases, discussed answers in a synchronous online small group chat, posted the group answers online, read the other groups’ answers and commented on them. This required three to seven days for feedback. The online class had only three additional opportunities for peer interaction in the form of asynchronous discussion board activities all occurring at the beginning of the semester. The questions posed were not high level critical thinking questions and therefore students typically posted one answer and did not comment substantially on previous postings. The instructor rarely provided feedback in these asynchronous discussion boards. Thus, the lower satisfaction in the areas of instructor rapport, peer interaction, and knowledge gains may be due, in part, to the limited delayed asynchronous feedback from the instructor in the online course as compared to more frequent immediate synchronous feedback in the classroom and lower quantity and quality of peer interaction. It is unknown how much unstructured student interaction such as study groups or comparing of scores between peers may have occurred. It is reasonable to assume that this occurs less in the online environment, thus online students may have limited ways of measuring themselves against others. Alterations in the structure of this online
e-41
course with close attention to improving communication, faculty and peer interaction and feedback may have improved student satisfaction in these areas. Instructors should recognize that prompt feedback and interaction with peers is often needed to build student confidence in learning outcomes and establish rapport with students. This may not occur as frequently in online classes as it does in traditional classroom settings. Based on written comments from course evaluations, students found the synchronous chats valuable and enjoyable but felt scheduling was sometimes difficult. When synchronous chats are not feasible, increasing the frequency of asynchronous chats and weekly assignments that require critical thinking and application of course material may be an effective way to improve student perceptions. High quality and quantity, synchronous and asynchronous, instructor and peer interaction can occur in online courses, and asynchronous discussion has been shown to actually increase some students’ abilities to interact.11, 12 For example, an asynchronous discussion board can provide shy students with enough time to carefully think out their responses in advance. Providing responses in writing rather than verbally may overcome public speaking barriers. It is important to deliberately focus on providing multiple opportunities for application of knowledge and critical thinking skills with frequent and prompt feedback and opportunities for high quality instructor and peer interaction. Interestingly, online students had a higher mean response to Q34) amount of work in other non-reading assignments; while classroom students had a higher mean response to Q37) I worked harder on this course than most courses I have taken. Indicators of amount of reading, difficulty of subject matter, and amount of effort exerted were similar. Because the classroom students did not have weekly homework assignments but did have weekly quizzes, it is likely that they did not consider study time to be an “assignment.” Because of the lack of weekly quizzes, the online students likely spent more time studying for exam. However, the pressure of weekly quizzes may have been perceived as more rigorous – thus the higher ratings of “worked harder.” The perception of “worked harder” may also reflect differences in letter grade distribution. It is unknown how students’ desire to take the course from this instructor (Q38) is relevant, because only one instructor teaches this course at this university. Although withdrawal rates and mean exam scores did not differ significantly, a difference in the letter grade distribution was noted. Besides the 300 exam points and 15 group case study points in both courses, the online students had 35 points from weekly homework activities; while the classroom students had eight 10-point quizzes (80 points). This indicates that a review of the point assignments for quizzes and activities in these courses may be indicated.
STUDY LIMITATIONS Students self-selected either the online or classroom course. It is possible that students self-selected appropriately based on
HALE, MIRAKIAN, AND DAY, Online v Classroom Pharmacology Course
their own understanding of their learning style preferences. As noted in other studies, the online students tended to be older and had more experience with online courses.13 It is also possible that some students selected the online course due to the common misperception that it would be “easier” or were forced into an online course due to schedule conflicts. A small percentage of students,
