Indicators Impacting Student Success in Distance Education Pamela K. Peak University of North Texas
[email protected] Kevin E. Kalinowski University of North Texas
[email protected] Tandra Tyler-Wood University of North Texas
[email protected] Jason B. Huett University of West Georgia
[email protected] Abstract: This paper suggests that simple motivational email messages address some of the needs of online students by increasing motivation and academic performance, while decreasing dropout rates. Participants are students enrolled in six online courses at a public university who were randomly divided into control and experimental groups. Interventions were developed using Keller’s ARCS model of motivation and delivered via email. Participants completed a pretest and posttest version of the Course Interest Survey (CIS), which measures a person’s level of attention, relevance, confidence, and satisfaction in a given situation, such as a student attending a course. Course cumulative points were used to measure academic performance. Reliability estimates for our data were slightly lower than those provided by Keller (1993). Although the subject-to-factor ratio was nearly 24:1, it is possible that low sample sizes enhanced the effects of sampling error, and produced estimates lower than previous studies.
Distance education is a typical educational delivery method and is predicted more than a 300% increase in terms of students served in the next five years (DETC, 2004) forcing researchers to examine all aspects of the distance learning environment. In order to determine the most appropriate and effective means for conducting distance education learning, more instructional studies are needed. According to Song (2000), determining what motivates and inspires distance learners is one particular area needing to be explored. Motivation is often perceived as one of the most important criteria of learning (Keller, 1987; Song & Keller, 2001). According to Meece, Anderman & Anderman (2006), current educational problems go beyond declining achievement scores to a more detrimental crisis in student motivation. Studies showing motivation accounted for 16% to 38% of the variations in overall student achievement (Means, Jonassen, and Dwyer, 1997). Though selfdirected learning environments, like distance education classes, demonstrated increased flexibility to meet specific learner needs and equity of educational opportunities no matter physical locale of learner (Kerka, 1996), they posed greater challenges to learner motivation than their face-to-face counterparts (Keller, 1987). Problems of web-based distance learning courses include loss of the sense of community and social isolation (Hiltz, 1998) as well as lack of supervision and reliance on learner initiative (Tuckman, 2003). Keller (1999) recognized that successful distance learning must incorporate student motivational tactics into the course design and therefore, created the ARCS model of motivation (Keller, 1987). “ARCS” stands for attention, relevance, confidence, and satisfaction and is a model designed to reveal that learner motivation can be influenced through external conditions. To gauge student motivation in relation to situational learning (i.e., a specific course being taught rather than a holistic view of motivation), Keller (1993) created the Course Interest Survey (CIS). The CIS was specifically designed to assess the four components of the ARCS model (attention, relevance, confidence, and satisfaction), as well as an overall (composite summative) motivation score.
Current Study Given that the literature already supports the contention that distance education initiatives require more work for faculty than traditional face-to-face classes (Foshay, Moller, & Huett, in press), there is a need for simpler approaches to motivating learners that are appropriate for the audience, the delivery system, and the course. Such methods should also be cost-effective, fit within the time restraints of the class and be easily integrated into the instruction. One such approach that shows promise is the creation of simple systematically designed mass-email messages based on established ARCS model principles (Huett, Kalinowski, Moller, & Huett, 2007). The purpose of the current study is to determine what effect motivational emails have on student motivation in distance education. Specifically, we inquired if motivation, as measured by Keller’s ARCS Model via the CIS, statistically and practically was higher for students given motivational emails versus those who were not.
Method Participants In order to answer the research question, we examined 96 students in six sections of an undergraduate special education survey course taught online at a north Texas university by two instructors. For each instructor, half of their sections were used as a control, and the remaining sections were treated with several simple, mass-mailed motivational emails throughout the semester. In total, there were 46 students in the control group and 50 students in the treatment group. Students were automatically assigned to the sections by the university registrar. The course content consisted of a series of self-paced modules taken in WebCT, and there was limited contact between students during the semester, so for this study it will be presumed that participants worked independently. Other than the motivational emails given to the treatment groups, there were no other perceivable differences between the groups. Procedures At the end of the course, all students were given the CIS through a web-based survey tool to measure the four factors of the ARCS Model (attention, relevance, confidence, and satisfaction), as well as an overall motivation score. The survey was voluntary and had no impact on their grade in the course. Measures were taken to ensure only students enrolled in the four sections took the survey, and no student took the survey more than once. Analyses For each student, the four latent factors of the ARCS model were computed from the 34 CIS questions as instructed by Keller (1993). Nine questions were reverse-worded by design and had to be recoded prior to the construction of the factors. The four factor scores were averaged to construct an overall motivation score. Cronbach' s coefficient alpha was computed for each of the four factors and the overall score. These results were compared to those published by Keller (1993). To answer the research question, independent samples t-tests were performed for each of the four ARCS factors as well as the overall motivation score. Finally, 95-percent confidence intervals were computed, as well as Cohen’s d effect sizes for statistically significant differences, to help assess the practical significance of the findings.
Results After computing the four measures of motivation and the overall score for each student, Cronbach' s (1951) coefficient alpha was computed for each factor as an estimate of reliability. The results, as well as a comparison to Keller' s (1993) initial reliability results, are found in Table 1.
Table 1: Cronbach' s alpha for ARCS-based Factors as Measured by the CIS Factor
Keller (1993)
Current Study
Attention (A)
.84
.67
Relevance (R)
.84
.77
Confidence (C)
.81
.81
Satisfaction (S)
.88
.76
Total Scale (ARCS)
.95
.91
Per the research design, independent samples t-tests were computed for the four ARCS factors as well as the overall motivation score. In all cases, non-significant values for Levene’s tests allowed us to assume equal variances for the two groups. The results are found in Table 2. Table 2: Group Differences for ARCS-based Factors Between Online Students Receiving Motivational Emails and Those Who Did Not No Emails Factor
M
SD
Attention (A)
3.73
0.626
Relevance (R)
4.49
Confidence (C) Satisfaction (S) Total Scale (ARCS)
Motivational Emails M
SD
t(94)
p
CI95
3.74
0.532
0.055
.956
(-0.24, 0.23)
0.557
4.64
0.380
1.533
.129
(-0.34, 0.04)
4.48
0.516
4.53
0.405
0.539
.591
(-0.24, 0.14)
4.29
0.573
4.34
0.523
0.407
.685
(-0.27, 0.18)
4.25
0.462
4.31
0.410
0.705
.482
(-0.24, 0.11)
Finally, 95-percent confidence intervals were computed for the measures. The results are found in Table 2. Cohen’s (1988) d effect size estimates were not calculated because the mean differences were not statistically significant.
Discussion Reliability estimates for our data were slightly lower than those provided by Keller (1993). Although the subject-to-factor ratio was nearly 24:1, it is possible that low sample sizes enhanced the effects of sampling error, and produced estimates lower than previous studies. The independent samples t-tests show that although the mean scores for the students receiving the treatment were consistently higher than those who did not receive the emails, there is not a statistically significant difference between the two groups. These results contrast a similar study by Huett, Kalinowski, Moller, & Huett (2007), which showed statistically significant increases and moderately high effect sizes when similar motivational emails were sent to undergraduate students taking a computer applications survey course. Furthermore, the previous study showed that the results from the treatment group did not differ from a face-to-face classroom section, suggesting that the emails had a possibility for addressing some of the motivational needs and retention concerns of online students. There are several explanations as to why the results from the current study differ from the study from which it was based. First, the two studies used students from different populations. Perhaps the positive results from the computer applications survey course do not generalize to other populations, such as students taking a special education survey course. Second, the initial motivational levels of students were not taken into account during analysis. Further analysis will use CIS pretest scores as a covariate to adjust for initial differences among groups, thereby enabling us to more precisely determine whether outcomes are due to the treatment effect or due to initial differences. Finally, the sample used may not have met the assumption of independence. Perhaps, students in the study had other classes together and spent time comparing experiences with this course.
Additional replication studies are under development to determine if simple, cost-effective, and easy-todesign email messages show a promise for addressing some of the motivational needs and retention concerns of online students. In addition to controlling for the initial motivational levels of students by way of a pretest, we are including additional instruments to determine if learning preferences, especially those impacting online learners, might influence student motivation.
References Cohen, J. (1988). Statistical power analysis for the behavioral sciences. (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates. Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika. 16, 297-334. Distance Education and Training Council (DETC). (2004) Distance education survey [report online]. Washington, DC: Author. Retrieved March 1, 2007, from http://www.detc.org/downloads/2004%20DE%20Survey.pdf Foshay, W.R., Moller, L., & Huett, J. (in press). Can instructional design deliver on the promise of the web? In S. Ferris and R. Zheng (Eds.), Online Instructional Modeling: Theories and Practice. Gabrielle, D. (2003). The effects of technology-mediated instructional strategies on motivation, performance, and self-directed learning. Unpublished doctoral dissertation, Florida State University, Tallahassee. Hiltz, S. R. (1998). Collaborative learning in asynchronous learning networks: building learning communities. WebNet 98 World Conference of the WWW, Internet, and Intranet Proceedings, 3rd, Orlando, FL. Huett, J. B., Kalinowski, K. E., Moller, L., & Huett, K. C. (2007). Improving the motivation and retention of distant students through the use of ARCS-based emails. Manuscript submitted for publication. Keller, J.M. (1987a). Strategies for stimulating the motivation to learn. Performance & Instruction, 26(8), 1-7. Keller, J.M. (1987b). Development and use of the ARCS model of motivational design. Journal of Instructional Development, 10(3), 2-10. Keller, J. M. (1993). Manual for instructional materials motivational survey (IMMS). Tallahassee, FL. Keller, J. M. (1999). Using the ARCS motivational process in computer-based instruction and distance education. New Directions For Teaching and Learning Vol. 78). San Francisco: Jossey-Bass Publishers ( pp. 39-47). Kerka, S. (1996). Distance learning, the Internet, and the World Wide Web. ERIC Clearinghouse on Adult Career and Vocational Education, Columbus, OH. (ERIC Document Reproduction Service No. ED 395 214) Means, T., Jonassen, D., & Dwyer, F. (1997). Enhancing relevance: Embedded ARCS strategies vs. purpose. Educational Technology Research and Development, 45, 5-17. Meece, J. L., Anderman, E.,M. & Anderman, L. H. (2006). Classroom goal structures, student motivation, and academic achievement. Annual Review of Psychology, 57, 487-503 Moller, L. (1993). The effects of confidence building strategies on learner motivation and achievement. Unpublished doctoral dissertation, Purdue University, West Lafayette. Moller, L., & Russell, J. (1994). An application the ARCS model confidence building strategies. Performance Improvement Quarterly, 7(4), 54-69. Song, S. H. (2000). Research issues of motivation in web-based instruction. Quarterly Review of Distance Education. 1(3), 225-229.
Song, S. H., & Keller, J. M. (2001). Effectiveness of motivationally adaptive computer-assisted instruction on the dynamic aspects of motivation. Educational Technology, Research & Development, 49, 5-22. Tuckman, B.W. (2003). Academic procrastinators: Their rationalizations and web-course performance. Journal of Personality and Social Behavior (in press). Visser, L. (1998). The development of motivational communication in distance education support. Den Haag: CIPGegevens Koninklijke Bibliotheek. Visser, L., Plomp, T., Arimault, R., & Kuiper, W. (2002). Motivating students at a distance: The case of an international audience. Educational Technology Research & Development, 50(2), 94-110.
