Educational Action Research, Volume 12, Number 3, 2004
Improving Web-based Instruction: using action research to enhance distance learning instruction KATHRYN J. LAMASTER San Diego State University, USA NANCY KNOP Ohio Wesleyan University, USA
ABSTRACT This study evolved as an instructor took over a course that recently had been converted to a web-based course. While navigating the web-based course the instructor and students recognised several issues and concerns (instructional alignment, lack of interactivity, poorly designed test questions and confusing instructions for assignments). Study participants were students enrolled in the online course. Data were collected from students, teaching assistants, and the instructor to explore the issues involved with this course, and included student focus group interviews, descriptive student surveys, student feedback, student work and an instructor journal. Qualitative methods were employed in this study and all data sources served as triangulating resources, allowing the researchers to confirm emerging trends. Data were organised chronologically and initial trends were sought. Within the chronological patterns that emerged, data were further organised into similar themes and categories. This process allowed analytical comparison of the different data forms (i.e. student interviews, instructor journals, etc.) Six central themes emerged: impersonality, interactivity, taking responsibility for independent work, course workload and accountability, technological skill and course alignment. In response to the data, several pedagogical changes were made to the course. Additionally, suggestions are made concerning future webbased course design.
Introduction Amidst recent technology advances, methods used to enhance and enable learning in higher education have drastically changed. Gone are the days when lecture and lab teaching and library research was the sole method driving student learning. Instead, today’s faculty find themselves and their
387
Kathryn J. LaMaster & Nancy Knop students propelled into a rapidly flowing river of available digital media. Due to the availability of digital media and ease of using it in instruction, many institutions have added web-based distance courses at an alarming rate and these rates are predicted to continue increasing (Institute for Higher Learning Policy, 2000). Distance education, historically called correspondence study, allows students to engage in course content without physically being at the instructional site (Steiner, 1995). As a result, distance education can open the boundaries of learning, and provide a flexible and engaging learning opportunity for students (Land, 2002). Today many academic institutions, perhaps feeling increasing financial or public demand pressure, are using available audio, video and computer technologies to create web-based distance courses to increase course availability and expand course offerings. The implementation of web-based instruction offers both boon and burden to higher education instruction and learning. On the one hand, ‘the flexibility and responsiveness of distance learning offers solutions to many problems, such as geographical isolation, access, equity, excellence, and funding’ (Ball & Crook, 1997, p. 13). Alternatively, while web-based instruction increases opportunities for students to attempt collegiate work and can enhance the learning that is already occurring, it also has the potential to allow learning to occur in increasingly non-collegial atmospheres, where students toil alone or with little help or supervision from the faculty member. Furthermore, there is the added burden on faculty members to first learn the technology, then create and manage instruction that best uses it to enhance learning. This is a huge responsibility for faculty and one that must be studied concurrently by the faculty delivering the instruction if they are to create meaningful learning for their students. Web-based distance instruction demands greater knowledge construction by the student than learning in a traditional classroom because there is less direct guidance from an instructor (Johassen, 1999; Mayer, 1999). Subsequently, educators have recognised that creating webbased courses requires different pedagogical practices while planning, creating, and delivering instruction than traditional courses (Johassen, 1999; Mayer, 1999).
Purpose Given the popularity of web-based distance education, demands on higher education to create web-based instructional opportunities and the complexity of creating good learning environments for students, the purpose of this study was to investigate how one instructor of an asynchronous webbased course studied, evaluated and amended the instructional environment she was directing. Her investigation included studying how students coped with an online environment and managed their learning experience, and how the process of studying student response to the course allowed her to amend the course and continue to study student interaction with course changes. Due to the complex nature of the data sought, this
388
IMPROVING WEB-BASED INSTRUCTION
was a qualitative study guided by the theoretical perspectives of action research (Greenwood & Levin, 1998) with the intent of describing, interpreting, and ultimately understanding emerging patterns and processes of how faculty can enhance web-based instructional environments.
Action Research Action research involves the researcher(s) and stakeholders in the research process to improve the environment for the stakeholders (Greenwood & Levin, 1998). In the case of this study, the course instructor and a peer faculty were the researchers. The stakeholders are the students in the course and the goal of the research process is to create a better learning environment to support challenging, interesting and motivating knowledge construction by the students. Several elements are innate to action research. These include the value of the research process for generating and reporting new knowledge, the importance of valued participation by the stakeholders, and the expectation that research should lead to changed practices (Greenwood & Levin, 1998). The action research process is a methodological process involving problem identification, action planning, implementation, evaluation and reflection (Carl & Kemmis, 1986). These processes are usually completed in a cyclic way and are often repeated beyond one cycle. In this study, the course instructor elected to take on new responsibilities after the completion of the first cycle. As a result, this study only involved one complete cycle of action research. The environment needed to support intended learning in higher education is very complex. Although teaching is planned with specific student learning objectives in mind, what students actually learn is nearly impossible to measure. As instructors, we assess student outcomes on projects, tests or tasks required in classes assuming that if students accomplish these successfully that they have learned. In addition, we guide them through the process of accomplishing these tasks in our weekly interactions with students so we hopefully have a good understanding of student actions aimed at satisfying course requirements. What is perhaps missing in online educational experiences is some understanding of how students negotiate the course to construct meaningful knowledge. By using qualitative methodologies guided by the theoretical premises of action research, we believe that instructors can learn more about the learning environment students are experiencing and amend courses to better meet students’ learning needs.
Context This study was conducted at a large university in the southwest corner of the United States and corresponded to a time when the university was experiencing increased student access demands. Supporting technology to create effective web-based distance education opportunities was one technique administratively promoted by the university to respond to the
389
Kathryn J. LaMaster & Nancy Knop increased access demand. During this time, enrollment at the university increased to over 28,000 students and the enrollment in the Kinesiology Department also dramatically increased. ‘Introduction to Kinesiology’, a survey course of the broader discipline, was the focus of this investigation. During this time, the course enrollment had increased by approximately 25%. This survey class was one of the first classes considered for web-based distance learning conversion and was a pioneer course on the campus. Selection of this course was based on content, ability to reach large numbers of students and department desire to be instructionally innovative. As a required course for all majors, ‘Introduction to Kinesiology’ has a yearly enrollment of approximately 375 students. Even though this course is offered exclusively online, there is no computer competency prerequisite. Since the course is designed as a prerequisite to the major, students must achieve a grade of C or better in order to enter the major. The web-based course content incorporated a survey approach of the Kinesiology discipline and was combined with a series of student projects. Course design consisted of a series of linked web pages using Adobe Pagemill (version 2.0) to present the course materials in text format. Delivery of the course was asynchronous (interaction between participants did not occur simultaneously) and since most students accessed the course via modem, there were few images or multimedia components to the course. However, there were a series of links to other websites embedded within the text to provide enrichment content and to contact the instructor (email link). Based on student feedback from the first two semesters minor changes had been implemented prior to this study. It was after these first two semesters that a different instructor was assigned the course. This instructor was relatively new to the department, unfamiliar with the course goals and content, but had extensive computer and pedagogical skills. Data for this study were collected during the first two semesters that this new instructor taught and coordinated the course. A major goal of the course, as established by the faculty, was to provide an overview of the Kinesiology subdisciplines (biomechanics, kinesiology, athletic training, teaching, coaching, etc.). Therefore, this information was represented on a majority of the web pages. Additional goals of developing writing skills and creating independent work were incorporated into the course through projects such as a term paper, library search, faculty interviews, behavior change project and résumé. A final goal, developing technologically competent students, was accomplished through computer tasks and use of skills developed during course navigation (Table I). The instructor was available in the classroom 2 days a week even though students were only required to attend the class the first and last day of the semester. Beyond these two required days, student attendance was optional. Since the instructor was present in the classroom, students could show up and ask questions concerning technology, projects and content. In addition to being available during the class period, the instructor was available to students through office hours, telephone and e-mail. Support
390
IMPROVING WEB-BASED INSTRUCTION
for the learning process was through the use of a textbook, links to other web pages and a supplemental study guide. Purpose Overview of Kinesiology subdiscipline
Instructional methods Readings in text Readings on WWW site Faculty interviews
Learning support WWW pages outlining important content, Multiple enrichment links
Assessment method Forced choice online tests Faculty interview write-up
Develop writing skill
Term paper, behavior change project, resume
Written directions available on-line, email queries, instructor availability hours
Assessment of these products by Graduate Teaching Assistant & Instructor
Teach and support process of creating independent work
Term paper, behavior change, library search
Written directions available on-line, email queries, instructor availability hours
Assessment of these products by Graduate Teaching Assistant & Instructor
Technological skill
Daily interaction with class and class tasks
Written directions available on-line, email queries, instructor availability hours
Students can access course materials. Know deadlines, what to do, how to get help
Table I. Initial purpose, instructional methods, support, and assessments.
Participants This study occurred over a two-semester period and involved one complete cycle of the action research process – problem identification, action planning, implementation, evaluation and reflection (Carl & Kemmis, 1986). The first stage of the study – problem identification and action planning – involved different participants than the last stages – implementation, evaluation and reflection. Study participants who were part of the first stage included 21 volunteers from the 200 students enrolled in the class during the spring semester, the course instructor and her graduate teaching assistant. Of the student participants most (15) students were in their third year, consistent with the course make-up, while three were first-year and two were second-year students. These participants represented academic achievement, as represented by their final grades, of the following grade ranges: six were A level students, seven were B level students, seven were C level students and one was a D level student.
391
Kathryn J. LaMaster & Nancy Knop Study participants who were part of the last stage of the study again included the instructor and her graduate teaching assistant, and approximately 30 students (also of varying academic years). The students were self-selected and were those that chose to attend the last day of the class in preparation for their final. These students participated in anonymous student-led focus group interviews guided by an open-ended survey questionnaire concerning the course. As a result of the anonymous participation, specific names and subsequent grade level breakdown are not available for these students.
Methods Qualitative methods (Lincoln & Guba, 1985) were employed at each stage of the study. During the first stage (semester) of the study data sources included: student semi-structured focus group interviews (Fontana & Frey, 1994), two short descriptive student surveys completed before each focus group, student feedback via email, phone or in person, student-completed course evaluations, student work, interviews of past students and the graduate teaching assistant, course materials (i.e. syllabus, email communications) and an instructor journal. Interviews, documented student feedback and course evaluations were intended to generate data about student perceptions of the class. The short surveys were created to gather descriptive data about students’ course grades, computer proficiency, intended major and career, and why they were studying this particular course. Evaluation of student work allowed inference of student learning throughout the course. The instructor journal and course documents created a chronological timeline of critical incidents and class occurrences that helped create context for the data analysis. All data sources also served as triangulating resources allowing the researchers to confirm emerging trends. A total of five focus groups were formed during stage one and each group was interviewed two times. Each focus group consisted of three to five participants and assignment to a focus group depended solely on when the students were available to meet. The first interview was conducted soon after the first course exam and occurred in the fourth week of class. It was intended as an initial fact-finding interview to elicit student response to the course, their learning strategies and perceptions of the role technology played in their learning. The second interview occurred during the last week of class. It was intended as a follow-up to the initial interview and as a chance to check themes that had emerged over the course of the semester. All interviews were conducted by the co-author of this study. She had no other involvement with the class. Prior to the interview, students were informed that the information they shared that day would have no influence on their grade nor would it be shared with the instructor until the end of the semester. In addition, they were told that their feedback was important and would be used to amend the course and create a better learning environment for future students.
392
IMPROVING WEB-BASED INSTRUCTION
During the second stage (semester) of the study when changes to the course were being implemented and evaluated, data sources included: • • • • • • • •
a short student survey; student feedback via email, phone or in person; student-completed course evaluations; student work; interviews with the graduate teaching assistant; course materials (i.e. syllabus, email communications); an instructor journal; a student-led focus group interview based on questionnaire.
an
open-ended
Documentation of instructor–student communication and instructor reflections provided perceptions as to whether students were better supported in the instructional process after the pedagogical and technological changes. The student-led focus group interviews administered during the final week of the semester focused on many of the same questions asked of students the prior semester as we attempted to identify the strategies, tactics and learning processes students were using to complete the course, and whether these were different from those used in the previous semester (Table II). Time Year 1 – fall
Year 1 – spring
Phase of action research Problem Identification
Research question
Action
How is the course going?
Instructor had just adopted course, realized change was needed
What content is being presented?
Student course feedback Student work Instructor journal Course materials Course evaluation Student questionnaire 2 Focus group interviews of students: st 1 interview th – 4 week of semester – 21 students, 5 focus groups nd 2 interview – Last week of semester Individual interviews Graduate teaching assistant Past successful students
How are students negotiating course demands?
393
Kathryn J. LaMaster & Nancy Knop Year 1 summer
Action Planning
What changes need to be made to create a better learning environment for students?
Consult literature Amend course materials Amend tests Align and reconstruct course tasks Increase student–student and student–teacher interactivity Research web-based course delivery systems Support better student time management Enhance student motivation Better guide student focus toward important concepts Increase student accountability
Year 2 fall
Implementation
How is the course going?
Student course feedback Student work Instructor journal Course materials Course evaluation Student questionnaire Student led focus group interviews (30 students) based on open-ended questionnaire Individual interviews Graduate teaching assistant
Year 2 – spring
Evaluation
What does this data tell us?
Analyze data
Year 2 – summer
Reflection
What is the impact of the changed curriculum?
Reflect on meaning of data and make suggestions for future course changes.
Table II. Research summary and timeline.
Data Analysis and Discussion Problem Identification Initially, a data timeline was used to situate all data within the context of the class before comparing the different data sources. Data were then organised chronologically and initial trends were sought. Within the chronological patterns that emerged, data were further organised into similar themes and categories (Guba & Lincoln, 1989). This process allowed
394
IMPROVING WEB-BASED INSTRUCTION
analytical comparison of the different data forms (i.e. student work and communications, instructor journals and interviews). Once themes had emerged, negative cases or cases that did not fit the emerging trends were sought (Rossman & Rallis, 1998). When found, these data served to challenge or add depth and breadth to the developing interpretation. Upon first analysis, data from the multiple sources accumulated around eight themes. We labeled each theme with a short descriptive heading that we felt summarised the sentiment of the data. These themes included: 1. Impersonality – data that suggested that this web-based course created an environment devoid of ‘human touch’ in a way that was detrimental to the learning process. 2. Interactivity – data that suggested that this web-based course diminished student opportunity to interact with the instructor or other students in a way that students or instructors felt added to the instructional process. 3. Struggles completing independent work – data that suggested students struggled with completing tasks on their own without consistent reminders or feedback from instructors. 4. Course alignment issues – data that suggested that there were perceived problems with the course construction and alignment of tasks. 5. Too much work – data that suggested tasks assigned in the class were perceived as overwhelming or too much for course of this nature. 6. Student accountability issues – data that suggested students were not being held accountable for completing the tasks in a way that consistently supported academic integrity and honesty. 7. Student strategies to meet course demands – data that suggested how students chose to meet the course demands relative to what they were being held accountable for. 8. Technological struggles – data that suggested students were struggling with accessing course information or completing tasks dependent on technological knowledge. Upon further analysis, several of the themes shared common data and were collapsed in three central themes. These are presented with representative data demonstrating development of that theme and discussed in the following paragraphs. Impersonality and interactivity. A common response from all focus groups revealed student discomfort with online courses. These responses varied, but are consistent with this comment by Mary, who said ‘computers should not teach people. That is the scariest thing. It is a valid source for research, but not for teaching me how to interact with other people’. It should be added that this comment, or one like it, was generally the first student response across all focus groups during the first round of interviews. Furthermore, this response was often unsolicited, and came after the interviewer explained the focus group process and initiated the discussion asking for general comments about the course. It is quite probable that
395
Kathryn J. LaMaster & Nancy Knop many students at this point in the semester were frustrated with the class and were blaming their frustration on the technological medium, rather than on the different academic processes needed to negotiate an online course. This theme continued to appear during the second round of interviews, but was not voiced as vehemently. Closely related to the perceived impersonality of a web-based course was the student perception that they were greatly missing opportunities for interacting with peers, instructors and professionals in this course. This was also demonstrated by Mary who voiced her concern with learning introductory knowledge of the kinesiology discipline, which is primarily a service-based discipline, exclusively via a web-based course. She suggested that students ‘need the interaction between people. They need to socialise and stuff. You can’t just go on the web to learn to do brain surgery and then do it the next day. You are not getting the experience of dealing with the person on the web. The hands-on experience.’ This sentiment was corroborated and extended to include what students perceived as a lack of interactivity when they did not have access to the instructor and peers in a classroom setting. One student, Mr Funny, summed up this feeling saying ‘Just being able to talk to other students and to the teacher. Just little questions that are not worth my time emailing the teacher about but if I were in a class setting I could find out the answer really easily. Little things like that bother me.’ While another student, Ben Hur, further highlighted student feelings of missed guidance and mentoring due to lack of ‘in-person’ instruction and management of a course by the instructor. To set the context, he is explaining that having ‘in-person’ instruction provides opportunities for unexpected learning, beyond class content. He related this example of stopping in to the instructor’s office during office hours: I walked in to talk to her [course instructor] ... I didn’t even talk to her about anything that pertained to Kinesiology. Just sat down and talked about my career goals. She is such a cool lady and you don’t ever get to see her or talk to her. She actually gave me some valuable advice. These comments were further supported by data from the written survey administered to students at the end of the first semester. Students were asked to create a list of course pros and cons relative to their course experience. Jennifer typified many students’ responses in her list of cons, which included, ‘not having contact with the teacher, not knowing exactly what is important, not knowing each other, not having peers’. The instructor also voiced concern over her lack of personal interactivity with students. For comparison, she suggested that in her ‘inperson’ courses she knew her students by name, interest and how well they were doing in her class. Before and after each class, many of these students would drop in to her office or meet her in the classroom to seek help about the content, tests, tasks or projects, seek career advice, or just stop to say ‘hi’ and visit. Relative to ‘Introduction to Kinesiology’, most of the
396
IMPROVING WEB-BASED INSTRUCTION
documented teacher–student communications were related to technological issues. Students would seek out the instructor, sometimes face-to-face, but frequently via email or phone, to get help downloading a document or signing up for a test. The most frequent source of regular communication from the instructor to the students was the regular emails she sent to remind students of upcoming assignments and deadlines. The instructor recognised most students’ names, but could seldom link a name to a face. This was a major pedagogical change for the instructor and, as a result, it appears that the web-based course was ‘less fun’ for her to teach as she, too, missed the student interaction. Taking responsibility for independent work. A second emerging theme related to how well students were taking responsibility for their work. Web-based courses, by their nature, allow students great independence in deciding when to do their work. At the same time, there is increasing responsibility on students to plan for and accomplish their work in a timely fashion without the benefit of weekly prompts, or solicited or unsolicited guidance that often occurs in a classroom. Students frequently felt that without the regular interaction of the instructor reminding them of their upcoming work and the weekly regimen of attending class that it was much harder to complete their work in an organised and timely manner. An example of this sentiment was stated by Steven: ‘it is like kind of hard to get into it when the teacher is not there. [You miss having] someone who is pushing you and getting you excited about the information.’ Ben Hur further clarifies this feeling saying, ‘there is too much freedom with web-based [courses]. If you say you will do it and time passes and you have other things to do it becomes low priority.’ At the same time, the independent nature of the course provided opportunity to some students. For example, Sue, who was returning to school after starting a family, could not have fit the course into her schedule had it been a ‘real time’ class. She relished the independent nature of the course because it allowed her to accomplish her academic goals without greatly compromising her daily family responsibilities. This suggests that although many students we interviewed were unable or unwilling to take responsibility for the independent nature of the work in this course, some students were quite prepared and willing. These students, however, tended to be more highly motivated to complete the course than their struggling peers. Motivation is an issue in most learning environments and this finding suggests that due to the increased reliance for students to complete work independently, lack of motivation to work independently of an instructor is a greater issue. This data indicated the need to embed time management and prioritisation strategies into the Internet course. Course alignment, workload and accountability. A final emerging theme was the relationship between the course alignment, workload, the accountability for accomplishing that workload and student strategies for accomplishing the workload within the constraints of the accountability system. Students
397
Kathryn J. LaMaster & Nancy Knop believed, and the instructor agreed, that the projects and tests needed to be more closely aligned with the major content concepts. Students felt that the projects were not related to the readings and tests. They further believed that none of the projects were related to any other project, so the skills or knowledge they learned doing one project were not helpful in completion of another project. This led them to consider several of the projects meaningless, although important to their grade. Interestingly, students could not come to consensus about which projects were most or least valuable. The value of the project to a student seemingly was determined by how hard or easy it was to accomplish or by how professionally interested they were in it. Concerning alignment of tests, this issue appeared to subdivide into two primary issues. First, there were a lot of pages located on the website and little way for the students to prioritise this material relative to its importance in the assessment process. The second issue had to do with the test itself. Many of the questions seemed to be based on memory and few on application. Memory questions often included recalling dates or locations, for example, ‘The landmark meeting to form the Association for the Advancement of PE was held at Harvard in what year?’ While application questions focused more on interacting with the information to produce a response, for example, providing resting heart rate, age and blood pressure in the question, and asking the response to be a calculation of target heart rate. As a result, students felt they were tested over arbitrary and minute details, and that it was nearly impossible to study for these tests because of the volume of material from which the questions might be taken. Based on the written survey completed at the end of the first semester, every student participating in the study felt that the course contained too much work. This perceived high workload consisted of reading web pages, completing several course projects, and studying the text and web pages for exams. They stated that there were too many pages of web-based material and, because of this reading volume, it was hard to know how or for what to study. Responding to the bulk of the reading, students used the following strategies. First, many students printed all of the primary web pages at the beginning of the semester or at the beginning of each new section. This strategy allowed students to study ‘on the beach’ as one student said and not be tied to the computer. After printing these pages, there seemed to be a difference in what strategies students chose. Students who were more successful in the course tended to return to the web home page at least once a week to check for announcements and review the course timeline. Students who weren’t as successful infrequently returned to the web site except to check their grades. Furthermore, more successful students tended to create and post for themselves a course timeline of readings, projects and tests. They also seemed to spend more time each week on course projects and were more likely to contact the course teaching assistant or instructor for help or clarification. Furthermore, students usually crammed before tests. Few kept up with readings on a regular basis. Instead, they tended to organise their
398
IMPROVING WEB-BASED INSTRUCTION
studying around the testing dates. Many students interviewed said they only had time to read the material once before taking a test. So it does not appear that students necessarily studied for these exams, but tried to get through the material and remember as much as possible right before the exam. Furthermore, this suggests that few students took advantage of the related links located on most of the primary content pages. However, a few students did confirm that they used many of the links as resources for completing course projects. Relative to completion of class projects, many students suggested that, because of the volume of work expected of them and the weak accountability system afforded by the independent nature of the work, the following strategies were used: • for the résumé project, students used a résumé template from their word processing software and minimally filled in the categories, rather than creating a beneficial and meaningful résumé they might actually use; • for a faculty interview task, many students avoided the actual interview, but satisfied the course requirements by creating interview write-ups from copied and pasted information from the faculty web sites; • for the behavior change project, students made up the project, data and their results. In general, students suggested that the weak accountability system and large workload created an environment where they took ‘less pride in work’. Most students, after enrolling in the course and experiencing the reading, testing and projects, seemed to have as their goal to do what they had to do to get through the course with a C. As a result, this course was seen by many students as a ‘hoop to jump through’, not as a course that would add to their knowledge or understanding of the discipline.
Action Planning Student feedback from focus groups, communications and surveys was vital to the reorganisation and improvement of this course. In direct response to student issues of impersonality and interactivity, course alignment and workload, and independent work, several changes were implemented. Impersonality and interactivity. First, more effective Internet course management packages were explored. One package, Blackboard Course Info (www.blackboard.com), was selected based upon the numerous internal features that related to each of the previously identified student concerns. Berger (1999) identified the need for interactivity and availability of the instructor to students. Course Info has a staff information section where students can access information about the instructor, such as office hours, phone number, research interests, etc. Additionally, there are student homepages that can be completed and provide information to all members in the course. This supports Berger’s (1999) comments that an instructor should have a web page for information about the course, then another area for students to include pictures and biographical information. Both of these
399
Kathryn J. LaMaster & Nancy Knop features work to reduce the impersonality of the Internet course. Interactivity features, such as threaded discussion board, chat feature and email are also embedded in Course Info. These features allow students to have quicker access to other students in the course. Previously, students had little access to other students in the course. These features made it easier to identify peers, communicate with them, and gain access to peer help or information. This supports Brown & Duguid’s (2002) notion that communities support the process of online learning. In addition, the discussion group and personal homepages allowed the instructor to better know the students, communicate with them and, by monitoring the discussion board, monitor emerging issues and concerns (Figure 1).
Figure 1. Communication and discussion boards.
Course Info opens to an announcement page that can be updated by the instructor as often as necessary (Figure 2). The package is also menudriven, which improves navigation and provides access to Course Documents, Assignments, Student Tools, Communication features and Instructor information. The web pages that were used in the previous course were uploaded into the Course Document section, organised by exam content, and this became the new course website. The Student Tools area provides access to grade checking, ability to update their homepage, a personal calendar and a digital drop box (allows electronic paper submission). These features also enhance student interactivity by making it easier to submit work and determine current grades. The grade access feature provides the additional benefit of encouraging more frequent student access to the web site. Coupled with the announcement page features that allow the instructor to create a beacon of new or important information for students (using color, flashing words or increased font size), these features were expected to increase student interactivity with the web site and enhance student motivation to get work done on time.
400
IMPROVING WEB-BASED INSTRUCTION
Figure 2. Announcement page and menu sidebar.
Taking responsibility for independent work. The instructor felt that the realigned course with increased and fairer accountability, and a streamlined workload would also increase student motivation to complete the course. In turn, if students were more motivated to complete the course, they would be more responsible for independent work. To further support student responsibility, the instructor recreated the course syllabus to include a schedule clearly showing all due dates in red and all assignments dates in blue. The color-coded syllabus created a more detailed schedule for students. In addition, the instructor added a time management piece early in the course that was composed of an assignment and a due date alerting system. For the assignment, students were directed to create a timeline using the personal calendar feature available on the homepage and date due sheet at the beginning of the course, reflecting good study habits and timely assignment creation. The instructor used the email system to alert and prompt students about upcoming due dates or test dates. One to two weeks prior to due work or tests, the instructor would send out mass emails prompting students of the impending task. Furthermore, the instructor planned to support independent student work by responding daily to all student queries. She planned to set aside time during each day and at weekends to check course email and respond. She also tried to make her responses positive and warm. Email, as a written media, can appear sterile and cold to people who barely know each other. The instructor planned to use prompt replies, varied punctuation, humor and personal responses to student queries to create a relationship with students in the more challenging environment of email. Course alignment, workload and accountability. Focus group interviews revealed that students felt that the course required ‘too much work’ and
401
Kathryn J. LaMaster & Nancy Knop that many of the tasks were ‘another hoop to jump through’, rather than a meaningful learning experience. In response to student concerns, course projects and tasks were re-evaluated. A revised summary of the initial course purpose, instructional methods, learning support, and accountability measures is found in Table III. Purpose
Instructional methods Readings in text book WWW pages Faculty search
Learning support
Develop writing skill
Interlinked projects.
Better directions Posted rubrics on-line. Easier email access to peers and instructor.
Assessment rubrics better guided Graduate Teaching Assistant & Instructor Assessment rubrics attached to student work
Teach & support process of creating independent work
Term paper Behavior change project Library search
Better directions Posted rubrics on-line. Easier email access to peers and instructor.
Assessment rubrics better guided Graduate Teaching Assistant Assessment rubrics attached to student work
Technological skill
Daily interaction with class and class tasks
New course software. More built in course accessibility & student communication features. Rewritten & tested directions for technological processes (i.e. downloading)
Students can access course materials. Know deadlines, what to do, how to get help
Overview of Kinesiology subdiscipline
More specific study guide for each content area. Use bolding to highlight important issues. Eliminate many enrichment links.
Assessment method Revised forced choice on-line tests Revised faculty search – supported by hard copy search evidence
Table III. Initial purpose and revised instructional methods, support, and assessments.
402
IMPROVING WEB-BASED INSTRUCTION
One major change that resulted from this re-evaluation was to interlink course projects. Previously, the projects seemed to stand alone, rather than relate to course content or other projects. Johassen (1999) suggests that ‘problem-based tasks that make sense, allow active and authentic involvement of learners, and reflect a cumulative building on past learning’ (p. 216), better guide and motivate students to take responsibility for constructing their own learning. The course projects were then evaluated and changed to become more personal problems that students would be motivated to solve consistent with Brown & Duguid’s (2002) notion of students’ academic need for ‘ever-more-intensive application’ (p. 233). For example, the library project had students electronically accessing the library, and completing random tasks designed to teach them how to digitally find and access information. Revisions to this project linked the library search to the term paper due later in the semester and, therefore, assisted students in acquiring necessary references for their topic. In addition, the term paper research served as the beginning of the behavior change project, which was the final project due in the semester. For example, a student interested in strength fitness might research the process of developing strength fitness specific to a personally determined goal (i.e. increasing sprint speed). Once the research was completed and the student was more knowledgeable, an appropriate behavior change aimed at increasing speed via strength training would be created and attempted. Further changes included evaluating the web page structure and especially the links from lecture pages to external Internet sites. The instructor re-evaluated the course website and decided to decrease the total number of web pages, especially the links. The links appeared to students as more information to know rather than enrichment, as it was intended. As a result some students would spend time studying the enrichment pages as if the material might be on the test. Later, these students would be frustrated, feeling that time had been wasted. Several of the links were eliminated in an effort to streamline the course readings. In addition, the instructor created better study guide outlines for each content subtopic, and highlighted important concepts using bold or underlined print. Other changes to the web pages included putting certain text in bold, spacing, refining the section study guides and reorganising the information. These changes are supported by design suggestions of web pages (Mayer, 1999). Relative to student accountability concerns, the instructor built assessment rubrics that better identified exactly how the students would earn points. These were posted on the website to guide student project development. During grading, the rubric was used and the completed assessment was stapled to the front of the paper. As a result, the student knew before handing in the project exactly what was expected and how the grading process would proceed. Additionally, students received better and more consistent feedback with their graded project. In another attempt to increase student accountability, the instructor created assessments that were more dependent on hard copy or downloaded
403
Kathryn J. LaMaster & Nancy Knop information. For example, with the ‘meet the faculty’ project where students previously had downloaded information from faculty web sites, rather than conducting face-to-face interviews, the project was changed to email queries to the faculty. To hold students accountable for completing this task, part of the assessment was turning in the hard copy email to document the query. Although the faculty had some objections to the impersonal nature of the project, many preferred it as they created a standard reply and sent to all queries. To further increase course alignment and effective student accountability for learning, the instructor re-evaluated the test bank questions. This was done in two ways. First, since testing was completed online the instructor had immediate access to student response data for all questions. Using this data, the instructor either eliminated or rewrote many questions that appeared to be bad or poorly written. Secondly, all questions were re-evaluated and rewritten to allow students to demonstrate several levels of learning – knowing, understanding and applying. It appeared that the knowing type questions were over represented in the test bank. In addition, test bank questions were re-evaluated for relevance to the central goals of the class. From re-evaluation, the instructor eliminated some questions that were not important points highlighted in the study guides, web content pages, text or projects. The questions eliminated tended to be less important points that were less fully covered in the course materials.
Evaluation and Reflection In this phase of the project, the instructor attempted to determine the impact of the changes made prior to the second semester. Since changes were made after reflecting on student response during and after the first semester delivery of the course, the instructor now wanted to see how these changes impacted the process and product of student learning during the second semester. The focus of research conducted in this phase of the study were to examine student response to the same issues raised in the first phase of the study – issues of impersonality and interactivity, course alignment and workload, and independent work. Focus group interviews conducted at the completion of the second semester, instructor reflections, student work, quantitative data reflecting student ‘hits’ on the course web site, and the nature and quantity of student communications with the instructor served as data for this phase of the project. Impersonality and interactivity. Focus group data from the second phase indicated that students valued the interactivity features of Course Info. Most students stated that they enjoyed the grade checking option and felt they checked it regularly. In addition, students stated that they were able to communicate using email and the discussion board and enjoyed that feature of the course software package. These features connected students to one another and reinforced the notion of peers being an important resource (Brown & Duguid, 2002). Instructor reflections suggest that
404
IMPROVING WEB-BASED INSTRUCTION
student Homepage construction ‘was great’. All students created homepages with required information of their major, however many incorporated personal information and pictures into their homepage. This personal web page allowed students to ‘meet’ other students. Students were also able to connect with other students via the homepages, discussion boards and email messages during the second semester of this study. In general, there were fewer complaints about the lack of interactivity with the updated online course. In addition, teaching evaluations improved slightly this semester, and the written comments from students were supportive of instructor interactions, friendliness, knowledge, promptness and availability. At the same time, the majority of student comments suggested that students would prefer the course to not be an Internet course, but would rather attend a class and have an instructor in front of them. While most of the data used to evaluate student perspectives of course changes occurred at the completion of the semester through focus group interviews, some data could be gathered throughout the semester using the new course management package. Tracking, the ability to determine student use of the website, created a history of student use of each of the menu options available. There were approximately 100,000 ‘hits’ to the course website during this first semester that Course Info was used. A majority (83%) of the ‘hits’ were to access the content (class notes, assignments, syllabus and instructor information) of the website, while students used the communication (8%) and student tools (9%) section less frequently. This data suggests that, unlike the previous semester, when students told us they would go on to the website early in the semester to print the pages needed to study for a test or prepare for a project maybe once or twice every 2 weeks, now students were actively using the website throughout the semester. It also suggests that students were using the communications tools (i.e. discussion board, direct email to students or instructor) and the student tools (i.e. student calendar creation, homepage access and student grades). Changes in the course features may have been more adequately meeting student needs and demands (Brown & Duguid, 2002). Taking responsibility for independent work. The students appreciated the email alert system created by the instructor to inform them of upcoming work completion or test dates. The alert system consisted of using the Course Announcement page tool in conjunction with emails distributed to all students. A problem with the alerting system, however, was that students had to access the Course Announcement page in a timely manner to see important updates. From the data gathered describing the number of ‘hits’ on the website, it seems many students were accessing the site, but there is no data to support that all students were accessing the site. Another problem with the alerting system was that many emails would ‘bounce back’ to the instructor if the student did not maintain their email account or if they had changed their account without updating the instructor. So there continued to be an issue with students taking responsibility for their own learning. Student grades also improved slightly
405
Kathryn J. LaMaster & Nancy Knop from a class grade point average of 2.09 to 2.14 on a 4.0 scale. Once again, this could be a result of better course organisation, clearer test questions, better content alignment, other pedagogical improvements or differences in student population from one semester to another. Course alignment, workload, and accountability. From the instructor’s perspective, realigning the tasks in a way that made sense to the students was the biggest accomplishment of the course changes. ‘The students might not enjoy the task’ she stated, ‘but once the project was put in the perspective of how it benefited them for the future projects it made the accountability better’. Accountability was further supported by better project directions and instructor interaction on the website discussion board. For example, when a student asked, ‘why are we doing this?’ the instructor would go into the discussion room and highlight why the task had merit. Ultimately, according to the instructor, it was increasing communication with the students and better task alignment that contributed to greater student accountability. The linking of the projects made the library project, term paper and behavior change more valuable to the students. From the instructor perspective, linking the projects eliminated student questions like ‘what kind of things should I look up for the library project’ because they could focus on the content of their behavior change project. In addition, fewer students indicated that they were going to change their major or viewed the course as a ‘hoop’ to jump through in their quest to be a Kinesiology major. At the same time, more students expected their grade to be representative of the effort and work they put into the course. Rubric creation made projects, in general, of higher quality and easier to assess, according to the instructor and graduate teaching assistant. At the same time, fewer students had questions about each project, evidenced by the decreased number of project-related email queries. Quality of work on the behavior change project seemed to increase somewhat because projects seemed more personal and genuine. However, at the same time there was still evidence of problems with students cheating through the behavior change project. Even with additional accountability students found ways to share work. For example, the instructor stated, ‘this last semester we got a behavior change project with another students’ name in the text, so basically the student had changed the cover page’. Furthermore, it is quite possible that a student who has enough knowledge could create a realistic-looking behavior change project without having actually gone through the process of changing a behavior.
Conclusions Students often find unanticipated ways of accomplishing tasks within the distance-learning environment due to the decreased ability of the instructor to mentor student progress and increased student freedom to interpret how
406
IMPROVING WEB-BASED INSTRUCTION
to accomplish course tasks. The methods they choose can alter the depth and breadth of learning that results from the course. In this study, a faculty instructor who had adopted leadership of a previously constructed webbased distance-learning course realised that the course needed to be altered to better guide and support rigorous and intended course goals. Collaborating with a peer faculty member, action research methods were built into the course structure to allow the course instructor to more clearly understand how students were accomplishing course objectives and, subsequently, alter the readings, tasks and tests to support a better learning environment. We do not intend to claim that the specific trends we found in this study can be generalised to another setting. Nor do we believe we can recommend a specific list of do’s or don’ts relative to online learning site construction, reconstruction or amending. Alternatively, the theory guiding action research has been supported as a research methodology sensitive enough to allow us to discern how students were accomplishing course objectives in a web-based distance-learning course. In addition, we learned a lot about redeveloping and amending an online course from the research process. Although we know learning to be context dependent, we hope that by sharing a summary of how and what we learned, readers contemplating creation, development or amendment of online courses will similarly use action research methods to monitor the impact of course content and pedagogy on student accomplishment of course learning goals.
Summary Using action research methods to identify potential course problems. In this study, during the problem identification phase, we used student focus group interviews and brief surveys, student work samples, logs of student– instructor interaction (email, phone or in-person), and instructor and graduate student reflections. Most of these methods could be built into tasks students complete for the class or standard operating procedures for the instructor. For example, the instructor may require all students to complete a short survey at the beginning of the semester and email the survey as an attached document to the instructor. This task has several purposes. First, it gives the instructor a student email address that apparently works. Secondly, it confirms that the student can attach a document. Thirdly, depending on the nature of the survey, it could give the instructor valuable information about the student’s course expectations, computer technology skills, or perhaps career goals or expectations. Although we used in-person focus group interviews, a similar process using an online chat room or discussion board might prove beneficial to elicit information about how students are constructing knowledge presented in the course. Here, students would be required to respond to several queries presented by the instructor throughout the semester. The queries could be constructed to allow students to share content understandings or interpretations, supported opinions about an issue or methods used to
407
Kathryn J. LaMaster & Nancy Knop accomplish a task. The nature of the queries would be determined by the instructor, and should be focused on determining how students are constructing knowledge and accomplishing tasks. In addition, students might have a place to submit queries to their peers and faculty of their own at several points during the semester relative to content or processes that they don’t completely understand. This area would be closely monitored by the faculty, and both questions and answers posted for all students. The content of the queries and responses would serve as an additional data source for the instructor. Using action research to plan successful changes. As an instructor plans for changes after gathering data during the problem identification phase of action research, he/she should consult the many resources available to guide on-line course construction. We found these sources to be particularly helpful: Berger (1999), Johassen (1999), Mayer (1999), Reigeluth (1999), Institute for Higher Learning Strategies (2000) and Quality Assurance Agency for Higher Education (2000). In addition, we found it useful to research new technology options. In our case, researching available online course construction packages and selecting a package that offers the most student interactivity options with the least amount of technological headaches was valuable. This was easier said than done and, in our case, meant experimenting with the packages and, more often, choosing what appears to be the best option and ‘learning as you go’. This process is time intensive and, even with a technologically adept instructor, improved over the semester and across semesters as the instructor learned how to use the power of the online course package functions. It would also be helpful to consult with instructional technology technicians about new software and hardware, and the efficient and effective methods of using both. Using action research methods to evaluate implementation. During the implementation phase, and subsequent evaluation and reflection of the data sources, data gathering methods similar to the problem identification phase were used. During this phase, however, data gathering was specifically focused on the major trends we found in the problem identification phase. In addition, due to the new course construction software we were able to use interactivity monitoring tools available with the course construction software. Regardless of what methods are used, it is important to continue through at least one full revolution of the action research process and preferable to continue using the process through several revolutions. In our study, we could not continue the process because the instructor left the course to take on other responsibilities. The process of actively researching online distance courses using action research methods decreases impersonality and enhances student interactivity. There is little doubt in the instructor’s mind that the action research process, even for the shorter duration of one complete cycle, enabled her to effectively alter the course to create a better learning
408
IMPROVING WEB-BASED INSTRUCTION
environment. Action research, by definition, involves the researcher(s) and stakeholders in the research process to improve the environment for the stakeholders (Greenwood & Levin, 1998). By involving the students in the process of studying what and how they were learning, the instructor felt greater student interactivity was elicited and strides were made toward decreasing the impersonality students felt with this online course. Action research methods embedded in the course, according to this instructor, created an avenue for students to voice important concerns or issues that was previously not available in the course, but that would have been available in an in-person classroom setting. Furthermore, the instructor felt that when students received feedback, help or saw course changes directly as a result of their voiced concerns, they were much more likely to become invested in the course.
Further Reflections on Web-based Instruction Since completing this action research project we feel much was gained in our understanding of how students managed the learning process that, in turn, allowed us to amend the course and better support student learning needs. At the same time, we learned a lot about web-based instruction in general. In light of our findings we were especially interested in reflecting on the role web-based instruction played in altering many of the learning methods we anticipated students would employ. We believe our three primary themes (impersonality and interactivity; course alignment, workload and accountability; and taking responsibility for independent work) provide great support for two primary educative goals in web-based construction: creating community and embedding critical thinking. The importance of creating community. Analysis of the data showed a predominance of the interactivity/impersonality issue throughout all phases of the project. This relates strongly to Brown & Duguid’s (2002) notion of the importance of peers in the educational process. ‘The more isolated learners are, whether physically or socially, the more they need access to peers, communities of practice and other social resources’ (Brown & Duguid, 2002, p. 227). A lack of interactivity in online learning reflects the ‘knowledge-delivery view of education’ where the students are vessels to be filled (Brown & Duguid, 2002, p. 221). What this perspective overlooks is the process of ‘learning to be’, which is considered an integral part of university life (Brown & Duguid, 2002, p. 221). Furthermore, it is the lack of interaction with peers that can prevent learning communities from developing. From our findings, we surmise that the lack of community development and peer support impacts upon student accountability, course structure, alignment and student willingness to take responsibility for learning. If the course were more personal, important peer–peer and student–faculty relationships would be pedagogically forged over the semester. We hypothesise that students might have taken more pride in their work and experienced better
409
Kathryn J. LaMaster & Nancy Knop mentoring into the profession through strong community bonds, and more positive and powerful discipline-based enculturation. What this suggests to us is that if we want students to learn to become knowledgeable and productive members of a particular discipline (in our case career professionals in health and physical activity) then we need to enculturate them to the practices and concepts important to the profession, rather than simply deliver information and hold them accountable for it. To do this, we must create opportunities throughout their college learning process for them to write and talk to peers and faculty about the practices and concepts (Garson, 2000). This is harder to do during web-based instruction, but could be done using pedagogically-creative online group-based tasks. Students could find or instructors could create online partnerships where students must communicate with each other, divide labor, and edit and critique each other's work in the task of creating a meaningful final project. The importance of embedding critical thinking skills. From the start of our action research project, students expressed concern over the volume of work expected of them. Throughout our project, students attempted to negotiate down the risk of the class by soliciting clearer definitions of what was expected. We responded by eliminating some work and better guiding student focus by creating study and test guides. Garson (2000) suggests that students attempt to overcome the isolation and lack of interactivity with faculty and peers by finding ways to increase course structure, which often serves to decrease learning autonomy. The more a class is defined by learning outcomes and test questions used to assess those outcomes, the more ‘education is narrowed toward training’ (Garson, 2000, p. 190). Teaching focused on information, rather than on the engagement of students in the practices of the profession is training (Brown & Duguid, 2002) and further serves to divorce students from opportunities to critically think within their discipline (Garson, 2000). In reflecting on the pedagogical decisions we made, we believe that streamlining the course and better directing student focus served to decrease student complaints about workload. Furthermore, it is quite possible that we have overused the testing process and have, in effect, limited the strength of the course as an enculturating process into the profession. At the same time, linking the learning tasks (library project, term paper and behavior change) and holding students accountable for discussing their findings in written form, as well as the enhanced access to peers and the instructor through the new software package probably served to increase the level of critical thinking that students needed to complete the class. We hypothesise, then, that by decreasing reliance on reading and testing, and increasing student opportunities to critically think, apply and share knowledge with peers and faculty, better learning and study methods appropriate for developing within the profession would be supported. Furthermore, reconstructing the webbased course in this manner would increase meaningful and educative student interactivity, enhance course alignment, better motivate students to
410
IMPROVING WEB-BASED INSTRUCTION
honestly engage in the course material, and better support student development into responsible and reliable professionals.
Correspondence Kathy LaMaster, Dept of ENS, San Diego State University, 5500 Campanile Drive, San Diego, CA 92182-7251, USA (
[email protected]).
References Ball, J. & Crook, B. (1997) Managing Change through Distance Learning, Community College Journal of Research & Practice, 21(1), pp. 13-23. Berger, N. (1999) Pioneering Experiences in Distance Learning: lessons learned, Journal of Management Education, 23, pp. 684-690. Brown, J.S. & Duguid, P. (2002) The Social Life of Information, 2nd edn. Boston: Harvard Business School Press. Carl, W. & Kemmis, S. (1986) Becoming Critical: education, knowledge and action research. Basingstoke: Falmer Press. Fontana, A. & Frey, J.H. (1994) Interviewing, in N.K. Denzin & Y.S. Lincoln (Eds) The Handbook of Qualitative Research. Thousand Oaks: Sage. Garson, G.D. (2000) The Role of Information Technology in Quality Education, in G.D. Garson (Ed.) Social Dimensions of Information Technology: issues for the new millennium. Hershey: Idea Publishing Group. Greenwood, D.J. & Levin, M. (1998) Introduction to Action Research. Thousand Oaks: Sage. Guba, E. & Lincoln, Y. (1989) Fourth Generation Evaluation. Newbury Park: Sage. Institute for Higher Learning Policy (2000) Quality on the Line: benchmarks for success in internet-based distance education. Available at: www.nea.org/he/ abouthe/Quality.pdf. Johassen, D. (1999) Designing Constructivist Learning Environments, in C.M. Reigeluth (Ed.) Instructional-Design Theories and Models: a new paradigm of instructional theory, vol. II. Mahwah: Lawrence Erlbaum. Land, D. (2002) Experiencing the Online Environment, USDLA Journal, 16(2), article 05. Available at: www.usdla.org/html/journal/FEB02_Issue/article05.html Lincoln, Y.S. & Guba, E.G. (1985) Naturalistic Inquiry. Beverly Hills: Sage. Mayer, R.H. (1999) Designing Instruction for Constructivist Learning, in C.M. Reigeluth (Ed.) Instructional-Design Theories and Models: a new paradigm of instructional theory, Vol. II. Mahwah: Lawrence Erlbaum. Quality Assurance Agency for Higher Education (2000) Guidelines for Distance Learning. Available at: www.qaa.ac.uk/public/dlg/guidelin.htm#Guideline%201 Reigeluth, C.M. (Ed.) (1999) Instructional-Design Theories and Models: a new paradigm of instructional theory, vol. II. Mahwah: Lawrence Erlbaum.
411
Kathryn J. LaMaster & Nancy Knop Rossman, G.B. & Rallis, S.F. (1998) Learning in the Field: an introduction to qualitative research. Thousand Oaks: Sage. Steiner, V. (1995) What is Distance Education, Distance Learning Resource Network. Available at: www.dlrn.org/library/dl/whatis.html.
412
