MODALITIES AND LEARNING IN COMPUTER SCIENCE Raymond Flood
Department for Continuing Education University of Oxford OX1 2JA
[email protected] http://www.kellogg.ox.ac.uk/ fellows/floodr.htm
Bob Lockhart Department for Continuing Education University of Oxford OX1 2JA
[email protected] http://www.kellogg.ox.ac.uk/ fellows/Lockhartr.htm
Pete Thomas
Computing Department Open University Milton Keynes MK7 6AA
[email protected] http://mcs.open.ac.uk/pgt2/
ABSTRACT In this paper we describe a part-time diploma course in computing delivered in two modes (face-to-face class teaching and electronically via the internet). We present results comparing the two modes in terms of student performance and their own perception of their educational experience. This project makes uses of a number of data sets involving performance data, student perceptions, demographics and attrition rates. In general, we found that students performed comparably in the two modes. Various issues are identified for further investigation particularly: the development of key skills, the importance of assessment in student motivation, and attrition in the Internet mode. Keywords Computing, modes, Internet mode, class-based mode, part-time education, key skills, satisfaction, attrition.
1. INTRODUCTION This paper reports on the findings of a project that compares an undergraduate computing course which is delivered in different modes. Our initial findings were first reported in the 2003 LTSN-ICS conference [7]. This paper updates and expands on those findings on the basis of further analysis and additional data and provides more details on the course. The Department for Continuing Education, University of Oxford, has been offering a classroombased, part-time undergraduate diploma in computing to adults for almost ten years. The diploma consists of two modules, each lasting for one year of study. In 1999, the department first presented the diploma in its Internet mode [3]. A major constraint on the development of the Internet delivered course was that it should differ from the class-based course only in the mode of its delivery and that the two courses should attract the same sort of students and have the same learning outcomes.
The Internet course is almost exclusively electronic, although there is a residential component in which the students experience group working and intensive study modelled on the class course study weekends. Internet students are allocated personal tutors to support their studies and mark their assignments; whereas the class-based students interact in face-to-face mode with their lecturers. This Internet course represents the first award-bearing electronic course offered by Oxford University and enquiries have exceeded the number of available places (currently 180). The class-based course can only accommodate about twenty students (limited by laboratory space). In North America there is considerable experience of adapting class based courses for electronic delivery. A typical model involves the presentation of existing materials in an electronic medium supported by a Web site, a single teacher and a range of communication facilities [5, 6, 8]. In contrast, our course was originally designed around the single tutor model used by the UK Open University [11] in which: •
students are divided into groups of about twenty, with each group supported by a single tutor;
•
communication between students and tutors occurs through electronic conferencing and email, both on a group and class basis;
•
there are purpose written materials, tailored for distance and electronic education [14].
In the initial presentation of the Internet course, we actually followed the “learning environments” model of having two tutors assigned to each tutor group [4]. The “lead tutor” provided academic support and marked assessments; the “support tutor” gave more informal contact and academic backup. In practice students and tutors became confused about these roles so we adopted the single tutor model. We have attempted to ensure that the two courses are comparable in that they: •
have the same admissions process and standards (so, attracting the same sorts of students);
•
are assessed in the same way (by making assignments and examinations as similar as possible, to the extent that they are interchangeable between the courses; identical assessments are not possible as the two courses are presented with different time-tables);
•
allow students to be able to move easily between the two modes of the course;
•
have the same processes in place to ensure educational and administrative standards are maintained.
The academic topics of both courses are identical and include such themes as: Computer Architecture, Operating Systems, Networking, Programming, Databases, and the World Wide Web. The two modes have the same learning objectives, are designed for the same groups of people and have the same assessment procedures. Where the courses differ is in their mode of delivery, the nature of the student support, and the timing of their presentations. Table 1 compares the essential features of the two modes. Class mode
Internet mode
Material delivered as lectures and handouts in two-hour sessions, once a week over thirty calendar weeks in both modules.
Material delivered as topics modelled on the subjects considered in the class course, tailored for the Internet.
Topics last for about six weeks, with material delivered incrementally throughout that time by a single lecturer.
Each successive topic is presented to the students in its entirety, at six-week intervals in the form of Web-based materials.
Topics subdivided into material covered in one evening class.
is
Topics subdivided into sessions that correspond to class course evenings.
Academic support comes from the lecturer and is available mainly at the evening sessions.
Academic support comes from a tutor who is allocated to the student at the start of the course. Students are in electronic contact with the tutor and may receive help at any time.
that
Student and lecturer interaction is usually synchronous - involving discussion occurring mainly at the lectures.
Student and tutor interaction is mainly asynchronous - involving email and an electronic conferencing system.
Student peer interaction would typically occur at lectures only.
Student peer interaction would typically occur electronically and involves the conferencing system. It occurs at no set time.
Students have three intensive weekend schools during the course of the year.
Students have one, six-day, summer school each year where they meet each other, have face-to-face teaching and work in teams derived from their tutor groups.
Each presentation of the course starts in October and lasts for two years.
Each presentation of the course starts in January (three months after the equivalent class-based course) and lasts for two years.
Assessment involves six written assignments per module and a three-hour unseen examination.
Assessment is exactly the same as in the class mode.
Table 1 – Essential features of the two courses.
Hence, we have a valuable opportunity to assess differences between two distinct delivery modes for computing courses [5,6,8,10,12]. Our preliminary studies [7] suggest that there is little to distinguish final student performance in the two modes but that the Internet course does experience higher attrition rates. In this paper we continue our investigation into whether the Web based course is comparable with the class-based course in terms of student performance and educational experience.
2. METHOD The methodology adopted for this investigation was an analysis of a number of data sets, involving: performance data, student perceptions and evaluations, demographics, and attrition rates. In the remainder of the paper we shall use the term presentation to mean an offering of the complete, 2-year course irrespective of mode. For example, presentation 3 refers to the offering of the class-based delivery of the course starting in October 2000, and to the Internet delivery starting in January 2001 (Internet students start three months later than their class-based equivalents). Thus, for a given presentation, the two cohorts of students will have studied essentially the same material but in different modes. Our initial investigations [7] used performance data consisting of continuous assessment and examination results from the period 1998 to 2002. In this paper we extend that work by analysing more recent data from 2003. In each presentation, the assessment and examination questions were set and moderated by the same group of teachers. Assessment grading, in the Internet course, was performed by individual tutors. For the purposes of our present study, our assumption is that our assessment policies are sufficiently objective to permit a direct comparison of results. Student perceptions of the courses were obtained through the Course Evaluation Questionnaire (CEQ) [2], which assesses student reception of teaching, attainment of goals and standards, appropriateness of assessment and workload, and the development of key skills. The CEQ is based upon a questionnaire developed by the Institute for the Advancement of University Learning, University of Oxford [9] which has been administered to several hundred full-time students of the University. Demographic information, including: age, gender, employment, ethnicity, disability and previous educational attainment, was obtained from admissions data. Information on student attrition rates comes from internal university records.
3. RESULTS 3.1 Student performance Table 2 summarises the assignment and examination grades for four successive presentations of the course. The data consists of the average assignment mark (expressed as a percentage) over all students over all assignments and the average examination mark over all students. Presentation 1
Average Assignment
Average Exam
No. Students
Module 1 Class
83.09
68.54
13
Module 1 Internet
78.49
75.69
32
Module 2 Class
79.2
66.0
13
Module 2 Internet
79.61
81.1
29
Average Assignment
Average Exam
No. Students
Module 1 Class
84.05
74.9
11
Module 1 Internet
80.99
76.88
50
Module 2 Class
77.8
65.36
10
Module 2 Internet
81.1
71.2
51
Average Assignment
Average Exam
No. Students
Module 1 Class
83.03
71.46
15
Module 1 Internet
76.56
69.06
77
Module 2 Class
78.37
72.64
14
Module 2 Internet
81.1
74.06
64
Average Assignment
Average Exam
No. Students
Module 1 Class
85.8
77.1
17
Module 1 Internet
78.1
72.1
66
Module 2 Class
72.1
69.4
16
Module 2 Internet
70.7
72.7
57
Presentation 2
Presentation 3
Presentation 4
Table 2-Assessment grades for four presentations
The data shows that the two modes of the course have very similar results in each presentation. There was a difference at the 5% level, using a t-test for the examination result for the second module of the first presentation - the phenomenon of students on the first presentation of a distance education course having relatively high grades is not uncommon. There was a similar difference at the 5% level in the average assignment score of the first module of the fourth presentation – this time, with the class mode having the higher value. We conclude that results over the first four presentations suggest that mode of delivery does not affect the performance of students who complete the course.
3.2 Course satisfaction indicators Course satisfaction data was obtained from the Course Evaluation Questionnaire (CEQ) which consisted of more than fifty questions. These questions asked the students to express the extent of their agreement with specified statements on the following five-point Lickert scale: 1
2
3
4
5
Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
Groups of questions in the CEQ are related to certain aspects of the learning environment [2,9]. There were questions in each of the following categories: 1. Good Teaching 2. Clear Goals and Standards 3. Appropriate Assessment 4. Appropriate Workload 5. Key Skills The most important consideration for us is the students’ overall satisfaction. The CEQ contains the key statement relating to student satisfaction: “Overall I am satisfied with the quality of this Diploma course”. The vast majority of students either agreed or strongly agreed that they were satisfied with the course. Table 3 contains the data for last three Internet course presentations. Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
Presentation 3
10
13
1
2
0
Presentation 4
28
24
0
0
0
Presentation 5
33
35
3
1
0
Table 3-Satisfaction data We correlated student responses to the five categories of questions with their response to the question on overall satisfaction, using Pearson’s correlation coefficient. The results for Presentation 4 are shown in Table 4. This presentation is the only one to have been fully analysed to-date (52 out of a maximum of 66 Internet students and 12 out of 17 class-based students responded to the questionnaire). Category
Good Teaching
Internet mode Correlation coefficient
Significance
0.400
Class mode Significance
(N = 52)
Correlation coefficient
< 0.01
0.257
Not significant
(N=12)
Clear Goals and Standards
0.365
< 0.01
0.590
< 0.05
Appropriate Assessment
0.254
Not significant
- 0.072
Not significant
Appropriate Workload
0.120
Not significant
- 0.080
Not significant
Key Skills
0.286
< 0.05
- 0.040
Not significant
Table 4-Satisfaction correlations for Presentation 4 For the Internet mode, the categories that can be related to student satisfaction appear to be: Good Teaching, Clear Goals and Standards, and Key Skills. For the class mode, the survey seems to indicate only the importance of clarity in setting clear goals and standards. Whilst we are aware that Pearson’s correlation coefficient is not particularly well suited to this type of data, we used it in order to compare our results with other results in the field. We have performed further analyses using, for example, Kendall’s tau (non-parametric) correlation, which supports the above results but with less significance.
3.3 Student Perceptions In this section we look at responses to individual questions on the CEQ where either there were strong opinions shared by the two groups or strong disagreements. Table 5 summarises the results. We note that all questions apart from 3 and 4 are expressed in a positive sense. In the case of all positive statements, the average response from the Internet-based students is lower than that for the class-based students, showing that they are more in agreement with the questions - as indicated by a negative number in the Difference column of Table 5. If questions 3 and 4 had been rephrased positively, as, for example: It has often been easy to discover what is expected of me in this course. To do well in this diploma, you really need more than a good memory. then one would have expected the differences in table 5 to have been reversed. That is, 0.12 for the reversed question 3 and –0.54 for the reversed question 4. Thus the responses to question 4 display the same characteristic as the responses to other questions expressed in a positive sense. Only in question 3 did we find that class-based students held stronger views. Question
1
The teaching staff normally give me helpful feedback on how I am doing
Average Internet
Class
1.86
1.92
Difference = Internet - Class -0.06
2
The teaching staff motivate me to do my best work
1.79
1.93
-0.14
3
It has often been hard to discover what is expected of me in this course.
3.81
3.93
-0.12
4
To do well in this diploma all you really need is a good memory
4.04
3.50
0.54
5
The course has helped to develop my ability to work as a team member.
2.15
3.14
-0.99
6
The course has developed my problem solving skills.
2.13
2.64
-0.51
7
The course has helped me to develop the ability to plan
2.10
2.71
-0.62
my own work 8
Course administration is effective in supporting my learning
1.58
2.43
-0.85
9
The way I have studied has been influenced by the demands of the assessments
1.98
2.07
-0.09
10
My course has stimulated my enthusiasm for further learning
1.50
2.29
-0.79
11
The use of IT helped me to learn
1.81
2.29
-0.48
12
Overall I am satisfied with the quality of this course
1.46
1.79
-0.32
Table 5-Course evaluation responses Questions 1 and 2 are representative of the responses to the appropriateness of the teaching and shows that both sets of students either agreed or strongly agreed with the positive statements about good teaching. Question 3 comes from the category of clear goals and standards and shows good agreement between both groups of students. Question 4 is representative of the category dealing with appropriate assessment and, as observed above, reveals that both sets of students felt that the assessment required more than just a good memory but that the Internet students felt this more strongly than their class-based counterparts. Questions 5, 6 and 7 are concerned with the development of key skills. The responses indicate that the Internet students felt that their team working abilities were enhanced whereas the class-based students did not. We believe that this is due to two factors which differentiate Internet from class-based students. First, that the Internet students were learning at a distance and had to work collaboratively using conference and e-mail facilities. Second that the Internet students are compelled to work in groups during the week-long summer school. Class-based students did not feel the necessity to collaborate on their studies and have more limited experience of group working. Question 8 shows that the Internet students were more positive about course administration and we believe that this is due to the fact that working at a distance naturally involves students having to interact more with the administration. Question 9, dealing with assessment, confirms established results about the importance of continuous assessment particularly in distance education [1]. Finally, questions 10 to 12 show very positive reactions to the course, the desire for further learning and the use of information technology in teaching (even for the class-based students, who were not required to use conferencing or email as part of their studies). The CEQ also contained the free response question, “What are the best aspects of this diploma course?” which revealed an interesting difference between the two modes of study. The Internet students tended to focus on teaching and support: “Very helpful, experienced tutors, feedback very good”; “Tutor group system for guidance/questions”; “Being able to organise your own work time”. Whereas the class-based students concentrated on course content: “A good mixture of theory and practice”; “Modules were very interesting, which kept my attention and (inspired me to) want to find out more about the subject”; “Material was very up to date”.
3.4 Demographics Our course application forms give us much demographic data and the Internet mode attracts students from all over the world. Two interesting points are: •
Both modes attract populations of students with mean age around 35. The class course attracts slightly older students.
•
The class mode consistently attracts a population in which about 50% of the students are male, whereas the corresponding figure for the Internet mode is approximately two-thirds.
3.5 Attrition Tables 6 and 7 show the attrition rates for presentations 3 and 4. There are clear differences between the Internet and class modes. There is significant drop-out on the Internet course but much less on the class-based course, which is in line with observed behavior in distance education; see, for example, [13]. In our system, students who discontinue their studies can indicate either that they wish to continue in the next presentation (Defer) or withdraw from studying the course altogether (Withdraw). We infer that those who defer are satisfied with the course but have overwhelming non-course reasons for withdrawing from the current presentation, whereas the group who withdraw includes people who are dissatisfied with the course. The figures show that, on both courses, the greatest drop-out occurs in the first module, and that, provided the students progress to the second module they are very likely to complete the course successfully (pass). The final column in these Tables shows the pass-rate calculated as the ratio of the number of people gaining credit divided by the difference between the number of people starting the course and the number of people deferring. Note that the number of students starting the second year of a presentation consists of the majority of the students who completed the first year of that presentation plus students who deferred from earlier presentations or transferred from one mode of the course to another. Initial nos.
Defer
Withdraw
Fail
Pass
Module 1 (Internet)
125
29 (23%)
16 (13%)
1
82%
Module 1 (Class)
18
1 (6%)
1 (6%)
0
94%
Module 2 (Internet)
80
6(8%)
7 (9%)
2
88%
Module 2 (Class)
15
0 (0%)
0 (0%)
0
100%
Table 6-Attrition and success rates for presentation 3. Initial nos.
Defer
Withdraw
Fail
Pass
Module 1 (Internet)
123
31 (25%)
23 (19%)
2
73%
Module 1 (Class)
18
0 (0%)
1 (6%)
0
94%
Module 2 (Internet)
69
7 (10%)
5 (7%)
0
92%
Module 2 (Class)
16
0(0%)
0 (0%)
0*
100%*
Table 7-Attrition and success rates for presentation 4. (The asterisk indicates that two students are still in the process of satisfying the regulations).
Tables 6 and 7 show that the Internet delivered mode has a much higher initial attrition rate than the class-based mode. This is typical of the observed patterns in previous presentations. Figures 1 and 2 show the Internet retention rates for presentations 3 and 4.
140 120 100 80 60 40 20 0 Reg
Topic 1
Topic 2
Topic 3
Topic 4
Topic 5
Topic 6
Progress
Progress
Topic 7
Topic 8
Topic 9
Topic 10
Topic 11
Topic 12
Pass
100 80 60 40 20 0
Figure 1-Retention in Presentation 3
140 120 100 80 60 40 20 0 Reg
Topic 1
Topic 2
Topic 3
Progress
Topic 7
Topic 8
Topic 9
Topic 4
Topic 5
Topic 6 Progress
100 80 60 40 20 0 Topic 10 Topic 11 Topic 12
Figure 2-Retention in Presentation 4
Pass
It seems clear that the further into the course the students get the more committed they become to finishing.
4. FUTURE WORK We intend to do considerably more analysis of the demographic and attrition data in order to look more closely at the reasons behind our attrition rates. Students who leave usually cite lifestyle changes as the primary cause, almost certainly, in good faith. However, students completing the first year seem, thereafter, relatively immune to such changes. When we compare the behaviour of Internet students with class-based students we see that Internet students have higher attrition rates. A priori this might suggest that the initial drop-out during the first year is related to the mode of delivery. However, similar behaviour has been observed in other distance education courses without electronic delivery. It seems to us that a key factor in drop-out is the students’ motivation which is closely related to the completion of assignments and we wish to examine this relationship in more detail. We also intend to investigate the issue of key skills and their acquisition, in the two modes.
5. SUMMARY There is little to distinguish final student performance in the two modes. The data we now have supports the view that students undertaking the course are neither advantaged nor disadvantaged educationally by delivery mode. Through the mechanism of the CEQ, students on the Internet mode certainly expressed high satisfaction with the course. In presentation 4, we found that students on the Internet mode related satisfaction to good teaching, clear goals and standards and key skills whereas on the class-based mode students related their satisfaction only to the clarity in setting clear goals and standards. The feedback on individual questions on the CEQ showed remarkable consistency of opinion between the two modes, with the Internet mode students almost always holding slightly stronger views (whether positively or negatively held). It is useful to note that administrative arrangements on the Internet mode are important. The responses to the free form question did reveal a difference between the two modes with respect to student satisfaction: Internet mode students focused on teaching and support whereas the class-based mode students focused more on course content. It will be interesting to see whether a similar pattern emerges for other presentations. The Internet mode has higher attrition rates. There is widespread support for the view that this is typical of distance education courses [13] but we need to undertake further work to see whether some part of the attrition rates can be related directly to delivery modes.
6. ACKNOWLEDGEMENTS We should like to thank Dr John Axford, course director of the Internet Diploma in Computing, Oxford, and Dr Paul Griffiths, for their support and helpful advice. This research was carried out while P.G.Thomas was Mawby Visiting Fellow, Kellogg College, Oxford. The work was partially funded by LTSN-ICS, and we gratefully acknowledge their support.
7. REFERENCES [1] Baath J. Assignments in distance education – an overview. Epistolodidaktika 1 13-20 (1994). [2] Cleat web site.
http://www.conted.ox.ac.uk/cleat/
[3] Computing course information web site: http://www.conted.ox.ac.uk/computing [4] Darby, J. Second Generation e-Learning realised through Learning Objects. http://www.aln.org/conference/proceedings/2001/ppt/01_darby.ppt [5] Dutton J., Dutton M. and Perry J. How do online students differ from lecture students? JALN 6 (1), 1-20 (2002). [6] Dutton J., Dutton M. and Perry J. Do online students perform as well as lecture students? Journal of Engineering Education 90 (1), 131-136 (2001). [7] Flood, RG. Lockhart R, Thomas P. G. Comparing modalities in learning in computer science. http://www.ics.ltsn.ac.uk/pub/conf2003/comparing_modalities.htm [8] Gagne M. and Shepherd M. A comparison between a distance and a traditional graduate accounting class. T.H.E. Journal Online 28 (9) 1-7 (2001). [9] Institute for the advancement of University Learning. http://www.learning.ox.ac.uk/iaul/default.asp [10]Meisalo V., Torvinen S., Suhonen, J. and Sutinen E. Formative evaluation scheme for a web-based course design. Proceedings of ITiCSE’02, Aarhus, Denmark. 132-133. (2002). [11]Open University. http://www.open.ac.uk/ [12]Richardson J.T.E, Morgan A. and Woodley A. Approaches to studying in distance education. Higher Education 37 (1) 23-55 (1999). [13]Sharma, R. C. Student retention in open and distance learning systems. Proceedings of the XV annual conference of the Asian Association of Open Universities, New Delhi, India (2002). [14]Thomas P.G., Carswell L., Price B. and Petre M. A holistic approach to supporting distance learning using the Internet: transformation, not translation. British Journal of Educational Technology 29 (2) 1-13 (1998). [All web references were valid on 03/06/04]
