EVALUATING THE EFFECTIVENESS OF RETENTION STRATEGIES USING SMS, WAP AND WWW STUDENT SUPPORT John Traxler
Brendan Riordan
National ICT Research Centre Priorslee Campus Telford, Shropshire TF2 9NT
[email protected] http://www.learninglab.org.uk/researchcentre
Senior Lecturer – University of Wolverhampton 35/49 Lichfield Street Wolverhampton WV1 1EL
[email protected] www.scit.wlv.ac.uk/brendan working on courses in classrooms, lecture theatres and laboratories. Evaluation has depended on a small stable repertoire of techniques. The changing political and social climate have forced educational institutions to address new constituencies, for examples access students without well established study skills and full-time students forced to hold down full-time jobs, to teach using new technologies, for example networked computers being used to teach asynchronously and at a distance, and to teach increasing numbers of students on static resources. This means that developmental and innovative projects working in these environments must adapt and explore more innovative and varied techniques to evaluate their effectiveness [1]. This paper describes briefly a small innovative project funded by LTSN [2] that uses blended mobile technologies and explores how such projects necessitate new evaluation techniques.
ABSTRACT In this paper, we examine the various strategies used to elicit feedback and evaluation from students involved in developmental and innovative educational projects. The examination springs from work being done on an LTSN-funded development project that uses a blend of mobile technologies to support students “at-risk”. As with many projects using new learning technologies with more and more diverse communities of students, it has needed to recognise that that the “classical” repertoire of questionnaire, focus group, observation and interview developed in the days of largely sedentary education are becoming inadequate for the richness and variety of current educational developments. The paper looks at work to develop and trial some alternatives, in keeping with the new ethos of technology supported learning. Keywords Evaluation techniques, m-learning, SMS texting, student retention, blended learning
Evaluation, in the context of the current discussion, is any attempt to access learners’ views about the value (actual, perceived or expected) of what they have experienced through the project. Any such evaluation might look for a number of different kinds of outcome from a project and for each must use an appropriate elicitation technique. The obvious outcome for any educational project is some cognitive change, where students are expected to have learnt something new about computing. An evaluation might also be looking for metacognitive change, where students have learned something about the process of learning. An evaluation may also look for affective changes in students, reflecting changed feelings, values or preferences. These are all purely individual changes and effects. An evaluation may look for social changes; perhaps in how students work with each other, or in how groups of students show increased collective skills
1. INTRODUCTION Evaluation is a key issue for any developmental project. Firstly because it informs the outside world about the effectiveness of the project, specifically in relation to project objectives, and secondly because it provides some insights to the funders on the utility and cost-effectiveness of their investment. For education projects, the development of evaluation strategies has historically focussed on face-to-face contact with students (and perhaps other stakeholders) Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission. 4th Annual LTSN-ICS Conference, NUI Galway © 2003 LTSN Centre for Information and Computer Sciences
54
technology have a social significance and a fashion dimension. Basic skills, literacy and numeracy are core to a sense of personal adequacy whilst marks and grades are the outward sign of academic progress
2. THE CHARACTERISTICS OF AN IDEAL EVALUATION Any evaluation is likely to face a number of practical and operational constraints. Nevertheless, it is useful to identify the characteristics of an ideal evaluation. These at least represent a point of departure. The ideal evaluation should be: •
Rigorous, meaning that conclusions must be trustworthy and transferable
•
Efficient, in terms of cost, effort, time
•
Lightweight, that is, less onerous than the learning itself.
•
Appropriate to the specific learning technologies, to the learners and to the ethos of the project
•
Authentic, in accessing what learners really mean, really feel
•
Consistent across
different groups or cohorts of learners
time, that is, the evaluation is reliably repeatable
whatever varied technologies are used
whatever courses, modules organisations are involved
whatever protocols, instruments, coding, analysis and presentations are developed
There may be cultural gaps, ethnic or gender gaps between evaluators and students. The former are more likely to be middle-class, possibly middleaged with the latter being young, (thanks to student loans) in debt and occupants of a completely different cultural space. This means that the two groups will speak essentially different languages (slang, jargon and txt messaging are literal examples) and dialogue between the two will be flawed by issues that are “not-worthmentioning” or “taken-for-granted”. Some of these ideas are adapted from the cognitive psychology of “knowing” [3]. Many of the development in pure mobile learning approximate or aspire to a style of learning that is spontaneous, unstructured, unpredictable and informal – at least in its short-term, local aspects – and it may be very difficult to devise evaluation strategies that harmonise with this ethos. This means evaluators will not always learn the truth, certainly not without using appropriate tools and techniques.
or
4. A REVIEW OF EVALUATION METHODS In order to provide a basis for innovation in evaluation, it pays to review the techniques and principles currently in use before asking which can be adapted to new situations whilst preserving their rigour and effectiveness. The classification is merely a convenient grouping.
3. WHY EVALUATION IS DIFFICULT The ideal evaluation will be compromised by practical and operational considerations. It will also be constrained by the very nature of students, the subjects they study and the society around them and by the means by which they study. To illustrate these points in a practical way, consider the following:
For software engineers, the problem of finding out what people want is probably familiar as requirements elicitation, and the problems with eliciting evaluation are broadly comparable to those of eliciting requirements. There have been several attempts to set requirements elicitation on systematic and rigorous foundations and this work is valuable in the current context [4]
If students are using SMS mobile ‘phones, PDAs or WAP as learning technologies, the interface and bandwidth through which the evaluation might take place is narrow, artificial, constrained and inconsistent.
4.1 Conventional Methods This heading covers the techniques that might be described as “classical” [5]. They are:
Students cannot or will not always be able to tell an evaluator about their abilities, knowledge, values, needs, preferences, goals or feelings, or any changes in them – they may not be able to put them into words, they may be too embarrassed to disclose or reveal.
•
Evaluation may raise issues of self-esteem, social standing and status – much of education is concerned with affluence and success signified by qualifications and employment, whilst ownership and competence with computer and ‘phone
55
Questionnaires: are driven by driven by the evaluator agenda rather than by student priorities, they need care and piloting. They can be indirect (“what do you think other students come here for?”) and they can be online, telephone-based, MCQs, free text or constrained answers.
• •
•
alien to conventional interpersonal interactions. They are used widely in market research. Examples include:
Focus groups: are driven by the most vocal students and are practically impossible to transcribe Interviews: are costly to transcribe and need careful preparation and delivery. They can be conducted by telephone. Observations: problematic to code and produce large amounts of irrelevant data. They do allow access to “back-versions”
These methods are all generally wordy and paperbased. They are potentially difficult to code and analyse, based on a face-to-face archetype of teaching and learning and depend largely on student self-report/self-presentation.
•
Card sorts: the students sorts packs of cards according to their own criteria
•
Repertory grids: the students sort more elaborate grids
•
Laddering: the evaluators attempt to elicit “the reasons behind the reasons”
In general, they are interview-based (though ‘phone-based versions have been used), repeatable and quick (though can only be conducted one-to-one), highly focussed and access students’ goals and values. They can be problematic because of their personal nature.
4.2 Courseware Evaluation This terms is being used to describe any of the host of evaluation methods [6] [7] [8] that have grown out of the need to evaluate the effectiveness of educational multimedia software packages, for example: pre/post testing, use of control groups, confidence logs and experiments.
4.5 Additional Constraints There are practical issues around the coding, transcription and analysis. Effort and expense vary enormously, for example:
They are often based around specific cognitive gains derived from formal models, settings and curricula. They are not necessarily intrinsically learner-centred and are based on individual, not social learning
4.3 Objective Techniques
•
Oral interview: a transcript takes 1:6 plus coding. The possibilities of transcription software are untried
•
Video observation: the effort is horrendous unless theory-driven
•
Card-sorts: software [10] does most of the work!
The nature of the conclusions needs consideration. The evaluation may only come up with conclusions using phrases like “may be typical of”, “indicative”, “probably”, “may influence”, “could contribute to”. There may be nothing statistical or quantitative and the evaluation may only serve to define or clarify future research questions
This phrase is taken to cover techniques where the technology not the evaluator generates the data upon which the evaluation is based. Examples include instrumentation, logs, hits, split screen video and perhaps quiz or test results eg MCQ These are not a direct measure of the learning or the learners but are interesting to compare, contrast or corroborate with learner selfreport especially if harvested consistently across devices. One current example of instrumentation, mentioned by Jon Trinder of Glasgow University, relates to an experiment based around PDAs:
5. EVALUATION OF LTSN PROJECT The LTSN project was speaking “blended” learning, consisting of a number of different media. In fact, the novel components, the SMS and WAP, were supplementary and intended to support learners “at-risk”. This meant that evaluation could use whichever conventional and innovative techniques were appropriate. Taking the “lightweight” principle as paramount in interacting with the students, the intended evaluation would have used
"...we have an application that runs on the PDA to log what it gets used for and when, in the hope of producing some actual results. Rather than trust users saying they've used the device, we can tell..." Because much mobile learning uses embedded and closed ad-hoc systems, this approach can be time-consuming. Where successful, it may then require sophisticated cross-referencing and statistical analysis.
4.4 Contrived Techniques This is probably the least familiar category. It describes those techniques, derived from the work of Personal Construct Theory and developed by Gordon Rugg and his collaborators [9], that are
56
•
end of year focus groups, based on a routine student meeting
•
WAP and VLE forms and asynchronous forum, moderated by the author [11]
•
paper questionnaire, filled in face-to-face
•
SMS text questions using Likert scale to create a virtual questionnaire
and intakes vary. This project was specifically targeted at “at-risk” students.
In the course of the trial, there was considerable positive informal feedback from students to the course leader (one of the project team) but in the event, only six students attended the proposed focus group (it had been scheduled after a revision session). There was also little take-up of VLE technology by the students and they vetoed WAP as being understandably expensive. The group was sent SMS queries inviting them to comment on aspects of the trial but few replied and thus there was little access to large-scale or systematic views. Each module has a university “Module Evaluation Form”; one student mentioned the best thing was “recieving (sic) text messages”.
The project has had several indirect outcomes.
It was however possible to deliver and analyse a conventional questionnaire to the students (Questions and responses in the Appendix). The first batch of questions revealed that the vast majority of the students thought the experiment was worthwhile, largely independently of the exact way the questions were phrased. The second batch of questions attempted to access, perhaps less successfully, preferences for the type of SMS intervention. This was probably influenced by those interventions to which the students had actually been exposed, hence the preference for “teaching material” Individualised messages are sharply differentiated, with a strong preference for coursework feedback and against individual admin. The other headings all received generalised approval but the questions did not probe preferences, reasons or opportunity costs, all of which would be needed to unpick the underlying thinking. The third batch of questions looked at the communications issue from the perspective of university broadcasting and dissemination. “Push” technologies, email and SMS, score more highly than “pull” technologies, web sites and notice boards, though students are indifferent to web sites but antagonistic to noticeboards.
45.35 46.34
•
The project team have secured funds from the University and collaboration with Sony to support a project using PDAs to support “students-at-risk” in computing.
•
The project team have mounted a national workshop on mobile learning in computing. This brought together computing lecturers and workers from other disciplines currently delivering HE content using mobile devices.
The evaluation of student responses was the last link in the chain and dependent on technical and organisation successes in all the preceding stages of the project. Being a developmental project attempting to integrate technologies and then devise teaching formats for them, these successes were not guaranteed or necessarily forth-coming. There was insufficient time to experiment with all the possibilities, specifically linking SMS to assessment marking and signalling remedial support. So, an evaluation of the evaluation would be that much can still be learnt about evaluation but the project was viewed as a successful experiment by staff and students.
7. REFERENCES [1] Traxler, J. (2002). Evaluating m-learning. Birmingham, UK: [2] Riordan, B. & Traxler, J. (2003). Supporting Computing Students at Risk Using Blended Technologies. Galway, Ireland: ICS-LTSN. [3] Rugg, G. & McGeorge, P. (1999). Questioning Methodology. University College Northampton Faculty of Management and Business.
There were 59 students in the module. Two students did not take the exam, although they had signed up for the SMS project. Their 'zero' results in the exam have therefore been removed. The score for the SMS group is slightly better than the non-SMS. There are however too many hidden variables to justify sophisticated analysis. SMS Group
The project team have made a presentation to the University IT Infrastructure Group, exploring the institutional implications for SMS support.
6. DISCUSSION
The free-text responses range from the imaginative to the pragmatic. Most provide a positive basis for improving the service
Whole Group
•
[4] Maiden, N. A. M., & Rugg, G. (1996). ACRE: Selecting Methods for Requirements Acquisition. Software Engineering Journal, 11, 183-192. [5] Sapsford, R., & Jupp, V. (Eds.). (1996). Data Collection and Analysis. London: SAGE Publications. [6] Harvey, J. (Ed.). (1998). LTDI Evaluation Cookbook. Edinburgh: Learning Technology Dissemination Initiative. http://www.icbl.hw.ac.uk/ltdi/ltdi-pub.htm
Non-SMS Group 43.37 Comparison within and across cohorts is, as ever in education, problematic because markers and assessment regimes change, the subject evolves
57
[7] Clow, D. (2000). Evaluation Methods and Procedures for studying learners' use of media. 2003(3/3/03). http://iet.open.ac.uk/plum/evaluation/contents. html
7)
with feedback on all assessments
8)
text messages being sent over the period of the semester instead of just right at the end
9)
some answers should be given to the [revision] questions
[8] Milne, J., & Heath, S. (1997). Evaluation Handbook, for successful CAL courseware development Aberdeen: University of Aberdeen.
10) messages only sent out within one part of the day eg between 12-1 11) a way of students responding back 12) texts seem to have come together and should maybe have been closer to the exam
[9] McGeorge, P., & Rugg, G. (1992). Uses of Contrived Knowledge Elicitation Techniques. Expert Systems, 9(3), 149-154
13) not so excessive. there were a load of messages overloading my inbox 14) don’t txt to [sic] early in the morning!! More information eg web site addresses
[10] Traxler, J. (2002). National ICT Research Centre Card Sort Software. http://www.learninglab.org.uk/researchcentre/q uestioning.asp?ses.
15)
Which messages are most useful: 46 returns
[11] Salmon, G. (2000). e-moderating - the key to teaching and learning online (F. Lockwood, Ed.). London: Kogan Page.
Most Useful
Fairly Useful
Useless
Didn’t answer
29
13
1
3
25
19
0
2
Teaching material such as revision tips
39
7
0
0
33
12
0
1
18
24
0
4
urgent like changes
8. APPENDIX – QUESTIONNAIRE DATA
admin room
General reminders, coursework deadlines
eg
46 returns
Yes, definitely
Maybe
No way
Didn’t answer
Are you pleased you took part in the experiment?
43
2
0
1
Did the experiment help in your studies?
40
5
0
1
Individual feedback coursework
Would you recommend the experiment to students?
38
7
0
1
Individual admin eg appointments with tutors
1) How could the experiment be improved ? Any ideas or suggestion
emails of information should have been sent, not just texts
on
Do you prefer communication by: all students mobile numbers should be chased up, feedback and texting should be accessed via users
2)
more help. Let us know changes to lectures. send results
3)
two way sms communication
4)
it was perfect
5)
no improvements except message to test messages
6)
the times that sms were sent out eg between 9 am and 9 pm some were being sent early and very late
for
[illegible]
58
46 returns
I prefer
I don’t mind
I don’t like
Didn’t answer
Consulting university websites
9
24
10
2
Receiving email to your university account
24
17
3
1
consulting university notice boards
8
14
20
3
receiving SMS
33
11
0
2
