Comparing the efficiency and effectiveness of different learning media designs for the preparation of skills training in a blended learning context N. Verliefde1, L. Stevens2, B. D’haenens3, A. Vermeyen4, R. Kaat5, M. De Gendt6, C. Van Caelenberg4, K. Stifkens7, S.Van Rampelberg4, T.Neijens, J. Van Den Bossche8 1
Association of the Catholique University of Leuven (BELGIUM) KAHO Sint-Lieven, Service for further education, Ghent (BELGIUM) 3 KAHO Sint-Lieven, Service for educational support and development, Ghent (BELGIUM) 4 KAHO Sint-Lieven, Aalst (BELGIUM) 5 Technical Institute Sint Carolus, campus Waas, (BELGIUM) 6 Hogeschool – Universiteit Brussel, Nursing Education, Brussels (BELGIUM) 7 Catholic University College Sint Lieven, Nursing Education, Sint Niklaas (BELGIUM) 8 Leuven Engineering and Sience Education Center (LESEC) (BELGIUM)
[email protected] ,
[email protected] ,
[email protected] ,
[email protected],
[email protected] ,
[email protected],
[email protected] ,
[email protected],
[email protected],
[email protected];
[email protected]; 2
Abstract This practice based research on effectiveness and efficiency of learning media design is meant to provide an answer to the professional bachelor programs nursing of three affiliated university colleges of the Catholic University of Leuven and one associate degree program for the optimization of their blended learning approach for the practical skills component of the programs. In particular we want to find the most optimal media design for students to prepare themselves independently for the practicum nursing skills. This replaces the demonstration of the skill that is normally done by the lecturer during the contact time of the practicum. We compare three stratified groups of hundred students. Each group prepares themselves for the real practicum of the next day by means of a different online medium: a short video with limited narration (spoken information: “what”), a long video with elaborate narration (more detailed spoken information: “what”, “how”, “why”), and a multimedia interactive learning module with learner control. We find supportive results for the decision to augment narration in the instruction videos from limited to elaborate narration. Integrating learning objects into a multi media module with learner control however, does not influence the learning outcomes/results (learning effectiveness) nor the time spent on the lab preparation assignment (learning efficiency). The final aim is to decide on basis of a cost-effectiveness evaluation on the future media design to be used in both the pre-service and post-service nursing programs. Keywords: learning effectiveness, media design, multimedia, blended learning, instructional media, practical skills education, nursing skills training, asynchronous learning
1
INTRODUCTION
The rise of new multimedia technologies for online learning, brings new opportunities for the practical components of educational programs to make the contact times during practica and skills lab more efficient. In the blended learning approach of the nursing programs involved in this project, the face-to face demonstration of the nursing skill is replaced by an online instruction video. This not only saves
time in se, but also allows students to better prepare themselves before coming to the skills lab, which allows them to spend more time practicing the skill with larger insight in the procedure of the skill. Blended learning is generally defined as an educational approach whereby face-to-face and online methods are combined [1]. In Napier’s [2] opinion ““Blended learning significantly reduces face-to-face instruction by incorporating rich, online learning experiences”. In our blended learning approach the focus is more on increasing the quality of the face-to-face instruction, rather than reducing it. The lecturer can shift his/her role from knowledge transfer to guidance of the learning process. We investigate whether the online learning experience we offer by means of the instruction videos can be made “richer”. The central question is: which choice of media and media design can offer the best support to nursing students to prepare themselves optimal for the practicum session/ skills lab ? In a preliminary satisfaction questionnaire on the quality of the instruction videos used in the technical nursing skills course, the main wish from the students was more elaborate narration (spoken information) in the video and to align the video better with the steps of the written procedural guidelines of the technical nursing skills manual. This however contradicts with assumptions and believes of the lecturers on effectiveness of the media design: - “If we do what students ask, explaining everything they need to know in the instruction video, they won’t look at their textbooks (technical manual) anymore !” The focus from the service offering further education nursing programs is more on “attracting” clients and is therefore considering a multimedia design of attractive (inter)active learning modules with learner control. Here cost-effectiveness questions raise: - “Is the time consuming production process of an interactive multimedia module worth the effort ? - “Is there an equivalent less time consuming way to add active learning activities to the passive watching of a video, by means of our learning management system?”. In this comparative analysis we investigate the effect of the educational design of online instruction video and of an interactive multimedia module on the learning process as well as on the learning outcomes
2
OBJECTIVE
To be able to make an evidence based decision on which media design to opt for in future, we compare: the effectiveness and efficiency of the use of different learning media designs for students to prepare themselves optimal for a practicum / skills lab.
3 RESEARCH QUESTIONS Learning effectiveness We are primarily interested in the learning effectiveness, the effects on the learning outcomes. Which media design realizes the largest level of recall and insight in the procedure of the practical nursing skill ? The two main research questions on effectiveness are: 1. Do elaborate self sufficient instruction films contribute to learning effectiveness as compared to short videos whereby one still has to read the written textbook or guidelines to understand the rationale behind certain steps of the procedure ? 2. Does a flexible interactive learning path of a multimedia module with integrated learning objects (text, video, self-evaluation questions with immediate feedback ) significantly enhance learning as compared to a linear structured instruction video offered separately from the other learning objects ? Learning efficiency Is the time spent on the assignment in the group of learners that obtain the learning objectives in a sufficient way (effectiveness) less in any one of the media designs ? Attractiveness of learning
Do students perceive the integrated learning module as more attractive than the short video or long video ?
4
EDUCATIONAL PRINCIPLES ON MEDIA DESIGN
In the following table we summarize the characteristics we take into consideration to improve the quality of the learning media design. They are derived from the highlighted wishes from our students as well as from the principles of the cognitive learning theory on multimedia from Mayer [3] and the Framework of Screenwriting principles for instructional video from Koumi [4]. Table 1: characteristics of the learning media design Media design 1: short instruction video
Media design 2: long instruction video
Media design 3: Interactive multimedia (learning module)
RESEARCH QUESTION 1 limited narration
elaborate narration RESEARCH QUESTION 2
Separate learning objects:
Separate learning objects:
Integrated learning objects
Audio-visual: video (short)
Audio-visual: video (long)
+ text (written guidelines)
+ Text (written guidelines)
Multimedia: video (long video), text, SEQ, with segmentation..
+ self evaluation questions added in LMS (blackboard)
+ Self evaluation questions (SEQ) added in LMS (blackboard)
Linear learning path
Linear learning path
Flexible learning learner control.
(Inter)activity: SEQ with delayed corrective feedback
(inter)activity: SEQ, with delayed corrective feedback
Interactivity: SEQ with immediate corrective feedback.
path
,with
Research queston 1: elaborating narration effect on learning ? Whether an elaborate video contributes to learning effectiveness as compared to a short video with limited narration, we need to know how much narration in the video is beneficial for learning and what should the content of the narration be ? In the preliminary satisfaction questionnaire and interview students suggest: - more narration explaining “what is done, how it is done and why it is done”. In the current video design the narration (spoken information) accompanying the moving images, is often limited to “what is done”. Lecturers limit the amount of information deliberately, because they are afraid students won’t look at their textbook / written procedural guidelines anymore if they explain too much in the video. The narration in the elaborate video contains: -“What”: Labeling the steps of the skill offers structure, matching the sequence to the written guidelines of the skill. -“How”: Stressing specific attention points on how to carry out a (sub)step of the skill or on how to make the right choice of materials e.g. “See to it that you stretch the skin of the injection place before injecting the needle !” e.g. “Watch out: the higher the gauge the thinner the needle !”
-“Why”: explaining the importance of the attention point, explaining the positive consequences when carried out correctly and the negative when it is omitted. e.g. “You chose a short needle when the patient is skinny, otherwise you might hit the bone.” Is what the students wish, being supported by Koumi’s theoretical framework of video design and screenwriting principles [4]? Indeed Koumi explains under his constructivist principle: “words should not duplicate pictures, they should add something extra”. But under the other hand he warns for overload.(Koumi’s 6.3 principle). If one is aware of the importance of a step in the procedure, has therefore been able to link the new knowledge to an existing knowledge construct in the brain, one is less likely to forget it. (constructivist principle). But if we overload, the student can not pay attention to the sequence of the steps anymore Research question 2, integrating learning objects, learner control, adding interactivity effect on learning ? Whether a flexible interactive multimedia module with integrated learning objects significantly enhances learning as compared to a linear structured instruction video offered separately from the other learning objects, we make the following assumptions: - Integrated learning objects; Mayer’s principles on multimedia design: Multi media principle: people learn better from words (written and spoken) and pictures than from words only. Segmenting: presenting in learner-paced segments Spacial contiguity: people learn better when corresponding graphics and printed text are placed near each other.
In the learning module of media design 3, we cut up the written procedural guidelines into sections (sub steps of the skill) which we put in text frames together with the fragment of the video illustrating this sub step. We still provide the full video version as well in one of the first frames of the learning module. However reading and listening to information at the same time does not go together (redundancy principle of Mayer). Students can chose which one to look at first, text or video, and switch easily between the two. - Flexible navigation, “learner control” of the learning path is more effective because the learner can control the learning path according to own preferences and in line with the learning style. He can chose for: linear or non-linear; holistic versus analytical (long video, short fragments), sequence of content by means of menu-bar in the left column, sequence of type of learning object: text / video,… However research on the Aptitude Treatment Interaction from Cronbach & Snow [5] show that not all learners profit from learner control. Snow states that low achievers and anxious or conformist type of learners tend to learn better in highly structured instructional environments. The high achievers and independent students profit from “learner control”, which is confirmed by Clark & Mayer [6].
Our student group, Flemish professional bachelor nursing students of the Association of the Catholic University of Leuven, are experienced by their lectures as being “docile, not critical, with a “tell us what to do attitude”. In general nurses conform to carrying out tasks according to protocols and decided by the doctors. Therefore it is not excluded that they might profit more from linear, structured instruction, than from free learner control. Another aspect which might undo the advantage of flexible navigation is the typical sequential nature of the skill to be learnt. The learning objectives are: to know and understand the necessary steps of a practical nursing skill and their proper sequence in the procedure of the total skill. - (Inter)active learning activities are more effective than instruction design whereby the learner only passively listens or watches a video. (Learning Piramid of Bales, constructivist learning theory)
“Interactive learning activities” are integrated in our learning module in the form of corrective feedback on “test yourself” questions used during the process of learning. The questions help to consolidate the insights learnt from watching the demonstration video and reading the textual
information. Questions in the learning module are integrated and attached to small learning sections/units.(cfr. Segmenting principle of Mayer) Students have control over which question they want to answer first, and get immediate feedback on their answer. With normal video (media design 1 & 2) we only got the possibility to add the “test yourself” questions with feedback as a separate learning object in the LMS (Blackboard). Students get only the possibility to answer the questions one after the other and they get the correct answers at the end of the test. We add the “Test yourself” questions also to the two other learning media designs in order to keep the learning activities as close to equal as possible. Clarks [7] states that “media in itself do not influence learning”, but the type of learning activity in the instructional design. Therefore if we want to isolate the effect of the media design itself, we have to keep the learning activities as close to equal as possible.
5
RESEARCH METHODOLOGY
In order to isolate the influence of the media design (MD) on learning, we compare three experimental groups of hundred students that only differ in the type of media design used. (see table 1, p.4) Experimental group 1: media design 1 (MD1) Experimental group 2: media design 2 (MD2) Experimental group 3: media design 3 (MD3) To rule out other influencing factors of the characteristics of the learning groups, we divide the students of the four participating institutions equally over the three experimental groups and we stratify them also on: type of enrolment (distance education or traditional education), prior work experience, previous education, gender, age and level of ability. The level of ability is defined by means of the st st study results of the previous semester (1 years, 1 semester) on the same practical nursing course. We use the normal distribution curve of each of the institutions to identify the high achievers, medium and low achievers. For research question 1: we compare MD1 with MD2, that only differ in elaboration of narration. For research question 2: we compare MD2 with MD3, whereby MD3 adds: integration of (segmented) learning objects, learner control and immediate character of the corrective feedback on self evaluation questions. In order to make an educational policy decision on the future use of media design based on the results of this research, the experimental learning task needs to be representative for the real education situation. Therefore we incorporate the experimental lab preparation assignment into the real education program. The day before the real practicum on the chosen skill “intramuscular injection” takes place, the students get two hours to prepare themselves to achieve the necessary insights in the procedure of the skill and to remember the steps in the right sequence. Keeping the research design both experimentally controlled as well as representative for the real educational situation is very challenging. We’ve involved all the lecturers in order to standardise the prior knowledge on the specific topic as well as on the related preparatory ones. As evaluation method to measure the effectiveness of the media designs, we conduct a written test (mainly multiple choice) at the beginning of the practicum, the next day. The test scores reflect the recall and insights in the steps of the procedure that are necessary to apply the skill correctly. A team of five subject lecturers identified these crucial insights, based on an analysis of frequent misconceptions an forgotten sub steps of the procedure observed in the students’ performance at the practical exams. It is also the same group of subject specialists who has decided on the content an narration in the video and the consolidating “test yourself” questions. The learning efficiency is measured by means of an online questionnaire that students had to fill in at the end of the lab preparation assignment. We compare the indicated time spent on the assignment and analyse if there is a difference between the three experimental groups? The online questionnaire also asks other details of the learning process: the use of the traditional written guidelines, the use of the questions with corrective feedback, the attractiveness of the media design, the pleasant way of learning, the perceived effectiveness of the learning media.
6
RESULTS
6.1 6.1.1
Learning effectiveness Research question 1: Comparing the learning results of media design 1, video with limited narration with media design 2, video with elaborate narration
The comparison (by means of a T-test) of the total test scores of the media design 1 group, (video with limited narration) with the media design 2 group, (video with elaborate narration) points out a significant difference (p=0,003) in favour of the elaborate video. The difference in mean score of the written test results is however only 0,5/10. Figure 1 shows the distribution of the test scores for the three types of media design.
Fig 1 Distribution of total written test score by media design
We can conclude that the students’ wish for more elaborate narration in the video pays off. Moreover do our analyses of the use of the written procedural guidelines (textbook) contradict the fear of the lectures that when using elaborate videos the students would not use the written guidelines anymore. In figure 2 we can see that there is almost no difference in percentage of respondents of the different media design groups in the use of the written procedural guidelines. 76% of students of the short video group say they have used the written guidelines, while also 74% of students of the elaborate video group say to have used them.
Percentage of respondents
100% 80% 60% short video
40%
elaborate video
20%
learning module
0% not used
used
Use of written procedural guidelines Fig 2 Use of written procedural guidelines by media design groups
6.1.2
Research question 2: Comparing the learning results from media design 2, (elaborate video), with media design 3, (learning module).
The group of learners using the learning module does not get higher results than the group using the elaborate video (media design 2). On the contrary, in table 2 we can see that the mean test score shows slightly lower results for the learning module group. The difference is so small, that it should be neglected. Table 2 Mean test scores by media design 1: Short video Mean test score
2: Elaborate video
5,59
3: Learning module
6,08
Total test score
5,94
/ 10
Our findings correspond with the findings of Snow and Mayer, that learner control is not necessary beneficial for all learners. It would be worth to further examine if the lecturers’ perceptions on the characteristics of this learner group of nurses, as being in general “conformist”, is being confirmed by empirical evidence. In general overlooking or bypassing information that might still be important, happens more with a flexible learning path, than with a linear structured learning path. [8] The results from our learning process questionnaire show in fig. 3 that students from the learning module (media design 3) bypass some “test yourself” questions (only 29% has made them all), while the other groups (media design 1 and 2) make significantly more “all the questions” (group 1: 79%, group 2: 67%). 90% 80% percentage of respondents
70% 60% 50% 40%
short video
30%
elaborate video
20%
learning module
10% 0% none
some
almost all
all
test yourself questions Fig 3: Use of “test yourself” questions grouped by media design
6.2
Learning efficiency
Figure 4 shows the analyses of study time spent by the different media design groups. There is no consistent difference found between the three experimental groups in time spent on the assignment.
Percentage of respondents
35% 30% 25% 20% short video
15%
elaborate video
10%
learning module
5% 0%
