Education Tech Research Dev (2012) 60:307–324 DOI 10.1007/s11423-011-9225-2 DEVELOPMENT ARTICLE
Learners’ perceptions and illusions of adaptivity in computer-based learning environments Mieke Vandewaetere • Sylke Vandercruysse • Geraldine Clarebout
Published online: 20 October 2011 Ó Association for Educational Communications and Technology 2011
Abstract Research on computer-based adaptive learning environments has shown exemplary growth. Although the mechanisms of effective adaptive instruction are unraveled systematically, little is known about the relative effect of learners’ perceptions of adaptivity in adaptive learning environments. As previous research has demonstrated that the learners’ view towards a learning environment strongly influences their learning outcomes and learning process, it can be discussed whether program-defined adaptivity is not only effective because of the underlying learner models, but also because the adaptivity is perceived and experienced as such by the learners. In this study, we apply the cognitive mediational paradigm and hypothesize that perceptions of adaptivity mediate the relation between adaptive instruction and learners’ motivations and learning outcomes. The results do not fully support the claim of the cognitive mediational paradigm. Both adaptivity and perceptions were related to motivation, but learners’ perceptions did not act as a mediating variable. Keywords Adaptive instruction Instructional perceptions Cognitive mediational paradigm Illusion of adaptivity
M. Vandewaetere (&) S. Vandercruysse G. Clarebout iTEC, Interdisciplinary Research on Technology, Communication and Education, K.U. Leuven—Kulak, E. Sabbelaan 53, 8500 Kortrijk, Belgium e-mail:
[email protected] S. Vandercruysse e-mail:
[email protected] G. Clarebout e-mail:
[email protected] M. Vandewaetere S. Vandercruysse G. Clarebout C.I.P.&T., Centre for Instructional Psychology and Technology, K.U. Leuven, Dekenstraat 2, 3000 Leuven, Belgium
123
308
M. Vandewaetere et al.
Introduction Since the advent of computers, a revolution in technology-based learning has occurred and research on computer-based adaptive learning environments has shown exemplary growth (Graesser et al. 2008). Although the mechanisms of effective adaptive instruction are unraveled systematically, little is known about the relative effect of learners’ perceptions of adaptivity in adaptive learning environments. As previous research has demonstrated that the learners’ view towards a learning environment strongly influences their learning outcomes and learning process, it can be discussed whether program-defined adaptivity is not only effective because of the underlying learner models, but also because the adaptivity is perceived and experienced as such by the learners. In this study, we apply the cognitive mediational paradigm and hypothesize that perceptions of adaptivity mediate the relation between adaptive instruction and learners’ motivations and learning outcomes. Approaches to adaptive instruction Learning environments are called adaptive when they are able to alter their behaviour (i.e., feedback, content, presentation, etc.) according to learner characteristics (Shute and Zapata-Rivera 2008). Research has demonstrated that, when instruction is adapted or accommodated to learners’ skills, needs, beliefs or knowledge, learners achieve learning goals more efficiently (for reviews see Cohen et al. 1982; Federico 1999; Kadiyala and Crynes 1998; Kulik et al. 1990). Examples of successful adaptive learning environments have, among others, been reported in literature on hypermedia and multimedia (e.g., Brusilovsky 2007), and intelligent tutoring systems (Anderson et al. 1995). Despite the accepted effectiveness of adaptive systems (for a review, see Vandewaetere et al. 2011), there is less consensus on what is worth to adapt for (e.g., cognitive style or learning style), and which individual learner characteristics should be taken into consideration when developing learner models for adaptive systems (Brusilovsky and Milla´n 2007). Typically, adaptation can occur on three learner parameters or a combination of them (Vandewaetere et al. 2011): cognition (such as prior knowledge or cognitive style), affect (such as motivation or frustration) and behaviour (such as number of attempts, need for feedback). Intelligent tutoring systems, for example, are able to adapt content and presentation based on errors and misconceptions (Mitrovic et al. 2003), while other research uses self-reports on mood to adapt instruction (Beal and Lee 2005). Adaptation can thus be unidirectional and system-controlled with the learner having no control on what is adapted, when and how it is adapted. Yet, adaptation can also start from the learner, with the learner (partly) controlling instruction and deciding what, how and when to adapt based on her own needs and interests (for an example, see Corbalan et al. 2009). In system-controlled or instructor-led adaptive systems all learners follow a predefined and fixed learning path that is defined by the instructor (Lawless and Brown 1997). Such systems are usually developed for novices as intended learners (Kalyuga 2008) and have major recognized benefits for learning. Yet, there are some drawbacks too. As a result of externally adapting instruction to the learner, adaptive systems have been developed without or with only sparse interaction between learner and environment. Learners do not know which of their actions result in reactions of the system and because of the one-way adaptation from instructor to learner, they are more prone to becoming dependent of prestructuralized instruction (Elen 2000). A second drawback of such systems is that adaptivity as such is not always recognized by the learner and that learners do not optimally make sense of using adaptive learning environments as tools (Kay 2001).
123
Learners’ perceptions and illusions
309
Prerequisites for effective instructional interventions Instead of viewing learners as passive recipients of (adapted) instruction, more recent views consider learners as active learners and stress the role of (meta)cognitive knowledge in learning (Gerjets and Hesse 2004; Lowyck et al. 2004; Struyven et al. 2008; Winne 2004). Learners are considered as active, self-regulating participants in the learning process and learners’ perceptions of learning influence how they learn (Entwistle 1991). As such, even in system-controlled adaptive learning environments, in which learners follow a predefined learning path, learners still have control over the cognitive operations they want to apply and how much effort they want to spend to parts of the instructional environment (Winne 2004). Instruction should therefore not be considered as a cause of learner behaviour, or learning outcomes, as Winne (2004) suggested. Rather, it is the learners’ self-regulation and perception that affects the effectiveness of instructions. Reformulated, there is a firm interaction pattern between learners’ perceptions and self-regulation skills, and learning outcomes. For instance, Winters, Greene, and Costich (2008) found evidence for the relation between self-regulation skills and having control or not. For learners with low self-regulation skills, working in a system-controlled adaptive learning environment is more beneficial compared to a learner-controlled environment (Eom and Reiser 2000). In line with this, it can be hypothesized that learners with high self-regulation skills perceive adaptive instruction differently as compared to learners with low self-regulation skills. The importance of learners’ conceptions and perceptions when studying the relationship between interventions and learning has been stressed in the cognitive mediational paradigm (Winne 1987; Luyten et al. 2001). In order to have effective instructional interventions, according to the cognitive mediational paradigm, several conditions and subconditions must be met (Martin 1984). First, the instructor’s behaviour (i.e., instructions) must be consistent with the intentions and purposes of learning. This means, for example, if an instructor expects a particular behaviour from the learners (e.g., asking for support), the instructor should use techniques to activate or stimulate this learner’s behaviour. Second, the learner must be willing to actively process the behaviour as intended by the instructor. This means that (1) the learner must attend to the cues given by the instructor, (2) the learner must correctly perceive the intentions of the instructor or environment, (3) the learner must be capable or must have the skills to execute the tasks or processes as intended by the instructor, and (4) the learner should have sufficiently high motivation to mediate the relation between instruction and learning outcomes. Next to consistent instructions and the learner’s cognitions, a third condition for effective cognitive mediation is the outcome of learning. Learning outcomes should accurately reflect the successful cognitive processing of the learner. Based on the cognitive mediational paradigm, we can predict that instructional interventions will be effective to the extent that the aforementioned conditions and subconditions are met. In their overview article, Lowyck and colleagues discuss research findings related to the mediating effect of instructional conceptions and perceptions on the effects of learning environments (Lowyck et al. 2004). Instructional interventions are always interpreted by learners, and in turn this interpretation influences the effects of the interventions (Struyven et al. 2008). The effect of instruction as such can be largely surpassed by the consequences of learners’ perceptions, knowledge and motivation. Applying the cognitive mediational paradigm to computer-based instruction has resulted in several guidelines for instructional design. In order for learners to optimally benefit from instruction and to interpret instruction, they must perceive it as such (Corbalan et al. 2009). Learners should also have knowledge about the intervention (Elen and
123
310
M. Vandewaetere et al.
Clarebout 2006) and should be sufficiently motivated (Perkins 1985). Not only do learners’ recognition, their perceptions, their knowledge and motivation influence the interpretations made by learners, interpretations can also vary across learners (Winne 2006). Winne also states that, if perceptions about how to participate in the learning process indeed differ between learners, the measurements of direct relations between factors of instruction (e.g., learner control or not, adaptive or not) and learning outcomes (e.g., higher learning, faster learning) are overridden (Winne 2006). This statement can be considered as a serious call for research taking into account the differences in perceptions and their influence on learning activities and learning outcomes. Indeed, learners’ perceptions must be considered as essential to an understanding of learning (Struyven et al. 2008). Summarizing, the cognitive mediational paradigm stresses the mediating role of learners’ perceptions and cognitions about learning and instruction. Learners’ perceptions about instructional interventions are related to the learner’s behaviour in a learning environment (Clarebout and Elen 2006), which in turn affects learning outcomes. With respect to technology-enhanced learning, Shuell and Farber (2001) concluded that the degree to which technology affected learning was more a function of learners’ perceptions, rather than of instructor’s use. More specific for hypermedia research, Gerjets and Hesse (2004) provided an overview on how learners’ perceptions and conceptions of educational technology might moderate the relationship between instructional design and learning outcomes. Not only are learner’s perceptions related to learner behaviour and learning outcomes, also motivation is assumed to be related with perceptions. Enhancing intrinsic motivation can be done by providing a higher perception of relevance to learners (Kinzie 1990). This is in fact the core notion of adaptive instruction: when instruction is adapted to the learner’s needs, motives or values, instruction will be intrinsically motivating (Keller 1983). Learners that perceive an environment as relevant to their needs will show greater intrinsic motivation. This was also demonstrated by Cordova and Lepper (1996), who showed that personalization led to increases in learners’ motivation, but also in their learning outcomes. Concluding, instead of focusing on the direct relations between instructional interventions and learning outcomes, the mediating role of learners’ perceptions can be investigated. The way learners perceive instruction and behave accordingly mediates the relation between instruction and outcomes. Applying this to the research on effectiveness of adaptive systems the following question arises: is an adaptive system effective because it is adaptive, or is such a system also or more effective because learners perceive it as adaptive? The present study While research on adaptive learning environments is still flourishing, the relative effect of perceptions of adaptivity has to our knowledge not been sketched. The cognitive mediational paradigm stresses that instructional effects, such as adaptivity, can be largely influenced by the learners’ perceptions. Learners interpret the instructional interventions and this in turn influences learning behaviour and learning outcomes. Reformulated, adaptivity as instructional intervention is interpreted by learners, and these interpretations will affect learning behaviour and learning outcomes such as course score and motivation. In turn, learners’ interpretations are hypothesized to be related to self-regulation skills. As Winne (1995) stated: self-regulation skills cannot be turned on or off. A learner’s representation is one of the primary factors determining whether self-regulation skills will benefit or hamper learning (Winne 2004).
123
Learners’ perceptions and illusions
311
In this study, we hypothesize that perceptions of adaptivity (experimentally induced by instructions) are related to motivation and learning outcomes in an adaptive learning environment (Hypothesis 1: The effect of adaptivity and instruction on motivation is mediated by perception of adaptivity; Hypothesis 2: The effect of adaptivity and instruction on learning outcomes is mediated by perception of adaptivity). Additionally, in line with Entwistle (1991), stating that learners interpret the instructional interventions and context, which in turn influences the learning process; and with the argumentation of Dror (2008) and Cordova and Lepper (1996), we also hypothesize that even the mere illusion of adaptivity (i.e., when learners perceive an environment as adaptive while it is not) can provide a means for improving intrinsic motivation and learning outcomes. As such, perception of adaptivity is entered as mediating variable when testing the hypotheses. In addition, since learners with low self-regulation skills benefit more from system-controlled adaptivity, we want to sketch how those skills are involved in the relation between learners’ perceptions and learning outcomes (Research question: In what way are selfregulation skills involved in the effects of perception and illusion?).
Method Participants One hundred twenty-two high school students participated in this study, as part of their English class. Two English teachers had confirmed participation in this study and their students were selected from the 11th and 12th grades of high school. In Flanders (Belgium), students follow compulsory English courses from the 8th grade on, so all participants followed English class 2 h a week. All participants already had some experience with English, which was necessary for learning the materials provided for this study. The age range was between 16 and 19 years. Sixty-nine females participated in this study (56%). Participants that did not complete the whole study (i.e., no post-test available or post-experimental questionnaire) were discarded from the analyses. This resulted in 99 participants having data on all measured variables. Design We adopted a 2 (adaptive or not) 9 2 (instruction of adaptivity or not) factorial betweensubjects design. This resulted in four conditions: in the first two conditions (both defined as non-adaptive) all participants received the same set of items focusing on English tenses. The last two conditions are adaptive, in that the administered items had previously been selected based on participants’ prior knowledge (as measured before the start of the experiment by means of a 15-item pre-test). Prior knowledge was defined as high, intermediate or low. Dependent on their prior knowledge, participants in the adaptive conditions received a course of easy, intermediate or difficult level. Next to the distinction adaptive versus non-adaptive, we also created two types of instruction. Either participants received instructions that they would work in a learning environment that was adapted to their prior knowledge, or they received no instructions of adaptivity. For an overview of the instructions, see Table 1. Summarizing, the four conditions are: (1) NoAdapNoIns: non-adaptive, default instruction; (2) NoAdapInst: non-adaptive, default instruction with additional instruction of adaptivity; (3) AdaptNoInst: adaptive, default instruction; and (4) AdapInst: adaptive,
123
312
M. Vandewaetere et al.
Table 1 Instructions per condition Course
Condition
Instructions
Nonadaptive
No instruction of adaptivity
NoAdapNoInst
Please complete these sets and questions. All questions in this course are the same for all users
Nonadaptive
Instruction of adaptivity = illusion
NoAdapInst
Please complete these sets and questions. The questions in this course have been selected based on your prior knowledge, i.e., based on the score you obtained on the pre-test previous to this session
Adaptive
No instruction of adaptivity
AdapNoInst
Please complete these sets and questions. All questions in this course are the same for all users
Adaptive
Instruction of adaptivity = perception
AdapIns
Please complete these sets and questions. The questions in this course have been selected based on your prior knowledge, i.e., based on the score you obtained on the pre-test previous to this session
default instruction with additional instruction of adaptivity. In condition 2 we hence created the illusion of adaptivity, while in condition 4, the perception of adaptivity is enhanced compared to condition three. Condition 1 served as control condition. Materials Item-based learning environment We used an item-based electronic language learning environment (adapted from the open source content management system Moodle, www.moodle.org), from which the course screen is represented in Fig. 1. In the environment, students were enrolled in one of the adaptive or non-adaptive courses and were asked to go through these courses. All courses contained items related to English tenses and verb conjugation. Items that focused on the same context (e.g., writing a letter, having a telephone conversation) were grouped into one item set. As such, courses comprised seven or eight item sets and participants were allowed to freely choose the order in which they completed the item sets. Participants could also request correct answer feedback and were allowed to consult additional instructional materials such as a general overview. This general overview presented a short introduction on the form, function and use of English tenses and was available as a pop-up screen. The whole study, including pre-test and post-test, was administered in this environment. Tracking and logging data were collected and stored on an external server. Afterwards, these data were converted into analytic data sets, ready for statistical analyses. Figure 1 presents the main course screen. The first topic (marked with A) presents the additional instructional materials and item sets. The second topic (marked with B) comprises the post-test. In Fig. 2, the screen for the first item set is presented. This first item set comprises four questions or items, all multiple choice, of which three are shown in this figure. With respect to the design of the learning environment, it should be noted that the environment used in this study, focuses more on assessment for learning as instructional
123
Learners’ perceptions and illusions
313
Fig. 1 Course screen from the learning environment: main course page
Fig. 2 Course screen from the learning environment: main course page
strategy rather than on learning as such. Following the taxonomy of Thelwall (2000), the learning environment in this study can be categorized into the area of formative assessment, comprising exercises that aim to consolidate learning. The emphasis on learning in this study was provided by the availability of the general and tense specific overviews and by providing feedback on learners’ responses, the latter being a key element of assessment for learning (Timmers and Veldkamp 2011). Introduction and pre-test The introduction and pre-test served three goals: first, by giving an introduction in the learning environment we wanted to familiarize the participants with the user interface and
123
314
M. Vandewaetere et al.
its possibilities, and second, we wanted to activate prior knowledge of participants (Merril 2002). The third goal was to define the prior knowledge level of the student (based on the pre-test). The introduction consisted of an overview of the most prominent tenses in English: future, past and present, simple and progressive. The overview of tenses comprised a theoretical overview of the function and use of each tense, exception rules and some practical worked-out examples. The school-like pre-test contained 15 multiple-choice items, with only one correct alternative possible. The test was not time-limited. The mean of the pre-test scores is 11.25 with a standard deviation of 2.34. Minimum and maximum test scores are 5 and 15, respectively. The reliability of the test, as measured by the KR-20 formula is a = .58. Based on the score obtained in the pre-test, only the participants in the adaptive learning conditions were assigned to one of the three courses that were adapted to their prior knowledge level. Seventeen participants scored below the 25th percentile of the pre-test and were assigned to the course with lowest difficulty, forty participants had pre-test scores lying between the 25th and 75th percentile and were thus assigned to the course with intermediate difficulty, while the reminder nineteen participants, who scored above the 75th percentile, were assigned to the course with highest difficulty level. Assessment for learning phase In the two non-adaptive conditions (non-adaptive with default instruction; and non-adaptive with additional instruction of adaptivity), all participants received the same course containing items of varying difficulty level. In the two adaptive conditions (adaptive with default instruction; and adaptive with additional instruction of adaptivity), participants received a course that was adapted to their prior knowledge level, as defined by the score on the pre-test. Item difficulty level was based on the categorization of the Common European Framework of Reference for Languages (CEFR; Council of Europe 2001). This framework involves three levels of language proficiency (A: basic speaker; B: independent speaker; and C: proficient speaker). Within each level, two sublevels are defined (e.g., A1: breakthrough or beginner; C2: Mastery or advanced). First, the items developed for this study were categorized into the six levels of the CEFR. In a next step, the English proficiency level of the participants was taken into account, based on the information that the participants’ teachers gave. Finally, the items were labelled as easy level, intermediate level, or difficult level, taking the participants’ English proficiency and the CEFR-categorization of the items into account. For instance, suppose that a participant is assigned to an adaptive condition. The participant obtained a pre-test score below the 25th percentile which indicates low English proficiency level. Hence, this participant was administered an adaptive course comprising a major part of easy items and a smaller amount of items of intermediate level. As a counterexample, suppose a participant’s pre-test score is above the 75th percentile. Hence this participant’s English proficiency level is among the highest of the group, and this participant was administered an adaptive course comprising a major part of difficult items, with a very small amount of items of intermediate level. Adaptation in this study was therefore not dynamic or run-time but rather static as it was defined before the participant entered the course (Vandewaetere et al. 2011). Items were presented as multiple choice questions (one correct alternative), focusing on recognition and reproduction. In the learning environment, participants could request feedback on their answer; they could freely navigate between the item sets and choose the order of completion. All items were binary scored (zero for wrong answer, one for correct answer).
123
Learners’ perceptions and illusions
315
Table 2 presents an overview of the items and item sets per course and per condition. The course for the low prior knowledge participants contained 31 items (18 of easy level, 13 of intermediate level) and the items were grouped into eight item sets. The course of intermediate and difficult level both contained 29 items. In the intermediate level course, three items were of easy level, 21 of intermediate level and five of difficult level. The items were organized in seven item sets. The difficult level course comprised eight items sets, and had nine items of intermediate level and 20 items of difficult level. The different amount of items between the courses is due to the fact that items had been grouped according to their context (e.g., having a telephone conversation) and that, in order to maintain the applicability of contexts in one item set of the easy-level course, some additional items were necessary. Test phase—post-test After the learning phase, all participants received a new set of 15 multiple choice items, with equal content for all conditions. This post-test was not time-limited and has a reliability (as measured by the KR-20 formula) of a = .54. The post-test has a mean value of 8.70, with a standard deviation of 2.60 and a minimum and maximum post-test score of 2 and 14, respectively. Since the mean value of the post-test is significantly lower than the mean value of the pre-test, the validity of this post-test cannot be assured. Hence, for the remainder of this chapter, post-test scores will not be involved in the analyses. Instead, a learner’s course score, i.e., the average score on all completed items during the learning phase, is used as learning outcome. Motivation and self-regulation questionnaire All participants completed rating scales related to their self-regulation after finishing the pre-test. Therefore we adapted the Learning Self-Regulation Questionnaire (SRQ-L; Black and Deci 2000; Williams and Deci 1996) so that this questionnaire would be applicable to language learning. Two subscales were administered: Autonomous Regulation (5 items) and Controlled Regulation (7 items). All items were measured on a seven-point rating scale. Reliability of the two subscales resulted in values a = .79 for the Autonomous
Table 2 Overview of participants, course materials and items per course Course
N
Course level
Number of item sets (items)
No instruction of adaptivity
15
Intermediate
7 (29)
Instruction of adaptivity = illusion
20
Intermediate
7 (29)
25
Easy
8 (31)
Intermediate
7 (29)
Difficult
8 (29)
Non-adaptive
Adaptive No instruction of adaptivity
Instruction of adaptivity = perception
39
Easy
8 (31)
Intermediate
7 (29)
Difficult
8 (29)
123
316
M. Vandewaetere et al.
Regulation subscale and a = .72 for the Controlled Regulation subscale. Correlation between the two subscales is positively significant (r = 0.38, p \ .01). Post-experimental motivation was measured after the learning phase. To assess the postexperimental motivation, and in line with the selection made by Corbalan et al. (2009), we selected four relevant subscales from the Intrinsic Motivation Inventory (IMI; McAuley et al. 1987; Plant and Ryan 1985). The four subscales are Interest/Enjoyment (7 items; a = .87); Perceived Competence (6 items; a = .91); Effort/Importance (5 items; a = .86); and Value/Usefulness (7 items; a = .85). All items were measured on a seven-point rating scale. Reliability of the post-experimental motivation questionnaire (as measured by the four subscales) is a = .91. As such, a participant’s post-experimental motivation was operationalised as the sum of the scores on the four subscales. Tracking and logging data Tracking and logging data were stored on an external SQL-server. Information that was extracted from this server was merged into one data file, ready for statistical analyses. This data file contained time spent while in the learning phase, time spent while in the test phase, number of attempts in the learning phase and score per item as measured by the first attempt and whether this attempt was correct (score = 1) or incorrect (score = 0). Based on the results of log file statistics, no participants had to be removed because of extreme values. Procedure Pre-experimental phase and learning phase Participants first received a short introduction on English tense conjugation and then completed the pre-test of 15 items. After completing the pre-test, self-regulated learning strategies were measured (SRQ-L; Black and Deci 2000; Williams and Deci 1996) and learners were subsequently directed to the learning environment. Participants were randomly assigned to one of the four conditions. In the two adaptive conditions learners were directed towards the most appropriate course dependent on their pre-test score (easy, intermediate, or difficult level). Participants in the two non-adaptive conditions received the same set of items (intermediate level), irrespective of their score on the pre-test. In the learning phase, all participants but one1 completed all items in the course they were assigned to. Items that were closely relevant (by means of goal and context) were grouped into item sets. No time limits were communicated to the participants although time was restricted to 50 min (the duration of one class hour). Previous tests had confirmed that all the items could be completed within this duration. Test phase—questionnaires and post-test After the learning phase, all participants received the selected subscales of the post-experimental motivation questionnaire (IMI; McAuley et al. 1987; Plant and Ryan 1985). After completing the questionnaires, the post-test was started and participants received one additional question, to be rated on a ten-point Likert scale. This question focused on the perception of adaptivity while working in the learning environment (‘‘To what extent do you consider these exercises as adaptive to your knowledge?’’) and was rated by all participants. 1
One participant did not answer one item.
123
Learners’ perceptions and illusions
317
Results For all analyses, a significance level of a = .05 was set. Different levels of participants’ prior knowledge were equally represented over conditions (F(3, 108) = .91, ns). Equality of conditions for controlled and autonomous regulation as separate constructs was also checked. A MANOVA, with the two subscales of the SRQ-L as dependent variables, showed that there was again no significant difference between the four conditions related to the scores on autonomous regulation and controlled regulation as measured before the start of the learning phase (Wilk’s k = 0.93, F(6, 224) = 1.40, ns). Table 3 gives an overview of the mean scores per condition (standard deviations between brackets) of all dependent variables measured in this study. Effect of instruction and adaptivity on motivation Two factors (adaptive or not; instruction of adaptivity or not) were entered in a model with motivation as dependent variable. There was a main effect of adaptivity (F(1,95) = 7.51, p = .007, g2p = .07) and a main effect of instruction (F(1,95) = 6.06, p = .016, g2p = .06). The interaction between adaptivity and instruction was also significant (F(1,95) = 7.08, p = .009, g2p = .07). More specifically, all courses resulted in equally high post-experimental motivation except for the non adaptive course in which no instruction of adaptivity was offered (control group). This interaction is graphically presented in Fig. 3. Based on the significant interaction between condition and instruction, both factors were merged into one factor with four levels in order to test for mediation of learners’ perceptions. Hypothesis 1: The effect of adaptivity on motivation is mediated by perception of adaptivity To test for mediation, the four-steps approach outlined by Baron and Kenny (e.g., Baron and Kenny 1986; Frazier et al. 2004) is used. The model to test is graphically represented in Fig. 4. Condition is considered as independent variable X, and motivation is the Table 3 Means (standard deviations) of all measured variables, represented per condition Non adaptive No instruction
Adaptive Instruction = illusion
No instruction
Instruction = perception
Pre-experimental Prior knowledge test (in %)
68.89 (14.62)
75.20 (15.69)
76.13 (14.38)
76.26 (16.67)
SRQ-L autonomous regulation
16.69 (4.32)
18.78 (3.20)
18.72 (3.05)
17.90 (3.28)
SQR-L controlled regulation
23.31 (4.92)
24.00 (6.15)
25.03 (4.00)
22.57 (5.23)
Learning outcomes Course score (in %)
59.07 (15.08)
57.25 (13.41)
60.85 (10.69)
60.27 (12.91)
Time on course (in minutes)
13.95 (3.14)
13.79 (3.55)
13.73 (3.97)
14.41 (3.38)
69.73 (17.29)
85.35 (13.75)
86.20 (13.28)
85.59 (13.92)
5.67 (1.99)
5.85 (1.60)
7.20 (1.58)
6.26 (1.77)
Post-experimental motivation (IMI) Perceptions Perceived adaptivity (max. = 10)
123
318
M. Vandewaetere et al.
Fig. 3 Interaction between adaptivity and instruction on motivation
Fig. 4 Mediation model with perceived adaptivity as hypothesized mediator
dependent variable Y. Perceived adaptivity (M) is hypothesized to be mediating the relation between condition and motivation. First, the direct effect (path c) of condition on motivation was tested by an ANOVA with condition as factor (four levels) and the total factor score on the post-experimental motivation questionnaire (IMI) revealed a significant effect of condition (F(3, 95) = 5.33, p = .002, g2p = .14). There was a conspicuous decrease in motivation for the condition in which there was no adaptivity and no instruction of adaptivity (M = 69.73, SD = 17.28). For the other conditions, there is no significant difference between perception of adaptivity (i.e., adaptivity with instruction; M = 85.59, SD = 13.92), adaptivity without instruction
123
Learners’ perceptions and illusions
319
(M = 86.20, SD = 13.28) and illusion of adaptivity (i.e., adaptivity with instruction; M = 85.35, SD = 13.75). Next, the relation between condition and the mediator, perceived adaptivity is tested (path a). Therefore, condition was entered as a factor, and perceived adaptivity was the dependent variable. Results show that perceived adaptivity was significantly related to condition, (F(3, 95) = 3.39, p = .021, g2p = .10). Perception of adaptivity was the highest for the adaptive courses without additional instruction of adaptivity (M = 7.20, SD = 1.58), then followed by adaptive courses with instruction of adaptivity (M = 6.26, SD = 1.77) and the regular courses with and without instruction of adaptivity (M = 5.85, SD = 1.60, and M = 5.67, SD = 1.99, respectively). This is graphically represented in Fig. 5. In the third step, the relation between condition and motivation is tested, but controlling for perceived adaptivity. As such, it is tested whether the mediator, perceived adaptivity, is related to motivation (path b) and what the relation is between condition and motivation, controlling for the mediator (path c’). An ANCOVA was done with condition as factor, motivation as dependent and perceived adaptivity as covariate. Condition was again related with motivation (F(3, 94) = 4.63, p = .005, g2p = .13). Also, perceived adaptivity was significantly related with motivation (F(1, 94) = 14.56, p \ .001, g2p = .13) The higher the perceived adaptivity, the higher the motivation (b = 3.03). The results of this analysis suggest a partial mediation effect of perceived adaptivity on the relation between condition and motivation. To test the significance of the mediation effect, the Sobel test was used resulting in a value of z = .31, ns. As such, it could not be confirmed that learners’ perceptions mediate the relation between condition and motivation.
Fig. 5 Effect of condition (four levels) on perception of adaptivity
123
320
M. Vandewaetere et al.
Hypothesis 2: The effect of adaptivity and instruction on learning outcomes is mediated by perception of adaptivity Learner performance was measured by the variables learning time and course score. As explained earlier, the post-test scores are discarded from further analyses because the test’s reliability is below satisfaction and the post-test scores are lower compared to the pre-test scores. Course score This score was computed as the mean of all item scores, completed during the learning phase, for each participant. A model was specified with two factors (adaptive or not; instruction of adaptivity or not) and with motivation as dependent variable. There was no main effect of adaptivity (F(1,89) = 2.56, ns) and instruction (F(1,89) = .11, ns). The interaction was also not significant (F(1,89) = .29, ns). Learning time Learning time is defined as the total time spent in order to complete the items. The timeon-course ranges from about 6 min to 24 min, with M = 14 min and SD = 3.50 min. An ANOVA with two factors (adaptivity and instruction) and learning time as dependent, again revealed that adaptivity (F(1,87) = .06, ns), instruction (F(1,87) = .11, ns) and their interaction (F(1,87) = .27, ns) were not significantly related with learning time. Because both instruction and adaptivity were not significantly related with learning outcomes, the test for mediation by learners’ perceptions was not performed. Research question: in what way are self-regulation skills involved in the effects of perceptions and illusion? Self-regulation skills were measured by the Autonomous Regulation and Controlled Regulation subscales of the SRQ-L (Black and Deci 2000; Williams and Deci 1996). Based on the research of Eom & Reiser (2000) who found that learners with low self-regulation skills benefit more from working in adaptive learning environments compared with not adaptive learning environments, it was expected that different degrees of self-regulation skills would be related to perception of adaptivity and motivation. Therefore, two separate ANCOVA’s were performed with self-regulation skills (autonomous regulation and controlled regulation) as covariates and either with perception of adaptivity or with motivation as dependents. Neither autonomous regulation (F(1, 95) = .79, ns), nor controlled regulation (F(1, 95) = .03, ns) was significantly related with perceived adaptivity. Participants with different levels of self-regulation skills did not perceive adaptivity differently. Related to motivation, only autonomous regulation significantly affected a participant’s post-experimental motivation (F(1, 93) = 22.04, p \ .001, g2p = .19). The higher the autonomous regulation, the higher the post-experimental motivation (b = 2.13). Controlled regulation did not show an effect on motivation (F(1,93) = .03, ns).
123
Learners’ perceptions and illusions
321
Discussion This study focused on the influence of learners’ perceptions and illusions of adaptivity and their effects on motivation, learning behaviour and learning outcomes. By an experimental design it was tested whether either adaptivity, instruction, or the combination of both would result in better learning outcomes. A 2 by 2 between-subjects design was applied, resulting in four conditions: (1) no adaptive course, with regular instruction; (2) no adaptive course with instruction of adaptivity (i.e., illusion of adaptivity); (3) adaptive course with regular instruction; and (4) adaptive course with instruction of adaptivity (i.e., enhanced perception of adaptivity). Results indicated that both adaptivity and instruction were related with post-experimental motivation. Also, conditions differed to the extent of participants’ perceived adaptivity. However, the effects were not in line with the expectations. It was hypothesized that working in an adaptive course would result in higher motivation and that instruction of adaptivity would also result in higher motivation. To control for mediating effects of the participants’ perceptions of adaptivity, a mediation analysis was done. However, a mediating role of perceptions, as suggested by the cognitive mediational paradigm, could not be confirmed. Related to learning outcomes, no effects of adaptivity and instruction were found in this study. As a consequence, some remarks should be made with respect to the design and implementation of this study. A first remark is that learners’ expectations, more than their ad hoc perceptions of adaptivity, influence the effectiveness of instructional interventions. Research has demonstrated that learners’ expectations influence perceptions of instruction, and in turn affect learning behaviour (Ko¨nings and Kirschner 2010). It might be the case that learners in the adaptive condition with additional instruction of adaptivity (i.e., perception of adaptivity) had higher expectations towards the learning environment compared with learners in the adaptive condition without additional instruction of adaptivity. For the participants in the adaptive with instruction condition, their expectations towards the environment might not have matched their actual perception of adaptivity, hence lowering their motivation towards the level of learners in the adaptive condition without additional instruction. Consequently, a warning for too high learner expectations, caused by instruction, is at its place here. Future research should investigate whether providing learners with information on instructional interventions might be a double-edged sword: beneficial for motivation in general, and for interest, enjoyment and perceived competence more specific, but also with the risk of causing too high expectations in learners and the consequences of this. Second, no effect of condition was found on learning outcomes. One possible reason is that learning time, although learner-paced, was rather the same for all learners since they tried to complete the items within one course hour of 50 min. In turn, learning time is probably too short due to the setting of this study (one class hour, time-limited) and the learning content (an extensive rehearsal of English tenses) might have been too various and elaborated to deal with by a limited amount of items and item sets. Also, the participants in this study were not accustomed to learn English in a computer-based learning environment. As a consequence, learners’ attitudes towards technology and computer-based learning might have influenced the results of this study. Future research, especially research situated in language learning, should focus on curriculum-embedded studies, smaller learning content and learning goals in order to maximize learning gains. Three last limitations of this study concern the design. First, a baseline motivation was not measured. This makes it somewhat difficult to compare motivational gains or losses
123
322
M. Vandewaetere et al.
that can be attributed to the intervention. Although complete randomization was used to strive towards equal motivation in all conditions, measuring the pre-experimental motivation would have been an advantage. Second, the pre- and post-test scores had unsatisfactory reliability. Above this, the post-test scores were lower compared to the pre-test scores. The post-test items were selected from a category of items with higher difficulty level compared to the items of the pre-test. Yet no conclusions could be made whether or not learning took place, since the pre and post-test were not of equal difficulty degree and did not meet the requirements for good reliability. Consequently, post-test scores were not included in the analyses and learning outcomes were measured by the learners’ course scores and time on course. A concluding remark is that although half of the participants received a non adaptive course, adaptivity by chance is still possible. Even in non adaptive courses the content, or a small part of it, might have matched the learner’s prior knowledge, and might thus have been adaptive. Future research should take this methodological issue into account when distinguishing between adaptive and non-adaptive learning environments. Also, careful experimentation with balanced designs is required when testing mediation effects of learners’ perceptions. To summarize, some possible implications of this research are discussed. As the importance of perception in and about learning is becoming established, this has consequences for teacher discourse and instructional design. Implementing instructional interventions should ideally always be accompanied by information for the learners, in order to establish their perception of the intervention. Simply offering adaptive learning environments to learners with the provision of information on how and why might be more beneficial than instructional interventions without additional information. Moreover, we suggest that learners’ perceptions about the instructional intervention learners are confronted with, are as important as the instructional intervention as such. Perhaps informing the learner is doing already half the work in establishing positive learning conditions to maintain or enhance motivation, and hence learning outcomes.
References Anderson, J. R., Corbett, A. T., Koedinger, K. R., & Pelletier, R. (1995). Cognitive tutors: Lessons learned. The Journal of the Learning Sciences, 4(2), 167–207. Baron, R. M., & Kenny, D. A. (1986). The moderator-mediator variable distinction in social psychological research: Conceptual, strategic and statistical considerations. Journal of Personality and Social Psychology, 51, 1173–1182. Beal, C. R., & Lee, H. (2005) Creating a pedagogical model that uses student self reports of motivation and mood to adapt ITS instruction. In Workshop on motivation and affect in educational software, at AIED2005. 12th international conference on artificial intelligence in education, Amsterdam, pp 39–46. Black, A. E., & Deci, E. L. (2000). The effects of instructors’ autonomy support and students’ autonomous motivation on learning organic chemistry: A self-determination theory. Science Education, 84, 740–756. Brusilovsky, P. (2007). Adaptive navigation support. In P. Brusilovsky, A. Kobsa, & W. Nejdl (Eds.), The adaptive web (pp. 263–290). Heidelberg: Springer. Brusilovsky, P., & Milla´n, E. (2007). User models for adaptive hypermedia and adaptive educational systems. In P. Brusilovsky, A. Kobsa, & W. Nejdl (Eds.), The adaptive web (pp. 3–53). Heidelberg: Springer. Clarebout, G., & Elen, J. (2006). Tool use in computer-based learning environments: Towards a research framework. Computers in Human Behavior, 22, 389–411. Cohen, P. A., Kulik, J. A., & Kulik, C. L. C. (1982). Educational outcomes of tutoring: A meta-analysis of findings. American Educational Research Journal, 19, 237–248.
123
Learners’ perceptions and illusions
323
Corbalan, G., Kester, L., & van Merrie¨nboer, J. J. G. (2009). Combining shared control with variability over surface features: Effects on transfer test performance and task involvement. Computers in Human Behavior, 25, 290–298. Cordova, D. I., & Lepper, M. R. (1996). Intrinsic motivation and the process of learning: Beneficial effects of contextualization, personalization, and choice. Journal of Educational Psychology, 88, 715–730. Council of Europe. (2001). Common European framework of reference for languages: Learning, teaching, assessment. Cambridge: Cambridge University Press. Dror, I. E. (2008). Technology enhanced learning: The good, the bad, and the ugly. Pragmatics & Cognition, 16, 215–223. Elen, J. (2000). Technologie voor en van het onderwijs. Een inleiding in onderwijstechnologische inzichten en realisaties [Technology for and of education. An introduction in educational technology insights and realisations]. Leuven: ACCO. Elen, J., & Clarebout, G. (2006). The use of instructional interventions: Lean learning environments as a solution for a design problem. In J. Elen & R. E. Clark (Eds.), Handling complexity in learning environments: Research and theory (pp. 185–200). Amsterdam: Elsevier. Entwistle, N. (1991). Approaches to learning and perceptions of the learning environments: Introduction to the special issue. Higher Education, 22, 201–204. Eom, W., & Reiser, R. A. (2000). The effects of self-regulation and instructional control on performance and motivation in computer-based instruction. International Journal of Instructional Media, 27, 247–260. Federico, P.-A. (1999). Hypermedia environments and adaptive instruction. Computers in Human Behavior, 15, 653–692. Frazier, P. A., Tix, A. P., & Baron, K. E. (2004). Testing moderator and mediator effects in counselling psychology research. Journal of Counseling Psychology, 51(1), 115–134. Gerjets, P., & Hesse, F. W. (2004). When are powerful learning environments effective? The role of learning activities and of students’ conceptions of educational technology. International Journal of Educational Research, 41, 445–465. Graesser, A. C., Chipman, P., & King, B. G. (2008). Computer-mediated technologies. In J. M. Spector, M. D. Merril, J. J. G. van Merrie¨nboer, & M. P. Driscoll (Eds.), Handbook of research on educational communications and technology (3rd ed., pp. 211–224). New York, NY: Taylor and Francis. Kadiyala, M., & Crynes, B. L. (1998). Where’s the proof? A review of literature on effectiveness of information technology in education. In 28th annual frontiers in education (FIE’98), Tempe, USA, pp. 33–37. Kalyuga, S. (2008). Implications of expertise reversal effect for adaptive multimedia learning. In J. Luca & E. Weippl (Eds.), Proceedings of world conference on educational multimedia, hypermedia and telecommunications 2008 (pp. 4167–4174). Chesapeake, VA: AACE. Kay, J. (2001). Learner control. User Modeling and User-Adapted Interaction, 11, 111–127. Keller, J. M. (1983). Motivational design and instruction. In C. M. Reigeluth (Ed.), Instructional-design theories and models: An overview of their current status (pp. 383–434). Hillsdale, NJ: Erlbaum. Kinzie, M. B. (1990). Requirements and benefits of effective interactive instruction: Learner control, selfregulation and continuing motivation. Educational Technology Research and Development, 38(1), 5–21. Ko¨nings, K., & Kirschner, P. (2010, August). Students’ expectations and perceptions on instruction: The role of students’ behavior. Paper presented at the SIG 6&7 meeting of the European association for ¨ lm, Germany. research on learning and instruction, U Kulik, C. L. C., Kulik, J. A., & Bangert-Drowns, R. L. (1990). Effectiveness of mastery learning programs: A meta-analysis. Review of Educational Research, 60, 265–299. Lawless, K. A., & Brown, S. (1997). Multimedia learning environments: Issues of learner control and navigation. Instructional Science, 25, 117–131. Lowyck, J., Elen, J., & Clarebout, G. (2004). Instructional conceptions: Analysis from an instructional design perspective. International Journal of Educational Research, 41, 429–444. Luyten, L., Lowyck, J., & Tuerlinckx, F. (2001). Task perception as mediating variable: A contribution to the validation of instructional knowledge. British Journal of Educational Psychology, 71, 203–223. Martin, J. (1984). The cognitive mediational paradigm for research on counseling. Journal of Counseling Psychology, 31(4), 558–571. McAuley, E., Duncan, T., & Tammen, V. V. (1987). Psychometric properties of the intrinsic motivation inventory in a competitive sport setting: A confirmatory factor analysis. Research Quarterly for Exercise and Sport, 60, 48–58. Merril, M. D. (2002). First principles of instruction. Educational Technology Research and Development, 50(3), 43–59.
123
324
M. Vandewaetere et al.
Mitrovic, A., Koedinger, K. R., & Martin, B. (2003). A comparative analysis of cognitive tutoring and constraint-based modeling. in Proceedings of the ninth international conference on user modeling UM 2003 (pp. 313–322). New York: Springer. Perkins, D. N. (1985). The fingertip effect: How information-processing technology shapes thinking. Educational Researcher, 14, 11–17. Plant, R. W., & Ryan, R. M. (1985). Intrinsic motivation and the effects of self-consciousness, selfawareness, and ego-involvement: An investigation of internally-controlling styles. Journal of Personality, 53, 435–449. Shuell, T. J., & Farber, S. L. (2001). Students’ perceptions of technology use in college courses. Journal of Educational Computing Research, 24, 119–138. Shute, V. J., & Zapata-Rivera, D. (2008). Adaptive technologies. In J. M. Spector, M. D. Merril, J. J. G. van Merrie¨nboer, & M. Driscoll (Eds.), Handbook of research on educational communications and technology (3rd ed., pp. 277–294). New York, NY: Taylor and Francis. Struyven, K., Dochy, F., Janssens, S., & Gielen, S. (2008). Students’ experiences with contrasting learning environments: The added value of students’ perceptions. Learning Environments Research, 11, 83–109. Thelwall, M. (2000). Computer-based assessment: A versatile educational tool. Computers & Education, 34, 37–49. Timmers, C., & Veldkamp, B. (2011). Attention paid to feedback by a computer-based assessment for learning on information literacy. Computers & Education, 56, 923–930. Vandewaetere, M., Desmet, P., & Clarebout, G. (2011). The value of learner characteristics in the development of computer-based adaptive learning environments. Computers in Human Behavior, 27, 118–130. Williams, G. C., & Deci, E. L. (1996). Internalization of biopsychosocial values by medical students: A test of self-determination theory. Journal of Personality and Social Psychology, 70, 767–779. Winne, P. H. (1987). Why process-product research cannot explain process-product finding and a proposed remedy: The cognitive mediational paradigm. Teaching and Teacher Education, 3(4), 333–356. Winne, P. H. (1995). Inherent details in self-regulated learning. Educational Psychologist, 30, 173–187. Winne, P. H. (2004). Students’ calibration of knowledge and learning processes: Implications for designing powerful software learning environments. International Journal of Educational Research, 41, 466–488. Winne, P. H. (2006). How software technologies can improve research on learning and bolster school reform. Educational Psychologist, 41, 5–17. Winters, F., Greene, J., & Costich, C. (2008). Self-regulation of learning within computer-based learning environments: A critical analysis. Educational Psychology Review, 20, 429–444.
Mieke Vandewaetere is a research assistant at the center for Interdisciplinary Research on Technology, Education and Communication. Sylke Vandercruysse is a research assistant at the center for Interdisciplinary Research on Technology, Education and Communication. Geraldine Clarebout is associate professor at the faculty of Psychology and Educational Sciences and works at the Center for Instructional Psychology and Technology at the University of Leuven.
123
