Using educational data mining methods to assess field ... - Springer Link

Education Tech Research Dev (2013) 61:521–548 DOI 10.1007/s11423-013-9298-1 RESEARCH ARTICLE

Using educational data mining methods to assess field-dependent and field-independent learners’ complex problem solving Charoula Angeli • Nicos Valanides

Published online: 26 April 2013 Ó Association for Educational Communications and Technology 2013

Abstract The present study investigated the problem-solving performance of 101 university students and their interactions with a computer modeling tool in order to solve a complex problem. Based on their performance on the hidden figures test, students were assigned to three groups of field-dependent (FD), field-mixed (FM), and field-independent (FI) learners, and were instructed to use integrated-format materials and Model-ItÒ in order to solve a problem about immigration policy. The results showed that there were significant differences among the three groups of learners in terms of their problem-solving performance. Consequently, the study employed educational data mining (EDM) methods in order to examine how FD and FI learners actually interacted with Model-ItÒ in order to solve the problem. The EDM methods provided rich analytical information and details about learners’ interactions with the computer tool. Implications for designing effective joint cognitive systems are discussed. Keywords Complex problem solving Field dependence–independence Computer modeling tools Educational data mining

Introduction People in their everyday life encounter problems that require immediate attention. Some problems are simple and require simple problem solving; other problems are complex and require complex problem solving (Frensch and Funke 1995; Fischer et al. 2012; Funke and Frensch 2007). According to Funke (2001), complex problems are characterized by (a) a large number of variables with high connectivity among them, (b) dynamic changes of these variables over time, (c) intransparency as not all problem variables lend themselves to direct C. Angeli (&) Department of Education, University of Cyprus, 11-13 Dramas Street, 1678 Nicosia, Cyprus e-mail: [email protected] N. Valanides School of Educational Sciences, Frederick University, P.O. Box 24729, 1303 Nicosia, Cyprus e-mail: [email protected]

123

522

C. Angeli, N. Valanides

observation, (d) time-delayed effects for the reason that not every action leads to immediate consequences, and (e) polytely, when a problem has multiple goals and some of the goals are contradictory requiring a reasonable trade-off. The two distinctive features of complex problems, namely, connectivity and dynamics, require computer tools for their realization, such as, dynamic computer visualizations and computer modeling tools (Funke 2001). Dynamic computer visualizations simulate complex phenomena allowing for a visual exploration of all factors that systemically affect the functioning of a system (Danner et al. 2011), while computer modeling tools allow the study of complex problems as interconnected components, whose behavior emerges from the interconnectedness of their components and thus counteract fragmentation (Penner 2000/2001; Sabelli 2006; van Merrienboer and Kirschner 2007). In relevant recent research, Angeli and Valanides (2004), and Angeli et al. (2009) investigated the effects of field dependence–independence (FD–I) on learners’ performance during complex problem solving with a computer modeling tool. In both experiments, field-independent (FI) learners were consistently associated with better problemsolving performance than field-dependent (FD) learners. On the basis of the outcomes of these studies, it would seem logical to recommend that only FI learners should be allowed to use computer modeling tools. Such a recommendation though would not recognize the fact that the percentage of FD learners in a classroom is much higher than the percentage of FI learners, and that a responsible educational system is supposed to provide equal learning opportunities for all learners. Therefore, it is important to invest research efforts in identifying ways of how all learners can equally benefit from learning with these tools. Educational data mining (EDM) in researching complex problem solving Research accounts on examining only the performance of a system are not enough, because they cannot establish principles about how technology affordances work with learners, or how learners actually interact with the affordances of technology to solve a problem (Salomon 1991; Stahl et al. 2006). More exploratory research methodologies are needed to investigate learners’ interactions with computer tools, as well as to assess learners’ complex problem solving with these tools, before deciding on the kind of training that may be helpful to employ in order to help those who need assistance. Educational data mining methods can be useful to employ in those cases where research examines students’ interactions with educational software in order to improve teaching (Baker and Yacef 2009; Munk and Drlik 2011a, b; Munk et al. 2010). Normally, students’ interactions with educational software result in very large data sets; and while these large data sets hold promise for improving teaching and learning, they can easily exceed human comprehension. In response to this, new research communities have emerged, such as the International EDM Society. EDM is an emerging discipline, concerned with developing methods for exploring the unique types of data that come from educational settings, and using those methods to better understand students and the settings in which they learn. EDM aims to improve the quality of analysis of large-scale educational data for the purpose of improving education by improving assessment, as well as intervention and planning (Siemens and Baker 2012). EDM approaches put more emphasis on reducing phenomena to components and analyzing individual components and relationships between them (Siemens and Baker 2012). Mining learners’ interactions with a computer tool may reveal otherwise hidden patterns or processes in complex problem solving, providing new and unique insights into how people with different cognitive styles problem solve with computer modeling tools.

123

Methods to assess field-dependent, independent learners

523

Purpose of the study On the basis of the above rationale, the authors herein engaged in research on firstly identifying any overall differences among groups of FD, FM, and FI learners’ complex problem-solving performance within the context of solving a complex systems problem with a computer modeling tool. Subsequently, the authors employed EDM methods in order to understand and assess the mechanisms of complex problem solving, and to identify FD and FI learners’ complex problem-solving patterns by examining their interactions with the computer modeling tool.

Literature review According to Frensch and Funke (1995), complex problem solving: occurs to overcome barriers between a given state and a desired goal state by means of behavioral and/or cognitive, multistep activities. The given state, goal state, and barriers between given state and goal state are complex, change dynamically during problem-solving, and are intransparent. The exact properties of the given state, goal state, and barriers are unknown to the solver at the outset. CPS (Complex problemsolving) implies the efficient interaction between a solver and the situational requirements of the task, and involves a solver’s cognitive, emotional, personal, and social abilities and knowledge (p. 18). Based on this definition, Frensch and Funke (1995) proposed a general theoretical framework, shown in Fig. 1, of complex problem solving. Their framework is based on the assumptions that (a) the theoretical goal is to understand the interplay among cognitive, motivational, personal, and social factors when complex, novel, dynamic and intransparent tasks are solved, and (b) the interplay among the various components can best be understood within an information processing perspective. As shown in Fig. 1, complex problem solving is viewed as the interaction between a problem solver and a complex problem in the context of a specific environment. The variables related to the problem solver are distinguished between domain-general and domain-specific knowledge, information processing (cognitive style, strategies, monitoring, and evaluation), and non-cognitive variables (motivation and personality attributes). The problem itself is depicted in terms of the barriers that exist between a given state and a goal state. The barriers are assumed to be complex, dynamically changing, and intransparent. The transition from the given state to the goal state is constrained by the problem solver’s knowledge and information processing capabilities, and by the tools that are available to the problem solver. The environment includes the resources that are available for problem solving, as well as feedback, expectations, cooperation, peer pressure, and disturbances. Clearly, research efforts that address which components within the problem solver, problem, and environment affect complex problem solving, and in what way, will be useful in terms of advancing the theory of complex problem solving. From this perspective, Angeli and Valanides (2004) and Angeli et al. (2009) systematically examined the extent to which an information-processing variable, that is, learners’ cognitive style of FD–I, affects performance within the context of solving a complex systems problem with a computer modeling tool. Cognitive style represents the characteristic mode of functioning shown by individuals in their perceptual and thinking behavior during complex problem solving and decision making (Morgan 1997; Schwering 1987). In

123

524


Fig. 1 A theoretical model of complex problem solving (adopted from Frensch and Funke 1995)

the literature, different cognitive styles are mentioned, but the most popular cognitive style, especially for instructional technology research, is FD–I (Witkin et al. 1977). Field dependence–independence is based on the individual’s reliance on the context to extract specific meaning, and it is directly related to how humans actually perceive, organize and process information (Witkin et al. 1977). These differences are often ascribed to the effects of the instructional context or of the prevailing field that is related to the complexity of the problem and the instructional materials supplementing teaching (Morgan 1997; Reiff 1996; Witkin et al. 1977). FD–I describes learners along a continuum such that individuals on one end are considered to be FD and individuals on the other end FI. Individuals who fall in the middle of the continuum are characterized as field-mixed (FM) (Liu and Reed 1994). The key difference between FD and FI learners is visual perceptiveness (Witkin et al. 1977). FD learners, who are asked to identify a simple geometric figure that is embedded in a complex figure will take longer to do than FI learners, or they may not be able to do it at all. According to Jonassen and Grabowski (1993), the prevailing context differentially influences FD, FM, and FI learners, and thus fundamental differences exist among them. FI learners, that is, those learners who can disentangle a field into its component parts, are not influenced by the existing structure of a field and thus can make choices independent of the perceptual field, are more successful in isolating important information from a complex whole, perform better on visual search tasks, and are more successful in analyzing ideas into their constituent parts and reorganizing them into new configurations (Davis 1991; Goodenough and Karp 1961; Snowman and Biehler 2003). FD learners, who are less

123


525

analytical and less attentive to detail, process information more globally, tend to see the perceptual field as a whole, and do not perform very well in a problem-solving space requiring the extraction of relevant information from a complex whole (Lambert 1981; Tannenbaum 1982). Angeli and Valanides (2004) investigated the extent to which instructional materials and learner FD–I differentially affected complex problem-solving performance with ModelItÒ, a dynamic systems computer modeling tool. The results of the experiment showed that students who received text-and-visual materials outperformed those who received text-only materials; that performance was significantly related to learners’ FD–I; and that there was a significant interaction effect between type of instructional materials and FD–I. Post hoc comparisons indicated that FI learners who received text-and-visual materials outperformed all other learners. In a follow-up experiment, Angeli et al. (2009) sought to improve the instructional design of the materials used in Angeli and Valanides (2004), and compared the effects of split-format and integrated-format materials on learners’ complex problem-solving performance. In the split-format condition, the model was first presented as a static diagram followed by its textual description below in a spatially-split format. In the integratedformat materials, the model was presented in an integrated format with its textual description such that all textual explanations were physically embedded into the diagram. The results showed that students who received the split-format materials reported a significantly lower problem-solving performance than students who received the integratedformat materials. There was also a significant interaction effect between type of materials and FD–I in terms of problem-solving performance. Post hoc comparisons indicated that FI learners who received the integrated-format materials outperformed both FD and FM learners. In essence, the results from both experiments showed that FI learners were consistently better than FD and FM learners, and that well-designed instructional materials did not lead to effective instruction and successful performance for all learners. Why were FI learners consistently better than FD and FM learners in both experiments? How did learners of different field type actually interact with the computer modeling tool to solve the problem? In order to answer these questions, the authors herein assumed both quantitative and more analytical research methods in order to report findings on learners’ complex problem-solving performance and interactions with the computer tool. In particular, the present study initially applied traditional quantitative research methods to identify differences among FD, FM, and FI learners, and then employed automated discovery methods, such as, cluster analysis and sequence, association and link analysis in order to assess and compare FD and FI learners’ interactions during complex problem solving with a computer modeling tool. Specifically, the study herein sought to provide answers to the following questions: 1. Are there any differences among FD, FM, and FI learners in terms of their complex problem-solving performance with a computer modeling tool? 2. Do certain sequences or patterns of computer interactions lead to better performance in complex problem solving? 3. Is there a difference between the sequences of computer interactions that FD and FI learners employed during complex problem solving? Based on the results of previous work (Angeli and Valanides 2004; Angeli et al. 2009; Burnett 2010), it is expected that FI learners will outperform all other learners in terms of problem-solving performance. The contribution of the current study to the existing body of research relates to the use of EDM methods in order to provide new insights into how

123

526


people with different cognitive styles interact with a computer modeling tool in order to solve a complex problem—a research and methodological focus that is currently missing from the current body of literature.

Methodology Participants One hundred and one first-year university students, who were enrolled in an instructional technology course, participated in the study. Of the 101 participants, 83 (82.18 %) were females, and 18 (17.82 %) males. The average age of the participants was 18.56 years (SD = .55). All students had basic computing skills and knew how to use Windows, Word, PowerPoint, and search the Internet for information using various search engines. The researchers explicitly asked the participants to state whether they had prior knowledge related to complex problems concepts, and none reported any familiarity. Similarly, students were asked whether they previously used computer modeling software to solve a problem, and all of them reported that they were unfamiliar with computer modeling software. Model-ItÒ Model-ItÒ (Metcalf et al. 2000; Stratford et al. 1998) was the computer modeling tool that the researchers used to create the model of immigration dynamics shown in Fig. 2, and the same tool that the research participants used to run the model and test different hypotheses. Model-ItÒ scaffolds the modeling process in three consecutive steps, namely, PLAN, BUILD, and TEST. During the first step (PLAN), the user creates the entities of the model. In the context of this research, the entities that were defined were Mexico and USA. During the second step (BUILD), several variables, such as, number of births, number of deaths, population, labor force, immigration rate, and unemployment were associated with each entity. These variables were also designated as independent or dependent based upon the direction of the relationship between them. For example, in Fig. 2, Mexican labor force is a dependent variable and Mexican number of births is an independent variable, because an increase in the number of new births in Mexico increases the population in Mexico and eventually Mexico’s labor force. In step two the user can also create relationships between variables. The relationship editor in Model-ItÒ is shown in Fig. 3. As shown in Fig. 3, relationships between variables in Model-ItÒ are formed in qualitative terms using verbal descriptions. Changes in a relationship are defined in terms of two orientations (i.e., increases or decreases) and different variations (e.g., about the same, a lot, a little, more and more, less and less). Subsequently, in step three (TEST), the user can run the model in order to test several hypotheses. For example, the value of an independent variable can be manipulated during run time to show how it can affect the value of a dependent variable, provided that the other independent variables remain constant or are controlled. During run time, a timer (see Fig. 4), counts time steps, which may represent whatever time interval the user conceptualizes, while a simulation graph, displayed at the bottom of the computer screen, shows how variables affect each other over a series of time steps. A screen capturing software was employed to capture all screen mouse movements into video files. The video files were later transcribed and analyzed using EDM methods in

123


527

Fig. 2 A complex systems model about immigration

Fig. 3 The relationship editor in Model-ItÒ

123

528


Fig. 4 Control of variables and testing a model in Model-ItÒ

order to understand how learners with different field types interacted with Model-ItÒ to solve the problem. Instructional problem Participants were given a complex problem about immigration dynamics to solve. Each participant had to individually run the computer model, shown in Fig. 2, in Model-ItÒ, in order to understand the situation at the USA–Mexico border regarding the illegal immigration of Mexicans to the United States, test different hypotheses, and finally suggest a solution to the problem. The model in Fig. 2 shows how an increase in the number of births in Mexico causes an increase in the population of Mexico, which gradually causes an increase in the labor force of Mexico and eventually an increase in the unemployment rate of Mexico. On the contrary, an increase in the number of deaths in Mexico causes a decrease in the population of Mexico, which eventually causes a decrease in the labor force of Mexico and a subsequent decrease in the unemployment rate of Mexico. An increase or decrease in the unemployment rate of Mexico causes a corresponding increase or decrease in the movement of Mexicans to the United States, and, eventually, an increase or decrease in the population, labor force, and unemployment rate of the United States. An increase in the number of businesses in the United States causes a decrease in the U.S. unemployment rate, while an increase in the movement of U.S. businesses to Mexico causes a decrease in the Mexican unemployment rate, but an increase in the U.S. unemployment rate. Overall, the model has five independent variables with high connectivity between them, and is associated with dynamic changes and time-delayed effects.

123


529

After opening the meters of all variables in the model, students can run the model in order to test several hypotheses by changing the value of one independent variable at a time while keeping the values of all other independent variables constant. Participants were given four immigration policies to test using the model in Model-ItÒ. The four immigration policies were (a) open border, (b) closed border, (c) job export, and (d) immigration. The open border policy encouraged the immigration of Mexicans to the United States as well as the movement of businesses from the United States to Mexico without imposing any restrictions. The closed border policy prohibited the immigration of Mexicans to the United States and the movement of businesses from the United States to Mexico. The job export policy created disincentives for American businesses to relocate to Mexico, such as higher taxes for goods produced in Mexico, which were intended for sale in the United States. Lastly, the immigration policy prohibited the immigration of Mexicans to the United States, but took no action regarding the movement of businesses from the United States to Mexico. Students were asked to form hypotheses based on these policies and test them using the model in Model-ItÒ. Then, they were asked to propose in writing which one of the four possible policies should be adopted in order to regulate as optimally as possible the situation at the USA–Mexico border. Learners’ responses to the problem were evaluated using an inductively developed rubric in order to assess the quality of their problemsolving performance. In essence, the instructional problem constituted a complex problem in the context of a specific problematic situation (USA–Mexico border). A given state of the problem was provided (illegal immigration of Mexicans to the United States) and learners were required to interact with it and suggest an optimal solution (desired goal) to the problem (French et al. 1963). The specific instructional problem has the characteristics of a complex problem because it incorporates variables, with high connectivity among them, that change dynamically over time (Funke 2001). In addition, the problem itself has multiple possible solutions depending upon the perspective that the problem solver assumes, while finding an optimal solution requires considering possible long-term effects of the full impact of each immigration policy given. Instructional materials All research participants received the same set of instructional materials. In the materials, the model in Fig. 2 was presented in an integrated format with its textual description. In essence, all textual explanations were physically embedded into the diagram. Several researchers (e.g., Feltovich et al. 2001; Narayanan and Hegarty 1998) have shown in their studies that complex problem solving imposes high load on learners’ cognitive resources, because of the highly interconnected nature of the elements that make up the problem. In other words, the intrinsic cognitive load imposed by the combination of the number of elements in the complex problem and the interactivity of those elements is so high that instructional efforts should be directed toward managing and reducing the extraneous cognitive load (Sweller et al. 1998; van Merrienboer et al. 2003). For this reason, the integrated-format materials were chosen to be used in this study, because according to the results of previous research studies, they do not promote instructional split attention and they do not increase extraneous cognitive load (Angeli et al. 2009; Ayres and Sweller 2005; Chandler and Sweller 1991, 1996).

123

530


Instruments Hidden figures test The hidden figures test (HFT) was used to determine learner’s field type (French et al. 1963). According to Rittschof (2010), the HFT is the most commonly used test for determining FD–I, because it is reliable and easy to administer. The test consists of 32 questions organized into two parts, A and B. Students are allowed 12 min for answering the questions in each part. Scores on the HFT can range from 0 to 32 (max = 32 points). For each question, the test presents five simple figures and asks learners to identify which one of them is embedded in a more complex figure. The test activity involved in the HFT has been described as perceptual disembedding (Rittschof 2010). According to Rittschof (2010), research studies that use perceptual disembedding tests, such as the HFT, treat FD–I as a continuous variable, while others specify cut-off scores for two or occasionally three levels of FD–I. In this study, the latter approach was followed. Research procedures First, the researcher administered the HFT (French et al. 1963) during three sessions scheduled at times convenient for the participants. Students’ average performance on the HFT was 13.69 (SD = 7.63). Students who scored 10 or lower were classified as FD, those who scored from 11 to 19 were classified as FM, and those who scored 20 or higher as FI. Based on this scheme, 35 students were classified as FD learners, 36 as FM learners, and 30 as FI learners. The cut-off scores were decided by considering the cut-off scores reported by other researchers (Angeli and Valanides 2004; Angeli et al. 2009; Chen and Macredie 2004; Daniels and Moore 2000; Khine 1996), so that meaningful comparisons of results across different studies can be made. Subsequently, each student participated in three 90-min weekly sessions. During the first 30 min of the first session, the researchers provided descriptions and explanations about complex problems and computer models, and, then, for the remaining 60 min the researchers taught students how to use Model-ItÒ. For example, during the 30-min lecture, the researchers explained that a computer model consists of entities, variables for each entity, and relationships between the variables of the entities. The differences between independent and dependent variables were also discussed. Students also learned how to run a model in Model-ItÒ and how to test hypotheses. During the second 90-min session, participants, with help from the researchers, developed computer models in Model-ItÒ for two phenomena, namely, the growth of plants and the economic growth of a family. In this session, most of the time was devoted to running the computer models in order to test different hypotheses and understand the need for controlling variables. During the first two sessions the researchers continuously scaffolded learners’ complex problem-solving process by providing them with assistance any time they faced a difficulty. The third 90-min research session was the research session during which the data for this study were collected, while the first two sessions were rather preparatory in nature. During the third research session students were given a set of instructional materials and they were asked to use the model in Model-ItÒ together with the instructional materials in order to solve the complex problem about immigration policy.

123


531

Data analysis Learners’ problem-solving performance was assessed with a rubric that was reported in previous related research (Angeli and Valanides 2004; Angeli et al. 2009). A one-way analysis of variance (ANOVA) with learners’ field type (FD, FM, and FI) as the classifying variable was performed in order to assess learners’ problem-solving performance. Pairwise comparisons and effect sizes were also computed. Regarding the EDM analyses, only data from 30 FD and 30 FI learners were considered while all data concerning FM learners were deliberately excluded from these analyses. It was thus possible to concentrate on the differences between the participants who fell in the two extreme ends (FD or FI) of the FD–I continuum. This deliberate decision was an attempt to simplify the consequent analyses and focus exclusively on the differences between FD and FI learners. For each video file captured using the screen capturing software, a transcript was created describing each participant’s interaction with Model-ItÒ. Each transcript consisted of a table with three columns, namely, STUDENT ID, TIME, and ACTION. STUDENT ID was student’s identification number, TIME represented the beginning and ending time of an action, so that the duration of an action could be calculated, and ACTION was a description of what the student was doing in Model-ItÒ during that time. ACTION was further coded into a sequence of computer or button actions each learner undertook in Model-ItÒ. Eleven actions were considered, namely, B, T, M, F, P, IV1, IV2, IV3, IV4, IV5, and S. B referred to the BUILD button learners pushed in order to see the model on the screen with all related variables and the relationships between them. T referred to the TEST button for accessing the needed test tools for the purpose of running the model. M referred to the button METER, which was used to open the meter of each variable in order to be able to observe changes in the values of the variable. F referred to fitting (organizing) the model on the screen, so that all meters could be simultaneously observed. P was the PLAY button that was used to run the model, IVn (n = 1–5) referred to each one of the five independent variables used in the model, and lastly, S referred to the STOP button that was used to terminate the run time of a model. The first data mining procedure that was carried out was a K-means cluster analysis procedure (Bramer 2007; Witten et al. 2011). The K-means cluster analysis procedure is an exploratory and analytical technique that is used to segment the data into similar and natural groupings, called clusters, so that individuals or objects in the same cluster are homogeneous, whereas at the same time there is heterogeneity across clusters (Witten et al. 2011). The data file that was used for this analysis contained data from only 60 students (30 FD and 30 FI). The variables contained in the first data file included: (a) student ID, (b) problem-solving performance, (c) HFT, (d) total duration in Model-ItÒ, (e) number of times the value of each of the five independent variable was changed, (f) total time of running the model, and (g) maximum number of meters that were opened. Then, another EDM method, the Sequence, Association and Link Analysis procedure (Nisbet et al. 2009) was employed. This procedure is specifically designed for extracting association rules and sequences from datasets. Association rules are extracted to determine which objects (i.e., button actions in this case) are closely associated together. Sequence analysis is concerned with an immediate subsequent action given a previous one. Link analysis provides information on the strength of the association rules or sequence rules (Nisbet et al. 2009). Once extracted, rules about associations or the sequences of actions, as they occur in a database, can be useful for organizing complex evidence and for quickly

123

532


extracting patterns and associations between individuals and actions (Nisbet et al. 2009). In education, the Sequence, Association, and Link Analysis procedure has been mostly used in e-learning research in order to find the most effective structure of e-learning courses, optimize the learning content of a course by providing sequences of closely associated modules, and recommend the most suitable learning path based on students’ behavior (Munk and Drlik 2011a, b; Munk et al. 2010).

Results Learners’ problem-solving performance Each written response to the problem (i.e., which one of the four immigration policies should be adopted in order to regulate as optimally as possible the situation at the USA– Mexico border) was assessed on the basis of an inductively constructed rubric that was reported in previous research studies (Angeli and Valanides 2004; Angeli et al. 2009). The scoring rubric, as shown in Table 1, had three mutually exclusive levels with scores ranging from 1 (low performance) to 3 (high performance). The specific criteria in Table 1 were used for evaluating learners’ problem-solving performance. One of the researchers and a trained rater independently evaluated students’ written responses. Cohen’s kappa was used to measure the agreement between the two raters, and it was found to be quite satisfactory (k = 0.83). The researcher and the rater easily resolved the observed disagreements, after discussion. Students’ problem-solving performance was found to be 1.37 (SD = .60) for FD students, and increased progressively to 1.78 (SD = .64) for FM learners, and to 2.10 (SD = .75) for FI learners. Subsequent tests confirmed normality and homogeneity of variances of students’ problem-solving performance (dependent variable), and a one-way analysis of variance (ANOVA) was performed with FD–I as the independent variable. The results indicated that performance was significantly related to learners’ FD–I, F(2, 98) = 9.04, p \ 0.01, n2 = 0.16. Post hoc comparisons using the Tukey HSD method indicated that both FI and FM learners outperformed FD learners. Consequently, two effect Table 1 Rubric for assessing learner problem-solving performance (adopted from Angeli and Valanides 2004) 3—High a. Reaches a decision by correctly interpreting the simulated outcomes of the model. b. Examines the consequences of all policies and identifies pros and cons of each policy. c. Considers possible long-term effects of the full impact of each policy and recognizes that ramifying may take a long time. 2—Medium a. Reaches a decision by correctly interpreting the simulated outcomes of the model. b. Examines the consequences of all policies and identifies pros and cons of each policy. c. Does not consider possible long-term effects of the full impact of each policy and does not recognize that ramifying may take a long time. 1—Low a. Reaches a decision, which is not based on accurate interpretations of the simulated outcomes of the model. b. Does not consider pros and cons of each policy and shows biased thinking. c. Does not consider possible long-term effects of the full impact of each policy and does not recognize that ramifying may take a long time.

123


533

sizes using Cohen’s d were calculated (Cohen 1988). The magnitude of the superior problem-solving performance of FI learners in comparison with FD learners’ performance was found to be very high (d = 0.99). Similarly, the better mean performance of FM learners over the mean performance of FD learners was also associated with a large effect size, which was found to be d = 0.66. K-means cluster analysis Three K-means cluster analyses were consequently performed with the number of clusters set to two, three, and four, respectively. Each cluster analysis used the ‘‘iterate and classify’’ method, with ten variables. The cluster analysis with the number of clusters set to four produced the most homogenous clusters, while exemplifying at the same time the heterogeneity across them. The final cluster centers of the ten variables used in this analysis and the number of FD and FI learners in each cluster are shown in Table 2. Sequence, association and link (SAL) analyses Based on the results of the K-means cluster analysis procedure, one data file was created for each cluster. The variables contained in the four data files included student ID, cluster number, action in Model-ItÒ (in terms of computer or button actions), and chronological order of action. For example, if student ID 456789, member of cluster 1, with FI as his field type (denoted as 3), first pushed the BUILD button followed by the buttons TEST and METER in Model-ItÒ, these data were recorded in three different rows and five columns as follows: 456789-1-3-B-0, 456789-1-3-T-1, 456789-1-3-M-2.

Table 2 Final cluster centers and number of FD and FI learners in each cluster Cluster 1

2

3

4

Problem-solving performance

1.17

1.38

2.47

1.44

HFT

9.50

5.62

20.05

15.00

Duration in Model-ItÒ

58.83

38.09

44.35

19.47

Number of times the value of IV1 was changed

3.83

1.62

3.53

2.19


4.50

3.38

3.53

2.50


2.83

0.31

2.11

0.63


1.50

0.31

2.11

0.56


0.33

0.38

2.11

0.50

Total time of running the model

29.58

14.15

29.68

7.61

Maximum number of meters opened

9.67

11.00

14.00

8.75

Number of FD and FI learners in each cluster FD–I 1,00 (FD)

9

13

0

8

3,00 (FI)

3

0

19

8

12

13

19

16

Total

IV1 Mexico: number of births, IV2 USA: movement of businesses, IV3 Mexico: number of deaths, IV4 USA: number of births; IV5 USA: number of deaths

123

534


Each of these data files was used as input to a SAL analysis in order to extract sequence rules (Nisbet et al. 2009) associated with how participants used the affordances of ModelItÒ in each cluster. Four SAL analyses, one for each cluster, were carried out. After specific recommendations from the data mining software, the minimum support was set to 0.55, the confidence level to 0.95, and the maximum size of a sequence to six items, for all SAL analyses. Sequence rules followed the general form: ‘‘If Body, then Head’’. The support value refers to the relative frequency of co-occurrences of the Body and the Head of each association rule, and the confidence value denotes the conditional probability of the Head of the association rule given the Body of the association rule (Nisbet et al. 2009). For each SAL analysis, the authors report a summary of sequential rules that were identified, and a table displaying the most frequent sequences of rules. SAL analysis for cluster 1 Table 3 shows the sequential rules, and Table 4 shows the most frequent sequences of rules for cluster 1.

Table 3 Summary of sequential rules produced for cluster 1 Body

?

Head

1

(B)

?

(T)

2

(B), (T), (M), (P)

?

(IV1)

3

(B), (T), (M), (P)

?

(IV2)

4

(B), (T), (M), (P), (P)

?

(IV1)

5

(B), (T), (M), (P), (S)

?

(IV1)

6

(B), (T), (M), (P), (S)

?

(IV2)

7

(B), (T), (M), (F)

?

(IV1)

8

(B), (T), (M), (F)

?

(IV2)

9

(B), (T), (M), (F)

?

(S)

10

(B), (T), (M), (F), (P)

?

(P)

11

(B), (T), (M), (F), (P)

?

(IV1)

12

(B), (T), (M), (F), (P)

?

(IV2)

13

(B), (T), (M), (F), (P)

?

(S)

14

(B), (T), (M), (F)

?

(S), (IV1)

15

(B), (T), (M), (F), (S)

?

(IV1)

16

(B), (T), (M), (F)

?

(S), (IV2)

17

(B), (T), (M), (F), (S)

?

(IV2)

18

(B), (T), (M), (S)

?

(IV1)

19

(B), (T), (M), (S)

?

(IV2)

20

(B), (T), (M), (S), (P)

?

(IV1)

21

(B), (T), (M), (S), (P)

?

(S)

22

(T), (P), (P), (IV2)

?

(IV3)

23

(T), (P), (P), (IV1), (IV2)

?

(IV3)

24

(T), (P), (IV1), (IV2)

?

(IV3)

B BUILD, T TEST, M METER, P PLAY, S STOP, F fit the model on the screen, IV1 change the value of Mexico: number of births, IV2 change the value of USA: movement of businesses, IV3 change the value of Mexico: number of deaths

123


535

Table 4 Frequent sequences of rules for cluster 1 Frequent sequences

Number of itemsets

Frequency

1

(B)

1.00

10.00

2

(T)

1.00

10.00

3

(M)

1.00

9.00

4

(P)

1.00

8.00

5

(F)

1.00

7.00

6

(S)

1.00

7.00

7

(B), (T), (M), (P), (IV1)

5.00

7.00

8

(B), (T), (M), (P), (IV2)

5.00

7.00

9

(B), (T), (M), (P), (IV3)

5.00

5.00

10

(B), (T), (P), (IV1), (IV5)

5.00

5.00

11

(B), (T), (P), (IV1), (IV2), (IV3)

6.00

6.00

12

(B), (T), (P), (IV1), (IV2), (IV5)

6.00

5.00

13

(B), (T), (P), (IV1), (IV3), (IV5)

6.00

5.00

14

(B), (T), (P), (IV2), (IV3)

5.00

6.00

15

(B), (T), (P), (IV2), (IV5)

5.00

5.00

16

(B), (T), (P), (IV2), (IV3), (IV5)

6.00

5.00

17

(B), (T), (P), (IV1), (IV2)

5.00

5.00

18

(B), (T), (P), (IV1), (IV3)

5.00

5.00

19

(B), (T), (P), (IV1), (IV2), (IV3)

6.00

5.00

20

(B), (T), (P), (IV2)

4.00

6.00

21

(B), (T), (P), (IV2), (IV3)

5.00

5.00

22

(B), (T), (P), (IV3)

4.00

5.00

23

(B), (T), (IV3), (IV5)

4.00

5.00

B BUILD, T TEST, M METER, P PLAY, S STOP, F fit the model on the screen, IV1 change the value of Mexico: number of births, IV2 change the value of USA: movement of businesses, IV3 change the value of Mexico: number of deaths, IV5 change the value of USA: number of deaths

As shown in Table 3, with the exception of the first rule which appears to be something that students just tried out, the first three items in rules 2–21 were exactly the same. Specifically, in these rules, action B was followed by action T, which was then followed by action M. This sequence of actions is meaningful as it shows that students were correctly pushing the BUILD button first to view the variables of the model as well as the relationships between them, followed by a push of the TEST button for accessing the tools needed in order to test the model, and finally by the button METER to open the meter of each variable. In addition, by pushing the BUILD button at the beginning of a new action sequence meant that the effects of any other previous action, such as for example P (PLAY) in rules 2 and 3, were automatically terminated or cancelled taking the values of all variables in the model back to their initial values. Interestingly, students in cluster 1 organized (the F action in rules 7–17) the model on the screen appropriately, so that all meters that were opened could be seen simultaneously. Of course, an F action becomes important when the learner actually runs the model to observe how the value of each affected variable changes. However, this was not always the case with students’ actions, as it is shown in rules 7–9 and 14–17. Thus, action F was an

123

536


important step in the problem-solving process in only four out of the eleven rules that appeared. Additionally, as shown in Table 3, students in cluster 1 attempted to examine the effects of two independent variables, namely, IV1 and IV2, repeatedly (rules 2–8, 11–12, 14–20), but these attempts were not always meaningful or productive. For example, the actions described in rules 5–6 and 14–19 were not productive, because no effect of changing the values of IV1 and IV2 could be observed after pushing the STOP button. Then in some other cases, students’ actions appeared to be pointless. For example, as shown in rules 4 and 10, students pushed the button PLAY twice in a row without any reason, or as shown in rules 14 and 15–21 students pushed the button STOP without having something to stop since the model was not running at that time. Students also attempted to examine the effects of IV3 (rules 22–24) occasionally, while they rarely attempted to examine the effects of IV4 and IV5. Students in cluster 1 repeated these actions over and over again. This pattern of problem-solving behavior with Model-ItÒ exemplified a mechanistic approach of testing and retesting (meaningfully or not) the model with the same variables (mostly IV1, IV2, and IV3) without showing some evidence of working systematically and strategically towards finding the best solution to the problem. The results in Table 4 indicate that there were some frequent actions that contained only one item (sequences 1–6), which appeared in isolation and not as part of a longer sequence. The rest of the frequent sequences contained up to six items (sequences 7–23). Only in sequences 7–9 students opened the meters of the variables before running the model, whereas in sequence 23 the PLAY button was not used for running the model. Sequences 10–19, 21, and 23 clearly demonstrate that students in cluster 1 did not correctly implement the control of variables strategy, since they were changing the values of more than one independent variable at a time. Thus, it was rather impossible to disentangle the effects of more than one variable at a time, because the relationships between the different variables had different orientations (i.e., increases or decreases) and different variations of change (e.g., about the same, a lot, a little, more and more, less and less). SAL analysis for cluster 2 Table 5 shows the sequential rules, and Table 6 the most frequent sequences of rules for cluster 2. As shown in Table 5, students in cluster 2 wondered around in Model-ItÒ (rules 1–38) in a disorderly, doubtful, and uncertain way about what they needed to do. For example, rules 1–35 indicate that students used to push the B (BUILD) button twice in a row without any specific reason. Action T (TEST) was the most frequent action following the B action, or the BB sequence of actions. In addition, action M (METER) was frequently used mainly after the sequences BB and BT or mainly after the sequence BBT. Interestingly, while students ran the model a total of 29 times, it was only in six of those cases (rules 39–44) that they actually examined the effects of IV2 (four times) or the effects of IV1 (twice) on the dependent variables. In general, students’ computer interactions included several repetitions of the same sequences or slightly different sequences. They also used to open the meters of several variables frequently, but without changing the values of the independent variables to observe their effects on the dependent variables. Also some actions were meaningless, such as for example in those cases where students used to push the STOP button without having the model running on the screen (rules 6, 7, 13–15, 25–26, 32–33). The majority of students’ actions seemed to be off target and irrelevant without showing evidence for a

123


537


?

Head

1

(B)

?

(B)

2

(B), (B)

?

(T)

3

(B), (B)

?

(M)

4

(B)

?

(B), (P)

5

(B), (B)

?

(P)

6

(B)

?

(B), (S)

7

(B), (B)

?

(S)

8

(B), (B)

?

(T), (M)

9

(B), (B), (T)

?

(M)

10

(B), (B)

?

(T), (M), (P)

11

(B), (B), (T)

?

(M), (P)

12

(B), (B), (T), (M)

?

(P)

13

(B), (B)

?

(T), (M), (S)

14

(B), (B), (T)

?

(M), (S)

15

(B), (B), (T), (M)

?

(S)

16

(B), (B)

?

(T), (M), (P), (S)

17

(B), (B), (T)

?

(M), (P), (S)

18

(B), (B), (T), (M)

?

(P), (S)

19

(B), (B), (T), (M), (P)

?

(S)

20

(B), (B)

?

(T), (P)

21

(B), (B), (T)

?

(P)

22

(B), (B)

?

(T), (P), (S)

23

(B), (B), (T)

?

(P), (S)

24

(B), (B), (T), (P)

?

(S)

25

(B), (B)

?

(T), (S)

26

(B), (B), (T)

?

(S)

27

(B), (B)

?

(M), (P)

28

(B), (B), (M)

?

(P)

29

(B), (B)

?

(M), (P), (S)

30

(B), (B), (M)

?

(P), (S)

31

(B), (B), (M), (P)

?

(S)

32

(B), (B)

?

(M), (S)

33

(B), (B), (M)

?

(S)

34

(B)

?

(B), (P), (S)

35

(B), (B)

?

(P), (S)

36

(B), (T)

?

(M)

37

(B)

?

(T), (P)

38

(B), (T)

?

(P)

39

(B)

?

(T), (M), (P), (IV1)

40

(B), (T)

?

(M), (P), (IV1)

41

(B)

?

(T), (M), (P), (IV2)

42

(B), (T)

?

(M), (P), (IV2)

43

(B), (T), (M)

?

(P), (IV2)

44

(B), (T), (M), (P)

?

(IV2)

B BUILD, T TEST, M METER, P PLAY, S STOP, F fit the model on the screen, IV1 change the value of Mexico: number of births, IV2 change the value of USA: movement of businesses

123

538



Number of itemsets

Frequency

1

(B)

1.00

13.00

2

(T)

1.00

13.00

3

(M)

1.00

13.00

4

(P)

1.00

13.00

5

(B), (T), (M), (P), (IV2)

5.00

9.00

6

(B), (T), (M), (P), (IV1)

5.00

8.00

B BUILD, T TEST, M METER, P PLAY, F fit the model on the screen, IV1 change the value of Mexico: number of births, IV2 change the value of USA: movement of businesses

systematic effort in terms of examining the effects of the IVs on the values of the dependent variables. As a matter of fact, most of the time learners appeared to be caught into a loop of repeating the same thing without having a clue of how to get out of this chaotic and muddled process. It is also worth mentioning that students in cluster 2 never organized the model on the screen in such a way, so that all meters could be observed simultaneously. These results corroborate the evidence presented in Table 6. Obviously, the most frequent sequences were single actions, which appeared in isolation and not as part of a longer sequence, such as, B, T, M, and P that were probably pursued for exploratory purposes, while the less frequent sequences contained up to only five items, which included meaningful sequential actions (sequences 5 and 6) that provided useful but limited information. SAL analysis for cluster 3 Table 7 shows the sequential rules, and Table 8 shows the most frequent sequences of rules for cluster 3. As shown in Table 7, students in rules 1–14 displayed the variables of the model (B) on the screen, organized the model on the screen in an appropriate way (F), opened the meters of the variables (M) in such a way that could all fit on the screen, and changed the value of one IV at a time to examine its effects on the dependent variables (rules 10–14). In three cases (rules 6, 7, 9), students pushed the STOP button even though the model was not running. The majority of the rules (rules 8–38) included BFTM as the initial sequence of actions followed by action P (rules 10–18). The sequence PP in rules 15–16 was obviously unnecessary and probably indicates exploratory activity. In rules 19–38, the initial sequence BFTM was followed by changes in the values of every possible combination of two IVs together, but without actual observation of the effects on the dependent variables, since no attempt was made to run the model. As shown in rules 39–50, students also attempted to examine the effects of every combination of three IVs together on the dependent variables after the BFT sequence of actions. Rules 51–57 demonstrate attempts to examine the effects of every combination of four or five IVs together on the dependent variables, although without running the actual model as well. Regarding the results in Table 8, rules 1–5 represent frequent single actions, which were not part of a longer sequence, indicating initial attempts to explore the affordances of Model-ItÒ. Sequences 6–10 represent well-executed series of six actions (the upper limit of items in each sequence of actions) involving one independent variable at a time leading to meaningful observations. In sequences 11–20 students attempted to examine the effects of

123


539


?

Head

1

(B)

?

(F)

2

(B)

?

(F), (T)

3

(B), (F)

?

(T)

4

(B)

?

(F), (P)

5

(B), (F)

?

(P)

6

(B)

?

(F), (S)

7

(B), (F)

?

(S)

8

(B), (F), (T), (M)

?

(P)

9

(B), (F), (T), (M)

?

(S)

10

(B), (F), (T), (M), (P)

?

(IV1)

11

(B), (F), (T), (M), (P)

?

(IV2)

12

(B), (F), (T), (M), (P)

?

(IV3)

13

(B), (F), (T), (M), (P)

?

(IV4)

14

(B), (F), (T), (M), (P)

?

(IV5)

15

(B), (F), (T), (M)

?

(P), (P)

16

(B), (F), (T), (M), (P)

?

(P)

17

(B), (F), (T), (M)

?

(P), (S)

18

(B), (F), (T), (M), (P)

?

(S)

19

(B), (F), (T), (M)

?

(IV1), (IV2)

20

(B), (F), (T), (M), (IV1)

?

(IV2)

21

(B), (F), (T), (M)

?

(IV1), (IV3)

22

(B), (F), (T), (M), (IV1)

?

(IV3)

23

(B), (F), (T), (M)

?

(IV1), (IV4)

24

(B), (F), (T), (M), (IV1)

?

(IV4)

25

(B), (F), (T), (M)

?

(IV1), (IV5)

26

(B), (F), (T), (M), (IV1)

?

(IV5)

27

(B), (F), (T), (M)

?

(IV2), (IV3)

28

(B), (F), (T), (M), (IV2)

?

(IV3)

29

(B), (F), (T), (M)

?

(IV2), (IV4)

30

(B), (F), (T), (M), (IV2)

?

(IV4)

31

(B), (F), (T), (M)

?

(IV2), (IV5)

32

(B), (F), (T), (M), (IV2)

?

(IV5)

33

(B), (F), (T), (M)

?

(IV3), (IV4)

34

(B), (F), (T), (M), (IV3)

?

(IV4)

35

(B), (F), (T), (M)

?

(IV3), (IV5)

36

(B), (F), (T), (M), (IV3)

?

(IV5)

37

(B), (F), (T), (M)

?

(IV4), (IV5)

38

(B), (F), (T), (M), (IV4)

?

(IV5)

39

(B), (F), (T), (IV1)

?

(IV2), (IV3)

40

(B), (F), (T), (IV1), (IV2)

?

(IV3)

41

(B), (F), (T), (IV1)

?

(IV2), (IV4)

42

(B), (F), (T), (IV1), (IV2)

?

(IV4)

43

(B), (F), (T), (IV1)

?

(IV2), (IV5)

123

540


Table 7 continued Body

?

Head

44

(B), (F), (T), (IV1), (IV2)

?

(IV5)

45

(B), (F), (T), (IV1)

?

(IV3), (IV4)

46

(B), (F), (T), (IV1), (IV3)

?

(IV4)

47

(B), (F), (T), (IV1)

?

(IV3), (IV5)

48

(B), (F), (T), (IV1), (IV3)

?

(IV5)

49

(B), (F), (T), (IV1)

?

(IV4), (IV5)

50

(B), (F), (T), (IV1), (IV4)

?

(IV5)

51

(B), (F), (IV2)

?

(IV3), (IV4), (IV5)

52

(B), (F), (IV2), (IV3)

?

(IV4), (IV5)

53

(B), (F), (IV2), (IV3), (IV4)

?

(IV5)

54

(M), (IV1)

?

(IV2), (IV3), (IV4), (IV5)

55

(M), (IV1), (IV2)

?

(IV3), (IV4), (IV5)

56

(M), (IV1), (IV2), (IV3)

?

(IV4), (IV5)

57

(M), (IV1), (IV2), (IV3), (IV4)

?

(IV5)

B BUILD, T TEST, M METER, S STOP, P PLAY, F fit the model on the screen, IV1 change the value of Mexico: number of births, IV2 change the value of USA: movement of businesses, IV3 change the value of Mexico: number of deaths, IV4 change the value of USA: number of births, IV5 change the value of USA: number of deaths

every possible combination of two IVs together, in sequences 21–26 students attempted to examine the effects of every possible combination of three IVs together, while in sequences 27–29 students attempted to examine the effects of every possible combination of four IVs together. Lastly, in sequence 30 all five IVs were considered. Interestingly, students never actually ran the model with changing the values of more than one IV at a time (rules 11–30). Based on the results shown in Tables 7 and 8, the majority of the students in cluster 3 exhibited combinatorial reasoning and the ability to control variables. SAL analysis for cluster 4 Regarding cluster 4, the summary of sequential rules is shown in Table 9, and the most frequent sequences of rules are presented in Table 10. As shown in Table 9, rules 1–5 included unnecessary actions, such as for example a push of the BUILD button twice in a row. More importantly, in three cases (rules, 2, 3, 5) students changed the value of IV1 after pushing the STOP button, meaning that the effect of IV1 on the dependent variables could not be observed, because the model was not running. A similar pattern was also observed in rules 8, 9, 12–14, 17, and 25. In addition, students did not always fit the model on the screen appropriately, and they did not always run the model (rules, 12, 14, 22, 25). There were also sequences of rules that did not lead to productive outcomes, such as, for example rules 12 and 14 where students pushed the STOP button at times when the model was not running. In other cases, students changed the values of two IVs (IV1 and IV2) before or after a PLAY action, indicating limitations in terms of their ability to control variables, although there were some cases where they changed the value of only one IV at a time, such as IV1 in rules 8, 13, 19, and 21, or IV2 in rules 6, 7, 18, and 20. The total evidence presented in Table 9 clearly demonstrates

123


541


Number of itemsets

Frequency

1

(B)

1.00

19.00

2

(F)

1.00

19.00

3

(T)

1.00

19.00

4

(M)

1.00

19.00

5

(P)

1.00

19.00

6

(B), (F), (T), (M), (P), (IV1)

6.00

10.00

7

(B), (F), (T), (M), (P), (IV2)

6.00

10.00

8

(B), (F), (T), (M), (P), (IV3)

6.00

10.00

9

(B), (F), (T), (M), (P), (IV4)

6.00

10.00

10

(B), (F), (T), (M), (P), (IV5)

6.00

10.00

11

(B), (F), (T), (M), (IV1), (IV2)

6.00

10.00

12

(B), (F), (T), (M), (IV1), (IV3)

6.00

10.00

13

(B), (F), (T), (M), (IV1), (IV4)

6.00

10.00

14

(B), (F), (T), (M), (IV1), (IV5)

6.00

10.00

15

(B), (F), (T), (M), (IV2), (IV3)

6.00

10.00

16

(B), (F), (T), (M), (IV2), (IV4)

6.00

10.00

17

(B), (F), (T), (M), (IV2), (IV5)

6.00

10.00

18

(B), (F), (T), (M), (IV3), (IV4)

6.00

10.00

19

(B), (F), (T), (M), (IV3), (IV5)

6.00

10.00

20

(B), (F), (T), (M), (IV4), (IV5)

6.00

10.00

21

(B), (F), (T), (IV1), (IV2), (IV3)

6.00

10.00

22

(B), (F), (T), (IV1), (IV2), (IV4)

6.00

10.00

23

(B), (F), (T), (IV1), (IV2), (IV5)

6.00

10.00

24

(B), (F), (T), (IV1), (IV3), (IV4)

6.00

10.00

25

(B), (F), (T), (IV1), (IV3), (IV5)

6.00

10.00

26

(B), (F), (T), (IV1), (IV4), (IV5)

6.00

10.00

27

(B), (T), (IV1), (IV2), (IV3), (IV4)

6.00

10.00

28

(B), (T), (IV1), (IV2), (IV3), (IV5)

6.00

10.00

29

(B), (T), (IV1), (IV2), (IV4), (IV5)

6.00

10.00

30

(P), (IV1), (IV2), (IV3), (IV4), (IV5)

6.00

10.00

B BUILD, T TEST, M METER, P PLAY, F fit the model on the screen, IV1 change the value of Mexico: number of births, IV2 change the value of USA: movement of businesses, IV3 change the value of Mexico: number of deaths, IV4 change the value of USA: number of births, IV5 change the value of USA: number of deaths

students’ limited ability to control variables, as well as their inability to work in a consistent and systematic way towards solving the complex problem. As shown in Table 10, the most frequent sequences were single actions (i.e., B, T, M, P), which were isolated actions and not part of a longer sequence and thus cannot provide any useful information other than indications of attempts to explore the affordances of Model-ItÒ. The other frequent sequences consisted of multiple actions where students mainly tested the effect of IV2 or the effect of IV1, by keeping constant or ignoring the other four IVs (sequences 5 and 6). However, as shown in sequence 7 they also changed the values of both IV1 and IV2 together, demonstrating a weakness in controlling variables.

123

542


Table 9 Sequences of sequential rules produced for cluster 4 Body

?

Head

1

(B), (B), (P)

?

(S)

2

(B), (B), (P)

?

(S), (IV1)

3

(B), (B), (P), (S)

?

(IV1)

4

(B), (B), (P)

?

(IV1)

5

(B), (B), (S)

?

(IV1)

6

(B), (T), (P), (IV2), (P)

?

(S)

7

(B), (T), (IV2), (P)

?

(S) (IV1)

8

(B), (T), (IV1), (P), (S)

?

9

(B), (T), (IV1), (S)

?

(IV1)

10

(B), (T), (IV1),(IV2), (P)

?

(S) (IV1)

11

(B), (T), (IV1), (IV2), (P)

?

12

(B), (T), (IV1), (IV2), (S)

?

(IV1)

13

(B), (M), (IV1), (P), (S)

?

(IV1)

14

(B), (M), (IV1), (S)

?

(IV1)

15

(B), (M), (IV1), (IV2), (P)

?

(S)

16

(B), (M), (IV1), (IV2), (P)

?

(IV1)

17

(B), (M), (IV1), (IV2), (IV2), (S)

?

(IV1)

18

(T), (M), (P), (IV2), (P)

?

(P)

19

(T), (M), (P), (IV1), (S)

?

(S)

20

(T), (M), (IV2), (P)

?

(P), (S)

21

(T), (M), (IV1), (P), (S)

?

(S)

22

(T), (M), (IV1), (S)

?

(S)

23

(T), (M), (IV1), (IV2), (P)

?

(S)

24

(T), (M), (IV1), (IV2), (P)

?

(IV1)

25

(T), (F), (IV1), (IV2), (S)

?

(IV1)

B BUILD, T TEST, M METER, P PLAY, S STOP, F fit the model on the screen, IV1 change the value of Mexico: number of births, IV2 change the value of USA: movement of businesses

Another single action, although less frequent, was the F action, which becomes meaningful only when it is used in conjunction with other actions, such as for example the M action for opening the meters of variables on the screen.

Discussion The results of the present study indicate that FI learners exhibited a significantly better problem-solving performance than FD learners, and FM learners also exhibited a significantly better problem-solving performance than FD learners during complex problemsolving with Model-ItÒ. These results corroborate the results of two previous related studies (Angeli and Valanides 2004; Angeli et al. 2009). These combined results clearly indicate that no matter how efficient the instructional design of the materials is, learners’ problem-solving performance may not be the best possible if learners’ field type prevents them from performing successfully with these materials during problem solving. In other words, the results of the study herein clearly indicated that FD learners were not able to

123


543


Number of item sets

Frequency

1

(B)

1.00

16.00

2

(T)

1.00

16.00

3

(M)

1.00

15.00

4

(P)

1.00

15.00

5

(B), (T), (P), (IV2)

4.00

13.00

6

(B), (T), (P), (IV1)

4.00

10.00

7

(B), (T), (P), (IV2), (IV1)

5.00

10.00

8

(F)

1.00

7.00

B BUILD, T TEST, M METER, P PLAY, F fit the model on the screen, IV1 change the value of Mexico: number of births, IV2 change the value of USA: movement of businesses

benefit from the integrated-format materials, which were shown to be efficient and effective in several other studies (Ayres and Sweller 2005; Chandler and Sweller 1991, 1996). Evidently, learners’ actual interactions with Model-ItÒ to solve the problem become important for explaining why their complex problem-solving performance with Model-ItÒ depended on their FD–I. The screen capturing software, which was employed for capturing learners’ interactions into video files as each learner was using Model-ItÒ to solve the problem, provided rich supplementary information for explaining the differential performance of FD and FI learners. The video files were transcribed and analyzed using two EDM methods, namely cluster analysis and SAL analyses. The results of a K-means cluster analysis for FD and FI learners only, resulted in a fourcluster solution that provided homogeneity within each cluster and heterogeneity across clusters. In terms of FD–I, clusters 2 and 3 exhibited maximum within-cluster homogeneity and maximum heterogeneity between clusters. Cluster 2 included exclusively only 13 FD learners and cluster 3 included exclusively only 19 FI learners. On the contrary, cluster 4 included 8 FD and 8 FI learners, and cluster 1 included 9 FD and 3 FI learners. Obviously only the differences between cluster 2 and cluster 3 can be attributed to FD–I, while any other differences among the four clusters signify that FD–I was not the only variable that contributed to the four-cluster organization of the data. Clearly, learners in cluster 3 (FI learners) had the highest problem-solving performance, spent the longest amount of time in running the model in Model-ItÒ, examined more often the effects of every IV on the dependent variables, and opened consistently all 14 possible meters. On the contrary, learners in cluster 2 (FD learners) had a significantly lower problem-solving performance, spent much less time in running the model in Model-ItÒ, opened consistently less meters than the 14 possible ones, and examined selectively only the effects of mainly IV2 and less frequently the effects of IV1 on the dependent variables, while they almost neglected examining the effects of the other three IVs. The performances of learners in clusters 1 and 4, which included both FD and FI learners, do not provide evidence contradicting the observed differences between FI and FD learners. Besides, in these clusters, FI learners’ mean performance on the HFT (20.05) was just above the score of 20, which was used as the cut-off score for separating FI from FM learners. Obviously, among the 101 students of the initial sample, there were not enough students with high HFT scores close to the maximum possible score of 32. As a matter of fact, the majority of FI learners’ scores on the HFT in these clusters were just

123

544


above the score of 20. The main differences between learners in clusters 1 and 4 was that cluster 1 learners spent a higher amount of time in Model-ItÒ, examined more often the effects of every IV on the dependent variables, and spent a higher amount of time in running the model in Model-ItÒ, than learners in cluster 4. The observed differences between FD and FI learners are strong indications of their different reliance on the context (model in Model-ItÒ) to extract specific meaning or the way they perceive, organize, and process information. More detailed information about learners’ interaction with Model-ItÒ was provided by the four SAL analyses. Nevertheless, the patterns and the frequent sequences of rules obtained for cluster 2 and cluster 3 are more informative. Members in cluster 3 (FI learners) clearly exhibited the ability to successfully explore the affordances of Model-ItÒ with sequences of meaningful actions. They were also very systematic in terms of opening and organizing all 14 possible meters on the screen, and in terms of examining the effects of one IV at a time on the dependent variables, while only few of their actions appeared to be pointless. Most attempts represented well-executed sequences of six actions (the upper limit of items in each sequence of actions) involving one independent variable at a time leading to meaningful observations. Although they attempted to examine the effects of several combinations of more than one variable, they never actually ran the model in order to change the values of multiple IVs concurrently. Thus, the majority of the students in cluster 3 exhibited combinatorial reasoning and the ability to control variables. They also navigated the model with ease, and examined carefully all subparts of the model before making a decision. In general, learners in cluster 3 exhibited complex problem-solving capabilities and it was only in a few cases that some FI learners performed unnecessary actions or sequences of actions that did not provide any useful outcomes. This finding, though, can be explained by the fact that in this study the number of items in each sequence of actions was limited to six. Thus, it could be the case that sequences that were considered non-productive in the study herein were actually meaningful and useful, because in reality they were longer than six items in such a way that any additional actions in the sequences led to useful observations eventually. In other words, having limited the rule sequences to only six items might have resulted in loss of useful data in the present study. On the contrary, the SAL analysis for cluster 2, which contained only FD learners, clearly showed that learners in cluster 2 mainly wondered around in Model-ItÒ, and were uncertain and doubtful about their actions. Their computer interactions were mostly meaningless repetitions of the same action or sequence of actions. Meaningful sequential actions were infrequent, and the results did not show a systematic effort for the purpose of examining the effects of the IVs on the dependent variables. For example, while students in this cluster often ran the model, it was only in a few cases that they were able to extract useful information from the outcomes. In addition, learners in cluster 2 did not open all meters of the model, they never organized the model efficiently on the screen, and they did not demonstrate combinatorial reasoning as well as the ability to control variables. In general, learners in cluster 2 appeared to be caught into a loop of repeating meaningless actions, while most of their frequent sequences were isolated single actions that were probably pursued for exploratory purposes. Nevertheless, their less frequent sequences contained only up to five items and represented some meaningful sequential actions that provided limited information. The SAL analysis for cluster 1 and cluster 4 also provided useful information and interesting insights of how learners attempted to explore the affordances of Model-ItÒ in order to inform their complex problem-solving effort. For example, learners in cluster 1, which contained 9 FD and 3 FI learners, exhibited a rather mechanistic approach of testing

123


545

and retesting the model using the same variables (mainly the first three IVs). They also exhibited behaviors similar to learners in cluster 2, that is, they did not open all possible meters on the screen, did not always organize the meters on the screen in a helpful way, and did not always control variables. Similarly, learners in cluster 4, which contained 8 FD and 8 FI learners, performed most of the time single actions, or sequences of actions that were not useful or productive. In conclusion, the cumulative evidence from the four SAL analyses clearly demonstrated that FD learners were not able to manage the complexity of the complex problem effectively, and their sequences of actions showed how lost FD learners were in the problem-solving space. This was evident from the fact that most FD learners did not even realize that they had to run the model by first opening all meters in order to test hypotheses by controlling variables. Those FD learners who ran the model did so by opening only a subset of the meters, an inappropriate tactic for testing a model since all meters needed to be open to observe how a change in one variable affected all other variables in the system. Obviously, the complexity of the system was overwhelming for the FD learners to manage effectively. FI learners, even though not all of them, appeared to be systematic in terms of testing hypotheses by changing the value of one IV at a time. Their sequences of actions showed that FI learners had a plan in mind and a strategy of how to collect data in order to solve the problem. Overall, the results of the cluster analysis in combination with the results of the SAL analyses clearly show that in addition to the variable of FD–I there are also other variables that seem to affect learners’ ability to cope with complex problems, such as for example the ability to control variables in a complex multivariate system.

Concluding remarks In the study herein, EDM methods provided useful information for explaining in detail how learners with different cognitive styles interacted with a computer tool to solve a complex problem. Specifically, EDM methods were useful in terms of identifying productive and meaningful computer interactions, as well as powerful in terms of discovering problematic sequences of problem-solving actions. Evidence from the study strongly suggests that the superior complex problem-solving performance of FI learners was connected to the sequences of computer interactions they employed, and that certain sequences or patterns of computer interactions were closely linked to FI learners’ better complex problemsolving performance, a finding that corroborates the results reported by other researchers (e.g., Schwering 1987; Dragon 2009; Burnett 2010) who also concluded that cognitive style was a significant variable in complex problem solving with computer tools. Obviously, learners and tools can work together as effective joint cognitive systems, provided that the cognitive characteristics of the learners are compatible with the corresponding characteristics and affordances of the tools (Dalal and Kasper 1994; Brezillon and Pomerol 1997). In this study, complex problem solving was viewed as the interaction between a problem solver and a complex problem in the context of a specific computerbased environment. The problem itself was represented in terms of the barriers that existed between a given state and a goal state, and the transition from the given state to the goal state was constrained by the problem solver’s knowledge and information processing capabilities, as well as by the affordances of the tool that was used to solve the problem. The findings of the study regarding learners’ actual interactions with the affordances of Model-ItÒ to solve the problem, are important, because they enable us to better understand the partnership between tools and humans during complex problem solving, and inform

123

546


research efforts in terms of finding ways to make this partnership between tools and humans as optimal as possible. Thus, ongoing investigations regarding the role of cognitive style on learners’ complex problem-solving performance with computer tools are important. Future investigations in this area should be carefully designed in order to avoid some of the limitations of the present study. In essence, future studies can greatly benefit from larger sample sizes, selection of FI learners with much higher HFT scores than those used in this study, inclusion of FM learners in the analyses, and employment of other EDM methods as well. Evidence from such investigations can provide additional support to the initial conclusions of the present study, and can be used to inform the design of effective joint cognitive systems by providing useful information about how learners with different cognitive styles problem solve with computer tools in specific complex contexts. Lastly, the authors also support the point of view that educational researchers should invest time and effort in designing and developing adaptive instructional systems in order to provide personalized learning experiences taking into consideration learners’ different cognitive needs. Acknowledgments We thank Constantinos Marinis for the extensive discussions we had with him about EDM methods.

References Angeli, C., & Valanides, N. (2004). Examining the effects of text-only and text-and-visual instructional materials on the achievement of field-dependent and field-independent learners during problem-solving with modeling software. Educational Technology Research and Development, 52(4), 23–36. Angeli, C., Valanides, N., & Kirschner, P. (2009). Field dependence–independence and instructional design effects on learners’ performance with a computer modeling tool. Computers in Human Behavior, 25, 1355–1366. Ayres, P., & Sweller, J. (2005). The split attention principle in multimedia learning. In R. Mayer (Ed.), The Cambridge handbook of multimedia learning (pp. 135–146). New York: Cambridge University Press. Baker, R., & Yacef, K. (2009). The state of educational data mining in 2009: A review and future visions. Journal of Educational Data Mining, 1(1), 3–17. Bramer, M. (2007). Principles of data mining. London: Springer. Brezillon, P., & Pomerol, J. C. (1997). Joint cognitive systems, cooperative systems and decision support systems: A cooperation in context. Proceedings of the European Conference on Cognitive Science, Manchester (pp. 129–139). Burnett, W. C. (2010). Cognitive style: A meta-analysis of the instructional implications for various integrated computer enhanced learning environments. Doctoral dissertation thesis: Indiana University of Pennsylvania, Indiana, PA. Chandler, P., & Sweller, J. (1991). Cognitive load theory and the format of instruction. Cognition and Instruction, 8, 293–332. Chandler, P., & Sweller, J. (1996). Cognitive load while learning to use a computer program. Applied Cognitive Psychology, 10, 1–20. Chen, S., & Macredie, R. D. (2004). Cognitive modeling of student learning in web-based instructional programs. International Journal of Human–Computer Interaction, 17, 375–402. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum. Dalal, K. P., & Kasper, G. M. (1994). The design of joint cognitive systems: Effect of cognitive coupling on performance. International Journal of Human–Computer Studies, 40, 677–702. Daniels, H. L., & Moore, D. M. (2000). Interaction of cognitive style and learner control in a hypermedia environment. International Journal of Instructional Media, 27, 369–384. Danner, D., Hagemann, D., Holt, D. V., Hager, M., Schankin, A., Wu¨stenberg, S., et al. (2011). Measuring performance in a complex problem-solving task: Reliability and validity of the tailorshop simulation. Journal of Individual Differences, 32, 225–233.

123


547

Davis, J. K. (1991). Educational implications of field dependence–independence. In S. Wapner & J. Demick (Eds.), Field dependence–independence: Cognitive styles across the lifespan (pp. 149–175). Hillsdale, NJ: Erlbaum. Dragon, K. (2009). Field dependence and student achievement in technology-based learning: A metaanalysis. Master thesis: University of Alberta, Edmonton, AB. Feltovich, P. J., Coulson, R. L., & Spiro, R. J. (2001). Learners’ (mis)understanding of important and difficult concepts. In K. D. Forbus & P. J. Feltovich (Eds.), Smart machines in education: The coming revolution in educational technology (pp. 349–375). Menlo Park: AAAI/MIT Press. Fischer, A., Greiff, S., & Funke, J. (2012). The process of solving complex problems. Journal of Problem Solving, 4(1), 19–42. French, J. W., Ekstrom, R. B., & Price, L. A. (1963). Kit of reference tests for cognitive skills. Princeton, NJ: Educational Testing Services. Frensch, P. A., & Funke, J. (Eds.). (1995). Complex problem-solving: The European perspective. Hillsdale, NJ: Erlbaum. Funke, J. (2001). Dynamic systems as tools for analysing human judgment. Thinking and Reasoning, 7, 69–89. Funke, J., & Frensch, P. A. (2007). Complex problem-solving: The European perspective: 10 years after. In D. H. Jonassen (Ed.), Learning to solve complex scientific problems (pp. 25–47). Hillsdale, NJ: Erlbaum. Goodenough, D. R., & Karp, S. A. (1961). Field dependence and intellectual functioning. Journal of Abnormal and Social Psychology, 63, 241–246. Jonassen, D. H., & Grabowski, B. L. (1993). Handbook of individual differences, learning and instruction. Hillsdale, NJ: Erlbaum. Khine, M. S. (1996). The interaction of cognitive styles with varying levels of feedback in multimedia presentation. International Journal of Instructional Media, 23, 229–237. Lambert, T. (1981). Effects of structure in pre-instructional strategies on memory for sentences in field dependent individuals. Unpublished doctoral dissertation, University of Southern California. Dissertation Abstracts International, 42, 1063. Liu, M., & Reed, W. M. (1994). The relationship between the learning strategies and learning styles in a hypermedia environment. Computers in Human Behavior, 10, 419–434. Metcalf, J. S., Krajcik, J., & Soloway, E. (2000). Model-It: A design retrospective. In M. J. Jacobson & R. B. Kozma (Eds.), Innovations in science and mathematics education (pp. 77–115). Hillsdale, NJ: Erlbaum. Morgan, H. (1997). Cognitive styles and classroom learning. Westport, CT: Praeger. Munk, M., & Drlik, M. (2011a). Impact of different pre-processing tasks on effective identification of users’ behavioral patterns in web-based educational systems. Procedia Computer Science, 4, 1640–1649. Munk, M., & Drlik, M. (2011b). Influence of different session timeouts thresholds on results of sequence rule analysis in educational data mining. Communications in Computer and Information Science, 166, 60–74. Munk, M., Kapusta, J., & Svec, P. (2010). Data preprocessing evaluation for web log mining: Reconstruction of activities of a web visitor. Procedia Computer Science, 1, 2273–2280. Narayanan, N. H., & Hegarty, M. (1998). On designing comprehensible interactive hypermedia manuals. International Journal of Human–Computer Studies, 48, 267–301. Nisbet, R., Elder, J., & Miner, G. (2009). Handbook of statistical analysis and data mining applications. New York: Elsevier. Penner, D. E. (2000/2001). Cognition, computers, and synthetic science: Building knowledge and meaning through modeling. Review of Research in Education, 25, 1–36. Reiff, J. (1996). At-risk middle level students or field dependent learners. Clearing House, 69, 231–234. Rittschof, K. A. (2010). Field dependence–independence as visuospatial and executive functioning in working memory: Implications for instructional systems design and research. Educational Technology Research and Development, 58, 99–114. Sabelli, N. H. (2006). Complexity, technology, science, and education. The Journal of the Learning Sciences, 15(1), 5–9. Salomon, G. (1991). Transcending the qualitative–quantitative debate: The analytic and systemic approaches to educational research. Educational Researcher, 20(6), 10–18. Schwering, R. E. (1987). Modeling supervisory judgment: Theoretical integration and systems evaluation. Unpublished doctoral dissertation, University of Kansas, Kansas. Siemens, G., & Baker, R. S. (2012). Learning analytics and educational data mining: Towards communication and collaboration. Vancouver: Paper presented at the International Conference on Learning Analytics & Knowledge.

123

548


Snowman, J., & Biehler, R. (2003). Psychology applied to teaching (10th ed.). Boston, MA: Houghton Mifflin Company. Stahl, G., Koschmann, T., & Suthers, D. (2006). Computer-supported collaborative learning: An historical perspective. In R. K. Sawyer (Ed.), Cambridge handbook of the learning sciences (pp. 409–426). Cambridge, MA: Cambridge University Press. Stratford, S. J., Krajcik, J., & Soloway, E. (1998). Secondary students’ dynamic modeling processes: Analyzing, reasoning about, synthesizing, and testing models of stream ecosystems. Journal of Science Education and Technology, 7, 215–234. Sweller, J., van Merrie¨nboer, J., & Paas, F. (1998). Cognitive architecture and instructional design. Educational Psychology Review, 10, 251–296. Tannenbaum, R. K. (1982). An investigation of the relationship(s) between selected instructional strategies and identified field dependent and field independent cognitive styles as evidence among high school students enrolled in studies of nutrition. Unpublished doctoral dissertation, St. John’s University. Dissertation Abstracts International, 43, 68. van Merrienboer, J. J. G., & Kirschner, P. A. (2007). Ten steps to complex learning. Mahwah, NJ: Erlbaum. van Merrienboer, J. J. G., Kirschner, P. A., & Kester, L. (2003). Taking the load off the learner’s mind: Instructional design for complex learning. Educational Psychologist, 38(1), 5–13. Witkin, H. A., Moore, C. A., Goodenough, D. R., & Cox, P. W. (1977). Field dependent and field independent cognitive styles and their educational implications. Review of Educational Research, 47, 1–64. Witten, I. H., Frank, E., & Hall, M. A. (2011). Data mining: Practical machine learning tools and techniques. New York: Elsevier.

Charoula Angeli studied at Indiana University in Bloomington, Indiana, USA (BS in Computer Science, MS in Computer Science, Ph.D. in Instructional Systems Technology). During the academic year 1998–1999 she was a postdoctoral fellow at the Learning Research and Development Center (LRDC) at the University of Pittsburgh. She is currently Associate Professor of Instructional Technology in the Department of Education at the University of Cyprus in Cyprus. Her research interests focus on the use of computers as mindtools, technological pedagogical content knowledge, teacher training, complex problem solving, learning and teaching analytics, and the design of adaptive learning environments. She has published extensively in well-respected referred journals and has participated in numerous research projects. In 2011 and 2012, she received the AERA/TACTL Outstanding Paper Award. Nicos Valanides is Professor of Science Education at Frederick University in Cyprus, and Director of the Internal Evaluation and Continuous Development Unit (IECDU) at Frederick University as well. He has undergraduate studies at the Aristotelian University of Thessaloniki (B.A. in Physics, B.A. in Law, 1985), and graduate studies at the American University of Beirut (Teaching Diploma, M.A. in Education: Teaching Sciences) and at the University of Albany, State University of New York, SUNY–Albany (M.Sc in Instructional Supervision, Ph.D. in Curriculum and Instruction and Educational Research). His research interests include teacher training, methodology of teaching and curricula for science education, the development of scientific reasoning and epistemological beliefs, complex problem solving, science-andtechnology literacy, the utilization of ICT in science education, blended learning, and the design of educational interventions and learning environments. He has published extensively in well-respected referred journals and has participated in numerous research projects. In 2011 and 2012, he received the AERA/TACTL Outstanding Paper Award.

123