Task-Adapted Concept Map Scaffolding to Support Quizzes in an Online Environment Thushari Atapattu, Katrina Falkner, Nickolas Falkner School of Computer Science, University of Adelaide, Australia {thushari.atapattu, katrina.falkner, nickolas.falkner}@adelaide.edu.au have been widely used in the educational context, particularly in supporting learners in adopting new concepts, and also in assisting learners to identify misconceptions and knowledge gaps through comparison with expert concept maps, and further gains in assisting knowledge organization, planning, and scaffolding purposes [1].
ABSTRACT This paper investigates the effect of different forms of concept maps as scaffolding techniques to support answering quizzes in an online learning environment. Concept maps which represent a course topic have being utilised as a scaffolding technique for learning the subject matters and problem solving. However, due to the typical amount of information presented in the topic concept maps, learners might feel overwhelmed, reducing their motivation and increasing the learners’ disorientation. In order to overcome this issue, a study was conducted with 59 undergraduates of a Software Engineering course to measure the effect of different forms of concept maps on learning. The study obtained statistically significant results when using concept maps adapted to given quizzes (known as task-adapted concept maps). Students’ reflections collected through a questionnaire were very positive towards task-adapted concept maps as a scaffolding technique.
Concept maps are particularly well suited to Computer Science, due to the diverse and complex range of concepts covered within the curricula, which can be well supported through various concept map activities including drawing concept maps by students or utilising expert constructed maps for scaffolding. Berges and Hubwieser [3] explore the semi-automated extraction of concept maps from textbooks to assist in Introductory Objectoriented Programming course; Sanders et al. [4] explore the use of manually constructed concept maps within an Introductory Object-oriented Programming course; Calvo et al. [5] adopt concept maps within an intermediate Database Principles courses to facilitate collaboration and understanding; and LarrazaMendiluze and Garay-Vitoria [6] explore an automatic comparison of manually generated concept maps within the area of Computer Architecture.
Categories and Subject Descriptors K.3.2 [Computing Milieux]: Computer Science Education – Computer and Information Science Education
General Terms
This research utilises concept maps generated from Computer Science lecture slides using a framework (known as concept map mining framework) developed by the authors for the purpose of scaffolding. The automated process of extracting concept-relationconcept triples to produce concept maps using Natural Language Processing (NLP) techniques and the evaluation of the framework with human experts has been broadly discussed in authors’ previous works [7, 8]. The particular interest of this paper is to demonstrate the usage of auto-generated concept maps as a scaffolding technique to facilitate answering quizzes in an online environment. Quizzes in this context are similar to formative or summative questions provided through online learning environments (e.g. Learning Management Systems) to engage or motivate students, guide learning, provide opportunity for practice and self-assessment [9]. Scaffolding is the support given during the learning process with the intention of helping the student achieves his/her learning goals. The concept of ‘scaffolding’ was originally introduced in the context of adults assisting children in acquiring knowledge or solving problems in informal learning environments [10]. Scaffolding is grounded in the Social Constructivism Theory of Vygotsky [11] and his popular concept known as the Zone of Proximal Development (ZPD) [12].
Human Factors
Keywords Concept map, scaffolding, lecture slide, meaningful learning, quizzes
1. INTRODUCTION Concept mapping is recognised as a valuable educational visualisation technique, which assists students in organising, sharing and representing knowledge. The use of concept maps as a way of supporting learning is well established [1]. According to the cognitive learning theory of Ausubel, which states “learning takes place by the integration of new concepts and propositions into existing concept and propositional framework held by the learner”, there has been shown to be a significant increase in meaningful learning when using concept maps [2]. Concept maps Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from
[email protected]. ITiCSE’15, July 4–8, 2015, Vilnius, Lithuania Copyright is held by the owner/author (s). Publication rights licensed to ACM ACM 978-1-4503-3440-2/15/07… $15.00 http://dx.doi.org/10.1145/2729094.2742634
Previous studies demonstrated that concept mappers who construct or utilise auto-generated concept maps possess significant improvements in problem solving, recall and conceptual knowledge [13, 14]. However, the existing approaches provide concept maps that represent a domain or topic (named topic concept maps) as scaffolding. Studies found that providing a topic concept map increases the risk of information overload which therefore rises the anxiety, stress, alienation and learning
272
disorientation among learners [15, 16]. This research specifically focuses on providing the most relevant information to answer quizzes. More precisely, the concept maps utilised as scaffolding in this work will be adapted to the given quiz. This process is known as ‘task-adapted knowledge organisation’. According to Novak and Canas [1], concept maps are more effective when they are produced to answer a question (known as a focus question) rather than using them to represent general knowledge in a domain or topic.
domain-specific knowledge is not viewed as a dichotomy, rather, both of them are important when performing reasoning tasks. Thus, it is understood that content-based scaffolding has benefits to improve the learning experience, however, it is challenging when performed manually, and hence, this paper provides the first attempt, according to the authors’ knowledge, that task-adaptation can be achievable through an automated approach.
3. METHODOLOGY In this research, we undertook a mixed method of data collection, combining quantitative and qualitative techniques.
This research investigates following two research questions. 1.
Does task-adapted concept map as scaffolding improve learning gain over other forms of scaffolding?
2.
What are the students’ opinions of concept mapping and scaffolding?
3.1 Experimental Design In our randomised experimental design, 59 students, aged over 18 years, who were enrolled in an Introductory Software Engineering course in University of Adelaide, participated in a computer-based experimental study and a post-test. The Software Engineering course, a core Computer Science course, contains a large set of new concepts that students are required to assimilate in their second year of undergraduate studies. Software testing topic was selected for the experiments. In addition to the concept-wise importance of Software Engineering course, the concept maps generated from this course demonstrated strong positive correlation (rs = .83) with human generated maps in a previous study of algorithm evaluation [8].
In this paper, we describe a randomised experiment to explore the effectiveness of different forms of concept map-based scaffolding in assisting learners in answering quizzes. In our study, we have found a statistically significant benefit when using task-adapted concept maps, in comparison to topic concept maps and lecture slides as scaffolding support. We undertook a further qualitative study on student reflections on their usage of various scaffolding techniques. Analysis of these qualitative reflections indicate a strong preference for task-adapted concept maps as a scaffolding technique, particularly for information- or theory-rich courses, but had no support for manual construction of said concept maps.
Ten multiple choice questions (MCQs) were constructed for the study which covered 68% of the important concepts in the ‘software testing’ topic. During the study, participants were expected to attempt 10 quizzes within 20 minutes through a webbased prototype. If attempted answer was incorrect, students were given an option to get ‘help’ through the scaffolding system (see Figure 1).
2. RELATED WORK Concept map-based scaffolding can be provided in many ways, among them, learner model-based scaffolding is one of the widespread techniques to offer personalised learning experience according to learners’ current knowledge and preferences. Recent research of concept map-based adaptive scaffolding can be found in the works of JSEM-HP [17], Chen et al. [18], and VLS-CCM [19].
The help was specific to the each group. The control group received lecture slides (LS) as a PDF file. This file had fewer slides than the original slide set. A subset of the original slide set which was related to each quiz was extracted manually. This process reduced the disadvantage faced by the control group having more content than the treatments. Treatment group 1 received the topic concept maps generated from lecture slides (CMap – Figure 3(a)). Treatment group 2 received the same concept maps as group 1; however, in their maps, the context to answer the quiz is highlighted (HLCMap – Figure 2). The highlighting process was performed manually using the IHMC CMap tools [24]. The aim of introducing the HLCMap group is to measure whether there is any effect of having scaffolding in between topic concept maps and task-adapted concept maps. Treatment group 3 received task-adapted concept maps (TSKCMap – Figure 3(b)) extracted from task-adapted scaffolding framework [25]. All the scaffolding resources were stored as image files or as PDF files.
Additionally, scaffolding can be provided based on a particular task or activity by emphasising the specific content knowledge required, reducing the information overload and increase learners’ motivation. However, this approach is not adopted widely in literature due to the excess of manual workload involved. For instance, Bulu and Pedersen [20] investigated the effect of domain-general and domain-specific scaffolds on ill-structured problem solving. Domain-specific scaffolds include specific content knowledge to use during problem solving and domaingeneral scaffolds are otherwise. A total of 332 students were allocated into groups and provided scaffolds through domainspecific and domain-general with either continuous or faded support. The results showed that the continuous domain-specific scaffolds outperformed the other conditions on the post-tests. A similar study with 578 middle school students showed curricular scaffolds that focus on the content and task (context-specific) are significantly better than generic explanation scaffolds in terms of students’ improvement and understanding of scientific content [21]. Eylon and Reif [22] suggested that “higher levels of the hierarchy should preferentially contain information most important for the domain of tasks”. This hypothesis was tested by comparing the effectiveness of two hierarchical knowledge organisations that contained the same knowledge, with one of them adapted to a set of given tasks. The results demonstrated that the task-adapted information group performed significantly better than the control group in the performance tasks. However, according to Perkins and Salomon [23], domain-general and
Once the participants completed the required learning through scaffolding, they were expected to go back to the initial question for which they requested help. However, the answers were shuffled at this stage to reduce guessing. This process could be repeated any number of times until the required study task is completed.
273
previous work [25]. The process of extracting task-adapted concept maps is shown in Figure 3. Question: Identify the interface types in component testing Question triples: (component testing, interface types, ?)
Figure 1: Screen shot of a quiz using web-based prototype
(a)
Figure 2: A subset of HLCMap for the question ‘Compare and contrast system testing and release testing’ After the main study, students were requested to complete a questionnaire reflecting their issues, if any, they encountered, their opinion about the system, new features they wanted to see and their willingness to use the system for future studies. The questionnaire contained 10 questions with a combination of 5point Likert scale (ranging from ‘strongly disagree’ to ‘strongly agree’) and open-ended questions. Students in the control groups who have not seen concept maps received a slightly different version of the original questionnaire.
(b) Figure 3: The task-adapted concept map scaffolding process (a) sub-set of a topic concept map (CMap) (b) task-adapted concept map (TSKCMap)
3.3 Quantitative Analysis
Participants who were involved in the first stage of the experiments were invited to participate in the post-tests with a two week gap. The paper-based, ten post-test questions included combinations of MCQs, fill-in-the-blanks and open-ended questions to minimize the opportunity to guess, allowing the actual learning gain between pre- and post-tests. Post-test included new questions as well as similar questions as experimental study, but they were rephrased and shuffled to reduce the possibility of memorisation.
In order to answer the first research question ‘Does task-adapted concept map as scaffolding improve learning gain over other forms of scaffolding?’, we conducted one-way ANOVA by comparing pre- and post-test scores. The response to the first attempt of each question was recorded as students’ prior knowledge on the topic (i.e. pre-test). Even though 59 students participated for the main study, only 30 students were able to participate to the post-tests. The descriptive statistics of students’ learning gain across different scaffolding groups are shown in Table 1. The scaffolding groups are indicated as LS – lecture slide, CMap – topic concept map, HLCMap – concept maps with context to answer the quiz is highlighted, and TSKCMap – taskadapted concept map in Table 1.
3.2 Task-adapted Scaffolding Framework In order to provide concept map-based scaffolding, our work developed a framework with the use of NLP techniques. Each question was processed through a question analyser to identify the question type (e.g. descriptive, comparison) and an annotator to extract noun-verb-noun triples (question triples), in order to extract task-adapted concept maps using the topic concept maps as a basis. The framework is broadly discussed in authors’
274
1. Table 1. Descriptive statistics of learning gain Group LS
Mean 2.5
Median 2.0
Standard deviation .755
n 7
CMap
3.2
1.4
.532
7
HLCMap
3.6
2.0
.777
7
TSKCMap
5.0
1.5
.496
9
Total
4.0
1.9
.351
30
What do you think about concept maps/lecture slides used in this study to answer questions? According to the statistics, students in the treatment groups were very positive about having concept maps as scaffolding. Among the treatment groups, 77%, 86% and 73% stated it was either ‘helpful’ or ‘very helpful’ while 30% of control group (LS) mentioned that it was difficult to utilise lecture slides to answer quizzes.
2.
According to the results, TSKCMap group had the numerically highest mean (M = 5.0, SD = 1.5, n = 9) while the control group (LS) had the smallest mean (M = 2.5, SD = 2.0, n = 7).
Which form of resources do you prefer if our system made available through CS Forums (Learning Management System) in future?
The three assumptions of ANOVA – test of homogeneity of variances, normal distribution of dependent variable, and independence of observation were not violated. Table 2 illustrates the results of mean comparison within and between groups. Table 2. Summary results of ANOVA
Between Groups Within Groups Total
Sum of Squares 28.292 79.008 107.300
df
Mean square
F
Sig.
3 26 29
9.431 3.039
3.103
.044
Figure 4. Students’ preferred form of scaffolding
Results of the one-way ANOVA indicated that the means between groups are significant; F(3,26) = 3.103, p = .044 (< .05), η2 = .263. The results of the post-hoc test indicated that the students in the task-adapted concept map scaffolding group demonstrated statistically significant learning gain compared to the students who received lecture slides or topic concept maps as scaffolding. However, the results were not statistically significant between TSKCMap and HLCMap groups [25].
The results showed that the students are extremely receptive to the use of concept maps and lecture slides including 80%, 80%, 54% and 82% in the LS, CMap, HLCMap and TSKCMap groups respectively. It appears students believe utilising concept maps generated from lecture slides would be useful as a supplement to traditional learning approaches. In addition, none of the participants out 55 who answered this question were interested in constructing the concept maps manually. This feedback can be used to support the rationale of providing students with autogenerated concept maps as scaffolding for learning.
3.4 Qualitative Analysis A comprehensive qualitative study was conducted to investigate the students’ reflections on the research. Prior to collecting the opinions, statistics about students’ prior experience on knowledge organisation techniques were collected (Table 3).
3.
Table 3. Students’ prior experience on knowledge organization techniques Prior experience No prior experience Have heard, but not used Previously used
Currently using
Technique
Concept maps Mind maps like inspiration Semantic networks Simple brainstorming ideas Concept maps Mind maps like popplet Knowledge maps
Would you like a tool which can extract partial concept maps to assist answering questions?
Number of students 10 11 11 21 1 1 3 10 1
According to the statistics, 62% of participants had previously used or are currently using, some form of knowledge organisation techniques.
Figure 5: Students’ opinion on task-adapted concept maps as scaffolding to answer questions The results indicated that the majority of the students in each group either agreed or strongly agreed to incorporate task-adapted concept maps including 80%, 80%, 67% and 71% in the LS, CMap, HLCMap and TSKCMap groups respectively.
275
4.
Which type of courses do you think this kind of tool will be more useful?
Statistics showed that majority of the participants in the LS group (89%) and the CMap group (83%) preferred to have concept maps for courses with less programming components or almost every course. In addition, 90% participants in the HLCMap group preferred to have concept maps for courses like Software Engineering and Operating Systems and 79% in the TSKCMap group chosen to have concept maps for every course. Some participants mentioned that they would like to see concept maps for the courses with more facts or heavy theory or concepts, emphasising “based around memory information courses, courses with heavy theory/concepts which interconnects with each other”. The feedback provides evidence that it would be beneficial for students to have concept maps generated from lecture slides to see the interconnections between concepts. Previous studies provided evidence that the knowledge organisation techniques are more effective than sequentially-arranged lecture slides [26, 27]. 5.
HLCMap
TSKCMap
From the feedback collected from the CMap and HLCMap groups, it is evident that there should be a mechanism to differentiate concepts that are more relevant to the context. Since, these two groups had no idea about task-adapted concept maps, they repeatedly mentioned the requirement of colour codes, ‘search’ option, zoom in/out or smaller/partial maps that focus on the more relevant information to learning (see highlighted comments in Table 5). In addition, all the groups were interested to have more details on specific concepts or relations by clicking them.
Write down any issues you had while interacting with concept maps Table 4: Pros and cons about concept maps as scaffolding
Group
CMap
HLCMap
TSKCMap
Feedback (Pros and cons) Too much information in concept map Some questions had no correct answer in the concept map Problems I had with the concept maps were that they were kind of bland. As more and more information is added to a concept map, it can be difficult to navigate Larger maps are more difficult to read Some questions required more information than showing Concept maps were useful for hints Some maps did not easily show the information needed to answer the question Question 10’s concept map was basically useless with help answering the question Not enough information provided in the concept tree
4. CONCLUSIONS This paper presented an approach to adopting the different forms of auto-generated concept maps as scaffolding to facilitate answering quizzes in an online environment. Several previous studies explored similar approaches of adopting domain-specific or domain-general scaffolds to improve the learning experience. However, the widespread adoption of content-based scaffolding is hindered by the high workload required to construct scaffolds corresponds to given tasks. As a solution, this paper proposed an automated approach. The concept maps generated from this work was compared with other forms of concept maps (topic concept maps or topic concept maps with the context required to answer the quiz is highlighted) and lecture slides (control) and the results illustrated a statistically significant difference between topic concept maps and lecture slides on the post-tests. In addition, students were extremely receptive on utilising concept maps as scaffolding as a supplementary to lecture slides, emphasising the ability to perceive inter-connections between concepts. The students who received topic concept maps expressed the importance of having less information in their maps or mechanism to differentiate most important information for context using colour codes.
According to the students’ comments, it is correct that some concept maps did not have adequate information to answer all questions. The particular concepts and their relationships to learn the specific skills were included, but not the direct answer. In contrast, participants in the first two treatment groups (CMap and HLCMap) criticised about excess of information in topic concept maps. 6.
What are the features you would like to suggest to improve our tool? Table 5: Suggestions for improvements of the system
Group
CMap
densities of information Colour code hierarchies or smaller maps In order to fix the problem of difficulty in navigating larger concept map, suggest to use colour to differentiate different components and branches Ability to search within the concept map, this would allow it to be useful for specific information or question rather than the full overview More easier to search A feature in which you can click on a concept to retrieve more information Add extra notes More details in relation labels Allow viewing the concept map at any time, not just when answering the question More explanations in concept maps It was quite good Being able to select on elements and have connected or related information and their paths highlighted Click on a specific concept or relations could bring up more details on that specific section
Suggestions for improvement Removing timer or making optional, gives more time to think and less pressure Use colour for help identifying important sections Improve appearance by providing partial concept maps that applies to topic Perhaps be able to view the concept maps before answering the question Need colours and switches Less concepts The concept map can be improved more to have zoom in/out to smaller or larger sections Perhaps a way to toggle between maps of higher and lower
As a future work, current study can be expanded using a larger student cohort and multiple CS courses. As per the suggestions of students’ through the questionnaire, an alternative form of taskadapted concept map can be introduced as a future work, in order to embed resources to concepts including underlying lecture slides or external resources such as multimedia. Additionally, there is a motivation to explore the effect of student-constructed taskadapted concept maps using an auto-generated topic concept maps as a basis. Within this study, we expect to investigate whether
276
students can identify the parts of a topic concept map that would be associated with a specific question. 15.
5. REFERENCES 1.
2. 3.
4.
5.
6.
7.
8.
9. 10. 11. 12. 13. 14.
Novak, J. D. and Canas, A. J. The Theory underlying Concept maps and How to construct and use them. Florida Institute for Human and Machine Cognition, Pensacola FI, 2006. Ausubel, D., Educational Psychology: A Cognitive View. Halt, Rinehart and Winston, New York and Toronto, 1968 Berges, M. and Hubwieser, P., Concept specification maps: displaying content structures. in Proceedings of the 18th ACM conference on Innovation and technology in computer science education, (Canterbury, UK, 2013), 291-296. Sanders, K., et al., Student understanding of object-oriented programming as expressed in concept maps. in Proceedings of the 39th SIGCSE technical symposium on Computer science education, (Portland, Oregon, USA, 2008), 332-336. Calvo, I., et al., The use of concept maps in Computer Engineering education to promote meaningful learning, creativity and collaboration. in Frontiers in Education Conference (FIE), (Rapid City, SD, 2011), IEEE. Larraza-Mendiluze, E. and Garay-Vitoria, N., Use of concept maps to analyze students' understanding of the I/O subsystem. in Proceedings of the 13th Koli Calling International Conference on Computing Education Research, (Finland, 2013), 67-76. Atapattu, T., Falkner, K. and Falkner, N., Acquistion of Triples of Knowledge from Lecture Notes: A Natural Langauge Processing Approach. in The 7th International Conference on Educational Data Mining, (London, UK, 2014), 193-196. Atapattu, T., Falkner, K. and Falkner, N., Evaluation of Concept Importance in Concept Maps Mined from Lecture Notes: Computer vs Human. in The 6th International Conference on Computer Supported Education, (Barcelona, Spain, 2014) Dillon, J. T. Questioning and Teaching: A manual of practice. Teachers College Press, New York, 1988. Wood, D., Bruner, J. S. and Ross, G. The role of tutoringin problem solving. Journal of Child Psychology and Psychiatry, 17(2), 89-100. Vygotsky, L. S. Mind in society: the development of higher psychological process. Harvard University Press, Cambridge, MA, 1978. Beed, P. L., Hawkins, E. M. and Roller, C. M. Moving Learners toward Independence: The Power of Scaffolded Instruction. The Reading Teacher, 44(9), 1991, 648-655. Okebukola, P. A. Can Good Concept Mappers be Good Problem Solvers in Science? Educational Psychology, 12(2), 1992, 113-129. Valerio, A. and Leake, D. B., Using Automatically Generated Concept Maps for Document Understanding: A
16.
17.
18.
19.
20.
21.
22. 23. 24. 25.
26. 27.
277
Human Subjects Experiment. in Fifth International Conference on Concept Mapping, (Valletta, Malta, 2012). Dias, P. and Sousa, A. P. Understanding navigation and disorientation in hypermedia learning environments. Journal of Educational Multimedia and Hypermedia, 6(2), 1997, 173-185. Edmunds, A. and Morris, A. The problem of information overload in business organisations: a review of the literature. International Journal of Information Management, 20(1), 2000, 17-28. Molina-Ortiz, F., Medina-Medina, N. and García-Cabrera, L., Applying a Semantic Hypermedia Model to Adaptive Concept Maps in Education. in Proceedings of the 11th International conference on Computer-aided Systems theory, (Las Palmas de Gran Canaria, Spain, 2007), 384-391. Chih-Ming, C., Chi-Jui, P. and Jer-Yeu, S., Ontology-based concept map for planning personalized learning path. in IEEE conference on Cybernetics and Intelligent Systems, (Chengdu, 2008), 1337-1342. Xu, N., et al., The Colored Concept Map and Its Application in Learning Assistance Program, in Proceedings of the 5th International conference on Hybrid Learning, (Guangzhou, China, 2012), 198-209. Bulu, S. and Pedersen, S. Scaffolding middle school students’ content knowledge and ill-structured problem solving in a problem-based hypermedia learning environment. Educational Technology Research and Development, 58(5), 2010, 507-529. McNeill, K. L. and Krajcik, J., Supporting Students' Construction of Scientific Expalnation through Generic versus Context-Specific Written Scaffolds. in Annual meeting of the American Educational Research Association, (San Francisco, 2006). Eylon, B.-S. and Reif, F. Effects of Knowledge Organization on Task Performance. Cognition and Instruction, 1(1), 1984, 5-44. Perkins, D. and Salomon, G. Are Cognitive Skills ContextBound? Educational Researcher, 18(1), 1989, 16-25. Canas, A. J., et al., CmapTools: A knowledge modeling and sharing environment. in First International Conference on Concept Mapping, (Pamplona, Spain, 2004). Atapattu, T., Falkner, K. and Falkner, N., Educational Question Answering motivated by Question-Specific Concept Maps. in 17th International Conference on Artificial Intelligence in Education (AIED 2015), (Madrid, Spain, 2015). Hall, R. H. and O'Donnell, A. M. Cognitive and Affective Outcomes of Learning from Knowledge Maps. Contemporary Educational Psychology, 21(1), 1996, 94-101. Kinchin, I. M., Chadha, D. and Kokotailo, P. Using PowerPoint as a lens to focus on linearity in teaching. Journal of Further and Higher Education, 32(4), 2008, 333346.
