Knowledge Representation and Assessment Using Concept Based ...

Knowledge Representation and Assessment Using Concept Based Learning Nandu C Nair Amrita E-Learning Research Lab Amrita Vishwa Vidyapeetham University Amritapuri,Kollam,India [email protected]

Archana J S Amrita E-Learning Research Lab Amrita Vishwa Vidyapeetham University Amritapuri,Kollam,India [email protected]

Shiffon Chatterjee Amrita E-Learning Research Lab Amrita Vishwa Vidyapeetham University Amritapuri,Kollam,India [email protected]

Kamal Bijlani Amrita E-Learning Research Lab Amrita Vishwa Vidyapeetham University Amritapuri,Kollam,India [email protected]

This approach of aligning knowledge representation and assessment to concepts can be termed Concept Based Learning (CBL) [2]. Existing work [1] in concept-based learning shows the importance of linking facts and skills with conceptual understanding. Concept-based techniques have been applied in a variety of domains, for example, in learning how to resolve conflicts in complex human interactions [3]. Concept-based instruction encourages deep learning as concepts involve certain levels of abstraction and generalizability which call for active participation by the learner in the intellectual process. However, there are challenges involved such as curriculum development, teacher training, and assessment. Our study addresses one of these challenges, concept based assessment.

Abstract— The process of learning can be improved with proper and timely feedback. This paper proposes a system that provides feedback for both teacher and student using conceptbased learning. Various types of concepts are defined, and assessment is done for each concept. The level of knowledge acquired by students can be clearly represented using this approach. The feedback provided is goal-oriented, active, personalized, timely, ongoing and consistent, which counts for the novelty of this approach. Results from this study show that concept-based assessment and feedback motivates the students and helps them improve their examination score. Keywords—concept; concept-based learning;assessment; examination score; knowledge representation.

I. INTRODUCTION

Assessment is a powerful aspect of the instructional process[9]. Feedback is an integral part of assessment which helps both the teacher and the students to make meaningful changes and to improve the teaching-learning process. For feedback to be effective, it should be goal-oriented, actionable, personalized, timely, ongoing and consistent.

One of the key challenges in instruction is to help students reorganize existing knowledge and to assimilate new knowledge [13][18][19][4]. To facilitate knowledge assimilation, information should be presented to the learner in a way that helps learners to make connections between prior and new knowledge and also to relate the facts they learn with real life scenarios. This enables the learner to analyze, evaluate, and form a deep understanding of the information acquired. Concepts are a way to organize knowledge and to make sense of learning. This paper proposes an approach to represent and assess course-level knowledge acquisition on the basis of concepts.

c 978-1-4799-8792-4/15/$31.00 2015 IEEE

The current study proposes a system for assessment of learning at the concept level in order to provide specific, targeted, ongoing feedback on strengths and weaknesses of students’ performance and to motivate them to improve performance in future. Both formative and summative assessment techniques are used in the current study. Formative assessment is performed as an ongoing part of the instructional process [15].

848

Performance of students is continuously monitored in order to identify gaps in the teaching-learning process and to improve upon them. More specifically, formative assessments help students to identify their strengths and weaknesses and to target areas that need work; and teachers to recognize where students are struggling and address problems immediately. The goal of summative assessment is to evaluate student learning at the end of an instructional unit by comparing it against some standard or benchmark. Assessments can be performed using different techniques like concept maps. Concept maps are used to reveal the understanding by organizing the subject matter as a graphical representation. The study, “Using concept maps to reveal understanding a two-tier analysis” [6] by Kinchin, Ian M suggests concept maps as a powerful classroom tool which performs a two tier analysis (qualitative and quantitative) to illustrate the conceptual changes at both the gross and specific levels. A meta-analysis [5] found that concept mapping benefitted students in terms of knowledge retention and transfer across different subjects and educational levels. Assessment of learning with concept maps indicates that use of concept maps in conjunction with feedback [8] tends to improve performance in problem-solving. A related approach is the use of short concept tests [17] to assess student understanding of the key concepts being taught in a class. Higher-order thinking skills [7] can also be measured through a concept-based approach. Our paper discusses an assessment technique where the knowledge structure of each course is classified into core and secondary concepts and student performance is assessed and represented in alignment with the concepts. The performance of the learner can be assessed at different levels - curriculum level, unit level, and learning step level. Curriculum level evaluation [10] is important for any education system. It provides the basis for curriculum policy decisions, for feedback on continuous curriculum adjustments and processes of curriculum implementation. Unit level evaluation [11] helps to find what students have achieved at the end of an instructional unit of a course. Learning step level evaluation [12] evaluates the extent to which students have advanced in skills, knowledge or attitude, that is, the progress of learning which occurs in each step. Methods could range from formal to informal testing to team assessment and self assessment [13] in order to have a global assessment of the student’s learning. Apart from the concept based assessment and feedback, our study also discusses how to represent the in depth knowledge acquisition of a student in a particular course. Visualizing the conceptual understanding of a subject is difficult. We address this issue in our study, by providing a mapping between each concept and score obtained in the examination. Thus one of the challenges tackled is the knowledge representation of the conceptual understanding of a

subject which is provided along with the feedback to the students. Further sections of the study are organized as follows: Section II details the proposed method, Section III outlines the solution approach, and Section IV describes experiments and results. Section V concludes the paper and outlines future work. II. PROPOSED METHOD This section describes our approach of defining a concept, how to categorize a student’s content level and application level knowledge, how to track their performance on each concept for a particular course, and how to visually represent this information in a convenient manner. A. Concept A concept means a general notion or idea. Concepts are the building blocks of a course. For the purpose of this paper, any knowledge, fact or skill is defined as a concept. It can be either a simple topic like Array, or a specific method like Dijikstra’s algorithm. The idea of concepts is not only restricted to higher level of abstraction of the subject content but it can be at unit level also. A typical example will be, for the course Advanced Data Structures and Algorithms, one of the concepts considered is the “data structures”. The concept “data structures” is a higher level of abstraction of concepts like priority queue, linked list etc. The proposed method classifies concepts in a course as core concepts and secondary concepts. A core concept is a fundamental, central concept that is essential and which is directly related to the learning outcomes. Secondary concepts are those which are associated with the core concepts in order to enable the learner to have a better understanding of the core concepts. The Table I list out some of the core and secondary concepts from three courses in M.Tech E-Learning Technologies. For example, the concept neighborhood pixels, classified as secondary concept in the course Digital Image Processing is required to understand one of the core concept boundary recognition. Table1 shows the sample list of core concepts and secondary concepts. For the study, three courses, namely Probability and Statistics for Research, Advanced Data Structures and Algorithms and Computer Graphics and Game Programming from an M.Tech program in E-Learning Technologies were chosen. The concept lists are based on the course syllabus and were prepared after discussing with respective faculty members.

2015 International Conference on Advances in Computing, Communications and Informatics (ICACCI)

849

B. Examination Pattern and Backward Design Setting an examination pattern in correspondence to the learning outcome requires a lot of attention. To check the understanding of a concept at the global level, the evaluation should focus on context part (domain-specific content knowledge) as well as the application part of a concept. While the context part refers to the subject knowledge of the concept, the application part is concerned with how the subject knowledge can be applied in the real world scenario. Testing the subject matter knowledge of the concept involves the use of direct questions like stating definitions, theorems and algorithms. Questions addressing the application part of a concept could involve simple as well as advanced application. Simple application refers to questions which are concerned with applying a single concept while advanced application requires higher levels of thinking involving multiple concepts to work out the solution. The other concepts could be either core or secondary. Consider the following questions to evaluate the concept data type. i.

List out the user defined data types.

ii.

Identify the data type of the result of a code fragment.

iii. Write a program to find the area of a square. The length of the square will be based on the user input. iv. Write a program to output the area of set of squares whose length varies from1to12cm.

TABLE I. Sl. No

1

SAMPLE LIST OF CORE CONCEPTS AND SECONDARY CONCEPTS Courses

Advanced Datastructures Algorithms

and

Core Concepts

Secondary Concepts

Array Priority queue Linked list

Radix Sort

Probability 2

3

850

Probability and Statistics for Research

Computer Graphics and Game Programming

Axioms of probability Discrete random variable Gagne’s principle Story board design Game attribute concepts

Induction Recursion Sampling distribution of mean Basic statistics Point estimation Direct instruction Discovery learning Situated cognition

These are some examples of direct, simple and advanced application questions respectively for the concept data type offered in the course Programming Concepts. Question (i) is direct question, checking the knowledge of the user defined data types. Question (ii) is simple application question checking knowledge of data type identification. Question (iii) is an advanced application level question, checking the concept of different data types. Here the student should be able to think at a higher level in the sense that the user input can be either integer or float or double or any other data types. Hence the program should be capable of handling user inputs of different data types. Question (iv) is also an advanced application level question as the student needs to understand and think for a solution such that the side of the square is an integer value and the program should be able to output the area of set of squares. Here the student needs to be aware of the concept data type as well as the concept of loops. The proposed study focuses on assessment at Context as well as Application part of a concept. Apart from this, the study follows the approach of backward design. It means that we develop the appropriate assessments for a unit of study so that we can give consistent feedback for the development of student learning . Instead of creating an assessment tool to test what was taught, the starting point is to define what students are expected to know, understand, and be able to do and then to assess student performance accordingly and provide feedback to improve performance. C. Concept based Assessment and Feedback In existing education systems, feedback is often limited to publishing of final scores in the examination. The emphasis is on the marks achieved. Often little attention is paid to the conceptual understanding actually achieved by the student. This is the gap which concept-based assessment attempts to address. Research shows how concept maps can be effectively used as tools to assess the organization of students’ knowledge structures [14] ,[16]. Monitoring of the level of understanding of the students’ knowledge in each and every concept is important for the learning process from both the instructor’s and students’ point of view. This can be achieved through concept-based assessment. In the proposed study, concept-wise scores are used to analyze the strengths and weaknesses of the students’ performance. This helps to provide specific feedback which likely to motivate students to perform better in future. D. Knowledge Representation The proposed system also helps the teachers to analyze the effectiveness of the teaching. As the number of students increases it becomes difficult for an instructor to keep track of the understanding of each student on each concept. This is achieved by considering the marks scored for each concept in the examination. This study proposes a method to visually represent the knowledge acquired by the students in the course. If majority of students show difficulty in


understanding a particular concept, then there is a need for the instructor to reflect upon the possible causes and to implement changes as required, in the method of teaching. Thus, there is a scope to provide personalized input for both the instructor and the students through course-level knowledge representation. III. SOLUTION APPROACH This section explains the architecture of the system in section III A and its implementation in section III B. Section III A describes in detail the different phases in the proposed system. To make learning better we require a curriculum which clearly defines the learning outcomes and their relevance in day-to-day life. Hence developing curriculum and courses which are based on concepts is an important step in the proposed approach. In addition to this, there is a need for an assessment system which evaluates student performance as well as motivates them to learn further. Thus the system should be capable of testing student understanding and improving it based on effective feedback. A. System Architecture Fig.1. shows the architecture of the system. The major components of the proposed model are concept based curriculum, concept based course, concept based assessment and database 1) Concept-based Curriculum The learning objectives of the educational program determine the significance of the concepts to be included in a course. The curriculum involves details reading instruction in order to help the learner to attain the goal. The objectives of the course depend on the type of program students taken. So curriculum for computer programming course for software branch and non-software branch vary. It is important to define a curriculum specifying the major topics needed to be learnt for a program based on the goal of the program. The proposed method suggests organizing the set of courses into set of concepts relevant to the educational program. Accordingly, in this study, concepts were defined based on inputs from the curriculum and guidance from the instructors. 2) Concept-based Course The organization of a course in conventional way is modules, units and topics. The proposed method suggests the entire course organized into set of concepts. Conceptbased Curriculum

Conceptbased Course

Conceptbased Assessment

Fig.1. Architecture of the proposed system

Conceptwise mark

Table 2 lists the sample list of concepts from the course Probability and Statistics for Research. 3) Concept-based Assessment The proposed assessment method is used to track the performance of students and thus give feedback to improve the performance in future. The assessment starts with two intermediate examinations conducted during the course. Hence the students benefit as the feedback based on these intermediate examinations is provided to them before the final examination. The procedure for performing the assessment is that the instructor decides the core concepts and prepares the question paper based on the concepts. The proposed system maps the question with the concepts in three ways; direct, simple application and advanced application. Direct question type checks the rote understanding of the concept; the score obtained for this type of question is termed as direct question score. Simple application question check show well the student understands the application of the concept. Advanced application checks higher levels of thinking involving multiple concepts. The cut-off score and the difficulty level is more for advanced application question than the simple application question. The score attained from these types of question are termed as simple application score and advanced applications core respectively. Hence this approach helps to generate an analysis on whether the students have content knowledge or real world knowledge of concepts. The detailed description of assessment is shown in block diagram in Fig.2. The instructors manage the system by inputting the scores obtained by the students into the system database.

TABLE II.

SAMPLE LIST OF CONCEPTS FROM THE COURSES TAKEN Sl. No

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Concepts

Data type Decision control Number conversion Operators Personalized system of instruction Programmed instruction Evolution of ET Definitions and domains of ET Digital images Space constraint of digital images Concept of neighbourhood Distribution transparency Scalability Interception threat Layered architecture


851

The traditional way of learning management system generates a global performance report based on the score obtained in the examination and on any other evaluation components. The score obtained by the proposed approach can be used as an indicator of how well a student understands different concepts and to motivate students to strengthen their performance as per specific concepts. The score for a concept question are in whole numbers but for easy processing of data, it is normalized for a concept by dividing the score of a question with total score for that question. So the values between 0 and 1 show the level of understanding of that concept in the scale of 0 to 1. Fig.3 shows the screenshot of the student mark list for a first semester course on Probability and Statistics for Research. The mathematical model used for this calculation is shown in (1) where M is the marks scored in the question and TS is the total score of the question. Score =M /TS

(1)

4) Database Database design is another important part of the proposed system. The students’ details, course details, concepts in the course, question papers and the marks of the students will be collected into the system using a database. MySQL server is used to create the database of the proposed system. B. Implementation As mentioned earlier in Section IIA, three courses were chosen. The instructor first inputs the concepts into the system through an interface. He furnishes the details like course name, category of the concepts (core or secondary). Then the evaluation questions are inputted to the system manually. Details regarding which concept is tested and whether it is a direct or application level question are also provided. The corresponding marks are also inputted into the system. The proposed system shows the performance in each concept by outputting a score and question type.

Concept

Direct question

Score in subject knowledge

Simple application question

Score in simple application knowledge

Advanced application question

Score in advanced application knowledge

For example, the student got 90% content understanding of the concept “array” and he scored 56% in simple application of array. The system output is visualized by means of a graph thus representing this knowledge to the outside world. IV. EXPERIMENTS AND RESULTS The experiments began within putting the core concepts in the courses mentioned earlier. The first periodical examination was attended by 11 students for the two courses Probability and Statistics for Research and Advanced Data Structures and Algorithms. There were 21 students for the course Computer Graphics and Game Programming. The score of the students for each question was used and the content knowledge score and application score was calculated from it. This was provided as feedback to students so that they could improve in the weak concepts identified. After the end semester examination, we analyzed the scores in the concepts for which the students had been given feedback. The results showed an improvement in the scores. The results presented in the section are derived from the data collected during the study. Individual assessment and concept wise assessment were performed with the help of the data. A. Course Analysis This type of assessment is for checking the progress of course for all students. Fig.4 shows the graphical representation of analysis of the examinations core on each concept for all students done in the course Computer Graphics and Game Programming. The X-axis represents the ID for students who are taking the course and Y-axis represents the interval in which score will map. It is between 0 and 1. Each concept is represented by a different color. As mentioned earlier, the score is mapped in the range between 0 and 1.If the score is 0, then there is no color corresponding to the concept. Similarly if the score is 1, then the corresponding bar is fully filled with color. Hence from this we can infer how the students performed for a course on each concept.

Score of students Fig.4. Course Analysis Assessment for the subject knowledge level of the course “Computer Graphics and Game Programming ” for the batch MTech E-Learning Technologies

Fig.2. Block diagram of concept based assessment

852


This kind of analysis also helps to check for situations where majority of students get less mark in a concept.Thus it provides feedback for the teacher to improve the score by implementing new techniques for teaching or adapting new resources for learning. B. Individual Assessment This accesses an individual student’s performance for a single course in the examination. Knowing the concepts will help the student to improve in future and to do better in the course. A graph is plotted as shown in Fig.5 shows the results of individual assessment for a student whose roll number is ‘elt13001’. The X-axis shows the eight concepts in the course Computer Graphics and Game Programming. Y-axis shows the normalized score for the student on each concept. C. Concept-wise Assessment The assessment shows how all students performed in a single concept. By analyzing this report, an instructor can decide whether he needs to teach the concept again or not. If majority of students find it difficult to understand the concept, then the teacher can adopt another instruction style and the institution can provide support for the teacher. A graph as in Fig.6, is plotted to show the concept wise assessment in ‘Formal elements’ from the course Computer Graphics and Game Programming.

Fig.5. Individual Assessment for the subject knowledge level of the student whose roll number is “elt13001” for the course “Computer Graphics and Game Programming”

Fig.6. Concept-wise Assessment for the subject knowledge level concept “Formal Elements” from the course “Computer Graphics and Game Programming”

D. Results The experiments were conducted to prove the efficiency of the proposed system. Concept-wise marks for the first periodical marks were entered for each student and relevant feedback was provided. The end semester marks were then compared with the marks obtained for the first periodical examination. The results are depicted in graphs in this section. In Fig.8, Fig.9 and Fig. 10, first bar shows the first periodical score of each student and second bar shows the end semester score. The X-axis of the graph indicates the students and Yaxis indicate the score ( between 0 and1). Fig.8. shows the comparison of results for the course Advanced Data Structures and Algorithms. The concept for comparison is “AVL tree insertion”. The score is between 0 and 1. From the figure, we can see that out of 10 students in the course, 7 students improved their marks. Fig 9 shows the result analysis of the course “Probability and Statistics for Research ” for the concept “axioms of probability ”. From the graph we can see that with proper feedback 7 students improved performance out of 11.Fig.10 shows the result comparison for the course “Computer Graphics and Game Programming” for the concept “game attribute”. From the graph we can see that out of 21 students,14 students improved the performance.

Fig.8. Result comparison of course “Advanced Data Structure and Algorithms”

Fig.9. Result comparison of course “Probability and Statistics for Research”


853

ACKNOWLEDGMENT We are grateful for the support and facilities provided by Amrita E-Learning Research Lab. REFERENCES Erickson, H. Lynn. Concept-based curriculum and instruction: Teaching beyond the facts. Corwin Press, 2002. [2] Schill, Bethany, and Linda Howell. "Concept-Based Learning." Science and Children 48.6 (2011): 40-45. [3] Galitsky, Boris, and Josep Lluis de la Rosa. "Concept-based learning of human behavior for customer relationship management." Information Sciences 181.10 (2011): 2016-2035. [4] Novak, J. D., & Cañas, A. J. (2006), “The theory underlying concept maps and how to construct them”, Florida Institute for Human and Machine Cognition, 1. [5] Nesbit, J. C., & Adesope, O. O. (2006), “Learning with concept and knowledge maps: A meta-analysis. Review of Educational Research” , 76(3), 413-448. [6] Kinchin, Ian M. “Using Concept Maps To Reveal Understanding: A Two-Tier Analysis.” School Science Review, v81 n296 p41-46 Mar2000, ISSN: ISSN-0036-6811. [7] McCoy, J. D., and Ketterlin-Geller, L. R. (2004), “Rethinking Instructional Delivery for Diverse Student Populations Serving All Learners with Concept-Based Instruction”, Intervention in School and Clinic, 40(2), 88-95. [8] Morse, D., & Jutras, F. (2008), “Implementing concept-based learning in a large undergraduate classroom”, CBE-Life Sciences Education, 7(2), 243-253. [9] Boud, David, Ruth Cohen, and Jane Sampson. "Peer learning and assessment." Assessment & Evaluation in Higher Education 24.4 (1999): 413-426. [10] Curriculum level evaluation, internet: http://www.ibe.unesco.org/ fileadmin/user_upload/COPs/Pages_documents/Resource_Packs/TTC D/ sitemap/Module_8/Module_8.html [11] Unit level, internet: http://staff.mq.edu.au/teaching/evaluation/resources_evaluation/develo ping_unit/assess_achievement/ [12] Learning step level evaluation, internet: http://www.icbl.hw.ac.uk/ltdi/implementing-it/eval.htm [13] Kirkpatrick Model of Evaluation, internet: http://www.isixsigma.com/dictionary/kirkpatrick-four-levelsevaluationmodel/ [14] Ruiz-Primo, Maria Araceli, Susan Elise Schultz, and Richard J. Shavelson. Concept map-based assessment in science: Two exploratory studies. Center for Research on Evaluation, Standards, and Student Testing, Graduate School of Education & Information Studies, University of California, Los Angeles, 1997. [15] Wiliam*, Dylan, Clare Lee, Christine Harrison, and Paul Black, "Teachers developing assessment for learning: Impact on student achievement." Assessment in Education 11.1 (2004): 49-65. [16] Kraiger, Kurt, Eduardo Salas, and Janis A. Cannon-Bowers. "Measuring knowledge organization as a method for assessing learning during training." Human Factors: The Journal of the Human Factors and Ergonomics Society 37.4 (1995): 804-816. [17] Concept Tests , internet: http:// www.cmu.edu/teaching/assessment/assesslearning/concepTests.html [18] Ausubel, D. (1963). The psychology of meaningful verbal learning. New York: Grune & Stratton. [19] Ausubel. D. (2001). The acquisition and retention of knowledge. A cognitivre view. Dordrecht: Kluwer Academic Press. [1]

Fig.10. The result comparison of the course Advanced Data Structures and Algorithms.

V. CONCLUSION AND FUTUREWORKS This paper proposed a system for course-level knowledge representation and assessment using concepts. . An attempt was made to provide timely and targeted feedback linked to students’ performance for different concepts within a course. Concept-wise strengths and weaknesses in learning were identified which helped instructors and students to improve the teaching-learning process and outcomes. For the purpose of experimentation, three courses namely Probability and Statistics for Research, Advanced Data Structures and Algorithms and Computer Graphics and Game Programming from M.Tech E-Learning Technologies were chosen. The students were given feedback on their performance in specific concepts in the intermediate examinations to motivate them to score more on their end semester examinations. Results from the study indicate that this approach led to an improvement in student performance. A brief summary of the results follows. For the course Computer Graphics and Game Programming, fourteen out of twenty one students improved the score for the end semester examination as a result of the concept-wise feedback given on their performance in the periodical examination. Similarly, for the course Probability and Statistics for Research, seven out of ten students improved the score for the end semester examinations. In the Advanced Data Structures and Algorithms course, there was an improvement in the performance of seven out of eleven students. The global view of concept-wise scores enabled teachers to identify whether the students were able to understand a particular concept. In our study, majority of the students were able to score average marks and above. So the feedback for the teachers, in general, was that their teaching methods and the instructional materials provided were appropriate for the course they were teaching. As future work, the current approach for concept-based learning and assessment can be implemented along a learning management system. Also as the number of students increase, entering marks into the system will be a tedious task. It can be simplified by automated recording of the front sheet of the examination paper.

854
