System

OPEN ACCESS EURASIA Journal of Mathematics Science and Technology Education ISSN 1305-8223 (online) 1305-8215 (print) 2017 13(1):19-34 DOI 10.12973/eurasia.2017.00602a

The Design and Development of Identification of Students' Misconceptions in Individualized Learning Environment (iSMILE) System Kaushal Kumar Bhagat Smart Learning Institute of Beijing Normal University, CHINA

N.P. Subheesh Indian Institute of Technology, Chennai, INDIA

Bani Bhattacharya Indian Institute of Technology, Kharagpur, INDIA

Chun-Yen Chang National Taiwan Normal University, TAIWAN Received 14 April 2016 ▪ Revised 07 June 2016 ▪ Accepted 30 June 2016

ABSTRACT With the rapid development of technology, incorporation of Information Communication Technology (ICT) for formative assessment purpose has been increasing over the past decade. This article describes the design and development of identification of students' misconceptions in an individualized learning environment (iSMILE) system that includes accommodations for students’ misconceptions in order to improve student’s conceptual understanding and finally learning outcome. The assessment process is carried out by preparing two-level multiple-choice questions. Misconceptions are identified for each instructional objective to make a root question and linked questions. Depending on the answers provided by the student in first level, the linked questions are given to get more information regarding the misconception. Based on the first and second level answers of students, feedbacks are provided with the misconception identified. One of the most important differences of our iSMILE system, with respect to the other developed systems, is the inclusion of misconception based feedback. The iSMILE system has been developed by using open source ICT tools (like MySQL, Apache, etc.) which make the system free of cost and accessible and editable by any instructor from anywhere and at any time. Keywords: Formative assessment; misconception; feedback; ICT

© Authors. Terms and conditions of Creative Commons Attribution 4.0 International (CC BY 4.0) apply. Correspondence: Chun-Yen Chang, National Taiwan Normal University, 88 Sec. 4 Ting-Chou Rd., Science Education Center, 116 Taipei, Taiwan [email protected]

K. K. Bhagat et al.

State of the literature  



Previous research studies have indicated that formative assessment is a central element of learning environment of the 21st century to improve students’ learning effectiveness. The main purpose of formative assessment is to provide meaningful feedback to teachers and students. Effective feedback can help to identify learner’s specific errors or misconceptions related with the concept. We have found that there is no available system, which provides misconception based feedbacks.

Contribution of this paper to the literature   

A formative assessment system is developed to identify students’ misconceptions. This system uses two-level multiple-choice questions for assessment purpose. Open source ICT tools (like MySQL, Apache, etc.) are used to design our system.

INTRODUCTION In general, assessment plays an important role in any educational system. It identifies, collects and interprets information about the learner’s conceptual understanding associated with instructional objectives (Farrell & Rushby, 2016; Rodrigues & Oliveira, 2014). Formative assessment is an integral part of assessment, which is an ongoing process of gathering data to measure the degree of attainment of course goals (Hsu, Chou, & Chang, 2011). Chin and Teou (2009, p. 1309) define formative assessment as “assessment that informs teachers about what students have learnt, indicates what students may be finding difficult, and helps teachers to adjust their teaching to maximize students’ learning.” Formative assessment is a central element of learning environment of the 21st century to improve students’ learning effectiveness (Bell & Cowie, 2001; Black & Wiliam, 1998; Redecker & Johannessen, 2013). Fuchs and Fuchs (1986) conducted a meta-analysis to examine the effect of formative evaluation on students’ learning achievement. They found an overall effect size of 0.70, which is moderate. The main purpose of formative assessment is to provide meaningful feedback to teachers and students. Students can monitor their learning progress using the feedbacks (Bell & Cowie, 2001). Effective feedback can help to identify learner’s specific errors or misconceptions related with the concept (Attali, 2015; Shute, 2008), improve learning and performance (Attali, 2015; Hattie & Timperley, 2007; Lefevre & Cox, 2016; Sadler, 1989; Yuan & Kim, 2015), increase learner’s confidence and motivation to learn (Yuan & Kim, 2015). With the introduction of technology, the effectiveness of feedback delivery can be improved with more accuracy (Pacheco-Venegas, López, & Andrade-Aréchiga, 2015). Technology can help the assessment process to collect, and analyze the data in a shorter period of time with more accuracy (Farrell & Rushby, 2016; Redecker & Johannessen, 2013; Yuan & Kim, 2015). Currently, many researchers have developed web-based/computer-based assessment systems and found the potential benefit in improving the learning effectiveness of the learners

20

EURASIA J Math Sci and Tech Ed

(Crisp & Ward, 2008; Hsu et al., 2011; Landauer, Lochbaum, & Dooley, 2009; Pacheco-Venegas et al., 2015; Rodrigues & Oliveira, 2014; Wang, 2008, 2010). Crisp and Ward (2008) developed Pedagogical Psychology Computer Assisted Assessment (PePCAA) system to facilitate understanding of pedagogical psychology. They found that PePCAA system motivated the students and provided deeper learning. Hsu et al. (2011) developed Edu-miner system by utilizing a series of text mining techniques to evaluate learner’s performance. Landauer et al. (2009) developed Write-ToLearn formative assessment tool to monitor progress in reading and writing. Pacheco-Venegas et al. (2015) developed MathDIP to provide automatic evaluation and feedback during problem solving in mathematics. Rodrigues and Oliveira (2014) developed AssiStudy as a formative assessment tool to monitor students’ learning progress. Wang (2010) developed graduated prompting assessment module of the WATA (GPAMWATA) to provide timely feedback while facing difficulties when attempting test items. From the above, we have found that there is no available system, which provides misconception based feedbacks. To fill this research gap, we have developed iSMILE system. This paper has outlined the iSMILE system, its components and architecture, working process, and application.

COMPONENTS AND ARCHITECTURE OF ISMILE SYSTEM The iSMILE system is composed of a database unit, a server unit, and a user interface, as shown in Fig. 1. In our system, MySQL and Apache Server are preferred as a database unit and a server unit respectively. PHP is used as a server-side language in order to design and code the user interface. We have used only open-source tools in our system. In an open-source system, the source code is available to all. Therefore, anyone can alter the code (to customize the software according to their needs), fix software-bugs and enhance security considerations. Thus, instructors with minimal software knowledge can also add new features and write new modules.

Fig. 1. The iSMILE system architecture.

21

K. K. Bhagat et al.

iSMILE’s architecture is based on the Model View Controller (MVC) design. According to the MVC design structure, the command-flow is initiated by a central controller. Upon initiation, the controller sends requests to a compatible handler application. The end user interacts with the intranet interface through this central controller. Controller handles all the inputs to the MVC model. For instance, a Graphical User Interface (GUI) controller accepts GUI inputs from the user and gives appropriate commands to the Model and View. Thus, command-actions based on respective inputs are executed. In case of an invalid-input is being given to the controller; the Model sends a command back to the controller to communicate through the View. Subsequently, the View layout displays a message through GUI like ‘inputerror’ and ‘try again’. The MVC design is illustrated in Fig. 2.

Fig. 2. MVC design.

22

EURASIA J Math Sci and Tech Ed

The three components of iSMILE system architecture are described as below Database Unit MySQL is used as the database unit. A database design of a database system is an important stage in system development. In design phase, the database construction is coded in any formal language supported by the database management system. A database design diagram is used to show how the data is organized by forming various database tables. The database design diagram is shown as Fig. 3.

Fig. 3. iSMILE’s Database Design Diagram.

Another design tool is the Data Flow Diagram (DFD). It is a graphical representation of the flow data. That is, how the data moves from the external data source or internal data storage to an external system or to an internal system. It is quite different from the Flow chart where one can determine the rate, order and circumstances of occurrence of a work but without any data input and output information, which is possible in case of DFD. iSMILE’s DFD is shown in Fig. 4.

23

K. K. Bhagat et al.

Fig. 4. iSMILE’s Data Flow Diagram.

Server Unit Apache HTTP web server software is used as a server unit in the iSMILE. We preferred Apache because it is free to download and is open-source. Hence, due to its open-source nature and add-ons support, anyone can adapt the Apache server for specific needs. In order to perform the HTTP server operations, Apache needs some other tools. Additional software tools required in the server side are: a) Server-side programming language: PHP b) Database tool: MySQL c) Compatible operating system: Windows/Linux d) Web browser supporting JavaScript: Google Chrome, Mozilla Firefox, etc.

24

EURASIA J Math Sci and Tech Ed

User Interface The user interface mainly has three stages. Login: There is a quality login window which can be used by both instructors as well as students, as shown in Fig. 5. If the login credentials are valid, the user will be redirected to respective dashboard screen.

Fig. 5. Login screen of iSMILE system.

Assessment: Assessment page is the most important page in our system, as shown in Fig. 6. It consists of two parts namely: (a) Assessment selection: From the given choices the student can select the assessment for taking on with the test. (b) Appearing for the test: Here the user can answer the questions. There is also an option to skip the question. If the user has answered the first-level question, the second-level question will be provided to the user based on the selection. For each option (choice) in the first-level question will have exactly one second level question.

25

K. K. Bhagat et al.

Fig. 6. Interface displaying students’ assessment screen of iSMILE system.

Feedback: The student will be provided with detailed feedback upon completion of the assessment. The feedback includes the questions attended, and the answers provided by the user. There will be separate section for feedback on each question. The feedback will be stored in the application, and the user can view it at any time, as shown in Fig. 7.

Fig. 7. Interface displaying feedback screen of iSMILE system.

26

EURASIA J Math Sci and Tech Ed

WORKING PROCESS OF THE ISMILE SYSTEM The login page of iSMILE user interface can be accessed in any web browser with a local address (e.g. http://localhost/iit/), and the option for sign-in is provided. In the first step, if the user has not registered, then he/she can register with ‘create account’ option. For registering a new instructor by assigning a user’s email id and password, one has to get the administrative privileges. Thus, through manual editing in server computer, new instructor login is created. In the second step, after opening http://localhost/phpmyadmin/ , the system administrator can add as many instructors as required. In the third step, user has to finish the registration process, and iSMILE system will identify the student or instructor, based on the sign-in credentials they are given. We have created two different types of registration to ensure security. When the instructor logs in the system, the application of the iSMILE provides the links (e.g., topics, questions and question-sets). If there are no question sets previously entered or if the instructor wants to create new question sets, he/she can use ‘create topics’, ‘create question-sets’ and ‘create question’ options to enter two-level objective questions on various topics. Each question will have two meta-data fields: instructional objective and misconception. After entering the second-level question based on the misconception, the instructor has to enter feedbacks for each possible option. The idea behind the feedback is to pinpoint the misconception. In the final step, entering all questions and feedbacks in a question-set, it has to be published by using the ‘publish’ button. All unpublished questions are visible to the instructor only and can be edited at any time before publishing. When a student logs in, all the available (published) question-sets are displayed to the student. He/she can attend the assessment and can view the feedback immediately. There is a provision to skip a question also. This is to avoid random guessing. The feedbacks are stored in the system for the students, and they can view previous feedbacks at any later login. The flow charts for student’s registration, login and assessment are shown in Fig. 8, Fig. 9, and Fig. 10 respectively. The working process is further illustrated using a model set of questions from the course of Digital System Design.

27

K. K. Bhagat et al.

Fig. 8. Flowchart of student’s registration.

Fig. 9. Flowchart of student’s login.

28

EURASIA J Math Sci and Tech Ed

Fig. 10. Flowchart of student’s assessment.

Instructional Objective: To write the characteristic (or function) tables for SR, JK, D and T flip-flops (FFs) Misconception: Similarity in characteristic equations of T and D; SR and JK FFs. (Root Question) 1.1.0: Which of the following is the characteristic equation of T flip flop?

29

K. K. Bhagat et al.

(Qn+1 denotes the next state) (a)

i/p

Qn+1

0

1

1

0

i/p

Qn+1

0

Qn

1

Qn 1

(b)

i/p

Qn+1

0

0

1

1

i/p

Qn+1

0

Qn 1

1

Qn

(c)

(d)

Correct answer is option (b). If the student’s answer is (a) for 1.1.0 (Root Question) then (Second Level Question) 1.1.1: What is the next state output of T flip flop, if its input is zero?

30

(a) Hold state

(c) Complement of present state

(b) 1

(d) 0

EURASIA J Math Sci and Tech Ed

Correct answer is option (a). Feedbacks: For second level answer (a), and so on. (A) Your idea regarding Toggle state of a flip-flop is improper. (B) Your answers are contradictory. You might be guessing. (C) Try to make a clear idea about Toggling. (D) You seem to be confused with hold state and Toggle state. If the student’s answer is (b) for 1.1.0 (Root Question), then (Second Level Question) 1.1.2 How a J K flip-flop is made to toggle its present state? (a) J=0, K=0 (b) J=1, K=0 (c) J=0, K=1 (d) J=1, K=1 Correct answer is option (d). Feedbacks: For second level answer (a), and so on. (A) You are confused with HOLD state and TOGGLE state of JK FFs (B) You have some problem in understanding the SET condition and TOGGLE states of JK FFS (C) You have idea about T FFs but don’t have proper idea regarding the TOGGLE and RESET state of JK FF (D) Your Answer is excellent. The instructional objective has been met. If the student’s answer is (c) for 1.1.0 (Root Question) then (Second Level Question) 1.1.3 What is the next state output of T flip flop if the input is 1? (a) 1

(c) Qn

(b) 0

(d) Qn

31

K. K. Bhagat et al.

Correct answer is option (d). Feedbacks: For second level answer (a), and so on. (A) You might be confused with T FF and D FF characteristic equations. (B) You are guessing, as your answers are contradictory. (C) You are guessing, as your answers are contradictory. (D) Your answers say the TOGGLE and HOLD states are confusing for you. If the student’s answer is (d) for 1.1.0 (Root Question), then (Second Level Question) 1.1.4 If J=0, K=0 in J K flip flop, then output Q n+1 (the next state output) will be (a) 1 (b) 0 (c) Qn (d)Qn Correct answer is option (d). Feedbacks: For second level answer (a), and so on. (A) Your first and second level answers say you are confused with D and T FFs. (B) It seems that you are confused with the HOLD and TOGGLE states (C) You are guessing, as your answers are contradictory. (D) Your idea regarding different states of T FFs is improper.

CONCLUSION To make the formative assessment more effective, this work presents iSMILE system, which is based on students’ misconceptions. This paper describes the design and features of iSMILE system and how it is better than the existing web-based formative evaluation tools. For every instructional objective, a root (first level) and four linked (second level) multiplechoice questions are prepared with respect to the identified misconceptions. At least one misconception is included in every instructional objective as a metadata field and based on this a two-level diagnostic test is carried out. Thus, based on the answers provided by the student to the first-level questions and second level questions, feedback is given to improve the learning skills of the student. In addition, feedback highlights the misconceptions associated with students’ conceptual understanding. This feature of iSMILE distinguishes

32

EURASIA J Math Sci and Tech Ed

itself from other available web-based formative assessment systems. Our system uses the open-source ICT tools (like MySQL, Apache); therefore, it is free of cost and can be accessed from anywhere and at any time. Our system has not yet been evaluated inside real classroom settings, which is a limitation of our work. For future research, we are planning to incorporate the iSMILE system into formal classroom settings, in order to evaluate the learning effectiveness of this software.

ACKNOWLEDGEMENTS This project is supported in part by the Ministry of Science and Technology, Taiwan, R.O.C. (previously named National Science Foundation of Taiwan) under the Grant number NSC1022515-S-103-001-H, “Aim for the Top University Project” sponsored by the Ministry of Education, Taiwan, R.O.C., and “International Research-Intensive Center of Excellence Program” of NTNU and Ministry of Science and Technology, Taiwan, R.O.C. under Grant no. MOST 104-2911-I-003-301.

REFERENCES Attali, Y. (2015). Effects of multiple-try feedback and question type during mathematics problem solving on performance in similar problems. Computers & Education, 86, 260-267. doi:10.1016/j.compedu.2015.08.011 Bell, B., & Cowie, B. (2001). The characteristics of formative assessment in science education. Science Education, 85(5), 536-553. doi:10.1002/sce.1022 Black, P., & Wiliam, D. (1998). Assessment and Classroom Learning. Assessment in Education: Principles, Policy & Practice, 5(1), 7-74. doi:10.1080/0969595980050102 Chin, C., & Teou, L. Y. (2009). Using Concept Cartoons in Formative Assessment: Scaffolding students’ argumentation. International Journal of Science Education, 31(10), 1307-1332. doi:10.1080/09500690801953179 Crisp, V., & Ward, C. (2008). The development of a formative scenario-based computer assisted assessment tool in psychology for teachers: The PePCAA project. Computers & Education, 50(4), 1509-1526. doi:10.1016/j.compedu.2007.02.004 Farrell, T., & Rushby, N. (2016). Assessment and learning technologies: An overview. British Journal of Educational Technology, 47(1), 106-120. doi:10.1111/bjet.12348 Fuchs, L. S., & Fuchs, D. (1986). Effects of systematic formative evaluation: A meta-analysis. Exceptional children, 53(3), 199-208. Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81-112. Hsu, J.-L., Chou, H.-W., & Chang, H.-H. (2011). EduMiner: Using text mining for automatic formative assessment. Expert Systems with Applications, 38(4), 3431-3439. doi:10.1016/j.eswa.2010.08.129 Landauer, T. K., Lochbaum, K. E., & Dooley, S. (2009). A New Formative Assessment Technology for Reading and Writing. Theory Into Practice, 48(1), 44-52. doi:10.1080/00405840802577593 Lefevre, D., & Cox, B. (2016). Feedback in technology-based instruction: Learner preferences. British Journal of Educational Technology, 47(2), 248-256. doi:10.1111/bjet.12227 Pacheco-Venegas, N. D., López, G., & Andrade-Aréchiga, M. (2015). Conceptualization, development and implementation of a web-based system for automatic evaluation of mathematical expressions. Computers & Education, 88, 15-28. doi:10.1016/j.compedu.2015.03.021

33

K. K. Bhagat et al. Redecker, C., & Johannessen, Ø. (2013). Changing Assessment — Towards a New Assessment Paradigm Using ICT. European Journal of Education, 48(1), 79-96. doi:10.1111/ejed.12018 Rodrigues, F., & Oliveira, P. (2014). A system for formative assessment and monitoring of students' progress. Computers & Education, 76, 30-41. doi:10.1016/j.compedu.2014.03.001 Sadler, D. R. (1989). Formative assessment and the design of instructional systems. Instructional Science, 18(2), 119-144. doi:10.1007/bf00117714 Shute, V. J. (2008). Focus on Formative Feedback. Review of Educational Research, 78(1), 153-189. doi:10.3102/0034654307313795 Wang, T.-H. (2008). Web-based quiz-game-like formative assessment: Development and evaluation. Computers & Education, 51(3), 1247-1263. doi:10.1016/j.compedu.2007.11.011 Wang, T.-H. (2010). Web-based dynamic assessment: Taking assessment as teaching and learning strategy for improving students’ e-Learning effectiveness. Computers & Education, 54(4), 11571166. doi:http://dx.doi.org/10.1016/j.compedu.2009.11.001 Yuan, J., & Kim, C. (2015). Effective Feedback Design Using Free Technologies. Journal of Educational Computing Research, 52(3), 408-434. doi:10.1177/0735633115571929

http://iserjournals.com/journals/eurasia

34