Towards Virtual Course Evaluation Using Web Intelligence M.E. Zorrilla1, D. Marín1, and E. Álvarez2 1
Department of Mathematics, Statistics and Computation, University of Cantabria. Avda. de los Castros s/n 39005 Santander, Spain {zorrillm,david.marin}@unican.es 2 Department of Applied Mathematics and Computer Science, University of Cantabria. Avda. de los Castros s/n 39005 Santander, Spain
[email protected]
Abstract. Web-based learning environments are now extensively used. To guarantee the success in the learning processes, instructors require tools which help them to understand how these systems are used by their students, so that they can undertake more informed actions. Therefore, the aim of this paper is to show a Monitoring and Analysis Tool for E-learning Platforms (MATEP) which is being developed in the University of Cantabria (UC) to help instructors in these tasks. For this, web intelligence techniques are used.
1 Introduction and Motivation In the last decade, the World Wide Web and new technologies have changed the concept of teaching. E-learning platforms are no longer expected to continue being data warehouses, but they are becoming a central element to the learning process [2]. The success of E-learning platforms such as WebCT or Moodle, is based on providing easy to use tools and offering students and teachers the possibility to connect and work “any-time, any-where”. Nevertheless, these systems present certain deficiencies as they lack a face-to-face student-teacher relationship, which is manifested in facts such as: teachers do not really control the evolution of their students, and students cannot express their problems in a natural way. Although some platforms offer reporting tools, the information which they provide is not enough to analyze the behaviour and evolution of each student. Futhermore, when the number of students and the diversity of interactions are high, the instructor has serious difficulties extracting useful information. Here is where web intelligence techniques play their role, given that these can generate from data statistics, analytic models and uncover meaningful patterns. Currently, there are many general tools on the market with which develop solutions for making decisions (SQL Server, Business Objects, Pentaho, …). However, there are no specific tools to monitor, understand and assess the distance learning process of the students, although some effort is being made by universities. CourseVis [3] is a tool that takes student tracking data collected by Content Management Systems and generates graphical representations that can be used by R. Moreno-Díaz et al. (Eds.): EUROCAST 2007, LNCS 4739, pp. 392–399, 2007. © Springer-Verlag Berlin Heidelberg 2007
Towards Virtual Course Evaluation Using Web Intelligence
393
instructors to gain an understanding of what is happening in distance learning classes. It directly uses web log files. It neither builds web sessions nor obtains behaviour patterns. Collaborative Analysis Tool (ColAT) [1] is a tool that offers interpretative views of the activity developed by students in a collaborative environment. It integrates the information of user actions from log files with contextual information (events, actions and activities) in order to reconstruct the learning process. It only shows statistic information. Mostow et al. in [6] describe a tool that shows a hierarchical representation of tutor-student interaction taken from log files. On the other hand, in the data mining field, we find particular solutions to specific goals. For example, Zaïane [9] suggests the use of web mining techniques to build an agent that could recommend on-line learning activities or shortcuts in a course web site based on learners’ access history. Tang et al. [8] are working in a recommender system that finds relevant content on the web and personalizes and adapts this content based on the system’s observation of the learners and the accumulated ratings given by the learners. Finally, it is worth mentioning TADA-ED [4], a tool which tries to integrate various visualization and data mining facilities to help teachers in the pedagogical discovering process. Our tool tries to join and integrate both perspectives: to offer instructors the useful information in static and OLAP reports and to show them the discovered student behaviour patterns. This paper presents the Monitoring and Analysis Tool for E-learning Platforms (MATEP) that is being developed to help instructors to understand how these environments are used by their students, so that they can consequently make better decisions. This would allow them to re-structure the course according to its use, design the course activities according to the resources they have used, propose activities which encourage students to follow the course regularly and so on. The paper is organised as follows. In section 2, we provide a general description of the Web Using Mining project inside which MATEP has been developed. In section 3, we specify the requirements demanded of the tool and show some reports obtained with it. In section 4, we give some recommendations based on the lessons learned. Finally, in section 5, we close with the conclusions drawn and the future work lines.
2 Web Using Mining Project This project [10] initially started as an attempt to give specific answers to professors who, committed to these new methods of learning based on new technologies, do not get the appropriate feedback compared to the feedback they get from their students with traditional teaching methods. What we finally did, though, was to propose and design a global solution that also offered answers to the rest of people involved in the teaching environment, such as students, academic people in charge and site administrators. The project is being developed following the stages of a web using mining project [7], although some stages have been added, such as the building of a data web house adapted to an e-learning environment [11], the generation of OLAP cubes [10], the developing of a reporting tool (MATEP) and the proposal to design, in the near future, a recommender agent. The architecture of the whole solution can be seen in [5].
394
M.E. Zorrilla, D. Marín, and E. Álvarez
3 Monitoring and Analysis Tool for E-Learning Platforms 3.1 Requirements Instructors require a usable and specific tool that allow them track online the learners’ activities and give them answers to general questions such as: − Regards the course follow-up: When do students connect to the system? Do they work online? Could the value of a session be measured in relation to learning objectives? This would help teachers to carry out continual evaluation. − Regards the course: How often do they use collaborative tools? What are the sequences of visited pages in each session, in which order, and how long do students stay in each one? What are the most frequent paths? This will allow teachers to discover if students follow the sequence established by them, to detect not-visited pages or to know which resources actually prefer the students (pdf, videotutorials, etc) among other things. In this way, teachers will have information to modify the structure of their courses and to adapt them to learners’ behaviour. − Regards students: What are the students’ profiles? Is there a relationship between their behaviour and their qualifications? Who leaves the course and when? In order for the application to be generally well accepted, the following requirements have been established: − Web application. The tool must be accessible through Internet. − Usable. It must present a clear and simple interface to find the information. Futhermore, this must not be more than two or three clicks. − Easy to interpret. The tool must include expressive and intuitive reports and graphics with the goal that instructors understand the information at a glance. − Interactive. Whenever possible, the tool must allow the professor’s interaction. (dynamic reports). − “Online” feedback. The webhouse must be fed every day from the e-learning platform in order for educators to have the information updated. 3.2 MATEP Tool Figure 1 shows the MATEP welcome page. Once the instructor identifies himself, he must select the folder associated to his interest. Reports are initially organized in three folders: course follow-up, course usage and student tracking. When he selects a folder, a new page with the available reports is shown (see Figure 2). Next, we present some of these reports. As the test data, we have selected a distance multimedia course of the UC which was held in the Spring semester 2006. We have the following information stored: for each session (obtained from the web log), we have the number of requested pages, time spent on each one and on the whole session, the date and time. Additionally, we have demographic information such as age and gender; academic information such as the degree course they are taking (background knowledge), partial and final marks and whether this is the first time they have selected this subject; and finally, course information such as page classification, estimated time to read and study the contents of each page and the course planning
Towards Virtual Course Evaluation Using Web Intelligence
395
Fig. 1. MATEP welcome page
Fig. 2. Reports in Course Usage folder
(submission of home-based tasks, exams,…) and organization (definition of study sessions, pages to visit in each study session, ...). Figure 3 shows two reports which summarize the global usage analysis of the course (Fig. 4a) and the usage per learner (Fig. 4b). Both include the number of sessions, the average time per session and the number of pages per session and, additionally, the second one also presents the average values of the course, so the instructor can compare the figures. These reports are very useful because they allow teachers to evaluate the usage of their course, to detect whether a student is about to drop out of the course, if the students connect to the system frequently, if the effort is greater than they planned, etc. As can be observed, the reports have parameters with which instructors can modify the query and visualize the information in an aggregated or more detailed way. Other interesting information for teachers when they have to design activities is to know how the students use the resources. To answer this question, the report in Figure 4 has been designed. The class and subclass attributes are established by instructors for each course.
396
M.E. Zorrilla, D. Marín, and E. Álvarez
Fig. 3. Global course usage and usage per learner
Fig. 4. Report about the usage of resources
In order to analyse the paths followed by the learners, the window in Figure 5 was developed. Teachers choose the initial page from which the analysis and the period of interest start and surf through the pages visited. For each one, some interesting information is shown such as the estimated time of study, the average and maximum time of stay, the number of times that the path has been followed and the number of students that did it. This graphic acquires more sense if the course structure is designed by study sessions, as our case is. It means that each student must follow the steps indicated in the programmed sessions which combine theory and practice. The yellow lines represent pages which do not correspond to the chosen study session. So, the instructor can extract conclusions and redesign the course if he feels it to be necessary.
Towards Virtual Course Evaluation Using Web Intelligence
397
Fig. 5. Path analysis
Fig. 6. Study session analysis
The report of Figure 6 offers more detailed information about the study session on a specific date. It allows the instructor to analyze when learners work and where they spend more time. This complements the report of Figure 7. A very challenging task, for professors, is to find relationships between students and web-navigation behaviours. For that, a clustering of sessions must be carried out previously. In figure 7, the results obtained descriptively can be seen. We identify 4 clusters: the first gathers very short sessions in a regular day probably to read the news; the second it is the same as the first but staying more time (read /write mail, study contents, do activities, etc.); the third gathers longer sessions (48 min. on average) on days previous to the submission of a task; and finally, the fourth collects very
398
M.E. Zorrilla, D. Marín, and E. Álvarez
Fig. 7. Web-navigation behavior cluster
long sessions from students that work at the last moment (close to a submission deadline). It can be concluded that the learners’ behaviour is similar to their behaviour in traditional teaching. The product selected to develop the whole system is BI-SQL Server 2005.
4 Recommendations Based on Lessons Learned Distance students require very explicit, well-organized and well-indexed web materials and resources. It is important to structure the entire course carefully before it starts and to be faithful to that structure throughout the course to provide students with a stable context in which to learn. We have checked that students navigate for the entire course in the first weeks and organize their work according to the submission dates. Likewise, these requirements are indispensable to do the course evaluation later. Besides, giving coherent names to files and organizing the course by study sessions is highly advisable. This will allow us to plot a more accurate study navigation path. On the other hand, the context cannot be forgotten. Its importance is vital in order to understand the students’ behavior. Events, news, exams, holidays, etc. influence their way of acting. So, all this information must be gathered and integrated. A good solution could be to store it in a data warehouse. Finally, the success of the tool will be measured by its usability and the information that it provides. So it is very important to make a suitable selection of the indicators to be calculated and of the graphics with which these measures are shown.
5 Conclusions and Future Work In this paper we present a tool, called MATEP, which provide dynamic reports and graphics components to track and assess the learning process in web-based platforms. It has been designed following the premises “easy to use” and “easy to interpret”, with the aim that instructors obtain advantages of its use quickly.
Towards Virtual Course Evaluation Using Web Intelligence
399
Integrating MATEP with e-learning platforms will allow professors to analyze and visualize aggregated and detailed data, discovering student behaviour patterns’, understanding how their courses are used, etc., that is, they will have in their hands quantitative and qualitative information with which they will be able to make better decisions. As future work, we will analyze those attributes that best characterize the learning process and build significant patterns which can be used by a recommender agent. We will also endeavour to obtain an association map between students and navigation sessions.
References 1. Avouris, N., Komis, V., Fiotakis, G., Margaritis, M., Voyiatzaki, G.: Logging of fingertip actions is not enough for analysis of learning activities. In: Proc. Workshop Usage Analysis in learning systems (AIED 2005), Amsterdam (2005) 2. Gradío, A., Peris, R., Pinazo, D., Jiménez, A.: Features of student-lecturer interaction in elearning. In: IV International Conference on Multimedia and Information and Communication Technologies in Education. Seville, Spain (2006) 3. Mazza, R., Dimitrova, V.: CourseVis: A graphical student monitoring tool for supporting instructors in web-based distance courses. International Journal of Human-Computer Studies 65(2), 125–139 (2007) 4. Merceron, A., Yacef, K.: Tada-ed for educational data mining. Interactive Multimedia Electronic Journal of Computer-Enhanced Learning (2005) 5. Millán, S., Zorrilla, M., Menasalvas, E.: Intelligent e-learning platforms infrastructure. In: Proceedings of XXXI Latin American Informatics conference (CLEI), Cali, Colombia (2005) 6. Mostow, J., Beck, J., Cen, H., Cuneo, A., Gouvea, E., Heiner, C.: An Educational Data Mining Tool to Browe Tutor-Student Interactions: Time Will Tell! 7. Srivastava, J., Cooley, R., Deshpande, M., Tan, P.: Web usage mining: discovery and applications of usage patterns from Web data. SIGKDD Explor., vol. 1(2). ACM Press, NewYork (2000) 8. Tang, C., McCalla, G.: Smart recommendation for an evolving e-learning system. International Journal on E-Learning 4(1), 105–129 (2005) 9. Zaïane, O.: Building a Recommender Agent for e-Learning Systems. In: Proceedings of the International Conference on Computers in Education (ICCE) (2002) 10. Zorrilla, M.E., Menasalvas, E., Marín, D., Mora, E., Segovia, J.: Web usage mining project for improving web-based learning sites. In: Moreno Díaz, R., Pichler, F., Quesada Arencibia, A. (eds.) EUROCAST 2005. LNCS, vol. 3643, pp. 205–210. Springer, Heidelberg (2005) 11. Zorrilla, M., Millán, S., Menasalvas, E.: Data webhouse to support web intelligence in elearning environments. In: Proc. of the IEEE International Conference on Granular Computing, Beijing, China (2005)
