Available Available online online at at www.sciencedirect.com www.sciencedirect.com
ScienceDirect
Available online at www.sciencedirect.com Procedia Procedia Computer Computer Science Science 00 00 (2017) (2017) 000–000 000–000
ScienceDirect
www.elsevier.com/locate/procedia www.elsevier.com/locate/procedia
Procedia Computer Science 112 (2017) 1835–1844
International Conference on Knowledge Based and Intelligent Information and Engineering Systems, KES2017, 6-8 September 2017, Marseille, France
Automatic data integration from Moodle course logs to pivot tables for time series cross section analysis Konomu DOBASHI* Faculty Faculty of of Modern Modern Chinese Chinese Studies, Studies, Aichi Aichi University University 4-60-6 Hiraike-cho 4-60-6 Hiraike-cho Nakamura-ku Nakamura-ku Nagoya-shi Nagoya-shi Aichi-ken, Aichi-ken, 453-8777 453-8777 Japan Japan
Abstract Abstract This This paper paper describes describes aa data data integration integration method method for for Moodle Moodle course course logs logs and and pivot pivot table table functions functions to to analyze analyze the the behavior behavior of of students’ students’ material material page page views views in in face-to-face face-to-face blended blended learning learning using using Moodle Moodle course course materials. materials. The The developed developed method method integrates integrates the the data data with with aa pivot pivot table table by by preprocessing preprocessing Moodle Moodle course course logs logs and and generates generates aa time time series series cross cross section section (TSCS) (TSCS) table table that that visualizes visualizes the the student's student's course course material material page page views. views. Experiments Experiments conducted conducted on on Moodle Moodle page page views views of of actual actual materials materials collected collected during during actual actual lessons lessons found found that that the the table table visualizes visualizes both both overall overall and and individual individual viewpoints. viewpoints. Reactions Reactions to to teacher teacher instructions instructions on on course course materials materials during during class class can can also also be be visualized visualized by by the the generated generated TSCS TSCS table. table. Moreover, Moreover, because because students students who who open open course course items items late late or or do do not not open open them them can can be be identified identified clearly, clearly, the the method method can can be be used used as as aa reference reference for for improving improving future future classes. classes. © © 2017 2017 The The Authors. Authors. Published Published by by Elsevier Elsevier B.V. B.V. © 2017 The Authors. Published by Elsevier B.V. Peer-review under responsibility of KES International. Peer-review under under responsibility responsibility of of KES KES International International. Peer-review Keywords: Keywords: time time series; series; cross cross section; section; page page views; views; visualization; visualization; educational educational data data mining; mining; Moodle; Moodle; pivot pivot table; table; data data integration integration
1. Introduction 1. Introduction In In recent recent years, years, research research and and development development have have been been conducted conducted on on learning learning management management systems systems (LMS), (LMS), e-book e-book systems, and similar systems. The digital materials and e-books installed in these systems have been used daily daily in in systems, and similar systems. The digital materials and e-books installed in these systems have been used classroom lessons and libraries. An LMS or e-book system can accumulate a user’s learning history, including classroom lessons and libraries. An LMS or e-book system can accumulate a user’s learning history, including
* * Corresponding Corresponding author. author. Tel.: Tel.: +81-52-564-6111. +81-52-564-6111. E-mail E-mail address: address:
[email protected] [email protected]
1877-0509 1877-0509 © © 2017 2017 The The Authors. Authors. Published Published by by Elsevier Elsevier B.V. B.V. Peer-review Peer-review under under responsibility responsibility of of KES KES International. International.
1877-0509 © 2017 The Authors. Published by Elsevier B.V. Peer-review under responsibility of KES International 10.1016/j.procs.2017.08.222
1836 2
Konomu DOBASHI / Procedia Computer Science 112 (2017) 1835–1844 Author name / Procedia Computer Science 00 (2017) 000–000
browsing history or test results. Therefore, research on educational data mining (EDM) has been conducted to analyze the accumulated learning history1 and develop a system called a learning dashboard2. Recent LMS such as Moodle3 or e-book systems such as BookLooper4 always accumulate the learning history of users. These systems both provide teaching materials to learners and collect the basic data necessary for learning history analysis. In general classes, it is necessary to analyze learners' behavior from multiple viewpoints when analyzing accumulated learning history. For example, in addition to analyzing the learning behavior of students during class, it is necessary to analyze their answering behavior when quizzes are conducted and the state of preparation and review out of class. Therefore, in classes involving a large number of students, face-to-face lessons are conducted using course materials installed on Moodle, which allows the accumulation and analysis of students' learning history. TSCS Monitor is a tool developed to analyze Moodle course logs during class using time series analysis5, 6. TSCS Monitor uses Excel macros and generates a time series cross section (TSCS) table that visualizes the course material page views of students. The TSCS table is generated using the pivot table function. The resulting table can analyze data from various perspectives and visualize the course material page views with time series tables and graphs. The function newly added to TSCS Monitor in this paper applies string processing to discretize the time data of course material page views recorded in Moodle course logs, in addition to the analysis in class, to enable analysis of the whole semester and multiple years. Specifically, for the time data of course material page views, new categories representing time, such as year, month, day, time, and day of the week, are generated and added to the course log. Using this method to discretize time data and cross tabulate with a pivot table, a TSCS table can be created. The experiments conducted on Moodle page views of actual materials collected during actual lessons found that the table visualizes both overall and individual viewpoints. Reactions to teacher instructions on course materials during class can also be visualized by the generated TSCS table. Moreover, because students who open course items late or do not open them can be identified clearly, the method can be used as a reference for improving future classes. 2. Related research It is extremely important for teachers to understand the behavior of students during class, and many studies employing information technology for this goal are underway. Raca photographed student face movements during class using a video camera and conducts research to investigate facial expression changes 7. Robbins8 and Mizutani9 conduct research to analyze student behavior from answers using Clicker. Moodle has a plugin system to integrate Clicker data. It is now possible to prepare questionnaires and gather answers from students in the class, which can then be used to investigate student actions during class10. BookLooper is a digital textbook delivery system for studying with e-books12. This software implements functions to compile student learning time, access frequency, marker number, and annotation number, allowing detailed analysis of student learning activity. Kiyota analyzed BookLooper log data and tried to seamlessly analyze the behavior of students before, during, and after class4. The author conducted correlation analysis between the number of preliminary exercises before lessons and the final exam. ProM was originally a tool for process mining but is now also used for EDM research 13, 14. Romero analyzed the response behavior of Moodle quizzes answered by students and generated a heuristic net to visualize state transitions 15 . Using ProM analysis, he attempted to find differences in the answer processes of quizzes for students with successful and rejected final exams. Mazza developed a system called CourseVis that tracks student behavior in online classes 16. This behavior can be visualized graphically, along with the status of student access to content pages following the course schedule 17, 18. GISMO, a similar tool, was developed as a plug-in system for Moodle and is currently used by many Moodle users 19, 20 . By installing GISMO into the Moodle reporting tool, Moodle course administrators can analyze student access activity for specific materials and resources, forum access, and quiz results. Analysis results are visualized by time series cross tables and graphs, allowing users to grasp the state of the class from an overall viewpoint, individual viewpoints, or for each of the course materials. However, the original Moodle logs provide only a rough picture of classroom situations and are thus minimally useful. Therefore, Dierenfeld21 generated various learning analytics methods using Excel pivot tables. Konstantinidis and Dobashi22 also developed Excel macros to process Moodle logs for analysis of page views and overall usage23.
Konomu DOBASHI / Procedia Computer Science 112 (2017) 1835–1844 Author name / Procedia Computer Science 00 (2017) 000–000
1837 3
Attempts to track the behavior of online visitors using access logs are increasingly common. For example, the behavior of visitors to online shopping sites is widely tracked to improve advertising messaging 24. Such efforts have led to the development of systems that track the behavior of students during lessons. Google Analytics provides a website analysis service that enables data analyses grounded in different perspectives 25. This service also helps educators improve course materials and lessons. Related Google Analytics offerings, such as data on aggregate visitors, website visit frequency, and website content viewed, are available in real time. Google’s services likewise allow for visual analysis of user behaviors for any operation on a page. As reported by Romero26, 27, 28 and Baker29, extensive research into EDM has been conducted, including studies of Moodle course logs. Verbert investigated various learning dashboard systems, developed a similar system, and reported that they are being used to support class improvement and understand student states 30. 3. Data integration method Auto-generation Moodle
Course logs Download
Pivot table TSCS table
Data integration
Teacher Interaction
Fig. 1. Outline of preprocessing and data integration
3.1. Moodle course log Fig. 1 shows the outline of preprocessing and data integration. The course material page views accumulated in Moodle course logs can be downloaded in Excel format for further analysis. To collect and accumulate Moodle course logs, digital materials such as PDF or Word files must be prepared on Moodle beforehand. The material files can be registered on Moodle and published to students according to the purpose of the lesson. Moodle is used for four primary purposes: (1) making course materials available for browsing when describing class contents, (2) performing a quiz during class, (3) referring to an external web page, (4) submitting a report at the end of term. To analyze learning history, it is desirable to clarify the purpose of collecting Moodle course logs. However, using the pivot table described below, it is possible to conduct analysis through repeated trial and error. Furthermore, the obtained results can be fed back to course material preparation and lesson management for improving future lessons. 3.2. Pivot table and TSCS analysis During face-to-face lessons with many students in the classroom, when Moodle is used to browse course materials or conduct quizzes, time series data is generated for each student. To efficiently process the time series data generated in such lessons, analyze them from various viewpoints, and organize them easily according to the purpose of analysis, a TSCS table is appropriate. The proposed TSCS table is designed to handle time series data in minutes or seconds and is based on the table long used for cross section analysis. Many of the research in relation to the TSCS analysis has been carried out so far. The TSCS analysis described in this paper is based on the theory of Fisher31 and Beck32. Both TSCS and panel data–based analyses can handle quantitative data and quantify qualitative data. TSCS data represent continuous observations of a single investigated unit. TSCS data can be used for both time series analysis and cross section analysis. In the TSCS table created with Excel (Tables 2–4), the observed objects are arranged in the row direction and the observation periods are arranged in the column direction to create a cross table. In the Moodle course log, all but the time data are discrete data and can be processed as qualitative data. Qualitative data in this paper are quantified by treating the data as selection problems. For example, when students click on a link that directs them to a course material, this action is assigned a value of 1; when students do not click
1838 4
Konomu DOBASHI / Procedia Computer Science 112 (2017) 1835–1844 Author name / Procedia Computer Science 00 (2017) 000–000
on the link, this behavior is quantified as 0. When students click the link more than once, the behavior is quantified by indicating the frequency of clicking. This method allows teachers to visualize student behavior as quantitative data. Excel has a pivot table function, which is mainly used to aggregate the frequency of multiple discrete data and create a frequency distribution cross table. Various graphs can be created from this cross table using the graph creation function of Excel. Furthermore, a system that automates processes such as the creation of cross tables and graphs can be built using Excel macro functions. The pivot table is suitable for aggregating discrete data but can be inconvenient for aggregating continuous data such as times. In Moodle course logs, time data are recorded as time series data, and all other data are recorded as discrete data. Therefore, by discretizing time data at appropriate intervals as described below, it is possible to generate the TSCS table with the pivot table and perform various analyzes. 3.3. Discretization of time data and string processing Moodle course logs accumulate time data for the opening of course material in the format “25/05/16, 14:29:40.” The same format is used for forum post times. In Moodle version 2.6.8 and later, these time data are recorded as character strings, but they have the same format as continuous data. Analysis of data in this form using the pivot table is extremely limited and complicates the generation of the TSCS table, which is the object of this paper. Therefore, to easily visualize the analysis results from a lesson, the original Moodle time data must be discretized at fixed time intervals. The proposed method uses string processing to generate categories representing time, such as month, day, day of the week, hour, minute, and time, thus allowing cross tabulation by the pivot table. Cross tabulation tends to be more complicated, and detailed aggregation becomes possible when the types of data to be processed increase, thereby increasing the possibility of multifaceted analysis of Moodle learning history. For example, it is possible to cross tabulate all students or individual students, all course materials or individual course materials, while changing the combination according to year, month, date, and time. The following section outlines string processing using Excel functions. In string processing, multiple processing methods can be considered by combining functions. For example, Moodle supports multiple display languages, including English, French, German, Japanese, and Chinese. Therefore, string processing corresponding to each language is required; here, processing is described using English. The string processing described below can be processed by the RIGHT function, MID function, SUBSTITUTE function, IF function, FIND function, and the string concatenation operator &. 3.4. Cross tabulation with year, month, and day as units If Moodle course logs are accumulated over several years, they can be cross tabulated by year using the Christian calendar year. "25/05/16, 14:29:40" is the data of the English display, and "2016" or the last two digits "16" are extracted as the Christian calendar part. In the case of the "2016" format, "20" is attached to the head of "16." Moodle course logs are accumulated for several years, and the Christian calendar is used when accumulating every year. When summing up data by month, data representing the month are extracted. For the month, "05/16" or "05" is extracted from "25/05/16, 14:29:40." Depending on the data sorting situation, the form "2016/05" may be more suitable. This value is used when totaling the data for every month. When analyzing data by specific days, such as the days on which classes were held, data representing days are extracted. "25/05" is extracted from "25/05/16, 14:29:40." This value is used when identifying and analyzing a specific date, such as a class day. 3.5. Cross tabulation with hour, minute, second, and time as units For the hour, "14" is extracted from "25/05/16, 14:29:40" and edited into the format of "hh:00." This value is used in conjunction with a specific date to analyze a specific hour, such as a class time. Detail in minutes can also be obtained. From "25/05/16, 14:29:40," the portion "14:29" is extracted. In this case, the seconds part is truncated and generated. It is also possible to extract data at different intervals as required, as described previously 5, 21. Time
Konomu DOBASHI / Procedia Computer Science 112 (2017) 1835–1844 Author name / Procedia Computer Science 00 (2017) 000–000
1839 5
extraction can be used to obtain more detail than the minute display. From "25/05/16, 14:29:40," the time part "14:29:40" is extracted. 3.6. Cross tabulation with enrollment year or student’s affiliation If a student’s ID number contains data for the enrollment year or student’s affiliation, those data can be added to the above time categories. By conducting analysis for each grade or student’s affiliation, this approach would increase the possibility of obtaining useful insight. 3.7. Example of data integration Table 1 shows the results after the above data integration. In Table 1, "Year" is shown in column B, "Date" in column C, "Month" in column D, "Time" in column E, "Hour" in column F, "Minute" in column G; "Day of the week" in column H; "Enrollment" in column J; "Affiliation" in column K; all data were generated using the developed macro. The label at the start of each column is a heading in the pivot table and must be created with an appropriate name. In the original Moodle course logs, these columns are followed by the K column, Event name, Description, Origin, and IP address, but they are omitted here due to paper width. Table 1. Moodle course logs after data integration (columns O and the following are omitted).
4. Experiment This section provides an example analysis of a Moodle course log collected from an actual class. After the Excel format file was downloaded from Moodle and the above data integration was conducted, learning analysis was conducted using the pivot table. 4.1. Status of daily course material page views Table 2 presents an automatically generated analysis of the Moodle course log collected after the spring semester of 2016 for a face-to-face class called "Introduction to Social Data Analysis." The content of the lesson was the foundations of statistics studied using Excel. A total of 57 students registered with Moodle, and 90-minute classes were held every Wednesday from 13:00 to 14:30 for 15 weeks. Column A displays the date data, as well as all teaching material files that the teacher (author) and students opened during the day. Column B and the following shows the number of page views for each course material for every hour. Table 2 displays data from 00:00 to 15:00 and omits those from 16:00 to 23:00 because of limited space and the small number of data. During class, students attended lectures while browsing course materials on Moodle. At the start of the lesson, five quizzes were conducted in 7 minutes. The main course items used during the day’s lesson were Files 6.1 to 6.5 (part X), which discuss the creation of a frequency distribution table. The quiz content concerned Files 4.0 to 4.9 (part V), about probability and probability distribution. Part V indicates course material related to the day’s quiz, which the students covered in the previous lesson. Part Z shows the headline of the quiz and the number of page views (392). Moodle was set up to display one quiz per page. As seen in part V, page views occurred from 9:00 to 12:00, likely in preparation for the quiz. Part X and Y shows the file names of the course materials used on the day of the lesson and the page views, which occurred mainly during the class period of 13:00 to 14:00. In Part Y, "File: Exercise 6" was used in the second half of the
1840 6
Konomu DOBASHI / Procedia Computer Science 112 (2017) 1835–1844 Author name / Procedia Computer Science 00 (2017) 000–000
Table 2. Automatically generated Moodle course log after data integration (columns R and beyond are omitted).
Table 3. Page views per course material (every minute), manually generated using parts V and W from Table 2.
lesson, and 55 page views therefore occurred at 14:00. Part W shows that course items related to the quizzes were opened, but it cannot be determined whether these items were opened during the quiz time. Table 3, manually generated from parts V and W in Table 2, shows timeseries data in time units. As shown in Table 3, results can be displayed for every minute to confirm opening in detail. These results show that the course items related to the quiz were open during the quiz time, from 13:00 to 13:07, although browsing also occurred in preparation for the quiz. Furthermore, by searching the course material page views for each student, it is possible to specify and display the corresponding students. By clicking on the corresponding numerical value cells in Table 2, it is also possible to display corresponding students in the same way.
Konomu DOBASHI / Procedia Computer Science 112 (2017) 1835–1844 Author name / Procedia Computer Science 00 (2017) 000–000
1841 7
The graph in Fig. 2 shows page views for the course material and quiz used in the day’s lesson, expressed in minutes for 24 hours. 12:59 is just before the start of the quiz, and 13:11 is just after the quiz is over. Therefore, the number of page views increases in this time period. Also, relatively more course material page views were seen at 13:25, 13:44, and 14:06. No viewing of course materials occurred after the class ended at 14:30.
Fig. 2. Automatically generated graph of parts X, Y, and Z, as well as the Entry page, of Table 2. The graph shows the page views for the course materials used in the day’s lesson. The bar graphs (left scale) show the page views of the quiz, and the line graphs (right scale) show the course material page views.
Fig. 3. Number of page views (line graph, left scale) of course materials related to the quiz and number of quiz page views (bar graph, right scale). This manually generated graph does not include the value of Total, described at the bottom of Table 3.
Fig. 3 shows a composite graph based on the data in Table 3 of course material page views related to the day’s quiz and the quiz page views. The graph shows all periods when the page views occurred: the first browsing occurred at 09:03 and the last browsing at 14:12. Page views also increased slightly just before the start of the quiz. 4.2. Comparison of opening of course items between student and teacher Table 4 compares a student's individual page views with the teacher's page views for the course items used in the lesson. The class was held on April 13, 2016. The table was generated by downloading only the course log of the day from Moodle. The time series data show when the course items were opened up to the second unit. The upper half of column A shows the student's ID and the names of the course items opened, and the lower half shows the ID of the teacher and the names of the course items opened.
1842 8
Konomu DOBASHI / Procedia Computer Science 112 (2017) 1835–1844 Author name / Procedia Computer Science 00 (2017) 000–000
Table 4. Page views per course material by Student01 and Teacher (every minute, generated semiautomatically).
During this lesson, the teacher logged in to Moodle using a student ID, displayed course items on the material presentation monitor for students, explained the contents, and instructed students to open the course items themselves. In column A of Table 4, the upper half shows the course items opened by the Student01, and the lower half shows the items opened by the teacher. Comparison shows that Student01 did not open the same course items as the teacher. The teacher opened 13 course item files during the lesson, but Student01 opened only 10 of the same files. Student01 did not open “File: 1.11 Printing,” “File: 1.12 Downloading data,” and “File: Exercise 1,” as shown by the absence of these files from column A for Student01. In addition, Student01 opened “URL: Japanese statistics,” which was not used by the teacher during the lesson. Notably, the teacher opened "File: 1.5 Basic operation on worksheet" at "13:32:48," but the student opened the same file at "13:37:57," after a lag of roughly 5 minutes. Similar comparisons can be used to examine the file opening patterns of students. 5. Discussion 5.1. Consideration of lessons from the TSCS table and graphs The results analyzed using the method proposed in this paper are shown in Tables 2 to 4. Table 2 shows in which hours of the day course materials were opened; opening is concentrated from 12:00 to 14:00. During the 12:00 period, the students seem to be reviewing the course items early to prepare for the quiz. However, using a TSCS table of each minute, student openings of course items were found to be concentrated from 12:49 to 12:59 when opening course items. Therefore, for many students, approximately 10 minutes before class is the review time for the quiz. Additionally, many students tend to browse course items only during class. Based on the number of students who logged in to Moodle on a daily basis, the majority did not to log in except on the class day. Most students log in to Moodle a few minutes before the quiz and open the course items during the quiz and lessons. Very few students opened course items outside of class hours or on days without classes. This situation shows that it is necessary to devise class improvements that give assignments for preparation and review beyond quizzes. During the lesson, one material presentation monitor is prepared for every two students, and students make it easy to see the course materials. Therefore, many students do not open course materials on their own personal computers. Table 4, for example, shows that the student did not open several course items. Because the students use
Konomu DOBASHI / Procedia Computer Science 112 (2017) 1835–1844 Author name / Procedia Computer Science 00 (2017) 000–000
1843 9
Excel during class, they tend to open Excel on the full screen, which hides the course material; instead, the students see the teaching materials on the material presentation monitor. The teacher demonstrates Excel operation during class, so the students may not be able to open course items on their own computers. 5.2. Effect on class improvement The author mainly creates TSCS tables after class and uses them to improve on the class structure. For example, a 7-minute quiz was used in the example lesson, but the TSCS table showed that many students finished the quiz early. For this reason, the author reduced the time of the quiz by 2 minutes. This change allowed the quiz to be conducted more smoothly, and few students finished early. Additionally, the TSCS table showed that course materials with less content shortened the presentation time of each file to students, and many students did not open these materials. Therefore, course materials with less content were integrated into other course materials, and greater caution was taken regarding the presentation time of each file. In the example class in Fig. 2, there was a clear tendency of decreased course material opening and participation in the second half of the class. It was therefore necessary to devise measures to maintain students' ability to concentrate. The TSCS table showed that some students opened teaching materials during the quiz time. Therefore, the author looked at the students’ PC screens during the quiz time and closed teaching materials other than the quiz. 5.3. Discover of abnormal page views In the TSCS table of this paper, all page views of course materials of students on Moodle is recorded. Therefore, regarding course materials page views during class, it is possible to clearly identify students who do not open the course materials, and students who have very few course materials can be found easily. Also, students who open the same course item frequently in a very short period of time can be found out. In Table 4, students who open the course item later than instruction of the teacher can be found easily. The author strongly urges students to avoid such abnormal page views during class. 5.4. The limit of Excel The maximum worksheet size in the current version of Excel is limited to 1,048,576 rows and 16,384 columns (Excel 2016). The accumulation of Moodle course logs depends on the nature of different lessons and the number of course materials. The author has found that over a 15-class semester, tens of thousands of data rows are accumulated, but Excel experiences no problems in processing this amount. Considering the size of the worksheet, teachers should be able to accumulate and process several years of data in Excel. 6. Conclusion This paper shows the results of analyzing student course material opening behavior by generating a TSCS table. The proposed method can switch analysis from comprehensive to individual perspectives. The TSCS table can be generated for the students as a whole or each student, for the entire course material or individual course items, and indicates the browsing status of the course materials. Analysis can also be conducted using different categories of time, such as month, day, hour, minute, and time. Acknowledgements This work was supported by JSPS KAKENHI Grant Number 15K00498. References 1. Romero, C. Ventura, S. Data mining in education. WIREs Data Mining Knowl Dis-cov 2013; 3: 12–27. doi: 10.1002/widm.1075.
1844 10
Konomu DOBASHI / Procedia Computer Science 112 (2017) 1835–1844 Author name / Procedia Computer Science 00 (2017) 000–000
2. Verbert, K. Govaerts, S. and Duval, E. et al. Learning dashboards: an overview and future research opportunities. Personal and Ubiquitous Computing 2013; 18, 6: 1499–1514. 3. Moodle, 2017. https://moodle.org/ 4. Kiyota, M., Mouri, K., Uosaki, N., Ogata, H. AETEL: Supporting Seamless Learning and Learning Log Recording with e-Book System. In: Proceedings of the 24th International Conference on Computers in Education. India: Asia-Pacific Society for Computers in Education 2016; 380-385. 5. Dobashi, K. A method of visualizing students' reactions by creation of a time series cross table from the in class page view history of the Learning Management System. Workshop Proceedings of the 23rd International Conference on Computers in Education. 2015; 423–432. 6. Dobashi, K. Development and Trial of Excel Macros for Time Series Cross Section Monitoring of Student Engagement: Analyzing Students' Page Views of Course Materials. Knowledge-Based and Intelligent Information & Engineering Systems: Proceedings of the 20th International Conference KES-2016, York, UK, 5-7 September 2016. Procedia Computer Science. 96, Elsevier 2016; 1086-1095. 7. Raca, M., Kidzinski, L., Dillenbourg, P. Translating head motion into attention-towards processing of student’s body-language. Proceedings of the 8th International Conference on Educational Data Mining. 2015. 8. Robbins, S. Beyond clickers: Using ClassQue for multidimensional electronic classroom interaction. In Proceedings of the 42nd ACM Technical Symposium on Computer Science Education 2011; 661-666. 10. Mizutani, K. A Study of Student Behavior in Classroom Response Systems. Liu, C.-C. et al. (Eds.) (2014). Proceedings of the 22nd International Conference on Computers in Education. Japan: Asia-Pacific Society for Computers in Education. 2014; 483-488 11. Chen, M. Visualizing the pulse of a classroom. Proceedings of the Eleventh ACM International Conference on Multimedia. Berkeley, CA, USA, November 2-8, 2003; 555–561. 12. BookLooper, http://www.kccs.co.jp/ict/cloud-booklooper/ 13. Van Dongen, B.F., De Medeiros, A.K., Verbeek, H.M.W., Weijters, A.J.M.M., Van der Aalst, W.M.P. The ProM Framework: A New Era in Process Mining Tool Support Applications and Theory of Petri Nets 2005 26th International Conference, ICATPN 2005. Miami, USA, June 20-25, Proceedings Lecture Notes in Computer Science 3536 2005; 444–454. 14. Verbeek, H.M.W., Buijs, J.C.A.M., Van Dongen, B.F., Van der Aalst, W.M.P. ProM 6: The Process Mining Toolkit, 8th International Conference on Business Process Management BPM'10. 14-16 September, Hoboken, New Jersey 2010; pp 34-39. 15. Romero, C., Cerezo, R., Bogarín, A. and Sánchez-Santillán, M. EDUCATIONAL PROCESS MINING. In: Data Mining and Learning Analytics: Applications in Educational Research. John Wiley & Sons, Inc., Hoboken, NJ, USA 2016; doi: 10.1002/9781118998205.ch1 16. Mazza, R., Dimitrova, V. CourseVis: Externalising Student Information to Facilitate Instructors in Distance Learning. Proceedings of the International conference in Artificial Intelligence in Education. Sydney Jul. 2003; 279–286. 17. Mazza, R., Dimitrova, V. Visualizing student tracking data to support instructors in web-based distance education. Proceedings of the 13th international World Wide Web conference on Alternate track papers & posters. May, 2004; 154–161. 18. Mazza, R., Dimitrova, V. CourseVis: A graphical student monitoring tool for supporting instructors in web-based distance courses. International Journal of Human-Computer Studies. Feb. 2007; 65(2): 125–139. 19. Mazza, R., Milani, C. Gismo: a graphical interactive student monitoring tool for course management systems. International Conference on Technology Enhanced Learning. Milan, Jan. 2004; 1–8. 20. Mazza, R., Botturi, L. Monitoring an online course with the GISMO tool: A case study. Journal of Interactive Learning Research 2007; 18(2), :251–265. 21. Dierenfeld, H. and Merceron, A. Learning Analytics with Excel Pivot Tables. In Proceedings of the 1st Moodle Research Conference (MRC2012). 2012:115–121. 22. Dobashi, K. Time series analysis of the in class page view history of digital teaching materials using a cross table. In Procedia Computer Science. 2015; 60:1032–1040. 23. Konstantinidis, A. Grafton, C. Using Excel Macros to Analyses Moodle Logs. In: 2nd Moodle Research Conference (MRC2013). 4th and 5th October, Sousse, Tunisia 2013; 33–39. 24. Sen, A., Dacin, P. A., Pattichis, C. Current trends in web data analysis. Communications of the ACM 2006; 49(11):85–91. 25. GOOGLE ANALYTICS, 2017. http://www.google.com/analytics/ 26. Romero, C. Ventura, S. Educational data mining: A survey from 1995 to 2005. Expert Systems with Applications. 2007; 33:135–146. 27. Romero, C., Ventura, S. Garcia, E. Data mining in course management systems -Moodle case study and tutorial-. Elsevier Science, Computers and Education. 2007;51(1):368–384 28. Romero, C. Ventura, S. Educational Data Mining: A Review of the State-of-the-Art. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).2010;40(6):601–618. 29. BAKER, R., YACEF, K. The State of Educational Data mining in 2009: A Review Future Visions. Journal of Educational Data Mining. Fall.2009;l1(1):3-16. 30. Govaerts, S., Verbert, K., Duval, E. and Pardo, A. The student activity meter for awareness and self-reflection. CHI EA '12 CHI '12 Extended Abstracts on Human Factors in Computing Systems. Austin, Texas, USA May 05 - 10, 2012;869-884. http://dx.doi.org/10.1145/2212776.2212860 31. Fisher, R.A. Statistical Methods and Scientific Inference, New York: Hafner Publishing 1973. 32. Beck, N. Time-Series Cross-Section Methods. In: Edited by Janet M. Box-Steffensmeier, Henry E. Brady, and David Collier. The Oxford Handbook of Political Methodology 2008.
