2015 2nd ACM International Conference on Mobile Software Engineering and Systems
Capturing Learner’s Activity Events from a Mobile Learning System using Adaptive Event Framework Juliet Mutahi∗ , Oliver Bent∗ , Andrew Kinai∗ , Komminist Weldemariam∗ , Bikram Sengupta† ∗ IBM
Research – Africa Nairobi, Kenya Email: {julimuta, oliverbent, andkinai, k.weldemariam}@ke.ibm.com † IBM Research – India Email:
[email protected] Abstract—Several mobile-enabled solutions for education transformation have been deployed in Africa. Drawbacks of these systems include; a predominant focus on disseminating bulk learning content; student outcomes are measured solely on quantitative performance metrics; a lack of instrumentation to capture fine-grained user interaction data; and finally none of them seem to focus on capturing the contextual factors affecting learners. Hence, there are few chances to fully empower educators to create effective interventions for their students. In this paper, we present an adaptive event framework library that can be embedded within a blended learning environment. It enables the capturing of fine-grained learners activity stream data, including learners sentiment and context information. We conducted a limited controlled experiment to evaluate the effectiveness of the event framework. The real time visualizations provide useful insights to teachers in understanding their classroom and/or individual student engagement and progress. Finally, we outline challenges and preliminary solution how such system can be deployed in resource constrained environments.
In resource constrained environments —characterized by intermittent or lack of network connectivity, as well as limited storage capabilities on the device, this approach can pose challenges by preventing data from being updated regularly to the server. These situations are particularly common in emerging markets like Africa and India. Furthermore, it is quite common now for learners in these environments to have access to mobile and tablet devices, which may come with preloaded content/applications, or may be loaded at educational institutions. In this paper, first we present an extensible and configurable event library that allows fine-grained data capture (e.g. user interaction with learning resources, feedback, sentiment and sensor/device level information) from a mobile-enabled blended learning system which we built. It consists of client side and server side components for loading, storing, analyzing, and distributing operational application data on top of Cloudant. We then present the results of our limited pilot towards understanding student engagement and context by applying cloud analytics using the collected data through the event framework. Finally, we online a preliminary solution to address the resource constrained scenario.
I. I NTRODUCTION The Smarter Planet [10] is increasingly instrumented, interconnected and intelligent. In the domain of Smarter Education, specifically, we are seeing the rise of instrumented applications and middleware that collect and process user-content interaction data at a fine-grained level [4], [7] —e.g., start, pause, stop, ffwd, rwnd, zoom-in, zoom-out, click, hover, happy, sad, bored, etc. Events are geo-tagged and have a corresponding timestamp associated. Such data can then be consumed by intelligent applications for adaptive and personalized learning —e.g. to infer learner engagement, understanding and progress (or lack thereof); to infer content effectiveness, popularity etc; and, to infer learner sentiment, context, affective states, etc. The availability and growing popularity of rich, multimedia learning content (e.g. videos, games, simulations) and sophisticated interfaces (e.g. smart phones, tablets) promote high interaction. A single user learning session can potentially generate thousands of interaction events of different types. Such events are generally first stored locally (e.g. in a phone/tablet) and then periodically sent over to a server where the data can be persisted and made available to consuming applications for analysis. For example Cloudant, a globally distributed database-as-a-service (DBaaS) which provides API support for disconnected use and server sync up [5]. 978-0-7695-5566-9/15 $31.00 © 2015 IEEE DOI 10.1109/MobileSoft.2015.23
II. R ELATED W ORK Tangerine [14] is a mobile/tablet based application designed to collect data on early grade reading and mathematics assessments. It has been also used in Kenya to generate school related operation survey questions and collect responses. Unfortunately, Tangerine focuses on collecting operational related data to facilitate adoption of early learning assessments for education decision making. Interactive context dialog systems in a learning environment allow learners to share their contextual situations and sentiment, experiences and knowledge with their peers and educators discussed in [8], [11]. Assessing learners cognitive context through an adaptive approach to collect learner’s current states based on an implicit assessment procedure by interpreting their behavioral indicators using an educational game is discussed in [9]. eSpark [6] explicitly tries to collect learner’s context (such as happy, exited, sick, etc.) when the learner initially logs-in to learning environment. From our experience and as noted in [9] only explicit-based data 109
collection, e.g. by presenting questionnaires on the screen would probably destroy the learner’s flow experience. A system that tracks the learners focus by tracking mouse and keyboard events and eye gaze presented in [12]. The collected data will be analyzed and used to detect possible intervention points. Our event framework extends this capability further by adding the context of the student while they were consuming that particular content item. The learner’s focus is packaged together with the sentiment the student had while consuming the content, their location and other sensor data. Our approach is thus able to adapt and implicitly ask relevant questions to determine that the student is having difficulty understanding a given content item. Finally, in contrast to emerging standards in the area such as Tincan [15], Caliper [3], or Activity Streams [2], the SAI event offers a much richer instrumentation facility through finegrained events. The standards currently support relatively highlevel events such as which user has accessed which content, has contacted which other user, when a session started/ended etc, whereas the SAI library allows the capture of time-tracked events of much finer granularity such as at which point in a content did a pause/rewind/ffwd happen and for how long, as well as various sensor events. The SAI events can be mapped back/defined as extensions to the events in the standards, allowing us an opportunity to propose standard enhancements going forward.
Figure 1: Event organization and Cloudant sync.
The application events are content dependent and mainly consist of user interaction data with learning content. There are three types of application event documents that are generated: text (pdf, ppt, etc), multimedia (audio and video), and assessment/quiz. Each event document has a set of interaction events that can be captured in a given session. The sentiment and contextual events are captured from learners through a context logic engine. The engine is configured to capture three forms of sentiment: generic, contentcentric and poll-based sentiments. Each of these garners emotional information from individual learners relative to the context of instrumentation. The context logic engine implements two levels of logic to determine the right sentiment question to ask the learner, the primary level is through the collection of explicit sentiment expression from current questions, polls or status updates while the second level implicitly collects the learner’s sentiment based on past sentiment, engagement and performance metric supplied from the analytics engine. The feedback include the well-being of learner constituencies, understanding learner opinion toward specific content, as well as their ability to adequately comprehend content within the specific mode of delivery. Historical sentiment data is used to create profiles to assist in identification of interventions for struggling learners and to better understand their learning styles and preferences for future content delivery. The sensor events include device level sensors such as the ACCELEROMETER, MICROPHONE, LIGHT SENSOR and GPS that are used to capture raw data about the learners context (dependent on the configuration settings). Summarization algorithms are performed on the raw data generated by the sensors in order to derive information about the learners environment. An example is where a three axis ACCELEROMETER is used by the event framework to determine the locomotive state of the learner. Sensor data alongside application events is used to derive the relationship between student engagement and the environmental context. The sensor data can be used during engagement analysis to determine the root cause of anomalous behaviors (could be extremely high or low engagement). An illustrative scenario about the engagement data would be a
III. E VENT F RAMEWORK This section discusses in detail the event framework, which is the basis for our data collection and generation in a blended learning environment. The aggregation of such fine-grained data forms what we call Student Activity Information (SAI) events which in turn will be consumed by the corresponding analytics engine to generate actionable insights —such as engagement, performance, and sentiment. Figure 1 shows a high-level organization of the event framework and its integration with our cloud backend. Application developers can easily integrate the library with their applications (and device) in order to capture granular events. All that is needed is to set a few configuration settings that provide a high degree of flexibility to end users (e.g., turn on/off certain events to log, frequency and storage destination). When the library is integrated with applications, the framework instruments user interactions with content and device, generates JSON documents, and writes to online or offline Cloudant sync engine [1] which automatically syncs with Cloudant storage. The event collection process begins when the system sends an open signal to the event listener and ends when a corresponding close signal has been received from the system. The framework takes advantage of the dictionary structure implemented within a JSON document. In what follows, we describe the various components of the event framework. The event framework collects data from three distinct types of interaction events: i) application event, ii) sentiment and contextual event, and iii) sensor event.
110
student application event document that indicates student disengagement with a content item. Within the same document, sensor logs indicate that device environment had low light quality and high noise level. IV. E VALUATION We ran a limited pilot and evaluated the event framework. Our evaluation was twofold: (i) measure the effectiveness of the event framework in generating fine-grained event streams, and (ii) study the usability of insights generated from the collected data with teachers. More specifically, we integrated the event framework library with our blended learning system (on tablet devices) and deployed in a Grade 5 classroom with 22 students. Two lessons were conducted: one in Science on the topic digestive system; and the other in Maths on the topic calculating perimeter. Two separate teachers taught their respective subjects with the system.
Figure 3: Number of JSON documents uploaded per minute.
While from this pattern of event generation it should be noted again that the quantity of data generated was not significant enough to impose resource constraints on the devices used for the experiment, which also had constant mobile data connectivity. The average document size uploaded by the event framework during this experiment was 5.4KB; Figure 3 shows the distribution of these documents uploaded over the lesson; with a total event data volume of 1.2MB captured for this small classroom size and single lesson.
”map ” : ” f u n c t i o n ( doc ) { i f ( doc . D o c u m e n t E v e n t L i s t ) { emit ({ ” c o u r s e I d ” : doc . D o c u m e n t E v e n t L i s t . c o u r s e I d , ” c o n t e n t I d ” : doc . D o c u m e n t E v e n t L i s t . c o n t e n t I d } , 1) ; } e l s e i f ( doc . V i d e o E v e n t s D o c u m e n t ) { emit ({ ” c o u r s e I d ” : doc . V i d e o E v e n t s D o c u m e n t . c o u r s e I d , ” c o n t e n t I d ” : doc . V i d e o E v e n t s D o c u m e n t . c o n t e n t I d } , 1) ; } }” , ” r e d u c e ” : ” sum ”
Listing 1: MapReduce function stored as a Design Document in Cloudant DBaaS. The collected event stream data was stored using the Cloudant DBaaS. We then used the MapReduce capabilities of Cloudant to manage and manipulate views of the data and store them as secondary indexes [13]. A sample javascript function to describe such a view is shown in Listing 1. In this example, summing over the key:value pairs emitted by the map function, returns the viewing activity for a particular content for the class.
Figure 4: Heatmap and corresponding frequency curve for the page views of a chosen content item. In our evaluation of the usability and insights generated from the data captured by the event framework, we presented to the teachers a dashboard organized into a set of highlevel interlinked views that can be drilled down for further details and first level insights. For instance, Figure 5 was used to present which content items were most active during the lessons —namely, showing resources that have been used by the students, with resource icon size proportional to the amount of students’ activity. If a particular content item is chosen we then present as in Figure 4 a heatmap and frequency curve of page visits the content item used in the lesson. The teachers commented that such heatmaps provide useful insight into the usage of content items, showing areas of high/low student engagement with the learning materials.
Figure 2: Number of events generated over a 50 minute lesson. Figure 2 shows the distribution of the learning events captured by the framework over the course of a 50 minute lesson.
111
generate and store the generated events. It then distributes event compaction needs over all the content that has been reviewed, for which existing events logs exist. Each content item can be partitioned into dynamic logical segments (temporal/spatial). Given a dynamic segment, with X number of event logs, the event summarizer may apply summarization to meet compaction needs. Finally, the summarized data is then sent to the server when network connection is established. Upon receiving a compacted event log corresponding to a content item from a user device, the server-side retrieve event logs for the same item as viewed by similar users in the past. It can then segment each historic event log to align with (temporal/spatial) segments of the received event log. For each aligned segment, it may use intelligent heuristics to reconstruct the received (summarized) event logs based on analytics on the event logs of that segment from past similar users. Finally, both the client and server side frameworks may use domain rules to (respectively) drop events and re-insert them when needed.
Figure 5: The class activity for the content items taught. V. L IMITATION
VI. F UTURE W ORK
As noted previously, users may only have intermittent internet connectivity, and sometimes not have connectivity for extended periods of time. As a result, as learning proceeds in resource constrained scenarios, user interaction data will not be stored beyond a point instrumented applications will have to stop collecting new data, or will delete existing data. Such missing data can lead to valuable user and content insights to be lost. Thus, there is a need for an intelligent event management system that intelligently stores, manages and processes learning interactions data in resource constrained environments.
In this paper, we presented a configurable event framework library. The ability to collect and generate fine-grained data differentiate our approach from other related work. We also reported on a limited pilot that demonstrate the effectiveness of the framework, and its limitations with respect to resource constraints. As our next step, we extend the current design and implementation of the adaptive event framework by handling problems with resource-constraints. Various analytics algorithms to provide real-time actionable insights to students and teachers are part of our future work. R EFERENCES [1] [2] [3] [4] [5] [6] [7] [8] [9]
Figure 6: Client-side event data collection and management. [10]
Figure 6 shows how events can be managed intelligently in resource constrained scenarios. It consists of a resource manager module that is built to monitors the resource levels, current event generation levels and estimates future event storage needs, based on which it computes existing event data compaction needs. For instance, the resource manager monitors the connectivity patterns in order to estimate when a connection may be probable to be available. It can then monitor ongoing and planned activities on the client and estimates the amount of resource that would be needed to
[11] [12] [13] [14] [15]
112
Cloudant Sync. https://cloudant.com/product/cloudant-features/sync/. Activity Stream Specification. http://activitystrea.ms. IMS Global: Caliper Analytics. http://www.imsglobal.org/caliper/. M. Chetlur, A. Tamhane, V. K. Reddy, B. Sengupta, M. Jain, P. Sukjunnimit, and R. Wagh. Edupal: Enabling blended learning in resource constrained environments. In ACM DEV, pages 73–82, 2014. Cloudant. https://cloudant.com/. The eSpark platform. https://esparklearning.com/. P. J. Guo, J. Kim, and R. Rubin. How video production affects student engagement: An empirical study of mooc videos. In L@S ’14, L@S ’14, pages 41–50, New York, NY, USA, 2014. ACM. P. A. Jaques and R. M. Vicari. A BDI Approach to Infer Student’s Emotions in an Intelligent Learning Environment. Comput. Educ., 49(2):360–384, 2007. S. K. D. A. Michael Bedek, Paul Seitlinger. Inferring a Learner’s Cognitive, Motivational and Emotional State in a Digital Educational Game. Electronic Journal of e-Learning, 10(2):172–184, 2012. S. Palmisano. A smarter planet: instrumented, interconnected, intelligent. Retrieved on Dec, 17:2008, 2008. L. Qu and W. L. Johnson. Detecting the Learner’s Motivational States in An Interactive Learning Environment. In AIED, pages 547–554. IOS Press, 2005. L. Qu, N. Wang, and W. L. Johnson. Choosing when to interact with learners. In IUI, pages 307–309. ACM, 2004. I. Software. Technical overview: Anatomy of the cloudant dbaas. Technical report, IBM Software Group, 2014. Tangerine. http://goo.gl/OTbVGj. Archive for the “Standards” Category. http://tincanapi.com/category/ standards/.
