is presented in this paper, exploiting ontologies and semantic augmentation. ... used: a) WordNet3 for lexical derivations and extraction of synonyms and ...
ViewS in User Generated Content for Enriching Learning Environments: A Semantic Sensing Approach Dimoklis Despotakis1, Vania Dimitrova1, Lydia Lau1, Dhavalkumar Thakker1, Antonio Ascolese2, Lucia Pannese2 1
School of Computing, University of Leeds, Leeds, LS2 9JT, United Kingdom {scdd, V.G.Dimitrova, l.m.s.lau, D.Thakker}@leeds.ac.uk 2 imaginary Srl, Milan, 50 - 20124, Italy {antonio.ascolese, lucia.pannese}@i-maginary.it
Abstract. Social user-generated content (e.g. comments, blogs) will play a key role in learning environments providing a rich source for capturing diverse viewpoints; and is particularly beneficial in ill-defined domains that encompass diverse interpretations. This paper presents ViewS - a framework for capturing viewpoints from user-generated textual content following a semantic sensing approach. It performs semantic augmentation using existing ontologies and presents the resultant semantic spaces in a visual way. ViewS was instantiated for interpersonal communication and validated in a study with comments on job interview videos, achieving over 82% precision. The potential of ViewS for enriching learning environments is illustrated in an exploratory study by analysing micro-blogging content collected within a learning simulator for interpersonal communication. A group interview with simulator designers evinced benefits for gaining insights into learner reactions and further simulator improvement. Keywords: Social content, Semantic Augmentation and Analysis, Viewpoints, Interpersonal Communication, Simulated Environments for Learning
1
Introduction
Social spaces are radically transforming the educational landscape. A new wave of intelligent learning environments that exploit social interactions to enrich learning environments is forming[1]. Notable successes include using socially generated content to augment learning experiences [2], facilitate search[3], aid informal learning through knowledge discovery or interactive exploration of social content [4], facilitate organisational learning and knowledge maturing. In the same line, social contributions are becoming invaluable source to augment existing systems, e.g. [5-7] and to build open user models [8-9]. Social spaces and user generated content provide a wealth of authentic and unbiased collection of different perspectives resulting from diverse backgrounds and personal experiences. This can bring new opportunities for informal learning of soft skills (e.g. communicating, planning, managing, advising, negotiating), which are ill-defined domains requiring awareness of multiple interpreadfa, p. 1, 2011. © Springer-Verlag Berlin Heidelberg 2011
tations and viewpoints [10]. There is a pressing demand for robust methods to get an insight into user generated content to empower learning of soft skills. Such a method is presented in this paper, exploiting ontologies and semantic augmentation. While semantic analysis of social content is revolutionizing human practices in the many areas (e.g. policy making, disaster response, open government), little attention has been paid at exploiting semantic technologies to gain an understanding of social content in order to empower learning environments. The approach presented in this paper is a step in this direction. We present a semantic social sensing approach which explores ontologies and semantic augmentation of social content to get an insight into diversity and identify interesting aspects that can be helpful for enriching a learning environment. While the approach can be seen as resembling open learner models of social interactions (e.g.[8-9, 11], it has crucial differences - we link social user generated content to ontology entities and provide interactive visualizations in the form of semantic maps for exploring such content. The work presented here is conducted within the EU project ImREAL 1 which examines the use of digital traces from social spaces to augment simulated environments for learning. We present a semantic social sensing approach adapted for user generated content on interpersonal communication (Sections 2 and 3), focusing on nonverbal communication (body language and emotion). This contributes to newly establishing research in social knowledge management for learning. The approach is then applied to one of the ImREAL use cases – a simulator for interpersonal communication in business settings (Section 4). We examine the potential for gaining an understanding of user reactions with the simulation and extending the simulation content. Our approach offers a new dimension in the established research strand on evaluating and extending simulated environments for learning by adding a novel way of sensing learners and content, in addition to traditional methods of log data analysis [12-13], measuring the learning effect [14-15] or eye tracking[16].
2
The ViewS Framework
ViewS (short for Viewpoint Semantics) - is a framework for capturing the semantics of textual user generated content (e.g. comments, stories, blogs). ViewS utilises existing semantic repositories (in the form of ontologies) with corresponding techniques for semantic augmentation of text input, and comprises of two phases ( see Figure 1). Phase 1: Text Processing. This involves: (i) Extraction of text surface form (SF) from the input text using Natural Language Processing (NLP) modules. The Stanford NLP2 parser is utilised for these modules for tokenisation, sentence splitting, part-ofspeech tagging and typed dependency extraction. The extracted SF includes exact text tokens (spans), stemmed tokens, and multi-token terms from the typed dependencies (negations are also included). (ii) Linguistic and semantic enrichment of the SF to provide enriched surface form (ESF). Freely available generic language resources are
1 2
http://www.imreal-project.eu Stanford Parser: http://nlp.stanford.edu/software/lex-parser.shtml
used: a) WordNet3 for lexical derivations and extraction of synonyms and antonyms (presented with negation); b) DISCO 4 corpus for deriving similar words based on cooccurrences in textual corpus by using the similarity queries; and c) Suggested Upper Merged Ontology - SUMO5, for detecting relevant senses by using lexical categories and semantic mappings of terms identified from WordNet. The resultant ESF allows for a broader set of textual terms to be mapped to ontologies. Phase 2: Semantic Annotation. The semantic annotation concerns the mapping of both SF and ESF to ontology entities, using one or more ontologies. Each ontology represents a specific dimension of the domain which we wish to analyse the viewpoints expressed in user generated content. For example, emotion and body language are the two chosen dimensions for examining user generated content in the domain of interpersonal communication. The algorithm for annotation prioritises the mapping of the SF. The result of the semantic annotation is a set of XML elements which semantically augment the textual content with ontology entities. These elements include the annotated textual token(s), the ontology entity URI, a negation operator (when such exists), the WordNet sense for the specific token (based on the SUMO mapping) and the corresponding ontology name.
Fig. 1. The ViewS6 semantic augmentation pipeline: a text surface form is extracted and enriched from the text processing Phase 1 and in Phase 2 is linked to ontologies.
3 4 5 6
WordNet: http://wordnet.princeton.edu/ DISCO: http://www.linguatools.de/disco/disco_en.html SUMO: http://www.ontologyportal.org/ Demo of the ViewS Semantic augmentation is available at: http://imash.leeds.ac.uk/services/ViewS
Instantiation of ViewS for Interpersonal Communication with a focus on Social Signals. In this paper, the domain interpersonal communication (IC) is chosen with the focus on two dimensions of social signals (emotion and body language). WordNet lexical categories and SUMO concepts for sense detection relevant to IC and social signals were selected7. For the semantic annotation phase we utilised two ontologies: (i) WordNet-Affect, which comprises a rich taxonomy of emotions with 304 concepts. The original XML format of WordNet-Affect was transformed to RDF/XML8 to enable semantic processing. (ii) A body language ontology9 which was built as part of the Activity Modelling Ontology [17] developed within the ImREAL EU Project. This body language ontology combined concepts in a survey article for social signal processing [18], a personal development site for business communications skills10, and a portion of SUMO. An example of a user comment on a job interview video, and the corresponding set of annotations, is shown in Table 1. Table 1. The annotation set of an example comment. Comment
" The applicant is not anxious. She appears very confident, although she is not greeting the interviewer and then sits and crosses her legs. She does not respect the interviewer. The interviewer might feel discomfort with the applicant's manners."
Text Token(s)
(ontology entity, {Ontologies presenting dimensions})
{not, anxious} appears confident Sits {not, greeting} Legs respect discomfort
(¬ anxiousness,{WNAffect, Body Language}),(¬ nervousness, {Body Language}), (facial_expression, {Body Language}), (face, {Body Language}) (confidence, {WNAffect, Body Language}), (authority, {Body Language}) (sitting , {Body Language}) (¬ greeting, {Body Language}) (legs, {Body Language}) (¬ regard, {WNAffect}), (¬ admiration, {WNAffect}) (nausea, {WNAffect}), (distress, {WNAffect}), (frustration, {WNAffect}), (anxiety, {WNAffect}), (¬comfortableness, {WNAffect}), (confusion, {WNAffect}) (crossed_legs_sitting, {Body Language})
{crosses, legs}
3
Validation of the Semantic Augmentation Mechanism
An experimental study was conducted to validate the precision of the semantic augmentation with ViewS in the above instantiation. Content was collected using a system which presented selected YouTube videos showing interesting job interview situations and asked users to comment at any point in the videos on interesting aspects they saw in the videos or could relate to their personal experience. The study involved 10 participants (5 male and 5 female). 183 user comments were collected. 1526 annotations were extracted with ViewS (average of 8.3 annotations per comment). Three 7 8 9 10
The selection was made by a social scientist expert in manual annotation of content for IC. The WNAffect taxonomy: http://imash.leeds.ac.uk/ontologies/WNAffect/WNAffect.owl BodyLanguage ontology: http://imash.leeds.ac.uk/ontologies/BodyLanguage/BodyLanguage.owl http://www.businessballs.com/body-language.htm
expert annotators (two social scientists with experience in content annotation and activity modelling and one psychologist) examined each comment and the corresponding semantic augmentation produced by ViewS. The average pair-wise interannotator specific agreement11 for correct annotation was 80%. The system achieved 89.97% micro-averaging precision 12 (i.e. the precision of the annotations for the whole corpus of comments) for correct ontology entity extraction (for WNAffect 96.84% and for Body Language 92.65% - entities are shared between the two ontologies) and 82.5% for correct identification of textual terms important to describe the textual comment (see Table 2 for a summary), considering the majority of responses by the three annotators. For these annotations the enrichment methods followed were more favorable than the surface form based (73% compared to 27%). The macroaveraging precision (i.e. the average of the precision for each comment) was 89.55% for correct ontology entity extraction and 82.72% for correctly identifying text terms to describe the comment. Table 2. Summary of the annotated content (183 user comments) over the two ontologies. SF
ESF
Total
Annotations
WNAffect Body Language
28 318
321 859
349 1117
Distinct Entities
WNAffect Body Language
12 82
89 142
101 (33.2% of 304) 224 (42.5% of 526)
Ontology
The validation study showed that ViewS performed very well with the instantiation for emotion and body language (see Section 2). The high precision (over 82% in different annotation aspects) signifies a reliable semantic augmentation mechanism to enable semantic analysis of IC related content.
4
Application of ViewS for Enriching an IC Simulator
4.1
The Simulator and Study Setup
To examine the potential of exploiting the semantic output of ViewS for enriching learning environments, we conducted an exploratory study that collected social content within an existing simulator for interpersonal communication in business settings. The simulator is developed by imaginary Srl within the framework of the ImREAL EU project. The study involved one of the simulation scenarios – the learner is a host who organises a business dinner involving several people from different nationalities. The interaction with the simulator is expected to promote awareness of the impor11
12
The prevalence of responses by the annotators lead to imbalanced distribution which results to low Kappa, even though the observed agreement (Po) is high. The specific agreement was reported following: D. V. Cicchetti and A. R. Feinstein, "High agreement but low kappa: II. Resolving the paradoxes," Journal of Clinical Epidemiology, vol. 43, pp. 551-558, 1990. For details regarding precision calculation please refer to: Sebastiani, F.: Machine learning in automated text categorization. ACM Comput. Surv. 34, 1-47 (2002)
tance of cultural variations in IC, focusing on differences in social norms and use of body language, and how this may influence a person’s expectations and emotions. It also aims to promote reflection on personal experiences in relevant IC contexts.
Fig. 2. Example learner interaction screen – the simulated situation is in the Dinner episode where the host has to decide about ordering food for his business guests.
The simulated scenario includes four episodes: Greetings (situations embed arriving on time, different norms about greetings, first impression, and use of body language), Dinner (situations embed use of body language and different preferences about food and drink), Bill (situations embed use of body language and different norms about payment), Goodbye (situations embed use of body language and different norms about greetings). Figure 2 illustrates the interface and the interaction features provided to the leaner to select a response and read/write micro-blogging comments. The simulator was used by 39 users who attended interactive sessions at learning technology workshops or responded to invitations sent to learning forums in Europe. The data was collected during the period 29 Oct 2012 – 15 Jan 2013. Micro-blogging comments (total of 193) were provided by 25 of the users, and were semantically augmented with ViewS for IC (see Table 3 for a summary). Table 3. Summary of the ViewS annotation of the micro-blogging content in the study.
Annotations Distinct Entities
Ontology
Episode
Greetings
Dinner
Bill
Goodbye
All
WNAffect
82
84
18
8
192
Body Language
311
236
100
76
723
WNAffect
36
36
11
5
57
Body Language
76
63
43
33
106
The output of ViewS was shown to two simulator designers (with background in Psychology) who were involved in the creation and improvement of the simulated scenario. The semantic augmentation output was visualised in the form of semantic
maps to enable exploration of both ontologies - WNAffect and Body language-, and identification of ontology entities that linked to user comments and their location over the ontologies (see examples in Figures 3, 4). The exploration allowed filtering based on episodes and user groups, and comparison of the resultant semantic maps. 4.2
Semantic Social Sensing with ViewS
The simulator designers were shown semantic maps providing: (i) overview of the annotations for each episode; (ii) comparison between different episodes; and (iii) exploration and comparison of user groups. For each semantic map, the designers were asked if they could see anything interesting and, if so, how it could be helpful for them. Designers’ observations and feedback were driven by the key challenges they were facing: (i) getting an insight of the user reactions with the simulator; and (ii) improving the simulation scenario to make it more realistic and engaging. Identified benefits and limitations of ViewS for simulator enrichment are sumarised below. Overview. The simulator designers were first shown an overview of the annotations of comments from all users for each simulation episode.
Fig. 3. The semantic map of annotations (highlighted nodes) for all comments on the Greetings episode. It shows a cluster of negative emotions around dislike and anger (top, centre), a cluster of positive emotions, and few ambiguous emotions on the right side.
Exploring the semantic maps, the designers noted that: (i) there were several clusters of positive and negative emotions in WNAffect (Figure 3); this confirmed the designers’ expectation that the simulation experience should trigger links to these emotions; (ii) the body language entities clustered around social interaction and psychological process, which was also seen as an indication of the desired effect of the simulation; (iii) the formed ontology clusters were also characterised by the designers as "hot-topics" on which more attention could be given in the simulation (e.g. by extending the feedback or by adding more situations); (iv) the ontology hierarchy was helpful to see the depth (abstract or specific) of the emotions and body language entities - the designers commented that this could give them suggestions for the range of emotions they could include in further situations or for improving the existing situations.
Gateway to comments. In addition to the semantic overview, the designers pointed out the need for examining the learners’ comments in order to get deeper insight into the content. The ontology clusters facilitated the examination of user comments in semantically close groups related to specific ontology regions. The de-
signers were specifically interested whether the comments were related to personal experience or to the simulated situation, which was not possible to analyse with ViewS. Manual inspection of several groups of comments indicated that most comments referred to either personal experience or rules the learners were following (see examples in Table 4). Some of the comments were seen as helpful to enrich the feedback provided to the learner or to add more options for response in the simulator. Table 4. Example comments and annotated WNAffect entities in the Greetings episode. Negative
Just two days ago I discussed with a friend of mine because he was really late...I felt so angry!
anger, fury
Positive
Being on-time helps to build confidence especially when meeting new people for the first or second time
confidence
Comparison between episodes. Comparison of semantic maps enabled comparing different episodes. For example, the content related to the Bill episode did not refer to many WNAffect entities (Figure 4, left), compared with the Greetings episode (Figure 3). The designers found such comparison useful because it provided a sensor of which simulation parts would require further improvement and in what direction (e.g. the designers noted that the Bill episode could be improved as it did not have many branches and situations, and hence did not provoke much user comments linking to emotion entities). Furthermore, semantic maps for the same episode were compared (an example is shown in Figure 4, right). The designers found such comparisons helpful for balancing elements of the simulation in the same or different situations.
Fig. 4. The WNAffect (left) and body language signal meaning (right) semantic maps of annotations (highlighted nodes) for the Bill episode. Example comment You can make a softer gesture with your palm when you want someone to hold and relax while you take care of things.
WNAffect easiness
Body Language gesture, relaxation, attention, interest, caution
Comparison between user groups. The designers were able to spatially examine the contributions from different user groups and see the distribution in the semantic map. Interesting observations were made. Comparing the semantic maps with WNAffect annotations of comments by 17-26 years old users and over 27 years old users, it was noted that WNAffect entities by the second user group were broader and covered different levels of abstraction, while the first group linked to a more limited set of entities. Similarly, comparison was made between male and female users, and was noted that the former referred to a broader set of WNAffect entities. The designers were reluctant to draw any conclusions regarding user groups. They noted however that the comparison between user groups could be useful when thinking of the target audiences during the simulation design process. In sum, there was overall a positive feedback about the potential of semantic exploration of social content to provide various ways of sensing what emotions or body language meanings the users noticed in the simulation and recalled from their personal experiences. This could be used to check the designers’ expectations for learners’ reactions with the simulator, or to identify areas for further improvement of the simulation situations, interaction, and feedback. The groupings of content were particularly helpful, and there was a strong desire to explore comments together with ontology concepts. Further text analysis to identify clusters of comments regarding textual content (e.g. personal experiences, rules, simulation feedback) could be helpful.
5
Conclusions
The paper presented a framework, called ViewS, which exploits ontologies and semantic augmentation techniques to analyse social user generated content and to explore diverse viewpoints presented in such content. ViewS provides a semantic sensing approach to get an insight into social content diversity and to identify interesting aspects that can be helpful for enriching a learning environment. ViewS has been validated in the ill-defined domain of interpersonal communication, focusing on social signals. ViewS distinguishes from other approaches by utilising a rich taxonomy of emotions and a prototypical ontology to describe body language for semantic annotation exploiting different enrichment methods. The potential of the approach for enriching learning environments was examined in an exploratory study with a simulated environment for interpersonal communication in business settings. Our immediate future work includes further experimental studies to examine the benefits of ViewS, which will include deeper evaluation with a broader range of simulator designers and relating comments to the learner performance and individual profiles. Further zooming into the content will be also investigated by exploiting different ontologies to capture ViewS on various dimensions (e.g. in the example presented in the paper, food habits were mentioned but not captured by the semantic augmentation). We are also currently implementing algorithms for semantic aggregation of annotations and content to enable semantic zooming and exploration at different abstraction levels of the semantic output generated by ViewS. Acknowledgments. The research leading to these results has received funding from the EU 7th Framework Programme under grant ICT 257831 (ImREAL project).
References 1. Vassileva, J.: Toward Social Learning Environments. Learning Technologies, IEEE Transactions on 1, 199-214 (2008) 2. Dragon, T., Floryan, M., Woolf, B., Murray, T.: Recognizing Dialogue Content in Student Collaborative Conversation. In: Aleven, V., Kay, J., Mostow, J. (eds.) ITS, vol. 6095, pp. 113-122. (2010) 3. Nejdl, W.: Exploiting User Generated Content to Improve Search. Proceedings of the 2009 conference on Artificial Intelligence in Education: Building Learning Systems that Care: From Knowledge Representation to Affective Modelling, pp. 5-5. IOS Press (2009) 4. Thakker, D., Despotakis, D., Dimitrova, V., Lau, L., Brna, P.: Taming digital traces for informal learning: a semantic-driven approach. Proceedings of the 7th ECTEL (2012) 5. Ziebarth, S., Malzahn, N., Hoppe, H.U.: Matchballs – A Multi-Agent-System for Ontology-Based Collaborative Learning Games. In: Herskovic, V., Hoppe, H.U., Jansen, M., Ziegler, J. (eds.) Collaboration and Technology, vol. 7493, pp. 208-222. (2012) 6. Ammari, A., Lau, L., Dimitrova, V.: Deriving Group Profiles from Social Media to Facilitate the Design of Simulated Environments for Learning. LAK (2012) 7. Kazai, G., Eickhoff, C., Brusilovsky, P.: Report on BooksOnline'11: 4th workshop on online books, complementary social media, and crowdsourcing. SIGIR 46, 43-50 (2012) 8. Vatrapu, R., Teplovs, C., Fujita, N., Bull, S.: Towards visual analytics for teachers' dynamic diagnostic pedagogical decision-making. Proceedings of the 1st International Conference on LAK, pp. 93-98. ACM, (2011) 9. Brusilovsky, P., Hsiao, I.H., Folajimi, Y.: QuizMap: Open Social Student Modeling and Adaptive Navigation Support with TreeMaps. In: Kloos, C., Gillet, D., Crespo García, R., Wild, F., Wolpers, M. (eds.) Towards Ubiquitous Learning, vol. 6964, pp. 71-82. (2011) 10. Lynch, C., Ashley, K.D., Pinkwart, N., Aleven, V.: Concepts, Structures, and Goals: Redefining Ill-Definedness. Int. J. Artif. Intell. Ed. 19, 253-266 (2009) 11. Mazza, R., Dimitrova, V.: CourseVis: A graphical student monitoring tool for supporting instructors in web-based distance courses. IJHCS 65, 125-139 (2007) 12. Thomas, P., Labat, J.-M., Muratet, M., Yessad, A.: How to Evaluate Competencies in Game-Based Learning Systems Automatically? In: Cerri, S., Clancey, W., Papadourakis, G., Panourgia, K. (eds.) Intelligent Tutoring Systems, vol. 7315, pp. 168-173. (2012) 13. Johnson, W.L., Ashish, N., Bodnar, S., Sagae, A.: Expecting the Unexpected: Warehousing and Analyzing Data from ITS Field Use. In: Aleven, V., Kay, J., Mostow, J. (eds.) ITS, vol. 6095, pp. 352-354. (2010) 14. Hays, M., Lane, H.C., Auerbach, D., Core, M.G., Gomboc, D., Rosenberg, M.: Feedback Specificity and the Learning of Intercultural Communication Skills. Proceedings of the 2009 conference on Artificial Intelligence in Education: Building Learning Systems that Care: From Knowledge Representation to Affective Modelling, pp. 391-398. (2009) 15. Wang, N., Pynadath, D., Marsella, S.: Toward Automatic Verification of Multiagent Systems for Training Simulations. In: Cerri, S., Clancey, W., Papadourakis, G., Panourgia, K. (eds.) Intelligent Tutoring Systems, vol. 7315, pp. 151-161. (2012) 16. Muir, M., Conati, C.: An Analysis of Attention to Student – Adaptive Hints in an Educational Game. In: Cerri, S., Clancey, W., Papadourakis, G., Panourgia, K. (eds.) Intelligent Tutoring Systems, vol. 7315, pp. 112-122. (2012) 17. Thakker, D., Dimitrova, V., Lau, L., Denaux, R., Karanasios, S., Yang-Turner, F.: A priori ontology modularisation in ill-defined domains. Proceedings of the 7th ICSS, pp. 167170. ACM, Graz, Austria (2011) 18. Vinciarelli, A., Pantic, M., Bourlard, H.: Social signal processing: Survey of an emerging domain. Image Vision Comput. 27, 1743-1759 (2009)
