University of Southern California/ Information Sciences Institute. 4676 Admiralty Way, Marina del Rey, CA 90292. {jihie, ychang, sencai, siddarth}@isi.edu.
In proceedings of the International Conference on Intelligent User Interfaces IUI-2012
PedConnect: An Intelligent Assistant for Teacher Social Networking Jihie Kim, Yu-Han Chang, Sen Cai and Siddharth Jain University of Southern California/ Information Sciences Institute 4676 Admiralty Way, Marina del Rey, CA 90292 {jihie, ychang, sencai, siddarth}@isi.edu ABSTRACT
Social networking has gained immense traction in many areas, including teaching and learning. Networking sites for teachers aim to facilitate teacher communication and information sharing, but fall short of their potential. In order to support more effective use of online resources and better communication among teachers, we develop a suite of new user modeling and recommendation capabilities within a middle school teacher networking site. We foster collaboration among novice and experienced teachers when they share similar interests, enabling new mentoring relationships, and promote the use of relevant educational resources. We illustrate our approach with an implemented system called PedConnect that analyzes user activities and presents intelligent suggestions for collaboration and resource use. Author Keywords
User profiling; teacher social networking; topic modeling. ACM Classification Keywords
H.5.2 [Information Interfaces and Presentations]: User Interfaces – User interface management systems; I.2.0 [Computing Methodologies]: Artificial Intelligence. General Terms
Human Factors, Algorithms. INTRODUCTION
Two challenges face educators today: shrinking resources and proliferating digital content. Some social networking sites for teachers facilitate teacher communication and information sharing. The Middle School Portal 2: Math and Science Pathways (MSP2, http://msteacher2.org) is a Ning based social networking site for middle school math and science teachers (see Figure 1). Such sites can help teachers maintain professional support and integrate new digital teaching methods and materials, without costly workshops and professional development courses.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. IUI’12, February 14–17, 2011, Lisbon, Portugal. Copyright 2012 ACM 978-1-4503-1048-2/12/02...$10.00.
Figure 1. MSP2: Social networking portal for middle school science and math teachers.
A major shortcoming of current social networking systems is their inability to assist teachers in navigating both the static content and dynamic social landscape of online communities. While search functionalities are supported in many sites, as the amount of related online resources grow, teachers need to be able to filter them with respect to their own individual interests and pedagogical characteristics. Another issue is lack of communication. Although the sites provide opportunities for teachers to collaborate and communicate within and across different school districts using online forums, blogs, and wikis, most of the participants remain silent. Among 1900 members on MSP2, only 10% are active. More than 70% of users who did not get their first question answered never came back to the forum. Although the MSP2 site has a few active users assisting novice teachers, they are not fully effective in guiding interactions since they cannot keep track of all the individuals. Teachers need intelligent services that that keep track of participant interests and needs, and connect them to resources or potential advisors within the social network site. This paper describes a new teacher assistant called PedConnect that uses an integrated approach for modeling user interests and providing assistance. The approach combines many sources of information, including forum and blog data, external topic ontologies, resource metadata, and user inputs, and generates interest models for typical teacher tasks and subject topics. Such interest models are used to promote participation in forums, connect teachers to others
who can potentially answer their questions, and suggest relevant educational resources. RELATED WORK
There have been several efforts to develop recommender systems in social networking sites. For example, Guy et al. [8] connects people who possibly have an existing relation over social network. Our work focuses on discovering mentorship among potentially unacquainted people, and modeling information seeking or providing activities than analyzing relational properties. Other work provides item recommendations based on user interests, including educational needs of children [2]. As many of our educational resources have metadata that describe the content of the resources, we were able to make use of the terms in generating richer models and relating them to contributions made by individual users. In forum analysis research there have been efforts to find experts or rate user reputations [3]. The quality of forum posts are analyzed and compared against reputations. In contrast, our work focuses on analyzing topic mentorship in terms of information sharing behavior in the forum, and connecting potential mentors to help seekers. Harper et al.’s work [9] described the difficulty of inviting users to participate in forums, suggesting that invitations worded to stress the social nature improved participation; this work may provide us with useful ways to tune the wording of PedConnect’s suggestions and invitations. Many papers from past IUI conferences are also relevant here, such as [7] describing a unified system for browsing and searching. APPROACH
from teacher handbooks [5,6,10] and profile questions used by the MSP2 site [11]. Table 1 lists the categories. After a few iterations of annotations, the kappa-values [4] for interannotator agreement are rather high (> 0.9). In generating models for individual users, several different sources of information are integrated to automatically calculate the degree of interest and mentorship in each area. The first source is user answers when they register: The user can specify which areas he needs help on and which areas he can help with. Another source is forum and blog contributions. More answers or comments on a topic area increase the degree of mentorship for that topic. Likewise, questions on a topic area increase the degree of help needed by the user on the topic. PedConnect also provides an interface for users to explicitly modify or add his/her interests. Using this information, mentor/mentee levels for each area for each user are calculated as follows: For user i and topic j, the topic mentor/mentee levels are: Tij = w1*(Profile answer for tag j) + w2*(User selection of tag j) + w3*(Normalized forum/blog contribution on tag j) The mentor/mentee levels for each user are updated as the user makes new contributions or changes his/her interests. If the topic mentor/mentee level is greater than a threshold then the user is classified as a mentor/mentee in that area. We plan to explore machine learning approaches as we collect more user feedback. Tags activity
Description After class activities Assessment/Testing/ assessment Examinations/Evaluation assignment Homework/Assignment Communication with Students outside communication classroom Curriculum/Class Component/ curriculum /Supplemental Materials demo In-class Knowledge demonstration discipline Discipline, learning environment It Integrating Technology parent Involving parents and caregiver special Special Needs students
Kappa 0.95 0.98 0.96 0.95 0.96 0.94 0.99 0.91 1.00 0.98
Table 1. Teacher Task Topics and Annotation Kappa Modeling Subject Areas
Figure 2. PedConnect: Connecting teachers to forums, resources and other participants. User Topic Interest Modeling Modeling Task Interests and Mentorship
We first model teacher interests using typical teacher tasks. We found most discussions within the site are about such teacher tasks and can be used to identify mentors and mentees in each topic. We derive the teacher task categories
We also model users based on their interest in Math and Science subject areas. First, we collect the domain concept terms in the National Science Digital Library math and science ontology [12]. We then we enrich the term set by retrieving relevant NSDL resources and extracting the additional domain terms used in resource metadata. We exploit the rich ‘description’ metadata in NSDL in generating the term lists that is used for representing subject interests. For each user, we keep track of term frequencies in his/her forum and blog contributions. The terms searched by the user are also included in modeling subject interests. As we remove old data over time, the term list reflects the most recent interests of each user. PedConnect recommends resources based on this term list.
Promoting Forum Participation
interface also allows the user to make changes to their profile subject terms if necessary.
Figure 5. PedConnect: ‘Resource Recommendation’ View Figure 3. PedConnect: ‘Interesting Discussions’ View
Using the above mentor/mentee levels for each topic, the system identifies recent discussions that the mentors can participate in. Potential discussion participants are determined by matching inferred user interests with inferred discussion topics. When topic mentors log into the system, PedConnect shows discussions where their contribution would be helpful under the “Interesting Discussions” tab (Figure 3). Promoting Communication among Teachers
PRELIMINARY RESULTS Prediction of forum participation
We use existing MSP2 data from 2008-2011 in evaluating our user topic model and forum recommendation. For each discussion thread, we produce a ranked list of potential participants, ordered by their degree of topic match against the initial post. We use the actual discussion participants subsequent to the initial post as a proxy for labeling “correct” user recommendations, i.e. a user that would have participated in the thread, had the user been recommended to be a participant. This rough approximation underestimates the likelihood of participation, since actual participation was done without the PedConnect suggestions, and most users simply do not browse through all the possible discussion threads to discover threads where they could participate.
Figure 4. PedConnect: ‘Your Questions’ View
PedConnect promotes communication among peer teachers in two ways. First, when the user logs into the system, under the “Your Questions” tab, for each question posted by the user, potential helpers who can respond to his/her questions are shown (Figure 4). These users share similar interests and have been answering related questions. Second, PedConnect recommends interesting discussions or other user’s questions in order to get the current user involved (Figure 3). Promoting Resource Usages
Figure 5 shows the resource tab of PedConnect. By matching the subject terms of resource descriptions with the popular keywords in the users’ conversation history, the most relevant resources are selected and presented to the user. The
Figure 6: Recall for forum participation prediction
We calculate the recall based on the top results in the list based on some threshold. For example, assume we choose a threshold of 10. We look at the actual set of participants, and calculate the percentage of those participants who appear within the top 10 users in PedConnect’s recommended list. The precision is harder to evaluate, as many users with matching topic interests do not often participate, potentially because they were not aware of the discussion. Figure 6 shows the results for different thresholds of top N users. It shows two recall scores for PedConnect: one that includes
scores for all users, even those without any data, and one that only includes scores for users who have interacted with MSP2 in some way, such as a post or blog item. We compare these curves with two other methods: a baseline where users are recommended with a random rank, and one heuristic based on a list ordered by a user’s activity level. We expected this heuristic to do quite well, since active users are likely to participate in many discussions. We do not plot the precision since it is misleading for the reasons already described, e.g. at the threshold of 30, with the recall above 0.6, the precision is less than 0.1. That is, only 2 or 3 of the 30 are “correct.” However, we may very well expect that many of the “incorrectly” recommended people might actually have participated in the thread, had they been made aware of the discussion. PedConnect performed significantly better than others due to its use of the context information (discussion topic). Further, if we remove users for whom we don’t have any information i.e. no profile answers and no mentor/mentee classification, the algorithm improved further. This shows that the more information we have about a user, the better we can predict whether the user is likely to participate in the discussion or not. Initial User Feedback Feature
Interesting Discussion
Resource Recommendation
Search Engine
Question category Is the task topic prediction accurate? Will you be able to help this discussion? Is the discussion topic prediction accurate? Is this discussion relevant to your interests? Is this math & science interest prediction accurate? Are the recommended resources relevant to your interests? Are recommended resources useful?
Avg. Rating 4 3.5 3.3 4.3 3.3 4.3 3.3
Are discussions relevant to the topics?
3.75
Are the recommended resources relevant to the topics?
3.5
Table 2. A summary of the user feedback.
Two teachers participated in the initial study and were asked to rate the results’ usefulness/relevancy/likelihood on a scale of 1-5 (Not at all, Slightly, Moderately, Very, Completely). For each question category, 2-3 items were presented for evaluation. Table 2 summarizes average ratings over each category, which contains 4-12 questions. The averages range between Very and Moderately. They liked the PedConnect concept and expected that the system will encourage participation. “I like what I see and think that it will encourage participation.”
CONCLUSION AND FUTURE WORK
We presented PedConnect, which assists social networking by promoting user interactions and resource usage. The system integrates diverse information from the social networking site and leverages the availability of rich metadata about educational resources. We identify user topic interests and mentorship using their participation in forums and blogs. We propose several different ways to promote user participation and user interactions using the topic interests and mentorship. The technique is applied in an important domain area, where successful networking and recommendations has the potential to greatly improve education by empowering teachers with knowledge from online peers, mentors, and resources. We will continue with this approach by integrating more sources of information to expand the user profiles, improve profile accuracy, and automatically model interests that potentially change over time. We will also investigate alternative ways to promote participation. The system is fully integrated with MSP2 and will be deployed in the coming months, and we will begin evaluation of user behavior in the deployed system. ACKNOWLEDGMENTS
This material is based upon work supported by the National Science Foundation under Grant No. 1044427. REFERENCES
1. Anderson, R.E. Social impacts of computing: Codes of professional ethics. Social Science Computing Review 10, 2, (1992). 2. Brusilovsky, P., L.N. et al., Social navigation for educational digital libraries. Procedia Computer Science 1,2 (2010). 3. Chen, B., Tseng, B., Yang, J., Guo, J. User Reputation in a Comment Rating Environment. In Proc. KDD 2011. 4. Cohen, J., A coefficient of agreement for nominal scales, Educational and Psychological Measurement, 20 (1960). 5. Danielson, C., et al. Implementing Framework for Teaching in Enhansing Professional Practice (2009). 6. Fredericks, A. D. The Teacher's Handbook: Strategies for Success. Rowman & Litlefield Education (2010). 7. Freyne, J., Smyth, B., Farzan, R., Brusilovsky, P., Collecting Community Wisdom: Integrating Social Search & Social Navigation. In Proc. IUI 2006. 8. Guy, I., et al., Do you know?: recommending people to invite onto your social network, In Proc. IUI 2009. 9. Harper F., et al., Inviting Users To Participate in Online Discussions. In Proc. IUI 2006. 10. Klein, M. B. New Teaching and Teacher Issues. Nova Science Publishers, 2006. 11. MSP2. The Middle School Portal 2 Math and Science Pathways, http://www.msteacher2.org/, 2011. 12. NSDL. The National Science Digital Library, http://nsdl.org, 2011.
