Predicting Individual Priorities of Shared Activities Using Support Vector Machines Lida Li*, Michael Muller**, Werner Geyer**, Casey Dugan**, Beth Brownholtz**, David R Millen** *School of EECS, Oregon State University 1148 Kelley Engineering Center Corvallis, OR 97331, USA Tel: 1-541-737-1646
**IBM T.J. Watson Research One Rogers Street Cambridge, MA 02116, USA Tel: 1-617-693-4791
[email protected]
{michael_muller, werner.geyer, cadugan, beth_brownholtz, david_r_millen}@us.ibm.com
ABSTRACT
1. INTRODUCTION
Activity-centric collaboration environments help knowledge workers to manage the context of their shared work activities by providing a representation for an activity and its resources. Activity management systems provide more structure and organization than email to execute the shared activity but, as the number of shared activities increases, it becomes more and more difficult for users to focus on important activities that need their attention. This paper describes a personalized activity prioritization approach implemented on top of the Lotus Connections Activities management system. Our prototype implementation allows each user to view activities ordered by her/his predicted priorities. The predictions are made using a ranking Support Vector Machine model trained with the user’s past interactions with the activities system. We describe the prioritization interface and the results of an offline experiment based on data from 13 users over 6-months. Our results show that our feature set derived from shared activity structures can significantly increase prediction accuracy compared to a recency baseline.
Knowledge workers typically manage multiple tasks with different people at the same time. The idea of activity-centric collaboration [15] is to support knowledge workers in managing and switching between multiple tasks by providing the computational concept of a work activity that represents the task and the resources involved. A series of activity management systems have been proposed, developed, or productized (e.g. [13], [15], [22], [27], or [28]). Although the definition of an activity varies across different authors, the many concepts are agreedupon. For example, Geyer et al. [15] describe an activity as a ”logical unit of work that incorporates all the tools, people, and resources needed to get the job done. Examples for activities are: preparing an executive meeting, planning a conference, closing a sale, […], or writing or responding to an RFP.” Activity management systems encourage knowledge workers to structure their everyday work as activities. Many systems support collaboration by allowing users to share content in the activity structure and by making changes or updates visible to other users. With an increasing number of activities, users experience an information overflow problem similar to the well-known problems of email: too many updates are coming in at the same time (e.g., [35]).
Categories and Subject Descriptors
H5.2 [Information interfaces and presentation]: User Interfaces. - Graphical user interfaces. G.3 [Probability and Statistics]: Experimental design, Probabilistic algorithms
Most activity management systems provide list views of activities ordered by recency of the latest modification to each activity or any of its component resources. A problem with this strategy is that all updates are treated equally, regardless of other factors such as the content of the update, the user’s role in the activity and the user’s relationship with the person who updated a component of the activity. To resolve this “activity overflow” problem, one solution is to rank activities by priority. Unlike ranking in web search engines, priorities of activities change more dynamically and may benefit from personalized models for each user.
General Terms
Design, Experimentation, Human Factors
Keywords
Activity-centric collaboration, prioritization, knowledge worker, ranking, support vector machine, activity flood
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. CIKM’07, November 6-8, 2007, Lisboa, Portugal. Copyright 2007 ACM 978-1-59593-803-9/07/0011...$5.00.
In this paper, we present a system and algorithm for prioritizing shared activities using machine learning techniques. We implemented our approach on top of the Lotus Connections Activities system (Lotus Activities) [26] and we integrated our user interface into Malibu ([16], [30]), a desktop client that provides access to Lotus Activities. We formalize the activity prioritization problem as a learning-to-rank problem, and we use a
515
selected document. The algorithm presented in this paper was inspired by Joachims [21]. We chose the Ranking SVM model fabecause it fits the pair-wise training data generated by users of our application very well. Also the kernel trick of it provides the ability to learn a non-linear decision boundary.
ranking Support Vector Machine (SVM) as our underlying model [21] Learning is supervised; first we retrieve activity data for each individual user from the centralized activity server. Then we create training data based on a user’s historic access and interaction patterns with his/her activities. During runtime, when new updates appear, the system predicts priorities of all activities based on the trained model and a feature set. The prediction results are shown in Malibu as a rank ordered list of activities.
Much research has been done on ranking web search results. (e.g. [1], [4], [5], [11], [21], [31], or [36]). Ranking web search results differs from ranking activities in that activity priorities are almost always personalized to the user, an activity is likely to change more frequently than a web page [9], and the set of individual activities to rank is much smaller than the number of web pages.
To verify the effectiveness of our system, we conducted an offline experiment with the data collected from 13 high-volume Lotus Activities users during a period of about 6 months. The experimental results indicate that our approach performs significantly better than conventional recency methods.
2.2 Activity Management Systems
Activity management systems can reduce information overflow, decrease search times, and improve interruption recovery because they support knowledge workers in maintaining multiple working contexts. Existing activity management systems fall into two categories: personal activity management systems and collaborative activity management systems.
The rest of the paper is organized as follows. Section 2 reviews related work. Section 3 introduces the Lotus Activities product and the Malibu client used for our work. Section 4 illustrates the architecture and the user interface of our prioritization system. Section 5 describes the ranking SVM and the feature set of our prioritization approach. Section 6 presents the experimental results, comparing our approach to two baseline recency methods. Section 7 summarizes the work and discusses future directions.
Personal activity management systems organize information on a user’s desktop in ways to provide information access in context, interruption recovery and information finding. For example, Bellotti et al. [2] introduce TaskMaster, a threaded activity-centric email client which enables users to associate incoming email messages with existing activities and other resources. TaskTracer [13] is an activity management system that allows users to label resources on the desktop as being part of an activity. The system learns based on the labels and can automatically associate resources on the desktop with activities. Both TaskMaster and TaskTracer require user input to define their activities. Mitchell et al. [25] use an unsupervised clustering approach to automatically extract activities from emails, online calendars, and contact names on the desktop.
2. RELATED WORK 2.1 Ranking Algorithms
Our goal of prioritizing activities can be reduced to the problem of finding a priority ordering or ranking across activities. Recently, machine learning approaches have treated ranking as a supervised learning problem. Researchers have developed ranking methods based on the classical classification methods including Perceptron, neural networks and support vector machines (SVM) (e.g. [1], [4], [11], [21], [19], or [36]). Cohen [11] proposes an early approach to learning ranks in domain-specific research. Training is done by user feedback in the form of preference judgments on two items. Cohen’s algorithm then learns a pair-wise linear preference function. The complete order of a set of items is learned by making it agree with the learned pair-wise preference function. The disadvantage of this approach is that finding the complete order with pair-wised preference function is NP-Complete.
Collaborative activity management systems support activity coordination and collaboration among multiple users. Knowledge workers are typically involved in multiple activities and collaborations (e.g. [7], [18], [20], [28], [28]). The Haystack project [31] provides integrated presentations of diverse media that are assumed to be shared, such as email, instant messaging and web pages. It enables some awareness of how others are manipulating the artifacts. Activity Explorer [15], Unified Activity Management [27], and the Lotus Activities product [26] manage multiple types of shared items, support structured collections of items, and provide access control at the level of items and collections. Activity Explorer also supports notifications of updates to activities or their components ([29], [32])
Herbrich et al. [19] and Crammer et al. [12] apply ordinal regression on SVMs and Perceptron models. Both of them create a connection between the classification and the ranking by modeling ranks as intervals of real numbers. Then the problem becomes learning a rank function that maps an input vector into a real number from the training data.
Existing research in this area focuses mainly on information organization and interruption recovery. The new work described in this paper helps knowledge workers to cope with activity overflow, i.e. with prioritizing an increased number of activities. This problem is similar to managing large numbers of email messages [35] or instant messages [18].
In learning a rank model, one difficult part is to obtain training data. Training based on ranked pairs of items is more popular in existing work because it is easier to get a partial order than the whole order especially given a large number of items. For example, a partial order can be extracted from explicit user feedback such as the rating of pages (see [5], [36]). Other more user-friendly approaches employ implicit user feedback by capturing user behavior. For example, Joachims [21] proposes an extension to SVMs using users’ clickthrough data on the search results to refine the ordering of future search results. Training pairs are generated based on the assumption that, in an ordered list of search results, a document selected by a user has a higher priority than other non-selected documents in the list before the
3. ACTIVITY MANAGEMENT SYSTEM 3.1 Lotus Activities
Our prototype leverages Lotus Activities, IBM’s activity management system ([22], [26]) which emerged from a multi-year research effort on activity-centric computing ([15], [27], [28]). Lotus Activities organizes and integrates resources, tools, and
516
G H B A
C
D
E
F
Figure 1. Activity web page with activity outline on left.
Figure 2. Malibu Side Bar
structured based on the response hierarchy in an activity, but it can also be reorganized post-hoc if a user wants to create their own structure within the activity.
people around the computational concept of a work activity, with the goal of increasing work quality and efficiency. The system consists of a centralized, web-based service and includes many extensions for existing desktop applications. The design of Lotus Activities was driven by the goal of organizing work into shared or private activities. In this system an “activity” consists of a structured set of related, shared resources representing a task or project. The set of related resources is structured as a hierarchical thread called an “activity outline”, representing the context of the task at hand. The activity outline (see (A) in Figure 1) is similar to an activity thread [15], with more explicit structure than a working sphere [18] and greater diversity of resources than a thrask [2]. Users add items to activities by posting either a response to an existing item or a new resource, such as a file, comment, chat, URL, task etc. Complex tasks can be structured through subactivities. The system is extensible, i.e. new resource types can be added to customize the system to particular application scenarios.
3.2 Malibu Personal Productivity Assistant
As noted above, a successful activity management system can lead to the problem of activity overflow. Malibu [16] was developed to assist knowledge workers in their activity-centric work providing a peripheral list of current activities and related resources. It runs as a desktop side bar (“Malibu Board”) that either slides out when users hover with their mouse at the left or right side of the screen or remains “sticky” and always visible on the desktop. The Malibu Board contains a series of configurable views, each one displaying one data source (see Figure 2): My Activities (D) provides access to Lotus Activities as described in the previous section; Dogear Bookmarks (E) displays bookmarks from the Dogear social bookmarking system [24]; and My Feeds (F), is a general purpose feed reader. Dynamic views are frequently updated with the most recent or most relevant items. As such, Malibu provides peripheral access to and awareness of multiple data sources while the user focuses on her main work on the desktop. With this design, Malibu becomes an always-on companion that can display information that is contextually relevant to the current desktop activity of the user, draw attention to new and important events, and provide quick access to data sources that need attention. Because of these characteristics, we decided to integrate the prioritization scheme into the Lotus Activities view in Malibu. Also, since Malibu is a desktop front end to activities, it allowed us to
Activities and resources within them can have simple metadata associated with them: name, description, tags, and optional due date (see (B) in Figure 1). Keyword-tagging allows users to find related resources in a single activity or across activities. Users can also search for terms within a specific activity, or across all their activities. Each activity has a web page associated with it (shown in Figure 1) so that users can see recent entries posted to the activity, navigate the activity outline, see all the entries in the activity organized by type, and see the history of the activity (C). Similar to Activity Explorer [15], the activity outline (A) is automatically
517
offload computation onto a client machine without modifying any of the Lotus Activities server code.
the user’s activities. The computed priority scores are sent to the Malibu UI to update the list of activities in the Activities view.
4. ACTIVITY PRIORITIZATION SYSTEM
Our design imposes no extra processing costs on existing Malibu users. The data retrieval and ranking component both work in the background with low computation overhead. Although the learning algorithm is supervised, we don’t bring extra user cost because the training data is from the logged history of activities stored in the centralized activity server.
In Malibu, the activity view (Figure 2 D) is the user interface to inspect and manipulate activities. The default version of Malibu is like other activity management approaches: All activities are listed in the order of their last update time by default. When there is a new update to one activity, the font of the activity becomes bold until the user accesses this activity. In Malibu, users can manually set an activity to be important or not important by right clicking an activity and selecting “Set Priority” in the context menu. By default, each activity is neither important nor non-important. Important activities are displayed with a purple dot and non-important activities are displayed with a blue dot. Activities are sorted by recency per default. Users can filter the view to display only the important activities. Our activity prioritization system is implemented as a Malibu plug-in that enhances the activity view of the original Malibu version of [16] and [30]. Figure 3 shows the modified activity view. It supports both manually-specified priorities and machine predicted priorities using the algorithm described in this paper. We added new functionality to the Activities display through a prioritization feature based on machine-learning. As shown in Figure 3, we decorate the top 5 activities with flags according to their predicted priority scores. In addition to recency, users can sort activities by predicted priory scores in each group (important, without-settings, non-important).
Figure 4. Architecture of the Activity Ranking System
5. PRIORTIZATION ALGORITHM
To build a personalized activity prioritization system, we formalize the problem as a learning-to-rank problem. We employ Ranking SVM [21] as the ranking algorithm. The learning features are created from shared activity data stored in the centralized activity server and the training data are retrieved from users’ access history.
5.1 Problem Formalization The problem of prioritizing activities is defined as follows: For v v v v all m activities (a1 , a 2 , K , a m ) , at time t , each activity ai is represented by an n-dimensional feature vector: ( x1 , x2 ,K, xn ) .
Figure 3. The activity view of Malibu integrated with the prioritizing system. The color of the dot indicating the user set importance and the activities with a flag are the top 5 activities in the predicted priority list. Figure 4 depicts the architecture of our activity prioritizing system. It consists of the data retrieval component, ranking component and a local database. The data retrieval component retrieves activity data periodically from the Lotus Activities server via its Java and Atom API. For performance consideration, the activity data is cached locally in an embedded database. At the same time, activity data are sent to the ranking component. The ranking component updates its local activity representation including feature values and generates the new priority scores for
Each feature
xk
indicates a specified attribute value at time t .
We are trying to find a ranking function R whose input is the vector
v v ai and the output is a score S i = R ( a i ), S i ∈ ℜ
.
The larger the score is, the higher the priority of the activity. The problem of prioritizing activities is to find an appropriate ranking function R .
518
To incorporate time as a factor, most features are either computed over time or weighted with an exponential decay factor (see, e.g., item 5 in Table 1). Also, in order to avoiding numeric scale problems, we normalize each attribute across all activities. All features are constructed from the data on the centralized Lotus Activities server.
In our setting, the training samples used to learn the ranking function are in the form of pairs of activities representing a partial order of priorities. For example, the input pair represents that the priority of activity priority of
v activity a j
v ai
v v ( ai , a j )
is higher than the
. We use pair-wise training because the
Each feature can be regarded as a weak measure to sort priorities of activities. We performed a feature analysis on the data collected from 13 real users for 6 months. Figure 5 shows the distribution of the rank of the activity being updated by individual features in our experimental dataset. While user recency and activity recency perform well when used as individual features, more than 30% of the time these two features fail in predicting that an activity is among the top five most important activities.
training sample can be inferred from users’ history and the Ranking SVM model works with pairs. The training pairs are generated as follows: We assume a daily time frame. At time t ,
v
if a user updates activity ai , we assume that the activity
v ai has
higher priority than all other activities not updated on the day of time t. For all activities, updated on the day of time t, we do not generate training samples. We make the assumption that a user updates activities following their cognitive (i.e., non-modeled) priorities. Although this assumption is not “perfect”, it is a good enough approximation and avoids imposing the extra cost on the users to supervise or correct the learning algorithm.
5.2 Ranking SVM
We employ Ranking SVM as our model to perform the task of learning the priorities of activities. The SVM approach has proved to be an effective learning method in classification and regression [34]. Ranking SVM [21] is an extension of standard SVM to support ranking. It can be directly applied to resolve the formalization of the activities prioritization problem that we described in Section 5.1. In Ranking SVM, each item is
5.2 Feature Construction
Finding suitable features is essential to build an effective ranking system. Our activity prioritization system utilizes the ten features in Table 1 to create a feature vector representing an activity.1 The features in Table 1 are intrinsic features, i.e. they are based on data from the activity only. Extrinsic features such as users’ recent desktop activities, emails or IM messages, are valuable too and we are looking into incorporating those in future work.
represented by an n-dimension vector
be thought of as a data point in the n-dimension space. The training samples are pairs of items in the form of the rank of
Lotus Activities provides rich information to understand the context of each activity. Every update to an activity is logged including the type of action that produced the update, the user, the updated entries and tags, and the timestamp of the update. The history record of an activity shows all updates and provides a detailed record of how a user accessed the activity. The threaded (i.e., hierarchical) structure of activities shows the interactions between the user (responses) and other members of the activities. The user-assigned tags categorize items and give clues about what the user has been working on. Time and date are very important since priorities of activities change over time. An activity’s priority can increase significantly when there is a new update.
training
v ai
v
v v (ai , a j ) with
being higher than the rank of a j . Given a set of
samples,
y = wi xi where
Ranking
SVM
finds
a
hyper-plane
the distance for each data point to the hyper-
plane has minimal discordance with the given training pairs. That
v v
means, for the training pair (ai , a j ) . The distance from the point
v ai to the hyper-plane v a j to the hyper-plane.
should be larger than the distance from Therefore, our ranking function computes
the distance from the given sample to the learned hyper-plane. Similar to a classification SVM, to find the hyperplane
The feature construction criterion is to find scores that intuitively reflect the importance of an activity to the user at the current time. We based our model on the following features:
y = wi xi
with
r w = ( w1 , w2 , K, wn ) ,
ranking SVM
resolves the following optimization problem: k 1 rr ww + C ∑ ξijk 2 1
•
a count how active the activity is, based on items (2) and (3) in Table 1
Minimize:
•
a measure of how actively the user is involved in the activity, based on items (1), (4) and (5) in Table 1
Subject to:
•
a measure of collaboration and social measures based on items (6), (7) and (9) in Table 1
For the k given training pairs
•
a simple indicator variable of the recency of a user’s joining an activity, which is 1 when users are added to the activity and 0 after a period of time, based on item (8) in Table 1
•
a measure of tag recency, based on item (10) in Table 1
1
(x1 , x2 ,K, xn ) and can
v v (aik , a jk )
rv rv waik > wa jk + 1 − ξ ijk
ξijk
is the slack variable and
C
is the trade-off parameter.
Ranking SVM learns a linear ranking function per default. However, it has been shown that different kernels can be applied to a Ranking SVM [21]. This allow us to create a nonlinear ranking function
These features are based in part in our experience with machinelearning models of user preferences for notifications in an activity-centric collaboration system [32].
519
Table 1. Features used to describe an activity No.
Name
Description of the feature
Priority assumption
1
User recency
Number of minutes since the last update of the current user
2
Activity recency
Number of minutes since the last update of any user.
3
Activity update frequency
Number of items updated by the user in the last 3 days divided by the number of members of the activity
Various recency scores assume that an activity is more important for a user if she has accessed it recently or if the activity has been accessed by others recently.
4
User update frequency
Number of items updated by the user in last 3 days divided by the number of members in the activity The score iskcomputed n as
S cr = ∑ oi / ∑ o j
5
User’s creative score
i =1
j =1
oi = β t −ti where t is the current time and ti the time when
the object was created and β ∈ (0,1) . o1 , K on are the
This score reflects how many items a user has created in an activity compared to other users. If the ratio for this user is high, we assume this is an important activity for her.
exponential decay scores of all objects in the activity.
o1 , K ok
6
Shared items within activity:
are the subset for objects created by the user.
Number of shared items in the activity of the current user compared to the user who updated the activity most recently. This is a social score comparing how much two users share in an activity.
Given the set of shared activities
7
Shared activities
k
i =1
Become member recently
between
the current user and the user who updated an activity most recently, and mi being the number of members of activity ai, this value is computed as
S so = ∑ 8
(a1 , a2 , K , ak )
A boolean variable indicating if the user has been added as a member to a to the activity within the past 24 hours
This score is based on the hypothesis that becoming a member of an activity might make the activity important to this user.
(a1 , a2 , K , ak )
This score assumes that if a pair of activities has been accessed on the same day very often in the past, then the coaccessed activity of the current activity has a high priority also today.
accessed on the
d i is the number of the days in the past the current activity and ai were
current day, this score is computed as follows:where 9
This is a social networking score that gives importance to updates of other users with whom the current user has a high number of shared activities in common (i.e. shared access).
1 mi
Given the activities Co-Accessed activities
This is a social networking score that gives importance to updates of other users with whom the current user has a high number of shared items in common within a single activity (i.e. shared access).
both accessed on the same day. n
S co = ∑ d i i =1
Given a tag score
10
Tag recency
where
n
S t = ∑ β t −ti i =1
t is the current time and ti
the time when the tag was
added by the current user and a decay factor β ∈ (0,1) , this score is computed for each activity as the sum of the scores St.of the tags that appear in the activity.
520
This score gives importance to an activity that contains many tags that a user has been recently using to tag all of his activities.
Figure 5. Rank distribution for individual feature computed with a dataset containing 13 users with 62 activities per user on average maximum number of activities per user was 117 and the minimum number was 20.
In our implementation, the Ranking SVM is implemented based on the Java package of LibSVM [8]. We extended the package to support ranking in addition to classification and regression. Since there is no efficient online update algorithm for SVMs, we retrain the Ranking SVM every 5 minutes. The training process runs in the background and is fast enough due to the small number of features. Our test on 13 subjects showed that the average training time on 6 month data took around 3 seconds and the average ranking time took less than 100 ms on an Intel Duo Core 1.6G HZ laptop with 1 GB memory.
6.2 Evaluation Method
Recall that our goal is to assist users to manage the large number of concurrent new updates of activities that may overwhelm them (e.g. [28], [32]). Hence, an ideal activity prioritization system should always (a) be able to identify the activity that the user is most likely to update, and (b) display that activity on the top of the Lotus Activities view in Malibu. This means that there is no extra cost of scrolling to find the right activity. On this basis, we performed an offline experiment as follows:
6. EXPERIMENTAL RESULTS
To evaluate the effectiveness of our activity prioritizing system, we ran an offline experiment with the data of the 13 internal users of Lotus Activities (also used to analyze our feature set in Section 5.1) We compared our Ranking SVM algorithm with the two baseline non-machine learning methods: Activity Recency and User Recency. In our experiments we measure prediction accuracy but also the cost for a user to scroll down in case an important activity is not visible in the activities view.
6.1 Data Collection
The data were collected from a database of the Lotus Activities server deployed on the IBM Intranet. It contained data from 1000 registered Lotus Activities users. This number includes many users who only have very smaller number of activities and updates on activities rarely. We selected 13 users with the largest number of activities. To avoid biased result, we also excluded users who worked on the Lotus Activities product team. The typical users were researchers, managers and software developers. The activities created by the subjects were related to research activities, meetings, presentations and software projects etc. The average number of activities per user was 62.69 (median 62). The
521
•
First we preprocessed the log data as follows: The actions logged in the centralized server include create /delete/undelete activity, create/delete entry, create/delete member and create/delete tag on the entry. We ignored the create/undelete activity actions because there was no way to predict the priority of a non-existent activity. Since the delete activity actions rarely happened in our dataset and they actually meant the activity was not used any more, we ignored them.
•
Then we sorted all the update actions by their timestamps.
•
Each time a user switched to a different activity, we treated the exact moment before the switching as a data point for our experiment and called the switched-to activity the true activity. There were a total of 2683 such data points in our dataset. For each of them, we generated feature vectors for all existing activities at that moment. These feature vectors were used to train and test our algorithm.
•
The entire set of data points was split into a training set and a testing set. The first 70% of the time ordered data points
In the user interface, the size of the activity view determines the number of activities visible without scrolling. If the true activity does not appear very high up on this list, or even worse fails to be visible, then an extra cost would be imposed on the user to find the activity. We assume that the larger the ranked position of an activity is, the higher the cost for the user. Figure 6 shows the percentage of failure cases in which the true activity failed to appear in the activity view as a function of size of the activity view. The error bars shows a 95% confidence interval computed using a binomial test. The number of failure cases (important activities not showing up in the activities view) is significantly lower compared to the two recency baseline methods.
were used for training and the remaining, most recent 30% were used for testing. For each training data point, training pairs consisted of the true activity and other activities not updated on the same day. •
For each testing data point, the ranking SVM algorithm sorted the activities into a list according to the predicted priorities. In this way, we simulated what the activity list looked like when a user was going to switch from one activity to another. If the activity that the user switched to was on the top of the list, that was evidence that the ranking algorithm had predicted accurately.
We used activity recency and user recency as a baseline in our experiment: Activity recency refers to sorting activities by their last update time (item 2 in Table 1). This is the current implementation in Lotus Activities and Malibu. User recency refers to sorting activities by the last update time of the current user (item 1 in Table 1).
In Figure 6, user recency has fewer failure cases than activity recency when the number of displayed activities is small, but more failure cases when the number increases. Analysis show that users often go to an activity that they have not touched for a long time and also, there are many updates in activities that might not be important or relevant to a user in particular if activities have many members . This is an indication that the ranking based on user recency might have a higher variance whereas activity recency might have lower accuracy. Neither recency method performs well. The Ranking SVM algorithm described in this paper improves on the simple recency method by incorporating other factors that influence the priority of an activity.
To evaluate the performance of our implementation, we measured the accuracy when true activity is among the top N (N=1, 3, 5) activities of the lists generated by our Ranking SVM algorithm and the two baseline methods2.
6.3 Results
Table 2 shows the prediction accuracy of the top 1, 3, 5 activities of the list and the mean and median rank of true activity in all test cases. In the experiment, Ranking SVM was configured with C=100 and a RBF kernel with ε =0.25 according to [9] based on a 10-fold cross-validation test on the training set. Table 2. Prediction accuracy of the top n activities Method User recency Activity Recency Ranking SVM
Top 1
Top 3
Top 5
Mean
(%)
(%)
(%)
(Median)
37.48±3.45
60.67±3.48
69.14±3.30
15.10±2.55
44.77±3.55
61.20±3.48
48.21±3.56
70.86±3.24
78.81±2.92
7.40 (3) 7.34 (4) 5.05 (2)
The accuracy is shown with a 95% confidence intervals computed by two tailed signed binomial test. The mean and median of the rank of true activity are also provided. The results show that the Ranking SVM method enhances the two recency baseline methods. The difference is statistically significantly in all top 1, 3 and 5 cases at a 95% confidence level. The algorithm has 78.81% accuracy of predicting an activity to be within the top 5 most important activities and it also has the lowest mean and median values. 2
Figure 6 Distributions of Failure Cases for different size of Activity View
7. CONCLUSION AND FUTURE WORK
In this paper, we proposed an approach to prioritize activities via machine-learning methods to resolve the activity overflow problem ([28], [32], [35]) in an activity-centric collaboration environment. We proposed and analyzed features used for our ranking algorithm, and we conducted an offline experiment with the data of a small number of high-volume Lotus Activities users. The results of this experiment are promising and indicate that the Ranking SVM model with the selected features performs significantly better than the two baseline recency methods without adding any extra user cost. We also implemented and integrated our activity prioritization system into the user interface of Malibu, an experimental personal productivity tool.
For completeness, we note that the User recency presents a special case. Since the data points in our test are the moments before a user switches to an activity, the last activity the user accessed will be always the first on the list according to user recency and it is never the true activity in our experiment. To fix this, we removed the first activity from the list of User recency in our evaluation process.
522
Retrieval (Seattle, Washington, USA, August 06 - 11, 2006). SIGIR '06. ACM Press, New York, NY, 186-193
There are several future directions to improve the current activity prioritization system. Our current feature set is not complete and additioal features can be added. For example, using extrinsic data in the feature set of the learning algorithm might further improve prediction accuracy. In addition to the Lotus Activities metadata, potential clues about a user’s current priorities could be found in emails, chat messages, or phone calls. Techniques including predicting the activities incoming emails belong to [14] and “speech acts” [6] could be applied to infer priorities of activities. For example, assuming that a user has an activity named “submitting report”, if s/he receives an email and an email classification system identifies the activity “submitting report” as the associated activity, we could argue that this activity has a higher priority because of the potential need of the user to go to this activity now as a consequence of the email. Vice versa, if a user never opens that email, the activity might also have a very low priority.
[6] Carvalho, V. R. and Cohen, W. W. 2005. On the collective classification of email "speech acts". In Proceedings of the 28th Annual international ACM SIGIR Conference on Research and Development in information Retrieval (Salvador, Brazil, August 15 - 19, 2005). SIGIR '05. ACM Press, New York, NY, 345-352. [7] Czerwinski, M., Horvitz, E., & Wilhite, S., “A diary study of task switching and interruptions,” in: Proc ACM CHI 2004. [8] Chang, C. and Lin, C. LIBSVM : a library for support vector machines, 2001. Software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm [9] Hsu, C., Chang C and Lin, C. “A practical guide to support vector classification” http://www.csie.ntu.edu.tw/~cjlin/papers/guide/guide.pdf
Another interesting direction is to include dates such as deadlines and milestones into the prioritization. Lotus Activities allows users to pick arbitrary names for their activities. We could automatically identify dates in names and incorporate them into the model. For example, an activity named “Malibu demo and talk on August 16” could be extremely important and updated frequently on the days before Aug 16 but not after. This kind of knowledge could be used as a strong, discriminating feature in our priority prediction model.
[10] Cho, J. and Garcia-Molina, H. 2000. The Evolution of the Web and Implications for an Incremental Crawler VLDB 2000. [11] Cohen, W. W. Schapire, R. E. Singer, Y Learning to Order Things In Advances in Neural Information Processing System 1998, pp. 451-457 [12] Crammer, K.and Singer, Y. Pranking with ranking. In Proceedings of the conference on Neural Information Processing Systems (NIPS), 2001
8. ACKNOWLEDGMENTS
[13] Dragunov, A. N., Dietterich, T. G., Johnsrude, K., McLaughlin, M., Li, L., and Herlocker, J. L. 2005. TaskTracer: a desktop environment to support multi-tasking knowledge workers. In Proceedings of the 10th international Conference on intelligent User interfaces (San Diego, California, USA, January 10 - 13, 2005). IUI '05. ACM Press, New York, NY, 75-82
We thank the Activities product team for their continuous support, Jianqiang Shen for inspirational research discussions, and all anonymous users of Activities who supported this study with their data.
9. REFERENCES
[1] Agichtein, E., Brill, E., and Dumais, S. Improving web search ranking by incorporating user behavior information. In Proceedings of the 29th Annual international ACM SIGIR Conference on Research and Development in information Retrieval (SIGIR '06), 19-26
[14] Dredze, M., Lau, T., and Kushmerick, N. 2006. Automatically classifying emails into activities. In Proceedings of the 11th international Conference on intelligent User interfaces (Sydney, Australia, January 29 February 01, 2006). IUI '06. ACM Press, New York, NY, 7077
[2] Bellotti, V., Ducheneaut, N., Howard, M., and Smith, I. 2003. Taking email to task: the design and evaluation of a task management centered email tool. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. CHI '03
[15] Geyer, W., Muller, M., Moore, M., Wilcox, E., Cheng, L., Brownholtz, B., Hill, C.R., Millen, D.R., “ActivityExplorer: Activity-Centric Collaboration from Research to Product,” IBM Systems Journal – Special Issue on Business Collaboration, 713-739
[3] Boardman, R., & Sasse, M.A. “Stuff goes into the computer, but it doesn’t come out” A cross-tool study of personal information management. Proc CHI 2004.
[16] Geyer, W., Brownholtz, B., Muller, M., Dugan, C., Wilcox, E., and Millen, D.R. Malibu Personal Productivity Assistant. Accepted Work in Progress Submission, CHI 2007, San Jose, CA, April 28 – May 3, 2007.
[4] Burges, C., Shaked, T., Renshaw, E., Lazier, A., Deeds, M., Hamilton, N., and Hullender, G. 2005. Learning to rank using gradient descent. In Proceedings of the 22nd international Conference on Machine Learning (Bonn, Germany, August 07 - 11, 2005). ICML '05, vol. 119. ACM Press, New York, NY, 89-96.
[17] Google Desktop. http://desktop.google.com/. [18] González, V., and Mark, G. Managing currents of work: Multi-tasking among multiplecollaborations. Proc ECSCW 2005.
[5] Cao, Y., Xu, J., Liu, T., Li, H., Huang, Y., and Hon, H. 2006. Adapting ranking SVM to document retrieval. In Proceedings of the 29th Annual international ACM SIGIR Conference on Research and Development in information
[19] Herbrich, R. Graepel, T. and Obermayer, K. Large margin rank boundaries for ordinal regression. In Advances in Large Margin Classifiers, pp 115--132. MIT Press, Cambridge, MA, 2000.
523
[27] Moran, T.P., Cozzi, A., & Farrell, S.P. (2005), “Unified activity management: supporting people in e-business,” Communications of the ACM, 48(12), pp. 67-70.
[20] Hudson, J. M., Christensen, J., Kellogg, W. A., and Erickson, T. 2002. "I'd be overwhelmed, but it's just one more thing to do": availability and interruption in research management. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: Changing Our World, Changing Ourselves (Minneapolis, Minnesota, USA, April 20 - 25, 2002). CHI '02. ACM Press, New York, NY, 97-104
[28] Muller, M.J. (2004). Activity graphs of the microstructure of complex work. Poster at CSCW 2004. [29] Muller, M.J., Geyer, W., Brownholtz, B., Wilcox, E., and Millen, D.R. One-hundred days in an activity-centric collaboration environment based on shared objects. Proc. CHI 2004.
[21] Joachims, T. Optimizing search engines using clickthrough data. In Proceedings of the Eighth ACM SIGKDD international Conference on Knowledge Discovery and Data Mining (Edmonton, Alberta, Canada, July 23 - 26, 2002). KDD '02. ACM Press, New York, NY, 133-142
[30] Muller, M.J., Geyer, W., Brownholtz, B., Dugan, C., Millen, D.R., and Wilcox, E. (2007). Tag-Based Metonymic Search in an Activity-Centric Aggregation Service. To appear in Proc ECSCW 2007, Limerick, Ireland, September 2007.
[22] Lotus Connections: Social Software for Business. http://www-142.ibm.com/software/swlotus/products/product3.nsf/wdocs/connectionshome, retrieved 4/26/2007.
[31] Quan, D., and Karger, D. R., (2003). 'Haystack: Metadata enabled information management'. Proceedings of UIST 2003.
[23] McCrickard D. S. & Chewar C. M. (2003) Attuning notification design to user goals and attention costs, Communications of ACM, 46 (3), 67-72
[32] Sen, S., Geyer, W., Muller, M.J., Moore, M., Brownholtz, B., Wilcox, E., & Millen, D.R. (2006). FeedMe: A collaborative alert filtering system. Proc. CSCW 2006.
[24] Millen, D. R., Feinberg, J., and Kerr, B. 2006. Dogear: Social bookmarking in the enterprise. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Montréal, Québec, Canada, April 22 - 27, 2006). R. Grinter, T. Rodden, P. Aoki, E. Cutrell, R. Jeffries, and G. Olson, Eds. CHI '06. ACM Press, New York, NY, 111-120
[33] Ringel, M. and Hirschberg, J. 2002. Automated message prioritization: making voicemail retrieval more efficient. In CHI '02 Extended Abstracts on Human Factors in Computing Systems (Minneapolis, Minnesota, USA, April 20 - 25, 2002). CHI '02. ACM Press, New York, NY, 592-593 [34] Vapnik, V. 1998. Statistical Learning Theory. WileyInterscience.
[25] Mitchell, T. M., Wang, S.H., Huang, Y. and Cheyer. A. Extracting knowledge about users' activities from raw workstation contents. In Proc.of AAAI-06, 2006.
[35] Whittaker, S., Sidner, C., “Email Overload: Exploring Personal Information Management of Email,” Proc. CHI’96.
[26] Moore, M., Estrada, M., Finley, T., Muller, M. and Geyer, W. Next generation activity-centric computing. Demo at ACM CSCW 2006.
[36] Xu, J., Cao, Y., Li, H., and Zhao, M. 2005. Ranking definitions with supervised learning methods. In Special interest Tracks and Posters of the 14th international Conference on World Wide Web (Chiba, Japan, May 10 14, 2005). WWW '05. ACM Press, 811-819.
524
