idence of activity: lines of text added to wiki pages, lines of committed code to. Subversion and ..... Acknowledgments We thank the Australasian Apple University Consortium ... in undergraduate computer science: Do things get better? In ACE ...
Narcissus: Group and Individual Models to Support Small Group Work
Kimberley Upton and Judy Kay
School of Information Technologies, University of Sydney, Sydney, NSW 2006, Australia {kupt2870, judy}@it.usyd.edu.au
Long term group work by small teams is a central part of many learning and workplace activities. Widespread group support tools such as wikis, version control systems and issue tracking systems are an invaluable aid for groups. They also have the potential to provide evidence for valuable models of the group activity. This paper describes Narcissus, designed as a new way to improve group-work by exploiting evidence from use of such group-work tools, to create a visual presentation of a group model. The Narcissus models and interfaces were designed to help groups function more eectively. It helps individuals see how well they are contributing to the group. It enables groups to assess contributions relative to plans. And it helps facilitators identify problems. The Narcissus interface supports scrutability and control over its models. We report a four part evaluation of Narcissus : individual level with 23 students; group level by 5 groups; facilitator level with 5 facilitators; and ne grained study with 8 students. Results indicate that all these groups were able to understand and use Narcissus and that they considered it eective in modelling the group activity in useful ways. They particularly valued the support for scrutability. Key contributions of this work are the creation of a scrutable and user controlled group model to support group work and to provide a new form of navigation interface for a complex groupware site. Abstract.
1
Introduction
Working eectively in groups is hard, particularly for projects that span months. Yet long term group activity is central to many workplace projects [10]. It also has an important role in education on two levels. One of these is the recognition that development of students' skills in group work is important. Secondly, there are learning bene ts from working in groups [2,6,11]. It is natural then that we have seen the emergence of many online tools designed to help groups collaborate more eectively over long periods. For example, the growth in wikis has enabled even large groups to work over long periods. Indeed, a range of online tools is increasingly being used to support collaboration. These tools present a promising potential for modelling individuals and groups and then exploiting these models to support group activity. Over long periods
of use, groups typically generate large amounts of content and huge traces of electronic activity and interaction. However, this overwhelming amount of detail is of limited value because it is hard for group members to see the big picture. This means that individual team members cannot readily see how well they are contributing. At the group level, it is di cult to use these electronic traces to help the group determine if it is functioning eectively. In a setting where there are group facilitators, the situation is even more challenging: they need to get a big picture understanding and yet, unlike the individuals in the group, they do not have intimate knowledge of the actual details of any of the work done. This means that they cannot make use of the electronic traces to see the progress of the group. Our approach has been to explore ways to analyse the electronic traces to create group models that can operate as
mirrors
which enable the individuals
and teams to see their progress. Mirroring is the technique of presenting team members with a visual summary of their own activities in order to support re ective learning. We take inspiration from other visualisations that create social translucence for groups [1,4,13]. However, that work explored supporting large groups or those using chat for social reasons. This is very dierent from our prime concern, long term work by small groups, as is typical in workplaces. The context of our research is a senior level capstone software development project which runs over a semester. Teams of around ve students use
Trac
[3],
an open-source web-based project management system. It has three key parts, each tightly integrated via hyperlinks. The documentation,
tem
Subversion
wiki
supports communication and
is for source code management, and the
ticket sys-
supports task management and reporting software bugs. Assessment takes
account of both the quality of the software produced and the group management. Students are also required to re ect on their group processes. Drury et al. [2] discuss the importance of re ection in overcoming lack of knowledge and experience in group work. This points to the value of mirroring tools to support re ection, both by individuals and as a group activity. The next section reviews related work, explaining how it has informed this work and how
cissus
2
Narcissus
diers from it. We then present the user view of
Nar-
followed by our evaluation experiments, their results and conclusions.
Related Work
In some of the key work on mirroring, there has been promising progress. Erickson's social translucence feedback tool [4] facilitated group interaction. For example, individuals who tended to dominate chat sessions became aware of this and modi ed their behaviour. This is precisely the goal of our mirroring approaches. In work somewhat closer to our own, the CodeSaw [5] temporal visualisation of collaborative software development was valuable for observing trends in source code contributions and a limited amount of communication. Also close to our work, with a focus on students in a software project course, Soller et al. [12] made use of online chat tools to support group work. The in-
terface was specially modi ed, making it feasible to use Speech Act Theory to analyse the nature of group interaction. However, no work has involved the multiple media of our context, based on a state-of-the-art collaboration tool as we wanted to use. Nor did we wish to restrict students to arti cial interfaces with special sentence starters or similar. So, we needed to take a dierent approach.
Some of our own foundation work resulted in a set of high level overview visualisations [7,8], one of which was called the
Wattle Tree, a visualisation rep-
resenting the activity of group members over time. Kay et al. [7] conducted a theoretical evaluation of this approach in terms of the
Big Five
theory of small
group work [10]. We have built upon that work, taking a similar overall approach to building the group model, in making use of very simple measures as the evidence of activity: lines of text added to wiki pages, lines of committed code to
Subversion
and counts of ticket activities. While these measures are clearly very
simple, by making them a very rough measure of the actual contributions made by each group member, we can readily explain the way that the model was built. It also facilitates the design of interfaces for users to control the way the group model is created.
Based on our previous experience, we formulated several new goals for supporting groups by building group models of activity on a state-of-the-art collaboration tool. One key new goal for
Narcissus
was to be readily extensible to
additional media. Previous work cited above dealt with a single medium, with the exception of our Wattle tree which could handle the three basic media in Trac but would not readily extend to other new media. A second core goal was that the new visualisation of the group model should be scrutable, meaning that it enables group members to see the precise evidence that contributed to each part of the visualised group model. Beyond this, we wanted to provide user control, such that the presentation of the model in the visualisation can be controlled by the user. This is important to take account of the variability in levels of activity by dierent groups and individuals within them as well as the potentially diverse reasons that group members may have for studying the model. Another key new goal for
Narcissus
was to make it serve as a form of navigation of the
vast collections of content within the group-work site. Essentially the visualised group model provides a new way for an individual to determine which parts of the group activity are important to them. Once they determine this, we wanted them to be able see the activity that contributed to any part of the visualised model. This goes beyond previous work described above but it is critical if the high level visualisation is to be easily used to scrutinise the Trac site to see why certain behaviours are observed. For example, if one person suddenly appears to be very active on the wiki, it is important to be able to easily see just what they did. Similarly, if an individual has very modest contributions, they may still be very valuable and may represent a large amount of work: hence it is important to have direct access to the actual contributions in order to see this.
3
Narcissus
Figure 1 illustrates the structure of the
Narcissus group model displays and their
associated support for scrutinising the model. It is for an actual group, but all names are anonymised. The very top shows the group name, in this case, it is Group Z. Next are hyperlinks to the three views: group, project, and ticket. In each of these, the left side of the page is the main visualisation of the group's model. All three views have a time-line down the left hand side, with the most recent date at the top. The
information panel
on the right provides a legend and
additional details and navigation as discussed below.
3.1
Group View
The
Group View is the main display of the group model and it is the default view.
It has a vertical column for each member of the group. In this gure, the models for 5 team members are displayed. The user can customise the display to include just those team members they wish. This is useful for ensuring only interesting parts of the group model appear. For example, students would typically exclude models for tutors and clients. The activities of each group member are mapped along the column. The media correspond to dierent colours and positions; leftmost, in purple is the wiki, next in blue is Subversion, and the third is tickets in green. The legend at the top of the information panel explains this. The level of activity on a resource for a given day is given a discrete score from one to four, with four the highest. The score determines whether a square is pale or brightly coloured. Grey indicates there was no activity on that resource on that day, by that group member. For example, Figure 1 shows that member4 (second column from the right) on the most recent day (the top row) had moderate activity on the wiki and using tickets, and no activity committing data to Subversion. The middle column for this member is entirely grey, except on the bottom two rows, indicating that they have had no Subversion activity for 18 of the 20 days shown. At the bottom of the columns is an
aggregate summary
of activity for each
group member. The coloured bars indicate the total level of activity for each group member. The grey bars indicate the average level for the group. These bars are semi-transparent, so activity above average appears as a bright coloured tip, and below average has a grey tip. For example, Figure 1 shows that member2 was more active on the wiki and Subversion than the group average (indicated because their coloured bar is longer than the group one for the group), but less active on tickets than the group average. The aggregate summaries enable a user to compare the overall activity of each group member. Figure 1 shows that member1 was the most active on Subversion, and member4 the least active.
3.2
The
Pro ject View
project view
displays a single vertical column summarising the combined
activity of all group members. As with the group view described above, each
The group view showing 20 days of activity of a group. Narcissus is structured with the visualisation on the left and an information panel on the right. Links for selecting one of three views are provided along the top. Fig. 1.
day is mapped vertically from bottom to top, each resource is indicated by a dierent colour (explained in the legend), and the level of activity is indicated by the brightness of the colour. Grey indicates that there was no activity on that resource on that day, by any member of the group. Unlike the group view, the project view does not provide an aggregate summary on the bottom. Since the group as a whole is considered, a total does not provide any more information than an average. The average level of activity for each resource is depicted by the width of each resource column. The average is relative to the span of the project, not to the number of group members. This is important for two reasons. First, activity must be maintained throughout the project. Secondly, the average is less sensitive to extreme group members. If one group member is under-performing, or conversely, dominating the group, their activity will not skew the average as it can in the group view.
3.3
Ticket View
Tickets are key for managing the group, providing rich and important information associated with tasks. The
ticket view
displays a history of ticket activity
for each group member. The lifespan of each ticket is plotted according to the group member who is responsible for the task associated with the ticket. The ticket view helps show the distribution of tasks amongst group members, which group members allocate tasks or maintain tickets, and point to other important behaviours, including evidence of collaboration.
3.4
Scrutability and user control
The term scrutability refers to the design of a system so that a user can scrutinise it to determine why it behaves in the way it does. In the context of the
Narcissus
group model visualisation, users can see a model of each team member's activity. If the user is able to understand the underlying process that generated this model, then the model is scrutable. Such scrutability is important for ensuring that users can appreciate just what the model means. But it is also a foundation for user control. In the case of
Narcissus, we consider that the user should be able
to control the ways that activity data is interpreted to build the group model. The activities are measured dierently for each resource. Contributions to the wiki and Subversion repository are measured according to the number of added lines. Tickets are scored according to the type of activity, such as creating, accepting, and resolving tickets, as well as adding comments at dierent stages of the task. This establishes another motivation for making the model scrutable: the underlying measures of the visualisations are very simple and it is important that this is clear to the users. As discussed earlier in this section,
Narcissus models activity on a 4 level Narcissus models scrutable, we
discrete scoring system. In the spirit of making
provide explanations of the way activity evidence is interpreted, such as the metrics outlined above. In the case of Figure 1, the scoring is explained in the right-hand information panel. This shows that the model has 4 levels of activity, corresponding to up to 50, 150, 300 or above 300 lines added on this medium. The user can control this, altering these thresholds which serve to interpret the evidence available for the model. This means that the user can decide on the cut points in terms of levels that they wanted modelled. In dierent situations and at dierent times, a user may want to set quite dierent levels.
Narcissus
provides another form of scrutability by linking each part of the
displayed model directly to the actual evidence used to infer the activity shown in the model. All three views provide this form of scrutability. The user can click on any component of the visualised model to see the evidence associated with it. For example, the hand cursor in Figure 1 shows where the user has clicked on a Subversion square for member2. This causes the display of the particular details shown in the right hand information pane, showing details of the ve change-sets committed by member2 on that day. The user can click on blue link with the
number 77 to display the actual change-set, a display within the Trac system, showing the additions and deletions to the source code in that commit action. There are corresponding links to the details for the other media. In all cases, the activities listed in the details sections provide a blue hyperlink to the actual activity on Trac. For the case of wiki activity, this is a hyperlink to the wiki page. Each ticket activity has a hyperlink to the ticket page, and Subversion activity has a hyperlink to the change-set, as just discussed. The links to Trac have three roles. One of these is scrutability. They also enable users to review the actual activities listed on the information panel, and make a qualitative judgement about the contributions that have been made. The third role is as a new means of navigation around the Trac site. Essentially, the
Narcissus
models show what
the group has been doing on each of the media over the duration of the project and the user can explore interesting or potentially problematic aspects of the group's work by scrutinising the site, via the links from the
3.5
Narcissus
models.
Implementation
The implementation of
Narcissus
involves extracting the desired data from Trac
then generating a group model which is presented in an active visualisation interface. The prototype was tested on Mozilla Firefox 2.0.0.9 on Mac OS X.
Narcissus
was written in Python to leverage the plugin support from Trac, and
Quartz, a 2D and PDF drawing library for Mac OS X. It is available as a plugin for the Trac site, and accessible from the Trac menu bar once installed. This means that the visualisation is available to the user in real time.
4
Evaluation
The key evaluation goals were to assess
Narcissus ' eectiveness in assisting in-
dividual students and groups to re ect, to see how the group is performing and then to nd on relevant details via the interactive facilities. Another key goal was to assess how well
Narcissus
supported facilitators, especially in identifying
indicators of likely problems in group and individual performance.
4.1
Experiments
Two groups of participants were recruited. Students came from the 2007 second semester capstone software project course and they participated near the end of the course. The facilitators were unfamiliar with the projects from that semester but had played that role at other times. This means they are familiar with the challenges of facilitating groups but they had to rely solely on the
Narcissus
model visualisations to gain insights into the performance of the students; they had no other knowledge of the groups. This is a challenging but useful test of
Narcissus
because it is important that facilitators be readily able to identify
potential problems with minimal time investment.
The approach was to observe users as they learn and explore the interface, and allow the participants to provide a qualitative assessment. The experiments began with use of
Narcissus, aided by an interactive tutorial. Then participants
completed a questionnaire. The questionnaire included questions that required the participants to analyse the visualised group model. The experiments used real student data from the current semester. Participants were provided with approximately three months of data, from the start of the project until the day of the experiment. The decision to use real data was to make the experiment as authentic as possible, and engage the participants. Students were provided with the data from their own project, which included changes to the wiki, changes to the ticketing system, and change-sets committed to the repository. Facilitators were provided with the data from all ve project groups from this semester.
Essentially, there were four parts in the evaluation: ne grained individual study with 8 students; group level by 5 groups; individual level with 23 students; and facilitator level with 5 facilitators.
Fine grained individual study In this experiment, participants used
cissus
Nar-
to observe recent project activity and report on their own behaviour in
relation to the group. From each of the ve project groups 1-2 students participated in this experiment, 8 in all.
Group and individual level study This experiment asked participants to
work collectively, using
Narcissus
to re ect on aspects of their group processes.
All ve of the project groups, with 23 individuals, participated. Their questionnaire had three Likert items [9] about the suitability of Participants also rated
ease of use
and
eectiveness
Narcissus
for real use.
of the interface on a scale
of 1 to 6, 6 being the highest rating.
Facilitator study In this experiment, participants used
Narcissus
to identify
and give examples of concerning groups or students. Five experienced facilitators participated.
4.2
Results
Of the three experiment categories, the group experiment most closely resembled real-world use. Students were able to work as a group to re ect on their group processes, and these re ections could then in turn be used in their nal report. Although the experiment itself was not related to their assessment in the unit of study, their participation rewarded them with additional evidence for re ection which they could use for assessment. We rst report the results of the ve groups. All worked through the tutorial eectively and then make use of the visualisations to answer the questionnaire. Figure 2 shows the ratings of ease of use and eectiveness (described to
Narcissus effectiveness 6
5
5
4
4 Rating
Rating
Narcissus ease of use 6
3
3
2
2
1
1
0
Previous vis effectiveness
0 G1
G2
G3
G4
G5
Avg
G1
G2
G3
Group
(a) Ease of use Fig. 2.
G4
G5
Avg
Group
(b) Eectiveness
Group ratings of the ease of use and eectiveness of
Narcissus.
S1
S2
S3
S4
S5
S6
S7
S8
Group view re ects activity
Y
Y
Y
Y
Y
N
Y
Y
Project view re ects activity
Y
N
Y
Y
Y
N
Y
Y
Project view useful for recent activity Y
Y
Y
Y
Y
N
Y
Y
Overall preference
A
G
P
A
G
G
G
Table 1.
S8.
G
Key: G = group view; P = project view; A = all three views Summary of task results from the student questionnaire, for students S1 ..
participants as `the ability for groups to use the visualisations to identify their strengths; and areas of potential improvement as a group'). All ve groups found
Narcissus
easy to use. All but one gave
Narcissus
an ease-of-use rating of 5, one
giving a rating of 4, resulting in an average rating of 4.8. Furthermore, all groups gave
Narcissus
a rating of 4 or 5 for eectiveness.
In the other two experiment categories, all participants were able to successfully complete the tasks included in the questionnaire. Each student identi ed his/her personal level of activity using the group view, the overall level of group activity using the project view, and a range of behaviour regarding ticket use by examining the ticket view. Table 1 details the results of the tasks in the student questionnaire, showing that students agreed that the group and project views accurately re ected activity, the project view was useful for viewing recent activity, and the preferred view overall was the group view. The facilitators used
Narcissus
to successfully identify both groups and stu-
dents of concern. They reported that the group view, like that in Figure 1, was the most useful for completing this task. Interestingly, all ve facilitators were observed to scroll straight to the aggregate summary when comparing the
Group Re ections Group 1 Periods of inactivity, low overall activity.
Facilitators' Observations
Co-ordinator's Assessment
Concerned about low overall activity.
Weaker group that improved, but made poor use of Trac.
Group 2 Well functioning group. Concerned that one group member is dominating. Group 3 Overall consistent activity, well distributed among group members.
No concerns.
Very strong group, with an excessively active leader. Strong group, good split of work.
Group 4 Well functioning group. Concerned about one or two of the group members.
Strong group with some under-performing group members.
Group 5 Well functioning group, Concerned about low no improvement overall activity. needed.
Consistently weak group and modest individual work.
Comparison of observations made by groups and facilitators in the user study, as well as those my by the co-ordinators of the unit of study. Table 2.
group view across the dierent groups. Three participants reported that the project view was useful in addition to the group view. One participant said of the project view, [the] relative performance of groups is clear. Table 2 shows the comments made by the groups, and facilitators as they reviewed the visualisations for each group. The rightmost column shows the independent assessment of each group made by the course co-ordinator, after grading the full semester's material and devoting considerable time to reviewing the full work of each group, as part of the assessment process. It is striking that the facilitators were able to quickly draw conclusions about the groups and these correspond closely to the course co-ordinator's knowledge. For the student groups, the re ections seemed to be excessively positive which may well be due to their unwillingness to share admissions of problems. Students and facilitators were both asked what they thought of the scrutability of
Narcissus. There was an overwhelmingly positive response to this feature;
all participants agreed that the active models and links down to the full details were useful. One of the students liked the links to Trac, remarking that it gave evidence for the
Narcissus model's representation of activity. Three students and
two facilitators noted that they valued the links to Trac. One facilitator found the link to Trac extremely useful for examining the quality of contributions, and another facilitator found it useful for identifying, then scrutinising abnormally large levels of activity, particularly source code committed to the repository.
Students and facilitators were also asked what they thought of the scrutable score section, and whether they found it useful. All but one student agreed that the detailing of the score calculation was useful, although there were caveats. One student felt it was only needed at the outset, to con rm fairness. Another student found it useful but only as a rough guide. Another student noted that counting added lines was a weak measure, but valued the explanation. The facilitators responded very positively to the scrutable scoring model. One facilitator thought it was useful to know the nature of each activity, and why certain activities were scored highly. Another facilitator was interested in understanding busy periods, and mentioned that knowing how the system works oers a sense of security. One participant thought the scoring system, coupled with the drill down interactivity, was crucial: Scrutability [is] very important for non-repudiability while a [visualisation] may indicate something is wrong, if it is not scrutable, discovering how to amend the problem is di cult. Worse, students may blame the visualisation for misrepresenting their work. The evaluation showed that while
Narcissus is useful for students to identify their
strengths and potential areas of improvement, facilitators identi ed many concerning behaviours that students did not. Furthermore, the inverse was true when facilitators identi ed concerning behaviour that could be explained by extenuating circumstances. In either case, this shows important promise as it means that
Narcissus provides facilitators with indicators of concerning behaviour. They can
then readily drill down to see the actual details. If it turns out that there is a problem, the facilitator is able to intervene early enough to remedy the situation.
5
Conclusion
Narcissus
was designed to support long term group-work based on the Trac col-
laborative software platform. As Trac has the same collaborative elements that are widely used for long term collaboration, our approach has broad applicability. Essentially,
Narcissus
makes it possible for group members and facilitators
to gain understanding of the way the group has been operating. It does this by presenting a group model of long term group-work activity. Our evaluations demonstrate
Narcissus
was usable, supporting re ection on
the way that each team member was contributing to the group. The students liked the interface, particularly for navigation. However, they understated group problems. By contrast, facilitators could quickly see how the groups were performing. This is a valuable foundation for identifying problems and helping groups address them early.
Narcissus
uses simple measures of activity. Our evaluations indicate that,
although participants recognised limitations of these, the active interface to the group model meant that they could scrutinise the evidence used to build the model. From this, they could apply much more informed and subtle reasoning to determine whether unusual patterns of activity indicated problems.
Project groups and other team-based work have a very large role in the workplace and in education. Our evaluations were in an educational context, but the groups were doing authentic large-scale team programming tasks. Our evaluations indicate that
Narcissus
oers a usable and useful tool for student
re ection and for facilitators to identify problems. Importantly, it demonstrates that a scrutable group model visualisation can provides a new mechanism for navigating a large, sophisticated and complex groupware site.
Acknowledgments We thank the Australasian Apple University Consortium
for its support of this project with their generous Honours Scholarship.
References 1. Judith Donath. A semantic approach to visualizing online conversations. Communications of the ACM, 45(4):4549, 2002. 2. Helen Drury, Judy Kay, and Warren Losberg. Student satisfaction with groupwork in undergraduate computer science: Do things get better? In ACE '03: Proceedings of the fth Australasian conference on Computing education, pages 7785, Darlinghurst, Australia, Australia, 2003. Australian Computer Society, Inc. 3. Edgewall Software. The Trac Project. Website, 2008. http://trac.edgewall.org/ veri ed 2008-02-01. 4. Thomas Erickson and Wendy A. Kellogg. Social translucence: an approach to designing systems that support social processes. ACM Transactions on ComputerHuman Interaction, 7(1):5983, 2000. 5. Eric Gilbert and Karrie Karahalios. CodeSaw: A social visualization of distributed software development. In Maria Cecília Calani Baranauskas, Philippe A. Palanque, Julio Abascal, and Simone Diniz Junqueira Barbosa, editors, INTERACT (2), volume 4663 of Lecture Notes in Computer Science, pages 303316. Springer, 2007. 6. D.W. Johnson and R.T. Johnson. Positive Interdependence: Key to Eective Cooperation. Interaction in cooperative groups: The theoretical anatomy of group learning, page 174, 1992. 7. Judy Kay, Nicolas Maisonneuve, Kalina Yacef, and Peter Reimann. The big ve and visualisations of team work activity. Proceedings of Intelligent Tutoring Systems, Taiwan, 2006. 8. Judy Kay, Nicolas Maisonneuve, Kalina Yacef, and Peter Reimann. Wattle tree: What'll it tell us? Technical Report 582, The University of Sydney, January 2006. 9. Rensis Likert. A technique for the measurement of attitudes. Archives of Psychology, 140:55, 1932. 10. Eduardo Salas, Dana E. Sims, and C. Shawn Burke. Is there a "Big Five" in Teamwork? Small Group Research, 36(5):555599, 2005. 11. Robert E Slavin. Cooperative learning: theory, research and practice. Englewood Clis, NJ: Prentice-Hall, 1990. 12. Amy L. Soller. Supporting social interaction in an intelligent collaborative learning system. International Journal of Arti cial Intelligence in Education, 12(1):4062, 2001. 13. Rebecca Xiong and Judith Donath. Peoplegarden: Creating data portraits for users. In UIST '99: Proceedings of the 12th annual ACM symposium on User interface software and technology, pages 3744, New York, NY, USA, 1999. ACM Press.
