Dynamic Meeting Annotation and Indexing Rick Kazman William Hunt Marilyn Mantei Department of Computer Science Department of Computer Science Department of Computer Science University of Waterloo University of Toronto University of Toronto
[email protected] [email protected] [email protected] Abstract: This paper presents research on a system which automatically indexes meetings where distributed participants are linked by voice, video, and shared computer applications. We approach the indexing of such meetings in four ways: 1) indexing by what people say (through speech recognition and lexical chaining), 2) indexing by meeting agendas, 3) indexing by people’s temporal interaction paradigms, and 4) indexing by the participants’ interactions with the application. We discuss each of the interaction paradigms we propose, the indexing techniques we use, and a prototype implementation of the system, called Jabber. Special attention is given to the user interface design issues of this dynamic annotation tool since meetings demand the full cognitive attention of participants making it difficult to introduce computer support tools. Keywords: CSCW, video conferencing, video annotation, distributed meeting support, information retrieval, automatic indexing
1 Introduction As more market and research emphasis is being placed on multimedia, video conferencing, computer-supported cooperative work, and telepresence, it becomes apparent that methods are needed for organizing the enormous amounts of data which such shared work will produce. Uncompressed real-time video alone can produce 150 kilobytes to 1.2 megabytes of data per second [5]. This data is currently stored as a linear stream, or, in the case of a multipoint video conference, as a set of parallel streams. Such enormous bodies of information, although valuable, are intractably locked in their inherently linear video format. Occasionally, videotaped meetings are indexed so that they may be queried like a database or browsed like an encyclopedia ([9], [17]), but only as a result of a large amount of manual labor. There exists little research into the management and automatic indexing of video repositories of meetings. In particular, we need ways to: 1.
organize and index meeting records so that people can access the stored information in a semantically meaningful, efficient way,
2.
provide user interfaces which make the annotation and retrieval of this information simple,
natural, and effective. Although this paper is primarily addressed at the indexation of distributed meetings, there are many other types of multiperson interactions which may be profitably indexed— lectures, requirements elicitation, joint work sessions, etc. For example, in requirements elicitation, a significant amount of effort is spent tracing requirements and collecting them into chains of request-rationale-requirement. Thus the goal of this work is to design and prototype tools for transforming distributed meetings involving some combination of video, audio, and application execution records into a structured repository. These tools are currently implemented as stand-alone prototypes, but are planned to be integrated with Intel’s ProShare video conferencing product for more thorough user testing. The prototype discussed in this paper is the Jabber system, a system which explores the human-computer interaction issues involved with annotating and, more importantly, searching through large meeting repositories.
2 The Problem Meetings both use extensive organizational resources and contain an enormous amount of corporate knowledge, culture and decisions. Myers [14] notes, for example, that large software development projects spend fifty percent of their budget on the interactive specification and design process. Because meetings take participants away from their normal work environment, often with significant travel and time costs incurred, organizations are becoming more and more interested in supporting long distance meetings through technology. Commercial products such as ProShare from Intel, Aspects from Group Technologies and Vis-a-Vis from Worldlinx are already available in the marketplace. Currently meetings are either not recorded or recorded in an abstracted, interpreted form (e.g., via an agenda or a set of minutes). Given that meetings consume so much personnel time and contain such important corporate information, more detailed recording mechanisms are desirable. This is especially true since our research indicates that post meeting recall and shared understanding among participants on what transpired at a meeting is rather poor [8]. Current distributed meeting support products allow corporations to maintain
detailed video and audio recordings of meetings, but not without the creation of new problems—that of efficiently accessing these records at a later time.
3 Background 3.1 Problems in Meetings Meetings are complex dynamic interactions creating much cognitive load on the individual participant partly because of the sheer volume of information exchanged and partly because of the social behaviors of other participants. In a report of the Groupware Project at Microelectronics and Computer Technology Corporation (MCC) [6], meeting participants expressed problems with: many people talking at once, people failing to listen to one another, people misunderstanding what was said, people disagreeing because of terminology differences and people failing to record important rationale or decisions. Minneman and Harrison [12] also report on meeting participants forgetting early discussions and decisions which slowed the meeting's progress. Although the social misbehavior in meetings is unlikely to be changed with computer support, better records of the meeting are likely to aid both the meeting process and meeting recall after the meeting. Note taking as a form of meeting record keeping unfortunately has its problems. Whittaker [19] and Minneman [12] found that meeting recorders did not have enough time to take adequate notes, were restricted from participating in the meeting by their note-taking task, and wrote notes that were too terse and missed important points of the meeting. Minneman also noted that reviews of meeting videotapes improved this situation but only at the cost of time-consuming serial access to the records, a process which impeded the flow of ideas in the design meetings. Chalfonte, Fish and Kraut [3] show that speech based notes are possible and much easier to make than written ones, but with the same serial access problem as video.
3.2 Low Post-Meeting Agreement One major impetus for finding ways to keep better records of meetings is the existing research showing poor postmeeting agreement between participants on what took place during the meeting. In a study investigating the efficacy of meeting support via shared writing tools, Olson Carte and Storrsten [15] found that the groups using shared tools generated a better meeting report and better design ideas. An extensive investigation of what groupware supported and unsupported groups took away from the meeting, however, showed that memory for what was accomplished was problematic in both cases. Less than 30% of the alternatives discussed and less than 3% of the design criteria agreed upon
were expressed in the final meeting reports. How much agreement was there for the participants on meeting content? Given the mandate to relay these meeting notes to a missing team member, large amounts of what took place was either forgotten or intentionally omitted. In addition, “new” ideas were added that had never been discussed, 9% for the unsupported groups and 15% for the supported groups. Although it was possible for all participants to take notes in the meeting not supported by the shared editor, only one unsupported group actually passed the final report around for all participants to view. In a second study, McCarthy, Miles and Monk [11] assessed how much common ground (mutual knowledge, beliefs, and assumptions) two people established while doing a textbased task. Each participant used a personal workstation located in different rooms. A private composing window was available to each participant in addition to a report space and a shared communication window. The report space was private for half of the pairs and public for the other half. The public report space helped participants establish a common ground for the design rationale portion of the task but not for the solutions that were proposed. Approximately 1/3 of the solutions and over 1/2 of the design rationales recalled in all conditions had no overlap. A closer examination of the rationales that were recalled showed that 1/4 of the rationales from the private report space condition and almost 1/2 from the public report space condition were fictitious, i.e., they were generated by the participants after the experiment. In a third study by Hunt and Mantei [8], a meeting of social scientists revising a questionnaire was observed. A postmeeting questionnaire revealed that only 60% of the decisions were remembered and only 36% of the design rationales were recalled. The recall test used for assessing the decisions was a simple recognition recall test in which participants were asked to indicate whether each question on the questionnaire had been deleted, kept or changed. During the first half of the meeting, a scribe captured the decisions in an issue-based meeting support tool. 100% of the decisions were captured by this method but the scribe captured only one rationale from many that were cited for each decision. In summary, substantial numbers of decisions and rationale for decisions were lost in experimental situations. This suggests that there is room for better support tools.
3.3 Approaches to Meeting Support Several meeting support systems address the aforementioned meeting problems: “Where Were We” is a prototype system that exploits the
capabilities of digital video to allow groups to include recordings of very recent events into real-time activities. The goals of the research are to see how playback-whilerecording and random access capabilities might substantially alter the activity of group work, to discover what difficulties arise in using emerging digital multimedia hardware and software facilities for this kind of application, and to further explore how to base novel systems design on necessarily partial and fragmented observations of actual work activity. Other video manipulation tools that can be used in meetings are ([7], [10]). Being able to see the structure of conversations or the structure of someone else's thinking [2] or the structure of a group's ideas (e.g., [4], [16]) can enhance understanding. The representations used by researchers in this area are generally either indented text or graphs with labeled nodes and links. The designers of one tool, the Designer's Notepad, have taken particular care in involving the users themselves [18]. This is to redress a major problem of structured tools, namely acceptability.
4 Heterogeneous Indexing Techniques By capturing the data involved in such meetings—what people say (audio), what people see (video), and what people do (computer applications), we create the possibility of using meetings as archives of information. This is already beginning to occur, for example, in software requirements elicitation. A significant problem in data capture is not one of hardware or software capabilities, but of the ergonomics of the user interface design: making a system that people will actually be willing to learn and use. This problem will only be solved by an analysis of interaction styles/metaphors, and the subsequent prototyping of candidate designs. Data capture is only one part of the process in creating repositories out of meetings. The data also needs to be appropriately structured and indexed so that it can be later queried and retrieved. This indexing can take many forms. The ones which are being explored in this project are:
• Indexing by actual content: meetings can also be indexed by what was said or done, rather than what was planned. A speech recognition system can be applied to the stored audio track in order to create text-based records of the meeting. The speech recognition system need not have perfect recognition because we are only interested in identifying topics, not each individual word. Thus, clusters of related words can be identified and used as indexes back into the original audio/video streams.
• Indexing by intended content: meetings can be indexed according to an explicit agenda which accompanies the
meeting. This agenda is used, in real time, to annotate the data streams to indicate the current topic or other aspects of the structure of the meeting. This puts a heavy real-time demand upon a users of the system— they would have to be constantly attentive to the data streams and discussion topics. Consequently the design of such an annotation system must be carefully studied, so that it provides a flexible and intuitive user interface.
• Indexing by temporal structure: meetings (or, in fact, any multi-agent interaction) can be indexed by their structure in terms of human interactions over time. Various prototypical forms of interaction—long uninterrupted oration, lively discussion, argument, brainstorming, question-and-answer—have different temporal characteristics with respect to the various data streams. We can view each of these forms as interaction idioms. These idioms can be used to provide a unique retrospective view of a meeting and a way of indexing the contents of a meeting.
• Indexing by application record: a log of a computer application’s activity can be kept and used as an index back into the audio/video streams. The events which will be logged will be application specific, but will typically be application object creations, deletions, and modifications as well as other common operations such as changing focus, grouping, and undo. The first three of these indexing techniques will be discussed in the following sections.
4.1 Indexing by Actual Content In order to index the actual content of captured meeting information we must perform speech recognition on the audio portions of the data streams. Each audio stream (one per participant) will be recognized and transcribed. For this purpose, we will use a commercially available speech recognition system.1 Currently we are manually transcribing the audio portion of recorded meetings. Simply recording the words which meeting participants say does not create a meaningful index. What we want is an index of topics, not spoken words. In order to accomplish this, we will use a technique known as lexical chaining that recovers semantic clusters of words from a discourse [13]. We view these semantic clusters as meeting topics or themes. Lexical chaining attempts to group words into sets, 1. We are evaluating both BBN’s HARK system and SRI’s DECIPHER. These systems support speaker independence, have relatively high accuracy and work with continuous speech in real time. In the usage scenario which we assume, each user will be located at her or his own workstation, making it trivial to discriminate and identify them.
using lexical relations between the words as a guide. For example in the following sentences: “Mary ate a peach. She likes fruit.” the superordinate relation (between peach and fruit) holds. In the sentences: “Mary likes green apples. She does not like red ones.” a systematic collocation relation (members of a set) holds. This process is not entirely trivial however. For example, in the sentences: “Mary spent 3 hours in the garden yesterday. She was digging potatoes.”, the relation is nonsystematic collocation. We will be able to extract a large number of these relationships from WordNet, a 90,000 word on-line thesaurus [1] which links words through a large number of semantic relationships. WordNet links synonyms, and antonyms, as any thesaurus would, but also links words according to a richer and more sophisticated set of relations such as: part-whole, sub-type, supertype, entailment, causality. We have already hand-simulated a lexical chaining algorithm on transcriptions of meetings and, in an informal user study, compared the results of this algorithm to the topics which human indexers choose when annotating those same transcripts. The results were encouraging, suggesting that a relatively simple lexical chaining algorithm can capture the vast majority of topics that humans choose. More work is needed to see if this technique scales up to larger samples of text. For example, we need to know how to parameterize the lexical chaining algorithm so that an appropriate number of themes is generated. The meaning of “appropriate” in this context can only be empirically determined—that is, the appropriate themes are the set of themes that the meeting participants found salient. We are in the midst of just such user testing.
4.2 Indexing by Intended Content The intended content of a meeting is expressed by its agenda. The agenda management portion of the Jabber system is concerned with usability issues in real-time annotation of streams of meeting information. Typically, user interfaces do a satisfactory job of handling 2-D interaction. Interaction in 3 spatial dimensions is often problematic. With Jabber we are investigating usability issues raised by interacting with a 3-D environment consisting of 2 spatial dimensions + time. The areas which we are addressing include:
• agenda-management (adding/deleting items, annotating, restructuring);
• associating agenda items with one or more streams in real-time;
• associating agenda items with past events (for often one may not realize that a meeting topic has switched until after it has occurred);
• automatically determining potential agenda items to
associate with spoken text. This will be done by attaching a set of user-selected, system-suggested keywords to an agenda item and then associating these keywords with the spoken text in real time. The final point sounds complex, but is achieved through interaction with the meeting participant. Using the lexical chaining techniques described in 4.1, we generate potential topic words from the written agenda and present these to the user. The user then culls the set of related words picking those that appropriately characterize the agenda items. These words are then compared to the real time lexical chains produced by Jabber from the spoken streams. Matches are used to tie the audio record to the agenda. This culling can be done prior to the meeting freeing the meeting participant from any need to explicitly time stamp agenda processes or indicate returns to prior agenda items.
4.3 Indexing by Temporal Structure Meetings are more than discussion topics and agenda items. They contain salient patterns of interpersonal relations— what we have called interaction idioms. These idioms can be identified by examining the patterns of interaction among the meeting participants over time. The identification of these idioms presents an intriguing new view on a meeting, allowing indexing in terms of how people interacted, as well as what they said. For example, Figure 1 gives examples of three different conversational idioms with which we are all undoubtedly familiar. (1) conversation Bill Phil (2) argument Bill Bob (3) monologue Bill Tom Figure 1: Examples of Interaction Idioms In our preliminary research into interaction idioms, we have discovered another important reason for analyzing multiagent interactions according to their temporal structure: changes in the temporal structure of a meeting always indicate decision points (even though these are often not recognized as such by the participants). These decisions may be minor ones, such as the decision to change the topic, but any indexing technique which can identify such decision points is potentially of enormous power in indexing meeting tran-
scripts, particularly when used synergistically with the other techniques that we are proposing.
4.4 Querying Meeting Repositories A meeting which has been recorded can be viewed as an arbitrary number of parallel data streams: the original video, audio and application records. One objective of the Jabber system is to provide a variety of indices which refer to one or more of those parallel streams. These indices could then be used singly, or in combination to query, browse, or summarize the data. The various forms of indexing the data involved in a meeting suggest different ways of retrieving the data. Indexing by intended content allows queries of the form: “take me to where we discussed agenda item 2 from the meeting of May 3rd”. Indexing by temporal structure allows queries of the form: “take me to our second long argument in last week’s meeting” or “show me all the decisions we made”. Indexing by actual content allows queries of the form: “show me all the places where we talked about baseball in last week’s meeting”, or “take me to where we deleted figure 3 from the paper”. The browsing and querying capabilities of Jabber will be discussed next.
5 Implementing Jabber 5.1 System Architecture The system architecture which we are implementing for Jabber, as depicted in Figure 2, consists of a number of independent PC workstations (currently two), each of which is running the ProShare video product. These workstations are all connected to a Sparc20 workstation, running Unix (chosen because of hardware requirements for the speech recognition systems)..
speech recognition and the lexical software. This workstation in turns provides annotations to each of the ProShare users, as will be described next in the description of the Jabber prototype.
5.2 Prototyping Jabber We built a prototype of Jabber in order to: 1) explore capturing the interaction paradigms we hypothesize, 2) evaluate the efficacy of the automatic indexing process, and 3) begin testing the user interface design. The prototype only provides for audio conferencing between distributed participants in its current instantiation. As the user hears the exchanges taking place in the audio conference, horizontal bars—one per speaker—are generated. These form a “time line” that moves across the screen. Symbols representing an analysis of the words spoken by the meeting participants appear in the horizontal bars at the position of their occurrence. These symbols represent words generated from the lexical chaining process. If a new theme is started, the new theme is shown as a special button below the time lines. In this fashion the display gives a snapshot summary of the distributed conversation taking place. Different views of the meeting exchange are provided in Jabber to allow users to see the relationship between the discussion and agenda items, the evolution of the major themes of the meeting and the summary of each participant's contribution. Three screen images from the Jabber prototype are shown below. Figure 3 shows the beginning of a meeting during which four themes have been established: information, file, stuff and question (this was a meeting about the design of a questionnaire). From this screen, one can easily determine who was talking about what, when. One can also use this interface for browsing (selecting a theme allows a user to retrieve more information about the discussion at that moment, including replaying the relevant audio portion).
PC + ProShare Sparc20 + speech recognition PC + ProShare
Figure 2: Jabber system architecture As users share information (audio and application records), this information is echoed to the workstation running the
Figure 3: Viewing a meeting by theme within participant Figure 3 shows a “theme view of the same portion of the
conversation. In this view themes are plotted on the time line, and the speaker is iconically represented (as an oval, a pentagon or an open square in this case). By viewing in this
Figure 4: Viewing a meeting by participant within theme mode, we see a meeting primarily as a set of themes, and secondarily as a set of speakers. Finally, Figure 5 shows a much larger segment of a different meeting (1 hour) indexed according to both themes (as indicated by the letters b, p, w, and o) and agenda items (as indicated by the numbers 1, 2, and 3). In this figure we can see, for example, that agenda item 3 was discussed briefly at the start of the meeting and then more thoroughly at the end of
the meeting.
5.3 User Interface Issues We are focussing on four major interface design issues in building Jabber. First, among these is the need to keep the cognitive effort to use the interface as low as possible. Verbal exchanges demand considerable attention from meeting participants. If desiring to speak, a participant must rehearse what he or she is going to say, adapt it to the changing focus of the person who is speaking, attend to nonverbal cues indicating when he or she can enter the conversation and also absorb the content of what is being said. If participants are not ready to speak, they must still process the verbalizations of the other speakers, interpret the content in terms of how they view the world and map the content on to the evolving structure of what is taking place in the meeting. Any additional cognitive demands such as note taking or needing to interact with a computer program make this difficult task all the more difficult. Thus far, we have designed Jabber to be automatic, not requiring input from the user. However, this does not mean it does not affect cognitive load. Jabber is like a television program without sound being displayed during the meeting discussions. The text alone is likely to attract attention. We have thus turned the words arising from the lexical analysis into symbols with their explanation placed in the lower portion of the screen. In this way, we have tried to create an interface where the user can focus
Figure 5: Agenda items and themes within participant
attention on the information being displayed but where this information is not demanding the user’s attention. A second user interface issue is the opposite of the first. Since lexical chaining is a crude form of generating meeting indices, it is likely to make some obvious mistakes. It would be useful for the user to observe and correct these during the meeting. When the user observes a misplaced word, he or she can: 1) move it to the appropriate chain, 2) create a new chain, or 3) remove it completely. The third user interface issue is one of presenting the appropriate summaries of the information being captured. Records of meetings such as minutes present summaries that do not reflect the meeting process. Discussions after meetings also indicate that users talk about who defended a particular issue, which group of individuals formed a meeting clique, etc. We have provided a time-based summary as the primary interface to allow the user to check the record keeping that is taking place simultaneously with the meeting. However, we know from meeting studies, that topics are dropped, returned to, dropped, returned to, etc. throughout the meeting. We have therefore provided a theme based summary and related this to the agenda items prepared for the meeting. We have also provided a “people” view so that users can see who is talking about what topic. At this point, we do not know if these views provide useful support for viewing the meeting taking place and/or if they will make a difference in what the user takes away as a summary of what transpired in the meeting. Our first goal with the interface is to make it one that accurately provides an index to the video and audio record of the meeting. Thus, we need to develop interactions with these views which allow meeting participants to adjust the summaries to reflect their perception of what transpired at the meeting. We have not made these additions as yet. In addition to the type of summary views to show, we need to present these views as an aggregation of information that is immediately recognizable and interpretable by the user. This interface tuning of the views and the presentation of a semi-unorganized set of words needs more work in our design. Our fourth user interface issue is one of the efficacy of the retrieval. Although the indexing may work well and the real time meeting go smoothly, this does not mean that the indices generated will allow a user to effectively query the video record at a later date. Words used within a meeting discussion do not necessarily make good query indices at a later date. This is especially true, at the beginning stages of a product design where more general terms are used until the design team starts to generate names for each of the units of the design. Retrieval of early meeting records could be difficult because of the naming ambiguities inherent in the meetings. We have built the prototype system in order to test this
efficacy on a user population. Our goal is that if we do seventy percent as well as human generated indices, we will feel that Jabber has the potential to be a useful annotation tool.
6 Lexical Chain Research 6.1 Generating Themes In order to address issues of theme generation we used a lexical chain generation program called LexC. LexC uses WordNet to create chains of textual sentences based on morphological identity, synonymy, and the shortest path between words within the relational network of WordNet. LexC allows one to parameterize the notion of chain. We used the default parameters, the more important being: number of sentences separating words (default 7 for non-identical words), and a word discard list of over 200 words (closed class words, such as connectives, determiners, and pronouns, as well as very common open class words). We applied LexC to the transcripts of four meetings which were 45 minutes, 1 hour, 1 hour, and 2 hours long, respectively.
6.2 Results An important issue associated with the lexical chaining process was whether it could work in real time, a necessary user interface constraint for Jabber. The run time of LexC was less than 10% of the meeting time in two cases and less than 25% for the other two. This shows that it is a feasible technology for real-time meeting annotation. The four meetings generated 30 to 60 chains of three or more words. About 60% of the words were in the discard list and from 15% to 25% were unrecognized. The results were encouraging as far as the speed of theme generation and number of themes created. However, the number of unrecognized words and size of some of the themes (chains) initiated further developments. These include: introducing repeated but unrecognized words as chains, increasing the size of the word discard list, and testing the appropriateness of generated themes.
7 Conclusions/Future Research Having completed a significant amount of prototyping (implementations, and pen-and-paper experiments) and user studies, we are now in the process of realizing the complete implementation architecture shown in Figure 2. We believe that giving users a variety of methods for interacting with the records of stored meetings will alleviate many of the potential problems associated with meetings and cooperative work. Users might browse or search the records of meetings after the fact, making meetings a useful repository
of information, they might use the information which Jabber gives them to better understand a meeting while it is in progress, or some participants might use Jabber to summarize or catch up on the results of the meeting in progress (if, for example, they came in late). Areas for future research in the area of supporting distributed meetings include:
• decision making aids/consensus tools • meeting summarization by: topic, participant, and time • performance measures of participants (which could be used by participants, observers, or management to assess and help improve an individual’s performance)
• identifying meeting “cliques” (sub-groups of meeting participants who are consistently not communicating with those outside of their clique—a common cause of misunderstandings in meetings).
• analysis based upon non-verbal communication (e.g. cues signifying attempts at interruption, assent, confusion, etc.)
• linking meetings to previous meetings to create a web of related discussions and information While some of these topics might be addressed in the current research program, most will likely be relegated to future research. Consideration of them will, however, affect the software architecture of the research prototypes which we build, since we will want to build our software with a view to future modifications.
8 Acknowledgments We gratefully acknowledge the help of David St.-Onge, who provided LexC. We also acknowledge support for this research from the Natural Sciences and Engineering Research Council of Canada and Intel Corporation.
9 References [1] R. Beckwith, C. Fellbaum, D. Gross, G. Miller, “WordNet: A lexical database organized on psycholinguistic principles”, In: U. Zernik (ed.), Lexical acquisition: Exploiting on-line resources to build a lexicon, Lawrence Erlbaum, Hillsdale, NJ, 1991, 211-232. [2] R. Boland, A. Makeshwari, D. Te'eni, D. Schwartz, V. Ramkrishnan, “Sharing Perspectives in Distributed Decision Making”, Proceedings of the ACM 1992 Conference on Computer-Supported Cooperative Work, 1992, 306-313. [3] B. Chalfonte, R. Fish, R. Kraut, “Expressive Richness: A Comparison of Speech and Text as Media for Revision”, Proceedings of the ACM Conference on Human Factors in
Computing Systems, 1991, 21-26. [4] J. Conklin, M. Begeman, “gIBIS: A Tool for All Reasons”, Proceedings of the ACM 1988 Conference on Computer-Supported Cooperative Work, 1988, 140-152. [5] B. Doyle, “Crunch Time for Digital Video”, New Media, March 1994, 47-49. [6] C. Ellis, S. Gibbs, G. Rein, “The Groupware Project: An Overview”, Technical Report STP-033-88. Microelectronics and Computer Technology Corporation, Austin, Texas. January 1988. [7] B. Harrison, R. Baecker, “Designing Video Annotation and Analysis Systems”, Proceedings of Graphics Interface '92, 1992, 157-166. [8] W. Hunt, M. Mantei, “What did We Really Decide? A Case Study of Meeting Participants’ Decisions and Subsequent Understanding”, unpublished ms., University of Toronto, 1994. [9] R. Kazman, J. Kominek, “Information Organization in Multimedia Resources”, Proceedings of SIGDOC ‘93, 1993, 149-162. [10] W. Mackay, G. Davenport, “Virtual Video Editing in Interactive Multimedia Applications”, Communications of the ACM 32(7), July 1989, 802-810. [11] J. McCarthy, V. Miles, A. Monk, “An Experimental Study of Common Ground in Text-based Communication”, Proceedings of the ACM Conference on Human Factors in Computing Systems, 1991, 209-215. [12] S. Minneman, S. Harrison, “Where Were We: Making and Using Near-Synchronous, Pre-Narrative Video”, Proceedings of the ACM Conference on Multimedia, 1993, 207214. [13] J. Morris, G. Hirst, “Lexical Cohesion, the thesaurus, and the structure of text”, Computational Linguistics, 17(1), March 1991, 211-232. [14] W. Myers, “MCC: Planning the revolution in software”, IEEE Software, November 1985. [15] G. Olson, J. Olson, M. Carte, M. Storrsten, “Small Group Design Meetings: An analysis of Collaboration”, Human Computer Interaction, 7(3 & 4), 1992, 347-374. [16] M. Scardamalia, C. Bereiter, “Higher Levels of Agency for Children in Knowledge Building: A Challenge for the Design of New Knowledge Media”, The Journal of the Learning Sciences, 1(1), 1991, 37-68. [17] S. Stevens, “Informedia“, Interactions, 1(4), 1994. [18] M. Twidale, T. Rodden, I. Summerville, “The Designers' Notepad: Supporting and Understanding Cooperative Design”, Proceedings of the Third European Conference on Computer-Supported Cooperative Work, 1993, 93-108.S. [19] Whittaker, P. Hyland, M. Wiley, “Filochat: Handwritten Notes Provide Access to Recorded Conversations”, Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI'94), Boston, MA, May 1994, 271277.
