CHI 2009 ~ Information Foraging
April 7th, 2009 ~ Boston, MA, USA
Remembrance of Things Tagged: How Tagging Effort Affects Tag Production and Human Memory Raluca Budiu Peter Pirolli Lichan Hong Nielsen Norman Group Palo Alto Research Center Palo Alto Research Center 48105 Warm Springs Blvd 3333 Coyote Hill Rd 3333 Coyote Hill Rd Fremont, CA 94539, USA Palo Alto, CA 94304, USA Palo Alto, CA 94304, USA
[email protected] [email protected] [email protected] Social tagging systems are examples of what Benkler [6] has defined as commons-based peer production systems, in which knowledge products (e.g., tags, Wikipedia articles, open source modules) are produced by decentralized, largely independent aggregates of users. The utility of such systems typically depends on having large user bases, and those participation rates are partly driven by having low cost-of-effort interaction and communication. A recent large-scale analysis and model of contributions to Wikipedia and similar systems [19] demonstrated the positive relation between lower interaction effort and increased participation. Consequently, in the development of our own social tagging system, SparTag.us [12], we have been motivated to develop techniques that lower the costs of producing tags and other annotations.
ABSTRACT
We developed a low-effort interaction method called Click2Tag for social bookmarking. Information foraging theory predicts that the production of tags will increase as the effort required to do so is lowered, while the amount of time invested decreases. However, models of human memory suggest that changes in the tagging process may affect subsequent human memory for the tagged material. We compared (1) low-effort tagging by mouse-clicking (Click2Tag), (2) traditional tagging by typing (type-to-tag), and (3) baseline, no tagging conditions. Our results suggest that (a) Click2Tag increases tagging rates, (b) Click2Tag improves recognition of facts from the tagged text when compared to type-to-tag, and (c) Click2Tag is comparable to the no-tagging baseline condition on recall measures. Results suggest that tagging by clicking strengthens the memory traces by repeated readings of relevant words in the text and, thus, improves recognition.
Lowering the cost of tagging effort should also have an impact on individual tag production. An extension of information foraging theory [15], discussed below, predicts that lowering the effort of producing tags should also increase individual tag production rates while decreasing the time devoted to tag production. However, there may be cognitive costs. Theories of memory [1] suggest that changes in the kind and amount of tagging effort may affect how well people remember the original content or tags. We want to avoid tagging techniques that provide low interaction costs, but make it harder for people to remember the content that they have tagged.
Author Keywords
Tagging, memory, foraging theory.
social
bookmarking,
information
ACM Classification Keywords
H5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous. INTRODUCTION
In recent years, there has been an explosion of social bookmarking systems (e.g., del.icio.us, diigo, ma.gnolia, CiteUlike). These systems allow users to generate labels or keywords (tags) to content encountered on the Web. These tags can later be used by the same user, or by others, to retrieve tagged information. The social aspects of these systems emerge from implicit or explicit sharing of tags among users.
In this paper, we unravel how different techniques for producing tags to Web content affect individual tag production and individual memory. In particular, we performed an experimental contrast of a lower interaction cost technique (Click2Tag), developed for our system SparTag.us, against a standard (higher interaction cost) type-to-tag technique, similar to ones used in popular tagging systems such as del.icio.us. Click2Tag allows users to simply mouse-click words in a text to have the words become tags for the content. Type-to-tag allows users to type their own tags for the content. Both of these tagging techniques were contrasted with a baseline condition of no tagging.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. CHI 2009, April 4–9, 2009, Boston, MA, USA. Copyright 2009 ACM 978-1-60558-246-7/09/04…$5.00
We examined the effect of these techniques on both recognition and recall tests for the original material. As we discuss below, previous memory research [1, 8, 17] leads us
615
CHI 2009 ~ Information Foraging
April 7th, 2009 ~ Boston, MA, USA
to hypothesize that Click2Tag would produce better recognition of the original material, but type-to-tag would produce superior recall. These results are predicted by differences in the way the techniques either strengthen or elaborate the memory traces for the original content.
while they are reading it. When tags are entered, they are inserted at the end of the paragraph and displayed within the same rendered web page. This design was inspired by annotation studies [13, 14] and by in-document collaboration systems [11], in which the content, and not some external form, is used as the setting for information sharing. Thus, Click2Tag combines synergistically with in situ tagging, ensuring that the focus of attention remains on the content at hand.
BACKGROUND: SPARTAG.US
Hong et al. [12] introduced the SparTag.us system. The design of SparTag.us was motivated by the objective of providing low-effort tagging and highlighting capabilities to users as they browse and read Web content. SparTag.us consists of two parts: A client-side browser extension plus a server. Tagging and highlighting functionality is made available in the same window displaying a Web page. Every paragraph is taggable (Figure 1), and text can be highlighted in yellow by click-and-drag. A searchable notebook collects together all tagged and highlighted material. Tags and highlights may be shared by setting various sharing permissions.
It is important to note that SparTag.us users are not constrained to use only the Click2Tag technique. Figure 2 shows that SparTag.us users actually also have the option of typing in any words they like, using the more standard typeto-tag technique.
Figure 1. SparTag.us: A user (e.g., “hong”) can click on individual words in the paragraph to tag the paragraph with that word.
Hong et al. [12] summarized GOMS analyses of several standard tagging interfaces as well as SparTag.us. Many, if not most standard systems, use a type-to-tag technique in which users can generate any string as a tag and enter it into a tag list. The systems analyzed by Hong et al. appeared to involve two types of significant costs: (1) interaction costs in terms of mouse clicks, button presses, and typing, and (2) attention-switching costs involved in switching attention back and forth between the material being read and various kinds of dialog boxes. SparTag.us was designed to reduce these two types of costs by integrating tagging into the flow of reading. Specifically, the Click2Tag technique makes two types of sub-page objects live and clickable: paragraphs of the web page and words of the paragraphs. According to Hong et al.’s analysis of del.icio.us data, a considerable portion of tags comes directly from words on the web pages. When people tag a web page in del.icio.us, there is about 50% chance that the tag word appeared in the content of the page. Making each word of the paragraphs clickable allows users to simply click on a word to add it to the tag list without typing, thus lowering the interaction costs of tag typing. This input method can be especially useful in cases where keyboard input is not the primary input means (e.g., iPhones, tablet readers).
Figure 2. Sequence of actions needed to tag a paragraph in SparTag.us. SparTag.us supports both Click2Tag and type-to-tag. EFFECTS OF COST OF TAGGING EFFORT Interaction Effort and Participation in Social Production
Social information foraging theories [15], as well as microeconomic theories of networked, commons-based peer production information economies [6] predict that as the costs of production of shareable knowledge (e.g., tags) are driven down, more individuals will participate and reap greater net benefits. Thus, reductions in the cost of tagging will improve the value of the system to the individual user. More tags, and presumably more useful tags for the
Click2Tag mitigates the attention-switching costs by enabling in situ tagging. With the Click2Tag interface, users can directly click on any paragraph to start tagging
616
CHI 2009 ~ Information Foraging
April 7th, 2009 ~ Boston, MA, USA
individual, will be generated as more people join a social tagging community.
reading it, and this is called the average between-patch time, tB.
Collections of people using social tagging systems such as del.icio.us, Connotea, and Clipmarks engage in information foraging and sensemaking activities. As discussed in Pirolli [15], the net benefits gained by users participating in a social information foraging community must be greater than solitary information foraging. Benkler [6] makes a similar point from the perspective of microeconomics. Social tagging systems are instances of networked information economies, involving the production, distribution, and consumption of information by decentralized individual actions over distributed networks. One crucial set of preconditions for the emergence of such economies is that the costs of knowledge production to the individual user must be very low. Reductions in the effort involved in tagging should increase the number of participants in a social tagging system.
Consistent with the information patch model, the current model assumes that tag production on a particular article produces diminishing returns as a function of time. In other words, on average, as time progresses, the user generates tags at an ever-diminishing rate. The cumulative production of tags on an average article may be characterized by a gain function, g(tw). The overall average rate of gain from tag production is (1) where G is the sum of all the gains from all tagging, TB is the total between-patch time (reading and interaction time in the current case) and TW is the total within-patch time (tagging in this case). Under some strong but relatively general assumptions [10, 15] the optimal time allocation to spend on tagging is tW*:
The assumption that social tagging systems are instances of such networked information economies has been one of the prime motivators for reducing the cost-of-effort of interaction with SparTag.us. However, there are theoretical reasons to expect changes at the level of the individual user, and those changes are the focus of the study presented in this paper.
R = g’(tW*)
(2)
where g’ is the marginal rate of (within-patch) tag production. Equation 2 captures the rule: Continue tag production until the marginal rate of gain for continued tagging drops below the overall rate of gain R.
Cost Reduction and Individual Tag Production
Figure 3 presents a graphical representation of this model familiar in optimal foraging theory [18]. Between-patch time is plotted horizontally from the origin to the left and within-patch time is plotted form the origin to the right. The curve g1 represents a hypothetical diminishing returns function for tag production. A line plotted from the intercept tB to a point tangent to g1 will have a slope equal to the overall average rate of gain from tag production R, and the point of tangency to g1 will be g1(tW*), thus giving us the optimal average time tW* to allocate to tagging.
The information patch foraging model from information foraging theory [15] can be used to make qualitative predictions about the effects of reducing the cost of producing tags. The information patch model addresses the optimal allocation of time in information patches (e.g., seeking information at a Web site) versus between-patch activities (e.g., navigating to another Web site). Extending the model to tag production, we assume a simple characterization of the tag producer’s task as involving a trade-off between reading + interaction time vs. tag production time. In the case of the current tagging model, we assume that the user’s tagging activity around an individual article constitutes a “patch” of productive activity of some value to the user. Imagine an idealized user who navigates the Web and reads articles. This idealized user iteratively navigates to a page, reads it, and moves onto the next. Now assume that this idealized user is also engaged in tagging the articles that were read. For each Web page, the user engages in a set of micro-tasks around the addition of tags (e.g., generating the tag from memory or from the just-read text and somehow entering it into a tagging system). So this idealized user’s time can be divided into time devoted to (a) interaction (navigation) plus reading and (b) tag generation. On each article, the user spends some amount of time, on average, engaged in tag generation activities, and this is called the average within-patch time, tW. The user also spends some amount of time, on average, navigating to the article and
Figure 3. An information patch model. Charnov's Marginal Value Theorem states that the rate-maximizing time to spend in patch, t*, occurs when the slope of the within-patch gain function g is equal to the average rate of gain, which is the slope of the tangent line R. The average rate of gain, R, increases with improvements in the gain function, while simultaneously decreasing the optimal time to allocate to tag production.
617
CHI 2009 ~ Information Foraging
April 7th, 2009 ~ Boston, MA, USA
Figure 3 also includes another gain function, g2 that represents the effects of lower time cost associated with producing tags. Going through the same graphical solution of plotting a tangent line to g2, one can see that the optimal time allocation to tagging is reduced while increasing the overall rate of gain R and increasing the number of tags produced.
words. Thus, we expect Click2Tag to produce better recognition performance than type-to-tag. Elaboration Effects on Recall Memory
Elaborateness of encoding of material (i.e., the extent to which people think of additional, possibly non-specified details related to new material) also generally improves aspects of memory. Memory research [7, 8] has shown that, when people are asked (or provided) with additional information that is highly semantically related to the content they are studying, they typically show superior recall when compared to content that has been processed less elaborately. One explanation [3] suggests that the memory traces that elaborate the original content provide additional retrieval routes to recall the content. This is because self-generated elaborations have some high likelihood of being re-generated at recall time as a retrieval route to the content. Since type-to-tag requires users to generate tags to associate with the original content, we expect it to produce more elaborative encodings and to improve recall performance.
The information patch model predicts that lowering the time cost of tag production will increase the number of tags produced per document by individuals while decreasing the amount of time spent on tagging. EFFECTS OF TAGGING TECHNIQUE ON MEMORY
One worry about Click2Tag might be that it trades off costs in tagging time for cognitive costs to subsequent memory. Human memory research suggests that Click2Tag and typeto-tag may have different effects on subsequent memory. Strengthening Effects on Recognition Memory
The strength of a memory trace captures the relationship of practice to memory performance. It has been shown that repeated practice increases strength, and strength decays as a function of time since last practice [1]. Reaction times and accuracy on recognition tests both improve with strength [17]. Click2Tag appears to encourage users to re-attend to the original content and, thus, increase the strength of memory traces for that content. Indeed, as shown in Figure 4a, eye tracking data from pilot studies in our lab suggest that, in the Click2Tag condition, participants often read a
METHOD
We designed an experiment to assess reading/tagging time, tag production, recognition, and recall in a no-tag (reading only) condition, type-to-tag condition, and Click2Tag condition. Participants
We recruited 27 participants who were each compensated with $20 in cash or Amazon gift certificates. Most participants were employees of PARC (18) or students at a local university (7). PARC employees were recruited through an internal mailing list; students and other participants were recruited by posting an ad to a university bulletin board and to craigslist. The participants were not required to have previous experience with tagging, but they had to have experience with reading news or other information on a computer screen. Materials
We selected 18 passages from news articles as well as from various web pages on the Internet. The passages reflected a variety of topics (medicine, education, general science, aviation, history, etc). On average, the passages were 267 words long (ranging from 253 to 279). Procedure
Participants went through three study-recall blocks. A study-recall block had two parts: in the first part, participants performed 6 study trials, and in the second part, participants performed 6 memory trials. Participants were instructed to perform these trials as fast and accurately as possible. The design of the study was within-subjects: all participants saw all conditions described below.
Figure 4. Eye tracking of reading when using two different tagging techniques versus no tagging. Red spots signify more time spent at those locations.
passage and then re-scan the passage to seek out words to click. In the no-tags condition (see Figure 4c), we see participants fixating more on relevant words when they read text, although perhaps less so than in the Click2Tag condition. In comparison, when participants have to type in tags (see Figure 4b), they tend to fixate less on any specific
Study trials. In each study trial, participants read a passage, selected randomly from the list of 18 passages. Participants were instructed to read at their own paces, but if they spent more than 2 minutes on a trial, they were automatically
618
CHI 2009 ~ Information Foraging
April 7th, 2009 ~ Boston, MA, USA
moved to the next trial. The trial could belong to one of three conditions as follows:
and within each block, there be 2 passages per condition. To keep them engaged in the experiment, at the end of each block, subjects were given feedback about their overall performance so far in the recognition tests. We did not give feedback after individual recognition trials.
• No-tags: In this condition, no tagging was performed. • Click2Tag: Participants had to tag the passage with relevant words by clicking on words from the passage. The tags were displayed in a box under the passage and could not be modified by the participants. • Type-to-tag: Participants had to tag the passage with any relevant tags that they could generate, and type those tags in a box under the passage. Subjects were not allowed to cut and paste in this condition.
For each trial, we measured the study time, the number of tags, the number of facts recalled and the recall time, the recognition time per question and the recognition accuracy. RESULTS Overview of the Results
Figure 5 provides a summary of the main measures of interest discussed in this section. The upper part of Figure 5 presents measures associated with the reading and tagging of articles when presented for study: the amount of time spent reading and tagging and the number of tags generated per article. The middle part presents two recall measures: the mean number of facts recalled per article and a recall efficiency score that normalizes for study time. The lower two groups of bars in Figure 5 present recognition measures for accuracy and time. In the statistical analyses that follow, we performed ANOVAs with subjects as the random factor using Block (0,1,2) and Condition (no-tags, Click2Tag, type-to-tag). When there was a correlation with study time, we performed ANCOVAs with study time as a covariate.
Memory trials. After 6 study trials, the participants completed 6 memory trials. A memory trial had two components, presented in the following order: • Recall: Participants were given two cues about one of the 6 passages they had previously studied (e.g., “One of the passages you read was about Christmas and Santa Claus”). Then they had to remember and type as many facts from that passage as possible. There was a time limit of 1.5 minutes per passage for this phase. • Recognition: Participants had to answer 6 true/false sentences (3 true and 3 false) about the passage they had just recalled. They had 1 minute to answer all questions.
Tag Production and Reading-Tagging Time
The order of presentation of questions was randomized, and the order of passages within a block was randomly generated for each participant, as was the assignment of a passage to a particular block or condition. The passages in the memory trials were presented in a random order, unrelated to the order in which the passages were studied. The only constraint was that there be 6 passages per block,
Figure 5 shows the reading and tagging times for the three conditions. There was a significant effect of condition (F(2,52)=52.72, p
