Using tag clouds to facilitate search: An evaluation

Using tag clouds to facilitate search: An evaluation Carol Shergold

Judith Good

IT Services University of Sussex Brighton, UK +44 (0)1273 873144

IDEAs Laboratory Department of Informatics University of Sussex +44 (0)1273 873228

[email protected]

[email protected]

Mirona Gheorghiu, John Davies TLDU University of Sussex +44 (0)1273 873219 M.A. [email protected] [email protected]

ABSTRACT In this paper, we describe an experiment designed to investigate the use of tag clouds as a method of search. We compared students’ use of tag clouds with that of input field search boxes. We were interested in user preference for one method over another, and in investigating whether the initial preference changed over time. Additionally, we were interested in the extent to which characteristics of the tag-cloud, namely variations in font size, have an influence on search strategy. During the experiment, participants were asked to search for a particular term and given the option of using either an input field search box, or a tag cloud in which the term was present. We found that, overall, tag clouds were used more frequently for search than search boxes. Furthermore, although font size had an effect in early trials (with an inverse relationship between font size and preference for using the tag cloud for search), this effect decreased in later trials. Finally, qualitative analysis of interviews with students suggested a number of themes which are relevant to the use of tag clouds as a search method, including the particular search strategy being used, and issues of font size variation with the tag clouds.

Categories and Subject Descriptors H5.2 [INFORMATION INTERFACES AND PRESENTATION (e.g., HCI)]: User Interfaces – Interaction Styles, Evaluation/Methodology

General Terms Design, Experimentation, Human Factors

Keywords tag clouds, evaluation, visualisation, navigation, usability, web design

interface,

search,

[2]. The SkillClouds project attempts to address these issues through an exploration of the use of social bookmarking software and tagging [4], and in particular through the tag cloud data visualisation technique that has become a distinctive feature of web 2.0 sites [1]. Our hypothesis is that representing transferable skills as a “skill cloud” will be an engaging way of visualising this information for students. In the study described in this paper, we investigated some foundational issues in the development of such representations, namely, how are tag clouds used for search and navigation, and how do they compare to more “traditional” methods such as search boxes? In the sections below, we describe tags and tag clouds in more depth, and consider related work in the area before going on to describe the study and its results.

2. TAGS, TAG CLOUDS AND SKILLCLOUDS A ‘tag’ is a keyword or descriptive term that is associated with a particular resource, such as a photograph, web page or blog posting, and which is used in order to facilitate the retrieval and dissemination of that resource. A ‘tag cloud’ is a weighted list of tags presented in paragraph layout. Typically, tags in a tag cloud are displayed in alphabetical order, and underlying attributes within the data set, such as the frequency of use of tags, are mapped to display features within the tag cloud such as font size or colour. In a tag cloud, each tag is generally a hyperlink to a list of the resources that have been tagged with that keyword. So tag clouds both display the key terms within a set of resources and provide a tool to use for search/navigation into those resources. A ‘tag’ is a type of metadata, but in many tagging systems the key point about tags is that they are user-generated and freely applied, rather than being based on an institutionally approved taxonomy (see [4] for a thorough review).

1. INTRODUCTION The ability of graduates to identify skills they have gained while at university is something that employers rate highly in selecting candidates [8]. However, these skills are not always transparent to students, but are hidden within the curriculum

© The Author 2007. Published by the British Computer Society

Figure 1. Tag cloud from del.icio.us based on one of the author’s tagging activities (CS)

Tag clouds are often associated with these user-generated tagging systems, often characteristic of web2.0 sites. In such systems, the weightings represented by the varying tag size will be directly meaningful to the user since they are the result of the many individual tagging activities that the user has undertaken. Tag clouds can also represent any set of weighted or unweighted terms, regardless of how these tags were derived. For example, tags can be derived from semantic frequency analysis, and there are many websites offering this service1. Tag clouds can provide interfaces into search tools, for examples to facilitate local services search 2 and job vacancy search 3. Tag clouds are an example of a search tool that promote a primarily browsing approach to the search task as opposed to the analytic approach required by search engines [6]. Browsestyle search is characterised by a lower cognitive load and is particularly suitable where the ‘anomalous state of knowledge’ paradox is present and where the information seekers may not have sufficient knowledge and understanding of the area to define an analytic approach to solving the problem. In the SkillClouds project4, tag clouds will be used to represent the transferable skills that students have acquired. The skill cloud will be automatically generated for each student, based on the merging or mashing-up of institutionally generated data (from a skills database associated with courses students have taken) and student generated data.

3. RELATED WORK Whilst there has been considerable discussion about the use of tag clouds in information architecture terms, and criticisms of usability of tag clouds (see [5] for a useful summary) there have been few systematic evaluations.

4. METHOD 4.1 Participants 116 undergraduate students participated in the experiment. They were recruited within a single week on the University of Sussex campus, from IT resource rooms, the Library and the Student Union. They were all asked to consent to taking part in a brief study that would last no more than 5 minutes.

4.2 Materials The experiment was conducted using a laptop with keyboard and mouse for input. A web application written in PHP handled the administration of the search experiment. Interviews were recorded using a small digital recorder.

4.3 Design The experiment was designed to investigate whether 1) observed search strategy (e.g. search input box or tag cloud) varied according to the tag size being presented (and whether this changed over time), and 2) whether search time varied according to the size of the tag. For the first hypothesis, search type is the dependent variable, and trial number and tag size are independent variables. For the second hypothesis, elapsed time is the dependent variable, and trial number, tag size and search type are independent variables, giving a 3x3x2 within subjects design.

4.4 Procedure We explained to participants that they would be shown a mock up of a web page that displayed services available in a local area, they would be asked to search for a particular service, and that this would be repeated for a total of three trials. The order in which the terms were presented was randomized.

Rivadeneira et al [7] explored the recognition and recall of terms displayed to users in a tag cloud-like layout, but did not invite participants to interact with the tag clouds, nor did they investigate participants’ perceptions of the task. They found that recall was associated with font size (words rendered in a larger font size were recalled more effectively) and location, with terms appearing in the upper left quadrant being recalled more effectively. Halvey and Keane [3] described work in which participants were asked to locate the name of a country as quickly as possible from a set of 10 country names randomly selected from a pool of 60. A number of different layouts were used, including randomly and alphabetically ordered vertical and horizontally presented lists, and randomly and alphabetically ordered tag clouds with random font size differences. They found that alphabetically ordered formats were searched most quickly, and that randomly ordered tag clouds were slowest. In contrast to these studies, we were interested in how users interact with tag clouds over more than one exposure to them, and in their choice of search strategy when presented with more than one alternative.

1

See http://www.tagcrowd.com/ for an example

2

See http://www.yell.com

3

See http://www.wiredsussex.com/Jobs/JobSearch.asp

4

See http://www.sussex.ac.uk/skillclouds

Figure 2. The experimental page Each target search term was present on the tag cloud. Each term had 8 letters. The term ‘florists’ was presented in a large font, ‘cinemas’ in a medium sized font and ‘printers’ in a small font. The tag cloud was identical for all searches. For each trial, the target search term, actual search term, search method and elapsed time was written to a database. At the end of the three trials, each participant was interviewed briefly. Analysis of audio recordings was undertaken to identify the key themes emerging from participants’ experiences. Data from 9 responses to the first trial, 7 to the second trial and 3 from the third trial were eliminated due to technical difficulties in writing the responses to the database, or because the participant searched for a term other than the target term.

5. RESULTS 5.1 Use of tag clouds for search 89% of participants used a tag cloud search in at least one trial, and 65% used the tag clouds for all three trials.

5.2 Does search strategy vary with tag size? Logistic regression analysis was undertaken. The results indicated that, overall, the size of the tag did not have an effect on preferred strategy. However, being presented with a small search target in the first trial significantly increased the probability that the participant would use the text search input box instead of the tag cloud (b = 1.73, SE = .83, p=.038, OR = 5.62). On subsequent trials, this effect was not present, suggesting that familiarisation had occurred as a result of a single exposure to the tag cloud.

100

Font size presented in trial 1


However, the majority of students who chose to use the tag cloud search did so because they felt it was easier, quicker or both: “Because I was unsure, so it’s easier to be given a list of options and choose one of those, than think on your own and type it in.” These observations neatly characterise the difference between the analytic and browse approach to search [6]. In fact, almost everyone used the terms ‘quicker’ and ‘easier’ to characterise and explain their search strategies, whether they had used the text entry box or the tag cloud. It became apparent that participants were using strategies to optimise their searching .

5.4.2 Tag Cloud as ‘random words’ For some students, the most effective way of optimising their search was to ignore the tag cloud. They assumed it was unlikely to repay their attention; its affordance as a search tool was not obvious to them:


90 80

“All search engines have a lot of other things on it, and it takes time to look through it...”

Percentage

70 60 tag cloud text box

50 40 30

“Maybe because it [the tag cloud] is like lots of random words, I didn’t look at [it]. Maybe if it was organised in another way...”

20 10

sm al lf on m t ed iu m fo nt la rg e fo nt

sm al lf on m t ed iu m fo nt la rg e fo nt

al

lf on m t ed iu m fo nt la rg e fo nt

0

sm

“I glanced at them [the tags] and saw how many there were and thought ‘No! I’ll just type it in!’”

Figure 3. Choice of search method according to font size of target search term over three trials

5.3 Findings in relation to search time A 3x3x2 analysis of variance was undertaken to test for the effects of search strategy, tag size and trial number on search time. We obtained the following results: • The decision to make a tag cloud search was quicker (M=7.09, SD=6.17) than a text box search (M=11.55, SD=7.77; F(1, 327) = 8.82, p=.003) • Searching for a medium or large target took significantly less time (M=7.87, SD=6.67; M=6.30, SD=5.07) than searching for a small target ((M=9.98, SD=7.87; F(2, 326) = 7.61,p=.001) • The first search task took significantly longer (M=13.46, SD=7.98) than the second or third tasks (M=5.64, SD=4.68, M=5.27, SD=3.49); F(2, 326) = 49.35, p