Envisioning the Web: User Expectations about the Cyber-experience Craig Scull City University of New York, New York Allen Milewski, David Millen AT&T Labs, Redbank, New Jersey
ABSTRACT An exploratory research project was undertaken to understand how novice college students and Web savvy librarians initially envisioned the Internet and how these representations changed over time and with experience. Users' representation of the Internet typically contained few meaningful reference points excepting "landmarks" such as search sites and frequently visited sites. For many of the users, the representation was largely procedural, and therefore organized primarily by time. All novice users conceptualized search engines as literally searching the entire Internet when a query was issued. Web savvy librarians understood the limitations of search engines better, but did still expect search engines to follow familiar organizational schemes and to indicate their cataloguing system. Although all users initially approached the Internet with high expectations of information credibility, expert users learned early on that "anyone can publish." In response to the lack of clear credibility conventions, librarians applied the same criteria they used with traditional sources. However, novice users retained high credibility expectations because their exposure was limited to the subscription-based services within their college library. And finally, during an assigned search task new users expected "step by step" instructions and self-evident cues to interaction. They were also overwhelmed and confused by the amount of information "help" displayed and became impatient when a context appropriate solution to their problem was not immediately offered.
INTRODUCTION The Internet is increasingly recognized for the vast array of information, services, meeting places, and communitiesof-interest that it offers. It is unfortunate that this vast resource of valuable information and services is not more easily accessible. Unlike the nation's telephone systems, road systems, and libraries, on the Internet there is no comprehensive organizational system. The result has been what some might call a chaotic state of affairs regarding the organization of literally millions of private and public web sites. It follows that information foraging has become recognized as an important activity supporting a number of interesting research projects (see, for example, Wexelblat & Maes, 1999; Pirolli and Card, 1995). These foraging services can be grouped into two main categories, active and passive. Passive information foraging is supported by push-based information services such as alerting services, on-line subscriptions, and listservs while active information foraging services include searchable indices and directories such as Alta-Vista, Yahoo, BigBook, and AOL "channels." Because search indices in particular have a strong media presence and definite advantages over other kinds of Internet services, they have become a gateway to the Internet for large numbers of people. While many users find search applications a useful means to finding timely and credible information, for a substantial number of others the experience is overwhelming and frustrating, as they sift through screens full of outdated and irrelevant information of dubious credibility. To better understand Internet foraging behavior, an exploratory research project was undertaken to examine how novice college students and Web savvy librarians initially envisioned the Internet and how these representations changed over time and with experience.
METHODOLOGY For the sake of conciseness we will broadly describe the methodology used in this study, introduce the criteria for participant selection, and present the interview guide in outline form. If the reader would like a full copy of the interview guide or a detailed description and rationale for the research methodologies used in the individual sections please contact the first author at
[email protected].
General approach Qualitative research methods were chosen for their particular strengths in uncovering the meanings of experience from the participants' own perspective. As a form of naturalistic inquiry, the semi-structured interview, was specifically chosen and it was followed by a search task on the Internet which allowed participants to demonstrate their navigational skills. Participants Participants in this study were selected from two groups. Librarians for this study were recruited from the new Science Industry and Business division (SIBL) of the New York Public Library (NYPL) , which is located in midtown Manhattan. SIBL is designed to be the most high tech library in the country with over 100 on-line computers connected to a T3 Internet connection. Additionally, SIBL has a large number of subscriptions to private databases on the Internet available to library patrons who use the library's computers. All of the 30 librarians at SIBL have MIS degrees (in most cases obtained less than 10 years ago). They teach free hands-on Internet classes to the public in two state of the art, on-line classrooms where over fifty thousand patrons have either acquired or improved their Internet skills. The librarians also use the Internet in their daily activities at the reference desks, and create both internal and ext ernal homepages for the NYPL organization. Students for this study were recruited from The Laboratory Institute of Merchandising (LIM) is a small private college of approximately 200 students located in midtown Manhattan. The school trains young adults for high level jobs in the fashion industry's top companies. Three computers in the library are dedicated to AOL access via a modem. LIM also subscribes to the Electric Library, a private database. All of the students interviewed were of typical college age and enrolled full-time with a busy schedule of classes and work projects. Five NYPL librarians, 3 females and 2 males, were selected on the basis of high Internet experience and low computer anxiety while five female LIM college students were chosen based on low Internet experience and high computer anxiety. Maurer's (1983) computer anxiety index was used to screen volunteers for participation in the study as well as a question regarding familiarity and experience with the Internet. Contrary to popular exp ectations it was not difficult to find students with high levels of computer anxiety. With the participants'permission all conversations were videotaped and selectively transcribed. Interview Procedure Partic ipants were interviewed individually and each session began by having the participant describe their personal experiences using the Internet. Participants were encouraged to create a historical time line that spanned from their first time on the Internet to the present. To elicit mental models of the Internet, they were also asked to sketch an early and a recent "map" of the Internet as they understood it. Using the most recent map, participants then explained how a search of the Internet would be carried out. The timeline and mapping execises were followed by several questions about the credibility of the information that they found on the Internet. Participants were then assigned an Internet search task and asked to find a specific piece of information using a standard web browser and a popular Internet search portal. During this hands-on task, participants were encouraged to "think aloud" and describe their search strategies and/or to discuss any problems or difficulties that they encountered. The final exercise was to have the participants help code their personal timeline and their Internet maps by adding small stickers to the drawings. These stickers suggested various emotional states such as happy, frustrated or sad. Participants were also encrouages to draw their own emoticons if they desired. The participants were encouraged to describe their rationale for selecting each of the the stickers that they applied.
RESULTS AND DISCUSSION
Cognitive Maps of the Internet The concept of cognitive maps has a long history in Environmental Psychology and Geography and it is most commonly used to refer to the mental representations people use to find their way through physical spaces such as buildings and transportation systems (Kitchin,1994.) This section will present the sketch map findings which were taken to be indicative of both librarians' and students' cognitive maps. Librarians' sketch maps of the entire Internet were primarily schematic while students' maps were either procedural or schematic. Moreover, the students approached the task with uneasiness because they felt they knew too little about the Internet. Librarians verbal descriptions accompanying their drawings demonstrated a thorough understanding of the connectivity within the distributed computer network that makes up the Internet. While librarians did apply familiar metaphors such as "pond," "brain," and "web" to the Internet, their sketch maps bore no relationship to spatial models or metrics of any kind:
It would appear that the chaotic structure of the Internet itself is in part responsible for why Librarians' prior experiences on-line had not been synthesized into a usable organizational framework at the scale of the entire Internet. However, at the scale of hyper-linked web sites which pertain to a more familiar level of experience, some forms of structure surfaced. In fact, our analysis of the librarians' Internet history narratives showed how memories of paths or "traveler's tales" are like those resulting from navigation in the physical environment. In both instances, the route memories are episodic in nature, meaning that they are organized sequentially by time. Librarians spontaneously offered personal accounts of information seeking, such as looking for educational web sites to add to the NYPL home page. The following quote exemplifies a typical traveler's tale across a locale of the web: PBS had a really wonderful home page they have a lot of wonderful education materials.. electronic fieldtrips and that led me to a lot of other science education links and then I got into, somehow during that whole PBS search, the Smithsonian homepage and museums. Searching through pages of information the librarian seems to recall a "route" marked by the items (landmarks) that pertained to her task. Similarly, people commonly remember only the relevant or distinct landmarks along their routes through complex and unfamiliar physical environments without necessarily developing relational or survey map knowledge [Bercaw, 1997]. Such environments may be further characterized as: poorly signed and mapped, undifferentiated, containing an inconsistent organizational system and too much visual obstruction. The use of a "route" metaphor stems from the librarian's own descriptions of "stepping" and "exploring" from a "starting point" at PBS:
I would use something like this [PBS] as a starting point and then you kinda go forward, two steps forwardone step back- two steps forward- one step back. Interviewer: What's the one step back? When you find something that's not really right and you literally go back and explore another link. (Lib-SJ12:41) The librarian repeats "two steps forward-one step back," showing she only needs to explore one level to make a relevancy and credibility check. Students didn't feel they could sketch a mental model because they thought the Internet was confusing and they knew too little about it. Students' maps can be categorized in two ways: those that were procedurally mapped (flowcharted) as it appears on the left chart, and those that were schematic in form and comparable to the librarians, on the right.
While it might be expected that students who produced procedural instead of schematic maps would have more difficulty with an assigned search task, the opposite was the case: I just thought it was really confusing and I didn't know how to get forward or how to get back. (Stu-DH7:01) This confusion may have resulted from the fact that the students had not yet connected their schematic understanding of Internet level connectivity to specific actions at the level of the browser. Possibly making matters worse, was the fact that these students had little general research experience.
User Expectations Newcomers Expected Search Engines To Have "Live" Access To Entire WWW A significant number of people seemed to believe that search engines actually comb the entire Internet each time a query is issued. For example, when we asked the students to explain exactly what happens when a search engine is used to find information we typically received responses such as: It searches whole wide world, everywhere, everywhere, Australia, North Pole South Pole, everywhere. (Stu-KP-40:46) Of further interest, when we initially presented this finding at a Web-related conference, we observed surprised reactions in the audience, several of whom commented along the lines of:
I thought it searched the whole Internet too. That comes as a complete surprise! An interesting question is "what were these expectations based on?" Our hypothesis is that most information services people encounter in their daily activities, such as telephone white pages and directory assistance, are nearly comprehensive of the areas they represent. It is also possible that these people simply overestimated the capabilities of our existing technology. This finding, however, was unforeseen because as professional researchers we overlooked the fact that not everybody knows that even the largest search engines index only one third of the Internet. Furthermore, the update periods for these indices range from a week to a month, with the largest search engines typically being at the high end of these estimates. Although uncommon at the time when this research was conducted, we acknowledge that now meta search engines have increased coverage because they search across multiple search engines simultaneously. However, this approach still misses millions of web sites. The implications of these ideas are twofold- one that searches pertaining to current events could leave people empty handed, and second that coverage, as an important difference between search engines, could be overlooked. Newcomers And Librarians Initially Expected WWW To Be Credible During their earliest encounters with the Internet both librarians and students had high expectations regarding the overall credibility of the information they would find. For students, the high expectations still held because their exposure was limited and at the same time socially facilitated by their friends. However, for librarians the expectations had been readjusted due to encounters with non-institutional web sites which had not met their expectations. During the Internet history narratives students described passive experiences using the Internet. With the exception of one or two confusing and distressing personal attempts, most of the students' experiences were guided by the school librarian or their friends who were in the "driver's seat." Both the students' friends and the librarian were accessing the "channels" on AOL as well as the Electric Library. The following provides a typical example of the kinds of mediated computer experiences the students described: It was just me and Mr. Sanchez, I would just watch when someone else would do it [AOL] because I was interested in it. (Stu-DH-4:33) Each time it was because I had to do it for a project. I don't do it on my own.. and I had to have, you know, assistance.. these two times were with friends. (Stu-KP-2:18) Because these students' experiences with the Internet were limited to private information services and friends' guidance, it is not surprising that their overall credibility models of the Internet remained high. However, librarians had Internet experiences that brought them into contact with a much greater range of information sources that altered their credibility models of the Internet. All the Librarians gave such examples of encounters with unofficial web pages that led to changes in their credibility expectations: I thought it [Internet] was very valid, that changed when I saw my first web page, because I saw this is not an official Lego site this is just some guy whose been to the Legoland theme park and who is really interested in Lego's. [Lib-AT-33:50] Librarians realized that "Anyone can publish" and therefore the credibility of the Internet could not be assumed, but would have to be determined on a site by site basis. The criteria for these site by site credibility evaluations are discussed in the next section. Librarians Expected To Find Clear And Consistent Credibility Cues Librarians apply many traditional credibility criteria such as timeliness, branding, and overall appearance when they evaluate web sites. However, it turns out that link reliability and site navigability are additional criteria that significantly impacted their credibility assessments.
Site navigability in particular impacts information accessibility because web sites are an unfamiliar environment in which traditional navigation strategies are inadequately supported or not applicable: .. and they [PBS Smithsonian web sites] looked better, they were easier to browse, when you get into something that's really hard to look at on the screen it's so easy to just back up and get out of there, so how it looks visually, how things are arranged on the screen, what menus look like, if it's well organized, is it easy to get home, or back, it gives it more credibility. (Lib-SJ-6S:00) I'm influenced (in terms of credibility) a lot by what a page looks like graphically, visually, if things are spelled wrong, if there are bad links, broken links, I might just get out of there and go somewhere else. Librarians also expressed concern over the timeliness of material on the Internet. In fact, one of the librarians cited an example that could have had severe consequences for a patron planning to transport a volatile substance overseas. Thanks to the librarian, the patron did not rely exclusively on the regulations found at the official airline web site, and double-checked the information by phone with an airline representative who explained that the regulations had changed since the site was last updated. Similarly, one of the authors also nearly missed an important bus departure after having referred to the official, but in reality outdated travel schedule found on the Internet. Further complicating the matter is the fact that ISP's store frequently visited sites in a cache to reduce bandwidth costs, thus making it difficult to access certain sites for up-to-date information even when they have been properly updated on the Internet. Librarians Expected Friendly And Standard Tools Cataloguing schemes on Internet search services range from the unclear to the completely opaque and this has prevented librarians' from acquiring a level of expertise that matches their skill with library based indices and online catalogs. In Alta Vista they don't really choose to link to them [web sites], they just identify that they exist and then apply some sort of software thing that will find them.. way above my head I don't know where-how they do that. (Lib-CS-57:37) We librarians we understand how info is indexed, I don't know for sure how these are indexed, are they looking for words in the summary? Some of them must index every word in the whole document.. I'm not quite sure. (Lib-SJ-59:28) Because librarians do not understand search's cataloguing systems as well as they understand other kinds of databases they experience difficulties in their job as well. People come up to the [reference] desk saying I heard I can get this on the Internet, where is it? And then you say "have you ever used the Internet?" and they say “no,” but they want it right now, they want it right now, it's an uncomfortable thing as a librarian. Not only is the predicament complicated by the patrons' expectations of immediate access to information, but it also assumed that librarians' expertise generalizes to the domain of the Internet because there is Internet access in the library. Because librarians are uncertain of the organizational system, they also experience a lack of closure when searching: There's always that feeling when you get something, that ooh, is there something better you could've gotten? (Lib-SJ-29:03) Well I can see how like people get totally sucked in, I can waste time just sortve' clicking on useless- it can suck you're time away, I sometimes think I waste too much time doing it here [in search]. (Lib-RF-68:34)
The second example illustrates another outcome of librarians' uncertainties, namely the tendency to be distracted and explore useless links. Librarians coped with their uneasiness about search engines by Yahoo'ing more frequently. They expressed a degree of comfort with Yahoo because it seemed to follow a more familiar organizational system: I think about Yahoo as this very organized system and like there are books on the shelf and here's the music section and look there's William Christy. (Lib-RF-61:39) Despite the fact that Yahoo contains a much smaller slice of the overall Internet, librarians weigh the familiarity of its cataloguing system quite heavily. Newcomers Expected Search Interface To Be Responsive To Their Needs The examples in this section illustrate how unclear affordances can impede navigation for newcomers. For example, during the assigned search task the students did not know that some text boxes must be clicked on with the mouse to permit text entry. When the computer answered students' keystrokes with "blip.. blip.." students next strategy was to seek "help." However, “help” proved to be not so helpful: I just want to know how to type in and no clue, it's not helping at all, it [help screen] gives you 10 more things.. This is stressful right now.. I'm mad, I'm angry, I don't know where to go, I don't know what to do. It just pushes me away. Why can't it just say "this is what you do"? (Stu-KP~280) I don't like reading all this [help], I just want them to tell me what I'm supposed to do. If I could talk to it and say "well, what am I supposed to do next?" it would be a lot easier. (Stu-C~175) Rather than offerring a context appropriate solution, the search program offers the student the entire instruction manual for its operations. The problem is twofold, because not only does the computer break social conventions for interaction (Norman, 1992) by answering the students with unhelpful "blips," but it also overwhelms them with a “thick” user manual lacking obvious and contextually defined entry points. The next example stems from misleading affordances in the results screen. One such problem was brought to our attention by one of the librarians in the context of teaching an Internet class to the public. The typical scenario presented was that students enter their queries but get sidetracked from the class activity because of alluring advertisements containing a "click here" message: They think it's [click here] part of the search engine.. and of course they're gonna "click here” because it's this graphical thing that flashes and it's on the subject they searched. Interviewer: How quickly do people realize that this is not a piece of information from the search engine? They don't, unless you tell them.. at least not right away. (Lib-AT-38:28) The unexpected layout and unclear boundaries between advertisements and the search application lead students unknowingly down the wrong “path.” The magnitude of the problem is even greater in view of the fact that these same people commonly think that searches take place “live” across the entire Internet, and therefore come to believe that the “click here” indicator will take them to a listing of the most relevant sites. Furthermore, as a result of such situations, the credibility of the search engine may also be compromised. A final example situated in the results screen illustrates how items in the actual data display lack self-evident codings indicating basic site characteristrics, such as access restrictions. On more than one occasion during the assigned search task students clicked inviting links on the data display only to find a password prompt facing them after the site had loaded. SUMMARY While neither experienced librarians nor novices showed specific evidence for spatial metaphors for the WWW, both interviews and cognitive maps indicated that many users follow procedural paths or routes in navigating the
web that are reminiscent if physical navigation. Consistent with this was an expectation of “information constancy”that is, that information discovered on one visit ought to continue to exist, unmodified across visits. Such paths seemed more prominent among librarians who had more organized experience with navigating information. With experience, cognitive maps moved from more mysterious, nebulous and fragmented to more crystallized and coherent with more reference point. A prominent finding in our interviews was that experienced librarians and novices alike have specific expectations about how search on the WWW works and of its information credibility. Moreover, these expectations are frequently violated, leading to some interesting modifications with experience. With respect to how the WWW works, many users, but especially novices, expected search engines to comb the entire Internet each time a new search was initiated. While in some cases, this expectation persisted across experience, librarians’ extensive experience with indices provided more realistic expectations. On the other hand, this experience with library-based indices also led to confusion. Since librarians are used to understanding the structure of catalog systems use, the relatively unstructured WWW was problematic. Also, with respect to information credibility, our interviews revealed changes with experience. Novices expected web sites to be high in credibility and rated them as such. More experienced librarians, on the other hand, had more conservative views of site credib ility, likely based on their experience that sites can be published by anyone. Similarly, librarians’ training and experience in assessing credibility of sources made them more attentive to potential cues to credibility in WWW material. Indeed, in many cases, the lack of such cues in the relatively cuedepleted WWW was frustrating. The current investigation was exploratory, and further research is required to understand the details of how expectations are formed and how they change. Moreover, the relationship between expectations and web usage behaviors and utility needs to be examined closely. Nonetheless, these results provide an initial picture of user expectations of the WWW and how they change with specific experiences in the information sciences.
REFERENCES Bercaw, S. L. (1997). Procedural versus declarative representation in cognitive maps. Unpublished manuscript. Kitchin, R. M. (1994). Cognitive maps: What are they and why study them? Journal of Environmental Psychology, 14, 1-19. Maurer, M. (1983). Development and validation of a measure of computer anxiety. Unpublished Master's Thesis. Ames: Iowa State University. Norman, D. (1992). Turn signals are the facial expressions of automobiles. Reading, Mass: Addison Wesley. Pirolli, P. & Card, S. (1995). Information Foraging in Information Access Environments. Proceedings of ACM CHI '95 Conference on Human Factors in Computing Systems. Ne w York: ACM Press. Wexelblat, A. & Maes, P. (1999). Footprints: History-Rich Tools for Information Foraging. Proceedings of ACM CHI '99 Conference on Human Factors in Computing Systems. New York: ACM Press.
