Evaluation of Ambient Displays Jennifer Mankoff1 and Anind Dey12 1
Electrical Engineering and Computer Science Department University of California at Berkeley Berkeley, CA 94720 2 Intel Research, Berkeley Intel Corporation Berkeley, CA 94720
[email protected],
[email protected]
Abstract. We are investigating a subset of notification displays we refer to as ambient displays. Ambient displays are peripheral, aesthetically pleasing displays of information which support awareness of some data. Our work focuses on design guidelines and evaluation methodologies for developing successful ambient displays. We call a display successful when it is acceptable to users (they want it in their environment), and it changes their awareness of or behavior with respect to some information.
1
Introduction
One of the biggest challenges facing ubiquitous computing in general, and ambient displays in particular, is evaluation. Two of the most effective techniques currently in use are formative ethnographies, and iterative, “living laboratories” in which applications are evolved over time as they are used by their creators. However, both of these techniques are costly and time consuming. Traditional techniques that have not yet been successfully applied to the ambient display domain include various modeling techniques such as GOMS and cognitive walkthroughs, automated usability evaluations and heuristic evaluation. Additionally, laws such as Fitts’ law and other guiding principles do not all translate into the domain of ambient displays. Our goal is to develop design guidelines and a corpus of evaluation techniques appropriate to ambient displays. We are in the process of surveying of existing GUI evaluation techniques in order to understand what fails and succeeds for ambient displays , and where it is possible to make appropriate modifications. To date, we have conducted two empirical evaluation of two ambient displays. We found that empirical techniques could be applied to ambient displays with little modification. We are also in the progress of exploring the use of heuristic evaluation for ambient displays. It appears that this will require a modification to Nielsen’s original heuristics, and we are in the midst of a survey to determine what modifications are appropriate.
2
1.1
Mankoff & Dey
Overview
We begin by providing background information on how researchers have evaluated existing ambient displays, and why evaluation of these systems is difficult. Next, we briefly discuss one of our empirical evaluations. This is followed by a description of our Heuristic Evaluation modification plans, and finally Conclusions and Author Bios.
2
Existing Ambient Display evaluations
The majority of ambient displays that have been published have not been evaluated, e.g. [12,4,3,6], or report only short evaluations with few details [7,2,8]. Some of the most extensive evaluation effort has been focused on determining exactly what users might want, using techniques such as surveys, interviews, and wizard of oz prototyping [9]. Finally, some displays have been shown in museum settings where they were used by hundreds of users but never tracked in detail [1,5,13]. In other words, some formative techniques such as interviews and wizard of oz prototyping and interviewing have been used successfully to determine potential areas of need. However, these techniques have not been verified completely since work has included the development, but not deployment or further evaluation of ambient displays. Additionally, since few such evaluations have been done, there are no definitive guidelines for good ambient display design, or for the situations in which ambient displays should be applied. However, the experience of several museum exhibits and some small empirical studies indicates that users are interested in and excited by ambient displays. Our goal is to contribute a better understanding of how to apply different evaluation techniques to ambient displays, and what design guidelines will lead to successful ambient displays. We plan to explore a range of different evaluation techniques, and to complete the design cycle by always putting our displays into real use in a context where we can monitor their effectiveness. To date, we have combined formative surveys with an empirical evaluation, and begun an analysis of the role of heuristic evaluation for ambient displays. We describe the empirical analysis next.
3
Empirical Evaluation
We hypothesized that ambient displays might be useful to undergraduate students working in windowless, basement Berkeley Computer Science labs. To determine what kinds of information would be useful to users, a preliminary survey was conducted, asking participants to rank in order information they would like to see in an ambient display. Some of the choices included how dark it is outside, the weather, the population of the computer labs, the network load on the login servers, when a bus was next scheduled to arrive, and sports statistics. The survey also queried the lab users with questions such as which bus lines they used
Evaluation of Ambient Displays
3
frequently and which sports teams they followed. Both the bus schedule and the amount of daylight ranked highly after the results of the survey were evaluated. There were some types of information that ranked even higher, such as how full the labs were and which servers had the most traffic. However, we decided that these data sources would not make effective ambient displays, since it is data that only needs to be obtained once before entering the lab, and not while one is working. Based on this input, we designed two ambient displays, the Bus Mobile and the Daylight Display (See Figure 1. The Bus Mobile is a mobile that hangs from the ceiling and has six bus numbers hanging from it, each representing a distinct route, which adjust their height according to the closeness of a bus. The Daylight Display consists of a floor lamp that adjusts its brightness according to the sunrise and sunset times, indicating the amount of outdoor light. Our hypothesis was that these displays would change user’s behavior or awareness with respect to each data source. For the effect of the Bus Mobile, we hypothesized that there would be an increase in the number of students who leave the lab within an optimal time frame before a given bus arrival, because students would like being able to catch a bus immediately instead of waiting at the stop for one to arrive. For the Daylight Display, we hypothesized that there would be an increase in the number of students leaving the lab in the time leading up to sunset. It is for safety reasons that we believe users of the lab would be interested in leaving before dark.
(a)
(b)
Fig. 1. (a) The BusMobile. (b) The Daylight Display.
4
Mankoff & Dey
The study was conducted in three phases. The first phase occurred in the week before the displays were to be deployed in the labs, when a questionnaire was distributed to the users of three labs, which they could fill out at their own leisure. During the second phase, users were exposed to our displays. Each display was located in one lab, while a third lab had no display and was used as a control. The questions on the survey covered various topics that could give us an indication of whether or not the information about bus arrivals and outdoor light levels affected a subject’s behavior. The third phase was identical to the first, in which no displays were present. This allowed us to gather more complete control data about changes that may have occurred over the 6 week study period that are not attributable to our displays. Although we are still analyzing the data, it appears that among user’s who were interested in the respective data displayed, over 80% found the Bus Mobile useful, while only 30% found the Daylight Display useful. Additionally, we found that the longer a student spent in the labs, the less interested a student was in the daylight display. In terms of actual behavior, our preliminary analysis found a small statistical effect of the Bus Display on how close to bus arrival times users left the lab. An important next step here is to learn more about why the daylight display was less appealing. We believe it may be that the display is only important to users as day turns to dusk and night, while users leaving at any time are interested in the bus display. This remains to be verified.
4
Heuristic Evaluation
We approached our empirical evaluation through direct application because there was sufficient evidence in the literature that this evaluation technique can be applied to ambient displays given the right set of hypothesis. We were not so certain about the applicability of Heuristic Evaluation [11]. We believe Heuristic Evaluation is important to ambient displays because of its potential to provide quick, inexpensive feedback about the possible problems with a specific display [10]. Because heuristic evaluation is based on a set of 10 heuristics that were designed for a particular domain (GUIs), we chose to examine the appropriateness of these heuristics to ambient displays. We found that with slight modifications and a few additions and deletions, Nielsen’s heuristics were a reasonable starting place. We decided to survey experts in the design and analysis of user interfaces, and experts in the design of ambient displays, asking them to rate the modified set of heuristics we had created and add any they thought we were missing. We are currently in the process of analyzing the resulting surveys (6 have been returned to us). Our goal is to revise our heuristics based on these results, and then ask a group of experienced user interface evaluators to evaluate the two displays described above using our heuristics. We will ask a separate group of evaluators
Evaluation of Ambient Displays
5
to conduct an evaluation using Nielsen’s original heuristics and compare the results. We hope that this will provide us with a good basis for understanding how to approach heuristic evaluation for ambient displays, and specifically which heuristics to use. Additionally, these heuristics, once validated, represent a potential seed for ambient display design guidelines.
5
Conclusions
Although this work is clearly very much in progress, that seems appropriate to a workshop setting. Clearly there is a lot of work left to be done in understanding where and when ambient displays should be applied, and how they should be built. Similarly, there is a lot yet to be discovered about appropriate evaluation techniques that might help to answer those questions.
6
Author Bios
Unfortunately, since Anind Dey is running the doctoral consortium, only Jen Mankoff will be able to attend the workshop, but we hope you’ll consider this submission in any case. Jennifer Mankoff, an assistant professor at the University of California at Berkeley, recently completed her PhD at the Georgia Institute of Technology. Her research focuses on supporting non-traditional forms of input and output used in ubiquitous computing and by people with special needs. Dr. Mankoff is an active member of the related research communities, and has served on the program and conference committees for the ACM assistive technology (ASSETS) and user interface systems and technology (UIST) conferences over the past few years. Anind Dey, a researcher at Intel Research, Berkeley, and an adjunct professor at UC Berkeley, also recently graduated from Georgia Institute of Technology. He is an expert in the area of context-aware computing. His current work ranges from sensor-driven applications to infrastructure problems, and includes projects on privacy, precision agriculture, end user programming tools for sensor networks, and ambient displays. Dr. Dey has participated in the conference and program committees for UIST and Ubicomp over the past few years.
7
Acknowledgments
Many students contributed to this work, including Lisa Chan, Morgan Ames, Chinmayi Bettadapur, Steven Chan, Julie Kientz, Gary Hsieh, Scott Lederer, Scott Carter, and Wai-Ling Ho-Ching
6
Mankoff & Dey
References 1. C. Chafe and G. Niemeyer. Oxygen flute. Installation by Artists, San Jose Museum of Art, October 13-2001 – June, 2002. 2. A. Chang, B. Resner, B. Koerner, H. Wang, and H. Ishii. Lumitouch: An emotional communication device. In Extended Abstracts of Conference on Human Factors in Computing Systems (CHI ’01), pp. 313–314, Seattle, Washington, March 2001. ACM Press. 3. L. Halln¨ as and J. Redstr¨ om. Slow technology; designing for reflection. Personal and Ubiquitous Computing, 5(3):201–212, 2001. 4. J. M. Heiner, S. E. Hudson, and K. Tanaka. The information percolator: Ambient information display in a decorative object. In ACM Symposium on User Interface Software and Technology, pp. 141–148. ACM Press, November 1999. 5. H. Ishii, S. Ren, and P. Frei. Pinwheels: Visualizing information flow in architectural space. In Extended Abstracts of Conference on Human Factors in Computing Systems (CHI ’01), pp. 111–112, Seattle, Washington, March 2001. ACM Press. 6. H. Ishii and B. Ullmer. Tangible bits: Towards seamless interfaces between people, bits and atoms. In Proceedings of the Conference on Human Factors in Computing systems. ACM Press, March 1997. 7. H. Ishii, C. Wisneski, S. Brave, A. Dahley, M. Gorbet, B. Ullmer, and P. Yarin. Ambient displays: Turning architectural space into an interface between people and digital information. In Proceedings of the First International Workshop on Cooperative Buildings (CoBuild ’98),. Springer-Verlag, February 1998. 8. E. Mynatt, M. Back, R. Want, M. Baer, and J. B. Ellis. Designing Audio Aura. In Proceedings of the Conference on Human Factors in Computing Systems (CHI ’98). ACM Press, 1998. 9. E. Mynatt, J. Rowan, S. Craighill, and A. Jacobs. Digital family portraits: Providing peace of mind for extended family members. In Proceedings of the 2001 ACM Conference on Human Factors in Computing Systems (CHI 2001), pp. 333–340, Seattle, Washington, March 2001. ACM Press. 10. J. Nielsen. Finding usability problems through heuristic evaluation. In Proceedings of Conference on Human Factors in Computing Systems (CHI’92), pp. 373–380. ACM Press, 1992. 11. J. Nielsen and R. Molich. Heuristic evaluation of user interfaces. In Proceedings of Conference on Human Factors in Computing Systems (CHI’90), pp. 249–256. ACM Press, 1990. 12. M. Weiser and J. S. Brown. Designing calm technology. Internet: http://www. ubiq.com/weiser/calmtech/calmtech.htm, 1995. 13. T. White and D. Small. An interactive poetic garden. In Extended Abstracts of Conference on Human Factors in Computing Systems (CHI ’98). ACM Press, 1998.
