Interaction With Co-located Devices Using Relative

¨ t Mu ¨nchen Ludwig-Maximilians-Universita Department “Institut f¨ ur Informatik” Lehr- und Forschungseinheit Medieninformatik Prof. Dr. Heinrich Hußmann

Diplomarbeit

Interaction With Co-located Devices Using Relative Spatial Information Sara Streng [email protected]

Bearbeitungszeitraum: Betreuer: Verantw. Hochschullehrer:

1. 11. 2006 bis 30. 4. 2007 Prof. Hans-Werner Gellersen Prof. Albrecht Schmidt

Abstract This project deals with interaction across multiple devices in mobile computing environments, and will investigate the use of spatial information to support such interactions. Therefore a variety of application scenarios is considered, which involve spatially co-located devices, objects, or people. The general research aim is to understand how such interactions can be supported with spatial information, that is how the devices are positioned and oriented with respect to each other. The first task is a thorough research of existing literature to identify prior work on mobile interactions that involve the use of spatial information. Based on this survey, a classification of spatial concepts is developed. Afterwards the focus is shifted to actions, which are triggered under spatial conditions. To simplify the development of such applications, a toolkit is designed and implemented. Finally, a new interaction technique is introduced, which is also based on spatial criteria. It involves small windows called gateways, which appear on the edge of the screen to represent a co-located device. By clicking on the gateway or dropping a file on it, the user is able to interact with the co-located device in an intuitive and easy way. The gateway interaction technique was evaluated in a user study. To ensure that the results are not limited by the capabilities of any particular sensor system, the Wizard of Oz approach was used to simulate the positioning system.

Zusammenfassung Ziel der Diplomarbeit ist es herauszufinden, wie die relative Position und Orientierung benachbarter Ger¨ate dazu verwendet werden kann, um die Interaktion zwischen diesen Ger¨ate zu vereinfachen. Dazu wird eine Vielfalt von Anwendungen betrachtet, die r¨aumliche Beziehungen benachbarter Ger¨ate, Objekte und Personen einbeziehen. Zun¨achst wird der aktuelle Forschungsstand auf diesem Gebiet analysiert und in einer ¨ Ubersicht dargestellt. Außerdem werden die Applikationen an Hand verschiedener Kriterien klassifiziert. Im Anschluß daran wird die automatische Ausf¨ uhrung von Aktionen bei Vorliegen bestimmter r¨aumlichen Bedingungen genauer betrachtet. Um die Entwicklung solcher Mechanismen zu vereinfachen, wurde ein Toolkit entwickelt. Schließlich wird eine neue Interaktionstechnik vorgestellt, die ebenfalls auf r¨aumlichen Kriterien beruht. Bei dieser Technik tauchen unter bestimmten Bedinungen kleine Fenster (sogenannte Gateways) am Bildschirmrand auf, die jeweils ein Ger¨at in der unmittelbaren Umgebung repr¨asentieren. Indem Nutzer auf das Gateway klicken oder eine Datei daraufziehen, k¨onnen sie auf einfache und intuitive Weise mit dem Ger¨at interagieren. Die Gateway Interaktionstechnik wurde in einer Benutzerstudie evaluiert. Um sicherzustellen, dass die Ergebnisse nicht von einem bestimmten Positionierungssystem beeinflußt werden, wurde der Wizard of Oz Ansatz gew¨ahlt, um die Positionsinformation zu erzeugen.

Definition of the Project The project deals with co-located devices, for example PDAs or laptops that are located in the same room or building. The question is how interaction between such devices can be enhanced by using information about their relative position and orientation. The first step in the project is an extensive literature research to find out about existing interaction techniques for mobile devices. Although a wide range of applications is examined, the focus is on those, which consider the relative position of at least two devices. A device’s location can either stand for the location of a person or a physical object or be self-dependent. Based on this survey a classification of spatial concepts for interaction is developed. There are several aspects to be explored, amongst others: 1. What is being located? (humans, devices, physical objects) 2. What are characteristics of the position information used? (e.g. whether orientation is considered or not, accuracy needed, ...) 3. How is the position information used? (a) Is the position presented to the user explicitly, implicitly or not at all? (b) What is the direction of the information flow? (user to application, application to user or both) (c) Which interaction techniques are used? (e.g. drag and drop, physically attaching devices to each other, pointing, ...) (d) What is the range of the spatial interaction? (within arm’s reach, within line of sight, ...) (e) Does the user interact with the physical world (e.g. tangible user interface), with an electronic device or both (computer-augmented world)? 4. What is the benefit for the user? (e.g. navigation, browsing and searching, object manipulation, ...) In a second step one interesting technique is selected and studied in more detail. In order to do this, an appropriate application is implemented for a user study. To be independent of any particular positioning system the Wizard of Oz approach is used to simulate spatial sensor data. The study is supposed to shed light on the usability of the selected interaction technique.

Ich erkl¨are hiermit, dass ich die vorliegende Arbeit selbstst¨andig angefertigt, alle Zitate als solche kenntlich gemacht sowie alle benutzten Quellen und Hilfsmittel angegeben habe. M¨ unchen, March 30, 2007 .........................................

Contents 1 Introduction 1.1 Motivation . . . . . . . . . . . . . . . 1.2 Aspects of Using Position Information 1.3 Objective . . . . . . . . . . . . . . . . 1.4 Focus . . . . . . . . . . . . . . . . . . 1.5 Outline . . . . . . . . . . . . . . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

2 Related Work 2.1 Surveys . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.1 Survey of Context-Aware Applications . . . . . . . . 2.1.2 Technological Progress in Location-Awareness . . . . 2.2 Classifications . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.1 Classification of Context-Aware Applications . . . . 2.2.2 Spatial Interaction Ranges . . . . . . . . . . . . . . . 2.2.3 Taxonomy of Location Systems . . . . . . . . . . . . 2.3 Toolkits and Frameworks . . . . . . . . . . . . . . . . . . . 2.3.1 Context Toolkit . . . . . . . . . . . . . . . . . . . . . 2.3.2 Prototyping Tool . . . . . . . . . . . . . . . . . . . . 2.3.3 Framework for Creating Context-Aware Applications 2.4 Wizard of Oz . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.1 Origin and Background . . . . . . . . . . . . . . . . 2.4.2 Simulating Positioning Systems . . . . . . . . . . . . 3 Applications Using Relative Spatial Information 3.1 Spatial Presentation . . . . . . . . . . . . . . . . . . . . 3.1.1 Spatialized Widgets . . . . . . . . . . . . . . . . 3.1.2 Map Alternatives . . . . . . . . . . . . . . . . . . 3.1.3 Spatial Presentation in Games . . . . . . . . . . 3.2 Location-Aware Applications . . . . . . . . . . . . . . . 3.2.1 All-Round Systems . . . . . . . . . . . . . . . . . 3.2.2 Spatially-Aware Displays . . . . . . . . . . . . . 3.2.3 Information Displays . . . . . . . . . . . . . . . . 3.2.4 Tourist Guides . . . . . . . . . . . . . . . . . . . 3.2.5 Teleporting . . . . . . . . . . . . . . . . . . . . . 3.2.6 Memory Aids . . . . . . . . . . . . . . . . . . . . 3.2.7 Other applications . . . . . . . . . . . . . . . . . 3.3 Position-Triggered Actions . . . . . . . . . . . . . . . . . 3.3.1 Games and Educational Applications . . . . . . . 3.3.2 Interaction between Co-located Devices . . . . . 3.3.3 Spatially Aware Local Communication . . . . . . 3.3.4 Physical Mobile Interaction Techniques . . . . . 3.4 Multi-Display Reaching . . . . . . . . . . . . . . . . . . 3.5 User Interface Migration . . . . . . . . . . . . . . . . . . 3.5.1 Presenting the Position of Possible Targets . . . 3.5.2 Spatial Interaction . . . . . . . . . . . . . . . . . 3.6 Motion-Tracked Handhelds . . . . . . . . . . . . . . . . 3.6.1 Windows to the Virtual World . . . . . . . . . . 3.6.2 Windows to the Physical World . . . . . . . . . . 3.6.3 Defining and Interacting with the Physical Space

. . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . .

. . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . .

. . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . .

. . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . .

. . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . .

. . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . .

. . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . .

. . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . .

. . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . .

. . . . .

1 1 1 2 3 4

. . . . . . . . . . . . . .

7 7 7 8 8 8 10 11 11 11 11 12 13 13 13

. . . . . . . . . . . . . . . . . . . . . . . . .

15 16 16 17 17 18 18 18 19 19 20 20 20 20 21 21 22 22 23 25 25 26 27 27 27 28 I

3.7 3.8

3.6.4 Handhelds in Tangible User Interface Systems . . . . . . . . . . . 3.6.5 Other Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tangible User Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.8.1 Use of Spatial Information . . . . . . . . . . . . . . . . . . . . . . 3.8.2 Presentation Forms . . . . . . . . . . . . . . . . . . . . . . . . . . 3.8.3 Accuracy of Position Information . . . . . . . . . . . . . . . . . . 3.8.4 Relative Position Between Humans, Devices or Physical Objects

4 Spatial Condition Toolkit 4.1 Purpose of the Toolkit . . . . . . . . . . . 4.2 Dissociation from Previous Work . . . . . 4.3 Different Uses of Spatial Information . . . 4.3.1 Direct Interaction with Positioning 4.3.2 Monitor . . . . . . . . . . . . . . . 4.3.3 Trigger Agent . . . . . . . . . . . . 4.4 Orientation, Location and Motion . . . . 4.5 Spatial Conditions . . . . . . . . . . . . . 4.5.1 Basic Conditions . . . . . . . . . . 4.5.2 Complex Conditions . . . . . . . . 4.6 A Typical Interaction Sequence . . . . . . 4.7 Adaptive vs. Dual Trigger Mode . . . . . 4.8 Implementation . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

5 Gateway Application 5.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 The Gateway Interaction Technique . . . . . . . . . . . . 5.3 Conditional and Scanning Mode . . . . . . . . . . . . . . 5.4 The Design . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4.1 Drawing the Right Level of Attention . . . . . . . 5.4.2 Button and Drag-and-Drop Funtionality . . . . . . 5.4.3 Avoiding Overlapping Gateways . . . . . . . . . . 5.5 Implementation . . . . . . . . . . . . . . . . . . . . . . . . 5.5.1 Package Overview . . . . . . . . . . . . . . . . . . 5.5.2 Using the SCT . . . . . . . . . . . . . . . . . . . . 5.5.3 Communication Between Gateway Application and 5.5.4 Calculating the Screen Coordinates . . . . . . . . . 6 User Study 6.1 Setup . . . . . . . . . . . . . . 6.1.1 Spatial Conditions . . . 6.1.2 Wizard of Oz . . . . . . 6.1.3 Staff . . . . . . . . . . . 6.1.4 Technical Requirements 6.2 Proceeding . . . . . . . . . . . 6.2.1 Introduction . . . . . . 6.2.2 Tasks . . . . . . . . . . 6.2.3 Scanning Mode . . . . . 6.3 Execution . . . . . . . . . . . . 6.4 Questionnaire . . . . . . . . . . 6.4.1 General Assessment . . II

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . .

. . . . . . . .

28 28 29 29 30 30 32 33

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

35 35 36 36 37 37 38 39 40 41 44 46 48 49

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Wizard . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

51 52 52 54 54 54 55 55 56 56 57 57 60

. . . . . . . . . . . .

63 63 63 64 65 65 66 66 66 67 67 68 68

. . . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

6.5

6.4.2 Design and Disturbing Factor . . 6.4.3 Button vs. Drag-and-Drop . . . 6.4.4 Conditional and Scanning Mode 6.4.5 Comments . . . . . . . . . . . . . Summary . . . . . . . . . . . . . . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

70 71 72 74 74

7 Conclusion 77 7.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 7.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 Appendices

81

A SCT Class Diagram

81

B Class Diagram of the Gateway Application with Wizard

82

C User Study Questionnaire

83

III

IV

1 1.1

Introduction Motivation

In our daily lives we constantly make use of position information. When people talk to each other, they are usually in the same room, close-by, and facing each other. If one wants to use a device, e.g. a computer, it has to be within reach. Thus, when interacting with daily objects, the interaction partner is often identified by spatial criteria, such as proximity, line of vision, and so forth. In other words, by looking at the spatial relationship of people, objects, and electronic devices, you can tell which interaction is possible or even likely. Traditional computing systems, however, mostly disregard position information. Desktops are not aware of their location because it is fixed. The interaction between computers is based on names and IP addresses, not on their locations. In the early days of computing this made perfectly sense because the primary objective was telecommunication - and for communication over great distances relative spatial information is irrelevant. In local settings, however, choosing interaction partners by spatial criteria is much more intuitive than using identities. Besides showing a map of co-located devices to the user, the relative position of devices with respect to the user can also be used to narrow down the number of communication partners. In particular, this applies to ubiquitous computing, which is the current trend of computational power moving into daily appliances. As computers intermingle with physical objects, their use should be as natural and intuitive as the use of physical objects. There should be no need for the user to find out a computer’s name or IP address in order to set up the connection manually. Instead, the system could be aware of the position of users, other devices and objects and act based on this information. A technological trend that pushed the use of position information was mobile phones. With the global positioning system (GPS) the outdoor position of mobile devices can be determined quite precisely. The idea of location based services (LBS) emerged, which consider the current location of mobile devices to provide enhanced services. If a mobile phone is equipped with a GPS receiver, it can e.g. guide the user to a desired destination or list nearby restaurants.

1.2

Aspects of Using Position Information

As an introduction to the application field that will be discussed in the survey, the principle of using spatial information is sketched in figure 1.1. Thereby some key aspects, which will play a role later in this work, are addressed beforehand. The spatial relation of at least two humans, electronic devices, physical objects or any combination of these, is measured. There are different technologies that can be used for the positioning. Most systems are based on signals, such as GPS, cell phone triangulation, WLAN, RFID, active badges, and so on. Alternatively video cameras can be used to capture pictures, which are later analyzed in order to retrieve the position of the recorded people or objects. Depending on the technology as well as the application’s requirements, the information can include any combination of location, orientation, and motion. Thus, in this thesis the terms “position” and “location” have different meanings whereas in common speech they are often used for the same thing. As both terms are used throughout the thesis, it is important to understand the difference. “Position”, which is equivalent to “position information”, is defined as umbrella term for any information, which can be provided by a positioning system. The “location” is only part of this information. It is usually determined by coordinates that specify where something is located. In chapter 4.4 the terms and definitions are discussed in more detail. 1

1

INTRODUCTION

Figure 1.1: The Use of Position Position information can be one-, two- or three-dimensional. Furthermore the accuracy varies. As positioning technologies were thoroughly discussed in previous work, e.g. [23], an examination of technologies is not included in this document. Once the position is acquired there are several ways to use it. Some applications explicitly show the spatial relations to the user, e.g. in a map. This can help the user to orient, navigate or find something. Other applications use the position to adapt the presentation, e.g. by showing a list of all accessible printers ordered by their distance. This is called implicit presentation because the user can draw conclusions from what he sees on the screen to at least part of the position information. Finally the spatial information can be used as input parameter without any effect on the presentation. For example, an application automatically synchronizes devices, which are placed next to each other. In order to start the synchronization, the user holds his PDA next to his notebook. In this case the position information is processed by the system, but not presented to the user. In the end, all this is done to create some kind of benefit for the user. If the position is shown to the user, the benefit is usually advanced navigation, browsing or searching. Otherwise the spatial information is probably used to start services or manipulate objects.

1.3

Objective

After briefly introducing the topic, the objective of this work is now specified. The first contribution is a survey and classification of applications that use relative spatial information. This is intended as orientation to anyone who is interested in location-enhanced systems. As the field is very broad and the applications are diverse, the survey is rather complex. Therefore various aspects of location-enhanced applications are selected and used to classify the applications. Following the survey, the focus is set to one part of it, which is applications that involve the automatic execution of actions under spatial conditions. An easy example is automatically synchronizing two (mobile) computers when they are physically next to each other. There are several applications that use such a mechanism, yet there is no uniform concept so far. To make up for this, a detailed concept is designed and a toolkit for such applications is created. The automatic triggering of actions is an interaction technique as such. Instead of dealing with menus or buttons, you e.g. simply place devices next to each other in order to initiate an action. However, this interaction technique is not suitable if triggering an 2

1.4

Focus

unrequested action may disturb the user. If the devices are placed next to each other unintentionally, the synchronization process might be started by mistake. This can be annoying when the process uses system resources that are needed for other purposes. Therefore an alternative interaction technique is presented as last contribution of this thesis. Instead of executing the service directly, the options are presented to the user in an unobtrusive way. This is done in the form of a small window, which appears on the edge of the screen. The window represents a gateway to the device, which the service is running on. Instead of triggering the service, the spatial conditions determine when the gateway windows appear. Then the user can manually execute the service with one mouse click or drag-and-drop if needed. Otherwise he can ignore the gateway without being significantly disturbed. In short, the gateway interaction technique takes advantage of the concept of triggering actions under spatial conditions while minimizing the risk of disturbing the user’s workflow. In addition to allowing users to intuitively interact with co-located devices, the gateway interaction technique can also be used to scan the environment and search for local services. Thereby the user issues a command in order to view gateway windows for all services in the room. As only little display space is claimed by the gateways, they are in particular suitable for small, mobile devices. The gateway interaction technique was evaluated in a user study. In particular it was examined whether it is attractive as well as easy and intuitive to use. The study proved that the gateway interaction technique provides easy access to local services. Participants liked both the gateways and services. They would also use the application if an appropriate mobile device is available.

1.4

Focus

As the application field examined in the survey is very broad and there are transitions to other computing areas, it should be clear where the line is drawn. The emphasis is on relative spatial relations as opposed to absolute positioning (e.g. GPS). For the interaction between co-located devices the distance between two devices is more relevant than global coordinates. However, absolute positioning systems cannot be completely disregarded because the relative position can always be derived from absolute statements. Furthermore we are not looking at the position and orientation of a single user or object but the relative position and orientation of multiple users or objects. For example we are not covering hand gestures or eye tracking, where the position and orientation of one human being is used as input for an application. For the same reason we are not interested in spatial input devices, such as 3D mice, because they only consider position changes, but not the relative position to other objects. Although in the survey various applications are discussed, the primary interest is in new interaction techniques, which are based on spatial criteria. Why this is such a promising application field is explained in the following paragraphs. Back in the days where no technology existed, people had no contact to remote places. The first change came with the postal service. Through this, communication over greater distances became possible. Because of the large number of potential destinations, addresses were needed to identify the communication partner. Later, with the spread of telephones the whole world became connected. Suddenly everybody could communicate with people at various places even without knowing their exact location. Soon after that, information technology (IT) made it possible to interact with remote devices. To manage the worldwide communication, it was necessary to introduce phone numbers and addresses. As the technology was designed for global networks, it is rather complex. IP addresses became the de facto standard for computer-based networks, but handling them is neither 3

1

INTRODUCTION

very convenient nor intuitive. Furthermore the access to computers needs to be restricted to prevent fraudulent use. Therefore complex mechanisms are needed, such as firewalls, authentication, encryption, and so forth. Because of all this, the interaction between computers is very difficult. The majority of people does not even have the essential background knowledge to set up a connection between two computers, printers or other devices in their own house. While all the hassle might be acceptable if one wants to connect to the other side of the world, it seems ridiculous if a user is sitting in front of his laptop, with a network printer next to him and he can’t manage to print a document. In other words, using address-based communication is a necessary effort for global interaction, but it is not suitable for local interaction. At least, the user should not be forced to deal with it, even if the underlying technology is still IP-based. Instead the number of potential targets can easily be narrowed down by spatial criteria. That way users can interact with local devices in an intuitive way, such as approaching an object or pointing at it. Furthermore the security issue is easier to handle in local environments. In office buildings or private households there are easy and effective arrangements like keys, which make sure only authorized people get access to restricted areas. In short, the difficult technology, which is intended and needed for long-distance connections, seems inappropriate for local environments. New interaction techniques, that consider the spatial relation of co-located devices, could be much more intuitive and easier to use. As with the increasing number of electronic devices, the communication between co-located devices becomes more and more prevalent, this is a very interesting and important issue.

1.5

Outline

After explaining the motivation and focus of this thesis, the structure is now outlined. In the next chapter related research work is presented. This includes surveys and classifications of location- and context-aware computing, toolkits, frameworks as well as Wizard of Oz techniques. Chapter 3 provides a detailed survey of applications, which use relative spatial information. This includes the presentation of position information, location-aware applications, interaction that is triggered when certain spatial conditions are fulfilled, multi-display reaching techniques, user interface migration and motion-tracked handhelds. Furthermore the field of tangible user interfaces is shortly brought up, but not discussed in detail as it is a quite separate field. In the end these applications are classified according to different criteria, e.g. the required accuracy of the positioning information, how the information is used, or in which way it is presented to the user. The wide range of applications examined in this survey is then narrowed down in chapter 4. The focus of attention is brought to applications, which automatically trigger actions when certain spatial conditions are fulfilled. For these applications a toolkit is created, the Spatial Condition Toolkit (SCT), which is based on a detailed concept of spatial conditions. The design of both is thoroughly explained. Furthermore the use of the toolkit is illustrated. Chapter 5 introduces the gateway interaction technique, which makes use of the SCT. Gateways are small windows that allow the user to interact with services in the environment. The advantage is that a service is not automatically executed when the spatial conditions are fulfilled. This may disturb the user if the service is not needed at that time. Instead a gateway window appears on the edge of the user’s screen to indicate that a local service can be used. The gateway’s position on the screen points to the direction which the according device is located. For example, if the printer is to the user’s left, the 4

1.5

Outline

gateway window is on the left side of the screen. In order to execute a service the user either has to click on the corresponding gateway or drag-and-drop a file onto the gateway window. In order to evaluate how intuitive the gateways are to use, a user study was conducted. For the study three services were implemented: a regular keyboard is used to enter text on small handhelds, slide shows run on a meeting display and files are sent to the printer. In order to execute one of these services, the user has to interact with the gateways. Chapter 6 explains the setup of the study, the spatial conditions under which the gateways appear as well as the technical requirements. Furthermore the proceeding and execution of the study are demonstrated. Finally the results of the study as well as the follow-up questionnaire are interpreted. In the last chapter the whole project is summarized and consequences are drawn. In addition the chapter gives an outlook to future work.

5

1

6

INTRODUCTION

2

Related Work

This chapter presents previous research that is relevant for this work. The first section addresses surveys of location- and context-aware applications. Second, classifications of such applications as well as taxonomies of location systems and spatial interaction ranges are discussed. While in a survey applications are only listed, classifications analyze the applications and assign them to different criteria. The third section presents toolkits and frameworks, which were developed for location-enhanced applications. Finally Wizard of Oz techniques are discussed. This is relevant as the Wizard of Oz approach was used in a study, which was conducted to evaluate the gateway interaction technique. Both the system and the user study will be addressed later on in the thesis. At first the origin and background of Wizard of Oz is explained. Then the challenge of simulating sensor-based systems, such as positioning systems, is analyzed.

2.1

Surveys

Although to this day there is no survey that focuses on the use of relative spatial information, there was a considerable amount of research done on context-aware applications. As most of these applications consider position information, they are relevant for this work. In the following subsections different surveys of context-aware applications are presented. Furthermore a discussion on the technological progress in location-aware applications is summarized. 2.1.1

Survey of Context-Aware Applications

Korkea-aho [30] made a survey of context-aware applications. As position is an important component of context information, a good portion of this domain can be considered as location-aware. At first, features and characteristics of the term “context” are defined. Then existing applications are grouped by application areas. This section covers office and meeting tools, tourist guides, fieldwork tools, memory aids and frameworks. The last section of the paper responds to context-awareness in ad-hoc networks. Chen and Kotz present a survey of context-aware mobile computing research. At first the paper [9] gives insight on technologies that can be used to equip handhelds with a wireless connection. Afterwards the term “context” is defined and a survey of context-aware applications is presented. Each application is described in a few sentences. Furthermore the authors distinguish active context, which influences the behavior of an application, from passive context, which is relevant but does not affect an application’s behavior (e.g. a map). For each application the active and passive context types are determined. Two applications (the office assistant and the adaptive GSM phone and PDA) mainly use other context types. All other applications are location-aware and will be discussed later in this work in more detail. These are: • call forwarding • teleporting (mapping user interfaces to the resource nearest to the user) • conference assistant • tour guides, other guide systems and augmented reality • field work • location-aware information delivery 7

2

RELATED WORK

• people and object pager • shopping assistant • mobisaic web browser, which reacts to potentially changing context information, such as the current location in a wireless network As both surveys were made in the year 2000, they are not up to date anymore. Also they do not point out similarities or compare the applications in any way. Since the list of applications has significantly grown over the years, simply itemizing all applications would be confusing. Thus more recent surveys should be more structured. 2.1.2

Technological Progress in Location-Awareness

Want and Schilit [63] illustrate the technological progress of location-aware computing. From a historical point of view they discuss the influence of Active Badge, an electronic badge which is carried by people to inform the system about their location, as well as ParcTab, GPS and short-range radio (e.g. Bluetooth). Furthermore the paper presents electronic equipment, that features location-awareness, such as GPS watches, Bluetoothenabled cell phones, PDAs and other handhelds.

2.2

Classifications

As for the surveys the majority of classifications was done for context-aware applications. In the following section different classification criteria that are also relevant for locationaware applications, are presented. Moreover a taxonomy of location systems is outlined as well as a layered software-engineering model, which is based on this taxonomy. Finally several divisions of the physical space into spatial interaction ranges are discussed. These ranges are typically concentric circles of different sizes, which are centered around the user. Such divisions are interesting as many applications adapt to the distance between the user and the device. An ambient display can e.g. turn into an information display when the user is close enough to read the information. 2.2.1

Classification of Context-Aware Applications

Schilit et al. [54] define four categories of context-aware applications. As all examples involve position information, the classification is highly relevant for this work. The first category is Proximate Selection. This means that in a graphical user interface, which allows the user to select objects in the physical environment, close objects are emphasized. An example is a list of people, which includes the distance to the user. The closer a person’s location, the bigger is the corresponding list item. Second is Automatic Contextual Reconfiguration, which describes the adaption of the system configuration depending on the context. For example when multiple people are in the same meeting room, collaborative services are automatically enabled. The third category is Contextual Information and Commands, which is information that is selected according to the user’s context. The underlying assumption is that depending on the user’s situation and location their actions can be predicted. For example in a kitchen people do other things then when they are at the office. Finally actions can be automatically triggered when a certain situation occurs. This is referred to as Context-triggered Actions. We will talk about this in detail later in this work. 8

2.2

Classifications

Figure 2.1: Context-Aware Categories These four categories can be classified according to two dimensions as shown in table 2.1. The first dimension examines whether the application is getting information or executing a command. The second one looks whether the action is triggered manually or automatically.

information command

manual proximate selection & contextual information contextual commands

automatic automatic contextual reconfiguration context-triggered actions

Table 2.1: Context-Aware Software Dimensions Based on the classification from Schilit and another one done by Pascoe [39], Dey and Abowd [15] proposed another categorization. The three categories of this categorization are 1. presentation of information and services to the user 2. automatic execution of a service 3. tagging of context to information for later retrieval In figure 2.1 some context-aware applications are assigned to these three categories (P, E, T is short for Presentation, Execution and Tagging). Furthermore the involved context types are specified (Activity, Identity, Location and Time). Another attempt to classify context-aware applications was done by Brown et al. [8]. Before presenting their ideas, they point out that “(...) it would be good if we could classify applications into neat and separate compartments, but the world is not like that. Thus context-aware applications blur into other kinds of applications (...)”1 . They suggest a division between continuous and discrete applications. In continuous applications the information that is shown to the user changes constantly, e.g. in a navigation system. 1

[8], page 3

9

2

RELATED WORK

Discrete applications show different information in different situations, but once a specific context is entered, the information stays the same. For example in a museum information is presented on a handheld for each exhibit once the visitor is close enough. 2.2.2

Spatial Interaction Ranges

Subramanian [57] presents a classification scheme for spatial object manipulation techniques. Although the focus is on three-dimensional input, which is the selection, positioning and deforming of 3D objects, the classification provides some aspects, which are interesting for interaction techniques in general. The paper discusses three issues. At first it is argued that users should be able to perform tasks without conscious attention. This is referred to as “naturalness and transparency” of user interfaces. Second and most relevant for this work is the division of the user’s surrounding into three regions: 1. The Personal Space is within arm’s reach of the user (about 2 m). 2. The Action Space is the space of a person’s public actions in which he can move, speak, throw, toss, and so on (about 30 m). 3. The Vista Space is everything beyond, limited by the visual horizon. The last issue of the classification is the degree of freedom of input and output devices, which is the least interesting point for this work. As we will see later, the division of space in different interaction ranges is used in various applications, e.g. in chapter 3.2.2 of this paper (Spatially-Aware Displays). The concept presented here was adopted from Cutting and Vishton [11]. Another segmentation was done by Stappers et al. [55]. They propose three different ranges: 1. The Small Range in which people can perform manipulations with their hands and fingers. 2. The Middle Range in which people can use their hands, arms and posture to grasp objects and make gestures. 3. The Large Range which provides an overview and feeling for the presence in the environment. The probably first approach in this vein was done by Hall [20] in 1963. He investigated people’s appreciation and use of personal space and defines four distances which apply to a typical Northern American: 1. intimate (less than 18 inches) 2. personal (up to 4 feet) 3. social (up to 12 feet) 4. public (more than 12 feet) For each situation there is a distance we find appropriate. If the actual distance is two small, we will automatically back off and the other way around. 10

2.3

2.2.3

Toolkits and Frameworks

Taxonomy of Location Systems

Hightower and Borriello [23] thoroughly analyzed all kinds of location systems and developed a taxonomy that helps understanding differences and criteria. They distinguish physical position information (typically coordinates) from symbolic information (e.g. “in the kitchen”) as well as absolute from relative position. Furthermore they respond to accuracy, costs, limitations and recognition mechanisms. Afterwards they present a survey of existing location systems and point out research directions. Based on this taxonomy the Location Stack [24] was developed, a layered softwareengineering model for location similar to the OSI model. The goal is to create one standard architecture instead of having individual solutions for each system. The Location Stack is based on five design principles that were extracted from previous studies. These are: 1. There are fundamental measurement types, which can be classified as distance, angle, proximity, asserted position, or non-geometric features. 2. There are standard ways to combine measurements. For example, in order to locate an object, one can combine the asserted position of the sensors with the distance. 3. There are standard object relation queries (e.g. proximity). 4. Uncertainty must be preserved. This means applications, which are using the location information, should keep in mind that sometimes measurement uncertainty is very high. 5. Applications usually adapt or react to activities. Finally the model is applied to two existing ubiquitous computing system to stress its practicability.

2.3

Toolkits and Frameworks

In this section existing toolkits and frameworks for location-aware applications are outlined. First the Context Toolkit is presented, a fundamental work for all context-aware applications. Next the functionality of the Topiary prototyping tool for location-enhanced applications is introduced. Third, a framework for context-aware applications is outlined, which is based on the Stick-e Notes application. 2.3.1

Context Toolkit

Dey and Abowd [53] have built the Context Toolkit, which simplifies the development of context-aware applications. They introduced context widgets, which mediate between the user and the environment. The purpose of these widgets is providing a uniform interface and thus hiding the underlying distributed sensor architecture. The context information provided by the widgets can be aggregated and interpreted. Thus the toolkit provides useful abstractions for high-level applications. 2.3.2

Prototyping Tool

Topiary [33] is a prototyping tool for location-enhanced applications. Using this tool designers can create active maps, which model the location of people, objects and places. On top of these maps, storyboards can be created. Therefore pages are sketched, which correspond to screenshots that appear on a mobile device. Links represent transitions between these pages. The transitions are either explicitly triggered by user interaction 11

2

RELATED WORK

Figure 2.2: Topiary’s Storyboard Workspace (e.g. when the user clicks on a button) or implicitly when the scenario associated with the link occurs. Figure 2.2 shows how the Topiary Storyboard interface looks like. On the left side there are different scenarios that were created beforehand. These scenarios can be dragged onto the large area on the right side. The large white squares are pages, that are shown on the mobile device. On the bottom there is a toolbar, which contains a pencil, eraser and other tools. Using the pencil, the designer can draw arrows that lead from one page to another. The arrows represent transitions. Blue transitions are explicit links, which originate from a GUI element, such as the OK button of the “Nearest Friends” page. Green transitions are implicit links. They are tied to situations or rather conditions that trigger the transition. For example, the arrow at the top is linked to the situation “Anyone moves near to Bob”. This means, if the map page is shown on the mobile device and someone moves near to Bob, the “Nearest Friends” page is shown. Finally the storyboards can be run on a mobile device using the Wizard of Oz approach to simulate the position information. For a detailed discussion on Wizard of Oz see chapter 2.4. 2.3.3

Framework for Creating Context-Aware Applications

Based on the Stick-e Notes application, which will be discussed in chapter 3.2.6, Brown [7] developed a framework for applications that present information to the user depending on the context. Stick-e Notes are electronic equivalents to Post-It notes, which incorporate context information. The information contained in the note is enclosed in Stick-e Documents. The supporting software consists of four components: 1. With SEPREPARE authors can create the Stick-e Documents. 12

2.4

Wizard of Oz

2. SEMANAGE deals with the management and broadcasting of the notes. 3. SETRIGGER causes the documents to be triggered when the required context is matched. 4. SESHOW stores the notes and presents them to the user. The goal of this framework is making the creation of such applications as easy as creating a web page.

2.4

Wizard of Oz

To be independent of any particular position system, the Wizard of Oz approach is used to enter the participants’ position information in the user study. This section explains the origin and background of Wizard of Oz experiments. Furthermore different methodologies for simulating a positioning system are described and compared. 2.4.1

Origin and Background

The idea of Wizard of Oz experiments was shaped by John F. Kelley [29]. He developed a natural language computer application, that processed unconstrained English language. Computer-naive experimental participants entered text on a keyboard to manage their calendars. The natural language recognition system exceeded the recognition rates of today’s complex systems by far and did not require any training. The participants were given the expression that they are interacting with a program that understands English as well as other humans. However, the answers were given by an experimenter, the ’Wizard’. The name of this methodology comes from the children’s book ’The Wonderful Wizard of Oz”’, in which a little man pretends to be a powerful wizard but really only pulls levers behind a curtain. Today the Wizard of Oz methodology is common practice in various fields, such as human factors, linguistics or experimental psychology. Furthermore it is used in iterative design processes as one type of high-fidelity prototyping. Like other prototyping techniques it allows developers to first evaluate the design and get feedback before the underlying logic is actually built. That way poor design can be corrected early and the effort of changing the implementation is minimized. It is also suitable for human computer interaction researchers to find out about human needs and preferences without having to implement a whole system. 2.4.2

Simulating Positioning Systems

There are a few additional challenges when it comes to sensor-based applications, such as positioning systems. In addition to the user’s explicit input, e.g. pressing a button, there is also implicit input created by sensors. First, this leads to more complex interaction sequences. Second the implicit input has to be controlled by the wizard. For example, in location-enhanced applications the change of a user’s position can lead to an action. Consequently the wizard has to keep track of the user’s position and make sure actions are triggered at the right moment. This is an additional task for the experimenter, which can lead to a significantly higher work load. There are two ways to manage this. One way is to manually trigger an action when the corresponding criteria are met. For example, every time a user is close enough to a display, the wizard presses a button and specific information is shown on the display. The other way is to constantly update the user’s position on a map, letting the application decide when to trigger an action. 13

2

RELATED WORK

Dow et al. [17] set up a study to compare several techniques. In the experiment some participants were using an application that played audio content depending on the user’s position. Other participants were in charge of the application and thus represented the wizards. The wizards had the choice of either interacting with a map or pressing buttons to pick the best audio content for the situation. On the map there were red cycles representing the location of audio segments and icons representing the users. When the wizard moved the user icon near the red cycles the corresponding audio content was played on the user’s PDA. The wizard operators felt that the buttons were much easier to understand and control than the map. However, if there is too much unstructured content the strategy does not work. For example, if the wizard has to choose between dozens of audio files, it might be easier to use the map and thereby let the system decide which file to play. In other words, the more complex the rules, the more suitable is the map solution. Furthermore, if the position is in some way presented to the user, the only solution is updating the user’s position on the map. For example, if a tourist guide only shows information about nearby points of interest, a wizard can manually trigger the presentation of the corresponding content. If, however, the user also sees a map of his environment, the position information has to be entered on a map.

14

Figure 3.1: Application Overview

3

Applications Using Relative Spatial Information

This chapter provides an overview of applications that make use of relative spatial information. Since position information can be applied in many different areas and ways, the number and variety of such applications is great. The survey is ordered by application areas. However, the separation is often not clear. Many examples would fit to several sections. Therefore there are cross-references where appropriate. In the first subsection the presentation of position information is examined. Different forms of spatial presentation are introduced, such as maps, compasses, 3D visualizations, lists and textual information. Furthermore their use is illustrated with an example. Spatial presentation is used by many applications for different reasons throughout chapter 3. Instead of listing all these applications in the first subsection, the topic is only introduced at this point. In the classification later in chapter 3.8.2 the spatial presentation is reconsidered and the applications are assigned to the different presentation forms. The second subsection presents several location- and context-aware applications. As location is an important part of context information, many context-aware applications are also location-aware and thus relevant for this survey. This includes spatially-aware displays, information displays, (tourist) guides, teleporting, and memory aids. The third subsection covers applications, which involve some form of action that is triggered when certain spatial criteria are fulfilled. Furthermore there are several multidisplay reaching techniques as well as user interface migration techniques that involve spatial information. Afterwards different forms of motion-tracked handhelds are analyzed. The last application type is tangible user interfaces. However, as this is quite a separate field, it is not discussed in detail. Subsequent to the enumeration, the applications are classified by different criteria, which are discussed in chapter 3.8 15

3

APPLICATIONS USING RELATIVE SPATIAL INFORMATION

3.1

Spatial Presentation

One of the most common utilization of position information is presenting spatial relations to the user in a descriptive and appropriate way. That way users can identify communication partners by their location. In local workspaces this is much more intuitive than using names, IP addresses and alike. Other benefits of spatial representation are providing information as well as helping the user to search, browse, orient or navigate. At first spatialized widgets are introduced and applied to different application scenarios. Using these widgets, spatial information can be included in user interfaces, e.g. in a map or as text labels. In the next subsection alternative solutions to classical maps are discussed. Finally the idea of using spatial representation in games is illustrated with an example.

3.1.1

Spatialized Widgets

Kortuem et al. [31] examined the use of spatial relations in user interfaces. They proposed a set of spatialized widgets to support interaction between mobile devices. Spatialized widgets are visual interaction elements, which adapt to spatial context information. Some widgets are considered as explicit, others as implicit. Explicit widgets visualize spatial relations. Typical examples are Spatialized Maps, Spatialized Compasses and Spatialized Labels. A map shows positions from a top-down view. A compass is a map where important relations are highlighted (e.g. the direction) while others are ignored (typically the distance). Spatialized Labels describe spatial relations in words, e.g. “directly behind you”. Implicit spatial widgets use the position information in a way that is not immediately obvious to the user. The only widget presented by Kortuem et al. is the Spatialized List, which is sorted by the items’ distances. In principle any presentation form can be implicitly spatially-aware, if the representation of an object adapts to the object’s position. For example, icons that represent close devices are bigger or otherwise highlighted. Although such presentation forms are not necessarily Spatialized Widgets, they are still implicitly spatially-aware. Within the Relate project several applications were developed that make use of spatial widgets. The Relate system proposed by Hazas et al. [22] provides fine-grained position information of co-located mobile devices. The devices are equipped with USB dongles that use ultrasound for peer-to-peer sensing. Thus, there is no firm infrastructure needed in the environment. The first application discussed is an awareness tool, which is basically a map that visualizes the position of nearby devices. This tool can help users to identify each other. In a meeting scenario e.g., several people are sitting around a table, each with a laptop in front of him. With the awareness tool they can immediately check who the other attendants are and find out more about them without disturbing the meeting. On top of the awareness tool the file transfer application was prototyped, which allows for intuitive interaction with nearby devices. The involved devices need to be connected and equipped with Relate dongles. The user interface consists of the awareness tool view explained earlier and a regular file browser. By dropping a file onto an icon on the map, the copying process is initialized. The clue is that the target computer is not identified by its name or IP address, but by the relative position to other devices in the network. The last example application that makes use of spatial widgets is spatially-aware service browsing. Here the objective is to discover services that are available in the current environment. An example is a list of services that is sorted by spatial criteria [31]. 16

3.1

Spatial Presentation

Figure 3.2: ARIS Iconic Interface [5]

3.1.2

Map Alternatives

Maps are the most common way to present an overview of a spatial region. However, they are restricted to a top-down view and therefore disregard the vertical dimension. As this can be a problem, sometimes other solutions are needed. One is to pull down the walls to their backsides. That way one can also see vertical objects, e.g. wall displays. Figure 3.2 demonstrates how this looks like, using the ARIS Iconic Interface as example. ARIS is used for application migration, which will be discussed in chapter 3.5. Using this interface users can select the target devices, which the application should be running on. The white rectangle in the center would be identical in a map. The gray rectangles represent the walls. On the top there are two plasma displays, on the right side there is a door. All this information would be hardly visible on a map. The big arrow indicates the current location and orientation in the workspace. Another alternative is representing the spatial information in 3D. Three-dimensional maps are common in many application areas, yet rarely used in this context. The Interactive 3D BreakAway Map, for example, provides a virtual view of buildings and allows the user to navigate them [10]. 3D representation also proved to be an effective tool for selecting devices in local workspaces, as we will see later (chapter 3.5).

3.1.3

Spatial Presentation in Games

Presenting spatial information can not only be used to select communication partners, but also to provide information. This is useful in many applications, for example in games. Unmasking Mister X [2] is a game, which tries to bridge the gap between virtual and physical environment. Each player wears a sensor device and a head-mounted display (HMD) or PDA. The team tries to identify one of the players as Mister X. On their HMD or PDA they see the sensor data of Mister X. By comparing this data to the behavior of their team mates, they try to unmask the mole. Part of the sensor data is spatial information, such as position and movement of the player. Since this information is presented to the user, this game can be categorized as spatial presentation. 17

3

APPLICATIONS USING RELATIVE SPATIAL INFORMATION

3.2

Location-Aware Applications

Position is an important component of context information - next to time, temperature, nearby people and more. Dey defines context as “any information that can be used to characterize the situation of an entity, where an entity can be a person, place, or physical or computational object”2 . He also discusses various other definitions in [15]. Contextaware applications adapt their functionality to the current context. As location is one of the most respected context types, many context-aware applications are location-aware and thus are relevant for this survey. Since the beginning of the 1990s context-aware applications were developed in various projects. Typical applications areas are information displays, tourist guides, teleporting and memory aids. Some research projects have a clear focus and thus can be assigned to one of these areas. In other projects, such as Active Badge or ParcTab, various applications were built, which makes it hard to classify them. The next subsection provides an overview by introducing the bigger projects. Subsequently the different application areas are presented. 3.2.1

All-Round Systems

One of the first location-aware applications was the Active Badge [62], an electronic device, which is worn by staff members. The badge transmits infrared signals to sensors placed around the building. That way their location is captured and location-based services can be provided. The initial idea was to route phone calls to the device, which is closest to the user. Over the years other applications were built as well. Also among the first context-aware projects was ParcTab [61]. The system is based on a wireless palm-sized mobile computer, which is operated with three buttons and a touch sensitive screen. Among various applications there are also some context-aware ones. Most of them are presenting position information. To mention only some examples, the ParcTab can be used to find nearby devices or people, who are also carrying a ParcTab. Another application is presenting information about the room the user is in or using the handheld as universal remote control. With the Sentient Computing System Addlesee et al. [1] created a system that reacts to the context and manner in which it is used. For this a model of the real world is needed. The according data is created by sensors. The system provides a wide range of applications. With Follow-Me, services are ubiquitously available to users. As the user moves around, interfaces of applications in use move to the nearest appropriate device. This is similar to the call forwarding application, which was developed in the Active Badge project, but includes other applications as well. Another example is spatial browsing, where the environment’s state is displayed to the user. That way you can find people, look at their current environment, find out whether they are busy, request a notification when a colleague leaves a room and so on. This work would also fit to chapter 3.1, spatial presentation. 3.2.2

Spatially-Aware Displays

Spatially-aware displays react to position information. Usually the content shown on the screen depends on the number and proximity of nearby people. For example Streitz et al. [56] present Hello.Wall, a location-dependent ambient display, that adjusts to the proximity of people walking by. Ambient displays (as opposed to common displays on PCs, notebooks, PDAs, etc.) are designed to display peripheral information without demanding 2

18

[53], page 1

3.2

Location-Aware Applications

Figure 3.3: Close and Intimate Interaction with the MirrorSpace [49] the users’ full attention. They reflect physical activities, the nature of movements and sound, todo-lists, deadlines etc. In the case of Hello.Wall there are three different zones of interaction: ambient, notification and cell interaction zone. The ambient mode is active when there are no people nearby. In this mode light patterns are shown. If a person is moving into the notification zone, the display changes from stand-by mode to notification mode. Depending on the application the information shown here can be either personal (and thus vary from user to user) or general (be the same for each user). If the user wants to see more details, she can move closer until she is in the cell interaction zone. Using a PDA-like device she can now interact with the display or store information on it. Another more playful idea is MirrorSpace, a combination of mirror and display [49]. In the center of the screen there’s a camera, which captures a live video stream and shares it with other MirrorSpaces. Thus, if two people are standing in front of two screens at different locations, they can see a superposed image of themselves on top of the other person (see figure 3.3). In addition each MirrorSpace is equipped with proximity sensors that measure the distance to objects in front of it. Objects that are far away from the screen are blurred, while close objects are shown in full detail. The idea is based on the study of a man’s appreciation and use of personal space by Hall (see chapter 2.2.2). 3.2.3

Information Displays

Spatially-aware displays react to the presence of people in front of it. Information displays are slightly different. They consider more than just their immediate environment. The In/Out Board [53], for example, shows which colleagues are currently in their office and which are not. The Conference Assistant [16] also displays relevant information depending on the user’s location, but is intended to run on wearable computers. The user can retrieve information about the presenter, presentation material or the timetable of events taking place in the room. 3.2.4

Tourist Guides

The commercially most successful context-aware applications are tourist guides. One purpose of guides is acting as navigation aid, e.g. car navigation systems. In addition they can provide information of nearby landmarks and allow the user to store the information on a handheld. Guides were developed in various research projects, e.g. the Georgia Tech Cyberguide [34], the GUIDE developed at the University of Lancaster [13] and [14], or the Smart Sight from Carnegie Mellon University [64]. 19

3

APPLICATIONS USING RELATIVE SPATIAL INFORMATION

There are also other forms of guides, e.g. the shopping assistant [3]. Shoppers carry a device, which - among other features - guides them through the store and provides information about products. 3.2.5

Teleporting

The term “teleporting” describes location-aware applications that dynamically map user interfaces to the appropriate device, which is closest to the user. The first application in this vain was routing calls to the nearest phone. Another example is Follow-Me. Both were already mentioned in chapter 3.2.1, which covered systems like the Active Badge or the Sentient Computing System. Other systems, where applications follow mobile users within buildings, are described in [21] or [43]. Similar is the People and Object Pager [8] that routes messages for visitors to the person closest to the visitor. 3.2.6

Memory Aids

Another context-aware application is recording notes, assigning them to the context in which they are taken and showing them when the same context appears again. This is useful because people remember things based on the context, that means when, where and under what circumstances they happened. Stick-e Notes [38], for example, are electronic versions of the yellow Post-It notes. Each note includes the context in which it was taken. Later when the same context occurs again, the note is automatically shown to the user. The Remembrance Agent [48] displays one-line summaries of textual information such as notes, emails or other documents, that might be relevant in the user’s current context. Forget-Me-Not [32] helps finding lost documents or remembering peoples’ names. Finally context-aware fieldwork tools, such as the rhino identification tool [41], the archaeological assistant tool [52] and the giraffe observation tool [40], allow users to take locationdependent notes. The notes can later be retrieved using a map that shows where they were taken. Yet another application that is similar in its purpose is comMotion, which can be used for location-aware information delivery. The system uses GPS position sensing and gradually learns about the locations that play a role in the user’s daily life. Reminder messages are attached to locations that are delivered when the user enters the according location. That way it e.g. reminds the user of his shopping list when he is near a grocery store. Although additional information can be shown on a display, the basic reminder function works with text-to-speech-synthesis [35]. 3.2.7

Other applications

An application, which does not fit to any of these groups is the Mobisaic Web Browser [60]. It relies on dynamic URLs, which depend on the current location of a mobile device within the wireless network. Active documents respond to dynamic URLs by automatically adapting their content. That way a web server can, for example, consider the user’s location as well as the current time in order to calculate the best bus route.

3.3

Position-Triggered Actions

Position information can be used to trigger interaction between co-located devices. In particular proximity is often used to initiate some form of action. Another example is the remote control metaphor where pointing with one device to another one triggers a service. The concept of triggering actions under spatial criteria is first illustrated with easy examples such as games and educational applications. The second subsection explains 20

3.3

Position-Triggered Actions

how spatial criteria allow for an intuitive interaction between co-located devices. Furthermore a service architecture is presented, which perfectly fits to such applications. Finally the results of an experimental comparison of the physical mobile interaction techniques touching, pointing and scanning are discussed. 3.3.1

Games and Educational Applications

A very descriptive example for proximity-triggered interaction is the Pirates game [18]. Multiple players, who are equipped with handheld computers, play throughout a physical environment. Thereby the players’ physical position is used to trigger game events. For example, only when a user is close to a room (which represents an island in the Pirates game) he is able to explore the island or engage in battle. The goal is to enhance the game experience as well as the social interaction. Next to proximity, pointing can play a role in games and alike. Geney [12], for example, is a collaborative educational application for children. Using Palm handhelds, which represent fish ponds, they explore genetic concepts, learn how to solve problems and cooperate with fellow classmates. By pointing to another handheld the students can exchange fish and thus arrange to mate fish in the same pond. 3.3.2

Interaction between Co-located Devices

Another example of interaction that is triggered under certain spatial conditions is Proximal Interactions. Rekimoto et al. [46] proposed a new way of establishing wireless connections among nearby devices with intuitive actions. Nowadays one has to know the IP address or name of the target device in order to setup a connection. Maintaining this information is tedious and entering it on a handheld can be cumbersome as well. Therefore a near field channel is used to exchange information that is necessary for setting up the connection. Depending on the technology the user has to point to a target device (infrared) or bring the two devices within proximity of each other (RFID) to initialize the transfer of the connection information. The connection is then automatically established and from then on the standard channel (e.g. WLAN) is used to transfer the data. The two steps are illustrated in figure 3.4. Rekimoto et al. also respond to security and authentication in ubiquitous computing environments. Traditional security policies, such as firewalls, are as inadequate as authentication by entering passwords. Instead physical and social barriers should be used to prohibit trespassers from using personal computers. In other words it is easier to lock the office door instead of locking every single device in the room. Next to setting up wireless connections between devices, other application scenarios are discussed. First is the mobile presentation, which allows people to start slideshows from a PDA on a presentation screen. By placing the PDA on the tag reader of the screen, the handheld is identified and accepted in the local network. Furthermore the screen is recognized as target destination and thus any selected presentation file automatically appears on the presentation screen. In addition the PDA can be used as remote control for the presentation. This allows the user to go the next slide by pressing a button on the handheld. Another scenario is the universal remote control. Having one remote control for each device is very frustrating, even today. In the future, however, the number of devices and objects equipped with computational power will steadily increase. Therefore the idea of using one PDA as universal remote control for all these devices is very interesting. Using the mobile device, the user simply points to the device he wants to control. Via infrared the connection information and controlling commands are transmitted. After the connection 21

3

APPLICATIONS USING RELATIVE SPATIAL INFORMATION

Figure 3.4: Proximal Interaction Model [46] is established, the controlling is done via WLAN so that the user can leave the infrared range and move around freely. 3.3.3

Spatially Aware Local Communication

Hupfeld and Beigl [27] introduce a model for local communication, the RAUM system. Local communication is defined as communication between partners located in the same physical environment. The authors argue that in ubiquitous computing the spatial relationship of communication partners is far more significant than their identities. If you want to talk to someone, you are standing in front of him, not turning your back to him. The same concept is transferred to inter-device communication. As for humans the spatial relationship defines an area of interest. Thus in spatially-aware communication the position information is used to select communication partners. For this type of communication the RAUM-architecture is built, an ISO/OSI conform layered model. RAUM stands for Relation of Application objects for communicating Ubicomp event Messages. The German word “Raum” means “space” as well as “room”. Based on this architecture several applications are described. The context-aware doorplate displays the name of the room, the name of people working there as well as events taking place. This application would also fit to chapter 3.1, spatial presentation. Furthermore they built the AwareKeyboard, a wireless computer keyboard, which can be used for text input on handhelds. Another example is an ElectronicManual for devices such as video recorders, which can be downloaded onto a handheld. Thereby the handheld can be used to read instructions for use and control the device. The precondition is that both the handheld and the device are within a certain spatial order. In order to use the services described above, the handheld needs to be in front of the device [4]. 3.3.4

Physical Mobile Interaction Techniques

Rukzio et al. [50] thoroughly analyzed the three physical mobile interaction techniques touching, pointing and scanning. The term physical mobile interaction is defined as communication between users, mobile devices and physical objects, which can include com22

3.4

Multi-Display Reaching

putational power or not. Touching means physically attaching the mobile device to the smart device one wants to interact with. Pointing is similar but there is no need for the user to be in reach of the smart device. Only line of sight is required. For scanning, the user enters a command on his mobile device to initiate the scanning process. Afterwards a list of available devices is presented and the user chooses one of them - again by entering a command on his mobile device. The first two interaction techniques - touching and pointing - involve spatial conditions. Touching is equivalent to triggering an action when the distance is closer than about 10 cm. Pointing requires line of sight. In order to compare the three interaction techniques at first a web-based questionnaire was used. The results were later verified by evaluating low and high fidelity prototypes. According to the results there are three criteria that users tend to consider when choosing the preferred mobile interaction technique: location, activity and motivation. The first and most important criterion is the location. Touching and pointing are the preferred techniques if the user is close to the device. Both techniques are quick and intuitive as they are based on everyday interactions. Furthermore they both avoid as much input on the mobile device as possible. Touching is more error resistant, secure and non-ambiguous but might require physical effort. If there is little motivation to make any physical effort, pointing is preferable if the device is not within the user’s reach but in line of sight. Scanning is indirect and thus more complex to use. Also the process of scanning and afterwards selecting the intended device is more time-consuming. Furthermore the cognitive effort is higher and there is more input required on the mobile device. Therefore scanning is only used when the user can not see the device or its tags. There are two more criteria that can disarrange this order of preference. Usually the motivation of approaching a device is very low. However, it can be increased for a number of reasons. • Security: If the device plays a critical role in the user’s life, he is more willing to make a physical effort in order to use the most secure interaction technique, which is touching. • Speed: If the selection process must be performed quickly, one might be motivated to move into the area of line of sight or reach (e.g. light control). • Intuitiveness: Especially the elderly who are not familiar with mobile phones might be more motivated to make a physical effort in order to avoid the necessary input to the mobile device. However, there is a maximum physical effort. This means the reasons mentioned so far are only practical if the physical effort is not too high. Finally the level of activity has an effect on the motivation to approach a device. In a lying or sitting position the barrier of physical effort is much higher than in a standing position. Rukzio et al. [51] also created the Physical User Interface Profile (PUIP), a UML based modeling tool for “physical mobile interaction”. The PUIP is designed to support the design, implementation, classification, comparison and evaluation of such interaction techniques. The modeling covers physical constraints, such as the distance between user and physical object, device features (e.g. screen size) as well as the temporal and social context.

3.4

Multi-Display Reaching

In our daily lives people often combine physical tools to achieve a task. With electronic devices, however, this is often not that easy. Nowadays, even the simple task of trans23

3

APPLICATIONS USING RELATIVE SPATIAL INFORMATION

Figure 3.5: Radar View ferring a digital object from one device to another is not intuitive at all. Several new interaction techniques were developed in order to change this. One is pick-and-drop [45], an interaction technique that allows convenient data transfer from one display to another. To pick up a digital object (e.g. an icon on the desktop) from one screen, the user taps on it with a stylus. The stylus is then moved to another screen and the icon is dropped by tapping on a target region. Similar to pick-and-drop is hyperdragging [47]. Instead of a stylus this interaction technique works with a mouse. By dragging an object over the edge of a screen it will jump to the physical environment, e.g. a table or wall. In the same way the object can be dragged to another display and dropped there. Thus not only interaction between multiple devices is supported but also interaction with the physical world. The user can for example add digital information to objects in the environment and retrieve it later on. To realize this, a camera-based recognition system and visual markers are used. There are some other interaction techniques for moving digital objects from one device to another. Most are pretty similar to pick-and-drop or involve a gesture, e.g. moving one’s hand and arm as if you are throw the digital object or use a slingshot. A different concept is the Radar View, which is illustrated in figure 3.5. The picture sketches a scenario where a PDA is located to the left of a tablet PC. The user is currently working on the tablet PC. As the user clicks on the icon (represented by a filled rectangle), a miniature representation of the devices in the environment appears, which looks like a map in form of a circle. The source, e.g. the icon, is in the center of the view. The possible targets are placed within the circle. Having an overview of the co-located devices and their position, the user can now drag the source to any of these devices. The advantage is that the Radar View works for remote target displays as well, whereas pick-and-drop and alike are only useful in adjacencies. Furthermore the Radar View can also be used to place objects in the physical environment. For example the scenario of figure 3.5 would also work if the PDA was replaced with a table. Nacenta et al. [37] compared all these different techniques based on the following characteristics: • topology (virtual or physical space, coupled, ...) • discrete destination space (whether the user cares about exact location) • range 24

3.5

ID based command line hierarchy

Spatial Presentation map (e.g. iCrafter) iconic interface (e.g. ARIS) 3D visualization

User Interface Migration

Spatial Interaction connecTables

Table 3.1: Different Ways of Controlling Application Migration • accuracy and resolution • feedback • input device • display and input area requirements (e.g. minimum size) • one-sided vs. two-sided (whether an action from recipient is required as well) • symmetric vs. asymmetric (moving objects in one or both directions) Besides the obvious advantage that the Radar View also works for remote targets, it was also more accurate and significantly faster than the other techniques.

3.5

User Interface Migration

A field of application that is in its intention very similar to multi-display reaching, is the migration of user interfaces. The objective is to migrate a user interface across different devices, e.g. from a laptop to a PDA or to a wall screen during run-time. This can involve multiple computing platforms. Also the whole application window can be migrated or only parts of it. For example it is possible to move a single toolbar onto a PDA while leaving the rest of the GUI on the desktop. Interesting in terms of spatial interaction is the question of how to control the migration. The easiest solution is using a command line in order to relocate an application. Thereby devices and applications are identified by name. As position information does not play any role in applications like this, they are not particularly interesting for this work. Other, more intuitive methods take the spatial relation of co-located devices into account. One way is presenting this information to the user, e.g. by showing an overview of the workspace in form of a two-dimensional map or 3D visualization. Finally there are interaction techniques which use spatial information as input source. For example, co-located connecTable displays act as one large display when they are physically attached to one another. An overview of the three different ways of controlling the migration of user interfaces is shown in table 3.1. The two latter approaches, which involve position information, are discussed in the following sections. 3.5.1

Presenting the Position of Possible Targets

A helpful method for controlling the migration of applications is presenting the spatial relationship of co-located devices to the user. In iCrafter [44] a map of the workspace is used to flexibly interact with services. Biehl et al. [5] propose ARIS (Application Relocater for Interactive Workspaces), an iconic interface that provides a miniature representation of the workspace. The advantage of this interface compared to a map was explained earlier in chapter 3.1.2. 25

3

APPLICATIONS USING RELATIVE SPATIAL INFORMATION

Figure 3.6: Screenshots of the Virtual Representation of a Real Migration [36]

Yet another form of spatial representation was proposed by Mass´o et al [36]. In their case study they used a 3D virtual reality system to represent the user’s real environment. In order to migrate a user interface, it first has to be described in UsiXML, an XMLcompliant USer Interface Description Language. In the window bar of such an interactive application there are additional buttons next to the usual close, minimize and maximize buttons. In particular, the migrate button (indicated with a capital “M”) can be used to “grab” the window in order to relocate it. As a result the GUI will follow the mouse cursor to the target device. The application which is subject to migration is regenerated to fit the target platform. Also, the user interface is adapted to fit the new screen size, color scheme, etc. Finally the application is launched on the target device. Figure 3.6 shows a sequence of screenshots that was taken from the virtual representation during the migration of an Instant Messaging application. In the foreground there is a Pocket PC, which is the target device. Initially the application is running on the laptop on the right. The window is grabbed (step 1), dragged to the pocket PC (step 2) and launched with an adapted resolution and look&feel (step 3).

3.5.2

Spatial Interaction

A different way of using position information is spatial interaction. Here the position information is directly involved in the interaction technique. For example, Tandler et al. [58] developed ConnecTable displays, which act as one large display when they are physically close to each other. This allows users to share their individual workspaces. A typical interaction technique is dragging digital objects over the edge of the screen onto another connecTable. Furthermore simultaneous views of an object can be created with a simple gesture. The views can then be shuffled to other displays. Each user can work on her view even after the connecTables are disconnected. A similar idea was realized by Hinckley et al. [25], who augmented tablet PCs with touch and tilt sensors. Therefore the devices recognize when they are bumped into another tablet PC and consequently act as one large display. If one of the tablets rests on the desk surface it acts as base tablet. This means, the tablet, which was moved by the user, annexes the base tablet’s screen. If both tablets are held in the user’s hands while they are bumped into each other, the two can share information. Thus depending on the position of the tablet PCs, different metaphors are used. In a way the prototypes are very similar to motion-tracked handhelds (chapter 3.6). 26

3.6

3.6

Motion-Tracked Handhelds

Motion-Tracked Handhelds

Various research projects examined how one can benefit from augmenting handheld devices with sensors. Typically proximity sensors, touch sensors and/or tilt sensors (accelerometers) are used. That way the devices are aware of their own position, orientation, or movement, sometimes even with respect to other devices or physical objects. As a result they can act as windows to larger virtual workspaces or to the physical world. Motiontracked handhelds can also be used to define virtual models of the physical world or interact with it. Furthermore they are also applied to Tangible User Interface Systems. In the next sections these projects are discussed in more detail. 3.6.1

Windows to the Virtual World

A typical example for motion-tracked displays is the Peephole Display [66], a handheld device that is used as a window on a large virtual workspace. By physically moving the display, the user can e.g. navigate through a large map or browse different parts of an image. Besides the typical panning and zooming there are more interaction techniques possible with Peephole Displays. Using one hand to pan and the other to draw, it is possible to draw continuous strokes on areas much wider than the screen (“draw-and-pan”). In the same way the user can extend the range of drag-and-drop (“drag-and-pan”). One can also switch between the detailed view and an overview to navigate through large information spaces [65]. 3.6.2

Windows to the Physical World

Another motion-tracked handheld is Fitzmaurice’s Chameleon prototype [19]. It emerged in one of the first projects dealing with relative position at all. The device is a combination of a 3D input controller and a video display. However, it is more than just a window on a large virtual world. The aim is to use Chameleon as a bridge between the digital and the physical world. As with the devices mentioned earlier the shown sector and level of detail can be controlled by the Chameleon device: moving the device to the left means shift to left or tilting means zooming. Fitzmaurice refers to this as the “Eye-in-hand metaphor”. In addition to tracking the user’s gestures and movements the device is aware of its own position and orientation with respect to the physical objects in the environment. That way it can act as an information lens. For example the user can stand in front of a map, which is pinned to a wall. By moving the device to a specific area he can request detailed weather information for that region. In that sense Chameleon could be described as window to the physical world, whereas the Peephole Display serves as window to the virtual world. Fitzmaurice mentions several scenarios in which such devices could be used. One vision is the portable surrogate office application, which offers remote access to the physical office environment. In the center of the room there is a 360◦ panoramic camera. Using the Chameleon device users can browse contents, such as white boards, books or calendars from anywhere in the world. In order to interact with these objects a spatial mapping to the physical world is needed. The user can then attach graphical notes or voice annotations by selecting objects with the cross-hair on the palmtop-unit. Indicator lights or touchsensitive LCD strips could be used to remind the user of the notes while he is working in his office. Another idea is the computer augmented library. The two tasks of querying and tracking down books are combined to one integrated process. The books and shelves emit semantic and navigational information. While the user walks through the aisles, those books that fulfill the search criteria are highlighted (again with indicator lights or LCD 27

3

APPLICATIONS USING RELATIVE SPATIAL INFORMATION

strips). By tapping on the touch-sensitive LCD strip the user can tell the search engine to focus on material with a similar content. Furthermore one can quickly have a look at the table of content of individual books or groups of books. All of the scenarios described by Fitzmaurice are only visions. It is not clear how they would be implemented and the proposed technologies might be out-of-date. 3.6.3

Defining and Interacting with the Physical Space

A development that goes even further is the iCam. Using position sensors in the environment this handheld device calculates its own position and orientation as well as the location of objects in its environment. It consists of a touch screen, a camera, orientation sensors, a location tag and various buttons and control elements. In addition, it disposes of a laser pointer that is used to select a physical object and calculate its relative position. There are basically two ways of using the iCam. First, one can define the physical space and place content in it. By tracing parts such as walls and doors a geometric model of the space is produced. Afterwards you can attach digital content, e.g. notes. Second one can scan the space with the camera and see the augmented digital content on top of the video stream. Applications like this, which overlay the real world with computer-generated information, are called Augmented Reality. A typical example application for the iCam is a mobile tour guide. In the first step points of interest are created and digital content is attached. In the second step tourists use the handheld to navigate through the space and retrieve the information. Furthermore the user can interact with the physical space, e.g. point to a light switch and press a virtual button. Compared to the devices and applications mentioned earlier in this section, the iCam is even more than a window to the physical world. It can be used to augment the physical world with text and pictures [42]. 3.6.4

Handhelds in Tangible User Interface Systems

The Active Lens is an arm-mounted flat panel display. It is part of the metaDESK, a tangible user interface developed by Ishii and Ullmer [28]. Figure 3.7 shows the components and their functionalities. The Active Lens is based on the metaphor of a jeweler’s magnifying lens. It is equipped with a magnetic-field position sensor in order to track its spatial position and orientation. Next to the Active Lens the metaDESK consists of a desk, which is a back-projected graphical surface, and a passive lens, an optically transparent surface through which the desk projects. Furthermore there are physical objects and instruments that can be placed on the desk. The purpose of the Active Lens is to display digital 3D information that is bound to physical objects. For example in a prototype described in [59] a physical model of the MIT’s Great Dome is placed on the desk. The system reacts to this by projecting a two-dimensional campus map onto the desk. Thereby the map is positioned correctly around the physical model. The Active Lens is used to show a navigable 3D model of the MIT campus buildings. 3.6.5

Other Features

Position sensors in handheld devices often only consider their own position and orientation. For example Hinckley et al. [26] used a palm-sized PC and augmented it with proximity sensors as well as touch and tilt sensors. Due to the sensors the prototype can detect when it is picked up and automatically power up. As for other handhelds tilting can be used to scroll the display. Furthermore the device recognizes in which way it is 28

3.7

Tangible User Interfaces

Figure 3.7: Hardware Architecture of the metaDESK [59] held and automatically switches between portrait and landscape mode. There are several applications like this, but as already mentioned in chapter 1.4, this is beyond the scope of this work.

3.7

Tangible User Interfaces

Contrary to Graphical User Interfaces (GUIs), which allow user interaction on a screen, Tangible User Interfaces (TUIs) emphasize physical interaction with physical objects. Tangible interaction includes touching, deforming and positioning of objects and thus allows for haptic feedback. We have discussed one example, the metaDESK earlier in chapter 3.6.3, because one the components is a motion-tracked handheld, which is aware of its position and orientation with respect to the desk. However, typically TUIs only consider the spatial relations of objects. An example illustrating this is the Illuminating Light [6]. Users directly arrange and manipulate physical objects on an augmented tabletop. These objects represent lasers, mirrors, lenses and other optical components. The system recognizes the position of these objects, calculates the behavior of the laser light and projects it onto the table. As tangible user interfaces are quite a separate field, we will abstain from a detailed analysis of TUIs. Interested readers are referred to [6], [28] and [59].

3.8

Classification

The applications that were presented in the last sections are now classified by different criteria. At first different ways of using the position information are examined. This includes spatial presentation, which can be either explicit or implicit, as well as locationaware actions and triggering. Second, different presentation forms are analyzed regarding the number of dimensions and accuracy shown. The third aspect is the accuracy of the position information that is required by an application. While some applications need very exact locations, others e.g. only consider whether a person is in a room or not. Finally applications are classified according to the type of entity, of which the position is sensed. Sometimes the relative position of human beings is of interest, sometimes physical objects or electronic devices are tracked, sometimes it is a combination of these. 29

3

APPLICATIONS USING RELATIVE SPATIAL INFORMATION

3.8.1

Use of Spatial Information

Spatial information can be used in different ways. It can be presented to the user (explicit presentation), affect what is presented to the user (implicit presentation), or the presentation is completely independent from the spatial information. In the latter case the position information can influence an action or it is used to trigger actions when certain spatial condition are fulfilled. If the position is explicitly presented to the user, he has to interpret the information himself. In the other cases the interpretation is done by the application. In figure 3.8 the previously discussed applications are grouped by these four categories. The according chapters, in which the applications were originally discussed, are listed in brackets. Explicit presentation means that the position information is shown to the user. The most common presentation forms, such as different kinds of maps, compasses or 3D visualization, were introduced in chapter 3.1. Showing the spatial relation of the user’s environment is often used in guides, navigation systems or games. Sometimes spatial presentation is also used to allow the user to select a communication partner or target platform by looking at the position of co-located devices. Examples were presented in chapter 3.4 (Multi-Display Reaching) and 3.5 (User Interface Migration). In other applications the position information is not shown to the user, but it affects what is presented to the user or how. This is called implicit presentation. For example a spatialized list contains items, which are ordered by their distance to the user. Other examples are spatially-aware displays or information displays, such as the In / Out Board. Yet other applications use the position information for purposes other than spatial representation. Location-aware actions produce different results depending on the position information. For example, phone calls can be routed to the device, which is nearest to the user’s current location. Similar applications were discussed in chapter 3.2.5 (Teleporting). Also part of this category are motion-tracked handhelds. Their position relative to other objects is often used to control what is shown on the handheld’s screen (see chapter 3.6). The last category is location-aware triggering. Memory aids are typical examples as recorded notes are usually shown when the same context - or location - is entered. Another example are the connecTables, which act as one display when they are attached to each other. This can be seen as service, which is triggered when the spatial condition “next to each other” is fulfilled. Similar to this behavior are some multi-display reaching techniques (chapter 3.4). Various other applications are mentioned in chapter 3.3 (Position-Triggered Actions). The presentation style is also an indication for the benefit for the user. The intention of presenting information is primarily providing a navigation aid or helping the user to search and browse. Otherwise the benefit is some kind of service or - in the case of TUIs - object manipulation. 3.8.2

Presentation Forms

As already discussed, there are different ways of representing position information: maps, compasses, 3D visualization, spatialized lists or labels. Each presentation form has its own characteristics. How the individual presentation form looks like is illustrated in figure 3.9. Figure 3.10 compares the accuracy of the provided information. In particular, it is analyzed how many dimensions are shown. Thereby location and motion (which is the change of location over time) is distinguished from orientation. Maps provide a top-down view and thus are two-dimensional. The location of an object is usually shown in form of icons. The orientation is often disregarded, but it can e.g. be indicated by arrows or - if the icon is not symmetric - by rotating the icon. 30

3.8

Classification

Figure 3.8: Ways of Using Spatial Information

Figure 3.9: Presentation Forms

31

3

APPLICATIONS USING RELATIVE SPATIAL INFORMATION

Figure 3.10: Accuracy of Presentation Forms Vertical objects, such as wall-displays, are hardly visible on regular maps. To find a remedy a map with pulled-down walls can be used. This adds the vertical dimension for objects that are placed along the walls. In the center of a room it is two-dimensional like a map. Compasses point to the direction in which other objects are located from the user’s point of view. They are radial and thus one-dimensional. As only the direction is indicated, it is hard to distinguish multiple devices, which are located in the same direction. Furthermore the orientation of other objects can not be represented. Although compasses point to an object, the orientation of this object is not clear. 3D visualizations show the user’s environment from one point of view. Therefore you can only see objects, which are located in one specific direction. If e.g. in a room two desks are located at opposing walls, it is not possible to show both of them in the same view. Also 3D visualizations are perspectively, which means it looks like a picture taken from the user’s point of view. Spatialized labels can contain any spatial information. However, as they are nongraphic, they are much less demonstrative than any of the other presentation forms. Finally, spatialized lists are sorted by one criterion, mostly the distance. 3.8.3

Accuracy of Position Information

Different applications require different accuracy regarding the position information. In order to draw a map or 3D visualization, the position information has to be very precise. Presence-aware applications, on the other hand, often just need to know whether the user is within a room or not (e.g. the In / Out Board). In figure 3.11 the previously discussed applications are ordered by the required accuracy of location. The same applies for motion, which is the changing of location over time. Very exact position information is usually needed for spatial presentation as well as interaction between devices in the same workspace, such as multi-display reaching techniques and user interface migration. Guides, teleporting and memory aids also work with medium accuracy and information displays are even more tolerant. Depending on the required accuracy different technologies are suitable. Information on technologies is not subject of this work, but can e.g. be found in [63]. Some applications incorporate the orientation. For example, pointing with one device to another can trigger an action (see chapter 3.3). This information is often provided by infrared as it operates with line of sight. Furthermore some positioning systems provide 32

3.8

Classification

Figure 3.11: Required Accuracy of Position Information relative orientation information, e.g. the Relate system. 3.8.4

Relative Position Between Humans, Devices or Physical Objects

The focus of this work are applications that use relative position information between different entities. An interesting question is which types of entities are taken into account, i.e. whether the application considers the location and orientation of a user with respect to a device, or the relative position of multiple devices, and so on. Thereby entities are defined as either humans, electronic devices or physical objects. Figure 3.12 provides an overview of previously discussed applications, each being assigned to the entities involved in the positioning. If all subsections are in the same group, only the subordinate section is listed. Applications can also be mentioned in more than one category (e.g. motion-tracked handhelds). As the position of human beings is often deduced from the position of a personal device (e.g. a PDA) the distinction between human and device is not clear. This is indicated with the dashed line between the corresponding rows and columns. How the line is drawn here, is best illustrated with examples. For a presence-aware display it is irrelevant whether the user’s proximity is sensed with a camera or an electronic device (e.g. an active badge), because the device is not the focus of attention. Thus, spatially-aware displays are considered as applications, which use the relative position of a human (i.e. the user) and a device (on which the applications are running, e.g. the display). The same applies to information displays, teleporting and memory aids. On the other hand, if the purpose of presenting spatial relationships is providing convenient interaction with other devices (e.g. spatial file transfer) the focus of attention is on the interacting devices, even though the user is controlling them. Devices and applications are also the focus when interaction between them is triggered by spatial conditions as well as for multi-display reaching and user interface migration. Motion-tracked handhelds can either consider their relative position to other devices (e.g. in the metaDESK) or to physical objects (e.g. when they act as windows to the physical world, see chapter 3.6.2). Guides use the relative position between the user and physical objects in order to adjust the shown section of a map or the presented information. A museum guide, for example, 33

3

APPLICATIONS USING RELATIVE SPATIAL INFORMATION

Figure 3.12: Subject to Positioning shows information about exhibits when the user is close to them. Another example is a smart home, which e.g. automatically switches the lights off when a user falls asleep on the sofa. Finally, tangible user interfaces consider the position of physical objects, e.g. building blocks, and interpret this information. Theoretically you could also think of applications that monitor the relative position of multiple people and react to this information. However, up to now such an application is not known.

34

4

Spatial Condition Toolkit

The survey and classification presented in chapter 3 examine the wide range of applications, which use relative spatial information in some way. Although all of them have this one thing in common, they are very diverse and difficult to pull together. For a profound work we therefore need to concentrate on a more concrete subject. The extent of chapter 3.3 (Position-Triggered Actions) shows that the automatic triggering of local services under certain spatial conditions is a common concept. The list of suitable services is long. First the proximity of multiple computers (such as PDAs, laptops, desktops, tablet PCs, and so on) can be used to initiate a synchronization or backup process, or transfer selected files from one computer to another. Second computer equipment, which is attached to one machine, can be “borrowed” to another one. For example a keyboard device, that is connected to a desktop computer, can be used to enter text one a handheld. Furthermore there are several stand-alone devices, which provide various services. Large wall displays can be used to show the content of a PDA’s screen. Alternatively sending a presentation file to a display could automatically start a slide show, which can be controlled with the PDA. In the same way printers, scanners or fax machines can automatically process sent files. Another idea is attaching electronic notes to nearby physical objects. As the technology develops, new services will emerge and existing ones will modify. Depending on the service and situation different criteria are suitable. Some applications, for example, use pointing to trigger a service, others are proximity-based, yet others consider the user’s orientation towards a device. For each of these examples different technologies stand to reason, e.g. infrared is suitable for pointing, bluetooth or sensor-based positioning systems for proximity, and so forth. Thereby the technology comes to the forth, which makes it hard to use conditions in a consistent way. In order to solve this problem we have developed a profound concept for spatial conditions and - based on this - the Spatial Condition Toolkit (SCT), which provides a uniform handling of conditions as well as the desired abstraction of the underlying technology. In the next subsection the purpose of this toolkit is explained. Second, the SCT is dissociated from previous work. Then some common ways of using spatial information are explained and thereby the functionality of the SCT is derived. The result is a layered structure which shows the toolkit’s main components. Before presenting a typical interaction sequence that demonstrates how applications can use these components, some basic questions have to be answered. In particular the term “spatial information” needs to be defined. Also the concept of spatial conditions is explained in detail.

4.1

Purpose of the Toolkit

Before explaining the intended purpose of the toolkit, the general function of toolkits is recapitulated. Toolkits are often used to simplify the use of common routines. Many applications share similar tasks, e.g. designing a graphical user interface (GUI). To save all these applications from implementing the GUIs from scratch, the abstract window toolkit (AWT) provides predefined GUI components. In other words, toolkits implement a particular functionality to reduce the implementation effort and error rate for applications that make use of this functionality. The SCT is intended for applications that use relative spatial information. One aim is to design an extensive interface for positioning systems in general that allows for convenient access to the position information. This interface defines which information should be provided by a positioning system and how it can be accessed. Furthermore the toolkit 35

4

SPATIAL CONDITION TOOLKIT

comprehends additional functionality, which is designed for location-triggered actions. The goal is to simplify the creation of applications that involve this kind of mechanism. Using the SCT, such applications can simply register pairs of spatial conditions and actions, which are automatically triggered when the condition is fulfilled. After the registration the SCT monitors the position information of involved people and objects, checks the conditions and if necessary automatically triggers the according action. Thus the implementation effort for applications is significantly reduced.

4.2

Dissociation from Previous Work

As described in chapter 2, different classifications, toolkits and frameworks for locationand context-aware applications were developed previous to this work. Although all of them differ from the SCT either in their focus or purpose, it is important to describe how they are linked and what is different. Most similar to the SCT is the Context Toolkit discussed in chapter 2.3.1. Both toolkits are implemented in Java and aim to simplify the process of building certain applications. The Context Toolkit is based on the concept of widgets, which provide context information while the details of how the information was sensed is hidden. It allows developers to rapidly prototype any context-aware application. As the SCT is specially designed for location-triggered actions, which is a special case of context-awareness, the Context Toolkit is useful for a significantly wider field of applications. For location-triggered actions, however, the SCT provides a lot of additional functionality. A classification, which demonstrates how location-triggered actions are exactly related to other context-aware fields, was presented in (2.2.1). Schilit describes four categories of context-aware applications: proximate selection, automatic contextual reconfiguration, contextual information and commands, and context-triggered actions. The SCT is clearly a toolkit for applications, which belong to the last category. However, as the SCT focuses on position information, only part of these applications are covered. Therefore the SCT is designed for position-triggered actions rather than context-triggered actions. Brown proposes a framework for context-aware applications, which is based on documents for the Stick-e Notes application. The software consists of several components, which were explained in 2.3.3. One of these components called SETRIGGER is responsible for triggering the notes when the according context occurs. Obviously its purpose is similar to the SCT, but the focus is different. This is illustrated in figure 4.1. The large oval stands for all context-triggered applications, no matter which application types are considered or which actions are triggered. The Stick-e architecture focuses on the triggering of documents. Therefore, applications where other actions are triggered, for example the execution of a keyboard service, are not included. The SCT focuses on position information and thus does not consider other context types. There is an overlap between these two focuses, which is applications, that show information contained in documents under spatial conditions.

4.3

Different Uses of Spatial Information

Depending on the requirements of an application different ways of using spatial information are suitable. In this chapter we will go into some common needs and how these needs can be satisfied with the toolkit. The result is a layered architecture, which includes the basic structure of the toolkit. 36

4.3

Different Uses of Spatial Information

Figure 4.1: Focus of Stick-e Framework vs. SCT

4.3.1

Direct Interaction with Positioning System

Navigation systems, for example, need to know the user’s position at any time in order to update the map correctly. The easiest way to realize this is shown in figure 4.2. There are two components. The positioning system provides the location of a specific person or object of interest. Most outdoor applications rely on the global positioning system (GPS) to monitor a GPS receiver in a mobile device. The second component is the application, which is using the spatial information provided by the positioning system. The application (in our example the navigation system) constantly receives position updates from the positioning system.

Figure 4.2: Application Receives Periodical Position Updates

4.3.2

Monitor

For navigation systems this simple architecture is sufficient because the user usually moves most of the time. However, sending the position information periodically is not suitable for all applications. If the tracked object changes its location only occasionally most of the information is sent for nothing. To save network resources a third component, the monitor, is created, which detects position changes. Only if the position has changed an update is sent to the application (see figure 4.3). Of course the monitor does not inhibit direct interaction with the positioning system for navigation systems and alike (referred to as “Application Type A” in figure 4.3). 37

4

SPATIAL CONDITION TOOLKIT

Figure 4.3: Monitor Sends Update When Position Changed

4.3.3

Trigger Agent

Another pattern that is very common especially for context-aware applications, is triggering an action under certain spatial conditions. For example, the rule for an ambient display could be ”if a user is facing the display, turn it on”. Here ”facing the display” is the spatial condition and ”turn the display on” is the action. To save implementation effort for applications like this, the Trigger Agent was built. The TriggerAgent is another component on top of the monitor, which checks all registered spatial conditions and as the case may be automatically triggers the action associated with it (see figure 4.4).

Figure 4.4: Toolkit Triggers Action When Spatial Condition is Met

Again it is still possible to directly interact with the positioning system or monitor. Thus we now have three different application types: Type A applications require constant position updates and therefore are directly linked to the positioning system. Type B applications are suitable for items which change their position infrequently. They once register the item at the monitor and from then on are informed when the item’s position has changed. Finally type C applications incorporate actions that are triggered when certain spatial conditions are met. They once register a pair of condition and action. Each time the monitor sends an update all registered conditions are checked and where appropriate the according action is triggered automatically. In figure 4.5 the responsibilities of all components are summarized. 38

4.4

Orientation, Location and Motion

Figure 4.5: Positioning System, Monitor, Toolkit and Application

4.4

Orientation, Location and Motion

The toolkit is designed for applications where spatial information is used to trigger some kind of activity. But what exactly is spatial information? Before going deeper into the toolkit functionality this needs to be answered. The main function of positioning systems is tracking things, which can change their position. These things can be physical objects, electronic devices or human beings. To embrace all these things with one word the generic term Trackable is introduced. Every Trackable has its own spatial information, which consists of three components (compare to 4.6): 1. location (where is the trackable?) 2. motion (where is the trackable going and how fast?) 3. orientation (in which direction is the trackable looking?)

Figure 4.6: Orientation, Location and Motion The difference between using motion and using location as trigger might not be immediately clear. After all, when a person moves she is changing her location. In other words: no matter if the user’s location or movement is the trigger for an action, a change will only occur while she is moving. However, when talking about location we are referring to a static situation. Something happens while someone is at a certain location. In contrast 39

4

SPATIAL CONDITION TOOLKIT

motion is dynamic: The action takes place while someone is changing her location. We will illustrate this with some examples. Imagine an employee with a PDA in his office. He points his PDA to his notebook and both automatically synchronize. In this case the action is caused by the PDA’s orientation alone. It does not matter where the device is located or if it is moving while it is pointing to the screen. Then the user places the PDA next to a nearby display to show its content to his associates. Here only the location is used to trigger the service. The application could e.g. work like this: While a PDA is closer than 5 cm the display service is triggered and the PDA’s content is shown on the screen. Finally an action can also be triggered by motion. An example is a guided museum tour: While the visitor is passing museum pieces, information is presented to him depending on how fast he walks by. If someone is standing directly in front of the object, all available information is shown. The faster he walks, the less details are shown. As soon as the visitor passes an object, information for the next piece will be shown. In this example the difference between location and motion is quite comprehensible. If only the location would be taken into account, the same information would be shown at the same distance, no matter if you walk towards it or move away from it. By using motion as a trigger, one can first consider directions and second, the speed (figure 4.7).

Figure 4.7: Museum Tour To sum up, spatial information of trackables consists of location, orientation and motion. Some applications only need a subset of this information. Relative position information is generated by comparing the location, orientation and/or motion of at least two trackables.

4.5

Spatial Conditions

As described earlier, the SCT is based on the concept of spatial conditions. In this chapter the concept is introduced, different forms of spatial conditions are defined and their use is illustrated. A spatial condition is defined as a boolean expression. At any moment the condition is either true or false. The complexity of conditions varies: The simplest form is a basic condition, which always refers to one aspect of location, orientation or motion with respect to a specific (set of) involved objects or people. For example “the user is facing the display” addresses the orientation of a user with respect to the display. In addition there are complex conditions which are composed of basic conditions. 40

4.5

Spatial Conditions

Figure 4.8: Basic Conditions Assigned to Orientation, Location and Motion 4.5.1

Basic Conditions

In figure 4.8 the most common basic conditions are grouped by location, orientation and motion. Each of them has quantitative and qualitative aspects. Quantitative aspects are determined by (a set of) numbers whereas qualitative aspects are determined by relations. Quantitative Aspects The quantitative aspect is more or less the definition of location, orientation and motion. Therefore they represent the attributes of the corresponding classes in the UML diagram shown in figure 4.9. The orientation can be specified with one number, which is the degree. The location is defined as x-, y- and z-coordinates. In 2-dimensional systems the z-coordinate is always 0. Motion is defined by three parameters: direction, velocity and acceleration. It is important to understand that direction is not the same as orientation (see chapter 4.6). A person can walk in a certain direction while looking (i.e. being oriented) in a different direction. Usually not all three parameters are taken into account. In the museum application of chapter 4.4 it would probably make sense to consider the direction and velocity. That means the slower a person walks, the more information is shown. However, sometimes it might just be interesting to know if a person moves at all. Then the direction can be disregarded. Finally, the acceleration can be used to determine whether the trackable is gaining speed or slowing down, etc. Qualitative Aspects One way of defining spatial conditions is using these quantitative definitions, e.g.: “if the speed is greater than 30 mph”. However for applications qualitative aspects that compare the position information of two or more locations are often more meaningful. Unfortunately some qualitative aspects such as “facing” or “left of” are not precisely defined. Of course, if two people are sitting in the same row, e.g. in a cinema, one of them is definitely “left of” the other. But what if they are sitting three rows and five 41

4

SPATIAL CONDITION TOOLKIT

Figure 4.9: Extract from UML Class Diagram: Trigger

Figure 4.10: Zones

seats apart? Is the condition “in front of” true or “left of” or both? The answer is not clear because people’s conceptions depend on the situation. For the toolkit, however, the rules have to be defined precisely. The definitions are explained in the following paragraphs. Contrary to the quantitative aspects, which are defined in the Motion, Location and Orientation classes, the qualitative aspects are attached to the trackable class (see figure 4.9). The reason is that it is often not immediately clear which of the three classes they are affiliated to.

Location Aspects As explained earlier, the location is specified by coordinates. By comparing coordinates one can find out whether a user is within a certain distance. You can also define zones and check whether a trackable is within this zone. A zone is defined as an area of any shape with a reference point that is set to a specific Location. The shape is defined within the inZone method which returns a boolean value (see figure 4.10). The easiest example is a range. Here the shape is a circle of a given radius with the center as reference point. Thus inZone() returns true if the distance between trackable and reference point is less than the radius, else false is returned (see code excerpt 4.11). By using zones you can also check whether two locations are in the same building, room or within line of sight, which is often more significant than distances. 42

4.5

Spatial Conditions

public boolean inZone(Trackable t) { double distance = t.getDistanceTo(super.getReferencePoint()); if(t.getDistanceTo(super.getReferencePoint()) < radius) return true; else return false; } Figure 4.11: Code Excerpt from Range.inZone()

Figure 4.12: Orientation Aspects: Left of, Right of, Facing, Turning Back to

Orientation Aspects Intuitively people think it is a matter of location whether an object is to your right, to your left, in front of you or behind you. However, if you think about it, you will realize it is more a question of orientation. After turning around (that is changing your orientation by 180◦ ) an object that was previously to your left is now to your right. Thereby the location did not change. In figure 4.12 this is illustrated. There are four similar sections. In the center of each section there is a circle which represents a trackable, e.g. the user. The star on the right side is a point of reference. To simplify matters the reference point is set to 0◦ . The line shows the user’s orientation. In order to define the four different cases the space is divided into right angles. According to the definition used by the toolkit the condition “facing” is true, if the deviation between orientation and the direction of the point of reference is less than or equal 45◦ (section A). Correspondingly for “right of” the allowed degree at the unit circle is between 45◦ and 135◦ , for “turning back to” it is between 135◦ and 225◦ and for “left of” it is between 225◦ and 315◦ (q.v. figure 4.13). 43

4

SPATIAL CONDITION TOOLKIT

Figure 4.13: Mathematical Definitions of Basic Orientation and Motion Conditions

Figure 4.14: Motion Aspects: Moving Towards / Away from Motion Aspects As for directions the rules are stricter (see 4.14). The allowed deviation for “moving towards” is only 20◦ compared to 45◦ for “facing”. The reason is that even with relatively small deviations one is still passing objects. By contrast, even when you are only roughly facing an object you can still see it. The opposite is the case if you are walking away from something. Even at a right angle the distance to the reference point increases. Thus for deviations between 20◦ and 90◦ neither “moving towards” nor “moving away from” is true while one of the orientation conditions (left, right, facing or turning back to) is true at all times. 4.5.2

Complex Conditions

In most situations basic conditions are insufficient. To control an ambient display, for example, it is not only interesting to know if the user is facing the display. At the same time the user should be within a certain range if personalized information is to be shown. Here a suitable spatial condition could be “the user is facing the display AND the distance between user and display is less than 5m.” Complex conditions like this are combined of basic conditions using the boolean operators “AND” “OR” and “NOT”. Applications that use the toolkit, can easily define their own customized conditions by creating a new class, which implements the Condition interface. On the top of figure 4.15 there is the Condition interface, which declares two methods. The evaluate method returns a boolean expression, which defines whether the condition is true or false. GetInvolvedTrackables returns a list of all trackables that play a role in the evaluation. These two methods need to be defined in a customized condition. A problem is that the evaluate method can not have any parameters since it is called generically from the TriggerAgent. Thus potential parameters have to be set in the con44

4.5

Spatial Conditions

structor of the condition object. Although it is simple to define conditions from scratch, some common conditions are predefined to prevent multiple implementation by different applications. An overview of predefined conditions and their relation is shown in figure 4.15.

Figure 4.15: Predefined Spatial Conditions Zone Condition As explained in chapter 4.5.1 the concept of zones is very common. The ZoneCondition class facilitates the use of conditions that are based on zones. Each ZoneCondition features one specific zone which can be specified by using the setZone method. The most important information is provided by two methods. InZone checks whether a specific trackable is within the zone. GetNrTrackablesInZone returns the number of trackables that are within the zone. Often the zone condition is combined with other conditions, e.g. “if the user is within a 3 meter range of the display AND facing it” (corresponding to the WithinZoneFacing condition). These conditions are sub-classes of the ZoneCondition. Their meanings can be derived from the class name. The according functionality is defined in the evaluate method. Quantifier Conditions A different approach is the QuantifierCondition. In boolean expressions there are often quantifiers, such as ALL or ANY. Normally these quantifiers have to be resolved in the evaluate method. This means the algorithm incorporates loops as shown in figures 4.16, 4.17 and 4.18, which will be explained later on. Instead of implementing the resolution in each condition that incorporates quantifiers, the prefabricated QuantifierCondition can be used. The purpose of this class is to provide the resolution algorithm and thereby reduce the implementation effort and error rate. We will demonstrate this with following example: “all trackables are at least 3 meters away from the display”. The corresponding mathematical expression is: ∀t ∈ involvedT rackable(distance(t, display) ≥ 3m) The QuantifierCondition is divided into two parts: on the one hand the declaration of quantifiers and trackables (here ALL and display) and on the other hand the inner condition “distance ≤ 3m”. The inner condition is defined in the method evaluateFor2 which returns a boolean value for two specific trackables. The method evaluate declared 45

4

SPATIAL CONDITION TOOLKIT

in the Condition interface runs through all potential trackables in a loop and combines the corresponding evaluateFor2 return values according to the chosen quantifier. The quantifiers and trackables are specified with two strings id1 and id2. Id1 always belongs to one specific trackable, in our case the display. The second is either 1. another specific trackable id (e.g. the user is facing the display) 2. ALL (e.g. all of the involved trackables are facing the display) or 3. ANY (e.g. at least one of the involved trackables is facing the display) In the first case evaluateFor2 is called only once with the two specific trackables as parameters (see figure 4.16). For the other cases the method is called for each of the involved trackables in a loop. In each iteration one of the parameters changes while the other one is fixed (in our example the display). In case of the ALL quantifier the evaluate method returns false as soon as one evaluateFor2 iteration is false (see figure 4.17). For the ANY quantifier true is returned as soon as one evaluateFor2 iteration returns true (see figure 4.18).

Figure 4.16: Code for Two Specific Trackables

Figure 4.17: Resolution of the ALL Quantifier

Figure 4.18: Resolution of the ANY Quantifier

4.6

A Typical Interaction Sequence

Now that we have gathered enough background information about the basic concepts, we can have a look at the typical interaction between applications and the SCT. The top-level application registers a pair of condition and action at the toolkit (see figure 4.20). More precisely the component within the toolkit which is responsible for this 46

4.6

A Typical Interaction Sequence

is the TriggerAgent. The term “trigger” is defined as combination of condition and action (compare figure 4.19). Since the same TriggerAgent instance can be used by multiple applications at the same time, it needs to know which application registers the trigger.

Figure 4.19: UML Class Diagram Extract: Trigger After storing the trigger in a list, the agent analyzes the condition in order to find all involved trackables. For example in the condition “if a user is facing the display” the involved trackables are user and display. These trackables are registered at the monitor. That way the TriggerAgent is informed of all position changes regarding the user and the display. To simplify matters there is only one involved trackable per trigger in figure 4.20. In reality the TriggerAgent runs through the array which is returned by the Condition.getInvolvedTrackables method and registers each trackable that is not already registered. After each update, which is initiated after a trackable moved, the conditions are checked and if applicable the according action is triggered.

Figure 4.20: Sequence Diagram In most cases the spatial condition controls when an action is started and stopped. For example the condition “user within two meters” triggers the action “display on”. The 47

4

SPATIAL CONDITION TOOLKIT

moment the user enters the two meter range the state of condition switches from false to true and the action is executed. As soon as the user leaves the range, the state of condition switches from true to false and the action is terminated. Therefore every action requires a start and stop method, which is called when the state of condition changes. As for the TriggerAgent this means the last state of condition needs to be saved in order to compare it to the next state. After an update was sent from the monitor the current state of condition is determined and compared to the last state. If both are true or both are false nothing happens. If the last state was false and the current state is true, the start method is called. If it is the other way around the stop method is called. The complete UML class diagram is shown in appendix A.

4.7

Adaptive vs. Dual Trigger Mode

In some cases the action that is triggered by calling the start method depends on the position information. For example, while the user is within the two meter range, the presented information adapts to the user’s proximity. For this case two more things need to be taken care of. First the application needs access to the position information. Therefore the condition which triggers the action is passed on as parameter for both the start and stop method. Each condition object has a list of trackables that are involved in the evaluation. Using this list the position information for each trackable can be retrieved and the action can be adapted. Second the TriggerAgent needs another way of handling the start and stop methods. As explained earlier nothing happens when the state of condition does not change. However, if the action adapts to the position information it makes sense to call the start method each time there is a position update. Thus the start method should also be called if the last state of condition was true and the current state is true as well. Consequently in the new mode the only case when nothing happens is when both the last and the current state of condition are false. We call this the adaptive mode since it is suitable for applications which adapt to position updates. The mode explained earlier is the dual mode since it is only used to switch something on or off (compare figure 4.21 and 4.22). To sum up, there are two trigger modes. The adaptive mode is used when the action adapts to position changes. To inform the application about the new position, the start method is called after each update while the condition is true. The condition object is passed on so that the application has access to the latest position information. If the application does not adapt to the position, the dual trigger mode is sufficient. In this mode the start method is only called when the state of condition changes from false to true. The application is not informed about position changes if the state of condition does not change. To allow a flexible handling the application can optionally specify the trigger mode for each trigger, that is for each combination of condition and action. By default the adaptive mode is active.

Figure 4.21: Adaptive Trigger Mode 48

4.8

Implementation

Figure 4.22: Dual Trigger Mode

4.8

Implementation

The SCT is implemented in Java. It consists of four packages: 1. trigger.* 2. trigger.conditions.* 3. trigger.zones.* 4. trigger.tools.* The main package comprises the components of the layered architecture shown in figure 4.5 (i.e. TriggerAgent, Monitor and Application) as well as the classes Trackable, Location, Motion, Orientation, Trigger and Action. The UML diagram in appendix A provides an overview of the classes. The content of the condition sub-package was discussed in chapter 4.5.2, which also includes an overview (figure 4.15). The zones package consists of the two classes Zone and Range, which are shown in 4.10. At last the tools package contains mathematical functions that are used in various places throughout the toolkit, e.g. calculating the degree from a target and destination point. Furthermore there is a LoggingManager class, which controls the debug level and thus the amount of information that is printed to the console during runtime.

49

4

50

SPATIAL CONDITION TOOLKIT

Figure 5.1: Service Maps and Menus

5

Gateway Application

Chapter 3 provides an overview of the broad field of applications, which use relative spatial information. In chapter 4 the focus is narrowed down to the automatic triggering of actions under spatial conditions. In this chapter a new interaction technique is presented, that is based on spatial criteria: the gateways. The motivation for this interaction technique is that the automatic triggering of services can disturb the user when the service is not needed. Better than directly executing the service is presenting the options to the user when the spatial conditions are fulfilled. That way the user is informed about services provided by nearby devices and he can manually choose which service to run. This should be done in an unobtrusive way, so that the user is not interrupted. There are several ways of doing this. One idea is shown in figure 5.1. The two boxes represent the content of a PDA’s screen. When the required conditions are true a small button appears on the screen (see left box). By pressing the button a map of the user’s local environment is shown (right box). In addition there is a list of available services for each device. By clicking on one of the menu items, the desired services is executed. An alternative idea is using gateways. The proceeding is similar but instead of presenting the position information on a map, the compass metaphor is used to position the button somewhere on the edge of the user’s screen. Thereby the position of the button indicates in which direction the device is located. The button functions as gateway to nearby devices. Gateways are much more intuitive to use than the service maps and menus mentioned in the first idea. Furthermore less mouse clicks are required to execute a services. In the following subsections the motivation for the gateway interaction technique is illustrated with some examples. Subsequently concept and design of the application are explained and the underlying implementation is outlined. Chapter 6 will present the results gained in the user study, which was conducted to evaluate the gateways. 51

5

GATEWAY APPLICATION

5.1

Motivation

Imagine you are visiting a company for a week. You plan to attend different meetings and give a talk about your area of expertise. While you are there, you get a temporary desk where you can work between the meetings. As usual you bring your notebook and PDA, since you prefer working with your personalized work environment. Nowadays, if you want to use local devices, like a printer, you first have to install driver software. This is usually quite a hassle. You have to ask colleagues for the name of the nearest printer, which often nobody knows by heart. Once they have figured it out you need to get the permission to use it since you don’t have a local account. Consequently you need to find out who is responsible for the administration and again bother your colleagues. After your request has been processed you can finally print. This procedure is often so annoying that people either don’t use the local services at all or they ask their colleagues to print for them. Of course, there is a reason why this is so complicated. First and most important is security. Since nowadays everything is connected and people from all over the world can use certain services, you somehow need to restrict access to these services. However, within an office building there is a certain level of security anyway, because not everyone can enter it. On the assumption that the company is not afraid of anyone sneaking into the building and printing a pile of paper unnoticed, the security issue can be disregarded in many cases. The second reason is a lack of convenient interaction techniques. According to the technology that is commonly used today, each machine has to go through the installation process in order to use a device. In all likelihood this will change in the future, as a great deal of research work is done in this field. Then there is no reason why the interaction with local services should be complicated anymore.

5.2

The Gateway Interaction Technique

The gateway interaction technique offers a convenient way of interaction with nearby devices. A gateway is a small window on the edge of the screen, which indicates that a local service can be used. The gateway’s position on the screen depends on the direction of the device which the service is running on. For example, if one is standing in front of a printer with a mobile device, a printer gateway is shown on the top of the screen. If the printer is to your left the gateway will appear on the left. Thereby the gateways are evocative of a compass. In figure 5.2 this is illustrated in more detail. In the left picture (section A) there is a user with his PDA indicated by a spot. The arrow represents the user’s orientation. As shown in section B the directions to the devices (printer to the left, display to the right and PC behind the user) are now mapped to the PDA’s screen. Section C shows how the gateways actually look like on the screen. When the user turns around the gateways are updated (see figure 5.3) In the design presented in figures 5.2 and 5.3 the distance to the device and its size are taken into account. The bigger the side the user is facing, the wider is the gateway and the farther the device, the narrower is the gateway. In other words the width of the gateway depends on the angle of incidence. The other dimension only depends on the screen size. Although this layout indicates distance and size of the represented device, the variable gateway size has a downside. Gateways contain buttons and icons. If it is too small the user will have difficulties to press the button or identify the icons. On the other hand if the gateway is too big, it superimposes a great portion of the screen. Therefore we chose to use constant gateway sizes as shown in figure 5.4 instead. 52

5.2

The Gateway Interaction Technique

Figure 5.2: Mapping of directions

Figure 5.3: Compass after the user turned to the left

Figure 5.4: Constant Gateway Sizes

53

5

GATEWAY APPLICATION

Each gateway can be used in two different ways. First it is a target for drag-and-drop. For example one can print a file by dragging it onto the printer gateway, send a file by dragging it onto the PC gateway or start a slide show by dragging it onto the display gateway. Second it can be used as a button. If you click on it, an options menu will be opened. This is an important functionality as drag-and-drop always triggers a default action without allowing for any specific settings. So, by dragging a file on the printer gateway the whole file will be printed with the default options, as if you would directly print (e.g. by clicking on the printer button). However, if you want to print only some pages or change the properties, you will have to click on the gateway.

5.3

Conditional and Scanning Mode

The gateways are useful for different purposes. First, a gateway can automatically appear when certain spatial conditions are met. Here the purpose is to indicate that the service exists and might be useful at that time. Imagine you are sitting in front of the PC entering text on your PDA. A small keyboard gateway appears on your screen. This adverts to rather use the keyboard instead of using the cumbersome text input methods provided by the PDA itself. The spatial conditions which cause the gateways to appear depend on the service. For example, the keyboard gateway is only shown if the PDA is within 50 cm of the keyboard, since the user needs the keyboard and the PDA to be within reach. When using a display service the user can be several meters away from the device, but he should be oriented towards it. If he is turning his back to the display he is hardly going to start using it. For more details see chapter 6.1.1. Alternatively the gateway application can be used to scan the environment for local services. In contrast to the conditional mode, this action is explicitly triggered by the user. To limit the number of simultaneously presented gateways, only services in the same room are shown. It does not make sense to indicate the direction of a service which is behind a wall. Also there are more suitable ways to present an overview of dozens of services in the whole building, e.g. hierarchies or lists.

5.4

The Design

For a new interaction technique like the gateway application, the design is essential. In the end the design determines whether the application is helpful and intuitive to use or frustrating. Thereby a number of things should be considered. First the gateways should attract an appropriate amount of attention. Second it has to be clear how the user can interact with the gateways. Moreover the gateways ought to be arranged in a way that makes sense to the users or, even better, provides valuable clues. 5.4.1

Drawing the Right Level of Attention

A key factor in making the gateway application successful is finding the right level of alertness, especially in the conditional mode. On the one hand the gateways have to draw attention in order to be useful. On the other hand they should claim as little display space as possible, so that the user is not disturbed if he does not want to use the service at that moment. Also, important user interface elements should not be superimposed. In particular gateways need to be placed next to the task bar, not on top of it. On notebooks and PCs this usually means that they are above the taskbar. If the taskbar is on the top, which is often the case for handhelds, the gateways would be below it. To avoid hiding other important information they could also be half transparent. 54

5.4

The Design

Figure 5.5: Original Gateway Draft

Figure 5.6: Final Gateway Design Furthermore it is possible to hide single gateways with one click as well as turn off the automatic appearing of gateways. 5.4.2

Button and Drag-and-Drop Funtionality

As explained earlier gateways have two functions: they serve as buttons and target areas for drag-and-drop. A fundamental question is how to design the gateways so that both meanings are obvious to the user. As for the button the gateway is highlighted when the mouse is placed over it. The same concept is used for buttons in most applications as well as for HTML-links. To indicate the drag-and-drop functionality we first thought about placing an arrow icon on the gateway (see figure 5.5). The problem is that next to the arrow there is another icon showing the represented device, e.g. a printer. During a test run several people said they find the two icons confusing because usually two icons imply two different buttons. In the final design (shown in figure 5.6) we therefore abandoned the arrow since the device icon is more informative. In addition a message will appear if the mouse hovers above the gateway for more than two seconds. The message is individual for each service. For the printer it would e.g. be ”Drag a file here to print it or click for more options”. 5.4.3

Avoiding Overlapping Gateways

Another issue is the distinction of multiple devices in the same direction. If, from the user’s point of view, there one device is behind another one, the according gateways will appear next to each other. The nearest device is also closest to the center of the screen (see Figure 5.7 ). That way, and by looking at the device icon, multiple devices in the same direction can be distinguished. The only exception is if two or more devices of the same type and the same size are on top of each other. In this very unlikely case the gateways would also be shown next to each other, but the user cannot tell which is which at first glance. In order to find out he would have to click on the device. 55

5

GATEWAY APPLICATION

Figure 5.7: Multiple Gateways in the Same Direction

5.5

Implementation

The gateway application is also implemented in Java. It makes use of the SCT since the appearing of gateways is triggered by spatial conditions. As for the SCT it was important to explain the implementation as detailed as possible because its functionality is intended for further usage. The gateway application is not presented in such detail for a number reasons. First of all it is widespread as it includes the service architecture, the wizard and much more. Thus an in-depth analysis would go beyond the scope of this work. Second, most of the functionality is only used internally, so there is no need for a detailed explanation. Third, there will be further development in the future. At the least some new services will be implemented and at some point the service architecture might be enhanced. At first a rough outline of the package structure is presented. In the succeeding sections the interrelation between the gateway application and the SCT is illustrated. Then it is explained how the wizard communicates with the application over the network. Finally we will go into the implementation of the actual core of the system and define how the gateway’s positions is calculated in consideration of the screen size as well as other gateways. 5.5.1

Package Overview

The core of the application consists of four packages: 1. gateways.* 2. gateways.devices.* 3. gateways.conditions.* 4. gateways.forms.* The root package contains the GatewayApplication and the toolbars. The MainToolbar is an extra window that allows the user to start and stop the application and set the mode. The AdminToolbar is an extension of the MainToolbar, which in addition allows the administrator to start and stop services and activate the wizard. Each gateway belongs to a Device, which is derived from the SCT class Trackable. In addition to the position information a Device has a specific size, provides services and has IP connectivity. Since the services are specific for each Device there is one sub-class for 56

5.5

Implementation

each of them (e.g. PrinterDevice, KeyboardDevice and DisplayDevice. Moreover there is the DeviceTransferObject class, which embraces only the most important facts, such as id, size and location. To save network resources, instances of these objects are sent over the network to the wizard instead of the extensive Device objects. This will be explained in more detail in chapter 5.5.3. The conditions which lead to the automatic appearance of gateways are defined in gateways.conditions.*. There is one condition per device. For the keyboard and printer gateways these conditions are derived from the predefined SCT condition WithinZone whereas for the display WithinZoneFacing is used. For more details on the spatial conditions see chapter 6.1.1. Furthermore there is another condition AlwaysTrue, which is used for the scanning mode. Actually in this mode the appearance of the gateways is not bound to any condition. To allow for a consistent approach the pseudo condition AlwaysTrue is created, which incorporates an evaluate method that always returns true. Each gateway has a GatewayController, which is responsible for showing the gateways at the right position. This controller also determines whether to show the HorizontalView or the VerticalView of the gateway. Chapter 5.5.2 will illustrate how these classes interrelate with the SCT, whereas chapter 5.5.4 demonstrates how the position is calculated. In addition there are various packages that provide the service functionality as well as the gateways.powoz.* package, which is responsible for the Position-Orientation-Wizardof-Oz. The latter includes the MouseDragAnalyzer and the RMI Interfaces WizardApplication and WizardMonitor, which will be explained later in chapter 5.5.3. 5.5.2

Using the SCT

Because in the conditional mode gateways are shown when certain spatial conditions are fulfilled, the gateway application is built on top of the SCT. Figure 5.8 shows the interrelation. The SCT checks registered conditions and automatically triggers the corresponding action. As for the gateways the action is popping up the windows, which in turn can be used to interact with the represented device. The class diagram in figure 5.9 gives more details. Since a complete class diagram would go beyond the scope, only selected attributes and methods are shown. More information about the gateway classes are shown in figure 5.10. On the right side there are the SCT classes, on the left side the gateway application. Each Trackable has its own Orientation, Motion and Location. On the one hand the user is a Trackable. On the other hand there are several Devices, which are extensions of the Trackable class. Devices have additional functionality such as a fixed size and IP connectivity. For each Device there is at least one Condition and Action. In the gateway application the Action is a GatewayController which implements the start and stop methods. The GatewayController has two GatewayView s (VerticalView and HorizontalView ) representing the actual window that is shown on the screen. When the start method is called, the GatewayController calculates the position on the screen, figures out whether the VerticalView or HorizontalView is needed and shows the window in the right position. Stop() will hide the gateway. At the same time the GatewayController has a Condition, which determines when start and stop is called. 5.5.3

Communication Between Gateway Application and Wizard

The gateway application and the wizard are running on different machines. There are two reasons why the two need to communicate. First the wizard needs a list of all devices in the room as well as their sizes and positions. These devices are painted onto the wizard 57

5

GATEWAY APPLICATION

Figure 5.8: Gateway Application is Using the SCT

Figure 5.9: Class Diagram of the Gateway Application and SCT

58

5.5

Implementation

Figure 5.10: Class Diagram of the Gateway Application map. That way the wizard can specify the users position relative to the devices and the results are much more accurate. Second, after each update on the map the new position information must be sent to the monitor. To enable this communication the Remote Method Invocation (RMI) technology is used. Figure 5.11 demonstrates this. On the left there is a box representing the wizard machine, on the right the handheld which the GatewayApplication is running on. In both cases the communication emanates from the wizard. Therefore two Remote interfaces are created on the right side, one for the Application (WizardApplication) and one for the Monitor (WizardMonitor ). The GatewayApplication and the Monitor export their RMI interface to a registry. The according code excerpt from the GatewayApplication is: Registry registry = LocateRegistry.getRegistry(); WizardApplication stub = (WizardApplication) UnicastRemoteObject.exportObject(this, 0); The registry service can run on any machine. If no IP address and port number is specified, it is running on the local host. The default registry port is 1099. Knowing the address of the registry, the MouseDragAnalyzer can now lookup the RMI interfaces and use their service. In the first place this is done in the setupDevices method in order to find out where the devices are located: Registry registry = LocateRegistry.getRegistry(‘‘ip’’, ‘‘port’’)); WizardApplication stub = (WizardApplication) registry.lookup(‘‘WizardApplication’’); devices = stub.getDeviceTransferObjects(); getDeviceTransferObjects() returns an array list of DeviceTransferObjects. Each of them is a reduced version of a Device that only contains its id, size and location. 59

5

GATEWAY APPLICATION

Figure 5.11: Remote Method Invocations (RMI) Whenever the wizard performs an update, the new location and orientation of the user is calculated and sent to the WizardMonitor using the update method. The according code excerpt from MouseDragAnalyzer.feedMonitor() is: Registry registry = LocateRegistry.getRegistry(‘‘ip’’, ‘‘port’’)); WizardMonitor stub = (WizardMonitor) registry.lookup(‘‘WizardMonitor’’); stub.update("user", p, o, null); Since the wizard cannot specify motions on the map, the last parameter is null. 5.5.4

Calculating the Screen Coordinates

In the gateway application there are two things that affect the position of a gateway on the screen. One is the position of the device relative to the user’s orientation. The other is the size of the display. Figure 5.12 illustrates how the gateway’s position can be calculated. Thereby two cases are distinguished. On the left the gateway is vertically aligned (i.e. on the left or right edge of the display) while the right side demonstrates horizontal gateways. The gray rectangle represents the display. α is the degree between the horizontal line and the line running from the display center to the upper right corner. β stands for the direction to the device. First we need to determine which edge the gateway is aligned to. How to do this is shown in table 5.1. Once it is clear whether we are dealing with a horizontal or vertical gateway, the only question is the distance to the center of the edge (referred to as distanceToCenter in figure 5.12). For the vertical gateway the maximal distance is half the screen height, which is sin α. The distanceToCenter is proportional to sin β. Using these equations the formula can be derived. The same works for horizontal gateways by replacing the screen height with the screen width and sine by cosine. This calculation translates the degree in which a device is located to display coordinates of the point where the degree hits the edge of the screen. From these coordinates the final position of the gateway can be determined easily. If the target position goes (partly) beyond the display, the gateway needs to be aligned to the display corner as shown in figure 5.13. 60

5.5

edge right top left bottom

Implementation

condition if(β < α k β > (360◦ − α)) if(β ≥ α && β ≤ (180◦ − α)) if(β > (180◦ − α) && β < (180◦ + α)) if(β ≥ (180◦ + α) && β ≤ (360◦ − α))

Table 5.1: How to Determine the Edge

Figure 5.12: Transformation from Degree to x- and y-Coordinates

Figure 5.13: Aligning a Gateway at the Display Corner

61

5

GATEWAY APPLICATION

Figure 5.14: Overlapping Gateways

Figure 5.15: Relocating an Overlapping Gateway Furthermore gateways are not supposed to overlap. Therefore whenever a gateway appears, the application is first asked whether the target area is occupied by another gateway. If so, one of the overlapping gateways will move away from the edge. In figure 5.14 a dark gateway is shown on the edge of the screen. The dashed rectangle marks the target position of another brighter gateway, which is calculated in the first iteration. Since there is an overlap between the dark and the dashed rectangles, a new target position for the brighter gateway is calculated in iteration two. In each iteration the display size is diminished by twice the gateway width in both dimension (compare figure 5.15). After recalculating the position the gateway width must be added to both coordinates in order to get proper values. This is repeated until the position of the new gateway does not overlap with any gateway. This procedure implicates that the gateway appearing most recently is closest to the center. Since we want to sort the gateways by their distance to the user and not by the time they appeared on the screen, all gateways have to be relocated starting with the farthest device whenever the application detects overlapping gateways.

62

6

User Study

In order to get feedback on the gateway interaction technique, a user study was conducted. In particular we wanted to find out whether the application is easy and intuitive to use. Before giving away the results, it is first demonstrated how the study was carried out. First of all the setup is summarized. In particular we will go into the spatial conditions that cause the gateways to appear. Furthermore we will explain how the wizard simulates the position information and what technical equipment and staff is needed. Then the proceeding is explained in detail. This includes the tasks the participants were asked to do and how they were instructed. Afterwards the execution is described and the questionnaire is evaluated. Finally the most important results are summarized.

6.1

Setup

The study took place in a meeting room. Distributed across this space were three devices: a keyboard, a printer and a large meeting display. Each device provided a local service. Participants were testing these services one after another. Therefor they were provided with a handheld device, which the gateway application was running on. For each service the user was asked to perform at least one task. In the conditional mode, which was chosen for this part of the study, the gateways only appear on the handheld when the user is close enough to the according device. That way the participants were forced to move across the room and get a feeling for the application’s behavior. A different approach would have been letting the user scan the environment and in doing so find out about available services. However, we expected that once the user has seen all gateways on the screen, he would not return to the conditional mode. As a consequence he would not have to move around and maybe not fully understand the underlying concept. In this case we would not have been able to evaluate the spatial conditions and many more aspects. Furthermore the course of action would not be structured and thus complicate some observations. 6.1.1

Spatial Conditions

In the conditional mode the gateways only appear if certain spatial conditions are fulfilled. As mentioned earlier these conditions are adjusted to the particular service. Figure 6.1 shows the room where the study took place as well as the location of the devices and the spatial conditions. The keyboard gateway only appears when the PDA is within 50 cm as the user needs both devices within reach. The orientation is not considered because the handheld can either be next to the keyboard, above it or below. For the large display the range is a lot wider because you can see it from far away. In addition there is a second condition. Imagine you are standing in a room and want to start a slide show on the meeting display. It is very unlikely that you are turning your back to the display at this moment, because you probably want to check if it actually works. Thus the display gateway will only appear if the user is facing it. Once the display service is started the user can change his orientation and move around as he wishes. For both keyboard and display movement does not play a role because the participant will in all likelihood stay in the same position while using the service or at least when the service is started. As for the printer scenario users often walk to the device to pick up the printed documents. Thus the gateway appears when the user is standing in front of it or moving towards it. To sum up the following spatial conditions are used: 63

6

USER STUDY

Figure 6.1: Spatial conditions • Keyboard: distance(user, keyboard) ≤ 0.5 m • Display: distance(user, display) ≤ 3 m AND user.orientation.towards(display) = true • Printer: distance(user, printer) ≤ 1 m OR user.motion.towards(printer) = true

6.1.2

Wizard of Oz

To be independent of any particular positioning system the Wizard of Oz approach was used in the study. Thereby the positioning information is simulated by the wizard who is using a map of the room where the study takes place, similar to the one in figure 6.2. The spot indicates the latest position whereas the line stands for the orientation. The wizard manually updates the user’s location and orientation on the map with a single mouse gesture. By pressing the left mouse button the wizard specifies the location. While keeping the mouse button pressed, the mouse cursor is then moved in the direction the user is holding the device. The point of origin (which corresponds to the user’s location) and the destination point are linked. The resulting line represents the user’s orientation. After each transaction the spot and line are repainted on the map and the position and orientation is updated in the system. Unfortunately this procedural method has a downside. The wizard can only update the user’s position in intervals of about one second. As a consequence the gateways jump from one position on the screen to the next. Ideally one would expect a smooth motion. However, this drawback is acceptable because most positioning systems do not provide the required update rate either. The low update rate implicates another problem. If the user’s movement was calculated from the few positions that were entered by the wizard, the accuracy would be very low. Thus it is much more effective for the wizard to detect if the user is moving towards the printer. On this account the functionality of calculating the movement from the position updates on the wizard map was not implemented. Instead the wizard cheats the system and pretends that the user is within range whenever the user walks to the printer. The only demand is that the user’s orientation is correct, so that the printer gateway appears in the right angle. 64

6.1

Setup

Figure 6.2: Wizard of Oz Map 6.1.3

Staff

To conduct the study two people were required. One person communicated with the users. He was responsible for the introduction and guided the users through their tasks. The other person was the wizard. Ideally the user should not recognize that the wizard is involved in the study. However, this was not realistic since she has to watch the user closely. To avoid that the participants get suspicious, they were told that both people are involved in the project and one of them is watching the process in order to take notes. 6.1.4

Technical Requirements

The gateway application was distributed across several machines, which are shown in figure 6.3. Display service, keyboard service and wizard each require a dedicated machine because of conflicting mouse and keyboard events. The display service machine was connected to a large flat screen representing the meeting display. Its functionality was listening to a port and starting a slide show. As all mouse and keyboard events are blocked during the slideshow neither keyboard service nor wizard can run on the same machine. The keyboard service machine listens to a port and if necessary catches key events. To stress that the keyboard belongs to a local machine (and thus is not connected to the handheld), a wired keyboard was chosen. As Java can only catch key events if the mouse cursor is placed on a Java window, the focus had to stay on this window during the study. Thus it was not possible to run the display service or wizard on this machine. Another computer was needed for the wizard to update the position information and take notes. The printer service could have run on any of these machines, since it is independent of any mouse or keyboard events. The only requirement was that the printer is installed on it. In addition a handheld was required, which the gateway application was running on. All of them needed to be connected to the same wireless network. 65

6

USER STUDY

Figure 6.3: Technical Setup Summing up, the technical equipment consisted of three computers (including keyboard, mouse and display), a handheld, a printer and a wireless router.

6.2

Proceeding

The course of action that is taken in a user study has to be planned carefully. This chapter presents the concept that was created up front. In the first phase the user was given a short introduction to the application as well as the process of the study. Then the tasks were performed in the conditional mode. Afterwards the scanning mode was demonstrated and finally a questionnaire was filled out. 6.2.1

Introduction

In the beginning of each session there was a general introduction concerning the intended purpose of the application. Then the services and tasks were explained. However, participants were not instructed on how to use the gateways. This was essential since one key question was whether the interaction technique is intuitive to use. Furthermore we made sure the users understood that the subject of evaluation was the application, not the user. We also pointed out that there is enough time to play around with the gateways. 6.2.2

Tasks

The first task was entering text on the handheld by typing on the keyboard. This service is activated by clicking on the gateway. A short message appears which confirms that the keyboard can now be used and how to end the service. The second service was starting a slide show. This application is intended for large meeting displays. Since that kind of display was not available, we used a large flat screen instead. An alternative was projecting the image onto a display mock up, e.g. a large white board. The advantage in the latter approach would have been a more realistic display size. However, this setup would implicate the risk of the beamer becoming the center of attention. In other words, users might want to interact with a beamer gateway 66

6.3

Execution

instead of the display gateway. Since beamer and display mock up are in different positions (and angles) this might confuse participants. On this account we opted for the large flat screen. To trigger the display service a slide show is dragged onto the display gateway. This is a shortcut to selecting the menu items ”Slide show” and ”View show” in Powerpoint. By clicking on the gateway one can select a file and send it, but the slide show will not start. By then the participants had experienced both the button and drag-and-drop functionalities with easy examples. At last they were asked to print two documents. The first one consisted of a single page, which was printed using default printer options. The easiest way to do that is drag-and-drop but the user can also click on the gateway and press the “OK” button without changing any settings. The second document was printed in gray scale. In this case drag-and-drop is not sufficient. In order to specify the color settings the participants had to click on the gateway and thereby open the options window. For each task we recorded whether the users figured out the easiest way to solve it and if so, how long it took.

6.2.3

Scanning Mode

So far the setup of the study was carried out in the conditional mode. If the user did not discover the scanning mode while playing around with the application we shortly explained what it is. This was necessary to avoid any confusion in the follow-up questionnaire.

6.3

Execution

The study was conducted with 15 users in consecutive sessions. All of them were students, 13 male and two female. 12 participants were between 21 and 25 years old, two between 26 and 30, one between 31 and 40. The study took about 30 minutes per person. In the beginning there was a short introduction of approximately five minutes. Then the participants performed the tasks, which took another five minutes. Afterwards they were shown the scanning mode and moved around to see how the multiple gateways simultaneously adapt to their position. The rest of the time was spent on the questionnaire. The original idea was to use the OQO, the smallest full-powered and full-featured PC when it came on the market in 2004. However, in the test run we had discovered that due to the small display (800x400 pixels), users find it very hard to do clicks. To avoid any influence on the test results we also installed the gateway application on the PaceBlade SlimBook tablet PC. Even with the large display (1024x768 pixels) the clicking was harder than with a mouse, but the improvement was immense. Altogether nine people used the OQO and six the PaceBlade. In figure 6.4 the average times for the three services are shown for the OQO, the PaceBlade and overall. Especially for the keyboard service, where clicking was inevitable, the difference is clear. Participants using the OQO performed the task in 2.44 seconds while those using the PaceBlade only needed 0.92 seconds. 67

6

USER STUDY

Figure 6.4: Time Used to Solve the Tasks

6.4

Questionnaire

In the end the participants were asked to fill out a questionnaire. That way we wanted to shed light on questions that could not be answered by monitoring. In order to avoid misunderstandings and find out as much details as possible, the questions were asked by word of mouth. That way we could dig deeper if the answers were inconsistent or unclear. Also the participants were more motivated to give feedback beyond the issues discussed in the questionnaire. The questionnaire consisted of four parts. In the first part the user’s general impression was captured. Second we wanted to find out how interrupting the gateways are and what can be done to improve this. Third the button functionality was compared to drag-anddrop. We asked which way was more obvious, which one was more appealing and how hard it was to figure out that there are two different ways of using the gateway. The last section focused on the user’s preference regarding scanning and conditional mode. For the complete questionnaire see appendix C. 6.4.1

General Assessment

All in all the feedback was very positive. Table 6.1 shows the average answers of the follow-up questionnaire. Figure 6.5 illustrates the distribution. The users were asked to give a ranking between 1 (not at all) and 5 (very much). All participants said, they like the application, 13 of the 15 would use it. They clearly find the application easy and intuitive to use, attractive and useful. Nobody thought it is frustrating. question average ? I like the application 4,47 I would use the application 4,47 The application is easy to use 4,40 The application is useful 4,13 The application is attractive 4,33 The application is intuitive to use 4,20 The application is frustrating 1,27 ? 1 = not at all, 5 = very much Table 6.1: General Assessment - Average Values From the three services provided in the study, the most popular was the printer service, followed by the meeting display, and finally the keyboard (see figure 6.6). The users also 68

6.4

Questionnaire

Figure 6.5: General Assessment - Distribution of Values made suggestions for other services that they would like to use, if the gateway application was running on their computer. One service is sending files to other handhelds, PCs, notebooks, mobile phones or hard disks. Another one is adding music to the playlist of a stereo or mp3-player or directly play a song. Furthermore one could copy the content of a blackboard to one’s personal computer or send a new entry. The same concept works for a calendar blackboard that you share with colleagues. People who liked the keyboard service also mentioned that they would like to use a mouse service. Finally someone named an electronic key service where you can send the key to a lock in order to open doors. In the same manner you could also replace the health insurance card and much more. We also asked some open questions. First we wanted to know where and under which circumstances people would rather use the gateways. Seven participants said, they would mainly use it in public buildings, at work or university, but probably not at home. Three people would use it especially in unfamiliar places and five would use it everywhere.

Figure 6.6: Popularity of Services 69

6

USER STUDY

The second question was which benefit they see in the application compared to traditional interaction techniques. The most frequent answers were “I can use devices on site, so I do not have to carry my own devices with me” and “it is convenient since there is no installation and configuration process”. Another common statement was that the application is very simple to use especially because of the ability to use drag-and-drop. Others mentioned that it is faster, allows for standard interfaces, it is fun and you can search for devices. All answers and the number of times corresponding answers were given are listed in table 6.2. answer convenient / no installation use on-site devices simple to use / allows for drag-and-drop search for devices that I can use faster fun standard interfaces

times 7 7 4 2 2 1 1

Table 6.2: Benefit Since many of the services imply that (personal) documents are sent to (public) machines, we were also interested in potential concerns about privacy issues. Six participants did not have any concerns, at least not for these services. They said for them personal privacy is not that much of a problem in general. Two users said, they would only share personal data if they would trust the service provider. For example they would rather use the application at university or work than in public places. Five people were concerned. Some of them said they would rather make the installation or configuration effort if they were not sure what happens to their sensitive files. They would only use the application for public or insensitive documents. Others would like to know which security standards are met. One person mentioned he would expect the system to warn the user whenever an unsafe service is used. Similar to Internet browsers a message could pop up, which informs the user of the involved risks and asks whether he wants to continue. 6.4.2

Design and Disturbing Factor

The first question regarding the design was, whether people would find the automatic appearance of gateways interrupting in real life. We consider this a design question because the most important objective was to create gateways that do not keep people from doing their work, regardless of the mode. The average answer was 2.07 (2 meaning “rather not”). When directly asked, twelve participants said, the gateways should not attract more or less attention. Three said the icons could be more colorful, but most people liked the simple design. Someone suggested that the gateways should be flashier immediately after they pop up, but then automatically disappear after a few seconds. Two others mentioned, they would prefer transparent gateway windows, so that the background is not completely hidden. Yet another user was confused by the red “X” that one can click to hide the gateway. He thought the icon indicates that the device is not connected. To avoid confusion, the color of the X could be changed into gray or black. Another way of annoying users are message boxes. On the one hand some form of feedback is needed to inform the user about what is happening. For example after dragging a presentation onto the display gateway, it takes a while for the document to be sent over the network. If there would not be some kind of confirmation that the command was 70

6.4

Questionnaire

Figure 6.7: Functionalities and Message Boxes accepted, users were irritated. On the other hand simple message boxes are often annoying because one has to press a button to get rid of them. 73% of the users thought the message boxes are helpful (see figure 6.7). On average they read 79 % of the text. Although this feedback is very positive, the usability could be improved by turning them into speech bubbles next to the gateway, which automatically fade out after a few seconds. Also, in the current version there is a small application window allowing the user to activate and deactivate the gateways or change the mode. Instead of having an own window, these options should be docked to the taskbar. 6.4.3

Button vs. Drag-and-Drop

Furthermore we wanted to find out how easy it is to discover that on the one hand you can click on the gateway and on the other hand use drag-and-drop. The average answer was 3.4 (neutral with a tendency to rather hard). The distribution is shown in figure 6.7. Almost 3/4 found the button functionality more obvious while all participants agreed that dragand-drop is easier to use. Interestingly we noticed that in this case the answers strongly depend on the order of services. Initially the users started with the keyboard service where they had to click on the gateway. Since they already knew this functionality, they did the same for the following services. After recognizing this we changed the order of services and asked some users to start with the display. All of these participants immediately used drag-and-drop. This indicates that the results are influenced by the setup of the study and probably in real life by the first device the gateway is used for. Thus, in all likelihood drag-and-drop is far more intuitive to use than the results indicate. Another way to help users discover the two functionalities is using tooltip messages, that are shown when the mouse hovers above the gateways. During the study only 3 participants (20 %) read the tooltip messages. However, one should consider that the participants worked with the application for only five minutes. One user spotted the tooltip message by accident while trying to hit the button with the stylus. The chances are that this would sooner or later happen to many users. One participant suggested to show an information bubble with instructions for use at the first time a gateway appears. This would certainly speed up the discovery of both functionalities as well. As for the tooltip messages the instructions depend on the service. The keyboard message should tell the user to click on the gateway in order to start it and 71

6

USER STUDY

Figure 6.8: How to Use the Modes and Conditions

how it can be stopped. Furthermore a short description of the service would be helpful, e.g. “The keyboard service allows you to enter text on your handheld by using a regular keyboard.” Analogous the printer and display instructions explain the provided service, as well as the difference between clicking on the gateway and using drag-and-drop. Since the instructions are different for each device, the information bubble should appear whenever a new gateway type pops up. In addition, the user should be able to look the information up. The best way of doing this is adding a help menu to the gateway window. The question is how this menu can be accessed without complicating the simple design. A regular click on the gateway is already reserved for starting services or setting properties. Alternatively clicks on the right mouse button could be used, which is often done in desktop applications where a mouse is available. However, this can be very difficult on handhelds as a stylus does not provide a convenient way of simulating this. On touchscreens right-clicks do not work at all. Another solution is replacing the tooltip messages with a menu that appears on mouse over. Since the first solution is faster if a mouse is available, it is probably best to implement both ways.

6.4.4

Conditional and Scanning Mode

In the last section of the questionnaire we wanted to find out how people want to use the modes and conditions. Four people would like to see gateways for all services in the room at all times, eleven would not. When asked whether they would like the gateways to automatically appear when they need a certain service the answers were nearly fifty-fifty (see figure 6.8). On average people would use the scanning mode 57 % of the time (accordingly 43 % for the conditional mode). The distribution is shown in figure 6.9. However, more revealing than these numbers were the users’ explanations. Most participants said they would use the scanning mode mainly when they enter a new room. When they are in a familiar environment - that is most of the time - about half of the users would use the conditional mode while the others would rather manually activate the gateways when they need it. 72

6.4

Questionnaire

Figure 6.9: Conditional vs. Scanning Mode

We also asked the participants which conditions they would expect an intelligent system to consider in the conditional mode. Although we focus on spatial conditions in this work, one could use other conditions as well. On the one hand showing the keyboard gateway makes only sense if a keyboard is nearby. On the other hand using the keyboard service will only help if the user is entering text. The most common answer was to take into account what application is currently used. This was suggested by seven people, who provided different examples. One was to show the keyboard gateway when the user opens a text editor. When you are entering text on a website, the keyboard service is probably not that important because usually only a few words are entered. Furthermore, after saving a text document the user might want to print it whereas after saving a presentation the display gateway should appear. Another interesting idea was considering the files that are shown in the file explorer window. If the current folder view contains at least one slide show, the display gateway is shown. The printer gateway appears if there are documents or pictures etc. This is a fantastic idea because that way you see the icon, which can be dragged onto the gateway. By contrast if the file is already open one can hardly use drag-and-drop. Two people suggested to monitor the customer’s habits. For example if the user opens Word and directly prints the document for the tenth time, the application should remember this and eventually automatically show the printer gateway. You could also combine this with spatial conditions: When the user enters a room again and again only to print a document and pick it up, the system can learn from this as well and show the printer gateway whenever the user enters the room. Furthermore the application could detect difficulties in using the mouse or keyboard and where appropriate offer the mouse or keyboard service. Yet another example is to adjust the occurrence of the keyboard gateway to the frequency the keyboard is used. In other words, if a user starts the keyboard service very often, the gateway will appear more often as well. Although in this work only spatial conditions were considered, it is good to know which features people are interested in. Coupling the gateway application with some kind of artificial intelligence seems to be very attractive for many users. 73

6

USER STUDY

6.4.5

Comments

In the end we asked for comments, suggestions for improvement and collected ideas for new features. The statements in the first paragraph regard the application, whereas the second paragraph focuses on the services. Features One idea was to show icons for programs that are not installed on the mobile machine but on another computer next to it. That way the user can switch to another work station instead of installing the program. Another user said he wanted to be able to tell the application that he is looking for a printer so that all printer gateways appear. Yet another suggestion was to show the distance to the device somewhere, e.g. “the next keyboard is three meters away”. Furthermore an option should be included to switch off the message boxes. Services Besides new features there were also some comments on the services. As for the display service people wanted to do more than only starting a slide show. They asked if one can go to the next slide or close the presentation using the handheld as controller. Others expected that the content of the handheld’s screen is simply shown on the display. That way the service could be used for things other than starting slide shows. In fact this is rather a new service for regular screens than an improvement for the current display service. Depending on the screen size you could offer a meeting display service, which starts and controls presentations, a regular display service, which sends the screen content to a larger screen, or both. One person mentioned he finds it awkward to use a large keyboard in combination with a tiny screen. Since usually you find keyboards and screens next to each other, it might make sense to combine the display and keyboard service especially for small handhelds. Regarding the keyboard service, one user would prefer if you would not have to click on the gateway in order to start it. Instead the application should ask the user with the first arriving keys whether the user accepts them or not. Another user said that he wants to see the options menu after dropping something onto the printer gateway. Since other users think different, it is probably best to let the users define these kind of actions themselves.

6.5

Summary

Altogether the feedback was very satisfying. People liked the idea of spatially discovering services very much. About half the people were also interested in an intelligent system that pops up gateways when the corresponding service is needed. Furthermore they agreed on the simple design and at the most would change an icon. It turned out that drag-and-drop is very intuitive to use. However it is not appropriate for all services. Thus clicking is needed as a second way of interacting with the gateway, which seems to unsettle users. For example the keyboard service is started by clicking on the gateway. We realized that people who started with this service continued clicking on the display and printer gateways because they were already used to it, whereas people who started with one of the other services intuitively used drag-and-drop. To avoid this, a speech bubble should pop up when a specific gateway appears for the first time and give instructions on how to use it. Another suggestion is facilitating customization. Some users do not like the direct printing and want to see the option menu when they drag a document on the printer gateway. Others would like the keyboard service to start automatically and accept arriving keys with a confirmation box. Since the users’ wishes are not consistent, an options menu 74

6.5

Summary

should be included that allows each user to specify her own preferences. In this menu the users could also set the icons and choose if the gateway should be transparent or automatically fade out after a few seconds.

75

6

76

USER STUDY

7

Conclusion

This thesis examined applications, which make use of relative spatial information. Relative position - as opposed to absolute position - considers the location, orientation, or motion of people or objects with respect to other people or objects. While each individual location can be defined by absolute coordinates, a relative statement can only be made for two or more entities. Particularly interesting is the distance, the relative orientation (i.e. whether the user is facing the display), or movement (i.e. whether the user is moving away from something or towards it). What relative position information can be used for was examined in a survey and classification, which was the first part of the thesis. In a next step, the automatic triggering of actions under spatial conditions was examined in more detail. A toolkit was developed to support the implementation of such mechanisms. Finally a new interaction technique was presented, which allows for an easy and intuitive interaction with co-located devices. In addition this technique can be used to discover local services. It is in particular suitable for mobile devices. The conclusion consists of two parts. First, the contribution of this work is recapitulated. This includes a summary of the survey, the toolkit, and the gateway interaction technique. Second, ideas for future work are presented.

7.1

Summary

In the first part of the thesis the wide field of applications using spatial information was presented in a survey. The goal was providing a detailed introduction to the research that was done previous to this work. Similar applications were grouped to make the survey more manageable. The survey first summarized different ways of presenting spatial information to the user, such as maps, compasses, 3D visualization, spatially ordered lists, and labels, which contain spatial descriptions. Second, context-aware applications were examined, as location is an important part of context. This included spatiallyaware displays, information displays, tourist guides, teleporting (mapping user interfaces, phone calls, etc. to the device which is closest to the user), memory aids as well as other applications. Another major field was the use of spatial criteria in order to trigger actions. This concept is mainly used for interaction between co-located devices. Moreover multidisplay reaching techniques were presented as well as the migration of user interfaces. Also, different ways of using motion-tracked handhelds were analyzed. Finally tangible user interfaces were brought up. However, since this is a quite separate field, it was not discussed in detail. Following the survey all these applications were classified according to several criteria. One question was, in which way the spatial information is used. The positions can be shown to the user to provide an overview, navigation aid, or improved ability to search and browse. Second, the position information can influence the presentation as in a list of objects, that are sorted by their distance to the user. Third, the spatial information can be processed by the system, which either adapts to it or takes an appropriate action. In the last case the presentation is completely independent from the spatial information or there is no presentation at all. If the spatial information is presented to the user, there are several ways to do so. Each presentation form has its own characteristics, such as number of dimensions, accuracy and so on. Another aspect was how accurate the information provided by the positioning system needs to be. Some applications require very precise spatial information while others just need to know the room in which something is currently located. 77

7

CONCLUSION

Finally it was examined whether the relative position between humans, devices or physical objects is considered. The intention of this survey was covering as many location-enhanced applications as possible. However, because of the large amount of applications this is hardly manageable. Even if the survey was complete today, it would not last long, as new applications constantly emerge. Nevertheless the survey gave a review of what relative spatial information can be used for. In addition the classification provided some aspects, which can help organizing the huge amount of existing as well as future applications. While the survey covered a wide range of applications, the second part zoomed in on the triggering of actions under spatial conditions. The Spatial Condition Toolkit (SCT) was developed and implemented for applications that use this concept. It is based on a profound model of spatial conditions and provides an abstraction of the underlying technology. The toolkit allows applications to register pairs of spatial conditions and actions, which are triggered when the according condition is fulfilled. Once a pair is registered, the toolkit observes the position information of involved people, devices or objects, checks the spatial conditions and - when appropriate - triggers the appendant action. The most prevalent spatial conditions are pre-implemented, e.g. whether the user is within a zone, facing an object or moving towards it, and so on. Additional conditions can easily be created by implementing one boolean method. Instructions and examples are provided and explained in detail. Furthermore a new interaction technique was presented, which is based on the concept of spatial criteria. In fact, the triggering of actions under spatial conditions is an interaction technique as such. If the user knows which criteria lead to which actions, he can manipulate his environment in order to fulfill the criteria when the according action is desired. However, automatic execution always bears the risk of triggering an action by mistake. This can disturb the user’s workflow. Therefore the gateway interaction technique was presented, which significantly reduces the risk of annoying the user. Gateways are small windows, which appear on the edge of the screen, when certain spatial conditions are fulfilled. In the easiest case the gateway appears when the user is within a specific distance of a device. Using the gateway window the user can then interact with the according device. On the one hand he can drag-and-drop a file onto the gateway to send it, on the other hand he can click on it to start other services. The gateway’s position on the screen is determined by the direction in which the device is located. Thus the gateway points to a device the same way a compass points north. Next to the automatic appearance of gateways under spatial conditions, users can also scan their environment for local services. In the scanning mode gateways for all services in the room are shown. This is an interesting application in particular for mobile devices. The objective was to design the gateways in an unobtrusive way and make them as easy and intuitive to use as possible. The implemented system was evaluated in a user study. Therefore the services were installed on different machines. 15 participants were given a handheld device, which the gateway application was running on. They were asked to test the services and perform several tasks. In the end each participant filled out a questionnaire. The feedback was very positive. People liked the application and would like to use it, provided they have an appropriate mobile device. The users also liked the simple design. Furthermore the study showed that users figure out at least one way of using the gateways quickly. Showing hints or instructions in speech bubbles could speed up the discovery of the second interaction possibility. About half the users would prefer the scanning mode, which means they would rather spatially discover local services when they are looking for something. This is especially 78

7.2

Future Work

helpful in new environments. Once the user knows which services are available in the room, he would rather use the automatic appearance under spatial conditions. This result shows that the gateway application can be used in multiple ways and thus is attractive for different user preferences.

7.2

Future Work

In the thesis the gateway interaction technique was designed, implemented and evaluated. Although the feedback gained in the user study was very positive, participants made some suggestions for improvement. One was to enable customized settings, such as switching off message boxes or defining whether the printing options menu should appear. This was not implemented, as the user study required uniform settings. Furthermore there were some ideas for new features. One example was showing icons for programs, which are not installed on the user’s machine, but on a co-located one. Provided with this information, the user can decide to switch to another computer if the functionality of this program is needed. Another participant would like to see all gateways of one particular device type, e.g. all printer gateways, if requested. Next to ideas on how the gateway interaction technique could be improved, there were also suggestions for new services. For the user study three exemplary services were implemented: 1. the keyboard service, which can be used to enter text on mobile devices, 2. the meeting display service, which accepts slideshows from mobile devices and starts them, and 3. the printer service, which prints documents that were sent from mobile devices. Participants mentioned the gateways could also be used to interact with other computers, stereos, TVs, hard disks, electronic keys, blackboards, or smart homes appliances (e.g. control the lights). However, in the long run developing proprietary services does not make sense. Instead the gateway interaction technique should cooperate with a sophisticated service architecture. This architecture should integrate a uniform service concept and provide dynamic access to local services. A service architecture, which is especially designed for the gateway interaction technique, is currently developed at Lancaster University. In chapter 4 the focus was set to the automatic triggering of actions under spatial conditions, as this is a concept, which is used by many applications. The gateway interaction technique uses spatial criteria to decide when to show the gateways. The premise is that spatial information indicates whether an interaction is possible or likely. For example, when the user sits in front of a keyboard while he is working with a handheld, a gateway appears which tells him that the keyboard can be used to enter text on the handheld. Using the distance between the user’s handheld and the keyboard as trigger makes sense, because in order to use the service, both devices need to be next to the user. However, considering more than just position information would be even better. In particular, it would be interesting to know whether the user is entering text. Thus a better rule would be “if the user is typing a long passage and the user is within reach of the keyboard device, the gateway should appear”. Thus, spatial criteria are only part of the information that should be considered for the gateway interaction technique. For a start the current engagement of the user should be observed and factored into the conditions, which lead to the triggering. Ideally the decision when to show the gateways could be made by an artificial intelligence (AI) system.

79

7

80

CONCLUSION

Appendices A

SCT Class Diagram

81

B

B

82

CLASS DIAGRAM OF THE GATEWAY APPLICATION WITH WIZARD

Class Diagram of the Gateway Application with Wizard

C

User Study Questionnaire

83

C

84

USER STUDY QUESTIONNAIRE

LIST OF FIGURES

List of Figures 1.1 2.1 2.2 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9 3.10 3.11 3.12 4.1 4.2 4.3 4.4 4.5 4.6 4.7 4.8 4.9 4.10 4.11 4.12 4.13 4.14 4.15 4.16 4.17 4.18 4.19 4.20 4.21 4.22 5.1 5.2 5.3 5.4 5.5 5.6 5.7 5.8 5.9 5.10 5.11 5.12

The Use of Position . . . . . . . . . . . . . . . . . . . . . . . . . . . . Context-Aware Categories . . . . . . . . . . . . . . . . . . . . . . . . . Topiary’s Storyboard Workspace . . . . . . . . . . . . . . . . . . . . . Application Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . ARIS Iconic Interface [5] . . . . . . . . . . . . . . . . . . . . . . . . . Close and Intimate Interaction with the MirrorSpace [49] . . . . . . . Proximal Interaction Model [46] . . . . . . . . . . . . . . . . . . . . . Radar View . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Screenshots of the Virtual Representation of a Real Migration [36] . . Hardware Architecture of the metaDESK [59] . . . . . . . . . . . . . Ways of Using Spatial Information . . . . . . . . . . . . . . . . . . . . Presentation Forms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Accuracy of Presentation Forms . . . . . . . . . . . . . . . . . . . . . . Required Accuracy of Position Information . . . . . . . . . . . . . . . Subject to Positioning . . . . . . . . . . . . . . . . . . . . . . . . . . . Focus of Stick-e Framework vs. SCT . . . . . . . . . . . . . . . . . . . Application Receives Periodical Position Updates . . . . . . . . . . . . Monitor Sends Update When Position Changed . . . . . . . . . . . . . Toolkit Triggers Action When Spatial Condition is Met . . . . . . . . Positioning System, Monitor, Toolkit and Application . . . . . . . . . Orientation, Location and Motion . . . . . . . . . . . . . . . . . . . . Museum Tour . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Basic Conditions Assigned to Orientation, Location and Motion . . . . Extract from UML Class Diagram: Trigger . . . . . . . . . . . . . . . Zones . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Code Excerpt from Range.inZone() . . . . . . . . . . . . . . . . . . . . Orientation Aspects: Left of, Right of, Facing, Turning Back to . . . . Mathematical Definitions of Basic Orientation and Motion Conditions Motion Aspects: Moving Towards / Away from . . . . . . . . . . . . . Predefined Spatial Conditions . . . . . . . . . . . . . . . . . . . . . . . Code for Two Specific Trackables . . . . . . . . . . . . . . . . . . . . . Resolution of the ALL Quantifier . . . . . . . . . . . . . . . . . . . . . Resolution of the ANY Quantifier . . . . . . . . . . . . . . . . . . . . . UML Class Diagram Extract: Trigger . . . . . . . . . . . . . . . . . . Sequence Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Adaptive Trigger Mode . . . . . . . . . . . . . . . . . . . . . . . . . . Dual Trigger Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Service Maps and Menus . . . . . . . . . . . . . . . . . . . . . . . . . . Mapping of directions . . . . . . . . . . . . . . . . . . . . . . . . . . . Compass after the user turned to the left . . . . . . . . . . . . . . . . Constant Gateway Sizes . . . . . . . . . . . . . . . . . . . . . . . . . . Original Gateway Draft . . . . . . . . . . . . . . . . . . . . . . . . . . Final Gateway Design . . . . . . . . . . . . . . . . . . . . . . . . . . . Multiple Gateways in the Same Direction . . . . . . . . . . . . . . . . Gateway Application is Using the SCT . . . . . . . . . . . . . . . . . . Class Diagram of the Gateway Application and SCT . . . . . . . . . . Class Diagram of the Gateway Application . . . . . . . . . . . . . . . Remote Method Invocations (RMI) . . . . . . . . . . . . . . . . . . . . Transformation from Degree to x- and y-Coordinates . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2 9 12 15 17 19 22 24 26 29 31 31 32 33 34 37 37 38 38 39 39 40 41 42 42 43 43 44 44 45 46 46 46 47 47 48 49 51 53 53 53 55 55 56 58 58 59 60 61 85

LIST OF FIGURES

5.13 5.14 5.15 6.1 6.2 6.3 6.4 6.5 6.6 6.7 6.8 6.9

86

Aligning a Gateway at the Display Corner . Overlapping Gateways . . . . . . . . . . . . Relocating an Overlapping Gateway . . . . Spatial conditions . . . . . . . . . . . . . . . Wizard of Oz Map . . . . . . . . . . . . . . Technical Setup . . . . . . . . . . . . . . . . Time Used to Solve the Tasks . . . . . . . . General Assessment - Distribution of Values Popularity of Services . . . . . . . . . . . . Functionalities and Message Boxes . . . . . How to Use the Modes and Conditions . . . Conditional vs. Scanning Mode . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

61 62 62 64 65 66 68 69 69 71 72 73

LIST OF TABLES

List of Tables 2.1 3.1 5.1 6.1 6.2

Context-Aware Software Dimensions . . . . . . . . . Different Ways of Controlling Application Migration How to Determine the Edge . . . . . . . . . . . . . . General Assessment - Average Values . . . . . . . . . Benefit . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

9 25 61 68 70

87

LIST OF TABLES

88

Content of the attached CD • figures/... contains figures (.jpg files) of the thesis, sorted by the following topics: – – – – – – –

application/... classification/... conditions/... introduction/... relatedwork/... study/... toolkit/...

• gateways/... SCT and gateway application – – – – – – –

build/... (executable Java classes) config/... (configuration for the services) data/... (images, test files for the services) doc/... (JavaDoc) lib/... (jars, that need to be included in the build path) src/... (source code) howto.txt (instructions on how to run the application)

• references/... contains the references in .pdf or .ps format, sorted by the following application fields: – – – – – – – –

location-aware applications/... motion-tracked handhelds/... multi-display reaching/... position-triggered actions/... related work/... spatial presentation/... tangible user interfaces/... user interface migration/...

• study/... – – – – –

consent form.doc questionnaire.doc questionnaire.pdf results.xls task sheet.doc

• thesis/... (latex and bibtex files) • topiary/... (executable code of the Topiary prototyping tool) • content.txt (discription of the CD’s content) • thesis.pdf (this document)

89

90

References [1] Mike Addlesee, Rupert Curwen, Steve Hodges, Joe Newman, Pete Steggles, Andy Ward, and Andy Hopper. Implementing a sentient computing system. Computer, 34(8):50–56, 2001. [2] Stavros Antifakos and Bernt Schiele. Bridging the gap between virtual and physical games using wearable sensors. In ISWC ’02: Proceedings of the 6th IEEE International Symposium on Wearable Computers, page 139, Washington, DC, USA, 2002. IEEE Computer Society. [3] Abhaya Asthana, Mark Cravatts, and Paul Krzyzanowski. An indoor wireless system for personalized shopping assistance. In IEEE Workshop on Mobile Computing Systems and Applications, Santa Cruz, CA, US, 1994. [4] Michael Beigl. Using spatial co-location for coordination in ubiquitous computing environments. In HUC ’99: Proceedings of the 1st international symposium on Handheld and Ubiquitous Computing, pages 259–273, London, UK, 1999. Springer-Verlag. [5] Jacob T. Biehl and Brian P. Bailey. Interfaces for managing applications and input in multi-device environments. In Proceedings of the Workshop on pervasive display infrastructures, interfaces and applications, 2006. [6] Scott Brave, Hiroshi Ishii, and Andrew Dahley. Tangible interfaces for remote collaboration and communication. Proceedings of the 1998 ACM conference on CSCW, 1998. [7] Peter J. Brown. The stick-e document: a framework for creating context aware applications. In Proceedings of Electronic Publishing, pages 259–272, 1996. [8] PJ Brown, J. D. Bovey, and X. Chen. Context-aware applications: from the laboratory to the marketplace. IEEE Personal Communications, 4(5):58–64, 1997. [9] Guanling Chen and David Kotz. A survey of context-aware mobile computing research. Technical Report TR2000-381, Dept. of Computer Science, Dartmouth College, November 2000. [10] Luca Chittaro, Vijay Kumar Gatla, and Subramanian Venkataraman. The interactive 3d breakaway map: A navigation and examination aid for multi-floor 3d worlds. In CW ’05: Proceedings of the 2005 International Conference on Cyberworlds, pages 59–66, Washington, DC, USA, 2005. IEEE Computer Society. [11] J. E. Cutting and P. M. Vishton. Perceiving layout and knowing distances: The integration, relative potency, and contextual use of different information about depth. Perception of Space and Motion, pages 69–117, 1995. [12] Arman Danesh, Kori Inkpen, Felix Lau, Keith Shu, and Kellogg Booth. Geneytm: designing a collaborative activity for the palmtm handheld computer. In CHI ’01: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 388–395, New York, NY, USA, 2001. ACM Press. [13] N. Davies, K. Mitchell, K. Cheverst, and G. Blair. Developing a context sensitive tourist guide, 1998. [14] Nigel Davies, Keith Cheverst, Keith Mitchell, and Alon Efrat. Using and determining location in a context-sensitive tour guide. Computer, 34(8):35–41, 2001. 91

[15] Anind K. Dey and Gregory D. Abowd. Towards a better understanding of context and context-awareness. In Technical Report GIT-GVU-99-22, Georgia Institute of Technology, College of Computing, 1999. [16] Anind K. Dey, Daniel Salber, Gregory D. Abowd, and Masayasu Futakawa. The conference assistant: Combining context-awareness with wearable computing. In ISWC ’99: Proceedings of the 3rd IEEE International Symposium on Wearable Computers, page 21, Washington, DC, USA, 1999. IEEE Computer Society. [17] Steven Dow, Jaemin Lee, Christopher Oezbek, Blair MacIntyre, Jay David Bolter, and Maribeth Gandy. Wizard of oz interfaces for mixed reality applications. In CHI ’05: CHI ’05 extended abstracts on Human factors in computing systems, pages 1339–1342, New York, NY, USA, 2005. ACM Press. [18] Jennifer Falk, Peter Ljungstrand, Staffan Bj¨ork, and Rebecca Hansson. Pirates: proximity-triggered interaction in a multi-player game. CHI ’01 extended abstracts on Human factors in computing systems, 2001. [19] George W. Fitzmaurice. Situated information spaces and palmtop computers. Communications of the ACM, Volume 36, Issue 7, 1993. [20] E. Hall. The Hidden Dimension: Man’s use of Space in Public and Private. Doubleday, New York, NY, 1966. [21] Andy Harter, Andy Hopper, Pete Steggles, Andy Ward, and Paul Webster. The anatomy of a context-aware application. Wirel. Netw., 8(2/3):187–197, 2002. [22] Mike Hazas, Christian Kray, Hans Gellersen, Henoc Agbota, and Gerd Kortuem. A relative positioning system for co-located mobile devices. Proceedings of the 3rd International Conference on Mobile Systems, 2005. [23] Jeffrey Hightower and Gaetano Borriello. Location systems for ubiquitous computing. Computer, 34(8):57–66, 2001. [24] Jeffrey Hightower, Barry Brumitt, and Gaetano Borriello. The location stack: A layered model for location in ubiquitous computing. In WMCSA ’02: Proceedings of the Fourth IEEE Workshop on Mobile Computing Systems and Applications, page 22, Washington, DC, USA, 2002. IEEE Computer Society. [25] Ken Hinckley. Synchronous gestures for multiple persons and computers. In UIST ’03: Proceedings of the 16th annual ACM symposium on User interface software and technology, pages 149–158, New York, NY, USA, 2003. ACM Press. [26] Ken Hinckley, Jeff Pierce, Mike Sinclair, and Eric Horvitz. Sensing techniques for mobile interaction. In UIST ’00: Proceedings of the 13th annual ACM symposium on User interface software and technology, pages 91–100, New York, NY, USA, 2000. ACM Press. [27] Felix Hupfeld and Michael Beigl. Spatially aware local communication in the raum system. In IDMS ’00: Proceedings of the 7th International Workshop on Interactive Distributed Multimedia Systems and Telecommunication Services, pages 285–296, London, UK, 2000. Springer-Verlag. [28] Hiroshi Ishii and Brygg Ullmer. Tangible bits: towards seamless interfaces between people, bits and atoms. In CHI ’97: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 234–241, New York, NY, USA, 1997. ACM Press. 92

[29] J. F. Kelley. An empirical methodology for writing user-friendly natural language computer applications. In CHI ’83: Proceedings of the SIGCHI conference on Human Factors in Computing Systems, pages 193–196, New York, NY, USA, 1983. ACM Press. [30] Mari Korkea-aho. Context-aware applications survey. In Internetworking Seminar, Spring 2000, 2000. [31] Gerd Kortuem, Christian Kray, and Hans Gellersen. Sensing and visualizing spatial relations of mobile devices. Proceedings of the 18th annual ACM symposium on User interface software and technology, 2005. [32] M. Lamming and M. Flynn. Forget-me-not: intimate computing in support of human memory. In Proceedings FRIEND21 Symposium on Next Generation Human Interfaces, 1994. [33] Yang Li, Jason I. Hong, and James A. Landay. Topiary: a tool for prototyping location-enhanced applications. In UIST ’04: Proceedings of the 17th annual ACM symposium on User interface software and technology, pages 217–226, New York, NY, USA, 2004. ACM Press. [34] Sue Long, Rob Kooper, Gregory D. Abowd, and Christopher G. Atkeson. Rapid prototyping of mobile context-aware applications: the cyberguide case study. In MobiCom ’96: Proceedings of the 2nd annual international conference on Mobile computing and networking, pages 97–107, New York, NY, USA, 1996. ACM Press. [35] Natalia Marmasse and Chris Schmandt. Location-aware information delivery with commotion. In HUC, pages 157–171, 2000. [36] Jos´e Pascual Molina Mass´o, Jean Vanderdonckt, and Pascual Gonz´alez L´opez. Direct manipulation of user interfaces for migration. In IUI ’06: Proceedings of the 11th international conference on Intelligent user interfaces, pages 140–147, New York, NY, USA, 2006. ACM Press. [37] Miguel A. Nacenta, Dzmitry Aliakseyeu, Sriram Subramanian, and Carl Gutwin. A comparison of techniques for multi-display reaching. Proceedings of the SIGCHI conference on Human factors in computing systems, 2005. [38] Jason Pascoe. The stick-e note architecture: extending the interface beyond the user. In IUI ’97: Proceedings of the 2nd international conference on Intelligent user interfaces, pages 261–264, New York, NY, USA, 1997. ACM Press. [39] Jason Pascoe. Adding generic contextual capabilities to wearable computers. In ISWC ’98: Proceedings of the 2nd IEEE International Symposium on Wearable Computers, page 92, Washington, DC, USA, 1998. IEEE Computer Society. [40] Jason Pascoe, Nick Ryan, and David Morse. Human computer giraffe interaction hci in the field. In Workshop on Human Computer Interaction with Mobile Devices, 1998. [41] Jason Pascoe, Nick Ryan, and David Morse. Issues in developing context-aware computing. In HUC ’99: Proceedings of the 1st international symposium on Handheld and Ubiquitous Computing, pages 208–221, London, UK, 1999. Springer-Verlag. [42] Shwetak N. Patel, Jun Rekimoto, and Gregory D. Abowd. icam: Precise at-a-distance interaction in the physical environment. Proceedings of Pervasive 2006, 2006. 93

[43] Thai-Lai Pham, Georg Schneider, and Stuart Goose. Exploiting location-based composite devices to support and facilitate situated ubiquitous computing. In Handheld and Ubiquitous Computing: Second International Symposium, HUC 2000, 2000. [44] Shankar Ponnekanti, Brian Lee, Armando Fox, Pat Hanrahan, and Terry Winograd. Icrafter: A service framework for ubiquitous computing environments. In UbiComp ’01: Proceedings of the 3rd international conference on Ubiquitous Computing, pages 56–75, London, UK, 2001. Springer-Verlag. [45] Jun Rekimoto. Multiple-computer user interfaces: A cooperative environment consisting of multiple digital devices. Lecture Notes in Computer Science, 1998. [46] Jun Rekimoto, Yuji Ayatsuka, Michimune Kohno, and Hauro Oba. Proximal interactions: A direct manipulation technique for wireless networking. Proceedings of INTERACT 2003, 2003. [47] Jun Rekimoto and Masanori Saitoh. Augmented surfaces: a spatially continuous work space for hybrid computing environments. In CHI ’99: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 378–385, New York, NY, USA, 1999. ACM Press. [48] Bradley J. Rhodes. The wearable remembrance agent: A system for augmented memory. In Proceedings of The First International Symposium on Wearable Computers (ISWC ’97), pages 123–128, Cambridge, Mass., USA, 1997. [49] N. Roussel, H. Evans, and H. Hansen. Mirrorspace: using proximity as an interface to video-mediated communication, 2004. [50] Enrico Rukzio, Karin Leichtenstern, Vic Callaghan, Paul Holleis, Albrecht Schmidt, and Jeannette Chin. An experimental comparison of physical mobile interaction techniques: Touching, pointing and scanning. Proceedings of the eighth international conference on Ubiquitous Computing, 2006. [51] Enrico Rukzio, Andreas Pleuss, and Lucia Terrenghi. The physical user interface profile (puip): modelling mobile interactions with the real world. In TAMODIA ’05: Proceedings of the 4th international workshop on Task models and diagrams, pages 95–102, New York, NY, USA, 2005. ACM Press. [52] Nick Ryan, Jason Pascoe, and David Morse. Enhanced reality fieldwork: the contextaware archaeological assistant. In Computer Applications in Archaeology, 1997. [53] Daniel Salber, Anind K. Dey, and Gregory D. Abowd. The context toolkit: aiding the development of context-enabled applications. In CHI ’99: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 434–441, New York, NY, USA, 1999. ACM Press. [54] Bill Schilit, Norman Adams, and Roy Want. Context-aware computing applications. In IEEE Workshop on Mobile Computing Systems and Applications, Santa Cruz, CA, US, 1994. [55] P.J. Stappers, J. Adriaanse, and I. Keller. User-centered design with and for virtual reality environments. International Journal of Human-Computer Studies, 2000. 94

[56] Norbert Streitz, Thorsten Prante, Carsten R¨ocker, Daniel Van Alphen, Carsten Magerkurth, Richard Stenzel, and Daniela Plewe. Ambient displays and mobile devices for the creation of social architectural spaces: Supporting informal communication and social awareness in organizations. In K. O’Hara et al. (eds.), Public and Situated Displays: Social and Interactional Aspects of Shared Display Technologies, pages 387–409. Kluwer Academic Publisher, 2003. [57] Sriram Subramanian and Wijnand IJsselsteijn. Survey and classification of spatial object manipulation techniques, 2000. [58] Peter Tandler, Thorsten Prante, Christian M¨ uller-Tomfelde, Norbert Streitz, and Ralf Steinmetz. Connectables: dynamic coupling of displays for the flexible creation of shared workspaces. In UIST ’01: Proceedings of the 14th annual ACM symposium on User interface software and technology, pages 11–20, New York, NY, USA, 2001. ACM Press. [59] Brygg Ullmer and Hiroshi Ishii. The metadesk: models and prototypes for tangible user interfaces. In UIST ’97: Proceedings of the 10th annual ACM symposium on User interface software and technology, pages 223–232, New York, NY, USA, 1997. ACM Press. [60] G. M. Voelker and B. N. Bershad. Mobisaic - an information system for a mobile wireless computing environment. Proc. Workshop Mobile Computing Systems and Applications, pages 185–190, 1994. [61] R. Want, B. N. Schilit, N. I. Adams, R. Gold, K. Petersen, D. Goldberg, J. R. Ellis, and M. Weiser. An overview of the PARCTAB ubiquitous computing experiment. IEEE Personal Communications, 2(6):28–33, Dec 1995. [62] Roy Want, Andy Hopper, Veronica Falc˜ao, and Jonathan Gibbons. The active badge location system. ACM Trans. Inf. Syst., 10(1):91–102, 1992. [63] Roy Want and Bill Schilit. Guest editors’ introduction: Expanding the horizons of location-aware computing. Computer, 34(8):31–34, 2001. [64] Jie Yang, Weiyi Yang, Matthias Denecke, and Alex Waibel. Smart sight: A tourist assistant system. In ISWC ’99: Proceedings of the 3rd IEEE International Symposium on Wearable Computers, page 73, Washington, DC, USA, 1999. IEEE Computer Society. [65] Ka-Ping Yee. Interaction techniques and applications for peephole displays. In CHI ’03: CHI ’03 extended abstracts on Human factors in computing systems, pages 636– 637, New York, NY, USA, 2003. ACM Press. [66] Ka-Ping Yee. Peephole displays: pen interaction on spatially aware handheld computers. In CHI ’03: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 1–8, New York, NY, USA, 2003. ACM Press.

95