Interaction and Visualisation across Multiple Displays in Ubiquitous Computing Environments Hannah Slay1,2,3 Bruce Thomas1,2 1
2
e-world Lab University of South Australia Mawson Lakes, SA 5095, Australia + 61 8 8302 3125
CRC for Enterprise Distributed Systems Technology University of South Australia Mawson Lakes, SA 5095, Australia + 61 8 8302 3464
3
Telkom Centre of Excellence Rhodes University Grahamstown, 6139, South Africa + 27 46 603 8623
Figure 1 Virtual representation of work environment
Abstract
1. Introduction
This paper describes the Universal Interaction Controller (UIC), a user interface framework and device designed to support interactions in ubiquitous computing environments, and the in-situ visualisation of ambient information in environments equipped with multiple heterogeneous displays. We describe the device and the infrastructure we have created to support it. We present the use of augmented reality to display information that is outside the bounds of traditional display surfaces.
Ubiquitous computing encompasses the notion of computing being woven into the backdrop of natural human interaction (Weiser 1993). The key step in the move towards these computing environments is a shift from the use of traditional workstations as a primary computing interface, to environmental or workspace interfaces. In these environments the traditional input devices of mice and keyboards will be augmented with more human-centric approaches. This project investigates new interaction and display techniques for ubiquitous computing environments that are equipped with multiple, heterogeneous displays.
Categories and Subject Descriptors H5.1 [Information interfaces and presentation]: Multimedia information systems; Artificial, augmented and virtual realities; H.2 [Information interfaces and presentation]: User Interfaces. Graphical user interfaces; User Interfaces. - Input devices and strategies
When ubiquitous computing environments were first proposed, a group of researchers at PARC in 1988 (Weiser 1991) recommended Virtual Reality (VR) as a type of environment that fulfilled all the properties of a ubiquitous computing environment. This was due to the fact that VR seeks to replace humans sensory and affecter systems (Rheingold 1991), in essence making the computer invisible to the user. In 1991, Weiser coined the notion of a ubiquitous computing environment, and criticised this belief, proposing that ubiquitous computing is in fact most diametrically opposed to VR (Weiser 1991). Instead of integrating computers into human activity so seamlessly that users do not realise they are actually interacting with a computer, VR focuses an enormous amount of computational power and physical apparatus on simulating rather than invisibly enhancing the world that already exists. In doing so, VR excludes offices, furniture and people who are not actively involved in the virtual world from the ubiquitous computing
General Terms Design, Human Factors. Keywords Interaction model, ubiquitous computing, interaction device, multiple display environment
Copyright © 2006 by the Association for Computing Machinery, Inc. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from Permissions Dept, ACM Inc., fax +1 (212) 869-0481 or e-mail
[email protected]. AFRIGRAPH 2006, Cape Town, South Africa, 25–27 January 2006. © 2006 ACM 1-59593-288-7/06/0001 $5.00
75
be used on a single display. A key feature is that the device is designed to be mobile in the room, and not fixed to a tabletop surface. The UIC is a physical device (referred to as Ukey), an interaction manager, and clipboard manager, which together provide seamless access and control of data across multiple displays. We envision the use of traditional keyboards and table-top mice for standard workstation activities such as word processing, and the UIC provides the user with the ability to manipulate data across multiple heterogeneous displays for activities such as select, copy, and paste. Figure 2 shows a user interacting with the UIC across multiple displays. Using the device, the user points at a display to select it, and then uses their stylus to perform fine grained interaction with the selected display. AR is used to support the visualisation of information across the new spaces opened up by ubiquitous computing environments where it can no longer be guaranteed that the user will be able to view information using traditional display techniques. To visualise room information such as connections between data on different display devices, it is proposed that the UIC can be used as a window to the virtual world. Figure 3 shows the prototype of this visualisation technique that has been implemented on a tablet PC. The figure depicts edges of a graph structure that extend outside the bounds of the physical wall mounted displays and laptop computer on the table.
environment (Weiser 1991). Even with the recent advances in computer technology, VR still cannot produce a real-time simulation at a reasonable cost that is photo-realistic enough to fool users into believing they are interacting in a different world. The computer is therefore not out of the way, but VR acts as a layer of abstraction between the user and the real world. Weiser was so critical of this comparison that he even referred to ubiquitous computing as “embodied virtuality” to illustrate that it draws computers out of their electronic shells and moves the techniques for altering computer readable data to the physical world (Weiser 1991). A related technology to VR is Augmented Reality (AR) that seeks to enhance the user’s reality, while still allowing them to interact with objects in the physical world. AR therefore does not suffer from the same problems as VR with respect to ubiquitous computing environments. AR can be seen to blur the visual and logical boundaries of physical computer spaces. Information no longer belongs to particular discrete computing devices, but extends to cover an environment.
Figure 2: User interacting across displays with UIC A strength of ubiquitous workspaces is the variety of display and interaction modalities that they support, which allows the user to select the best suited device for a specific classes of application. Our project investigates appropriate interaction devices to control, manipulate, and visualise information across a large range of display devices, applications and tasks in the context of command and control centres (Bright 2004). In particular, this project investigates the use of information visualisation in such environments (Slay, Thomas et al. 2001; Slay and Thomas 2003). To date, a limited number of interaction techniques for ubiquitous computing environments have been reported. The techniques that have been proposed are discussed in the related work section, and focus more on control of the devices in the environment themselves rather than the data stored on them. Users are often forced to use traditional interaction techniques which are not suited for multi-display, collaborative environments.
Figure 3: Room Visualisation technique This remainder of this paper is separated into four sections. The next section will provide an overview of related work in the fields of human computer interaction within ubiquitous computing environments. After this we then describe the UIC and the infrastructure that we created to support it. Following this is an overview of the AR visualisation technique that we created to provide an in-situ visualisation of information within ubiquitous computing environments. The final section provides some concluding remarks and future work.
This paper presents the Universal Interaction Controller (UIC) as our proposed solution, created in particular to support interaction in ubiquitous computing environments. The UIC is intended to allow more intuitive control of data across multiple displays, whilst still being accurate enough to
2. RELATED WORK We are investigating interaction techniques and technologies to support ubiquitous computing environments. Project
76
access to menu options. The third mode allows a user to “snarf” the text from a screen, and displays it for edit on the PDA
Oxygen at MIT has a focus on combining speech and vision to interact within intelligent spaces to such an extent that they use no keyboards or mice in their environment (Brooks, Coen et al. 1997). Instead they rely on tracking head, eye and arm movements and positions, along with the capture of speech from microphone arrays in the environments to allow users to interact intuitively with the room.
Richardson, Stafford-Fraser, Wood, and Hopper suggest the use of Virtual Network Computing to provide an image of the remote display in a window on the users desktop to allow users to control multiple displays (Richardson, StaffordFraser et al. 1998). To control multiple computers, the user simply switches between windows. For this technique to be useful there is a requirement of a large display window. The display on a handheld device for example would not be able to provide adequate resolution or granularity to allow control of a large screen display.
The Gaia project at the University of Illinois aims to create an operating system for physical spaces that are equipped with displays and devices (Román and Campbell 2000). To interact in these environments with heterogeneous devices, Gaia applications are required to provide interfaces that can be transformed to suit the requirements of PDAs, laptops and large screen displays.
The XWand (Wilson and Shafer 2003) is a 6DOF wireless pointing device; the novel application of this device is the selection gesture control of physical objects in a user’s environment. For example by pointing at a lamp, users can either perform a gesture to turn it on or off, or press the button on the handle of the wand to toggle its state. To employ the XWand, a model of the environment must be created by pointing at each of the devices to be controlled and creating a Gaussian blob with mean and covariance values for each area of interest. As well as allowing users control of devices in their environment, the XWand provides a cursor control function that allows users to take control of the cursor of the display device they are pointing at. Because the XWand is an absolute 6DOF device, to perform fine grained manipulation of the cursor position, the user is forced to hold their arm relatively still for long periods of time, causing them discomfort and muscle fatigue. The XWand device is extended in the WorldCursor (Wilson and Pham 2003) system that uses a laser spot instead of the geometric model previously used to determine object selection in an environment This system supports interaction with multiple displays in an environment, but similarly to XWand suffers from arm fatigue, hand jitter and latency introduced by direction detection. The designers of WorldCursor found that a drawback of their system was that control of small displays from a distance was difficult because of the small “mousing surface”.
Stanford’s Interactive Room Operating System (iROS) provides a different operating system level approach to the design of ubiquitous computing infrastructure (Johanson 2002; Johanson, Fox et al. 2002). The main communication mechanism in iROS is the event heap (a tuplespace), a mechanism by which multiple users, machines and applications can all simultaneously interact as consumers and generators of system events. Publishing and subscribing to the event heap is sufficient to participate in the room infrastructure while allowing components to remain very loosely coupled. The iROS data heap provides for more long termed storage of data. The data is keyed with an arbitrary number of attributes, and data is retrieved by a query specifying attributes that must be matched. By using attributes instead of locations, the data can be distributed over a number of physical devices. Numerous techniques have been proposed to support interaction across displays. A key field that provides similar techniques is AR / VR for selection of objects at a distance. Although similar, these techniques are not included in this discussion as they are more concerned with interaction with the remote object itself rather than controlling the input of a remote machine (Bowman and Hodges 1997). PebbleDraw is a drawing application from Carnegie Mellon (Myers, Stiel et al. 1998) that utilizes techniques for sharing a display between multiple participants. By connecting their PDAs to a shared display, users can either take turns controlling the one shared cursor, or have control of their own cursors. This investigation focused more on one display collaboration rather than controlling multiple displays.
A novel input technique that supports the input of position and selection events is manual and gaze input cascaded (MAGIC) pointing (Zhai, Morimoto et al. 1999). MAGIC uses a combination of eye tracking for coarse grained position input, and manual device control for fine grained position and selection input.
A related technique is Semantic Snarfing (Myers, Hong Peck et al. 2001) that allows users to create a copy of a portion of a large display onto their PDA. This technique is named after the popular UNIX term snarf that describes the action of acquiring a file or set of data across a network. Semantic Snarfing can operate in three modes: screen capture, menu snarfing, and text snarfing. In the first mode, the remote screen can be captured as an image. When viewing an image of the remote screen, stylus movements can be mapped directly to mouse movements on the remote display, or sent to the remote PC and drawn using a transparent overlay, allowing the user to annotate a screen. The second mode allows a menu to be “snarfed” from the remote PC and displayed on the PDA. This allows a user more fine grained
Pointer redirection techniques map discrete displays into a continuous display; a logical mapping between displays describes how screens are connected together. Once the screen topology has been created, the user may move their cursor to the edge of one display, and the cursor and mouse movements are automatically forwarded onto the next logical display. Gaia supports a pointer redirection system called Clicky (Andrews, Sampemane et al. 2004), and iROS uses an application called PointRight (Johanson, Hutchins et al. 2000). Both Clicky and PointRight provide the similar functionality for their respective environments. However, Clicky and PointRight only provide an intuitive solution for a
77
Figure 4 depicts the UIC architecture in terms of the major software components and how an application is connected to the UIC. The architecture is split into computing device specific components and ubiquitous services for the workspace. Each computing device is able to communicate to the Event Heap and Data Heap and has a set of applications. The UIC supports the following two major functions for an application: 1) replacing the cursor control with a portable device for multiple heterogeneous displays and 2) clipboards suitable for ubiquitous workspaces. In Figure 4 a number of computing devices (Computer 1…n) are depicted, each with clipboard suitable applications and legacy applications. KegLite represents where user developed applications are placed in the architecture. All application communication for cursor control is through MS Windows and one Interaction Client for each computing device; therefore the cursor control functionality operates with all Windows applications. The clipboard control is through the Clipboard Manager, and this functionality must be known to the application to operate properly. Each computing device may have multiple displays, and these displays are abstracted behind the MS Windows system. The Clipboard Manager listens for events from the PDA Client and other instances of KegLite, and uses these to copy and paste data between computers.
linear arrangement of displays, where connections between displays may only be specified between edges of physical display devices. Also, a reasonable assumption is for ubiquitous workspaces to be equipped with mobile devices such as laptops that may be moved around the environment, rendering the logical mapping illogical. With AR, an optical combiner is required to superimpose the computer generated images onto the view of the physical world. Two techniques are used (Rolland, Holloway et al. 1994) to combine the physical and virtual images: optical and video. Some head mounted displays (HMDs), referred to as optical see-through HMDs, combine the physical and virtual images together by allowing the user to see the physical world through semi-transparent displays. Virtual objects are overlayed onto the real world by projecting the images onto half silvered mirrors. The other technique used is to capture a video of the physical world, overlay the computer generated images onto it within the computer and display it back to the user as another video stream, referred to as video seethrough. Although the introduction of video processing adds a delay of the final presentation of information, an advantage of this technique is its ability to create a more convincingly merged physical world (Rolland, Holloway et al. 1994). The resultant video stream is often displayed back to the user on a HMD, but a recent trend has been to display it back on a tracked PDA or other hand held device (Bertelsen and Nielsen 2000; Beier, Billert et al. 2003; Wagner and Schmalstieg 2003). PDAs are typically used to display the AR system to the user when they are interacting with colocated users in collaborative environments. A known problem with HMDs is that they cover the user’s eyes, concealing important visual cues such as gaze and facial expressions that humans use when undertaking face to face interactions with other people.
3. UNIVERSAL INTERACTION CONTROLLER Figure 4: UIC Architecture
Our ubiquitous workspace is built from the LiveSpaces project, an overarching project that is addressing how physical spaces such as meeting rooms can be augmented with a range of display technologies, personal information appliances, speech and natural language interfaces, interaction devices and contextual sensors to provide for future interactive/intelligent workspaces (Bright 2004). The LiveSpaces environment is fitted with heterogeneous displays including rear projected and front projected wall displays, large plasma screens, laptops, and LCD panels for the display of ambient information. The LiveSpaces software infrastructure has been created to form the basis for integrating, controlling and coordinating activities and technologies within these future workspace environments. The environment is equipped with 6DOF tracking facilities to determine the position and orientation of marked objects in the environment (Slay, Vernik et al. 2003; Slay, Thomas et al. 2004). Software components in the LiveSpaces infrastructure are written in Java, C++, C, and Python. The LiveSpaces project builds on iROS as one of its primary communication mechanisms between distributed components.
The following ubiquitous workspace services for cursor control are depicted in Figure 4: Interaction Manager, VeWorld, PDA client, and Track Devices. The Interaction Manager has two operations, selecting the display for cursor control and redirecting cursor control. The Interaction Manager listens for cursor control commands from the PDA Client, along with the selected display from Ve-World and posts interactions task to the Interaction Client for the appropriate computing device. The PDA client monitors pen cursor events from the physical PDA that forms the basis of the Ukey and posts these events onto the event heap. VeWorld subscribes to 6DOF position and orientation (pose) data of the Ukey from Track Devices service. From this pose information, Ve-World posts which display the Ukey is pointing at onto the event heap. The Track Devices service provides a tracker abstraction for the 6DOF tracking infrastructure of the LiveSpace. Ve-World is a Java3D based application that is used to both view a dynamically updated representation of a work
78
• Quantify The user specifies a numerical quantity. Minimal requirements: value, and the device the quantity should be sent to.
environment and to monitor for screen selection events. It receives pose data from the Tracking Device service. When a collision occurs between a virtual ray that extends from the top of the Ukey in the direction that it is pointing and a display, an event is posted on the event heap that records the IP address of the currently selected display device. This application will be discussed in more detail in Section 4.
• Text The user specifies an alphanumeric value. Minimal requirements: value, and the device the text should be sent to. Implementation. The Interaction Manager is designed to mediate interactions between disparate devices, accepting input from heterogeneous modalities and translating it into a uniform output that can be understood by traditional client machines. This requires a model of the interaction space as defined in the previous section, so that all input can be classified and forwarded onto the correct client.
The UIC consists of three major components that work together to provide the seamless access and control of data across multiple displays: Interaction Manager, Clipboard Manager and the physical device referred to as Ukey. Each of these components will now be discussed further.
3.1 Interaction Manager
The interaction manager was implemented with a clientserver architecture, where the client runs on each computer to be controlled and the server is run on a centralised computer. The server subscribes to the event heap for events showing the currently selected display (as monitored by Ve-World application) and the output of the Ukey. The selected display is used to determine which display to forward the user interaction tasks to. The server then maps the input task to one of the tasks specified in the interaction model and sends an event over event heap to .event with a field that states which interaction task needs to be performed.
The Interaction Manager is designed to mediate the interactions that occur between applications on a set of computing devices and forward them to the correct recipients in order to allow different devices to control other displays in a ubiquitous workspace. We define a model of the interaction space to ensure that support can be provided for all interactions that take place in the environment. This section provides an overview of the interaction model, followed by the implementation details for our environment. Interaction Model. An earlier version of this model was proposed in (Slay and Thomas 2003), but after additional work, this model has been extended to its current format. Our interaction model is based on a model developed by Foley, Wallace, and Chan (Foley, Wallace et al. 1984), which was derived by studying interactive graphics systems and categorizing their observations into six fundamental types of interaction. The environments Foley, Wallace, and Chan studied consisted of single two dimensional displays, where input is provided by one or two devices per screen. Our model extends the initial model to cover interaction in ubiquitous computing environments with the following characteristics: three dimensional, collaborative, multi display and multi modal input. We extended the following six interaction tasks from their model to our ubiquitous workspace interaction model:
On the client side, each Interaction Client subscribes to the event heap for events of type .event where specifies the device’s own hostname. The client is written to provide an instantiation for each of the interaction tasks for the Windows operating system. As an example, if a select event is received, this would be performed on the client side by firing two left mouse button clicks. If a position event is received, this would result in a mouse movement event being fired. The current implementation of the interaction client supports the select, position, path, quantify, and text tasks. Orient is not supported because native support for changes in orientation is not provided by the Windows operating system.
3.2 Clipboard Manager
• Select The user selects one or more objects from a list of alternatives. Minimal requirements: object or list of selected objects and the device that the selection should take place on.
Foley et al. defined the following four controlling tasks that directly modify the object being displayed: stretch, sketch, manipulate and shape (Foley, Wallace et al. 1984). These four tasks are appropriate for single display graphical based applications, and we define two new controlling tasks: copy and paste that are important to ubiquitous computing environments. The remainder of this section provides an overview of our clipboard model to support these two new tasks for ubiquitous computing environments, along with implementation details within our environment.
• Position The user specifies a position of a selected item in 1, 2, or 3 dimensions. Minimal requirements: dimensionality, position, and the device the position should take place on. • Orient The user specifies a new orientation of a selected item with 1, 2 or 3 degrees of freedom. Minimal requirements: degrees of freedom, orientation, and the device the orientation should take place on.
Clipboard Model. Our Clipboard Model has previously been reported (Slay, Thomas et al. 2005), but this section provides an overview. This model defines a clipboard as a repository that is used to temporarily hold data to be transferred between devices, windows, or applications.
• Path The user specifies a list of positions and orientations over time with 1 – 6 degrees of freedom. Minimal requirements: degrees of freedom, position or orientation, time, and the device the path should take place on.
The reference points from which the repositories can be defined are the infrastructure, and an infrastructure is defined
79
whereas the UIC relies on a physical mapping between the screens. The UIC approach creates an added requirement of a tracked environment, but it is considered a tolerable tradeoff for intuitive control of multiple heterogeneous displays.
as a computer, set of applications, and a set of displays. We initially investigated three different infrastructures: personal, walk-up, and ubiquitous. Personal infrastructures move with the user, such as smart badges and PDA’s. When the personal infrastructure is in use, the distance between the user and the devices is on the scale of 0 – 2 metres. Walk-up infrastructures are fixed repositories such as information kiosks and automatic tellers. Walk-up infrastructure is typically used over distances of 0 – 10 metres. Ubiquitous infrastructures are computing enhanced physical spaces, such as intelligent meeting rooms. Ubiquitous infrastructures typically operate over distances of 1 – 1000 metres. Implementation. The Clipboard Manager is implemented as a Java API to support Copy and Paste tasks for applications in a ubiquitous workspace, and supports the transfer of the following forms of data: files, byte arrays, and arrays of serialized objects. The methods defined in the API transfer the data to and from the application and a uniquely identified clipboard. Clipboards are identified by a unique attribute pair of type and name of the clipboard. The type of clipboard is one of the following categories: device, user, application, computer, walk-up display, or ubiquitous infrastructure. The name identifies the clipboard to allow multiple clipboards of a particular type to be created.
Figure 5: Ukey handheld device The XWand takes an approach that can be seen to be a combination of the Clicky / PointRight approach and the UIC approach. Like the UIC, the XWand allows users to select a display by pointing at it. As with Clicky / PointRight, the XWand works on a lower level than the Ukey, being concerned primarily with wand (mouse) movements and button presses instead of the interaction task that is being performed. The XWand is the only physical device supported by its architecture. The architecture that supports the UIC has been designed to be generic enough to support interaction and control within the environment from a number of different devices, and the Ukey is just one instantiation. Unlike the XWand, the pose of the Ukey is only used for coarse grain selection of objects (such as large screen displays, laptop screens), allowing the user more limb movement. This increase in movement allows the user to reposition fatigued limbs, whilst still providing control of the desired object.
Copy tasks require the following data: a type, a name, and an object to store. After receiving this data, the clipboard manager creates metadata describing the object and then associates the object with the metadata. It then places both the metadata and the object onto the data heap. After the object has been successfully added to the data heap, the clipboard manager posts a message to the event heap notifying all interested parties of the addition. An event is added to the event heap to notify clients of clipboard additions because the event heap allows users to subscribe for events that match a template and the data heap does not. Although this technique involves replication of data, it provides a simple method for automatic notification of clipboard changes.
As shown in Figure 5, the Ukey is a tracked object that allows users to interact with displays by pointing at them. By 6DOF tracking of the displays, a dynamically updated model can be created, showing the position of displays in the environment. The goal of the system was to optically track displays in the environment (laptops and wall displays) and the Ukey using the Passive Detection Framework (PDF), an optical based 6DOF tracking facility we created for work volumes (Slay, Vernik et al. 2003; Slay, Thomas et al. 2004). Because of current accuracy limitations, this is not a feasible solution. Instead, the Ukey is tracked with a Polhemus tracker which provides far more accurate tracking results over small distances, and the other displays are placed in known locations. The future works section of this paper will explain our approach to overcome the limitations to the optical tracking solution. A user study has been undertaken comparing user’s performance with the Ukey, the leading off the shelf wireless pointing device, and a traditional mouse. The results of this study are presented in (Slay and Thomas 2006).
Paste tasks require the name and type of clipboard they require information from. This information is combined into a template, and the data heap is queried for all results that match the template. The default behaviour of the clipboard manager is to return the most recently added object that matches the search criteria, and there is a second method to return all matches to a user supplied template. The clipboard manager was first implemented as a service for the iROS framework. This allowed it to dynamically advertise the methods that it could perform, allowing clients to discover it and then invoke its methods over the event heap. Because of performance limitations however, the current version of the manager was created that can be referenced from within a program.
3.3 Ukey The UIC, Clicky, and PointRight all share the aim of creating a continuous mapping between discrete displays in an environment. Clicky and PointRight create a logical mapping between screen positions and connections between displays,
The hardware and software aspects of the Ukey will now be discussed separately
80
refers to the position of the cursor. When the Copy button is pressed, the currently selected object on the currently selected display is sent to the clipboard manager and assigned to the Ukey’s clipboard. Finally, a visual representation of the object is added to the paste container.
Hardware. The Ukey employs an IPAQ 5450 with a Polhemus sensor mounted on an arm, 20 cm from the top of the device. The sensor was mounted this distance from the PDA for two main reasons: visual cues to the user of pointing direction and magnetic interference reduction. The PDA is composed of ferrous components, and it creates magnetic interference to the tracker’s sensor. After brief experimentation, it was found that the sensor was required to be mounted at least 10cm from the PDA to obtain accurate position and orientation data.
The paste container consists of 12 clipboard containers and one Clear all Clipboards button. The Ukey can be set to use a device based or a session based clipboard. When using a device based clipboard, the information copied using the Ukey is stored in a clipboard with name unisa.eworld.ipaq.5450 and type device clipboard. When using a ubiquitous infrastructure clipboard, any information copied in the environment will be stored in a clipboard with name unisa.eworld.lab and type ubiquitous infrastructure. With this clipboard type, any information copied within the workspace can be viewed and pasted using the Ukey. The current implementation of the device only provides 12 clipboard containers. This is not a limit on the total number of objects that can be copied, but the number that can be displayed at one time and is due to the small screen size of the Ukey. When objects are copied, they are placed in the next empty clipboard container. If all containers are full, the object is used to overwrite the clipboard container that has the oldest timestamp. To paste an object displayed in a copy container, the user selects a destination display device with an up and down pointing gesture. Once the display is selected, the user taps on the copy container that contains the information to be pasted. Finally, the Clear all Clipboards button sets the contents of each of the clipboard containers to be empty, and sets the next available container to the first container.
Software. The GUI for the Ukey was implemented in Java for the Crème JVM for Windows Mobile 2003. As shown in Figure 6, the user interface of the UIC consists of two primary containers: the modify container (left image) and the paste container (right image). Both containers are controlled with three buttons and one label that are clearly visible on the interface: Modify button, Paste button, Exit button and the selected display field.
Figure 6: Modify and paste containers of Ukey The Modify button activates the modify container, and the Paste button activates the paste container as shown in Figure 6. The Exit button closes the application. The selected display field indicates the name of the currently selected display. This information is updated by subscribing to the event heap for all events showing the currently selected display. All pen movements or button presses are automatically sent via the Interaction Manager to the computing device associated with the currently selected display.
4. VE-WORLD As previously mentioned, Ve-world is used to graphically monitor for screen selections. It is also used to visualise information that extends through the boundaries of display devices within an environment. This section discusses the two functions provided by this application.
4.1 Screen Selection
The modify container consists of four widgets: pen control area, Select button, Move button, and Copy button. In the pen control area, stylus movements on this area are translated into cursor movement messages being sent to the Interaction Manager. The Interaction Manager then forwards these messages on to the appropriate display, in the form of a path interaction task detailing the relative movements of the stylus over time. When the Select button is pressed, a select task message is sent to the Interaction Manager, who then forwards a message to the appropriate Interaction Client telling it to perform a select task. The Move button is a modal button that when pressed once, maps all movements of the users stylus on the pen control area to a movement of the currently selected object on the currently selected display. When pressed a second time, the mode is switched to stop move and all further stylus movements on the pen control area are mapped to movements of the selected display’s cursor. If an object is currently selected, the Interaction Manager sends a path request specifying the new position for the currently selected object. Otherwise the path request
Two sets of information are required to monitor for screen selections: the pose of all displays within an environment; and the pose of the selection tool. Within our environment, objects are coarsely tracked (3+ cm accuracy) using the Passive Detection Framework (PDF), an ARToolkit based framework (Kato and Billinghurst 1999) that we created to determine the pose of marked objects within an environment. To avoid the occlusion issues that traditional optical based tracking techniques face, the PDF uses multiple roof mounted cameras in order to view an environment from multiple angles. Using the PDF, fiducial markers are attached to the objects (including wall mounted displays and laptops) to be tracked and their pose is calculated and posted to the iROS event heap for consumption by other applications. Refer to (Slay, Vernik et al. 2003) for more details on the framework itself, and (Slay, Thomas et al. 2004) for an overview of the rapid configuration and adaptability of the PDF.
81
ReferenceSystem field set to a value of absolute provide the transformation of the fiducial marker in absolute coordinates, so when an event of this type is received the position of the virtual environment can be changed to equal the new position. Events with ReferenceSystem field to a value of relative calculate the transformation of the object with respect to the transmitter so are in a relative coordinate system. Before a position update can be made with this event, the event must be transformed into the absolute coordinate system by multiplying the 6DOF position and orientation of the tracker transmitter (as specified in the XML configuration file) by the 6DOF position and orientation specified in the event. This was performed to minimize the amount of replicated information in the system, and therefore make it more efficient to reconfigure the system when the room is rearranged.
Ve-World is a Java3D based application that is used to both view a dynamically updated representation of a work environment and to monitor for screen selection events. It takes an XML file describing the configuration of a work environment, and the mapping between fiducial markers and their virtual counterparts and creates a Java3D model of the scene. The fiducial marker mapping depicts the name of the fiducial marker, the IP address or hostname of the machine it is attached to, and a brief description that details the size of the device and the type of device (for example whether the device is a laptop, a PDA, a large screen display). Virtual representations of devices are added to the scene, updating their positions when an event is received from the PDF over the iROS event heap. The Ukey pose is also updated through a subscription to tracker events. When a collision occurs between the virtual representation of the Ukey and a display, an event is sent to the iROS event heap containing a unique identifier for the display. In the case where each computer is attached to one display, this field contains the IP address or hostname of the computer. When more than one display is attached to one computer, the IP address or hostname of the computer is concatenated with a number to allow each display to be uniquely identified. For example, if a laptop with hostname knoxville.levels.unisa.edu.au is configured to use an external display, the laptop can be uniquely referred to as knoxville.levels.unisa.edu.au:1 and the external display can be uniquely referenced as knoxville.levels.unisa.edu.au:2. Figure 1 shows a screen capture of the Ve-World program, showing the extent of the model. For a new environment, a simple model of the room is required to be created, showing basic features such as walls, floors and doors. All items within the room that users want modelled can then have a fiducial marker attached to them, and as long as a model can be associated with each marker, the room and all of its components can be dynamically tracked. Room components that users wish to be modelled but not tracked can by included in the virtual world by specifying their position, size and choosing from a list of generic descriptions of office furniture and devices. The list of currently modelled components is: file cabinet, desk, table, end table, cabinet, whiteboard, smart board, plasma screen, tracker, PDA, and laptop. All of these components except the PDA can be seen in Figure 1. Figure 7 shows a closer view of the scene, including a model of a PDA.
Figure 7: Ve-World screen capture showing selection of display Figure 7 shows a screen capture of the Ve-World system. In this figure, one SmartBoard, two plasma displays, one laptop, one large screen display and a Ukey (as a PDA) can be seen. A set of axis extend from the roof showing the position of the origin and axis of the Polhemus transmitter. In this figure, the large screen display on the right hand side is being selected by the Ukey. As described earlier, in the current configuration of the e-World Lab, the Ukey is tracked using the Polhemus tracker so it is added to the scene graph as a child of the tracker rather than of the room itself, allowing the scene graph to determine the position of the Ukey in room coordinates.
This modelling process is beneficial in particular when room components such as desks, computers and large screens are rearranged to suit the changing needs of the work environment’s occupants. Although it is not necessary to add any objects to the virtual environment except for those displays that the user wishes to control, other objects provide users with a more complete spatial reference system to interact within.
As can be seen from Figure 7, the model of the Ukey includes a virtual ray that extends along the length of the right hand side of the device depicting the pointing direction. A display is selected when a collision occurs between the virtual ray and one of the modelled displays in the environment. To reduce muscle fatigue, a display remains selected until a collision occurs between the virtual ray and another display, allowing the user to reposition their arms whilst still having control of the display. Once a selection has taken place, an event is posted to the iROS event heap showing the currently selected display. The remainder of this section describes the
As well as providing a dynamically updated view of the work environment, Ve-World is also used to determine if a user has selected a display. As input, Ve-World subscribes for all new events from the PDF. Each of these events has an associated device ID that refers to one of the display devices or the Ukey. Pose information is either specified as an absolute position or a relative position. All events with the
82
implementation of the Ve-World application as a novel AR visualisation technique within IIWEs.
4.2 AR Visualisation The Ve-World application allows users to visualise room information that is not associated with a particular computer or device. Currently this is implemented for use in two different configurations: in VR on a standard desktop computer viewed through a HMD; and in AR viewed on a tracked tablet. Figure 3 shows a user viewing the environment with a tracked tablet PC as a window to the virtual world. The tablet allows them to view information that extends across the bounds of one display in situ, such as the connections between objects on different machines. In this figure, the user is viewing a connection between a laptop, and a front-projected whiteboard. As she moves the tablet PC around the environment, the viewpoint in the virtual world is updated to the tablet’s new pose. Figure 8 shows a screen capture of the Ve-World application showing connections between display contents. These connections are dynamic and are anchored to positions on each display device. If the laptop shown in Figure 8 was tracked and was moved about the environment, the model of the laptop and its connections to the large wall displays would be automatically resized to demonstrate these changes.
Figure 8 Ve-World visualisation
5. CONCLUSION AND FUTURE WORK This paper has presented our Ukey to support interaction and control in a ubiquitous computing environment with multiple heterogeneous displays. It has detailed the components that constitute the UIC: the physical device, the interaction manager and the clipboard manager. This paper also presents a novel visualisation technique for ubiquitous computing environments. By using the UIC as a window to a virtual world, the device can provide the user with a visualisation of information that extends through the bounds of traditional display devices. This information is presented on the UIC as opposed to a traditional head mounted display as it reveals the user’s eyes to other collocated participants in the environment, an important visual cue in collaboration.
A handheld device was chosen as the base platform when VeWorld is used in collaborative environments in order to ensure that the participant’s eyes are not obscured from other co-located participants. HMDs are a useful display device as they are head mounted and therefore do not require the user to hold anything in their hands, but when used in collaborative environments they obscure useful communication cues such as eye gaze and facial expressions (Mogilev, Kiyokawa et al. 2002).
We are currently investigating improvements to the tracking infrastructure, the PDF, to make it accurate enough to be used as the sole tracking facility in the environment. Using a new calibration program to determine the distortion of each camera’s lens, our previous version of PDF was accurate enough to track an object’s position to 3cm, but our current version that is based on using stereo cameras and can perform with an accuracy of 1cm over a distance of 1.25m
We are currently investigating the porting of this application to the Pocket PC platform. This configuration would enable users to interact with their Ukey as usual, and when they want to view information that is not attached to a particular display they could hold their Ukey in front of their eyes and view the virtual world through the 6DOF tracked PDA screen. From a hardware perspective, this is not a difficult step. There already is a pre-requisite for the Ukey to be tracked, so this pose information could be re-used and mapped to move the viewpoint around the virtual environment, as demonstrated with the tablet PC application. Researchers from the Studierstube project (Wagner and Schmalstieg 2003) have demonstrated that an unmodified PDA can be used to execute and display feedback from a stand-alone, self-tracking, AR application. This demonstrates that there are no hardware or software problems involved in transforming the Ve-World application to the Pocket PC platform.
6. ACKNOWLEDGMENTS Some of the work reported in this paper has been funded in part by the Co-operative Research Centre for Enterprise Distributed Systems Technology (DSTC) through the Australian Federal Government's CRC Programme (Department of Education, Science, and Training). The remainder of the work was undertaken under the auspices of the Centre of Excellence in Distributed Multimedia at Rhodes University, with financial support from Telkom, Business Connexion, Verso, Comverse and THRIP.
83
RICHARDSON, T., Q. STAFFORD-FRASER, K. R. WOOD and A. HOPPER (1998). "Virtual network computing." IEEE Internet Computing 2(1): 33. ROLLAND, J., R. HOLLOWAY and H. FUCHS (1994). "A comparison of optical and video see-through headmounted displays." SPIE: Teleconferencing and Telepresence Technologies 2351: 293-307. ROMÁN, M. and R. H. CAMPBELL (2000). Gaia: enabling active spaces. ACM SIGOPS European Workshop 2000. SLAY, H. and B. THOMAS (2003). An Interaction Model for Universal Interaction and Control in Multi Display Environments. International Symposium on Information and Communication Technologies, Trinity College Dublin, Ireland. SLAY, H. and B. THOMAS (2006). "Evaluation of a Universal Interaction and Control Device for use within Multiple Heterogeneous Display Ubiquitous Environments." to Appear in Conferences in Research and Practice in Information Technology 40. SLAY, H., B. THOMAS and R. VERNIK (2005). A Clipboard Model for Ubiquitous Computing Environments. Winter International Symposium on Information and Communication Technologies, Cape Town, South Africa. SLAY, H., B. THOMAS, R. VERNIK and M. PHILLIPS (2001). "Interaction Modes for Augmented Reality Visualisation." Conferences in Research and Practice in Information Technology 9. SLAY, H., B. THOMAS, R. VERNIK and W. PIEKARSKI (2004). A Rapidly Adaptive Collaborative Ubiquitous Computing Environment to Allow Passive Detection of Marked Objects. Asia Pacific Conference on Computer Human Interaction, Rotorua, New Zealand. SLAY, H., R. VERNIK and B. THOMAS (2003). Using ARToolkit for Passive Tracking and Presentation in Ubiquitous Workspaces. Second IEEE International Workshop on Augmented Reality Toolkit, Waseda University, Japan. WAGNER, D. and D. SCHMALSTIEG (2003). First steps towards handheld augmented reality. Proceedings. Seventh IEEE International Symposium on Wearable Computers. WEISER, M. (1991). "The computer for the 21st century." Scientific American 265(3): 66-75. WEISER, M. (1993). "Some computer science issues in ubiquitous computing." Communications of The ACM 36(7): 74-84. WILSON, A. and H. PHAM (2003). Pointing in Intelligent Environments with the WorldCursor. Interact 2003, Zurich, Switzerland. WILSON, A. and S. SHAFER (2003). XWand: UI for intelligent spaces. CHI'03, Ft. Lauderdale, FL, United States, Association for Computing Machinery. ZHAI, S., C. MORIMOTO and S. IHDE (1999). Manual and gaze input cascaded (MAGIC) pointing. CHI 1999, Pittsburgh, PA, USA, ACM, New York, NY, USA.
7. REFERENCES ANDREWS, C. R., G. SAMPEMANE, A. WEILER and R. H. CAMPBELL (2004). Clicky: User-centric Input for Active Spaces. Urbana-Champaign, University of Illinois. BEIER, D., R. BILLERT, B. BRUDERLIN, D. STICHLING and B. KLEINJOHANN (2003). Marker-less vision based tracking for mobile augmented reality. Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. BERTELSEN, O. W. and C. NIELSEN (2000). Augmented Reality as a Design Tool for Mobile Interfaces. Symposium on Designing Interactive Systems, Brooklyn, New York, USA. BOWMAN, D. and L. HODGES (1997). An Evaluation of Techniques for Grabbing and Manipulating Remote Objects in Immersive Virtual Environments. Symposium on Interactive 3D Graphics, Providence, Rhode Island, ACM Press. BRIGHT, D. (2004). "LiveSpaces: An Interactive Ubiquitous Workspace Architecture for the Enterprise." Embedded and Ubiquitous Computing: 982-993. BROOKS, R. A., M. COEN, D. DANG, J. DE BONET, J. KRAMER, T. LOZANO-PEREZ, J. MELLOR, P. POOK, C. STAUER, L. STEIN, M. TORRANCE and W. MICHAEL (1997). The Intelligent Room Project. Second International Conference on Cognitive Technology, Los Alamitos, CA, USA., IEEE Computing Society. FOLEY, J., V. WALLACE and P. CHAN (1984). "The Human Factors of Computer Graphics Interaction Techniques." IEEE Computer Graphics and Applications 4(11): 13-48. JOHANSON, B. (2002). Application Coordination Infrastructure for Ubiquitous Computing Rooms. Electrical Engineering. Stanford, Stanford University: 211. JOHANSON, B., A. FOX and T. WINOGRAD (2002). "The Interactive Workspaces Project: Experiences with Ubiquitous Computing Rooms." IEEE Pervasive Computing 1(2): 67-74. JOHANSON, B., G. HUTCHINS and T. WINOGRAD (2000). PointRight: A System for Pointer/Keyboard Redirection Among Multiple Displays and Machines. Stanford, Stanford University: 1-2. KATO, H. and M. BILLINGHURST (1999). Marker Tracking and HMD Calibration for a Video-based Augmented Reality Conferencing System. International Workshop on Augmented Reality, San Francisco USA. MOGILEV, D., K. KIYOKAWA, M. BILLINGHURST and J. PAIR (2002). AR Pad: A face-to-face AR Collaboration. ACM Conference on Human Factors in Computing Systems, Minneapolis, ACM Press. MYERS, B., C. HONG PECK, J. NICHOLS, D. KONG and R. MILLER (2001). Interacting at a Distance Using Semantic Snarfing. Third International Conference on Ubiquitous Computing, Atlanta, Georgia. MYERS, B. A., H. STIEL and R. GARGIULO (1998). "Collaboration using multiple PDAs connected to a PC." ACM Computer Supported Collaborative Work: 285-294. RHEINGOLD, H. (1991). Virutal Reality. New York, USA, Summit Books.
84
