TACT: Mobile Terminal for Session Manipulation

TACT: Mobile Terminal for Session Manipulation M. Kohno, Y. Ayatsuka and J. Rekimoto Interaction Laboratory, Sony Computer Science Laboratories, Inc. 3-14-13, Takanawa-Muse Bldg., Shinagawa-ku, Tokyo, 141-0022, Japan {mkohno, aya, rekimoto}@csl.sony.co.jp ABSTRACT Wireless networking is becoming anytime and everywhere, but it is still hard to communicate between computing objects. If a user can easily make a connection between the user’s terminal and surrounding computers, the user can temporarily utilize the computers’ resources, including a display or a camera. There are many projects tackling this problem, but the user operation issue still remained unsolved. This paper describes TACT, a newly created mobile device as a general interaction tool for ubiquitous computing environments. It is dedicated to operate various kinds of sessions, e.g. voice telecommunication and file transfers between remote/surrounding computers. A user with TACT can manipulate the endpoints of a certain session. This paper also shows a system that provides easy-to-use operations to start a videoconference. It clarifies that transferring the addresses of surrounding computers through an established voice session is useful to realize interaction in ubiquitous computing environments. Keywords: IP phone, endpoint manipulation, ubiquitous computing, VoIP, FEEL Phone.

1 INTRODUCTION Wireless networking is becoming anytime and everywhere, but it is still hard to communicate between computing objects. For example, we already have a number of computing objects around us. Many people always carry mobile phones and some people often carry notebook computers. A mobile phone and a notebook computer are both wireless networking objects, but require a cable and nuisance operations to connect them. In addition, what kinds of data they can exchange is strictly limited; for instance, they cannot even synchronize the current time. In a ubiquitous computing environment, it will become essential that a user can temporarily utilize the resources of surrounding embedded computers, including a display or a camera. In such case, it is important that the user can easily make a connection between the user’s terminal and those computers and operate them. However, it is troublesome that the user has to input or choose the network address of the corresponding computer. Because it may be hard to know such addresses if many computing objects are avaialbe. Instead, it is familiar for users not to take care of underlying network technology, such as IP addresses.

Figure 1: A snapshot of TACT Our goal is to develop fundamental technology and a concrete device that can realize the above features. Towards this end, we created TACT as a general interaction tool for ubiquitous computing environments (Figure 1). It is a newly created mobile device and dedicated to operate various kinds of sessions, including audio, video and data transmission between remote/surrounding computers. We designed TACT as a replacement for present mobile phones so that people can always carry it around. Whenever a user wants to utilize computing resource located nearby the user, the user can open a session and manipulate its endpoints without caring about network addresses.

2

RELATED WORK

Our work is essentially based on Rekimoto’s “pick-anddrop” [1] technique. Originally it was an intuitive user interface to transfer a file between different computers. We extended this idea by incorporating telephone metaphor [2]. Therefore this paper focuses on the details of integration into a small mobile device and appliances. There are several research projects on session management in ubiquitous computing environments. A.Harter et al proposed BatTeleporting [3]. It redirects a X-Window console to the nearest display of a user detected by the ActiveBat system. Redirection of a window session is imple-

mented by using Virtual Network Computing (VNC). While it provides a simple operation for redirection, it is hard to do more complex operations. Because the ActiveBat only has two buttons and no LCD screen. In addition, the destination of a session teleport is decided only by location. Thus if there are more than two candidates near the user, to repeated Badge button presses are still needed. In our work, we incorporate alternative, but intuitive user actions to target a candidate. The Aware Home [4] is a set of projects that aims at creating a home environment that is aware of its occupant’s whereabouts and activities. It contains some context-aware sub-projects. Q.T.Tran made a system that helps a resident remember past activities [5]. Since it is a video based system, the resident does not need carry anything to be supported. This approach is effective in enhancing human memory, but it cannot make any operation easier. In other words, the resident can remember or learn knowledge by the system, but the entire operation procedure is not affected. In order to integrate controls of audio sessions into a single mobile device, IP-phone is an adequate option. There are many projects that exploiting IP-phone (e.g. [6-8]). They are focusing on how to utilize context information, such as location and presence. On the other hand, this paper focuses on providing an easy way to open connections to surrounding computers by integrating intuitive user interaction techniques. Note that past research projects of universal remote control (e.g. Pebbles [9]) focused on how to render various control panels on a small LCD screen. They lacked how to initially connect to a device in a user-friendly manner. In terms of integrating audio and data session, desktop conference tools (e.g. NetMeeting and Polycom) and several instant messenger applications are already available. They provide high-quality IP-based audio/video connections, and they can launch another session to share data. But they can share document only at the computer on which those applications are running. On the other hand, our approach is trying to share document by exploiting surrounding computers. As networking aspects, there are a number of protocols proposed for mobile ad-hoc networks. This paper focuses on software platforms that support context-aware systems. A.K.Dey et al proposed a context-based software infrastructure solution for smart environments [10]. They provided a uniform support for building and executing context-aware applications. We adopt their ideas and UPnP [11] for realizing target discovery existing in a connection link and communication among them.

3

DESIGN OF SESSION MANIPULATOR

In this section, features indispensable in a session manipulation device are discussed. We designed TACT in

consideration of the following two issues: (1) providing easy ways to make a connection between computing objects, and (2) integrating session control functions into a portable communication device like a mobile phone.

3.1 Requirements Considering the above issues, we decided the basic features of TACT as the following: 1.

Portability: it can be used as a replacement of present mobile phones. 2. Close-range communication: a user can specify the connection device collocating to the user without care of network addresses. A user can operate the computing resources of the connected device, and can transfer files from/to the device. 3. Remote communication: a user can transfer audio, data files and presence information with a remote user. 4. Session control: a user can also generate a session that is subordinate to an already connected session. Note that a user may transfer private data though an established connection. Therefore, even though the user can easily start a session, it must be secure so that nobody can snoop the session packets.

3.2

Design Principles

In order to satisfy these requirements, we decided the following design principles. To realize the second requirement, we adopt infrared (IR) communication for initial connection establishment. Communications via IR signals require that a sender and a receiver physically locate on line-of-sight. Thus we can consider that they physically collocate when they can exchange their identities via IR signals. By using this condition as a trigger to open a connection, a user no longer need to input the destination address; the user only need to bring the TACT closer to the target IR receiver. We adopted several input/output devices onto TACT. When it connects to a surrounding computing object, it becomes a dedicated remote control by downloading the menu and event table from the object. What will happen when an input device is activated (e.g. a tact switch is pressed) depends on the current event table. In addition, an object may be a conventional PC. If TACT can become its remote mouse, a user is assured of at least minimum operations on the PC. Therefore, we mounted a mouse pointer on TACT and defined a packet format to transmit mouse and keyboard events. TACT can be used as a tool to transfer a file from one to another. A user can pick up a file and drop down to the destination object without consideration of their network names (DNS hostnames or IP addresses). We provide two

VoIP (H.323)

protocol engine

user interface (Qt/ Embedded)

infrared transceiver

session control

access ticket / content manager

speaker microphone

wireless LAN

infrared

FLASH ROM

buttons pointer

LCD

Appliance mouse/kbd event handler

infrared transceiver

4

IMPLEMENTATION

This section describes the implementation of TACT and a typical appliance that can connect to TACT.

4.1

TACT

According to the design principles described in the previous section, we created TACT, a prototype handheld ter-

protocol engine

TACT ID (in frared) messages over 802.11b

Figure 2: Hardware and software components of TACT methods to transfer a file. The first one is the straightforward way, that is, sending the real contents of a file between TACT and a connected target. Another one is to only send/receive a small fragment that represents resource information. The real data is directly transferred to a device when it tries to access the resource. For privacy protection, we defined an access ticket that represents access control information of a resource. It contains the resource location (URL), its expiration date and the identities of the accessible hosts. It also contains the message integrity check value signed by the owner device of the resource. As the size of an access ticket is independent of the real file size, a user can carry many tickets with a limited amount of memory. As an approach to the third requirement, we utilized one of H.323 VoIP implementation. Since the H.323 protocol defines a packet type that can transport any arbitrary string message, we exploited it by encapsulating various data types into this packet type. We defined an internal message format encapsulated into the string field. For the last requirement, we built a mechanism that allows TACT to exchange an endpoint address of the current close-range connection, via an active VoIP session. It allows us to pick up, drop down and create a session endpoint. This mechanism enables to create a subordinate session to an already established VoIP session.

applicat ion manager

TACT

Figure 3: Components of an appliance minal. Figure 1 is a snapshot of TACT. The overall software and hardware components are illustrated in Figure 2. We used Dragonball MX.1 196MHz as CPU, 64MB SDRAM, 32MB FLASH ROM. As a network interface, 802.11b wireless LAN is used. An IrDA device is mounted for ID transmission in a close-range. Since this device can also transmit traditional remote control signals, we can control commercial electrical devices, such as TV and VCR. As input devices, four push buttons, an eight-direction pointer and a jog dial are mounted. A small LCD screen, a pair of speakers and a microphone are also mounted. As an extended feature, we adopted a USB host interface for wired connection. It allows us to connect various kinds of commercial peripheral devices, such as a digital camera and a printer. From the software viewpoint, we used Linux as the operating system of TACT because we can utilize a number of software resources in the world. We wrote several device drivers for its dedicated hardware devices. We used Qt/Embedded as the application framework because we can write programs without depending on the underlying hardware platform. In order to run Qt/Embedded on TACT, we wrote some hardware adaptor classes. We wrote the VoIP component by using OpenH323 [12] library. It is running as a single process and communicates with other components by a UNIX domain socket. It makes or hangs up a phone call when the protocol manager requests. It is also listening to an incoming call and notifies an event to the protocol engine when a call arrives. It is

1: audio sess ion 3: exchange addresses

2a: proximal sessions

2b

4: video session

Figure 5: Connection sequence of starting a videoconference session by using TACT Figure 4: Opening a connection with TACT responsible for sending and receiving of special packets encapsulated in H.323 string message packets.

4.2

Appliance

We create several prototype appliances by using Windows PCs. The visibility of a PC itself depends on the type of an appliance because each has its own external appearance. An appliance system consists of four components: event handler, application manager, infrared transceiver and protocol engine. Figure 3 illustrates the relationship of the components. The protocol engine is responsible for communication with TACT. It recognizes the type and body of a packet from TACT and dispatches other components according to the type. The infrared transceiver component opens an infrared device and waits for a TACT ID. When it detects a signal, it tries to convert it to a network address by querying to an address server, which is responsible for the mapping table of TACT ID and network address. The mapping table is automatically maintained since TACT registers itself to the server when it becomes active and when its network configuration has changed. When the protocol engine receives a mouse/keyboard event message, it requests the mouse/keyboard event handler to emulate the corresponding event. The handler converts the request to a Windows message. The application manager component maintains session information. When TACT drops an access ticket, the protocol engine requests it to start an appropriate application program. It then launches a new process according to the content type of the ticket. In our current implementation, the filename extension of a resource location is used.

4.3 Example Use Case Figure 4 illustrates an actual use of TACT. In this figure, a user opens a connection between TACT and an appliance (APP) by bringing it closer to the IR receiver of APP. When a connection is established, it downloads a menu and an event table from APP and renders the menu on the LCD. Since APP is a picture browser, the user can scroll and magnify pictures by rotating the jog dial. When the user picks up a picture by pushing a button, APP generates an access ticket and gives it to TACT. Then the user can drop the ticket into another appliance (e.g. a wall projector) in a difference place to show the picture. Note that during a connection is established, white LEDs of both TACT and APP are blinking synchronously. We implemented this feature to visualize invisible wireless connections.

5 TACT AND VIDEOCONFERENCE Sometimes we use a videoconference system, such as NetMeeting and Polycom, to office work. When the number of participants is not small, a large screen is used to display the peer’s scene. Since such system tends to be placed at a fixed, shared location (e.g. meeting room), the address book stored in the system might not be useful. Therefore the most troublesome operation in its usage is initiating a new video session. We have observed many users saying the addresses of conference systems to each other using mobile phones. They often take more than five minutes to set up a session. This problem is easily solved with TACT. Figure 5 illustrates the sequence of this task. When a user wants to begin a videoconference, user at first makes a phone call with TACT (a). The user then looks for a conference system having a display and a camera, and opens a connection to the system by physically bringing the terminal closer (2a

of current mobile phones can transmit data through IR ports, a user has to choose a deeper menu item to use the function, and the number of available operations is limited. In addition, since entire data is transmitted via IR, it requires much amount of time to keep devices on line-of-sight. As a result, the possibility of failure becomes very high. On the other hand, we focused on how easy a user can operate surrounding resources. TACT only transmits its ID via IR. Since every other communication is done via wireless LAN, we no longer need keep on line-of-sight for long time.

6.2

Sessions

We provide two ways to close a session. We have the close menu item displayed on the LCD. When a user chooses it, the current connection is disconnected. Another way is more interesting. If a user opened a videoconference by using a VoIP connection (as described in the previous section), when the VoIP connection is closed, the videoconference session automatically closes. Since TACT manages its session information, all subordinate sessions are immediately closed when the master session is closed. This is another advantage of our work from past researches. From a usability viewpoint, each endpoint of an appliance should provide only small number of services, just like a “stationary.” Otherwise a user has to choose a menu on a small screen. Alternatively, incorporating the idea proposed by Holland [13] will make the user’s choice easier. Figure 6: Snapshots of starting a videoconference and 2b). When the connection is established, the terminal immediately becomes a dedicated remote control of the system and a menu appears on the LCD screen. The menu contains “start videoconference,” which transfers the address of the conference system to the peer TACT with which the user is talking. The address information is encapsulated into a string message of H.323 and transmitted (3) and sent to the conference system. The systems then establish a video session (4). Therefore the user can begin a videoconference only by simple actions (pushing a jog dial button and a single click for confirmation.) Figure 6 shows a pair of snapshots of this scenario.

6 6.1

DISCUSSIONS

VS. Mobile Phones

As described in the introductory section, exploiting various kinds of surrounding computing resources is one of key features of ubiquitous computing. Even though some kinds

6.3 Hardware Considerations At this moment, the most difficult problem relevant to the TACT hardware is its battery life. We can use only one and a half hour. We are planning to improve it toward a battery conscious system. Shih’s work [14] is helpful for this issue. Mounting a camera on TACT is interesting improvement. It allows us to take a picture and immediately send to a friend, your notebook or a nearby printer with simple operations. Since TACT has only four buttons, it is difficult to input text. We are thinking to create an attachable keypad.

7 CONCLUSIONS A mobile terminal, which is designed as a general interaction tool for ubiquitous computing environments, has been introduced. The concept of a session manipulator been described and its design principles have been discussed. The first prototypes of TACT and example appliances were shown. We think that the videoconference system described in this paper is a suggestive demo about the essence of TACT. We have built another kinds of interactive systems control-

lable with TACT. With these systems, a user can easily control multiple sessions like a “conductor.” As our future direction, we are going to establish a security model in ubiquitous computing environments. The access ticket model described in this paper will be effective, what kind of information should be included in a ticket must be much more considered.

ACKNOWLEDGMENT First of all, we are grateful to Haruo Oba for the appearance design of both TACT and the proximity sensor devices. We are also grateful to all the members in Wearable Solution Division, Sony Corporation for hardware design and implementation of TACT. We are also thankful to Satoko Nagara for proofreading our manuscript. We are grateful to all the other members of the Interaction Laboratory, Sony CSL.

REFERENCES [1] J. Rekimoto. Pick-and-Drop: A Direct Manipulation Technique for Multiple Computer Environments. In the 11th ACM Symposium on User Interface Software and Technology (UIST'98), pp. 31-39, 1998. [2] Michimune Kohno and Jun Rekimoto. New Generation of IP-Phone Enabled Mobile Devices. In 4th International Symposium on Human Computer Interaction with Mobile Devices (MobileHCI2002), pp. 319323, September 2002. [3] A. Harter, A. Hopper, P. Steggles, A. Ward and P. Webster. The Anatomy of a Context-Aware Application. In Proceedings of MOBICOM’99, pp. 59-68, September 1999. [4] D.J. Moore, I.A. Essa and M.H. Hayes III. Exploiting Human Actions and Object Context for Recognition Tasks. In Proceedings of IEEE International Conference on Computer Vision, pp. 80-86, September 1999. [5] Q. Tran, K. Truong, E. Mynatt and G.Abowd. What Was I Cooking? Part of the “As We May Live” Demo invited to ACM2001, March, 2001. [6] Paul H. Dietz and William S. Yerazunis. Real-Time Audio Buffering for Telephone Applications. In Proceedings of the 14th ACM Symposium on User Interface Software and Technology (UIST'01), pp. 193-194, 2001. [7] Chris Schmandt, Jang Kim, Kwan Lee, Gerardo Valljo, and Mark Ackerman. Mediated Voice Communication via Mobile IP. In Proceedings of the 15th ACM Symposium on User Interface Software and Technology (UIST'02), volume 4, pp. 141-150, October 2002. [8] Stacie Hibino and Audris Mockus. handiMessenger: Awareness-Enhanced Universal Communication for Mobile Users. In 4th International Symposium on Hu-

man Computer Interaction with Mobile Devices (MobileHCI2002), pp. 170-183, September 2002. [9] Jeffrey Nichols, Brad A. Myers, Michael Higgins, Joseph Hughes, Thomas K. Harris, Roni Rosenfeld, and Mathilde Pignol. Generating Remote Control Interfaces for Complex Appliances. In Proceedings of the 15th ACM Symposium on User Interface Software and Technology (UIST'02), volume 4, pp. 161-170, October 2002. [10] A.K. Dey, G.D. Abowd and D. Salber. A ContextBased Infrastructure for Smart Environment. In Proceedings of 1st International Workshop on Managing Interactions in Smart Environments (MANSE’99), pp.1-6, 1999. [11] Microsoft Corporation. Understanding Universal Plug and Play: A White Paper. http://www.upnp.org/, 2000. [12] OpenH323 Project. H.323 standards. http://www.openh323.org/, 1998. [13] Simon Holland, David R. Morse, and Henrik Gedenryd. Direct Combination: A New User Interaction Principle for Mobile and Ubiquitous HCI. In 4th International Symposium on Human Computer Interaction with Mobile Devices (MobileHCI2002), pp. 108-122, September 2002. [14] Eugen Shih, Paramvir Bahl and Michael J. Sinclair. Wake on Wireless: An Event Driven Energy Saving Strategy for Battery Operated Devices. In Proceedings of 8th International Conference on Mobile Computing and Networking (MOBICOM’02), September 2002.