A Context-Dependent Environment for Web Application Migration

A Context-Dependent Environment for Web Application Migration Giuseppe Ghiani, Fabio Paternò

Jussi Polet, Ville Antila, Jani Mäntyjärvi

CNR-ISTI, HIIS Via Moruzzi 1 56124 Pisa, Italy

VTT Technical Research Centre of Finland Kaitoväylä 1 90571 Oulu, Finland

{giuseppe.ghiani, fabio.paterno}@isti.cnr.it

{jussi.polet, ville.antila, jani.mantyjarvi}@vtt.fi

ABSTRACT

1. INTRODUCTION

The growing amount of interactive devices and displays around us opens up new possibilities for creating multi-device collaborative environments, enhancing task continuity and migration across devices. The challenges in current multi-device user interface migration environments include sub-optimal techniques for target selection and process initiation. We argue that these aspects can be enhanced by increasing the context-awareness of the multidevice environments.

Recent studies [5] show that many users have different devices available at home as well as at work. People are thus increasingly accessing applications through diverse types of interactive devices, depending on their current location and needs.

In this paper, we present a multi-device environment supporting automatic migration of Web applications across interactive devices, according to the user’s context of use. The environment exploits a Web-based platform, which manages migration from the source device to the target one, and an embedded context monitoring mobile application, which detects relevant context parameters and triggers the migration of the Web application running on the source device to the target device, according to predefined rules. The context parameters and rules can be selected and configured according to the requirements of the migration use case, such as to detect the current location, the presence of the target device and source device stability.

Categories and Subject Descriptors H.5 Information Interfaces And Presentation; H.5.2 User Interfaces.

General Terms Design, Human Factors.

Keywords Migratory Web User Interfaces, Context-awareness, Multi-device Environments, Continuity.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. AVI ‘12, May 21-25, 2012, Capri Island, Italy Copyright © 2012 ACM 978-1-4503-1287-5/12/05... $10.00

Typical example applications accessed in different places and with heterogeneous devices are the Web ones. In order to support users in such ubiquitous interactions, we have designed and developed an integrated solution for migrating Web applications according to information related to the context of use, such as location and physical activity. With Web applications migration we mean migrating the front end of the application (i.e. the Web page and its state information) across different devices. The back end, instead, lies always in the original application server. Migration of Web applications is managed by a Web-based platform that relies on a proxy server for real-time annotation of existing Web pages. The proxy injects additional scripts on the navigated pages that enable migration on request, while preserving the original functionalities of the Web application. Context monitoring allows triggering the migration automatically, by retrieving values from diverse sources, such as embedded sensors, and applying them to custom rules. The starting point for obtaining an integrated environment for context-based migration was a previously developed multi-device and multi-user Migration Platform [6] supporting explicit (i.e. manually triggered) migration. Such platform was evaluated through a usability test, and the participants reported their appreciation for the proposed techniques, though they recommended some enhancements in order to improve the overall support. Among such recommendations, there was a need to ease the strategy for triggering the migration, i.e. to make the migration more automatic and intelligent. In order to facilitate migration, and particularly for mobile users, we have thus opted for providing the possibility of a fully automated way to generate triggers based on the context-of-use information. This approach suits well for that need to exploit their technological offer while freely moving, especially when accessing applications that require long sessions. Thus, besides providing task continuity to the user, the automatic migration trigger eases and fastens the whole migration task and minimizes the user effort.

In this work we provide the motivations, architecture overview and some use cases for the context-based migration environment. In particular, in the following sections, after citing related work, we provide a couple of example scenarios for context-based migration, describe the integrated architecture of our environment and give some final remarks.

2. RELATED WORK Dearman and Pierce [5] have focused on the techniques people use to access multiple devices and found that one of the main challenge is how to support seamless device change. Support for migratory user interfaces aim to provide useful techniques for users accessing information through multiple devices. Wäljas et al. [12] have also studied how people access various multi-device applications during a few weeks, and have proposed an initial framework for analyzing cross-platform service user experience, which is composed of three main aspects: composition (whether the service provides appropriate system composition), continuity (fluency of content and task migration), and consistency . There has been earlier research exploring the space of interaction techniques and enabling technologies for sharing information and session between devices in multi-device environments. For example, Pick-and-Drop [9] was a first interesting solution to support the dynamic graphical selection of application elements in one device and easily move them in another device. However, that approach was mainly limited to move data across devices while people would like also to move entire interactive applications. Berry et al. [2] deal with view sharing in cooperative work environments, and tackle the privacy issues of a presenter which wants to show/hide parts of the view according to the type of audience. The solution focusses on collaborative tasks, while in this paper we focus on single-user scenarios in which our platform can be useful. Deep Shot [3] is a solution for automatic migration of user interfaces across devices with state preservation. Migration of an interface, or part of it, is triggered by “shooting” it with a mobile device camera. In our platform, migration is triggered in a fullautomated way without requiring explicit user interaction. Also, the authors of Deep Shot state that this approach is compatible even with native applications, but that existing ones need to be modified in order to support the deep shooting/posting. Our platform for Web migration instead relies on a proxy that injects all the needed support to the navigated pages, and does not require any additional software or browser plugin on client side. Enabling technologies for communication, such as RFID and Bluetooth, can be used to sense the co-location of target devices in the proximity of the user [4, 10]. Moreover, different characteristics of proximity in interacting in ubiquitous computing environments have been identified [7]. There has been research on techniques that facilitate interaction, taking into account the proximity factors between people and computing devices, such as location (e.g. room or building), position (close to a display), movement (coming towards a display) and orientation (facing towards the display). These factors can facilitate better contextual knowledge for the system about the current situation (e.g. whether the public display should be facilitating interaction with someone or just displaying information) [1, 8]. In this paper, we demonstrate the ability to use a simple sensing application as an enabler of context-awareness in a multi-device

environment, allowing the system to automatically detect some variables of the situation, such as the location, position, identity and information about the user’s current activity (e.g. moving or still). Furthermore, we discuss the possibilities of using this information to allow the system to perform automatic or assisted migration of session from a device to another. This type of automatic trigger can be related to the work on implicit humancomputer interaction driven by the context discussed in [11], in which the system acts proactively on the basis of context information.

3. EXAMPLE SCENARIOS In the following, we present two example scenarios we aim to user test with the current developed versions of the integration of the Migration Platform and Context Monitor. We are focusing on the smartphone stability and the Bluetooth presence/proximity information.

3.1 Scenario 1 A user is walking and browsing some Web page via the Migration Client on her/his Android smartphone’s Web browser. The smartphone is running the Context Monitor as well. The Context Monitor on the smartphone is set to detect the presence/proximity (Bluetooth) of the user’s office desktop computer and the stability of the smartphone, so that when the user arrives to her/his office desk and sets the smartphone on the table, the browsed Web page is migrated from the smartphone to the desktop computer (which is also running the Migration Client), thus ensuring the continuity of the browsing experience.

3.2 Scenario 2 A user is browsing some Web page via the Migration Client on her/his office desktop computer and decides to leave. When the Context Monitor notices that the smartphone is no longer stable and still and that the office desktop computer is no longer nearby (Bluetooth), the browsed Web page is migrated from the desktop computer to the smartphone. The user arrives to a meeting room and sets the smartphone on the meeting room table. The Context Monitor recognizes that the smartphone is again still and stable and that the meeting room large screen media computer is nearby, so the Web page is migrated from the smartphone to the large screen media computer for all the meeting participants to see the Web page.

4. CONTEXT-AWARENESS IN USER INTERFACE MIGRATION During the usability test of previous solutions in multi-device UI migration, we noticed that some issues concerning the usability/user experience of the migration process are related to target acquisition (meaning the selection of target device for the migration) and process initiation (the “triggering” of UI migration process). These both aspects can be enhanced by contextawareness of the system, which means the support of a context manager able to provide relevant information regarding the users, the devices, the surrounding environment, and social relations. The benefits of such context-awareness in UI migration are: 

Automatic target selection – using information about the environment we can arrive at a decision about the most preferred device as the target device for the migration;



Automatic or suggested process initiation – based on the sensed situation it is possible to either trigger a migration process automatically (implicit migration) or to suggest the possibility of migration to a nearby device (assisted migration).

5. UNDERLYING ARCHITECTURE The main modules of the architecture that enables context-based migration are the devices involved in the migration with the migration client running on them, the migration server, and the context monitor that should run on the involved devices as well.

5.1 General Architecture for Migration In order to be involved in a context-based migration, the source (originating) and the target (destination) device must run the Migration Client, which is a simple Web application for logging in the environment and for navigating the Web via the Migration proxy server. In the source device the Migration Client is typically already running, since it is used to navigate the pages via the Migration proxy. However, the user has to ensure that the Migration Client is already active also in the target device in order to manage the incoming migration request when a context-based migration is triggered. It is worth considering that, since the context-based migration assumes that the involved devices are already active, having the Migration Client running in all the involved devices is a reasonable constraint.

The Context Monitor, depending on its configuration and deployment, is aware of several parameters. For instance, it might know the position of Device A, the preferences of the user, and the IDs of nearby devices. As soon as a migration is implicitly desirable (e.g., Device A gets in close proximity of Device B), the Context Monitor sends an external migration trigger to the Migration Platform (5) specifying the source and the target devices’ ID. The Migration Platform forwards the request to the Migration Client of the source device (6) which, in turn, invokes the previously injected JavaScript procedure of the navigated page. The procedure serializes the Document Object Model (DOM) of the page by including the interaction state (e.g. content of FORM elements) and sends it to the Migration Platform (7), which saves it locally. The Migration Client running on the target device is then notified (8) with the link to access the migrated page and automatically opens it on a new browser window/tab. It is worth noting that the Migration Client continuously communicates to the Migration Platform server and provides the unique ID of the device and the user credentials. Thus, since each instance of Migration Client is uniquely identified, more than one potential target device can be active (i.e. can run the Migration Client) at the same time and in the same environment. The actual target device will be the only one that would manage the incoming migration, since its ID is specified by the custom rules.

5.2 Context Monitor Overview The Context Monitor is a mobile application for Android smartphones, which runs on the background and monitors the desired context parameters. These parameters can be, for example, the proximity of other devices (Bluetooth etc.), current physical activity of the user (fall-detection, still, walking, running detected using the device accelerometer), current virtual activity of the user (currently running applications and currently used applications on the device, call and SMS activity etc.), location (GPS, WLAN, cell-tower ID) and other, such as sensor-based cues (audio, light etc.). Figure 2 illustrates these kinds of context parameters.

Cell ID, GPS, WLAN, location… Stabile/still, walking, running, face-up, facedown, activity…

Figure 1. Overview of the architecture for context-based migration. Figure 1 shows the simplified architecture for the environment and the main communications between the modules. It is assumed that Device A (i.e. smartphone) and Device B (i.e. desktop computer) have the Migration Client active, and that a Context Monitor is running. When the user navigates a Web page (1) on Device A, the request is caught by the Proxy of the Migration Platform that forwards the request to the application server (2). The application server provides back the requested resource (3) to the Proxy. The Proxy annotates the resource (e.g., the HTML page) by adding JavaScript code for enabling migration and responds to the device (4).

Presence of other devices/people, NFC, Bluetooth, social…

UI migration

Audio, Used applications

Figure 2. Illustration of context-awareness in UI migration. The desired context parameters can be chosen and configured, so that the user can set, when the Web application should be migrated from the smartphone to another device and vice versa. Thus, for example, the user can set the target devices for the migration and also the “context-dependent rules”, such as “perform migration from the smartphone when it is still, facing upwards and near my office desktop computer to the office

desktop computer” and “perform migration from the office desktop computer to the smartphone, when the smartphone is not still and the office desktop computer is not nearby”.

6. CONCLUSION AND FUTURE DEVELOPMENT We have implemented the first versions of Migration Platform and Context Monitor and tested the functionality to be working. The current version of the Context Monitor only supports the device stability (still or moving) and the proximity of another device(s) (Bluetooth) for the upcoming user test purposes. We will expand the Context Monitor to support multiple types of context parameters and to be more configurable. The Migration Platform will also be developed further to enable more parameterized external triggering, for example, triggering the external migration of only certain UI elements. We are going to perform a laboratory user test (N~20) for the environment, where the users will test the functionality in the two example scenarios presented in this paper. We will also differentiate the implicity/explicity of the migration, i.e. have three different cases in the test; 1) explicit migration, where the user will initiate the migration of the Web page on her/his own behalf, 2) assisted migration, where the user is queried (when the context parameters match), whether s/he wants to migrate the Web page and 3) implicit migration, in which the Web page is automatically migrated whenever the context parameters match.

8. REFERENCES [1] Ballendat, T., Marquardt, N. and Greenberg, S. Proxemic Interaction: Designing for a Proximity and Orientationaware Environment. In: ACM International Conference on Interactive Tabletops and Surfaces, Anonymous ACM, 121130, 2010. [2] Berry, L., Bartram, L., and Booth, K. S. 2005. Role-based Control of Shared Application Views. In: Proceedings of the 18th Annual ACM Symposium on User interface Software and Technology. UIST '05, 23-32. [3] Chang, T.H. and Li, Y. Deep Shot: A Framework for Migrating Tasks Across Devices Using Mobile Phone Cameras. In: Proceedings of CHI ‘11, ACM Press, 2011, pp. 2163-2172. [4] Cheverst, K., Dix, A., Fitton, D., Kray, C., Rouncefield, M., Sas, C., Saslis-Lagoudakis, G. and Sheridan, J. G. Exploring Bluetooth-based Mobile Phone Interaction with the Hermes Photo Display. In: Proceedings of the 7th International Conference on Human Computer Interaction with Mobile Devices & Services, ACM, 47-54, 2005. [5] Dearman, D. and Pierce, J. S. It's on My Other Computer! Computing with Multiple Devices. CHI 2008, pages 11441153. [6] Ghiani, G., Paternò, F., Santoro, C. Push and Pull of Web User Interfaces in Multi-Device Environments. To appear in: Proceedings of AVI ‘12, May 21-25, 2012, Capri Island, Italy, ACM.

We are also going to perform a diary-type of study (N~20), in which the participants will take notes of their everyday situations, in which they would find context-aware UI migration to be a useful feature. The participants will be asked to take notes of each situation, in which they would find the migration of the whole UI or just parts of it (task migration) useful and between which devices. The length of this study will be approximately two weeks, during which the participants will take notes of their ideas and thoughts about context-aware UI migration. We will also interview the participants at the end of the study.

[8] Marquardt, N., Diaz-Marino, R., Boring, S. and Greenberg, S. The Proximity Toolkit: Prototyping Proxemic Interactions in Ubiquitous Computing Ecologies. In: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, ACM, 315-326, 2011.

A further evaluation will be devoted to study how the users can benefit from the context-aware migration, and will be carried out by letting them freely exploit the system as they wish for a few days.

[9] Rekimoto, J. Pick-and-Drop: A Direct Manipulation Technique for Multiple Computer Environments. ACM Symposium on User Interface Software and Technology 1997: 31-39.

Through the user tests we aim to gain insight into the functionality of the current developed version of the environment and to get further suggestions and thoughts about context-aware Web application migration for multi-device environments.

[10] Rekimoto, J., Ayatsuka, Y., Kohno, M. and Oba, H. Proximal Interactions: A Direct Manipulation Technique for Wireless Networking. In: Proceedings of INTERACT, 2003.

7. ACKNOWLEDGMENTS We gratefully acknowledge support from the EU ARTEMIS SMARCOS Project (http://www.smarcos-project.eu).

[7] Greenberg, S., Marquardt, N., Ballendat, T., Diaz-Marino, R. and Wang, M. Proxemic Interactions: The New Ubicomp? Interactions 18, 42-50, 2011.

[11] Schmidt, A. Implicit Human Computer Interaction Through Context. Personal and Ubiquitous Computing 4(2/3), 2000, 191-199. [12] Wäljas, M., Segerståhl, K., Väänänen-Vainio-Mattila, K. and Oinas-Kukkonen, H. Cross-Platform Service User Experience: A Field Study and an Initial Framework. In Proc. of MobileHCI ‘10, ACM, 2010.