Visual separation in mobile multi-display environments

Paper Session: Mobile

UIST’11, October 16–19, 2011, Santa Barbara, CA, USA

Visual Separation in Mobile Multi-Display Environments Jessica R. Cauchard1, Markus Löchtefeld2, Pourang Irani3, Johannes Schoening2, Antonio Krüger2, Mike Fraser1, Sriram Subramanian1 1

University of Bristol Bristol Interaction & Graphics Dept. of Computer Science Woodland Rd, Bristol BS8 1UB, UK {cauchard, fraser, sriram}@cs.bris.ac.uk

2

German Research Center for Artificial Intelligence (DFKI) Stuhlsatzenhausweg 3 66123 Saarbrücken, Germany {markus.loechtefeld, johannes.schoening, krueger}@dfki.de

ABSTRACT

3

University of Manitoba Computer Science Department, Human-Computer Interaction lab Winnipeg, Manitoba, Canada. [email protected]

devices. In the case of projector phones, it is expected that the market will grow to 20 million units by 2015 [12]. Unlike traditional handsets, these devices offer a large projected display, in addition to the existing display, that can be viewed by more than one person at a time.

Projector phones, handheld game consoles and many other mobile devices increasingly include more than one display, and therefore present a new breed of mobile Multi-Display Environments (MDEs) to users. Existing studies illustrate the effects of visual separation between displays in MDEs and suggest interaction techniques that mitigate these effects. Currently, mobile devices with heterogeneous displays such as projector phones are often designed without reference to visual separation issues; therefore it is critical to establish whether concerns and opportunities raised in the existing MDE literature apply to the emerging category of Mobile MDEs (MMDEs). This paper investigates the effects of visual separation in the context of MMDEs and contrasts these with fixed MDE results, and explores design factors for Mobile MDEs. Our study uses a novel eye-tracking methodology for measuring switches in visual context between displays and identifies that MMDEs offer increased design flexibility over traditional MDEs in terms of visual separation. We discuss these results and identify several design implications.

Such mobile multi-display environments operate by providing visual information on different screens. Figure 1 illustrates two examples of projecting on the floor or the wall to create a large screen real-estate. Often the larger display allows sharing public information, while private information can be kept on the device’s screen for the owner’s eyes only, or to principally support input feedback. Additionally, two physical displays on one mobile device can afford more display real estate than a single-display mobile-device. While MMDEs can be used in both, singleuser contexts (using multiple displays to partition tasks) and multi-user contexts (e.g. for privacy setting), the design space of MMDEs needs refinement for single-user contexts before considering aspects of multi-user environments.

ACM Classification: H5.2 [Information interfaces and presentation]: User Interfaces. - Graphical user interfaces. General terms: Design, Experimentation, Human Factors. Keywords: Mobile Multi-Display Environment, Mobile

Projection, Visual Separation, Handheld Devices, Eye Tracking, Visual Search. INTRODUCTION

Clamshell phones, handheld dual-display game consoles, projector-enhanced tablet PCs and cameras are steadily increasing the number and forms of multi-display mobile

Figure 1 – Design implications: In Mobile MultiDisplay environments, displays should ideally be in the same field of view.

The existing MDE literature shows how such device ecologies are affected by the unavoidable visual separation effects caused by multiple displays [28, 29]. Simply defined, visual separation is the division of information across space in MDEs. Non-continuous presentation of information can be inefficient to interact with, if it is not

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. UIST’11, October 16–19, 2011, Santa Barbara, CA, USA. Copyright © 2011 ACM 978-1-4503-0716-1/11/10... $10.00.

451

Paper Session: Mobile

UIST’11, October 16–19, 2011, Santa Barbara, CA, USA

handled properly [5, 18]. Research in fixed multi-display environments has shown that visual separation of content can affect performance [13, 22]. Tan and Czerwinski [29] found a significant detrimental effect when dividing information across multiple displays at different depths for the same separation angle. Likewise, Su and Bailey [28] found that when positioning large displays through workspaces, the relative depth between displays can affect users’ performance.

et al. [21] present techniques using a network to link multiple single displays in order to share co-located display spaces. Cao et al. [6] present a multi-user interaction technique that allows individual handheld projectors to interact simultaneously. Finally, Siftables [24] provide a set of tangible interactive objects, each equipped with a single display that can be combined in order to manipulate data and information. They support tangible interaction effects, such as removing a physical item from a pile to delete associated virtual data. In all the described cases, each individual display can easily be moved and re-oriented depending on the desired situation. The users can then intuitively reduce visual separation effects.

On one hand, direct control over the projection space and the closeness of the display on a mobile device could mitigate the effects of visual separation on mobile MDEs. However, on the other hand, mobile devices can create conditions whereby their mobility accentuates static MDE problems. For example, mobile projector phones have an inherent depth differential between the phone’s screen and the projection. Prior work in MDEs would suggest negative visual separation effects due to this depth gap. With a lack of understanding of how visual separation affects usability and performance, it is hard to identify appropriate designs and suitable interaction techniques or adapt these devices to specific applications.

Mobile single-device-multi-display environments provide more than one display on a single mobile device. This type of environment has gained a lot of popularity with the growth of embedded pico-projectors in existing devices such as phones, cameras, camcorders and even tablet PCs. Traditionally these displays have been fixed relative to one another, such as with a mobile projector phone where the projection lens is normally fixed at an orthogonal angle to the mobile phone’s screen. Some devices, such as the Nintendo DS™ present reconfigurable hardware capabilities in between the two screens. Unfortunately, these capabilities are not currently exploited by software applications. Nonetheless, increasing numbers of singledevice-multi-display environments exploit a reconfigurable multi-display layout, as the Codex [17] where two screens are hinged and can be rearranged into different positions. Despite the possibility of re-orienting these devices, many single-device-multi-display environments do not allow the user to rearrange displays in order to simultaneously visualise information. This bears the question of whether we can immediately transfer guidelines from research on fixed multi-display contexts to mobile single-device-multidisplay environments.

Borrowing on principles derived from research in MDEs, we map out factors that could negatively impact visual separation in MMDEs. This motivated our user study which includes an innovative eye-tracking methodology to measure visual separation effects. Our principal contributions are that: 1. Visual separation does not impair the viability of MMDEs; 2. Displays should ideally be in the same field of view (Figure 1); 3. Factors such as: handling, portability and unsteadiness do not exacerbate visual separation. 4. We present some implications of our study on MMDE design. TYPES OF MOBILE MULTI-DISPLAY ENVIRONMENTS

In this section we explore existing work in MMDEs. MMDEs are either partially mobile (i.e. a mobile component imported in a traditional MDE) or fully mobile (i.e. a mobile device that supports more than one display).

FACTORS INTENSIFYING THE IMPACT OF VISUAL SEPARATION

Having reviewed existing MMDEs, in this section we review existing work on visual separation in MDEs and the visual separation challenges for single-device-multi-display mobile environments. Factors amplifying the effects of visual separation have been studied for a range of multidisplay configurations including when displays are of different sizes, when placed at different distances from the user, if oriented at different relative angles and when separated by surrounding bezels or frames.

Partially mobile MDEs include environments where a mobile device is imported inside a traditional MDE, for example Greenberg et al. [14] present an environment in which a PDA is used in conjunction with shared public displays. Since the mobile component can be flexibly reoriented relative to the existing MDE, the visual separation effects of this component in the overall environment may be mitigated by the ability to easily reorient the device. This could minimise visual separation between displays and thus current research in fixed MDEs is likely to hold in the partially mobile MDE case.

Size and Depth

Mandryk et al. [22] show that users are faster at interacting between two identical and continuous monitors compared to using a secondary monitor of smaller size placed with a small gap to the primary screen. Pointer warping techniques such as Mouse Ether [2] and frame memory pointer [4] propose cursor movement techniques that can help reduce the effects of visual separation across displays of different sizes in heterogeneous MDEs.

There are many existing examples of fully mobile MDEs in the literature, which can be divided into two categories, multi-device-single-display and single-device-multi-display. Mobile multi-device-single-display environments are created when individual single-display mobile devices are brought together to create a new MDE. For example Lyons

452

Paper Session: Mobile

UIST’11, October 16–19, 2011, Santa Barbara, CA, USA

Early literature in ergonomics [1] advises that documents and screen be kept at the same distance from the user for data-entry tasks that require rapid shifts between both elements, to reduce costs in switching views. Recently, Tan and Czerwinski [29] show a detrimental effect due to visual separation when a screen and a projector are placed at different depths within the same visual field. These negative effects can be reduced with techniques such as the Perspective Cursor [25], that remaps the ordinary mouse cursor in a complex heterogeneous MDE depending on the perspective of each user regardless of their position.

example of single-device-multi-display with 6 screens fitted in a cubic arrangement. Held in the palm, it can be moved to explore content on the 2 or 3 faces in front of the user while other faces remain hidden at the back of the cube. Bezels

In MDEs, Tan and Czerwinski [29] found no effects of visual separation due to bezels and physical distance between screens alone. Yang et al. [30] found minimal visual separation effects between Lens-Mouse (a mouse with screen on top) and the monitor. Task performance in Yang et al.’s study [30] degraded in their dual-monitor condition attributed to distance and not bezels. Contrarily, Bi et al. [5] found that splitting symbols across two displays with a bezel in the middle was detrimental in a search task. Bi et al. [5] also found that interacting with data was faster with no bezel compared to a tiled screen. Forlines et al. [13] show that for an individual user; having information split across multiple vertical screens is detrimental in terms of reaction time to accomplish a visual search task compared to a single vertical screen. Stitching [18] is an interaction technique designed to reduce visual separation effects by using a pen interface to draw interaction lines across multiple displays.

In most single-device-multi-display mobile environments, the screens used are set to have similar characteristics and dimensions, and are often at the same distance from the user (i.e. where the device is held). However, in projectorenhanced mobile devices, screens and projections vary in size and distance depending on the proximity to the projection surface. This is the case for mobile projector phones, projector-cameras or camcorders and e-book readers equipped with projectors. This category of devices contains a small personal screen and a larger projection area. Although absolute size and distance can be configured by manipulating the device, relative size and distance between displays are typically fixed and may cause visual separation effects due to angular or focal displacement.

Chen et al. [9] present a dual-display e-book reader and shows advantages of using multiple screens for reading. For example, information can be separated on both screens through the bezel for multi-document reading. Moreover, the device supports interaction techniques that draw on real books, such as moving one screen towards the other to ‘turn pages’. In addition, the screens can be detached and reassembled for different modes of use. Devices with dual screens separated by a bezel already exist, such as phones, laptops or even game consoles as the dual-screen Nintendo DSi™ or dual-touch-screen Toshiba Libretto® laptop.

Angular separation / Field of view (FOV)

Tan and Czerwinski [29] show greater visual separation effects of depth when the data is separated by a 55° angle (i.e. outside the useful FOV) compared to a 27° angle (i.e. inside the useful FOV). Su and Bailey [28] studied visual separation for multiple large displays and found negative effects when the secondary screen is situated on the same horizontal plane as the primary screen but at an angle of 70° relative to the user, at the periphery of their field of view. Their study also showed a negative effect when the second screen was completely behind the user (i.e. in a completely separate FOV); however, they found no effect when the secondary screen was oriented at an angle from the first screen and were both at the same distance from the user. Following their experiment, they presented a set of guidelines on how to position two large displays relative to each other: the displays should stay on the same horizontal plane, at no more than a 45° subtended visual angle and should not be placed behind a user; in other words both displays should stay within the user’s FOV.

MMDEs vs. MDEs

In all the above designs, MMDEs have very different characteristics to traditional MDEs. We have identified inherent size and depth gaps which create potential angular and focal separation in the case of projector-enhanced mobile devices or individual displays placed in separate fields of view such as clamshell phones. Previous research in MDEs shows that multiple screens need to be placed within the same useful FOV of the user to avoid negative effects of visual separation [28] and also that specific interaction techniques need to be applied if the size of the displays differs. Yet, MMDE designs do not necessarily follow these guidelines because the studies presume a fixed position and orientation and no or limited control over changing display placement during the task. It is therefore essential to determine whether visual separation effects previously demonstrated in fixed MDEs translate to MMDEs. In the remainder of this paper, we explore the design space for MMDEs and we determine if the negative effects of visual separation in MMDEs can be reduced by aligning displays within the same field of view.

Some single-device-multi-display environments are designed with the displays in different fields of view. For example, some clamshell phones are equipped with both an internal and an external display, such as the Samsung Alias™ 2. With this configuration, the screens are on different sides of the phone (i.e. in a different FOV) and cannot be used simultaneously. Codex [17] is a dual-screen device that works with a hinge between the screens and offers different functionalities for different rotational ‘postures’ of the screens, that can be in same or different FOV depending on context. Z-Agon [23] is another

453

Paper Session: Mobile

UIST’11, October 16–19, 2011, Santa Barbara, CA, USA

DESIGN FACTORS FOR MOBILE MDES

separation effects can configurations only.

There is a fixed number of ways to position displays together in a single-device-multi-display mobile environment. When such devices possess more than two displays, these design considerations apply to each pair of displays individually. The displays can either be separated by: distance vertically (Figure 2a,b,c) or horizontally (Figure 2d,e,f), an angle (Figure 2g,h,i), or any combination of those conditions. The displays can be separated by any distance d that will vary depending on the devices’ design: from a few centimeters wide such as the size of a bezel or a hinge (Figure 2 left column) up to a few meters wide in the case of a projector enhanced device (Figure 2 middle and right columns). When the displays are separated by an angle α (Figure 2g,h,i), α can be of any value (0-360°) along any axis in the cartesian space.

then

be

studied

for

fixed

In Figure 2, we present possible layouts of two displays: two screens (left column), a screen and a projector (middle column) and two projectors (right column). In the two screens case, the screens are unlikely to be more than a few centimetres apart in order for the device to be handheld; the design is therefore similar to traditional screens in MDEs separated by a bezel. The visual separation effects are then likely to be similar to the effects of bezels in MDEs. However, bezels do not affect visual separation as long as information is not cut across the bezel [5] and appropriate interaction techniques are implemented [18]. We have therefore chosen not to explore visual separation effects for this configuration. In the case of a screen and a projector, the displays have by default heterogeneous characteristics, such as different sizes and resolutions and are moreover separated by depth. The literature on MDEs shows that depth can be an important factor when managing visual separation effects. Moreover, the position of the projector lens on the device itself will determine if both displays will be in the same field of view or not. We believe that visual separation effects will be at their strongest in this type of environment, hence our decision to run the user study with a projector enhanced mobile device.

When the displays are close to each other or at a small angle from each other, they are in the same field of view. However when d or α have high values, the displays are in different fields of view. In fixed MDEs, displays tend to be in the same field of view, which is not the case in current mobile MDEs. In our user study, we will determine whether placing the displays in different fields of view increase visual separation effects.

The two projectors case is similar in characteristics to traditional large displays MDEs, such as two projection spaces that will display either on the same, on an orthogonal or on opposite planes, characteristics that have already been explored in the MDE literature. Yet, dualprojectors mobile devices present some interesting features such as the ability to display at different depths depending on the surrounding environment, as when displaying on an uneven wall. Nonetheless, in most multi-projector cases, the projections will either be separated in distance (depth), in plane or in size of projection. We believe that those issues are similar to the ones encountered by “a screen and a projector” case and that any experimental results obtained for the former configuration will apply to this category too.

Figure 2 - Possible layouts for two displays on a mobile device for different types of displays: screenscreen (a,d,g); screen-projector (b,e,h) and projector-projector (c,f,i). Displays can be separated horizontally, (a,b,c), vertically (d,e,f), by an angle across any plane (g,h,i) or any combination of the above positions. In the screen-projector cases (b,e,h), the displays are further separated by depth due to the inherent properties of each display

STUDY

The purpose of this study is to identify the effects of visual separation on single-device-multi-display mobile environment when the multiple displays are in the same field of view and when they are not, as well as when the device is fixed or mobile. We run the study using a projector-enhanced mobile device since the embedded displays are by design of different sizes and displaying at different depths. We study the “a screen and a projector” case over the “two-projector” case since this configuration is more prominent in current devices. We expect that the lack of physical connection between displays will generate greater effects of visual separation.

Depending on the design of the device, the displays are either relatively fixed: always at the same distance and angle from each other or reconfigurable: the distance and angle between the displays is context-dependent such as in Codex [17]. Reconfigurable devices are especially interesting since they can adapt to different contexts by rearranging the displays with respect to one another. Experimentally, reconfigurable displays can be simplified to devices that offer a set of fixed configurations, visual

Our experimental setup includes the following aspects of mobility: handling (participant can handle the device as

454

Paper Session: Mobile

UIST’11, October 16–19, 2011, Santa Barbara, CA, USA

they feel comfortable), portability (implies that the size and distance of the projection will vary), unsteadiness (jitter is not compensated as in a real-life scenario) but not actually moving between rooms in order to allow comparison between results in the fixed and the mobile settings.

measure visual search tasks [19]. The experimental room was darkened to optimize the projector viewing conditions.

Task

The task chosen for this experiment is a visual search task. Visual search is a typical task for analysing visual separation [13]. Tan et al. [29] use different types of task including text comparison as it is “representative of tasks in which the user must cross reference and compare content displayed in multiple locations” (p.4). Chen and Chien [8] use a similar task when looking at effects on visual performance on small screens. In our experiment, we chose an image comparison over a text comparison task, since the laser projector’s resolution could affect reading accuracy.

Figure 4 – Phone and projector used for the user study fitted with the Floor setting mirror.

The independent variables were:  Position of the projection relative to the phone’s screen: in the same field of view (Floor), in different fields of view separated by one angular plane (Front) or by two angular planes (Side)  Mobility: whether the device is fixed on a tripod or handheld by the user: mobile setting. Position

The projection spaces relative to the screen are described on Figure 5: The Front projection corresponds to the alignment of the phone and the projector. A mirror is placed at the top of the projector lens and oriented at 60° downwards for the Floor condition, as shown in Figure 4, and 40° sideways for the Side condition. In order to reduce the keystone effect introduced by the mirror, we projected at a resolution smaller than the projector’s maximum one.

Figure 3 – a. Left: Example of pattern displayed on the screen – b. Centre: Matching sparse version – c. Right: 3x3 grid displayed on the projection.

The task chosen consisted of matching a pattern on the screen (Figure 3a) with a sparse version of the same pattern (Figure 3b) positioned inside a projected 3x3 grid of competing matches (Figure 3c). This makes use of the different display sizes, showing the initial pattern only and a keypad on the small display and the 9-pattern grid on the larger projected display. The sparse versions are randomly created by deleting half of the items from the initial pattern and replacing them with blank cases. The competing patterns in the grid are other sparse versions of the initial pattern for which 5 items are permuted in order to look similar but not match the initial pattern.

Front

Side

In a pilot study with 4 participants we presented the participants with two types of patterns: a matrix filled with letters ‘P’ and ‘B’ and a matrix filled with coloured shapes: circles and triangles. The letter-based task was very long to perform and extremely tiring for the user while results obtained were similar to the shape-based task. We then decided to run the study with the shape-based task only.

Floor Figure 5 - Example of task pattern and grid of sparse patterns in the three positions in fixed setting: Front, Floor and Side. For each trial, the grid is only displayed in one setting only.

The participant would select a matching pattern on the projection by pressing the corresponding number on the numeric keypad on the screen below the initial pattern. Depending on the answer, the participant could receive positive audio-feedback and continue to the next trial or receive negative audio-feedback and would have to repeat the same trial until the correct matching pattern was found.

Mobility

In terms of mobility, the device was either set at a fixed position on a tripod (fixed setting) or held by the user (mobile setting).

Experimental design

In the fixed setting, the assembly phone-projector is placed on top of a tripod and the participant had to stand on footsteps marked by the tripod. The position of each projection space (on the floor, front and side walls) was

For the study we used a Google Nexus One with touch screen combined to a Microvision ShowWX laser picoprojector (Figure 4). The study makes use of a portable eye tracker since these systems have already been used to

455

Paper Session: Mobile

UIST’11, October 16–19, 2011, Santa Barbara, CA, USA

predefined in order to set a constant position and aspect ratio of the projection for all participants. We ensured that all three projection spaces were the same distance from the device (110cm) and would therefore always have the same size (middle of the projected grid fixed at 60cm wide).

In summary the experimental design was: 12 participants x 2 mobility factors x 3 positions x 8 trials = 576 data points. Measures

 Number of context switches between the screen and the projected display. Previous studies on visual separation do not measure the number of context switches. However the problems induced by context switches are quantified in mobile projector-phone studies [16] as well as in some MDE studies [3, 11]. This is measured by the portable eye tracker. The number of context switches is computed by the eye tracking software in the fixed setting using the IR markers and is then manually verified through analysis of the eye tracker video. In the mobile setting, the switches are manually counted at the video analysis stage (Figure 6) since the position of the projection space is not constrained in this setting.

In the mobile setting, the user is holding the device and can use any projection surface at any distance or size that they are comfortable with. The user was free to move around the room with the device. The distance to the wall and the size of the projection would then vary depending on user’s movements. We did not impose any restriction on how the user would hold the device. Nonetheless, we observed that most users held the device in the non-dominant hand and touched the screen with the dominant hand, while other users held the device in both hands and used their thumbs to touch the screen. None held the device with their dominant hand. Eye tracking procedure

The context switches were measured using a mobile eye tracker: Tobii® Glasses that recorded eye movements at 30 Hz. This eye tracker is non-intrusive as it is low weight (75 grams glasses) and fully mobile so participants could roam freely. Some InfraRed (IR) markers were positioned around the various display spaces (in the fixed setting) to allow automatic data mapping and help repositioning the projected image at the same place for each participant. The eye tracker records both a video of the scene and where the user is looking in the scene.

Figure 6 – Snapshots from the eye tracker video Left: User is looking at the projection. Right: User is looking at the phone

 Completion time and number of errors in performing each trial, including number of false positives. These are typical measures in visual separation studies [13, 28, 29] and allow comparing participants’ efficiency for different experimental settings. The completion time is timed between the start of the task to its successful completion.

Hypothesis

Based on the literature review and our preliminary exploration of the issues, we expected display configurations (relative positions of displays within same or in different fields-of-view), and whether the device is being held (mobility), to significantly affect performance and produce visual separation effects.

 Position preferred - NASA TLX: This test assesses subjective information on a 7-point Likert scale for mental, physical and temporal demand; performance; effort and frustration. We have combined this traditional subjective workload questionnaire with some personalised questions aimed at gathering user preference data.

We presumed visual separation effects to be less important when the screen and the projection are in the same field of view (floor setting) than when the projection is in a different field of view than the screen (front and side settings). We also expected for participants to compensate visual separation effects when holding the device since they could themselves reconfigure the display areas adaptively.

Results

We used a repeated measures ANOVA test for the number of context switches, completion time and number of errors. We used the univariate ANOVA test for the NASA-TLX results analyses with subject as a random factor.

Procedure

Twelve volunteers (5 men) aged between 24 and 35 years old (μ=28.6) were recruited from within one of our universities. All our participants were familiar with touchscreen technology and all had normal colour vision. We used a within-subjects design where position and mobility were counterbalanced across participants.

 Number of context switches: We found a main effect for position (F(2,94)=62.817, p0.05), as well as no significant difference in error-rates for the different positions (F(2,94)=1.049, p>0.05) and mobility (F(1,95)=1.143, p>0.05). The average error rate across all conditions was 8.9%. Figure 7 right shows the average trial completion times across all conditions. 40

Fr

Fl

S

40

30

30

20

20

10

10

0

0 Sta

Mob

Fr

Fl

During the experiment, participants had no problems using a mobile dual-display device even with very heterogeneous displays in terms of size, resolution and depth of the displays. In a case where tasks are divided across displays, single-device-multi-display environments can outperform today’s single display devices. Moreover, the tasks can make use of the different displays’ characteristics, such as our experiment uses the phone’s screen to display a single pattern and a keypad and the large projection space to display a large grid of 9 patterns.

S

We conclude that mobile multi-display environments with heterogeneous displays are a viable solution. Sta

B. Displays should ideally be in the same field of view.

Mob

The eye tracker recorded significantly more eye context switches in the floor condition: over 30% more than in the other positions; whereas we found no significance in completion time and error rate across the three positions in both fixed and mobile settings. This important result would have been overlooked should we have used traditional task performance measures only.

Figure 7 – Average of context switches (left) and average task completion time (right) for each conditions: Fr: Front, Fl: Floor, S: Side.

 NASA TLX: We only found a significant difference in temporal demand (“How hurried or rushed was the pace of the task?”) for position (F(2,22)=4.086, p=0.031). Floor is perceived as faster than Front and both are perceived as faster than Side (means for temporal demand for Floor is 3.67, Front is 3.83 and Side is 4.33 on a 7-point Likert scale). For all other variables no significant effect was found.

Our task was complex enough that participants could not simply memorise the whole pattern and find the matching pattern on the projection. The results show that the number of eye context switches does not affect task performance and that there is a higher number of context switches when both displays are in the same field of view. This suggests context switches are a lot cheaper to perform when both displays are in the same field of view (Floor setting) as they only require a simple eye movement and little or no head and neck movements, unlike the Side and Front conditions where participants reported discomfort. One participant said about the Side setting: “It was very uncomfortable to constantly turn my head during the experiment”. We also believe that the higher number of context switches in the Floor condition is due to the fact that context switches can be considered as epistemic [20], using the active memory to store the position of the geometric shapes in the pattern. Instead of having to remember the positions in the pattern, users could externalise their thought processes by switching context more often. This could also explain why this setting appeared as being faster paced to the participants.

 Position preferred. In the fixed setting, 75% of participants chose the Front position with the remaining participants preferring the Floor. In the mobile setting, half of the participants preferred the Floor, 42% the Front and 8% the Side. When asked what their favourite position was overall, 75% favoured a mobile position compared to a fixed one (Figure 8). 6 4 2 0

We recommend that as a default displays in MMDEs should have the displays aligned in the same field of view.

Figure 8 – Overall preferred position for each participant (Fr: Front, Fl: Floor, S: Side)

C. Mobility factors do not exacerbate visual separation.

Discussion

Since we found no significant difference between mobile and static setting in terms of error rate, task completion time or context switches, we believe that the following mobility factors: handling, portability, projection size and unsteadiness, have no particular effect on visual separation.

In this section, we discuss the above results on four related themes: A. Viability of mobile MDEs, B. Dual-display configurations, C. Substantiation of mobile uses and D. Design implications A. Visual separation does not impair the viability of MMDEs

Participants’ wrist and hand movements in the mobile setting did not help compensate the effects of visual separation. A possible reason could be that they were already compensating for the jitter of the projection resulting from the participants holding the device in their hands. Since none of the participants mentioned jitter as a problem during the experiment and in the post-study

The results of the study show that visual separation effects did not prevent users from carrying out the task, which is reflected through the low error rate of only 8.9% over all tasks and conditions. This result is valid for both the static and the mobile conditions.

457

Paper Session: Mobile

UIST’11, October 16–19, 2011, Santa Barbara, CA, USA

questionnaire, we conclude that they instinctively compensated for any mobility-induced jitter effects. Our experiment showed no more visual separation effects between mobile and static settings, even though the projection space and display size were varying; and since participants showed a strong preference for the mobile setting, the investigation of mobile scenarios is justified.

partially occluding the second display. This case is especially likely to occur in MMDEs where there is a depth gap between the displays as in mobile projector phones.

In the following sections, we present some design implications for future MDEs that emerge from our discussion in terms of type of displays, display physical arrangements, flexibility of design and mobility.

Besides, arranging displays in the same field of view is not trivial in a mobile environment where external factors influence how the user holds the device and on which surfaces content can be displayed. These external factors range from luminosity and glare to the available projection spaces, number of users viewing the content and the type of information being displayed. The usage of a steerable projection could overcome these environmental issues, as proposed by Pinhanez [27] for static and by Cauchard et al. [7] for mobile projection. Moreover, a steerable projection can reduce visual separation effects in MDEs by automatically reconfiguring the alignment of the displays according to the context the device is used in.

Type of displays for MMDEs

Flexibility of design

Factors such as: handling, portability and unsteadiness do not exacerbate visual separation. Although we did not explicitly test fully mobile conditions, we anticipate that those results are transferable to those environments. D. Design implications

Prior research conducted in MDE suggests that displays arranged on different planes or separated by more than 45° angle result in lower task performance and provide negative visual separation effects [28]. However, in our study, we find no significant task performance differences, whether in time completion or error rate, across the different settings. These results show that guidelines for MDEs are not directly applicable to MMDEs. One explanation could be that MMDEs use a comparably small display, close to the user compared to traditional MDEs. This shows that although it is preferable for the user to have both displays in the same field of view, there is more flexibility in the alignment of displays in MMDEs than in MDEs.

Our experiment demonstrates that mobile MDEs are viable, which includes heterogeneous dual-display solutions. Although dual-display solutions for mobile devices are technically possible, they are currently under-exploited by manufacturers. Our study demonstrates that these solutions should be envisaged more often since visual separation effects do not present issues for carrying out activities where tasks are distributed across displays, such as in our experiment. This is also valid for activities wherein the user chooses which display to use depending on application and context needs. Those scenarios of use are consistent with most common uses of MDEs as described by Grudin [15]. Additionally, most existing dual-display mobile devices are designed with multiple displays of similar types, whereas heterogeneous displays offer more potential, such as the ability to choose where to display depending on the context without generating negative visual separation effects. While current usage of heterogeneous dual-display mobile devices is often limited to one display at a time, we encourage designers to consider exploiting both displays simultaneously. This would also allow more flexibility in the choice of interaction technique; such as in Chen’s ebook reader [9]

This is especially the case for applications that do not require epistemic actions from the user, and for which the need for rapid context switching is not crucial. For those applications, manufacturers have more freedom to position the projection unit wherever it best suits the device ecology. This could result in smaller devices since the projection unit could be placed where it fits best without generating visual separation effects on performance. In this case, a wide range of interaction techniques can be supported for which displays do not need to be aligned, such as foot interaction on the floor [7] or even shadows on the projection [10] for any other projection setting.

Physical arrangements of displays

Our experiment shows that having both displays in the same field of view is paramount for applications that make use of both displays. There is evidence that users can reduce the amount of information they have to remember and can instead use active memory to recall information by switching gaze between displays more frequently. This is particularly important for applications that suffer from heavily cluttered displays, such as map applications. This pattern of increased context switches to alleviate cognitive load is equally important when one display is also used to facilitate input to the other display. For example, a projector phone’s touch screen can be used to manipulate content in the projected space. In this situation, the displays must be arranged within the same field of view. When both displays are in the same field of view, one display can be

Floor Projection

Our experimental results also illustrates that projecting on the floor is a promising option. When a projector-phone is held horizontally, the user can have both displays in the same field of view by projecting on the floor. This is especially useful for street navigation applications where users can follow directional arrows on the floor instead of reading a map on a small screen. Moreover, unlike a wall, the floor is a surface that is constantly available for projection. However, projecting on the floor is not straightforward and involves careful technical considerations. As is the case for any projection surface, the floor can be uneven, as for example on a cobbled street. There is also the issue of the

458

Paper Session: Mobile

UIST’11, October 16–19, 2011, Santa Barbara, CA, USA

user paying attention to the projected display only, while being inattentive to their surroundings. This can be very dangerous, so it is important to design a system that will retain users’ awareness to hazards.

physical arrangements of the displays, flexibility of design of MMDEs and mobility. Additionally, we establish that through the use of an eye tracker, we were able to highlight interesting differences between different physical arrangements of displays; that may not have been revealed with trial completion time and accuracy alone. This further suggests that eye tracking is an interesting way to investigate visual separation issues.

The choice of interaction technique will partially depend on the position of the floor projection with respect to the user. Projecting close to the user allows foot interaction or even a full body interaction; projecting further away from the users’ body will require indirect input such as through buttons or sensors on the projecting mobile device. The position of the projection can also be adapted to the user’s pace. For example, if the person is walking, the projection could be further away from their body [26] and move closer when the user stops walking to allow for direct interaction.

ACKNOWLEDGEMENTS

Partial funding for this work was provided by EPSRC (grant number EP/G058334/1) and MobileVCE (www.mobilevce.com) as part of the User Interactions for Breakthrough Services research program. We would like to thank Dr Peter Bennett in the Bristol Interaction and Graphics lab for the illustrations.

Mobility

In our study we find no more visual separation effects when the device is held than when the device is fixed on a tripod. Most current single-device-multi-display environments are built for scenarios of use in which the device is placed on a surface. Our study shows that various factors of mobility are worth investigating, such as when the user is walking while holding the device; or stopping by to obtain contextual information about the area they are walking by; or when using a QR-code on a poster for example. Many contextual applications could benefit from true mobility and new interaction paradigms could be envisaged, such as the use of haptic while on the move.

REFERENCES [1] Ankrum, D. R. Visual Ergonomics in the Office —

Guidelines. Technical Report, 1999. [2] Baudisch, P., Cutrell, E., Hinckley, K. and Gruen, R.

Mouse ether: accelerating the acquisition of targets across multi-monitor displays. In CHI '04 EA, ACM, 2004, pp. 1379-1382. [3] Benko, H. and Feiner, S. Multi-monitor mouse. In CHI

'05 EA, ACM, 2005, pp. 1208-1211. [4] Benko, H. and Feiner, S. Pointer warping in

heterogeneous multi-monitor environments. In Proceedings of Graphics Interface 2007, ACM, 2007, pp. 111-117.

FUTURE WORK

The focus of the present work has been to explore visual separation issues for a single user working in a mobile multi-display environment. However there are equally interesting future scenarios which involve multi-user collaboration in mobile multi-display environments. Supporting collaboration in a MMDE requires positioning displays in order to improve coordination and awareness while potentially increasing visual separation. One direction of future research is to explore how visual separation issues affect collaboration and coordination between multiple users.

[5] Bi, X., Bae, S.-H. and Balakrishnan, R. Effects of

interior bezels of tiled-monitor large displays on visual search, tunnel steering, and target selection. In CHI 2010, ACM, 2010, pp. 65-74. [6] Cao, X., Forlines, C. and Balakrishnan, R. Multi-user

interaction using handheld projectors. In UIST 2007, ACM, 2007, pp. 43-52. [7] Cauchard, J. R., Fraser, M. and Subramanian, S.

Offsetting Displays on Mobile Projector Phones. In Ubiprojection Workshop on Personal Projection at Pervasive 2010, pp. 3.

CONCLUSION

In this paper we presented design factors for an emerging category of devices: Mobile Multi-Display Environments (MMDEs). We then investigated visual separation effects for MMDEs compared to the current literature of visual separation in fixed MDEs. Through an innovative eye tracking methodology, we compared different angular separations of a projection and a screen: two displays of different sizes and at different distances from the user. We determined that although task performance was not affected by the displays being in the same or in different fields of view, the number of eye context switches was over 30% higher in the condition where both displays were in the same field of view. We also tested various factors of mobility in our experiment and concluded that they did not affect visual separation. We finally present design implications in terms of types of displays used in MMDEs,

[8] Chen, C.-H. and Chien, Y.-H. Effects of RSVP display

design on visual performance in accomplishing dual tasks with small screens. International Journal of Design, 1, 2007, pp. 27-35. [9] Chen, N., Guimbretiere, F., Dixon, M., Lewis, C. and

Agrawala, M. Navigation techniques for dual-display e-book readers. In CHI'08, ACM, pp. 1779-1788. [10] Cowan, L. and Li, K. ShadowPuppets: Supporting

Collocated Interaction with Mobile Projector Phones Using Hand Shadows. In CHI 2011, ACM [11] Dickie, C., Hart, J., Vertegaal, R. and Eiser, A.

LookPoint: an evaluation of eye input for hands-free switching of input devices between multiple computers. In OZCHI '06, ACM, 2006, pp. 119-126.

459

Paper Session: Mobile

UIST’11, October 16–19, 2011, Santa Barbara, CA, USA

[12] Dickson, F. Pico Projectors: One Reason Bigger Isn't

[22] Mandryk, R. L., Rodgers, M. E. and Inkpen, K. M.

Better. Technical Report IN1004722WH, In-Stat, 2010.

Sticky widgets: pseudo-haptic widget enhancements for multi-monitor displays. In CHI '05 EA, ACM, 2005, pp. 1621-1624.

[13] Forlines, C., Shen, C., Wigdor, D. and Balakrishnan,

[23] Matsumoto, T., Horiguchi, D., Nakashima, S. and

R. Exploring the effects of group size and display configuration on visual search. In CSCW '06, ACM, 2006, pp. 11-20.

Okude, N. Z-agon: mobile multi-display browser cube. In CHI '06 EA, ACM, 2006, pp. 351-356.

[14] Greenberg, S., Boyle, M. and Laberge, J. PDAs and

[24] Merrill, D., Kalanithi, J. and Maes, P. Siftables:

shared public displays: Making personal information public, and public information personal. Personal and Ubiquitous Computing, 3, 1999, pp. 54-64.

towards sensor network user interfaces. In TEI'07 ACM, 2007, pp. 75-78. [25] Nacenta, M. A., Sallam, S., Champoux, B.,

[15] Grudin, J. Partitioning digital worlds: focal and

Subramanian, S. and Gutwin, C. Perspective cursor: perspective-based interaction for multi-display environments. In CHI'06, ACM, 2006, pp. 289-298.

peripheral awareness in multiple monitor use. In CHI 2001, ACM, 2001, pp. 458-465. [16] Hang, A., Rukzio, E. and Greaves, A. Projector phone:

[26] Ota, S., Takegawa, Y., Terada, T. and Tsukamoto, M.

a study of using mobile phones with integrated projector for interaction with maps. In MobileHCI 2008, ACM, 2008, pp. 207-216.

A Method for Wearable Projector Selection that Considers the Viewability of Projected Images. Comput. Entertain. 8, 2010, pp. 1-16.

[17] Hinckley, K., Dixon, M., Sarin, R., Guimbretiere, F.

[27] Pinhanez, C., Kjeldsen, R., Levas, A., Pingali, G.,

and Balakrishnan, R. Codex: a dual screen tablet computer. In CHI'09, ACM, 2009, pp. 1933-1942.

Podlaseck, M. and Sukaviriya, N. Applications of steerable projector-camera systems. In IEEE International Workshop on Projector-Camera Systems at ICCV 2003, IEEE Computer Society Press.

[18] Hinckley, K., Ramos, G., Guimbretiere, F., Baudisch,

P. and Smith, M. Stitching: pen gestures that span multiple displays. In AVI 2004, ACM, pp. 23-31.

[28] Su, R. and Bailey, B. P. Put Them Where? Towards

Guidelines for Positioning Large Displays. In Proc. of Graphics Interface '05, 2005, pp. 337-349.

[19] Kern, D., Marshall, P. and Schmidt, A. Gazemarks:

gaze-based visual placeholders to ease attention switching. In CHI 2010, ACM, 2010, pp. 2093-2102.

[29] Tan, D. S. and Czerwinski, M. Effects of Visual

Separation and Physical Discontinuities when Distributing Information across Multiple Displays. In Interact 2003, pp. 252-255.

[20] Kirsh, D. and Maglio, P. On Distinguishing Epistemic

from Pragmatic Action. Cognitive Science, 18, 1994, pp. 513-549.

[30] Yang, X.-D., Mak, E., McCallum, D., Irani, P., Cao, X.

[21] Lyons, K., Pering, T., Rosario, B., Sud, S. and Want,

and Izadi, S. LensMouse: augmenting the mouse with an interactive touch display. In CHI 2010, ACM, 2010, pp. 2431-2440.

R. Multi-display Composition: Supporting Display Sharing for Collocated Mobile Devices. In Proceedings of the 12th IFIP TC 13 International Conference on Human-Computer Interaction: Part I, Springer-Verlag, 2009, pp. 758-771.

460