Drag-and-Pop: Accelerating Drag-and-Drop on Large Screens Patrick Baudisch 3084 Emerson St. Palo Alto, CA 94306 +1 (650) 493 3614
[email protected]
Benjamin B. Bederson HCIL, CS Department, Univ. of Maryland College Park, MD 20742 +1 (301) 405-2764
[email protected]
Alexander Zierlinger Maila Push Frankfurter Str. 10 64293 Darmstadt, Germany +49 (6151) 493 752
[email protected]
ABSTRACT
Drag-and-pop is an interaction technique designed to accelerate drag-and-drop on large screens. By animating potential targets and bringing them to the dragged object, drag-and-pop reduces the user effort required for dragging an object across the screen to a desired target. To preserve users’ spatial memory, targets are not moved away from their original location, but are instead stretched using a rubber band-like visualization. This paper presents the rationale, algorithm, and visualization techniques of drag-and-pop with the example of Windows desktop icons. Keywords
Drag-and-drop, pop-up menus, interaction techniques. INTRODUCTION
This paper presents drag-and-pop, an enhancement to dragand-drop. Drag-and-drop is an interaction technique for transferring or copying information using a mouse or other pointing devices [2]. A user first “grabs” an object, such as an icon, by pressing a mouse button on it, then “drags” it towards a desired position on the screen, and “drops” it at that location by releasing the button. Objects can be dropped onto a target object in order to initiate a specific action, e.g. dropping a file icon onto a printer icon will print that file [9]. The strength of drag-and-drop compared to traditional cut/copy/paste is that users are not required to interact with a hidden clipboard. Direct manipulation is used to give continuous visual feedback about valid targets as the user drags the object across the screen [2, 8]. The problem that we address in this paper is that drag-anddrop was developed for the affordances of a desktop computer screen, and does not scale well to large screens, especially if they are pen-operated, such as the Xerox Liveboard [6] or if they are finger-operated such as the Smartboard (http://www.smarttec.com). Pen interaction is problematic because it is difficult for users to keep the pen tip in contact with the display surface during long drag interactions, so that users often drop objects accidentally [7]. Penbased drag-and-drop does not work at all across display units of multi-display systems, such as the DynaWall [4].
Figure 1: In drag-and-pop, each valid target icon in direction of the drag motion creates a linked tip icon that approaches the dragged object. Dropping onto a tip icon saves mouse travel to distant targets.
To overcome this problem, several alternative interaction techniques based on point-and-click have been proposed, such as pick-and-drop [7] and ‘taking’ [4]. The more general problem with drag-and-drop on large screens results from the size of the display. Drag-and-drop, as well as pick-and-drop, requires a manipulation effort that grows with the distance to the target, according to Fitts’ Law [3]. This is especially significant for systems using the physically more demanding pen-based interaction style, but also for mouse-based large screen systems, such as focus plus context screens [1]. Several interaction techniques have been proposed that facilitate overcoming distances on the screen; some of them are in principle applicable to drag-and-drop. For instance, ‘throwing’ objects allows users to accelerate an object with a small gesture; the object then continues its trajectory based on its inertia [9]. The imprecision of human motor skills and the limitations of mice prevent throwing from being used for reliable drag-and-drop. A technique to reduce this problem is based on eye tracking. Manual And Gaze Input Cascaded pointing (MAGIC) [10] uses eye tracking to move the mouse cursor roughly to the target area, from where the user guides the cursor manually. The performance of MAGIC users, however, is limited because the
(selected icon) towards the icons located at the top left of the screen. The icons of valid targets (called base icons), develop copies of themselves (called tip icons) that approach the dragged icon using a smooth and fast animation. If the user drops the selected icon onto one of the tip icons, the corresponding action is started and all tip icons retract to their bases. Alternatively, the user may ignore the tip icons and continue the drag motion to a different target. Moving away from the tip icons or dragging past them makes tip icons disappear prematurely, which can be used intentionally for dropping an icon onto space occluded by tip icons.
Figure 2: With drag-and-pop, accessing icons on the other side of the screen does not necessarily require a longer mouse movement than accessing icons that are located more closely. Clusters of target icons are compressed by removing white space.
manual correction cannot start until the cursor reappears in the target area, which was found to negate the time saving gained from the reduction of movement amplitude [10]. An alternative that allows avoiding the affordances of dragand-drop-based handling of icons entirely is to instead use variations of pop-up menus (e.g. [5]). Desktop operating systems, such as Microsoft Windows, use ‘context menus’ for this purpose, and include a ‘send to’ option that provides a selection of predefined targets. Pop-up menus have some clear benefits over drag-anddrop. They minimize mouse travel by always appearing at the current cursor position and they work when the desktop is covered by other objects. On the other hand, icon-based interactions allow a broader set of choices since the entire desktop is available and they allow users to take advantage of spatial memory by allowing for free arrangement of icons in two-dimensional space. By using a consistent interface, users can plan drag-and-drop interactions in advance while users of context menus must scan the menu, since the menus can be different for each icon and use. DRAG-AND-POP
We address the issues identified above by introducing an interaction technique which enhances drag-and-drop. Dragand-pop is designed to combine the benefits of drag-anddrop with the benefits of pop-up menus. Drag-and-pop is based on icons, so it allows users to benefit from free spatial arrangements. However, icons are rearranged during drag-and-drop, resulting in an interaction similar to a popup menu. Figure 1 shows an example drag-and-pop interaction. The user has started dragging the icon at the right of the screen
The purpose of drag-and-pop is to shortcut the mouse travel involved in drag-and-drop. Tip icons serve as shortcuts. If the user drops an icon onto one of them, a drag interaction that would otherwise extend to the corresponding base icon is abbreviated.
The remainder of this paper is structured as follows. We first present the algorithms for computing the spatial arrangement of tip icons. Next, we describe the visualization techniques we applied for helping users keep track of their targets during the interaction. Finally, we briefly explain the current implementation of drag-and-pop, and then conclude with a summary of its benefits and limitations and our plan for future work.
COMPUTING THE LAYOUT OF THE TIP ICONS
In order to maximize time savings, drag-and-pop tries to optimize the selection of tip icons and their positioning with respect to the mouse cursor. In order to reduce clutter, drag-and-pop selects only a subset of all icons when generating tip icons. Only icons of compatible type are considered. The selection of candidates is further restricted based on the direction of the mouse motion initiating the drag interaction. This is motivated by the notion that users will start dragging towards the desired target, independent of whether they expect drag-and-pop to appear or not. In the current prototype, icons inside a ± 20° sector of the initial mouse motion angle are considered candidates for tip icons. Figure 1 shows an example. Drag-and-pop computes a layout for the tip icons of all candidate targets. The goal is to obtain fast access to all tip icons by using a compact arrangement. At the same time, the spatial relationship among candidates should be preserved so that users may use their spatial memory to rapidly identify the desired target within the candidate set. Spatial arrangement is especially important for distinguishing between multiple documents represented by identical icons, as illustrated by the cluster of folders in Figure 2. We use the following bottom-up clustering algorithm that condenses the field of candidates by compressing large distances, while preserving clusters of collocated icons.
Init:
Each icon is a cluster. Initialize the desired minimum distance d between icons to some small value, such as twice the size of an icon.
Step 1: Merge all pairs of clusters that are closer than d into one cluster. Step 2: Increase d in order to visually distinguish originally distant clusters.
compress
skew
compress overlay
skew h etc str
Step 3: Scale down the desktop space, but not the clusters inside it, until the distance between the first pair of clusters hits d. Step 4: Repeat until all clusters are merged into one. After the compressed tip icon layout has been computed, drag-and-pop attempts to position the tip icon layout such that the most likely candidate targets are most accessible. To accomplish this, drag-and-pop estimates where the mouse cursor will be after the user notices the activation of drag-and-pop and had a chance to stops the mouse motion. The current drag-and-pop version uses a heuristic function based on drag direction and speed for that purpose; future versions might use machine learning techniques to optimize this function. Drag-and-pop places the tip icon layout such that the sum over tip icon to cursor distance x candidate probability is minimized. Tip candidate probabilities are computed as a Gaussian function of the difference between the angle of the user’s initial drag motion and the actual angle to the candidate target. The layout algorithm described above lets drag-and-pop scale up to handling larger numbers of icons well. Since the layout algorithm works by removing white space, it will move all tip icons close to the cursor and remove a lot of white space when there are few tip icons; it will move them less when there are more tip icons. If the desktop is completely full of icons, then the tip icons won't move at all, and drag-and-pop effectively degrades to a null action. We are also considering a range of other approaches to scale up drag-and-pop, such as limiting the number of tip icons, dynamically changing the angular selection sector, and choosing tip icons adaptively based on use patterns. VISUALIZATIONS
Moving icons during a drag interaction, as done by dragand-pop, involves the risk of disorienting the user. At some time before or during the drag interaction, users identify the desired target icon. If users lose eye contact with this target, they have to find it again; this requires a time-consuming reorientation similar to the one users undergo when opening a context-sensitive pop-up menu. To prevent disorientation, drag-and-pop uses customized visualization techniques to keep users aware of their target and to get them reoriented rapidly in case they should lose it. Maintaining awareness of the target
Drag-and-pop uses slow-in-slow-out animation to allow users to track the desired tip icon while it develops [8]. Because tip icons originate directly on top of their base icons, users who are fixating their target at the beginning of a drag interaction can visually trace the tip icon. In the current
Figure 3: The rubber band that connects each tip icon with its base results from combining horizontally and vertically skewed icons.
version of drag-and-pop, the tip icons animation is set to 1500ms for first-time users. With repeated usage, this time converges rapidly to 200ms in order to speed up the interaction. Additional usage data is required for determining optimal values. Drag-and-pop renders tip icons and their rubber bands on top of all other desktop content. This always-on-top property, illustrated by giving tip icons a drop shadow (Figure 3), provides users with continuous visual access to the desired target. As a side effect, it allows dropping onto targets occluded by application windows, because the tip icons of occluded icons “break through” occluding desktop content. Translucency prevents rubber bands from occluding desktop content. For systems with multiple display units, tip icons always push into the display unit where the mouse cursor is located. Thus, tip icons can be used to provide users with access to off-screen targets, i.e. targets that are clipped at window boundaries, that are in the non-displayed part of a virtual frame buffer, or that are located on the screens of other machines [7]. Drag-and-pop keeps tip icon layouts compact and concise by reducing target windows to icons; the icon in the left corner of the window’s title bar serves as the base icon that develops the tip icon. In case a window provides additional targets inside, e.g. in case of a directory window or a CD burning program featuring a directory widget, these targets may develop their own tip icons. To enhance visual clarity, additional visual grouping may be applied to icons collocated in such folders. Helping disorientated users re-identify their target
Users who have lost sight of their target icon may reidentify their target by visually scanning all candidates. Drag-and-pop supports an alternative strategy. The combination of base icon, rubber band, and tip icon allows users to look at the base icon and then find the tip icon by tracing the rubber band. For a distant icon, this is still timeconsuming. Drag-and-pop therefore uses a special trick to help users shortcut this backtrack: rubber bands are designed to approximate the color schemes and textures of the
icons to which they are attached (Figure 3). This provides additional information for identifying an icon and may make it unnecessary for the user to go back to the base icon. Users only go back until the combination of color-scheme recognition and spatial information allows them to recognize the icon. In addition, the color scheme helps to prevent users from confusing different rubber bands with one another on the way back to the tip icon. We have experimented with several methods for creating drag-and-pop rubber bands. The method shown in Figure 3 creates rubber band textures by averaging the icon texture. This is done by compressing copies of the base icon to 1pixel thickness. The shown method uses pairs of rubber bands to avoid degenerating of the rubber band for certain drag directions. To visually communicate that tip icons will return to their bases, the rubber band’s elasticity is emphasized by making the middle section get thinner as the rubber band is stretched. The shape of the stretched band also gives users an additional visual cue about the locations of base icon and tip icon, even if users see only a fragment of the rubber band (see the rubber band of the “My Documents” icon in Figure 1). IMPLEMENTATION
Drag-and-pop is currently implemented in Macromedia Flash (http://www.macromedia.com). We are working on a native implementation for Microsoft Windows. CONCLUSIONS AND FUTURE WORK
Compared to traditional interaction techniques, we see three benefits of drag-and-pop. First, we expect drag-and-pop to be faster than traditional drag-and-drop. According to Fitts’ Law [3], targets which are smaller or farther away take longer to acquire. Dragand-pop shortens the distance, while the target size remains the same, so it should have the desired effect—although there may be countering effects of disorientation. The reduction of long-distance cursor motion should also reduce manual stress and fatigue. We also expect drag-and-pop to be faster than context menus that list possible targets, because drag-and-pop users can use their spatial memory for pre-selecting target candidates while users of contextdependent menus have to visually scan the individual menu entries first. Second, drag-and-pop makes peripheral and off-screen information more accessible by bringing it to the user’s focus of attention (the cursor position). On wall-size screens, this makes distant icons easier to recognize. On multi-display systems, it allows accessing the other display units. By bringing target icons into the user’s primary screen and on top of other desktop content, drag-and-pop facilitates dragand-drop on cluttered desktops, as well as access to offscreen content. Third, drag-and-pop integrates into existing systems well. Unlike interaction techniques based on hidden clipboards, drag-and-pop is a natural extension to drag-and-drop, so
that the basic interaction concept, as well as all derived interactions, such as file copy and creation of links remain unchanged. Drag-and-pop is limited in that the need for an extra deactivation gesture is created when rearranging icons on the desktop. If users conduct extensive reorganizations of their icons, this may be cumbersome. We also see the potential risk that drag-and-pop might contribute to visual overload. While informal test users liked the graphic design and animation of drag-and-pop, it may turn out that long-term users find the animation distracting. Additional user feedback will allow us to understand these issues better and to better evaluate the various design options. Our future work includes initiating a controlled experiment comparing task execution times of drag-and-pop with those of traditional drag-and-drop and those of pop-up menus. While we initially developed drag-and-pop for large screens, we also plan to examine carefully for highresolution desktop screens. An important issue that remains to be addressed is how drag-and-pop scales up on real users’ desktops. We plan on looking at a broad range of desktop layouts to determine what the most effective algorithms to scale drag-and-pop up will be, as well as to examine the long term usage patterns and interactions of users. ACKNOWLEDGMENTS
Thanks to Diane Kelly, Dieter Böcker, Peter Tandler, Lance & Nathan Good, and Amanda Williams for their comments. REFERENCES 1. Baudisch, P., Good, N., and Stewart, P. Focus plus Context Screens: Combining Display Technology with Visualization Techniques. In Proc of UIST ‘01, pp.31–40, 2001. 2. Beaudouin-Lafon, M. Instrumental Interaction: An Interaction Model for Designing Post-WIMP Interfaces. In Proc. CHI ’00. pp. 446–453, 2000. 3. Fitts. P.M. The information capacity of the human motor system in controlling the amplitude of movement. Journal of Experimental Psychology, 27, pp. 381-391. 4. Geißler, J. Shuffle, Throw or Take It! Working Efficiently with an Interactive Wall. In CHI '98 Late–Breaking Results, pp. 265–266, 1998. 5. Guimbretière, F., Stone, M., Winograd, T. Fluid Interaction with Hi–res Wall–size Displays. Proc. UIST ‘01, pp. 218–30. 6. Elrod, S., et. al. Liveboard: a large interactive display supporting group meetings, presentations, and remote collaboration. In Proc. CHI '92, pp. 599-607, 1992. 7. Rekimoto, J. Pick–and–Drop: A Direct Manipulation Technique for Multiple Computer Environments. In Proc. UIST'97, pp. 31–39, 1997. 8. Shneiderman, B. Designing the User Interface: Strategies for Effective Human-Computer Interaction. Third edition. Reading MA: Addison-Wesley, 1998. 9. Wagner, A., Curran, P., O'Brien, R. Drag me, drop me, treat me like an object. In Proc CHI ’95. pp. 525–530, 1995. 10. Zhai, S., Morimoto, C. Ihde, S. Manual And Gaze Input Cascaded (MAGIC) Pointing. Proc. CHI’99, pp. 246-253, 1999.
