Method for Visualizing Undone Operations Based on ...

Method for Visualizing Undone Operations Based on Changes on Desktop Screen Arisa Sakamoto, Takuya Katayama Grad. Sch. of Engineering, Kobe University 1-1 Rokkoudaichou Nadaku, Kobe, 657-8501, JAPAN

{sakamoto_arisa@stu, takuya.kata@port}.kobeu.ac.jp

Tsutomu Terada

Masahiko Tsukamoto

Grad. Sch. of Engineering, Kobe University PRESTO, JST 1-1 Rokkoudaichou Nadaku, Kobe, 657-8501, JAPAN

Grad. Sch. of Engineering, Kobe University 1-1 Rokkoudaichou Nadaku, Kobe, 657-8501, JAPAN

[email protected]

ABSTRACT The undo operation of graphical user interfaces is widely used. However, conventional implementations of undo have two problems: 1) users may not notice undone parts since the operation is instantaneous, and 2) it is difficult to know the undone contents if the user temporarily interrupts a work the user works on multiple operations in parallel. Although there have been several methods developed that improve undo operations, they are for specific applications. Here, we devised a visualization for undone operations that, is independent of the running applications and based on the idea that an undo operation causes display to change. Our method specifies the area affected by the undo operation by screens. We implementedseveral visualization methods including emphasizing the undone area, presenting the undone content, and notifying the user that the undone is not visible.

Categories and Subject Descriptors H.5.2 [INFORMATION INTERFACES AND PRESENTATION]: User Interfaces—Graphical user interfaces (GUI), Prototyping; H1.2 [User/Machine Systems]: Human factors

General Terms DESIGN, HUMAN FACTORS

Keywords user interface, undo operation, visualization, changes on desktop screen

even though they know where the undo operation took place. Some commercial products and studies have tackled these issues. For example, Microsoft Office has a function to present a history of past operations that users can undo if they choose the pull-down menu next to the ’Revert’ button. Adobe Photoshop image editing software has a ’History Panel’ that users can check anytime. Users can confirm the operations that they want to undo before they actually undo then and they can undo all operations after one they specify. However, it is difficult to for them to see exactly which operations have been undone because such historical representations tend to be too simple or specialized to the application. For example, simple representations such as "Input” and "Font color” put users at a disadvantage. There is another problem in which users may miss the undone part because of an unexpected undo operation and they may not find the undone operations outside the visible area. In this research, we devised visualizations of undone operations that are independent of the running applications. Our method specifies the area affected by the undo operation by using image comparison techniques to detect visual changes on the screen. We examined several visualization methods including emphasizing the undone area, showing the undone contents, and notifying the user when an undone part is outside the visible area, in order to resolve three problems: (1) users may not notice undone parts, (2) it is difficult to be certain about the content undone, (3) users cannot see undo operations that are outside the visible area. The reminder of this paper is organized as follows. Section 2 describes the related work, and Section 3 details our proposed methods, and Section 4 presents our conclusion.

2. 1. INTRODUCTION The undo operation of graphical user interfaces (GUI) cancels the last operations. However, conventional implementations of the undo operations have the following problems: 1) users may not notice that an undo operation has taken place since it is instantaneously and 2) they may not know which content was undone, Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. SAC’14 March 24-28, 2014, Gyeongju, Korea. Copyright 2014 ACM ...$15.00.

[email protected]

RELATED WORK

The undo operations of most applications cannot selectively undo only the intended operation without undoing all the subsequent ones because they undo by undoing operations all the way back to the intended one. Adobe Photoshop, however, has a feature that selectively undoes only the intended operations, which called ‘Selective Undo’ [1], and does not require the user to take account of all operations of the sequence. However, these developments are for specific applications and are dependent of them. Regional Undo/Redo [2] recovers input operations in each region by using a large interactive whiteboard designed for it. This method undoes the operation in the intended region without affecting ones in other regions even if multiple users simultaneously work on the whiteboard. Caroline et al. presented Dwell-and-Spring [3], an interaction technique that uses the metaphor of springs to enable the

Undo

Undo Pasting the former state on the actual one Filling red

Drawing a line

Opacity: 75%

Opacity: 50%

Opacity: 25%

Opacity: 0%

Figure 1: Red Rectangle Mode user to undo direct manipulations. Though this technique allows the user to cancel the current manipulation or undo the last one, he/she can already do that during a press-drag-release interaction. We previously showed how users can use a history of past operations that can be undone in Office and Photoshop. The operations, however, had simple representations; therefore, it was difficult to find the command or object state that the user needs. There have been a number of studies attempting to resolve this problem. Masuda et al. designed a specialized graphical history browser to use selective undo effectively and easily [4]. It enables the user to undo the intended operation simply by choosing a snapshot that records the intended operation. In addition, the browser presents only the operations related to a chosen object. Although above researches proposed methods to resolve the problems described in Section 1, they are designed for specialized applications. By contrast, our system works independently of the running applications.

3. PROPOSED METHODS As mentioned above, the problems of conventional undo operations are that (1) users might perform undo parts without knowing they have done so, (2) it is difficult to know with certainty what the undone content is, and (3) users cannot see undo operations outside the visible area. The proposed methods use the graphical changes to the desktop screen that are caused by undo operations in each application window. We use simultaneous presses of the Ctrl -key and Z -key as a trigger of our visualization method. The method consists of the following three visualization functions.

3.1 Emphasizing Undone Area The undone area is emphasized with two different effects, called Red Rectangle Mode and Smooth Transition Mode. Red Rectangle Mode emphasizes the undone area with red rectangles. It is good for situations where the user trouble finding the undone area. Smooth Transition Mode provides a gradated slow motion view of the undo.

3.1.1

Red Rectangle Mode

Upon pressing the Ctrl -key and Z -key simultaneously, this mode provides a screenshot before the undo occurred. Then, one second after releasing the keys it shows the state of the desktop screen after the undo operation. Then, it compares the RGB values of pixels in each screenshot and draws the smallest red rectangles that enclose areas including pixels that have different RGB values in going between screenshots (Figure 1).

3.1.2

Smooth Transition Mode

The system provides the two screenshots in the manner mentioned above, except that it draws the screenshot after undo operation in layers by changing the opacity gradually. The opacity isdecreased from 100% to 0% by 1% every 30 milliseconds. Finally, as shown in Figure 2, it presents the undo operation in slow motion instead of instantly as is usually done, the leftmost screenshot showing the state before the undo operation and the rightmost screenshot showing the state after the undo operation.

Changing the opacity of the pasted one gradually

Figure 2: Smooth Transition Mode

3.2

Showing Undone Content

The second function presents the undone content to user. The presented content includes visualizations of non-visible operations, such as a choosing a color for a line in a paint application or choosing a font in documentation application. It thus overcomes the difficulty of recovering the past states for lack of a visible operation history. The function specifies the period of the undo operation (described below). Hereinafter, we explain the procedure of this function.

3.2.1

Recording Input Content

This function presents a history of the keyboard and mouse input. To achieve this, the system hooks each input event and allocates a screenshot that is within one second of that event. To shorten the processing time, it keeps the input events and the screenshots in memory for 60 seconds. In addition, it records only the screenshot on the active window to exclude changes on the desktop screen that are unrelated to the undo operation. KeyDown events, MouseDown, MouseUp, and MouseMove events are recorded. By combining the three events, the system distinguishes between cursor moves and mouse drags.

3.2.2

Specifying Undone Content

The proposed system specifies the undone content. In particular, it specifies the past state in relation to the state after the undo operation from the history of desktop screen transitions kept in memory. We call the period from the specified past state to the state after the undo operation undone period. The function uses the Bhattacharyya distance on the histograms of luminance of the screenshots being compared (Expression (1)).Bhattacharyya distance is an index of similarity between images. In Expression (1), R denotes similarity, H1 , H2 the corresponding sets of components in the compared histograms of luminance, and N the number of bins in each histogram. R = 0 means that the two compared screenshots are the same. We define an occasion that R is at a minimum other than zero as one in which the operations have been undone because we think the screenshot after an undo operation is the most similar in the screenshots before it. R=

s

1− √

P 1 I 2 ¯ H ¯ H 1 2N

(∵H¯k = N1

P

√

H1 (I)H2 (I)

J Hk (J)

)

(1) (2)

The processing time is shortened as follows: (1) if R = 0, the system finishes the comparison, and the occasion is regarded as before the undo operation; (2) if R 6= 0, the system finishes the comparison when R exceeds 0.05 after it has fallen below 0.05 once, and the occasion that has the minimum R is regarded as before the undo operation. For example, Figure 3 shows a situation in which the operation of erasing three black lines enclosed in the orange circle has been undone and each screenshot shows the working state at that time.

(a) 16 sec before

(b) 11 sec before

(c) 4 sec before

Operations 1. Deleting 3 letters by BackSpace-key 2. Typing ‘’HOKKAIDOU” 3. Converting to Kanji by Space-key 4. Confirming by Enter-key

(d) After undoing

Undo Extracting keyboard input

Figure 3: Compared screenshots: Erasing three black lines Table 1: Bhattacharyya distance for each screenshot Screenshots 4 sec before undoing (Figure 3(c)) 11 sec before undoing (Figure 3(b)) 16 sec before undoing (Figure 3(a))

R 2.086 ×10−3 9.153 ×10−4 0.000

The labels in the figure show how long the screenshot is before the undo operation. Moreover, Table 1 shows the Bhattacharyya distances between screenshots before the undo (Figures 3(a), 3(b), 3(c)) and after the undo (Figure 3(d)). In the example, the R = 0 condition in comparisons with Figure 3(a) is reached, at which time the comparison finishes and the function judges that the state has gone back 16 seconds before the undo operation.

3.2.3

Figure 4: Showing undo content: Keyboard event

MouseDown

Pressing left mouse button Filling jaggedly (Dragging)

Undo MouseDrag Filling red MouseUp

Releasing the button

Showing Content

The system presents the undone content based on the undone period that has been calculated above. It combines keyboard and mouse inputs related to each screenshot recorded before the undone operations in a history and shows a sequence of input events captured by the combinations of inputs. Figure 4 shows the an example presenting an input event history for a keyboard, and Figure 5 shows an example presenting an input event history for a mouse (mouse tracks are presented with lines). Keyboard inputs are presented as a series of pressed keys and the undone area with a frame. In Figure 4, the BackSpace -key has deleted letters originally inputted, in this case, alphabetical keys, the Space -key has been used to convert alphabetical text into Kanji characters, and the Enter -key has been used to confirm the correctness of the selected Kanji characters in the area enclosed with the orange line. Three kinds of mouse input are presented: (1) MouseDrag as a sky-blue solid line, (2) MouseDown as a red dot, (3) MouseUp as a pink dot. Figure 5 presents the clicks to indicate the start and finish points of the line and the drag to draw it.

3.3 Undo Notification As mentioned in Section 1, conventional undo operations do not allow users to see performed outside the visible area. On the other hand, although certain applications the changes indicates on a toolbar or quick-access-toolbar, they do not indicate the undone content. We devised a function that notifies users when an undo operation has been performed outside the visible area and shows them the undone content if they want to know that. We used a blinking window to inform users of changes that occur outside the visible area. Specifically, the system gets the handle of an active window when the user presses the Ctrl -key and Z -key simultaneously and the taskbar and the title bar on the window blink if the system detects undo operation being performed outside the visible area of the window whose handle is the same as the handle that has been captured. If no operations are detected outside the visible are, the changes as a result of undos on the desktop screen are emphasized using the modes described above.

4. CONCLUSIONS We proposed a visualization mechanism for undone operations

Figure 5: Showing undo content: Mouse event that is independent of the running applications and based on changes to the screen and a history of past operations on the user interface. We have implemented and evaluated several visualization methods including emphasizing the undone area, undone content presentation, and notification that the undone part is outside the visible area to resolve the following problems: (1) users cannot notice undone parts, (2) it is difficult to know what the undone content is, and (3) users cannot see undo operations outside the visible area of the screen. We will improve our methods based on our own experience and the survey results of the evaluation and perform another evaluation. Our goal is to resolve the problems with the undo operation and devise a more convenient undo operation.

5.

ACKNOWLEDGMENTS

This research was supported in part by a Grant in aid for Precursory Research for Embryonic Science and Technology (PRESTO) from the Japan Science and Technology Agency.

6.

REFERENCES

[1] T. Beriage: A Selective Undo Mechanism for Graphical User Interfaces Based On Command Objects, ACM Transaction on Computer-Human Interaction, Vol. 1, No. 3, pp. 269–294 (1994). [2] T. Seifried, C. Rendl, M. Haller and S. Scott: Regional Undo / Redo Techniques for Large Interactive Surfaces, Proc. of SIGCHI Conference on Human Factors in Computing Systems (CHI2012), pp. 2855–2864 (2012). [3] C. Appert, O. Chapuis, and E. Pietriga: Dwell-and-Spring: Undo for direct manipulation, Proc. of SIGCHI Conference on Human Factors in Computing Systems (CHI2012), pp. 1957–1966 (2012). [4] H. Masuda and A. Imamiya: Design of a Graphical History Browser with Undo Facility, and Visual Search Analysis, Journal of Systems and Computers in Japan Archive, Vol. 35, No. 12, pp. 32–45 (2004).