Generation of an Operation Learning Support System by Log Analysis Hajime Iwata
Junko Shirogane
Yoshiaki Fukazawa
Department of Network and Communication Kanagawa Institute of Technology Atsugi-City, Kanagawa, Japan
[email protected]
Department of Communication Tokyo Woman’s Christian University Suginami-ku, Tokyo, Japan
[email protected]
Department of Computer Science and Engineering Waseda University Shinjuku-ku, Tokyo, Japan
[email protected]
Abstract— End users' experiences, which differ drastically from user to user, influence the perceived usability. Thus, to make software user friendly, it is important to support learning operation methods for end users. To this end, tutorial systems have been proposed. Although referring to the performance of various users has been effective in learning operation methods using tutorial systems, which operation sequences are more frequently performed must be analyzed. Herein we propose a method to generate a tutorial system by analyzing the operation logs of end users and by extracting window sequences from source programs within the software.
II. PURPOSE OF OUR WORK In our method, a tutorial system is generated by analyzing end users' operation logs. In many cases, users begin to use functions from a specific window, called the “main window”. Thus, the main window is considered to be the starting point for using the software, and our method assumes that this window is displayed when the software starts. To generate a tutorial system, both the sequence of frequently performed operations by end users and a description of the operation method for end users are required. Our method employs operation logs to acquire sequences of operation methods. However, two difficulties arise in analyzing operation logs:
Keywords- component; Log Analysis; Tutorial System; Window Sequences
I.
INTRODUCTION
z
Recently, software has been developed considering usability [1]. However, the perceived usability is influenced by end users' experiences; thus, it is difficult to develop usable software for all end users. Additionally, software has numerous functions, which is another reason end users often find software difficult to use. Hence, providing support is important to resolve this situation as well as an effective means for end users to learn frequently performed operation methods. To learn how to operate software, end users often employ a tutorial system that simulates the software running [2]. End users can learn the sequences of operations as if they were actually using the software. The advantage of this type of tutorial system is that end users interactively learn how to operate the software. Thereby, users can understand operation methods effectively using the tutorial system. However, there are currently few built-in tutorial systems for software. Because the structures of software have become highly complicated and have many functions, the development of a tutorial system places a heavy burden on software developers and is expensive. In this paper, we propose a method to automatically generate a tutorial system based on the operation logs of end users and the source programs. Using our method, a tutorial system can be easily developed.
z
Extracting individual units of sequence in the operation method (unit flow) Determining frequently used combinations of unit flow
A unit flow indicates a sequence in the operation method from the beginning to the end of a function. In our method, a function begins by operating the main window, continues during window switching, and ends upon returning to the main window. Thus, a unit flow is extracted from the operation logs by analyzing window switching. As mentioned above, in many cases, functions are selected from the main window. Thus, one unit flow consists of the following: 1. End user opens the main window 2. Main window switches to another window 3. Window in closes and main window returns Figure 1 shows an example of unit flow where the main window for software of an address book is displayed on the left. This main window is the starting point for operation functions such as “Registration” and “Confirmation”. Clicking the “Registration” button in the main window causes another window, which registers a new address, to be displayed. In this new window, the end user inputs an address and then pushes the “Confirm” button, which completes the registration,
207
window of the address book software. The end user selects a combination flow operation from the combo box, and clicks the “Tutorial start” button to learn the selected operation. In this example, the end user selects “[Registration] next to [Confirmation] next to [Edit]” combination flow.
closes the window, and returns the end user to the main window. After this, the end user can operate the main window again. These window sequences are written in the operation logs. Our system analyzes this “Registration flow” from the operation logs. In our method, like these users operation flows are called unit flows.
Figure 2. Example of a tutorial system
III. SYSTEM ARCHITECTURE
Figures 3 and Figure 4 depict the system architecture of our system, which consists of six steps: 1. 2. 3. 4. 5. 6. Figure 1. Unit flow of "Registration of address information"
Frequently performed unit flows extracted from the operation logs are important. Thus, frequent unit flows are shown to developers, who then select unit flows appropriate for learning operation methods. However, unit flow is too detailed to show to end users as the operation. For example, even if only the tutorial of “Registration of address information” is repeatedly seen, end users cannot understand the flow of the entire operation. So, in our method, our system connects unit flows to show them as tutorials to end users. A combination flow is a set of connected unit flows. Because our system analyzes common combination flows, it sets the appearance of unit flows that follow the base flow in the operation log, and then counts these appearances. An example of a commonly used combination flow in the address book software is the end user “Registration of address information” followed by “Print address information onto an envelope”. Figure 2 shows an example of a tutorial system for software from an address book. The right side is a window of the tutorial system generated by our method, while the left is a
Adding logging tools Extracting unit flows from the operation logs Analyzing operation methods Selecting the appropriate unit flows Adding descriptions of the operation methods Adding a tutorial system into target automatically
Figure 3. System summary
208
software
z z z
Inputting each text field (name, nickname, address, telephone number and e-mail address) Selecting sex, i.e. “Male” Clicking the “Apply” button
Figure 5 shows an example of a log. The tag is time when end user's operation occurred. The tag is variable name in the source program of the software. In this example, “entryBtn” is the variable name for the “Registration” button and “txtName” is the variable name for the text field to input “Name”. The tag is the variable name for the operated window in this operation timing. In this example, “AddressbookManager” is the variable name of the window when an end user clicks the “entryBtn” button. is the end user's operation of the software. If this widget is a text input type (i.e. text field and text area), this tag is a character determined by the end user’s typing. In this example, the action is “Click” for the “entryBtn” button and the action is “A” for the “txtName” text field. Figure 4. Our system architecture
A. Addition of a logging tool In our method, our system adds a logging tool to target software. This logging tool records the user’s actions, which are GUI widgets of software that the end user operates, such as clicking buttons and inputting text fields. To automatically generate the tutorial system, the following items are obtained by the logging tool: z z z z
Time that actions occur Widget names of end users’ actions Window names that widgets operated by end users are arranged Types of end users' actions
The widget name recorded depends on the operation of the end user because widgets are grouped according to type of action. Moreover, operated widgets include text fields or text area and input strings. To extract a widget name from the source programs, our method generates a source program analysis tool using JavaCC (Java Compiler Compiler) [3]. JavaCC is a parser generator and a lexical analyzer generator for Java programming. The extracted data includes variables of each window from the source program as well as widget names and types in each window. Our system generates a logging tool based on these extracted data as a source program of AspectJ [4]. AspectJ is one of the software tools that uses aspect-oriented programming for Java programming. Using the address book software as an example, the operation of registering a new address encompasses the following: z
Figure 5. Example of a log file
B. Extraction of unit flows End user’s operations occur a number of times in one widget. Examples of these operations include selecting an item in the radio buttons, choosing items in the list, and rewriting to the text field. All detailed operations are recorded in operation logs. In our method, log entries for the same widget are merged in each window. Our system analyzes the merged log entries because operations to a widget are represented once in the generated tutorial system. For each widget in the same window, our system records the end users’ actions. We assume that unit flow begins with end user’s operation of the main window, and it ends when all
Clicking the “Registration” button
209
other windows are closed and the end user returns to the main window. Additionally, we assume that the unit flow continues if a new window is opened. Hence, unit flow indicates the purpose of the users.
C - 1. Reappearance of one unit flow When an end user learns an operation by a tutorial system, the end user watches same operation only once. If one unit flow is repeated with a combination flow, then it is not one large combination but a series of smaller combination flows. This means the end user will watch the same operation procedure in a tutorial system once, and in this case, the end of the operation is unclear. Thus, this combination flow is not suitable as a tutorial, and the sequence of unit flow is aborted before the combination flow is analyzed. For example, to create a combination flow in the address book software when [“Register flow” next to “Confirm flow” next to “Register flow” or “Confirm flow”], the appropriate combination flow is [“Register flow” next to “Confirm flow”].
C. Analyses of the operation methods Unit flows are used to generate a tutorial system. How often a unit flow is analyzed using the operation log. Our system counts the number of appearances for each unit flow. The most frequent unit flow is defined as the base flow, and typically other unit flows are connected to the base flow. A set of connected unit flows is called a combination flow in our method. Our system sets the appearance of unit flows that follow the base flow in the operation log, and counts these appearances. Moreover, our system contains an aborting condition to control the combination flows, which is invoked under three different conditions: 1. 2. 3.
C - 2. Insufficient occurrence of unit flows appearing in succession After the logs are analyzed, the frequencies of unit flows are counted. Low frequency combination flows are not suitable for tutorials. Thus, combination flows with low occurrences are aborted. For example, in the address book software, combinations where “Register flow” is preceded by another unit flow [i.e. after “Confirm flow” or “Edit flow” are aborted because they are rare.
Reappearance of one unit flow Frequency of unit flows appearing in combination is low Insufficient combinations with beginning with one type of unit flow
Figure 6 is an example of determining combination flow processes from the operation logs of the address book software. “Register flow” means a unit flow of registration, “Confirm flow” indicates a unit flow of confirmation, and “Edit flow” means a unit flow of edit.
Figure 7. Example of an analyzed combinaion flow list
C - 3. Appearance of few combination flows beginning from one type of unit flow Combination flows with a low frequency of appearance are not employed by end users, and thus, are unsuitable candidates for the tutorial. For example in the address book software, combination flows beginning with “Edit flow” are rare; thus, combinations flows beginning with “Edit flow” are removed as candidates in the tutorial. Figure 7 shows an example of extracted combination flows from the address book software operation logs.
Figure 6. Example of creating combinaion flows
210
D. Selection of appropriate unit flows Of the analyzed combination flows, many are suitable for tutorial operations. Unit flows included in a combination flow are presented to developers as part of the combination flow because unit flows may not be appropriate for tutorial operations. For examples, “Return from the error of the operation” and “Interruption of the operation because end user canceled” are not suitable as tutorial operations. Thus, the developer removes unit flows unsuitable for a tutorial operation.
TABLE I.
E. Description of data for presentation After specifying combination flows, target software developers are required to add extra descriptions, which are texts to explain how to operate the software functions. These extra descriptions are displayed to end users as explanations once the tutorial system is initiated. In this case, some widget types, such as text fields, require text data. The tutorial system should show an example of the input operation. Because software developers cannot describe all combination flows, developers describe each unit flow. Our system labels a combination flow based on the order of appearance of the unit flow.
2. 3.
z z z z
Mail Order System
Unit flows analyzed by hands
54
90
Unit flows analyzed by our method
54
90
Window switches
848
768
Number of operation logs in the total operation records Number of unit flows of the total analyzed by our method Number of total analyzed combination flows by our method Number of combination flows that appear multiple times TABLE II.
A process for displaying a sequences of operations based on a combination flow is generated for a tutorial. An operation procedure based on the unit flow is constructed. According to the constructed operation procedure, a process to demonstrate the mouse pointer movement from the location of the GUI widgets and end user operation is generated by unit flow.
RESULT OF COMBINATION FLOW Video Rental System
Mail Order System
Total operations
2562
3758
Total unit flows
54
90
Total combination flows
11
21
Combination flows appears more times
3
4
Table 2 shows the results. From the 54 unit flows, our system extracted 11 combination flows for the video system. Similarly, 21 combination flows were extracted from the 90 unit flows for the mail order system. Each combination flow is used in multiple procedures by the end user. Thus, by counting each different combination flow once resulted in three and five combination flows for the video rental system and mail order system, respectively. The generated combination flows can be evaluated as either resisting data or confirming registration content.
Additionally, target software developers can add the generated tutorial system to the target software by compiling the generated Java source programs and weaving the generated AspectJ source programs into the target software. IV.
Video Rental System
Both the data analyzed by hand and using our method gave the same number of extracted unit flows. Consequently, it was determined that our method is capable of analyzing unit flow. Next four elements, which are listed below, were examined from the operation logs from the mail order software and video rental software.
F. Automatic generation and weaving of a tutorial system into target software A tutorial system is automatically generated after extracting the combination flows and information on how to operate functions by unit flow. This procedure is as follows: 1.
RESULT OF UNIT FLOW
EVALUTATION
We evaluated the accuracy of our method in analyzing unit flows from the log files. The target software samples are a video rental system and a mail order system [5]. The login window was excluded from the evaluation because our method requires the first displayed window to be the main window. Three elements were evaluated: Number of unit flows analyzed by hand, number of unit flow analyzed by our method, and the number of windows switches. Table 1 shows the results.
From these results, our method analyzed the procedure of typical operations via the operation logs. Because these combination flows are the end users' operating procedures that correspond to the source program of the software, it is possible to reproduce the operations. Therefore, we conclude that our method is effective in the operation learning support of software.
211
V.
RELATED WORKS
VI.
CONCLUSION
In this paper, we propose a method for developers to weave a tutorial system into existing software based on the operation logs. Using this generated tutorial system, end users can easily learn operations of the software. Future studies will examine situations where the initial starting window is not the main window and automatically removal of unit flows not suitable as tutorial operations.
Flaithri et al. have proposed a method to create an auditory help system that focuses on the task-oriented user interface model [6]. In this method, a user model to achieve the operation for the tutorial must be prepared. D. V. Koznov et al. have proposed a method to create user documentation, a support site, and a help system [7]. In this method, product line paradigm and technologies for developing documentation include means for the reuse of parts of the documents. Additionally, Miura et al. have proposed a method to create a demonstration-authoring tool for Java applets [8]. In this method, developers add event-driven functions to a Java applet, and the event for automation is recordable from the operation. Although this method creates help functions for Java applet software, a new rule to introduce the concept of event-driven functions must be defined to generate a tutorial system. Our method presents event-driven functions to the developer by narrowing the number of combinations to analyze based on window transitions. Therefore, our method imposes a smaller burden than the aforementioned methods.
REFERENCES [1]
[2]
[3] [4] [5] [6]
[7]
[8]
212
M. B. Rosson and J. M. Carroll, “Usability Engineering: Scenario-Based Development of Human Computer Interaction”,Morgan Kaufmann Pub., 2001. B.Shneiderman and C.Plaisant, “Designing the User Interface: Strategies for Effective Human-Computer Interaction 4th Edition”, Addison Wesley, 2004. JavaCC HOME, https://javacc.dev.java.net/ The AspectJ project, http://eclipse.org/aspectj/ G. Schneider, and J. P. Winter, “Applying Use Cases: A Practical Guide”, Addison-Wesley Professional, 1998 F. Neff, A. Kehoe and I. J. Pitt, “User Modeling to Support the Development of an Auditory Help System”, Proc.of 10th International Conference on Text, Speech and Dialogue (TSD2007), pp.390-397, 2007 D. V. Koznov and K. Y. Romanovsky,” DocLine: A method for software product lines documentation development”, Programming and Computing Software Vol.34, 4 pp 216-224, 2008 M. Miura and J. Tanaka: “Jedemo Demonstrational Authoring Tool for Java Applets”, Proc. of the 5th World Conference on Integrated Design and Process Technology (IDPT2000), 2000.
