Actionable Knowledge Model for GUI Regression Testing Zunliang Yin1, Chunyan Miao1, Yuan Miao2 and Zhiqi Shen1 1 Nanyang Technological University, Singapore {ps6984132e, ascymiao, zqshen}@ntu.edu.sg 2 Victoria University, Melbourne, Australia
[email protected] Abstract GUI regression testing is challenging as GUIs are changed frequently during the whole SDLC. Currently, there are no effective techniques for doing GUI regression testing. In this paper, a novel Actionable Knowledge Model (AKM) is proposed for representing GUI test cases. The proposed model is adaptive to the change of GUIs. A multi-agent based framework that infusing AKM for GUI application regression testing is developed to facilitate test case creation, execution and repair. Key words: Software testing, GUI regression testing, Test case repair, Multi-agent.
1. Introduction Applications with Graphical User Interfaces (GUIs) are user friendly, easy to use and have been widely adopted in now days. But, testing applications with GUIs becomes complicated as both the underling application and its GUIs need to be tested. Following lists some major difficulties [1]: 1) The number of GUI states to be verified is enormous. 2) The permutation of input and events are extremely large. 3) There may be logic dependencies between the GUIs and the underling applications. For example, the format of an input filed must obey a set of rules. The above difficulties make automated test cases generation of GUI applications very difficult. The number of test cases to be generated is extremely large. The GUI states to be verified are needed to be produced before a test case is executed. “Although a few automated GUI test case generation schemes have been proposed [2], in practice, test cases are still being generated manually using capture/replay tools” [3]. Capture/replay approach has been used widely in regression testing of GUI applications. Practices show that this method is very time consuming and is not adaptive to the change of GUIs of an application. Most of the GUI
applications have logic dependencies between the underlying applications and the GUI elements. In addition, the permutation of input and events are extremely large. Therefore, it is not possible to generate and repair test cases fully automatically merely relying on GUIs of a running application. In this paper we present a new regression testing technique for GUIs regression testing. Instead of keeping each test case in a separate script file and in sequence format, we represent it in an Actionable Knowledge Model and we infuse agent technology to help the tester partially repair a test case without recording it from the beginning. The contributions of this paper include: 1) An Actionable Knowledge Model to present all the test cases of a GUI application. 2) Algorithms to maintain the Actionable Knowledge Model. 3) Integration of this regression testing technology into an agent based framework. In the next section we will discuss related works. Section 3 proposes Actionable Knowledge Model. Section 4 presents the agent based framework. Section 5 gives the conclusion and the outlook of the future research.
2. Related Work Capture/replay tools, such as IBM rational robot [4] and WinRunner [5] capture the interaction between user and the GUI application and store them into scrip files. At a later time, these scripts could be re-executed (replayed) to verify if a GUI application still conforms to the previous recorded interactions. Using “capture/replay” approach, each test case is stored in a script file. Such approaches have the following drawbacks: 1) It makes the knowledge about a GUI application not shareable across test cases. If the GUI is modified (GUI elements are added, deleted and renamed), we need to modify all the script files that contain the modified GUI action. For a tester who does not have programming skills, all the test cases will need to be rerecorded. 2) It is not adaptive to the change of the GUIs of an application. Any change of the GUIs will cause some of the manually generated test cases become
Proceedings of the 2005 IEEE/WIC/ACM International Conference on Intelligent Agent Technology (IAT’05) 0-7695-2416-8/05 $20.00 © 2005
IEEE
invalid and discarded and need to be re-recorded as well. In fact, GUIs of an application at testing stage are modified often due to new requirement. In order to repair test cases when the GUIs of an application have changed, in [3] control-flow graph (GCFG) and a GUI call graph (G-call graph) are used to represent the knowledge of GUIs. A G-CFG statically represents all possible interactions among the events in a component. A G-call graph shows the invoke relationships among all the components in the GUI. These graphs are automatically obtained from the GUI by performing a traversal of the GUI’s event structure [3]. Based on the use of G-CFG and G-call tree a test case repair algorithm is proposed. With the help of the G-CFGs and G-call tree for both the original and modified GUI, this algorithm repairs repairable test cases by skipping events or inserts a single new event until a legal event sequence is obtained [3]. This algorithm is suitable for the GUI applications which the input data fields have no constraints set to it, for example, a notepad application. We can enter anything in the notepad’s text field. But to automatically repair a test case generated manually by using a “capture/replay” tool, the GUI structure of a GUI application must be obtained first. As we know, not all GUI applications are like notepad. Most of the GUI applications have logic dependencies between the underlying application and GUI elements. Navigating most of the GUI applications relies on the input data from keyboard. These input data must follow some built-in constraints, such as the format of email address, identity card number, account number etc... If the correct input data can not be generated automatically and correctly, these tools could not traverse the GUI application completely and the GUI structure could not be obtained automatically. In such a case, the tester needs to generate the GUI structure manually. It is time consuming as well to make all the GUIs to appear. Even the GUI structure can be obtained; there will be problems when repairing the test case automatically using the algorithm proposed in [3]. For instance, if the algorithm tries to insert is a keyboard input event which has bounded constraints to the format of the input data, the generated event sequences will be invalid sequences.
3. Actionable Knowledge Model We define a GUI test case T as a pair (S0,a1;a2;…;an), consisting of state S0, which is the initial state for T, and an action sequence a1;a2;…;an performed by the tester during recording time [3]. A GUI action is a mouse event or keyboard inputs performed on a GUI element. For example, mouse clicks on a button, keyboard input of a text string in a text box, etc.
In simple, we can consider the process of capturing the user’s action into script files by using a “capture/replay” tool as re-construction of the knowledge about a GUI application. The capture tool does not only capture the action that could be performed on a GUI element, but also the relationships between GUI actions. In this paper, we divide user actions of a GUI test case into two categories: GUI related actions and non-GUI related actions. GUI related actions are used to navigate a GUI application or verify states of a GUI component. Non-GUI related actions are all the other actions in the test case besides GUI related actions, such as initialization action of a test case, database states verification actions, etc. Actions sequence a1;a2;…;an are ordered by the time they are performed during recording time. If no errors happened during the execution of a test case, a1 is always the first action, an is the last action and ai+1 is performed immediately after ai. Here, all the test cases of a GUI application are integrated and represented in a proposed Actionable Knowledge Model. This Actionable Knowledgeable Model includes two parts: Actionable Knowledge Graph (AKG) and Sub-Goals-List (SGL). An AKG for a GUI application is 2-tuple where: 1. V is a set of vertices representing all the actions in all the test cases of a GUI application. 2. E⊆V×V is a set of directed edges between vertices. An edge (vx,vy)∈E if the action represented by vy follows the action represented by vx at the time when the tester creates test cases. As all test cases of a GUI application are integrated into one AKG, test cases are able to share knowledge in terms of action information and relationship between two actions. For example, if several test cases have the same action flow vx→vy, then only two vertices, vx and vy, and one edge are needed for these test cases. At playing time, when a test case is to be executed, it is reconstructed from the AKG. But, the AKG is only able to memorize actions information and relationship between two actions. The ordering information of actions in a test case is missed out as more test cases are added to the AKG. To re-produce test cases correctly with an AKG, further information is needed besides the AKG. Our method is to use one or more sub-goal vertices to represent a test case. A sub-goal vertex is either the last vertex which represents the last action of a test case or is a vertex in an application’s AKG, which in-degree is greater than one. By default, a test case is represented by the last vertex as its sub-goal vertex. If a later added test case shares a vertex in the AKG with a former added test case, then the
Proceedings of the 2005 IEEE/WIC/ACM International Conference on Intelligent Agent Technology (IAT’05) 0-7695-2416-8/05 $20.00 © 2005
IEEE
newly added test case uses its last vertex and this shared vertex as its sub-goal vertices. To differentiate the shared vertex which is used by different test cases, we use an alias name, which has a suffix identified by the in-degree of the vertex at the time it is created. A SGL consists of a list of sub-goals in the order, which they are created. In this paper, all the test cases of a GUI application are represented as AKG and SGL. For example, Figure 1 shows an AKG of a GUI application, which includes two test cases, T1 and T2. T1 has an action flow, A→C→D, and T2 has an action flow, B→C→E. Using the information provided by the AKG shown in Figure 1, we are not able to re-produce test case T1 and T2. We may get wrong result. For example, T1 becomes A→C→E and T2 becomes B→C→D. The problem is caused by vertex C, where two test cases are met. T1
A
actions information and relationship between the actions of these test cases are shared in the AKG. This makes it is possible to repair one test case and make all the other invalid test cases become valid.
4. Introduction Framework
D
Assume T1 is created first, T2 is being added to the AKG. Since C already exists in the application’s AKG, if action C is the action being performed by the tester, then C becomes a sub-goal vertex of test case T2. After T2 has been added to Figure 1, T2 is represented using sub-goal vertices C and E. As T1 is created before T2, it is represented only using sub-goal vertex D. These sub-goal vertices are to be remembered when adding a new test case to the AKG. To differentiate vertex C in T1 and T2, we give an alias name C1 to represent vertex C in test case T2 and remember the C1’s preceding vertex name B as well. As a result the SGL of test case T1 is [D] and the SGL of test case T2 is [C1, E]. When reproducing a test case represented in AKG, each sub-goal in a SGL is used to create one test case segment. All the test case segments are combined together in the same order as they are in the SGL. As the sub-goal is only the vertex which in-degree is greater than one or the last vertex of a test case in the AKG of a GUI application, the new representation is shorter than the original test case action sequence. This implies a higher level representation of a test case. When a GUI application is modified, one or more test cases may become invalid. If such a test case is executed, an unexpected result will occur. As the test cases in this paper are represented in the form of AKG and SGL, the
Mediating Agent
KB Managing Agent
Replaying Agent
Tester
Interface Agent
Figure 2. agent based framework
KB
Architecture of multiGUI regression testing
User interface agent: User interface agent is a GUI agent. It is used to accept instructions from the tester and displays information generated by the mediating agent. The commands that the user could use includes “start test case”, “stop test case”, “pause test case”, “execute test case”, “repair test case”, etc. These commands are sent to the mediating agent for interpreting. Result information and further requests from the mediating agent are also displayed on the GUI of the interface agent. Capturing agent: Capturing agent is used to capture the tester’s action or GUI states when the tester operates on the GUI application under test. This agent is also used to capture the GUI states when a test case is re-executing by the agent system. Whenever a request comes from the mediating agent, it will start to capture the elements of the GUI or user input events. The capturing agent will inform
Proceedings of the 2005 IEEE/WIC/ACM International Conference on Intelligent Agent Technology (IAT’05)
IEEE
Based
Capturing Agent
Figure 1. AKG example
0-7695-2416-8/05 $20.00 © 2005
Agent
E
GUI Application
B
the
“Multi-agent systems are computational systems in which several semi-autonomous agents interact or work together to perform some set of tasks or satisfy some sets of goals” [6]. Agent oriented software engineering has been advocated as a promising trends [7]. In this research we infuse agent technology into automated software testing. To facilitate test case creation, execution and repair using our regression testing technology, an agent based framework is proposed. The architecture of this framework is shown in Figure 2. The proposed AKG model is used as the actionable knowledge model of the intelligent KB agents. The functionalities of each agent are briefly described as below.
C T2
to
the mediating agents the captured results. It could also stop and pause the capturing operation when a request comes from the mediating agent. Playing agent: Playing agent is used to perform the actions in the test case. There may be one or more playing agents in the system, such as GUI playing agent that is used to play the GUI related actions and database playing agent which is used to execute database related actions. Whenever a request comes from the mediating agent it will play the requested action and inform the mediating agent when it finishes the playing action. KB managing agent: KB agent is used to do the reasoning work for the mediating agent, including creation and search of the Actionable Knowledge. The KB managing agent does the search work based on the AKG and SGL. The AKG and SGL could be considered as its environment. This environment is created when new test cases are executed manually by the tester. Test cases could be reconstructed from the AKG and SGL for problem solving. A set of operators are defined for the KB managing agent to operate the AKG and SGL. These operators include: add_action_to_test_case(a,t), where a is a GUI action or state action, t is the name of the test case. This operator adds a new action a to test case t. delete_action_from_test_case(a,t), where a is a GUI action or state action, t is the name of the test case. This operator deletes action a from test case t. retrieve_test_case(t), where t is the test case name. This operator reconstructs the actions flow of a test case t from the AKG and SGL. find_test_cases(a), where a is a actions name. All the test cases the include action a will be retrieved. successor_action(a), where a is a action. All the successor actions and their test case names are returned. If a returned action does not have a test case name, this action is in an invalid GUI action sequence. When the tester is creating a test case, the test case name and actions are received from the mediator agent. The KB managing agent adds each action of this test case to the AKG and creates a SGL for this test case. It could also suggest and execute flowing actions for the tester. When the tester is doing regression testing, a set of top level testing goals are created based on the SGLs. Subgoals are also generated for each top level goals. For each sub-goal, the KB managing agent retrieves actions from the AKG. These actions will be sent to the playing agent for execution. If all the actions are executed by the playing agent successfully, this sub-goal is achieved. Otherwise the top level goal is dropped and another top level goal is selected. When the tester is repairing test cases, according to the action name the user keys in, it finds all the test cases that include this action and selects one test case for execution. If the playing agent has executed all the actions which
precede this action successfully, an interactive process begins. The KB managing agent adds all of the user’s new actions to the test case’s AKG and SGL and drops the actions which are between the user’s newly inputted first action and last action from the AKG and SGL with the agreement of the tester. This will continue until all the affected test cases are repaired. Mediating agent: This agent plays the role of mediating in the system. It processes the tester’s command, relays information between agents. It also records all the testing logs
5. Conclusion In this paper, we propose a multi-agent based framework for automating the GUI application regression testing. To make a test case more adaptable to the change of the GUI of a GUI application, we define an Actionable Knowledge Model to represent GUI test cases. Algorithms to enable the agents to operate the Actionable Knowledge Model are also proposed. A multi-agent based framework which uses the Actionable Knowledge Model to facilitate test case creation, execution and repair has been presented. A prototype multi-agent system is under developed. Evaluations for using proposed framework in real world applications will be carried out.
6. References [1] D. J. Kasik and H. G. George, “Generating test cases for GUI responsibilities using complete interaction sequences”, Proceedings of the International Symposium on Software Reliability Engineering, Oct 2000, pp. 110-121. [2] A. M. Memon, M. E. Pollack, and M. L. Soffa, “Hierarchical GUI Test Case Generation Using Automated Planning”, IEEE Transactions on Software Engineering, vol. 27, no. 2, Feb 2001, pp.144-155. [3] A. M. Memon and M. L, Soffa, “Regression testing of GUIs”, Proceedings of the 9th European software engineering conference held jointly with 10th ACM SIGSOFT international symposium on Foundations of software engineering, Vol. 28, No. 5, pp. 118-127, Sept 2003. [4] Rational Robot, http://www306.ibm.com/software/awdtools/tester/robot/, Jan, 2004. [5] WinRunner, “Test automation for the enterprise”, http://www.mercuryinteractive.com/products/winrunner/, Jan, 2004. [6] Victor R. Lesser, “Multiagent Systems: An Emerging Subdiscipline of AI”, ACM Computing Surveys, Vol. 27, Issue 3, Sep 1995, pp. 340 -342. [7] N. Jennings, "On Agent-Based Software En-gineering", Artificial Intelligence, Vol.117 (2), 2000, pp. 277-296.
Proceedings of the 2005 IEEE/WIC/ACM International Conference on Intelligent Agent Technology (IAT’05) 0-7695-2416-8/05 $20.00 © 2005
IEEE
