2010 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery
Scenario-Based Approach for Blackbox Load Testing of Online Game Servers Chang-Sik Cho, Dong-Chun Lee, Kang-Min Sohn, Chang-Joon Park
Ji-Hoon Kang Dept. of Computer Science and Engineering Chungnam National University Deajon, Korea
[email protected]
Smart Game Team ETRI Deajon, Korea {cscho, bluepine, sogarian, chjpark}@etri.re.kr
online games. To meet such requirements for game servers, many game testers are involved in testing, and a large number of game players actually participate in beta testing before deploying a game. In order to reduce such testing time, the use of testing automation is recommended [2-5]. Traditional server load testing technologies such as LoadRunner [6] have been applied on online game testing, but various game actions cannot be easily simulated by simply and partly modifying the game packets of actual players. Thus, The Sims Online (TSO) automation [3], [7] and the Virtual Environment Network User Simulator (VENUS) system [8], [9] used a virtual client engine with a subset of actual game clients. Both the TSO automation and the VENUS system have saved many person-hours, leading to better game quality, and have been more efficient than manual testing. However, they both have some weak points. First, both use a subset of the main game client for testing the client code. In other words, blackbox testing is not supported. Though using an existing game client has minimized the need to create new code, the test client code should be rewritten for new game testing. Their approaches can be applied to game developers only and are not adequate for game publishers because the developed game client code must be supplied and rewritten for game testing. These days, testers in game publishing side, need to test the games without the game source code. Second, the TSO automation and the VENUS system do not support scenario-based testing. Their testing has been achieved using the unit of the simple command script. Though both have provided more flexible testing actions compared with traditional server load testing technology, it is difficult for them to test interactive actions such as multiuser collaborative play. Complex and various test scenarios such as party play and waypoint movement still have not been supported. However, game testers want to simulate complex scenarios and mimic real testing environments instead of doing simple load testing. In this paper, we propose blackbox testing and scenario-
Abstract— Simply having a large beta test cannot consistently provide stability and performance to game servers, which are major issues in online game development. Therefore, test automations have been used in order to reduce the testing time of online games by simulating highly repetitive tasks and emulating server loads. However, in previous approaches, blackbox testing and scenario-based testing are not supported because they use prerecorded packets of real players as templates, or reuse a subset of the main game client for the test client. In this paper, we propose blackbox testing and scenario-based testing of online games as well as simple load testing. Instead of rewriting the virtual client dummy code, only the game description language and virtual game map are redefined when a new game is to be tested. In addition, an actual testing environment can be mimicked more closely, because complex and various scenarios such as attack, party play, and waypoint movement can be tested through combining actions. We have applied our tools on several online games to verify the effectiveness of our method. Online Game Testing; Load Test; Massive Virtual Users; Blackbox Testing; Scenario-based Testing; Game Description Language; Virtual Game Map (keywords)
I.
INTRODUCTION
A massively multiplayer online game (MMOG) is a multiplayer video game that can be played via a game server over the Internet with players from around the world. MMOGs can enable players to cooperate and compete with each other on a large scale, and include a variety of gameplay types, representing many video game genres[1]. MMOGs are deployed using a client-server system architecture and can actually be run on multiple separate servers. The servers must be able to handle and verify a large number of connections. Thus, the stability and performance of game servers have become major issues in 978-0-7695-4235-5/10 $26.00 © 2010 IEEE DOI 10.1109/CyberC.2010.54
259
based testing of online games, as well as simple load testing. The game description language and virtual game map are used to achieve these goals. The massive virtual clients automatically generate packet loads to test the stability of game servers as in the existing approaches. However, by describing game logics and test scenarios using the game description language and virtual game map, the test client code itself does not need to be rewritten for new game testing. The virtual clients automatically generate packet loads according to the defined game description language, and complex scenarios such as attack, party play, and waypoint movement can be tested by combining actions. The rest of this paper is organized as follows. Section II shows examples of game testing scenarios and actual game packets to help explain the conceptual model of our method. Section III explains the syntax of the game description language and section IV shows the implementation results. Section V shows the comparison with previous approaches. Finally, section VI concludes our paper with a description of future work. II.
game. The TSO team used a subset of the main game client to create a test client in which the GUIs are mimicked via a script-driven control system. LoadRunner has been used to control up to thousands of simulated users against a given candidate server cluster. A bridge code has also been used to hook LoadRunner into a TSO test client. The method of TSO automation cannot be used on the publishing side because testers cannot access the game source code. Also, in the TSO automation, testing has been achieved by a simple command script, not by test scenarios. The VENUS system has been developed to provide a beta test environment to game developers in order to test online games efficiently. Using a virtual client engine in the VENUS system, the user can easily simulate online games for beta testing. The load-limit point and bottle-neck of a game server can be verified by simulating a massive number of virtual clients in an online game environment. Further, the VENUS system is very helpful because it stores an internal game data from the game server in a database. The VENUS system has been applied to many different online games to test its usefulness. However, dummy game clients should be rewritten by using the VENUS Software Development Kit (SDK) for new game testing. Also, its testing is focused on simple load testing instead of scenario-based testing.
RELATED WORK
Developing and deploying a permanent massively multiplayer world is very difficult due to the distributed nature and large scale nature of the game system. In general, game developers use a set of beta tests to achieve stability for the online game server. Selected beta testers play the game and send feedback information to the game testers. Then, the game developers analyze the error reports from the feedback and debug the game applications. However, this ordinary testing process consumes a lot of development resources such as cost and time. In many cases, the error reports from the beta testers provide only vague clues of a particular error to the testers, which makes it quite difficult for the developers to repeat the error environment under the same conditions that the beta tester experienced [2]. Server testing technology using virtual clients has been actively studied. Industry-standard load testing solutions such as LoadRunner [6], QALoad [10], and e-Load [11] can emulate hundreds or thousands of concurrent users by reusing captured packets. These solutions enable measuring end-to-end performance, diagnosing applications and system bottlenecks, and tuning for better performance. They help reduce the costs and time required to test and deploy new applications and systems. However, their key infrastructure components are Web servers, database servers, and so on [12-14], so cannot be applied easily to game servers. It is difficult to test the stability of an on-line game server application using a conventional packet reuse method because an on-line game server employs complicated logic compared with a general server application [15-16]. Online game server testing has introduced new research issues in this area. Game test automation has been used in The Sims Online
III.
ONLINE GAME TESTING SCENARIOS
Conceptually, a test scenario is a sequential list of actions with corresponding positions. The actions are meaningful user activities such as logins, movement, combat, and so on. Fig. 1 shows an example of a test scenario. As shown in the example, a test scenario is made up of a sequence list of a position and action pair.
Scenario = ( List (position, action))
Figure 1. Example of a game testing scenario.
260
The test scenario of Fig.1 is described as follows. ((A, Generate_Users) (B, Gather_Minerals) (D, Sell_Items))
Hunt_Mobs)
(C,
A, B, C and D are the position data and Generated_Users, Hunt_Mobs, Gather_Minerals and Sell_Items are the names of actions. In our method, the position information is represented in a virtual game map and the actions are represented in the game description language. That is, all the game logics are described by defining the game description language and virtual game map, so the test client code does not need to be rewritten when a new game is to be tested. A. Virtual game map
Figure 2. Virtual game map.
A virtual game map is a collection of x, y, and z positions and contains the texture information and building objects. The process by which a virtual game map is generated is as follows. At first, the virtual game map is blank. The actual game players or virtual users randomly and massively move around the game world. The game world is gradually searched using the position information gathered by game users’ movements. The packet analyzing tool determines which areas the game users are and are not allowed to visit. Finally, the tester can modify the map textures and place building objects such as villages, stores, and hunting grounds manually.
B. Example of game packets To explain the game description language conceptually, Fig. 3 shows an example of ELMA game protocols. ELMA is a typical MMORPG in its alpha testing period, and is used to test the feasibility of our system. In ELMA, the packets always start with 0x21, and the timestamp, packet length, protocol ID, and parameters follow. Its endianness is little endian, and it has no encryption rules. The protocol ID is in 3-byte hexadecimal form, and the number of parameters is dependent on each protocol ID. As shown in Fig. 3, the IDs of the move protocol are 0x46, 0x6, and 0x1, and those of the attack protocol are 0x51, 0x7, and 0x2. The move protocol consists of four parameters, and the attack protocol consists of five parameters. The parameters of the attack protocol are character ID, attack type, damages, target ID, and animation type, and the data types are short, integer, integer, byte and integer.
In a virtual game map, game objects such as virtual users, actual game players, and NPC objects are displayed. The testers can view the position and status of game objects by selecting them in the virtual map. Also, the tester can command the actions for a designated game object. Waypoints can be pointed to by mouse clicks, and a group of virtual users can move according to the particular waypoint.
Figure 3. Example of ELMA game packets.
261
IV.
string is defined with the size.
GAME DESCRIPTION LANGUAGE
The game description language consists of actions, packet protocols, and packet generation rules. The actions consist of a series of packet protocols, and the packet protocols define the individual packet messages transmitted between the game server and game client. The packet generation rules define the overall packet structure rules such as encryption, endianess, timestamp, and so on. Fig. 4 shows the syntax of the game description language, and Fig. 5 shows the relationship between the scenario and game description language.
Examples of ELMA elements are as follows. (Starter, byte, 1, (0x21)) (TimeStamp, Int, 1, (0xcf, 0xc, 0x5, 0x4) (Move_protocol_ID, byte, 3, (0x46, 0x6, 0x1)) (Characte_ ID, Short, 1, (0xb2, 0x7, 0x2)) (Attack_Type, Int,1, (0x4, 0x0, 0xde, 0x3)) (Target_ID, byte, 7, (0xd1,0x48, 0xd0,0x3f,0x5d,0xf5 0x2a))
The protocol consists of a name and one or more elements.
Game_Description_Language= (Protocol, Action, Generation_Rules) Protocol = (Element) Action = (Time_Constraint, Socket, Protocol, Parameter) Generation_Rules = (Endianess_Rule, Encryption_Rule, TimeStamp_Rule, HeartBeat_Rule, ...)
Protocol= (name, list (element)) The ELMA attack protocol example shown in Fig. 3 is as follows. (ATTACK, (Starter, TimeStamp, Length, Attack_protocol_ID, Attack_Type, Damages, Target_ID, Animation_ID))
Element = (Name, Type, Length, Byte_string) Figure 4. The syntax of the game description language.
An action is a sequence of protocols. It consists of the time constraint, socket, and protocol. Time constraints describe the time relationship for two protocols. They include send, receive, sendWait, and delay. There are many kinds of servers such as a gateway server, login server, world server, and so on, so the socket description should also be defined with each protocol.
Figure 5.
Action = (Time_Constraint, Socket, Protocol, Parameter) Time_Constraint = (Send, Receive, SendWait, Delay) Send : Send protocol Receive : Receive protocol SendWait : Send protocol and Wait until receiving server’s response Protocol specified in parameter Delay: Wait and delay the time intervals (ms) specified in parameter Socket: Network IP and port description Protocol: Previously defined protocol name
The relationship between scenario and game description language.
An element is the basic component of the game description language. It consists of name, type, size and byte_string.
An example of ELMA login actions is as follows.
Element = (Name, Type, Size, Byte_string)
##(Send)(Socket1)(SendDefault_ID)## ##(SendWait)(Socket1)(Login _ID)(Login_ID_Responce)## ##(Delay)(Socket1)300## ##(SendWait)(Socket1)(Char_List)(Char_Lis_Responce)## ##(Receive)(Socket1)(Live_Msg_ID)## ##(SendWait)(Socket1)(Enter_World)( Enter_World _Responce)##
The byte_sting is for the reuse of actually captured packets. Not all packet protocols need to be defined in the game description language. In some cases, the captured byte string can be reused as it is. In this case, the date type is fixed with the byte, and the length of the captured byte
262
V.
IMPLEMENTATION RESULTS
virtual game map. The process of defining the game description language is as follows. First, the position and length of the packet header and protocol ID is analyzed. Then, the packet generation rules such as encryption and endianess are analyzed. After the overall structure and the packet generation rules are defined, individual protocols such as a move, attack, and skill are analyzed. Finally, actions are defined. Fig. 7 shows screenshots of the packet analyzing tool. It shows the process of defining the ATTACK protocol, defining in particular the Attack_Type parameter. After the user selects a packet range in the individual packet data pane, the name, type, and size of the element are defined. On the right-hand side, there is a protocol view pane listing the already defined protocols.
We have implemented a VENUS II system, which is named after the previous VENUS system, and it has been applied on four online games to verify the effectiveness of our method. The applied game genres [1] include MMOG, multiplayer online game (MOG), and casual game. In MOG and casual game, the testing is rather simple, includes entering, leaving, and re-entering rooms repetitively, and a virtual game map is not used. In this section, the application to ELMA will be explained in detail. A. Overall system architecture The VENUS II system consists of a packet analyzing tool and virtual user control tool. Fig. 6 illustrates the overall architecture of the VENUS II system. The actual game packets transmitted between a game client and game server are captured and stored. Then, the stored packet lists are analyzed, and the game description language and virtual game map are generated. The Winpcap [17] library has been used for filtering and capturing the game packets. Game Client
Packet Analyzing Tool Virtual Game Map
Virtual Game Map Analyzer
N E T W O R K
Packet Capure
Game Description Language Analyzer
Game Description Language
Virtual User Control Tool User Interaction Manager
Virtual Game Map Manager
Virtual User Manager Virtual User 1
……
Figure 7. Packet analyzing tool defining the game description language.
Virtual User n
C. Virtual user control tool
Game Server
A virtual user is an activity entity of a single online game user. To the main game server, these test clients look identical to an actual connected game player. To generate a heavy load and provide a set of stresses to the game servers, a plurality of virtual users should be created and controlled. Fig. 8 shows a scene of the virtual user control tool. In the virtual user control tool, there are three window panes, a user interaction pane, virtual game map pane, and virtual user property pane. In the virtual game map in the center, the virtual users and mobs are displayed according to their position using circles and triangles. The red bar below the shape indicates the HP value of the game objects, and the color of the shape indicates the states of the objects such as in combat, walking, dead, and so on. When the tester selects a virtual user in the virtual game map, the detailed properties of the virtual user are shown in the property pane window. The virtual user control tool generates an actual load when the tester inputs the actions in the user interaction
Figure 6. The system architecture of VENUS II.
The game description language and virtual game map are referred to in the virtual user control tool. The virtual user control tool consists of a user interaction manager, virtual game map manager, and virtual user manager. The virtual game map manager shows the position and status of game objects. The user interaction manager constructs test scenarios by selecting the game objects in the virtual game map and inputting the actions sequentially. The virtual user manager generates actual packet loads according to the defined game description language. B.
Packet analyzing tool
The packet analyzing tool captures and analyzes packet data, and generates the game description language and
263
position and hunting mobs, i.e., it is a waypoint scenario. The activities of the virtual users can be viewed and verified from the viewpoint of an actual player’s game client.
pane. A large number of virtual users can be generated with group and action commands being applied to the group of virtual users at the same time. The group of virtual users can select and attack a victim mob at one time, or each virtual user can select and attack its own mob independently. The action is converted into a sequence of protocols internally, and the generation rule is applied to each protocol. Sending of the packet protocol can be delayed for the designated time interval according to the time constraint defined in the actions. The login/logout, directional movement, and attack actions are provided with button control, and test scenarios such as party play and waypoint movement are supported. In our experiment, 1,000 ELMA virtual users have been generated and are controlled in a 2.8Ghz Intel Core2 Quad CPU PC.
Figure 9. A screenshot of the actual ELMA game client.
VI.
COMPARISON OF ONLINE GAME TESTING
Table 1 shows a comparison of existing game testing approaches and the proposed method. Our approach is compared with existing ones in the viewpoint of the blackbox testing and scenario-based testing. The TSO automation and the VENUS system are based on whitebox testing. Their approaches can be applied to game developers only and are not adequate for game testers because the developed game client code must be supplied and rewritten for game testing.
Figure 8. Virtual user control tool
Fig. 9 shows a scene from an actual ELMA game client when the virtual users are moving toward a designated
TABLE I. Approaches Categories
Blackbox testing
Scenariobased testing
COMPARISON WITH PREVIOUS APPROACHES
HP LoadRunner [6]
TSO Automation [7]
VENUS [8]
Our Paper
Reuse game client code partialy whitebox
Capture, analyze and generate GDL blackbox
Main idea
Reuse captured data
Backbox/Whitebox Requiring game client code Easy application to another game
blackbox
Reuse game client code partialy whitebox
No
Yes
Yes
No
Hard
Impossible
Impossible
Easy
Game map support
No
No
No
Yes
Unit of testing
Protocol
Action
Action
Scenario
Flexibility to various test scenario Dynamic game data support
Hard
Easy
Easy
Easy
Parameterization and correlation
ActionScript (proprietary)
SDK (code itself)
GDL (action and protocol)
264
REFERENCES
HP LoadRunner supports blackbox testing by reusing captured packet data. And it supports partially the dynamic content with parameterization and correlation. The dynamically created server responses and the correlated data can be handled easily in the general server such as web server, but it is hard for them to be applied to game testing because an on-line game server employs complicated logic compared with a general server application. The TSO automation and the VENUS system reuse the game client code itself, so various scenarios can be tested easily, but the game map is not supported in scenario testing and their testing unit is action. Our approach overcomes the weakness of previous approaches and supports blackbox testing and scenariobased testing. Game logics are easily expressed with game description language and game map, and testing scenarios are easy and intuitive.
[1] [2] [3] [4]
[5]
[6] [7] [8]
[9]
VII. CONCLUSIONS
[10] [11] [12]
In this paper, along with simple load testing, we propose blackbox testing and scenario-based testing of online games. In previous test automation, the developers must provide their client code to the game testers, because a test client code is based on a dummy game client code without a GUI. In our approach, the game logic is described using a game description language and virtual game map, so the test client code does not need to be rewritten when a new game is to be tested. The virtual clients automatically generate a packet load according to the game description language. Moreover, complex scenarios such as attack, party play, and waypoint movement can be tested by combining actions. To verify the effectiveness of our method, we have applied the VENUS II system on several online games. Among them, application to the ELMA online game has been explained in detail. The login/logout, directional movement, and attack actions are defined using the game description language in the packet analyzing tool, and the virtual user control tool has generated and controlled 1,000 ELMA virtual users in a 2.8Ghz Intel Core2 Quad CPU PC. The stability of the ELMA game severs has been verified using our tools. Future research is planned to improve our VENUS II system, provide more intelligent test scenarios, and make it a standard model for online game testing.
[13] [14]
[15]
[16]
[17]
ACKNOWLEDGMENT This work was supported by the IT R&D program of MKE/IITA. [2009-S-041-01, Online Game Quality Assurance Technology Development using Scenario Control of Massive Virtual User]
265
Game genres. [Online]. Available at http://en.wikipedia.org/wiki/Game_genres. C. P. Schultz, R. Bryant, and T. Langdell, Game Testing All In One, Thomson Course Technology PTR, 2005. L. Mellon, B. Dalton, G. Walton, and D. West, "Large-scale engineering for online and offline games," in GDC 2007, Spring 2007. Y. Deshpande, R. Jenkins, and S. Taylor, “Use of simulation to test client-server models”, Proceedings of the 1996 Winter Simulation Conference, pp. 1210-1217, November 1996. D. Wangerin, C. DeCoro, L. Campos, H. Coyote and I. Scherson, “A modular client-server discrete event simulator for networked computers”, Proceedings of the 35th Annual Simulation Symposium, pp. 125-133, April 2002. LoadRunner. [Online]. Available: http://www.hp.com L. Mellon, “Automated testing of massively multi-player systems: Lessons learned from The Sims Online,” GDC 2003, Spring 2003. B. H. Lim, J. R. Kim, and K. H. Shim, “A load testing architecture for network virtual environment,” Proceedings of 8th International Conference on Advanced Communication Technology, pp. 848-852, February 2006. Y. W. Jung, B. H. Lim, K. H. Sim, H. J. Lee, I. Park, J. Chung, and J. Lee, “VENUS: The online game simulator using massively virtual clients,” in Proceedings of AsiaSim 2004, pp. 589-596, October 2004. QALoad. [Online]. Available: http://www.compuware.com/ Empirix, OneSight. [Online]. Available at http://www.empirix.com/ S. Elbaum, S. Karre, and G. Rothermel, “Improving web application testing with user session data,” Proceedings of 25th International Conference on Software Engineering, 3-10, pp. 49-59, May 2003. D. A. Menasce, “Load testing of web sites,” IEEE Internet Computing, pp. 70-74, July-August 2002. F. Ricca and P. Tonella, “Analysis and testing of web applications,” Proceedings of the International Conference on Software Engineering, pp. 25–34, May 2001. W. Feng, F. Chang, W. Feng, and J. Walpole “Provisioning online games: A traffic analysis of a busy Counter-Strike server,” Proceedings of the second ACM SIGCOMM Workshop on Internet measurement workshop, pp. 151-156, November 2002. T. Fritsch, H. Ritter, and J. Schiller, “Network effects on games: The effect of latency and network limitations on MMORPGs : A field study of everquest2”, Proceedings of 4th ACM SIGCOMM Workshop on Network and System Support for Games NetGames '05, ACM Press, pp. 1-7, October 2005. WinPcap: The Windows Packet Capture Library. [Online]. Available at http://www.winpcap.org/
