Computer games played over a network are categorized in the three main ... exist several advanced networking topologies, which can support a high number.
A Proxy Server-Network for Real-time Computer Games? Jens M¨ uller1 , Stefan Fischer2 , Sergei Gorlatch1 , and Martin Mauve3 1
Westf¨ alische Wilhelms-Universit¨ at M¨ unster, Germany 2 Technische Universit¨ at Braunschweig, Germany 3 Heinrich-Heine-Universit¨ at, D¨ usseldorf, Germany
Abstract. Computer games played over the Internet have recently become an important class of distributed applications. In this paper we present a novel proxy server-network topology aiming at improved scalability of multiplayer games and low latency in client-server data transmission. We present a mechanism to efficiently synchronize the distributed state of a game based on the concept of eventual consistency. We analyse the benefits of our approach compared to commonly used client-server and peer-to-peer topologies, and present first experimental results.
1
Introduction
Computer games played over a network are categorized in the three main genres: First Person Shooter Games (FPS), Real-time Strategy Games (RTS) and Massive Multiplayer Online Roleplaying Games (MMORPG). For MMORPG, there exist several advanced networking topologies, which can support a high number of participating clients [1]. In contrast, the basic concepts of network communication for FPS and RTS games have not been altered for years. Commonly used architectures like client-server and peer-to-peer do not provide the required scalability for massive-multiplayer sessions. Although other communication architectures like peer-to-peer server-networks or hierarchical server-networks exist, their efficient usage for FPS and RTS games is still not well understood (an overview can be found in [2]). In this paper, we present our work on a Proxy Server-Network for Real-time Computer Games. Our main goal is to improve network scalability for FPS and RTS games in comparison to traditional topologies, thus allowing games with higher numbers of players. A further objective is to enhance responsiveness of massive-multiplayer, Internet based games by providing low communication latency between clients and servers. The paper is organized as follows: In Sect. 2 we introduce our architecture and propose a concept for an efficient synchronisation of the distributed game state. We describe our implementation of the architecture as a C++ API and present experimental results on its scalability in Sect. 3. Section 4 compares our approach with related work and concludes the paper. ?
c
Springer-Verlag, 2004. In proceedings of Euro-Par 2004 Parallel Processing, LNCS 3149. URL: http://www.springer.com/comp/lncs
2
Architecture
Commonly used communication architectures like client-server or peer-to-peer topologies provide a variety of positive aspects and can be implemented quite easily, which leads to their high acceptance for computer games. However, there are several disadvantages in both concepts, resulting in poor quality of service and inhibiting novel game designs with a higher number of participating users. Client-server and peer-to-peer topologies do not scale well when the number of users increases. Additionally, the server in a client-server setup forms a single point of failure for the whole session. While the peer-to-peer approach eliminates the problem of a single point of failure, it allows a hacked client to cheat, since submitted game updates are not filtered by a server instance and hidden information can be made visible to the player [3]. 2.1
The Proxy Server-Network
The proposed proxy server architecture, which is derived from a peer-to-peer server-network [2], is shown in Fig. 1. The initial idea was sketched in [4]. clients
clients
[...]
[...]
proxy server
proxy server
ISP 1 client proxyserver communication inter proxyserver communication
[...]
clients proxy server
ISP 2
[...]
proxy server clients
ISP 3
Fig. 1. Example session using the proxy server architecture
The key idea of our architecture is the use of several interconnected proxy servers for a single game session. Each proxy server has a full view of the global game state, allowing it to serve arbitrary clients. Each client communicates with a single proxy to send user actions and receive updates of the game state. Proxy servers process user actions and forward them to other proxies using multicast (IP or application level) for synchronizing the distributed game state. For the sake of low latency in Internet based sessions, proxy servers should be located at different Internet Service Providers (ISP), such that each client connects to a proxy at its ISP. In our current implementation, the setup of
servers is done manually, but future work includes a dynamic setup of proxies respecting current user demand. MMORPG commonly use an approach orthogonal to ours: Instead of replicating the game world, they partition it among servers, which forces clients to change the server when the user moves into another region. While this approach works well for the relatively slow game flow in MMORPG, it cannot be applied to fast paced games like FPS and RTS, in which recurrent pauses resulting from server changes are annoying for users. Since each proxy has full view of the game state, our proxy server-network avoids reconnects during a session. As we show in Sect. 3.1, the described proxy network scales better with an increasing number of clients than traditional topologies. 2.2
Managing the Distributed Game State
Since the game state is replicated at the proxy servers, our architecture requires mechanisms to provide adequate consistency of the replicated data. There is a classical trade off between scalability, responsiveness, and consistency in distributed simulations in general. Implementing strong consistency models like causal or even sequential consistency [5] would increase the amount of inter-proxy communication: Proxies would have to order changes of the game state using Lamport timestamps or physical clock synchronisation. This would delay the transmission of acknowledgements of user actions to clients, thereby reducing the responsiveness of the game. As reported in [6], a scalable distributed system with real-time performance is only possible if not all participating processes share the same view on common data at one time. Therefore, we propose to use the concept of eventual consistency [5] for the game state. To implement eventual consistency, only a single process is allowed to alter specific parts of replicated data. Changes have to be propagated to other processes, ensuring consistency of the replicated state as soon as the update notification arrives. (5) remote proxy updates local game state and checks interactions
(5) remote proxy updates local game state and checks interactions
(6) proxies submit results of interaction evaluation
[...]
(7) proxy informs affected clients
(4) proxy notifies other servers
(7) proxy informs affected clients
(2) proxy validates user input and updates local game state on success (1) client submits user action
[...]
(3) proxy sends acknowledgement
Fig. 2. Acknowledging and forwarding a single user action
The main steps in our concept of transmitting and processing a single user action in the proxy server-network are depicted in Fig. 2. User actions transferred from clients (1) are categorized in movements and interactions. In case of movement commands, the server has to check if the user input is legal to prevent cheating before changing the state (2), and to inform other proxies about updated position values (4), which in turn update the local copies accordingly (5). In this way eventual consistency for the changed part of the game state is guaranteed and the clients receive acknowledgments for movement commands in a relatively short time (3). In case of interactions, notified proxies additionally have to check whether local clients are affected. If so, the responsible proxy updates the game state of its local client, and informs other servers in turn (6). For all state updates received from other proxy servers, each proxy evaluates which local clients are affected and informs them (7). In our architecture, general parts of the game state that are not associated with a specific client are assigned to single proxy servers for maintenance, which ensures eventual consistency for these parts of the game state as well. Such data contain manipulable parts of the gameworld like collectible items or avatars not controlled by the users (Non-Player Characters). To avoid a single point of failure, applications have to implement a proper election method to reassign responsibility to another proxy in case of a server failure. For data representing states of clients, a reassignment of responsibility is not required: If a proxy process fails, clients connected to it will loose connection to the session anyway, while for all other processes the game will continue to be playable. The presented concept for managing the distributed state allows an efficient synchronisation of the game state in combination with fast acknowledgment of user actions. However, due to the use of eventual consistency, execution of commands can occur in a different order than their initation by the users. In the following, we discuss the impact of this fact on the playing conditions.
2.3
Discussion of Game Correctness
A computer game provides equal game opportunities for all users, if the order of processed state updates is the same as the order of their initiation by the users. Additionally, in continuous simulations like real-time games, the point in time at which actions are processed has to be considered, too. Let us consider an example interaction between two clients, c1 and c2 , in a real-time game: The user at c1 issue an action affecting c2 (e. g. shooting at the avatar managed by c2 with an instant-hit weapon), while the user at c2 issues a movement command some time later which would nevertheless prevent his avatar from being affected if processed before the action of the user at c1 . Figure 3 depicts four situations for the communication between processes in our example, comparing a client-server setup (a) to our architecture (b), in which additional methods for improved correctness of the processing order are applied (c), (d). Let l(c, p) denote communication latency between c and p.
c2
s
c1
(a)
p1
p2
c2
l(c1,p1)
l(c2 ,s)
l(c1 ,s)
c1
(b)
l(c2 ,p2)
l(p1,p2)
time (c)
c1
p1
l(c1,p1) l(p1,p2)
p2
c2
c1
(d) l(c2 ,p2)
p1
l(c1,p1)
p2 1/f
l(p1,p2)
c2 l(c2 ,p2)
Fig. 3. Comparison of processing orders: Traditional client-server topology (a), proxy server-network (b), proxy server-network with client latency levelling (c), proxy servernetwork with client latency levelling and action reordering (d)
(a) An incorrect processing order is possible in the traditional client-server topology: client c2 has an advantage in time over c1 of l(c1 , s) − l(c2 , s), due to which c2 can dodge the hit although c1 aimed correctly and shot first. (b) The proxy server-network has lower latency l(c1 , p1 ) < l(c1 , s), but requires additional time l(p1 , p2 ) before the shot action reaches proxy p2 . Therefore, the processing order at p2 is incorrect as well. (c) Our implementation uses client latency levelling: communication between a proxy and clients with low latency is delayed; latencies are thus adjusted to an equal level l(c1 , p1 ) = l(c2 , p2 ), which reduces the advantage in time for c2 to l(p1 , p2 ). However, actions in the depicted example are still processed in an incorrect order. An upper limit for client latency levelling is configurable, e. g. by a session administrator, to avoid an unresponsive game for all clients due to a single slow connection of a specific client. (d) In real-time games, updates of the state usually are done at fixed points in time, so called ticks, discretizing the actual continuous game simulation. With a tickrate f , proxy p2 can reorder actions if they are received in the same tick of length 1/f . This leads to the correct processing order.
This analysis shows that our architecture provides a degree of equal game opportunities comparable to the client-server setup, which is commonly used for FPS games. The provided correctness of the processing order is even better when actions are reordered by the game application. However, if proxies receive interactions at different ticks then reordering is not possible, since the first action received has already been processed. If the action processed first was a movement command of a local client, proxies can hold a backlog of old positions and apply remote actions with hindsight performing a timewarp [11]. In the relatively rare case that the acknowledgment sent for the first action is not reversible, we accept an incorrect processing order for the benefit of high responsiveness for clients.
3
Implementation and Analysis
The implementation of a game using our architecture consists of two parts: the functionality for the network communication, and the game specific data structures and algorithms for the simulation itself. We implemented the communication of the server-network as a C++ API using the TCP/IP protocol stack. Games using this API only have to implement the game specific part and to handle user commands according to the procedure depicted in Fig. 2. Our implementation includes two different APIs, designated to clients and servers. Each of them provides a virtual callback-class, which has to be implemented by the game application to receive incoming data. For transmission of data, each API offers a sender-class providing methods to initiate the connection and send data to the network. The implementation provides functionality to send and receive byte arrays of arbitrary length. It offers different stages of reliability in data transfer, automatically fragmenting and acknowledging packets when using UDP and IP multicast for inter-proxy communication. 3.1
Scalability: Analysis and Experimental Results
In the computer games context, scalability is the ability of an architecture to maintain service when the number of participating players increases. The two kinds of resources utilized are computing cycles of the host machines and network bandwidth. With the common unavailability of IP Multicast for home computer Internet dial-up connections, utilized bandwidth of the peer-to-peer topology is quadratic to the number of participating clients, which disqualifies this architecture for large scale Internet-based sessions. Therefore we focus on a comparison of the proxy-network to a traditional client-server topology in our analysis. The computation intensive tasks in simulating the game are the validation of user inputs and the filtering of client updates (steps (2) and (7) in Fig. 2). The proxy server-network scales better than the client-server topology, because these tasks are distributed among all participating proxies. Utilized bandwidth in client-server communication of the proxy architecture and client-server topology is equal: each client only receives state updates if it is affected. However, the proxy server-network utilizes additional bandwidth in inter-server communication, which will be taken into account in the analysis. We implemented the following test game to measure the gain in scalability. Each client has control over a single avatar, other avatars are visible if they reside in the avatar’s viewing radius. The environment, internally represented by a quadtree, is defined as a bordered plane, on which rectangular solid blocks define regions avatars cannot walk into. Avatars can catch each other if in range. After a successful catch, the winner gains a point and the caught avatar is taken out of the game for some time. Although this game is quite simple, its basic concept is comparable to sophisticated FPS games. We measured the maximum number of supported clients by adding clients to the session until the servers become congested, i. e., when calculation of a tick takes longer than the time available. The tickrate of the evaluation sessions
was 50 updates per second, which is comparable to tickrates in Internet based FPS and RTS games commonly ranging from 10 to 40. Each proxy server was run exclusively on a Linux host with 1,7 GHz Intel Pentium 4. Figure 4 depicts our experimental results. The left plot shows the maximum number of supported clients with different numbers of proxies. The right plot depicts the average total bandwidth at a single proxy server in different session setups.
250
supported clients
200 150 100 50 0
number of proxy servers 0
1
2
3
4
5
bandwidth at single server in KByte/s
1200
single server 2 proxies 3 proxies 4 proxies 5 proxies
1000 800 600 400 200 0
number of clients 0
50
100
150
200
250
Fig. 4. Experimental results. Left: maximum supported clients with different numbers of proxy servers; Right: utilized bandwidth at a single proxy
The results for computational scalability (left plot) show that our proxy architecture supports a much higher number of clients than a traditional clientserver topology whose capacity is shown by ◦. The absolute gain in the number of supported clients decreases slightly for each additionally attached proxy, due to increased calculation for processing remote clients’ actions. Therefore we expect a game dependent, maximum number of clients for which an acceptable responsiveness is provided by our architecture. The total utilized bandwidth at a single server (right plot) consists of the bandwidth of the client and proxy communication, each with incoming and outgoing bandwidth. While the bandwidth of inter-proxy communication grows only linearly with the the number of participating clients, the outgoing client bandwidth increases quadratically. In order to compensate the growing utilized bandwidth for increasing numbers of participating clients, additional proxy servers have to be added. Figure 4 (right) shows that in our test game, for approximately 25 additional clients joining the game one additional proxy server has to be added to the session in order to avoid an overload of servers due to high bandwidth. The deviation of total average bandwidth was below 3% throughout multiple measurements for the same setup. The available number of host computers and network hardware capabilities of our testbed subnet limited us to a maximum of 240 clients with an overall bandwidth of about 6 MByte/s. With the bandwidth distributed to several subnets as in Internet based sessions, more proxies and therefore more clients can participate in a game.
4
Conclusion and Related Work
For multiuser simulations in general and real-time computer games in particular, there is a trade off between responsiveness, scalability, and consistency. Our proxy server-network weakens consistency for the benefit of higher responsiveness and improved scalability. At the same time, we do not sacrifice correctness of processing, which is still comparable to the commonly used client-server setup. There has been active research in the domain of Distributed and Collaborative Virtual Environments (DVEs resp. CVEs). Existing DVE systems like MASSIVE [7] or DIVE [6], do not aim at maximized responsiveness for the benefit of stronger consistency. However, to achieve high responsiveness in games with direct, hand-eye coordinated movement like FPS, latency has to be below 150 ms [8]. Our architecture does not introduce additional delay for acknowledgment of actions, which only depend on the round trip times to the ISP. OpenSkies, developed by Cybernet, is a commercial server-network developed for massive multiplayer online games [9]. In contrast to our proxy-network, servers in OpenSkies only filter communication to interested hosts and do not maintain the application state at all. In comparison to the spatial partitioning commonly used for MMORPG [1], our architecture does not require reconnects of clients in a running session. RING [10] proposes a network topology similar to ours, but updates are already filtered in inter-server communication. RING is specially designed for densely occluded virtual environments, while we aim at supporting arbitrary environments in computer games.
References 1. Cai et al.: A scalable architecture for supporting interactive games on the Internet. In: Proc. of the 16th Workshop on Par. and Dist. Simulation, IEEE (2002) 60–67 2. Smed, J., Kaukoranta, T., Hakonen, H.: Aspects of networking in multiplayer computer games. In: Proc. of Int. Conf. on Application and Development of Computer Games in the 21st Century, Hong Kong SAR, China (2001) 74–81 3. Pritchard, M.: How to hurt the hackers: The scoop on Internet cheating and how you can combat it. Game Developer (2000) 4. Fischer, S., Mauve, M., Widmer, J.: A generic proxy system for networked computer games. In: NetGames2002 Proceedings. (2002) 25–28 5. Tanenbaum, A., van Steen, M.: Distributed Systems. Prentice Hall (2002) 6. Frecon, E., Stenius, M.: DIVE: A scalable network architecture for distributed virtual environments. Distributed Systems Engineering Journal (1998) 7. Greenhalgh, C., Purbrick, J., D.Snowdon: Inside massive-3: Flexible support for data consistency and world structuring. In: Proceedings of the Third ACM Conference on Collaborative Virtual Environments (CVE 2000). (2000) 119–127 8. Armitage, G.: Sensitivity of Quake3 players To network latency, IMW2001 poster 9. Cybernet Systems: Openskies network architecture http://www.openskies.net/ 10. Funkhouser, T.A.: RING: A client-server system for multi-user virtual environments. In: Symposium on Interactive 3D Graphics. (1995) 85–92 11. Mauve et al.: Local-lag and Timewarp: providing consistency for replicated continuous applications. IEEE Transactions on Multimedia, February 2004.
