Adaptive Server Selection for Large Scale Interactive Online Games Kang-Won Lee ∗ IBM T.J. Watson Research Hawthorne, NY 10532, U.S.A.
Bong-Jun Ko Columbia University New York, NY 10027, U.S.A.
Seraphin Calo IBM T.J. Watson Research Hawthorne, NY 10532, U.S.A.
Abstract Large scale interactive online games aim to support a very large number of game players simultaneously. To support hundreds of thousands of concurrent players, game providers have so far focused on developing highly scalable game server architectures and extensible network infrastructures. Recently, distributed online games are beginning to incorporate more interactive features and action sequences; thus, it becomes increasingly important to provision server resources in an efficient manner to support real-time interaction between the users. In this paper, we present a novel distributed algorithm to select game servers for a group of clients participating in a large scale interactive online game session. The goal of server selection is to minimize server resource usage while satisfying a real-time delay constraint. We develop a synchronization delay model for interactive games and formulate the server selection problem, and prove that the considered problem is NP-hard. The proposed algorithm, called zoom-in-zoom-out, is adaptive to session dynamics (e.g., clients join) and lets the clients select appropriate servers in a distributed manner such that the server resource is efficiently utilized. Using simulation, we study the performance of the proposed algorithm and show that it is simple, yet effective in achieving its design goal. In particular, we show that the performance of our algorithm is comparable to, or sometimes even better than, that of centralized greedy algorithms, which require global information and extensive computations. Key words: Distributed systems, MMOG, On-line game infrastructure, Resource allocation, Distributed Algorithm, Synchronization delay model
Preprint submitted to Elsevier Science
28 November 2005
1 Introduction
Large scale interactive online games, such as Massively Multi-player Online Games (MMOG), aim to support a very large number of game players simultaneously. In practice, MMOG providers often are required to support hundreds of thousands of geographically distributed users at the same time. For example, it has been reported that about 165,000 concurrent users play Lineage [1] in Taiwan [2]. To support such a large number of players, MMOG providers have so far focused on developing highly scalable game server architectures and supporting network infrastructures that span wide geographical regions while satisfying loose real time requirements [3,4]. Recently, however, large scale online games are beginning to incorporate more interactive features and action sequences (e.g., first person player-to-player combat) to provide a realistic game experience [5]. Thus it becomes increasingly important to provision enough server resources to support real-time interaction between users. There are several challenges in augmenting MMOGs with such interactive features. First, unlike in online first-person-shooting (FPS)-type games where a small number of users that are geographically close to each other get assigned to the same game session, MMOG games must maintain a persistent virtual world view for a large number of game players that are distributed over the network. Second, the maintenance of a long-lived persistent world mandates a server-based game architecture, where clients interact with a central server that keeps track of the game states. However, it is well known that the conventional server-client architecture does not scale well as the number of clients increases [6]. To overcome this limitation, a mirrored server architecture has been proposed [7–9], where a set of distributed game servers are allocated and orchestrated to support a large number of distributed clients. In this architecture, the game servers are typically interconnected via well provisioned network links, and each game client is directed to connect to a nearby game server. The game servers simulate the game state according to the actions of the players and maintain a consistent game state among themselves so that all the clients can have a consistent view of the game world. In parallel with this architectural development, proposals have been made to dynamically provision game servers on the fly exploiting emerging Grid technologies [10,11,3]. According to these on-demand game architectures, game servers can be dynamically provisioned to accommodate the capacity requirements as the ∗ Corresponding author. Email:
[email protected]; Tel.:+1-914-784-7228; Fax: +1914-784-6205.
2
user demand changes. In addition to efficient resource utilization, such on-demand game architecture removes the burden of server and network management from the game service provider. Also by multiplexing and sharing a large scale infrastructure across multiple games, the service provider can try out new titles without having to make a large initial investment in infrastructure [10]. Thus it now becomes an important issue to dynamically provision and utilize shared server resources in an efficient manner. In this paper, we present a novel distributed algorithm that selects game servers for a group of clients participating in large scale interactive online games. The goal of the server selection algorithm is to select the minimum number of servers from all available servers, while satisfying the real-time delay constraint of the game. To solve this problem, we first develop a synchronization delay model for interactive distributed games, formulate the server selection problem, and prove that finding an optimal solution of this problem is NP-hard. We then propose a novel heuristic algorithm for the considered server selection problem that can be easily implemented in distributed environments. The proposed algorithm, called zoom-in-zoom-out, is adaptive to session dynamics (e.g., clients join) and lets the clients select appropriate servers in a distributed manner such that the server resource is efficiently utilized. Through simulation-based performance evaluation, we study the performance of the proposed algorithm with various zoom-in-zoom-out techniques and show that it is simple yet, effective in achieving its design goal. In particular, we show that the performance of our algorithm is comparable to, or sometimes even better than, that of centralized greedy algorithms, which require global information and excessive computation. The remainder of this paper is organized as follows. Section 2 presents synchronization delay model and problem formulation. Section 3 presents the proposed server selection algorithm. Section 4 presents the performance of the proposed algorithm varying parameters and in comparison with more expensive greedy algorithms. Section 5 presents an overview of related work, and finally Section 6 concludes the paper.
2 Problem Formulation
2.1 System model
We assume a mirrored-server architecture, where multiple game servers are interconnected via well-provisioned links. In our model, a server refers to an entity that collectively maintains a persistent virtual world, calculates the game state based on the events generated by the players. A client refers to an entity that renders and presents the game states to the player. We call the network interconnection between 3
mirrored servers the server network. The server network is assumed to be private and dedicated to the game service provider. We further assume that the virtual game world is divided into multiple regions, wherein relatively few game players (compared to the number of users in the entire game world) directly interact with each other. In general, a region is defined by the geography or architectural structure in the virtual world, and the game players in the same region may engage in combat or first person shooting actions. We call the persistent state of a region a session. A single session may span multiple game servers to accommodate geographically distant clients. Each client is assigned to one of the servers, called the contact server of the client, which is responsible for forwarding the client’s action events to all the other servers participating in the same session. Upon receiving clients’ action, each game server independently calculates a new game state, and sends the updated state to the directly connected clients. We note that this client and server relation effectively defines a one-to-one mapping from the clients to the servers. We call this mapping the server allocation or just allocation. When a player moves from one region to another region, the client may need to hand-off from the current contact server to another server, which hosts the session for the new region. Discovering and subscribing to a new server can be facilitated by techniques presented in [12–14]. For accurate game simulation and presentation, game events generated by the players must be time-stamped according to a global clock or must be serialized in a total order. In practice, however, having a global clock or determining a total order of game player’s actions is infeasible. Instead, most practical game systems adopt a synchronization mechanism, which divides time into discrete slots and let each server wait for events generated by remote clients to arrive before executing game simulation [15,16,7,17]. Depending on the scheme, inconsistencies due to late events are resolved by rolling back the game state, and lost events are deadreckoned based on the action history. In this paper, we do not assume a particular synchronization mechanism. Instead we just assume that the synchronization delay must be greater than the latency for the events generated by clients to reach the farthest server, and for the results of game simulation to reach all the local clients. This condition must be met because we assume that each server simulates the game state independently and for that it has to receive all the events from the farthest client before performing the simulation. Based on this condition, we define the synchronization delay as follows: the synchronization delay between a client and a server is the time difference between the instance that the client sends its players’ actions and the instance that the client renders a new game state in response to the actions sent to the server. This synchronization delay depends on the network latency from the client to the server (upstream latency), processing time at the server, and the network latency from the server to the client (downstream latency). To synchronize game play and interaction amongst all players participating in the same session, we must take into account synchronization delays between all clients 4
Fig. 1. Example of game server architecture with three sessions
in the session and the corresponding servers. More specifically, a game server must calculate a new game state after receiving the action events from the farthest client. Otherwise, the action from the farthest client will not be synchronized with others. Similarly, at the client side, a new game state should not be presented to the players until the same game state is delivered to the farthest client from the server. Otherwise, the game simulation becomes unfair to the farthest clients [18].
2.2 Delay model and problem statement Let Cg denote a set of game clients that participate in a game session g and let S denote a set of available game servers. Servers in S form an undirected connected graph, called the server network. Let ds (j, k) denote the shortest distance (or latency) between servers sj and sk , and let dc (i, j) denote the distance between a client ci and a server sj . Suppose client ci is mapped to server sj in some allocation Ag . Then we say server sj serves client ci under Ag . We define Cg,j ⊂ Cg as the set of clients that are directly connected to the server sj , i.e., Cg,j = {ci ∈ Cg |sj serves ci }. We also define the session server set, S(Ag ) under the allocation Ag as the set of servers that serve at least one client in C, i.e., S(Ag ) = {sj ∈ S|Cg,j 6= φ}. Let Du (ci , sk ) be the upstream distance between a client ci and a server sk . We note that Du (ci , sk ) is defined not only for a client and a server that are directly connected, but also for a client and a server connected indirectly via some other servers forwarding the client ci ’s actions to server sk through the shortest path in the server graph, i.e., Du (ci , sk ) = dc (ci , sj ) + ds (sj , sk ), where sj is the contact server for ci . On the other hand, the downstream distance from sj to a client ci ∈ Cg,j is just dc (sj , ci ) as ci receives game states from its contact server sj . Figure 1 presents an example. There are three game sessions: (i) session g1 consisting of clients c1 , c2 , c3 , c7 , and c8 , represented in gray; (ii) session g2 consisting of clients c4 and c5 , represented by double lines; and (iii) session g3 consisting of clients c6 and c9 , represented in white. The server allocations for each session are S(Ag1 ) = {S1 , S2 , S5 }, S(Ag2 ) = {S2 , S4 }, and S(Ag3 ) = {S3 , S4 }. To illustrate 5
the latency calculation, take the events generated by client c5 in session g2 . In this case, the upstream latency from c5 to s2 is Du (c5 , s2 ) = dc (c5 , s4 ) + ds (s4 , s2 ). The downstream latency from s4 to c5 is just dc (s4 , c5 ). We now consider the overall synchronization delay for a session. As previously discussed, for a server to simulate game state in a fair manner it must wait until all the action data from all the clients in the session to arrive. In other words, a server sj must wait for maxci ∈Cg,j Du (ci , sj ) before game simulation. Then it processes the action data and sends out updates to all the clients it serves. This game state update takes up to maxci ∈Cg,j dc (sj , cj ) for the farthest client. Since this condition must hold for all servers, the overall session synchronization delay Ds (A) under allocation A can be succinctly written as follows:
Ds (A) = max {max Du (ci , sj ) + max dc (ci , sj )}. sj ∈S(A) ci ∈Cg
ci ∈Cg,j
Based on this session synchronization model, we now formulate our server selection problem. Our goal in this paper is to find an algorithm that can minimize the number of servers allocated to a session, while satisfying a given synchronization delay requirement. 1 Restated formally, the problem is: given a network topology consisting of a set of servers S, a set of clients C, a set of links between the servers and the clients that are annotated with latency between the nodes, and a real-time delay requirement ∆ for a game session, find a server allocation Amin that minimizes |S(A)| subject to Ds (A) ≤ ∆ and |S(A)| ≥ 1. This problem is NP-hard as an efficient solution to this problem can effectively solve other NP-hard problems.
Theorem 1 The minimum game server allocation problem is NP-hard. Proof: Consider, in our model, the case when the latency between every pair of servers is zero. Further assume that the latency between any client and server pair is ∆/2 if there is a link, where ∆ is the sync-delay bound in our model. Then the optimization goal in this specific case would be to minimize the number of servers subject to the condition that every client is within distance ∆/2 from its contact server, to which it is allocated. This is exactly a set-covering problem, in which a set is defined by the set of clients that are within the distance ∆/2 from each server. Since our problem generalizes the set-covering problem, which is a well-known NP-hard problem, our problem is also NP-hard. 2 1
In this paper, we only consider the server selection problem for a single game session. The problem of optimally allocating servers for multiple dynamic game sessions is a very challenging problem of its own and we leave it as a future research topic.
6
(a) The topology
(b) Allocation A1
(c) Allocation A2
(d) Allocation A3
Fig. 2. Examples of game server allocation
3 Server Selection Algorithm
3.1 Impact of Server Allocation
In this section, we first analyze the impact of server allocation using an example. Figure 2 illustrates a simple example with 3 servers, s1 , s2 , and s3 , and 2 clients, c1 and c2 . Figure 2(a) shows a network configuration, where the solid lines represent the latency between the servers in the server network, and the dashed lines represent the latency between each server-client pair. Figures 2(b) – 2(d) show three different server allocations, A1 , A2 , and A3 , where allocated session servers are marked in gray. According to the session delay model, the overall delay for allocation A1 is 10 with both maximum upstream and downstream delays being 5 (between c2 and s1 ). Allocating s2 instead of s1 as the session server (allocation A2 ) reduces the synchronization delay to 6 (upstream/downstream 3 each). Note that, though the end-to-end distances between the clients are the same in A1 and A2 , the synchronization delay is reduced by placing the server at the center of the network. Finally, if we allocate two servers at the edges of the server network as in allocation A3 , the synchronization delay decreases to 4 with the upstream latency 3 (e.g., along the path c1 -s1 -s2 -s3 ) and the downstream latency 7
reduced to 1. In this case, the reduction comes from placing the servers near the clients at the expense of using two servers. From this example, we can draw the following intuition to design our server selection algorithm: • If we had to choose only one server to minimize the overall synchronization delay, then it would be optimal to select a server that minimizes the maximum distance (not the average) to all clients. In graph theory, such a node is called the center of a network. 2 In this paper, we call it a core server. • If it is faster to forward packets via the contact server over the server network than to send packets to a remote server directly over the public Internet, allocating servers near the “edge” of the server network (and close to the clients) reduces the overall synchronization delay. However, this comes at the expense of increasing the number of servers in the session. Based on these observations, we design a distributed server selection algorithm as follows.
3.2 Server Selection Algorithm
We design our server selection algorithm based on our observation presented in the previous section: the synchronization delay of a session is small when clients are connected to the “edge” servers; and it increases as the clients are connected to servers near the core server. Since our goal is to reduce the number of session servers, at a high level, our algorithm attempts to reduce the “diameter” of the network of session servers while keeping the synchronization delay below the bound, ∆. To this end, each client visits a sequence of servers that are successively closer to the core server. This procedure has an effect of increasing the chance for more clients to share common server, and thereby decreasing the number of total session servers, at the expense of increased synchronization delay. Given a set of clients C, a set of servers S, link configuration, and delay requirement ∆, our algorithm performs the following procedure. • STEP 1: Initially allocate each client ci ∈ C to the closest server in S. 3 Denote this initial contact server of ci by s0 (ci ). This produces an initial allocation A. We assume that the session has been provisioned to satisfy the delay requirement 2
Formally, the eccentricity of the vertex v in a graph G is the maximum distance from v to any other vertex. The center of a graph G is the set of vertices of eccentricity equal to the radius, where the radius of G is the minimum eccentricity among the vertices of G. 3 This can be done by roundtrip-time measurement based on game server directory in the region.
8
∆, i.e., Ds (A) ≤ ∆. Otherwise, the violation is marked, a high level session regrouping module is called, and then exit. • STEP 2: Find the core server, s∗ ∈ S, of the session that minimizes the maximum distance to the clients. 4 Finding the core of a network in a distributed system has many interesting applications (e.g., core-based multicast tree construction), and thus has been studied in various contexts. In this paper we employ a tournament-based method studied in [19]. • STEP 3 (Zoom-In): At each client ci , do the following. Migrate each client, ci , from the current contact server, sk (ci ), to another server, sk+1 (ci ), that is further from ci and is closer to s∗ than sk (ci ). Using this as a new contact server sk+1 (ci ), calculate the session synchronization delay. If the delay is within the bound ∆, then mark the server sk+1 (ci ) to be the “new contact server”, and notify all clients in the session of the successful migration. When notified of a successful migration of some other client, mark the current server to be the “new contact server”. Repeat this step for all clients until all clients arrive at the core server s∗ , at which point each client migrates to the “new contact server”. • STEP 4 (Zoom-Out): At each client ci , do the following. Let S 0 be the current set of contact servers for all clients. Then among the servers in S 0 that are further from the core server s∗ than the current contact server, find the closest server to ci . Then probe the session synchronization delay to see whether the session synchronization delay would still be within the bound ∆ if the client migrated to this new server. If yes, ci migrates to the new server. Otherwise, take no action. Repeat this step for all clients until no client can migrate to a server in S 0 that is further from s∗ than the current server s(ci ). To ensure correctness, we assume the existence of a mechanism to ensure that clients do not perform STEPs 3 and 4 concurrently. In practice, such a serialization can be obtained by using a token-passing scheme amongst clients. The procedure in STEP 3 has the effect of moving a cluster of the session servers toward the core server; hence we call it zoom-in. With this procedure, the number of session servers tends to decrease, but the overall synchronization delay tends to increase because the latency between clients and their contact server increases. This zoom-in process terminates when all clients have reached the core server s∗ . At this point, each client ci connects to the last server in the migration procedure that makes the overall synchronization delay below the bound. Note that, in the zoom-in procedure, we attempt to move all clients up to the core server even after the synchronization delay exceeds the delay bound. This is because the synchronization delay does not always monotonically increase, but sometimes decreases as we move clients from the edge to the core. Such fluctuations in the synchronization delay are frequent particularly when the location of a new 4
The core is not necessarily the same server for multiple game session since the core is determined by the distribution of the clients in a particular game session.
9
c0
c0 c2
c2
S*
S*
c1
c1
c3
c3
c4
c4
(a) Initial allocation
(b) Zoom-In
c0
c0 c2
c2
S*
S*
c1
c3
c1
c3
c4 (c) Zoom-Out
c4
(d) Final allocation Fig. 3. ZIZO algorithm
contact server deviates from the shortest path from the client to the core server. Therefore, if we terminate the zoom-in process when we first encounter increase in the synchronization delay, then the process may trap in a local minimum point. By iterating the zoom-in step all the way up to the core server, the client can explore the entire search space and avoid stopping at local minima. By performing the procedure in STEP 4, we seek to further reduce the number of session servers by “freeing up” some servers among those obtained in STEP 3 that do not contribute to decreasing the synchronization delay. This is possible because after the zoom-in procedure, servers near the core server are most likely to have been selected, but they may be removed without increasing the synchronization delay. This process is called zoom-out since it tends to remove servers from the core and proceed to the outside, i.e., to the boundary of the session server cluster. Note that, in zooming-out process, since the clients attempt to migrate to another server among the set of servers that are currently selected, STEP 4 can only reduce the number of session servers. If all clients could connect to the core server in STEP 3, this is an optimal selection and we do not need to perform STEP 4. 10
Note that, in practice, the information shared by the session clients can be exchange through the “server network” rather than maintaining a separate channel for signaling between the clients. For instance, in STEP 3 of the algorithm, the notification for successful migration of a client can be sent to all the servers in the corresponding session. More specifically, when a client successfully migrates to a new server, that server notifies of this migration to all the other session servers in S(Ag ). Then, each session server, in turn, informs its clients that they can mark the current server as the new contact server. This approach is more scalable than having each client individually contact all the session clients since the number of session servers is typically small compared to that of clients, and we can minimize the session state information maintained by each client. Figure 3 illustrates the above procedure through an example. In Figure 3(a), each of five clients, c0 , · · · , c4 is initially allocated to the closest server. In this allocation, we have five servers participating in a game session by the clients. Now, in Figure 3(b), each client incrementally probes servers that are closer to the core server, for example, client c2 migrates twice toward the core server as long as the delay bound is satisfied. As a result of the zoom-in procedure, three servers near the core server are selected. Now, clients further seek to reduce the number of session servers by moving out from the core server. For example, in Figure 3(c), two clients, c1 and c2 , were able to migrate to the session servers closer than the core server, with s∗ no longer being allocated to any client. The final server allocation is shown in Figure 3(d), with only two servers being selected by the procedure. In what follows, we present several issues in implementing this algorithm in a scalable and efficient manner.
3.3 Implementing the Selection Algorithm
3.3.1 Minimizing migration overhead The first issue concerns game session migration. Although a game architecture based on on-demand technologies enables our dynamic server selection, migrating a game session from one server to another is a relatively expensive procedure. As a result, probing a new server and trying to migrate the game session in each zoom-in and zoom-out step by a client is not a good idea. We address this issue by handling the zoom-in and zoom-out steps in the control and management plane, not in the actual game session plane. To support this operation, a server that has been probed by a client runs a simulation on synchronization delay computation and maintains a virtual state. The clients keep probing the next possible server following the server selection algorithm until they reach a steady state. Only when the entire zoom-in and zoom-out process terminates, do the clients coordinate and move the game session to the newly allocated servers. In this way, we can minimize the management 11
and session migration cost. In addition, we note that the search process need not occur every time when there is a change in the session. A practical implementation may control the frequency of server search procedures by a specifying a threshold or a damping factor. For example, the propose server selection algorithm may be triggered only when certain number of servers have been added to the session.
3.3.2 Scalable server search method The performance of the “zoom-in” and “zoom-out” procedures in our proposed algorithm relies on the effectiveness of the server search algorithm. We consider three types of search methods: (i) full search, (ii) partial search, and (iii) tree-based search. In the full search method, each client searches for the next server to migrate among all servers in S. This may be costly if S is large, but will provide the most accurate result. In the partial search method, when we discover a core server s∗ , each server discovers up to k closest servers that are closer to s∗ than itself, and up to k closest servers that are further from s∗ , where k < |S|. When a client searches for the next server, it considers only the servers in these sets as the candidates. In order to provide an estimate of the distance from its clients to the other servers, each server evaluates the distances to the other servers based on the distance in the public network domain, on which the clients’ data travel to the servers. The tree-based method is similar in spirit to the partial search method in that each client utilizes the set of servers discovered by the current server in searching for the next server. The difference is that each server, si , selects only one server, p(si ), which is the closest to itself among those closer to the core server s∗ . Then si registers p(si ) as its parent server, and informs p(si ) that it has done so. This mechanism effectively forms a core-based tree, with s∗ being the root of the tree, along which each client selects the next server when it migrates from one server to another. More specifically, when zooming in, each client migrates to the parent server of the current server; hence there is no need to “search” for the next server. On the other hand, when clients zoom-out, they search for the next server that is further from the core server among the children nodes of the current server in the server tree. The partial search and the tree-based method enable the clients to narrow the set of the candidate servers within which the clients search for the next server at each step. They are an approximation based on a hypothesis that when a client selects the next server, it will be most likely on the shortest path from the client to the core server. The above three different search methods represent different levels of trade-off between the search space and the cost: the full search method will examine all possible 12
alternatives at high complexity, whereas the tree-based method will consider only those servers that are on the shortest path, but does not incur search cost. We expect that the partial search method will perform similar to the full search method as we increase k.
3.3.3 Coordinating client migration Next, we consider two different types of client migration strategies for the zoomin and zoom-out process. In a naive implementation, each client may individually probe candidate servers and may make a decision on migration. In this case, it is possible that, while some clients connected to a server could change their contact server, others may fail to migrate because it would violate the overall delay constraints. Therefore this uncoordinated migration of clients may result in a suboptimal result by temporarily increasing the number of servers allocated to the session. An alternative approach we consider in this paper is to coordinate the migration of all clients on the same server simultaneously, and move them only when all the clients can migrate to new servers.
3.3.4 Server capacity The delay model presented in Section 2 and our base-line algorithm in Section 3.2 do not take into account the processing time at the server. However, if a server needs to handle a lot of game clients, its processing time can be a bottleneck in overall latency in practice. While we mostly leave this issue as a future research topic (e.g., incorporating the impact of the number of clients in a server into the delay model), we take a simple heuristic approach to handle this case in this paper. When the number of clients at a server concerns, we impose an upper bound on the number of clients that can be connected to a server, up to which we assume the processing time remains constant. Then we slightly modify our algorithm to deal with the bounded capacity of the servers as follows. When a client performs zoom-in and zoom-out process, it first searches the best server to migrate to; however, if the server has already reached its bound on the number of clients, then it attempts to migrate to the next best server, and so on, until it finds a server that can accommodate more clients. More specifically, in STEP 3 of the algorithm, if a client ci is currently connected to a server s(ci ), it first attempt to migrate to the closest server among the servers that are closer to the core server and further from ci . If this server is full, ci attempts to migrate to the next closest server, and so on. In zoom-out process, the client searches for a server to migrate to in a similar manner, but only among the set of current contact servers. We evaluate this “capacitated” version of the algorithm in Section 4. 13
3.3.5 Overhead of the algorithm Finally, we study the computational overhead of the proposed algorithm. Overall we observe that the computation overhead of the algorithm incurred at each client is moderate. The step 1 requires a communication overhead of O(|S|) for each client for exact computation. In most practical situations, however, it will be O(l) where l ¿ |S| is the number of servers in the client’s region. The step 2 requires a total of log2 |S| tournaments for the entire session. The step 3 requires either O(1) or O(k) depending on the type of the zoom-in algorithm (see Section 3.3). The step 4 requires O(m) where m ¿ |S| is the number of session servers. In the next section, we present the effectiveness of the design alternatives considered above using a simulation-based study.
4 Performance Evaluation
In this section, we evaluate the performance of the proposed algorithm through simulations. For simulation, we use a synthetic, two-level, hierarchical topology generated the BRITE topology generator [20]. Specifically, we generate 50 AS domains derived by Barab´asi-Albert model [21], and for each AS domain, we then generate 100 nodes connected by Waxman model. Out of these 5,000 nodes generated (50 AS × 100 nodes per AS), we randomly select 100 servers and 50 clients to participate in the same game session for each simulation scenario. We then measure the performance of the server selection schemes by averaging the results obtained from 50 simulation runs using 50 different sets of servers and clients selected randomly. We emphasize that we select a relatively small number of clients (with respect to that of the servers) in the simulation since our goal is to evaluate the performance of the algorithm for a “single” session. In the aggregate, however, there will be many such sessions and the total number of clients will be much larger than that of servers. To emulate a well-provisioned server network with little congestion, the inter-server latency is set to be smaller than the client-to-server latency. In particular, we study the cases when the latency between two servers is reduced to 10%, 30%, and 50% of the latency in the underlying topology produced by the topology generator. We denote this latency reduction factor by parameter f (= 0.1, 0.3, 0.5). We also use some abbreviated notations to represent different types of client migration approaches during the Zoom-In and Zoom-Out process as explained in Section 3.3. The first set of indices represents the way clients select the next server to migrate: • Tree: Tree-based search (clients migrate along the shortest-path tree to the core 14
server.) • Part: Partial search (clients migrate by searching in the candidate server set.) • Full: Full search (clients migrate by searching for the next server among “all” servers.) Particularly in the partial search cases, we denote by k the maximum number of candidate servers given by the current server. The second set of indices represents two different ways of moving clients from one server to another: • Un: Uncoordinated migration (each client migrates to a new server individually.) • Co: Coordinated migration (all clients connected to a server migrate simultaneously.) We have also evaluated the effect of the migration order of the clients by comparing several approaches, e.g., round-robin, random order, and furthest-client-first (from the core). Overall, we find that the order in which clients migrate does not affect the end result of the simulation. In this section, we present the case of the round-robin migration. For evaluation, we compare the performance of the proposed algorithm with two centralized server selection algorithms, namely, greedy server selection and client clustering algorithms. In what follows, we briefly describe these centralized algorithms before showing the performance results of the proposed scheme.
4.1 Reference centralized algorithms
The greedy selection algorithm attempts to add a new server to a session server set by exhaustively searching for a server that, when added, results in the smallest synchronization delay. More specifically, at each step of the greedy selection, we calculate the synchronization delay when a non-session server has been temporarily added to the session server set and each client is connected to the closest session server. After evaluating all non-session servers, we select a server that produces the lowest synchronization delay when added. We keep adding servers to the session server set in this manner until the synchronization delay becomes smaller than the bound. This algorithm is similar to the “greedy algorithm” presented in [22] in the context of the Web server replica placement problem. 5 In [22], the greedy algorithm has been shown to perform closely to the optimal solution. The second reference algorithm, called client clustering algorithm, also attempts 5
The goal of server replica placement problem in [22] is to choose M replicas among N potential locations to minimize the client’s access latency.
15
to select servers in a greedy manner. But, this algorithm is different from the first algorithm in that, while the greedy algorithm keeps adding a server that would minimize the synchronization delay, the clustering algorithm adds the server that would maximize the number of clients that will be served by it. Conceptually, let us assume that each server is assigned a disk of radius x. We say the server “covers” the clients that are within the radius from the server if the server-client distance is smaller than x, i.e., dc (ci , sj ) ≤ x. Using this model, we select a server that covers the most number of clients that are not yet covered by any server in the session server set. We keep adding servers until all clients are covered. Now we repeat this process for all possible disk sizes for each server, and then select the smallest size session server set for which the synchronization delay is within bound. 6 For the set-covering problem, this selection strategy is known to have a O(1 + log N ) performance bound [23]. Note that these centralized algorithms are not only impractical to be applied in a distributed networking environment, but also are computationally expensive: the greedy selection must evaluate the new synchronization delay for each of the servers that have not yet been added to the session server set, and the clustering algorithm must select the session server set for all possible values of server disk size.
4.2 Number of servers selected
Figure 4 compares the number of servers allocated by the centralized algorithms and the proposed algorithms with full server search when the inter-server latency is relatively small (f = 0.1 in Figure 4(a)), moderate (f = 0.3 in Figure 4(b)), and large (f = 0.5 in Figure 4(c)). The y-axis represents the number of servers selected by each algorithm and the x-axis represents the delay bound ∆. In general, we observe that, when the full server search method is employed, our proposed algorithm performs as well as, or sometimes even better than the centralized algorithms (when f = 0.5 and ∆ < 30). As the delay bound increases, all algorithms find that fewer servers can meet the delay requirement. In particular, when the delay bound is sufficiently large, we note that all the algorithms can find the optimal server allocation, which is to allocate only one server. When the inter-server latency is relatively large, more servers are needed to satisfy the synchronization delay bound, particularly when the delay bound is tight. This is because the inter-server latency becomes the bottleneck and therefore it is better for the clients to connect to a nearby server when the delay bound is small. On the other hand, if the server-to-server delay is relatively small, it leaves some “slack” in the client-to-server latency, and it allows the clients to connect to a relatively farther server. As a result, the number of servers required to meet the delay bound is 6
If the current disk radius is x, we increase the radius by the minimum of the distance from each server to its closest client that is not yet covered by a server.
16
25
ZIZO-Full-Un ZIZO-Full-Co Greedy Clustering
20
# of servers selected
# of servers selected
25
15 10 5
ZIZO-Full-Un ZIZO-Full-Co Greedy Clustering
20 15 10 5
0
0 25
30
35 40 delay bound (msec)
45
50
25
30
(a) f = 0.1
45
50
(b) f = 0.3
25
# of servers selected
35 40 delay bound (msec)
ZIZO-Full-Un ZIZO-Full-Co Greedy Clustering
20 15 10 5 0 25
30
35 40 delay bound (msec)
45
50
(c) f = 0.5 Fig. 4. The number of servers selected: full search vs centralized, server capacity unbounded
reduced. Note that our proposed algorithm sometimes finds better solutions than the centralized algorithms do (especially when the delay bound is tight). We observe this to be the case because the centralized algorithms tend to select the servers near the center of the network and keep them in the selected set, while our algorithm can remove redundant servers near the center during the zoom-out process. We verify the effectiveness of the zoom-out in Section 4.4. We further notice that the coordinated client migration offers little benefit. This observation implies that the independent decision made by each client results in as good a server allocation as the case with explicit client coordination. In Figure 5, we evaluate the effectiveness of the partial server search mechanisms with different values of k, i.e., the maximum number of servers that each client searches, when f is 0.3. Figure 5(a) shows the case when clients migrate individually, while Figure 5(b) shows coordinated migration. We observe that, as k increases, the performance of partial search will become close to that of full search in both the figures. In particular, searching for a next server among (maximum) 5 candidate servers produces significant improvement over the tree-based migration, and k = 20 (out of 100 servers) results in a performance that is very close to the full search case. From this result, we can conclude that it is worthwhile for each client to search for the next server from a small set of candidate servers instead of 17
20
ZIZO-Tree-Un ZIZO-Part-Un (k=5) ZIZO-Part-Un (k=10) ZIZO-Part-Un (k=20) ZIZO-Full-Un
15
# of servers selected
# of servers selected
20
10
5
0
ZIZO-Tree-Co ZIZO-Part-Co (k=5) ZIZO-Part-Co (k=10) ZIZO-Part-Co (k=20) ZIZO-Full-Co
15
10
5
0 25
30
35 40 delay bound (msec)
45
50
25
(a) Uncoordinated migration
30
35 40 delay bound (msec)
45
50
(b) Coordinated migration
Fig. 5. The number of servers selected: partial search, server capacity unbounded, f = 0.3
25
ZIZO-Tree-Un ZIZO-Tree-Co ZIZO-Part-Un (k=5) ZIZO-Part-Co (k=5) ZIZO-Part-Un (k=10) ZIZO-Part-Co (k=10) ZIZO-Full-Un ZIZO-Full-Co
# of servers
20 15 10 5 0 25
30
35 40 delay bound (msec)
45
50
Fig. 6. The number of servers selected: bounded server capacity, f = 0.3
searching from a tree. So far we have seen the case when there is no bound on the server capacity. However, in practice, each server will be limited to handling up to certain number of clients. Figure 6 shows the case where the maximum number of clients that each server can handle is 10. In this case, our algorithm first searches the best server to migrate to; however, if the server is full, it searches for the next best server, and so on, until it finds a server that can accommodate more clients. From the figure, the performance of the algorithm improves as we expand the search space for each client by increasing k. However, when the number of candidate servers is very small as in tree-based migration method, the algorithm performs quite poorly. In particular, even when the delay bound is sufficiently large, the tree-based algorithms fail to reach the optimal number of servers (which is 5 servers). This is because migration along a fixed server tree severely limits the search space for each client, and the chance of the same server being contacted for candidate server for migration is relatively high. However, we find that the partial search with large k and full search can find the optimal selection. 18
6
10 8 max # of migrations
# of migrations per client
9 5 4 3 ZIZO-Full-Un ZIZO-Part-Un (k=10) ZIZO-Tree-Un ZIZO-Full-Co ZIZO-Part-Co (k=10) ZIZO-Tree-Co
2 1 0 25
30
35 40 # of clients
7 6 5 4 ZIZO-Full-Un ZIZO-Part-Un (k=10) ZIZO-Tree-Un ZIZO-Full-Co ZIZO-Part-Co (k=10) ZIZO-Tree-Co
3 2 1 0
45
50
25
(a) Average number of migrations per client
30
35 40 # of clients
45
50
(b) Maximum number of migrations
Fig. 7. The number of server migrations of clients: server capacity unbounded, f = 0.3
4.3 Number of migrations
Figures 7(a) and 7(b) plot the average and maximum number of migrations of 50 clients, respectively, with different types of ZIZO algorithms. Each point in the figures is obtained by averaging over the values resulted from 50 simulation runs. Recall that these migrations do not happen in the data session domain but in the control domain. Therefore, the average number of migrations indicates the computation overhead and the message exchange overhead of each algorithm rather than the actual session migration overhead. On the other hand, values in Figure 7(b) (the number of server migrations of the client that visited the most servers) provide a rough indication of the time required to create a session since a game session can be started after all clients have finished the migration steps and selected their servers to connect. In the figure, while the number of required migration steps is slightly larger when the delay bound is smaller, it is somewhat insensitive to the difference in the synchronization delay. This is because, during the zoom-in procedure, all clients first attempt to migrate up to the core servers (regardless of the sync-delay bound), and relatively few client migrations occur in the zoom-out procedure. When we compare different migration/searching strategies, in general, the coordinated client migration methods incur less migration steps than the uncoordinated ones, and the tree-based search scheme results in less number of migrations than the search-based ones. This result is expected because in the tree-based search algorithms at each client there are only a few servers that the client can choose from. On the other hand, we observe that the non-tree search algorithms do not incur an excessive number of migrations either. The difference between the two is only a constant factor and does not change much due to the simulation parameter. In particular, the coordinated migration with partial search in moderate number of candidate servers (e.g., ZIZO-Part-Co(k = 10)) appears to strike a good balance between the complexity and the performance observed in Figure 5. 19
# of servers-ZIZO/# of servers-ZI
1.1 1 0.9 0.8 ZIZO-Tree-Un ZIZO-Tree-Co ZIZO-Part-Un (k=10) ZIZO-Part-Co (k=10) ZIZO-Full-Un ZIZO-Full-Co
0.7 0.6 25
30
35 40 delay bound (msec)
45
50
Fig. 8. Further benefit of Zoom-Out: server capacity unbounded, f = 0.3
4.4 Impact of Zoom-Out
In Figure 8, we quantify the additional benefit obtained by the zoom-out procedure, compared to the case when only zoom-in has been performed. Here we plot the ratio of the number of servers after performing the whole zoom-in-zoom-out algorithm to the best case when all clients have migrated up to the core server without zoomout. We observe that, when the delay bound is tight (25 to 35 msec), 10 to 20 percent of unnecessary servers are further removed from the selected server set by zooming-out clients outward from the core server. This result partially accounts for the sub-par performance of the greedy algorithms presented in Figure 4.
4.5 Adaptivity to incremental client join
In this section, we evaluate the adaptivity of the proposed algorithm by considering the case when the clients incrementally join the gaming network. We handle this incremental growth of game session as follows. First, when a new client arrives, this client connects to the closest server, and probes the synchronization delay. If the synchronization delay is larger than the delay bound, the new client is notified of delay violation and rejected. Otherwise, it is accepted and the client connects to the closest server. We then perform our proposed algorithm periodically after every p succesful joins to the session in an effort to reduce the number of session servers. With this approach, the number of servers would temporarily increase as more clients join until the server selection algorithm is performed in the next round. Thus we expect a tradeoff between the frequency of running the algorithm (i.e., how many times the clients migrate servers overall) and the level of keeping the temporary growth of the set of session servers. In this section, we show the results when the reconfiguration period p has been set to 10. 7 7
An alternative approach to periodical server selection would be to have a threshold in the number of servers and to trigger the ZIZO process whenever the size (or the increase in size) of the session server set exceeds the threshold. The net effect of the periodic method and the threshold-based method is similar. Thus we only present the periodic recomputation
20
25
20
20 # of session servers
# of session servers
25
15 10 ZIZO-Full-Un ZIZO-Part-Un (k=10) ZIZO-Tree-Un ZIZO-Full-Co ZIZO-Part-Co (k=10) ZIZO-Tree-Co
5 0 10
20
30 # of clients
40
ZIZO-Full-Un ZIZO-Part-Un (k=10) ZIZO-Tree-Un ZIZO-Full-Co ZIZO-Part-Co (k=10) ZIZO-Tree-Co
15 10 5 0
50
10
(a) delay bound = 25 msec
20
30 # of clients
40
50
(b) delay bound = 35 msec
Fig. 9. Number of servers selected when clients join incrementally: p = 10, unbounded server capacity, f = 0.3
4 # of migrations per client
# of migrations per client
4
3
2
1
0
ZIZO-Full-Un ZIZO-Part-Un (k=10) ZIZO-Tree-Un ZIZO-Full-Co ZIZO-Part-Co (k=10) ZIZO-Tree-Co 10
20
3
2
1
0 30 # of clients
40
50
ZIZO-Full-Un ZIZO-Part-Un (k=10) ZIZO-Tree-Un ZIZO-Full-Co ZIZO-Part-Co (k=10) ZIZO-Tree-Co 10
(a) delay bound = 25 msec
20
30 # of clients
40
50
(b) delay bound = 35 msec
Fig. 10. Number of migrations per client when clients join incrementally: p = 10, unbounded server capacity, f = 0.3
In Figure 9, we show the number of session servers allocated as new clients join the game session, when the delay bound is relatively tight (Figure 9(a)) and moderate (Figure 9(b)). The x-axis of the figures represents the number of clients that have joined thus far. We see that, as more clients join the game, the number of selected servers temporarily grows, and then drops significantly at the point that the algorithm is performed. When we compare these results to those in Figure 5, which present the case where all the clients are given initially, we can see that running the algorithm “on-line” produces as efficient results as the running it “off-line” since the numbers of servers allocated when the last client finally has joined is close to the corresponding values in Figure 5. Figure 10 plots the average number of migration steps performed by clients. Here we observe that the number of migrations per client tend to decrease as the game session grows. In particular, when the delay bound is moderate (Figure 10(b)), the overhead of the session change monotonically decreases as more clients join. This is because, when a game has a loose delay bound, the server selection by the exscenario in this paper.
21
isting clients tends to form a cluster near the center of the network as more clients join, and therefore, there is not much need for the clients to move their servers to find a better selection. We conclude from these results that, when the game session grows incrementally, our algorithm is effective and scalable in terms of the server selection result and the overhead for session reconfiguration and convergence.
5 Related Work
In the context of real-time group communication, our problem shares some similarity with the delay-constrained multicast tree construction problem [24,25]. The goal of the delay-constrained multicast tree construction problem is to construct a lowest-cost multicast tree with an additional constraint on the upper bound of end-to-end delay between the sender and the receivers. Our work differs in that the delay present in server-client gaming networks is not just an end-to-end delay but a nonlinear combination of client-to-server and server-to-server delay as given in our delay model. Our work is related to the dual of the minimum K-center problem [26], in which the number of servers is to be minimized when the maximum distance between clients and the nearest server is given. In the server selection problem we have considered we must take the inter-server delay into account as well. The (log(N ) + 1)approximation bound is known for this problem, and we are currently investigating the possibility of finding a formal approximation bound for our problem. Our server selection algorithm has been developed for a mirrored server architecture. Recently, the mirrored server architecture has been proposed for distributed gaming [7–9] in order to overcome the limited scalability of the classic client-server architecture. The mirrored server architecture provides many benefits of peer-topeer architecture without burdening the clients with game simulation and administrative functions. To find the closest server among the mirrors, a client can use ping or Internet distance service to find servers in its proximity. In mirrored server architecture, the servers connected via a well provisioned private network to ensure low latency between them. And since the server network is privately owned, advanced features such as IP multicast can be leveraged to facilitate efficient event delivery. In large scale games, the virtual game world is divided into multiple regions or zones [12–14] and only the users in the same zone directly interact with each other. Therefore, it is important to logically group the users in the same zone together and process the events and actions between them. To support clustering of clients and communication between them one can leverage pub-sub technologies and distributed hashing. In [12], the authors proposed to use distributed hash table (DHT) 22
for rendezvous to find a server in a mirrored server architecture. In [13], the authors construct a Voronoi diagram based on the coordinates of each node in the virtual world and maintain peer-to-peer connection between the nodes to hand the dynamics of the game play. In [14], the authors proposed a range queries that can support multiple attributes, and explicit load balancing. Such algorithms can be used for communications in a game session once the server allocation has been resolved by using our algorithm. For efficiency, in the mirrored server architecture, each game server simulates the game state independently based on the game events it receives. Since there is a delay in event delivery in distributed game environments, the game state at each server must be synchronized with each other. For this many synchronization algorithms have been proposed. One of the simplest approaches is the lockstep synchronization. In the lockstep synchronization, as the name suggests, all the events in the system must be acknowledged by all members and processed in lockstep. Since events are processed in sequence there is no inconsistency in the game state. However, since it cannot guarantee that the game will play at the wall clock rate, the lockstep synchronization is not applicable for online games. Another simple mechanism, called the bucket synchronization [16], delays the game simulation for a fixed time period, or bucket, to prevent misordering of events due to network delay. Bucket synchronization is widely used in distributed games for its simplicity. However, its performance and user experience is highly sensitive to the bucket size. Optimistic synchronization algorithms try to reduce the synchronization delay by simulating game play with minimal delay and rolling back the game state when inconsistency has been detected. Time warp synchronization and trailing state synchronization (TSS) [7] are examples of optimistic algorithm. In time warp synchronization, a snapshot of the game state is taken before execution of each event. When a late event arrives which require re-computation, the game state gets rolled back to one of the previous snapshots and game simulation gets re-executed with the new events. TSS maintains multiple game states with different synchronization delays instead of keeping snapshots. Each state processes the events after its own delay and the “leading” state (the state with the smallest delay) rolls back to one of the “trailing” states if inconsistency is discovered. Since it does not need to maintain many snapshots it is more efficient in terms of space complexity. Another interesting issue in distributed gaming environment is to simulate the game play in a fair manner. This problem arises from the fact that different clients may experience different latency to the server, and the game players who are closer to the server can react quickly before other players who are connected over delayed link, thus have unfair advantage. To address this issue, Guo et al. have proposed a proxybased algorithm to ensure so-called fair order service [27]. The main concept of this algorithm is to have a client side proxy, which marks the reaction time of the user, and forwards this information to the server. When the server receives messages, it processes them considering both the reaction time and the arrival time of the 23
messages. The server selection algorithm proposed in this paper does not depend on a particular synchronization technique. We just assume that all servers must receive data from all clients for computing a new game state, which is a necessary condition for most synchronization models. Shaikh et al. [10] and Saha et al. [11] have proposed an on-line games hosting platform by developing middleware based on existing grid components. The proposed gaming architecture is based on utility computing model to support multiple games by leveraging off-the-shelf software components that were originally developed for business applications. For example, they use IBM Tivoli Intelligent Orchestrator (TIO) [28] as the provisioning module for game servers. In addition to performing addition and removal of servers according to the workload dynamics, the provisioning module performs administrative tasks such as (a) new game deployment, (b) directory management of games and players, (c) player redirection to server, and (d) game content distribution. In the commercial world, Butterfly.net [3] is implementing the idea of building a scalable gaming infrastructure based on grid computing technology. However, both these works focus on architectural issues, leaving such issues as server allocation unaddressed.
6 Conclusion
In this paper, we presented a novel distributed algorithm that dynamically selects game servers for a group of game clients participating in large scale interactive online games. The goal of server selection is to minimize server resource usage while satisfying the real-time delay constraint. The proposed algorithm, called zoom-inzoom-out, is adaptive to session dynamics and lets the clients select appropriate servers in a distributed manner such that the number of servers used by the game session is significantly reduced compared to the case when clients select the closest servers. We have considered various zoom-in techniques that the clients can implement; namely (i) tree-based search, (ii) partial server search, and (iii) full server search. We have also considered two different types of client migration; (i) coordinated migration and (ii) uncoordinated migration. Using simulation, we have shown that the proposed zoom-in-zoom-out algorithm results in comparable or better performance than the centralized greedy algorithms, which are based on global information and expensive computation. We also show that, in practice, partial server search algorithm with uncoordinated client migration strikes good balance between the performance and complexity. In particular, we have shown that this scheme can find the optimal solution in many cases and performs closely to the full search algorithm. Furthermore, we have shown that zoom-out process is essential for reducing the number of servers especially when the delay bound is tight. In the scenarios we have considered, the zoom-out process can save up to 20% of server resources. Finally, we have shown that our proposed algorithm is not only effective for a static configuration but also is adaptive to the dynamics in the client set. 24
References
[1] Lineage, http://www.lineage.net/. [2] NCsoft, http://www.ncsoft.net/kor/ncnewsdata/View.asp. [3] Butterfly.net, http://www.butterfly.net/. [4] Terazona, http://www.zona.net/. [5] PlanetSide, http://www.planetside.com/. [6] J. D. Pellegrino, C. Dovrolis, Bandwidth Requirement and State Consistency in Three Multiplayer Game Architectures, in: Proceedings of NetGames, 2003. [7] E. Cronin, B. Filstrup, A. Kurc, A Distributed Multiplayer Game Server System, Tech. rep., Univ. of Michigan (May 2001). [8] D. Bauer, S. Rooney, P. Scotton, Network Infrastructure for Massively Distributed Games, in: Proceedings of NetGames, 2002. [9] M. Mauve, S. Fischer, J. Widmer, A Generic Proxy System for Networked Computer Games, in: Proceedings of NetGames, 2002. [10] A. Shaikh, S. Sahu, M. Rosu, M. Shea, D. Saha, Implementation of a Service Platform for Online Games, in: Proceedings of NetGames, 2004. [11] D. Saha, S. Sahu, A. Shaikh, A Service Platform for OnLine Games, in: Proceedings of NetGames, 2003. [12] T. Iimura, H. Hazeyama, Y. Kadobayashi, Zoned Federation of Game Servers: a Peer-to-Peer Approach to Scalable Multi-player Online Games, in: Proceedings of NetGame, 2004. [13] S.-Y. Hu, G.-M. Liao, Scalable Peer-to-Peer Networked Virtual Environment, in: Proceedings of NetGame, 2004. [14] A. R. Bharambe, M. Agrawal, S. Seshan, Mercury: Supporting Scalable MultiAttribute Range Queries, in: Proceedings of ACM SIGCOMM, 2004. [15] L. Gautier, C. Diot, J. Kurose, End-to-end Transmission Control Mechanisms for Multiparty Interactive Applications on the Internet, in: Proceedings of IEEE Infocom, 1999. [16] C. Diot, L. Gautier, A Distributed Architecture for Multiplayer Interactive Applications on the Internet, IEEE Networks Magazine, vol. 13, no. 4. [17] Y.-J. Lin, S. P. K. Guo, Sync-MS: Synchronized Messaging Service for Real-Time Multi-Player Distributed Games, in: Proceedings of IEEE International Conference on Network Protocols (ICNP), 2002. [18] P. Bettner, M. Terrano, 1500 Archers on a 28.8: Network Programming in Age of Empires and Beyond, in: Game Developers Conference, 2001.
25
[19] D. G. Thaler, C. V. Ravishankar, Distributed Center-Location algorithms, IEEE Journal on Selected Areas in Communications, vol. 15, issue 3. [20] BRITE, http://www.cs.bu.edu/brite/. [21] A. L. Barab´asi, R. Albert, Emergence of Scaling in Random Networks, Science (1999) 509 – 512. [22] L. Qiu, V. Padmanabham, G. Voelker, On the Placement of Web Server Replicas, in: Proceedings of 20th IEEE INFOCOM, 2001. [23] T. H. Cormen, C. E. Leiserson, R. L. Rivest, C. Stein, Introduction to Algorithms, 2nd Edition, MIT Press, 2001. [24] M. Parsa, Q. Zhu, J. J. Garcia-Luna-Aceves, An Iterative Algorithm for DelayConstrained Minimum-Cost Multicasting, IEEE/ACM Transactions on Networking 6 (4). [25] H. F. Salama, D. S. Reeves, Y. Viniotis, An Efficient Delay-Constrained Minimum Spanning Tree Heuristic, TR-96/46, Tech. rep., North Carolina State University (1996). [26] J. Bar-Ilan, D. Peleg, Approximation Algorithms for Selecting Network Centers, in: Proceedings of 2nd Workshop on Algorithms and Data Structures, Lecture Notes in Comput. Sci. 519, 1991, pp. 343–354. [27] K. Guo, S. Mukerjee, S. Rangrajan, S. Paul, A Fair Message Exchange Framework for Distributed Multi-Player Games, in: Proceedings of NetGames, 2003. [28] IBM, IBM Tivoli Intelligent Orchestrator http://www-306.ibm.com/software/tivoli/products/intell-orch/.
26
