This article was originally published in a journal published by Elsevier, and the attached copy is provided by Elsevier for the author’s benefit and for the benefit of the author’s institution, for non-commercial research and educational use including without limitation use in instruction at your institution, sending it to specific colleagues that you know, and providing a copy to your institution’s administrator. All other uses, reproduction and distribution, including without limitation commercial reprints, selling or licensing copies or access, or posting on open internet sites, your personal or institution’s website or repository, are prohibited. For exceptions, permission may be sought for such use through Elsevier’s permissions site at: http://www.elsevier.com/locate/permissionusematerial
Computer Communications 29 (2006) 3553–3566 www.elsevier.com/locate/comcom
py
A network-centric approach to enhancing the interactivity of large-scale distributed virtual environments
co
Duong Nguyen Binh Ta *, Suiping Zhou
School of Computer Engineering, Nanyang Technological University, Singapore 639798, Singapore
al
Received 25 December 2005; received in revised form 31 May 2006; accepted 31 May 2006 Available online 27 June 2006
Abstract
pe
rs
on
In Distributed Virtual Environments (DVEs), where many simultaneous human users interact with each other in a shared, 3D virtual world, enhancing interactivity is of crucial importance, since the success of a DVE greatly depends on users’ perceptions. In general, enhancing interactivity for large-scale DVEs with thousands of geographically distributed users faces various challenges such as large Internet latencies, high resource demands and high potential of security threats. In this paper, we propose a network-centric approach to enhance the interactivity of DVEs by directly minimizing the network latencies in client–server communications. We consider two key problems: the server placement problem and the client assignment problem. We have formulated these two problems and proved that they are both NP-hard. We then adapt and develop several degree-based server placement approaches and greedy client assignment algorithms. Extensive simulation studies using realistic models have shown that our approach is very effective in enhancing the interactivity of large-scale DVEs. 2006 Elsevier B.V. All rights reserved. Keywords: Distributed Virtual Environments; Geographically distributed server architecture; Server placement; Client assignment; Interactivity enhancement
r's
1. Introduction
Au
th o
In recent years, advances in networking technologies, computer graphic and CPU power have popularized the deployments of Distributed Virtual Environments (DVEs). Large-scale DVEs refer to a class of applications that enable thousands of geographically distributed users1 to simultaneously interact with each other in a shared, computer-generated 3D virtual world, where each client is represented by an avatar [30]. A client controls the behavior of his/her avatar by various inputs, and the updates of an avatar’s state need to be sent to other clients in the same zone of the virtual world to support the interactions among *
Corresponding author. Tel.: +6591186750; fax: +6567926559. E-mail addresses:
[email protected] (D. N. B. Ta),
[email protected] (S. Zhou). 1 In this paper, the terms ‘‘user’’, ‘‘participant’’ and ‘‘client’’ all refer to DVE users and are used interchangeably. 0140-3664/$ - see front matter 2006 Elsevier B.V. All rights reserved. doi:10.1016/j.comcom.2006.05.015
clients. Prominent applications of DVEs include online games, military simulations, collaborative designs, virtual shopping mall, etc. The latest study by the market research company IDC (http://www.idc.com) on the online gaming market of Asia/Pacific (excluding Japan) revealed that the subscription revenue is about US$1.09 billion in 2004, an increase at about 30% compared to 2003. Also according to this study, this market will be more than doubled in 2009. Enhancing the interactivity of DVEs is of crucial importance, since the users may not feel that they are interacting with other elements in the virtual world if the responses from the DVE are much slower than what the users experience in real-life. However, in general, seeking a good balance between enhancing interactivity and other important issues in DVEs such as maintaining consistency, managing resource bottlenecks and securing the virtual world system under various technical problems such as large Internet latency, high resource demands and high
D. N. B. Ta, S. Zhou / Computer Communications 29 (2006) 3553–3566
co
py
and the servers. This problem has been studied extensively in the context of Web replica placement [10,27,24]. However, to our knowledge there is no existing work that assesses the suitability of Web replica placement approaches to the server placement problem in DVEs. In this paper, we look at the state-of-the-art results in the Web replica placement problem. We then discuss the difference between the server placement problem in DVEs and the Web replica placement problem. Finally, we suggest some possible approaches adapted from Web replica placement strategies that we believe appropriate to the server placement problem in DVEs. 2.1. Web replica placement problem
al
The explosive growth of the World Wide Web in the late 1990’s had demanded efficient content delivery architectures to offer better service to web users at lower costs. A Content Distribution Network (CDN) is a system of many (hundreds or thousands) well-connected servers deployed over the Internet to transparently deliver web contents to end users by allocating the replicas of contents at the edges of the Internet, i.e., close to the users. Hence, in large CDNs like Akamai [1], the Web replica placement is crucial to the system performance. Informally stated, the Web replica placement problem concerns how to choose K web replicas among N potential sites (i.e., network locations) in the network, where K < N, to minimize a given objective function, usually the clients’ access latencies, or the total bandwidth consumption. After the replicas are in place, each web client connects to its closest replica to retrieve the contents needed. Here, it is assumed that each client needs to access only one replica to get all of its interested contents. The Web replica placement is usually formulated as a minimum K-median problem, which is a well-known NP-hard problem. In [27], the authors proposed a greedy placement algorithm that is shown to be able to perform very well compared to a ‘‘super-optimal’’ algorithm3 that uses Lagrangian relaxation with subgradient optimization. The authors showed in [27] that the median performance in terms of client–server communication delays of the greedy algorithm is within a factor of 1.1–1.5 of the optimal.
Au
th o
r's
pe
rs
potential of security threats is a very challenging task. In this paper, we have proposed a network-centric approach to enhancing the interactivity of large-scale DVEs. Our methodology is different from the traditional applicationcentric approaches, e.g., dead reckoning, which aims to ‘‘hide’’ the effect of network latencies by predicting the state of remote users at the application level. Since the activities of human users in DVEs are highly unpredictable, such approach may result in visual perception errors, or in more serious cases, inconsistent DVE states [34]. Our network-centric approach is based on a geographically distributed server architecture (GDSA) [21,9], in which multiple geographically distributed servers are connected to each other. Each client is connected to one of these servers, and clients interact with each other through these servers. This server-based architecture essentially provides good ways to implement consistency control, resource management and security mechanisms. In addition, in order to deal with large-scale DVEs with hundreds, or even thousands of clients interacting simultaneously, usually the virtual world is spatially partitioned into many distinct zones, with each zone managed by only one server, as in [3]. A client only interacts with other clients in the same zone,2 and may move to other zones. As a server only needs to handle one or more zones instead of the entire virtual world, the system is more scalable. In this paper, we refer to such a partitioning approach as the zone-based approach. In this paper, we consider two key problems with the GDSA and the zone-based partitioning approach: the server placement problem and the client assignment problem. Both problems aim to reduce the client–server communication delays, but the former achieves this goal by finding good locations to place the servers, while the latter does so by assigning clients to appropriate servers. We have formulated the two problems and proved that they are both NP-hard. To address these two problems, we have developed and adapted several server placement approaches and some client assignment algorithms. Extensive simulation studies have shown that our approach is very effective in enhancing the interactivity of large-scale DVEs. The rest of the paper is organized as follows. Section 2 introduces and addresses the server placement problem in DVEs. The client assignment problem and algorithms are presented in Section 3. Section 4 discusses the relations between server placement and client assignment. Simulation study is described in Section 5, and Section 6 concludes the paper.
on
3554
2.2. Server placement problem in DVEs
A key question in the GDSA is where to place the servers to reduce the communication delay between the clients
Although the Web replica placement problem has been well studied in the context of CDNs with promising results, the appropriateness of the proposed approaches to the server placement problem in DVEs has not yet been studied. The main difference between the Web replica placement problem and the server placement problem in DVEs lies in the ‘‘connection’’ of clients to servers. In the
2 For simplicity, we say that a client is in a zone if its avatar is currently residing in that zone.
3 The ‘‘super-optimal’’ solution may be better than the optimal, but may not be feasible.
2. Server placement problem
3555
Fig. 1. Max-router placement approach.
al
placement. In this approach, servers are randomly placed in the network.
on
Remark 2.1. The computational cost of the random placement approach is O (Mm), where M is the total number of routers, and m is the number of servers.
2.3.2. Degree-based placement The degree-based server placement approaches aim to place the servers at some key locations in the network. These key locations are usually the network nodes (routers or ASes) with high node degrees. The degree-based approaches have been shown to perform comparably to the greedy server placement algorithm in [27] in the context of the Web replica placement problem [24]. However, whether they are appropriate for DVEs needs to be further investigated.
r's
pe
rs
first problem, it is relatively straightforward to let web clients connect to their closest replicas, hence the greedy placement approach is feasible. However, in DVEs, the large virtual world is partitioned into multiple distinct zones, and each zone is hosted by a separate server. A client cannot simply connect to its closest server since that server may not manage its zone. Hence, the greedy placement approach is not appropriate for the server placement problem in DVEs, as it assumes that clients always connect to their closest servers. So, for the server placement problem in DVEs, we need an approach that can place the servers in appropriate network locations to reduce clients’ communication latencies without knowing in advance which server the clients will connect to. In [10,24], the authors have shown that by placing Web replicas at some key network locations, i.e., at the network nodes (routers or Autonomous Systems (ASes)) with high node degrees, the clients’ communication latencies can be significantly reduced, regardless of the clients’s locations in the network. We feel that such approaches are applicable to the server placement problem in DVEs. However, the mechanisms in [24,10] were only evaluated under the assumption that clients would eventually connect to their closest servers, although this assumption was not used in deciding server placements. Hence, in this paper, we adapt the placement approaches in [24,10] to address the server placement problem in DVEs, and evaluate the appropriateness of these approaches in DVEs. To our knowledge, our work is the first to consider server placements to enhance the interactivity of DVEs. In the following section, we suggest several possible server placement approaches.
co
py
D. N. B. Ta, S. Zhou / Computer Communications 29 (2006) 3553–3566
2.3.1. Random placement The random placement serves as a basic approach for comparison in the sense that any good server placement approach should perform better than the random
4 Indeed, network hop counts are good indications of round-trip network delays [23].
Au
th o
2.3. Server placement approaches for DVEs
2.3.3. Max-router In the Max-router placement approach, we sort all the routers in the network in descending order of their node degrees using Quicksort, and place servers at the routers with the highest node degrees. Intuitively, a node with larger node degree is likely to be closer to other nodes, hence it is a good choice for server placement. For example in Fig. 1, router B has the largest node degree (which is 5). If a server is placed there, then the network distances (in number of hops4) from client c1 and c2 to the server are 2 and 1, respectively. However, if the server is placed at router A, which has a node degree of 1, then the above client– server distances become 3 and 2, respectively. Thus, placing servers at nodes with large node degree may effectively reduce the client–server communication delay, which implies good interactivity.
The computational cost of the Max-router approach is O (M logM) + O (m), or where M is the total number of routers in topology, and m is the number of servers,
py
In CDNs, after the web replica placement is done, assigning clients to servers is relatively straightforward, i.e., each client is connected to its closest server. However, in the context of DVEs with the zone-based partitioning approach, this simple client assignment approach is not appropriate, since a client may interact in a zone which is not hosted by the client’s closest server. Hence, in this section, independent of any specific server placement, we formulate a new problem, termed the client assignment problem (CAP). This problem essentially concerns how to assign clients to distributed servers to reduce the client– server communication delays. 3.1. Definitions
al
Before formulating the CAP, we introduce the following notations and concepts: • ci – A client in the DVE. • C = {c1, . . . , ck} – The set that consists of all clients in the DVE. • zi – A zone in the DVE. This is also used to denote a set that consists of all clients in a zone. • Z = {z1, . . . , zn} – The set that consists of all zones in the DVE. • si – A server in the DVE. • S = {s1, . . . , sm} – The set that consists of all servers in the DVE. • Rsi – The resource consumption on a server si. This can be measured by CPU usage, network bandwidth usage, etc. Since the network bandwidth often represents the major operating cost in current server-based online games [18], in this paper, we assume that the server CPU is not a bottleneck, and measure the resource consumption by the network bandwidth usage. • Rzi – The total amount of resource, i.e., network bandwidth, used by all the clients that are interacting in zone zi on the server which is hosting zi. • C si – The resource capacity of a server si. • d ci sj – The round-trip network delay between a client ci and a server sj. • D – The delay bound of a DVE. The delay bound indicates the required upper bound of the round-trip communication delay between a client and its server to guarantee the interactivity of the DVE. For different types of DVEs, there are different delay bound requirements. For example, First Person Shooter (FPS) games typically require a delay bound of 250 ms [16], while carracing games have much more stringent latency requirement, at about 100 ms [25]. It should be noted that the communication delay between a client and its server is different from the network delay between the client and its server. The communication delay is the sum of the network delay and the processing delay at the server. However, in this paper we assume that the server CPU is
rs
2.3.4. Max-AS/Max-router The Max-AS/Max-router placement approach is similar to the Max-router approach in the sense that they both aim to place servers at nodes with large degree in the network topology. The main difference between the two approaches is that Max-AS/Max-router uses a hierarchical placement approach which is based on the structural nature of real network topologies. First, Max-AS/Max-router approach selects the ASes with the largest AS-level node degrees. In each AS, we place one server at the router with the largest node degree. By spreading servers over multiple ASes, this approach may further reduce network latencies for geographically distributed clients compared to the Max-router approach. To implement the Max-AS/Max-router approach, we sort all the routers according to the node degrees of their corresponding ASes (each router belongs to an AS). If there are two routers with the same AS-level node degree, we further consider their router-level node degrees. We then iterate over the sorted list of all routers to select one best router in each AS to place the server.
3. Client assignment problem
on
Remark 2.2. placement O (M logM), the network m > M.
D. N. B. Ta, S. Zhou / Computer Communications 29 (2006) 3553–3566
pe
Remark 2.3. The computational cost of the Max-AS/Maxrouter placement approach is O (M logM) + O (M), or O (M logM), where M is the total number of routers.
Au
th o
r's
2.3.5. Other placement approaches As comparative references, we also consider some alternative server placement approaches, namely Min-router, Max-AS/Min-router and Min-AS/Max-router. The Minrouter approach, which place the servers at the nodes with the minimum node degrees, may serve as an ‘‘upper bound’’ on how bad a server placement approach can be. The Max-AS/Min-router5 and Min-AS/Max-router6 may help to identify which (AS or router) is the major factor in the hierarchical server placement approach. However, note that in practice, sometimes we have to adopt these methods for server placement since typically the routers with high node degrees may be very busy due to the high volumes of network traffic, thus it may be impossible to add any more service to them [24]. Moreover, some economic and administrative constraints may make it harder to place servers at the selected network locations.
5 This approach is similar to Max-AS/Max-router, but in each AS, the router with minimum node degree is selected to place the server for that AS. 6 This approach is similar to Max-AS/Max-router, but we select the ASes with minimum node degrees, and in each selected AS we select the router with maximum node degree to place the server.
co
3556
D. N. B. Ta, S. Zhou / Computer Communications 29 (2006) 3553–3566
3557
number of clients with QoS in the DVE would be maximized. The client assignment problem is formulated as follows.
Fig. 2. Geographically distributed server architecture.
m X n X i¼1
subject to n X Rzj xij 6 C si ;
not a bottleneck, thus the client–server communication delay is determined by the client–server network delay. In this paper, the terms ‘‘network delay’’ and ‘‘communication delay’’ are used interchangeably.
C ij xij
8i 2 I;
ð2Þ
ð3Þ
m X
al
j¼1
xij ¼ 1;
i¼1
on
xij 2 f0; 1g;
8j 2 J ;
ð4Þ
8i 2 I; 8j 2 J :
ð5Þ
Remark 3.1. In Definition 3.1, constraint (3) ensures that the capacity of each server will not be exceeded. Constraint (4) means that each zone is assigned to only one server.
r's
pe
rs
For interactive applications like DVEs, the client–server communication delay is the most important Quality of Service (QoS) parameter that the system provides to clients [16]. In this paper, we say that a client is with QoS or without QoS if the communication delay between the client and its server is smaller or larger than the delay bound, respectively. To illustrate the client assignment problem, we consider the example in Fig. 2. Assuming that d c1 s1 ¼ 50 ms; d c1 s4 ¼ 80 ms; d c2 s1 ¼ 150 ms; d c2 s4 ¼ 50 ms; d c4 s1 ¼ 200 ms; d c4 s4 ¼ 50 ms and the delay bound of this DVE is 100 ms, then it would be better for the two clients c2 and c4 which are interacting in the zone z1 of the virtual world if z1 is hosted by server s4 instead of s1. For client c1, her experience would not be damaged when zone z1 is hosted by s4 since d c1 s4 < 100 ms. From this example, it is obvious that appropriate assignment of clients to servers7 is crucial to enhance the interactivity of DVEs.
j¼1
co
CðX Þ ¼
py
Definition 3.1. Client assignment problem (CAP) Let I = {1, . . . , m} and J = {1, . . . , n} be the set of indexes of servers and zones in the system, respectively. For each i 2 I and j 2 J, given the cost Cij of assigning zone zj to server si as defined in Eq. (1), find an assignment matrix X = (xij), with xij = 1 if zone zj is assigned to server si or xij = 0 otherwise, which minimizes the total cost
th o
3.2. Formulation
Au
With the geographically distributed server architecture and the zone-based partitioning approach, the CAP concerns how to assign each zone (with all of its clients) of the virtual world to an appropriate server so that the total number of clients with QoS in the system is maximized. To formulate this problem, we first propose the following metric to measure the ‘‘cost’’ of assigning a zone zj to a server si C ij ¼ jfck 2 zj : d ck si > Dgj
ð1Þ
where j Æ j denotes the cardinality of a set. Cij measures the number of clients in a zone zj that do not satisfy the delay bound D, i.e., without QoS. Therefore, by minimizing the total cost of the assignment, the total 7 Recall that the assignment of clients to servers is determined by the assignment of zones to servers.
Theorem 3.1. The client assignment problem (CAP) as described in Definition 3.1 is NP-hard.
Proof. We consider in our model the special case where the resource capacity C si of each server is the same. We further assume that the cost Cij when assigning a zone zj,j 2 J = {1, . . . , n} to server si,i 2 I = {1, . . . , m} has the special form Di 1, where D>1 is a constant, i.e. assigning any zone to server s1 costs C1j = 1, to server s2 costs C2j = D, to server s3 costs C3j = D2, etc. Then the optimization goal of CAP in this specific case would be to assign all the zones to the smallest possible number M of servers, with server indexes range from 1 to M. This is exactly the well-known Bin Packing problem [17], in which a bin corresponds to a server and an item corresponds to a zone. Since the CAP generalizes the Bin Packing problem, which is NP-hard, CAP is also NP-hard. h 3.3. Related work To the best of our knowledge, there is no existing work that directly addresses the client assignment problem in DVEs as we have described. Research on how to assign clients to servers in DVEs is usually formulated as a load distribution problem in a locally distributed server architecture, i.e., all the servers are placed in the same machine room [31,20]. Such approaches may damage the interactivity of the DVE, since clients may be far away (in terms of network delays) from the servers.
3558
D. N. B. Ta, S. Zhou / Computer Communications 29 (2006) 3553–3566
co
al
pe
rs
on
3.4. Assignment algorithms Since the CAP is NP-hard, we seek some heuristic solutions instead of the optimal one. We propose three algorithms for the CAP. The first one randomly assigns the zones, while the second one is a greedy heuristics to minimize the number of clients without QoS in the system. The third algorithm is also a greedy heuristics similar to the second one, except that it uses a different cost metric.
with the highest value of lij and with sufficient resource capacity. This procedure is adapted from the well-known approach used to solve the Generalized Assignment Problem [28]. Let us illustrate the effectiveness of the Greedy-1 algorithm using an example. Assuming that the DVE has two zones z1 and z2, and two servers s1 and s2. Each server can only take one zone, due to its capacity constraint. Let us further assume that assigning z1 to s1 or s2 costs C11 = 10 or C21 = 20, respectively. Similarly, assigning z2 to s1 or s2 costs C12 = 5 or C22 = 10, respectively. The desirability difference q1 of zone z1 is equal to l11 l21 = C11 (C21) = 10 (20) = 10, while q2 of z2 is equal to l12 l22 = C12 (C22) = 5 (10) = 5. Hence, the Greedy-1 algorithm will select z1 to assign first. As a result, z1 is assigned to server s1, and z2 is assigned to server s2, since each server can only take one zone. The total cost of this assignment is 10 + 10 = 20, i.e., in total there are 20 clients that are without QoS in the system. However, if we do not follow the procedure of Greedy1, then z2 may be chosen to assign first. In this case, z2 will be assigned to s1, thus z1 has to be assigned to s2. The total cost in this case would be 20 + 5 = 25, which is larger than the cost obtained using Greedy-1. Thus, by sorting the zones according to their desirability differences q, Greedy-1 can effectively reduce the number of clients without QoS in the system. We analyze the computational cost of the Greedy-1 algorithm. Let m, n and k denote the number of servers, the number of zones and the total number of clients, respectively. The Greedy-1 algorithm consists of four parts. The first part (from line 2 to line 8) is to find all the values of lij. The cost of this part is O (mk). The second part (from line 9 to line 12) is to find qj for all zj 2 Z. In the line 10, Quicksort is used. Hence, the cost of this part is O (n)[O (m logm) + O (1)], or O (mn logm). The third part (line 13) is to sort the list of n values of qj. Using Quicksort, the cost is O (n logn). The final part (from line 14 to line 27) is to assign zones to servers. There are m servers and n zones, hence the cost of this part is O (nm). Hence, the overall computational cost of the Greedy-1 algorithm is O (mk) + O (mn logm) + O (n logn) + O (nm), or O (max{O (mk), O (mn logm), O (n logn)}).
py
Recently, the interest in investigating server and network architectures to reduce the effect of network latency for DVEs has been increased. In [22,29], the authors introduced the concept of latency driven distribution (LDD) for the distribution of a DVE over the networking architecture, as opposed to the traditional resource driven distribution (RDD), i.e., load distribution. Our client assignment problem shares the idea of the LDD concept. However, the authors of [22,29] only investigate the provision of immersive audio communication in DVEs within the context of the LDD concept. In their work, two clients that belong to two adjacent DVE zones are able to hear each other. However, several important kinds of interactions that may happen in a DVE such as shooting enemies, manipulating objects, changing object ownership, etc., were not studied. These interactions may raise complicated consistency issues, if the clients in adjacent zones are managed by different physical server machines.
r's
3.4.1. Random assignment (Random) In the random assignment algorithm, zones are assigned to randomly selected servers with the only concern of not to overload the servers. In this algorithm, the following procedure is repeated until all zones have been assigned: first we select a random zone zj, and then a random server si with sufficient capacity is selected to take zj, i.e., the target server of all clients in zj is set to si.
th o
Remark 3.2. The computational cost of the Random algorithm is O (mn), where n is the number of zones and m is the number of servers.
Au
3.4.2. Greedy assignment – algorithm 1 (Greedy-1) Since the random assignment algorithm is oblivious to client–server network delays when assigning zones to servers, the obtained performance in terms of the number of clients with QoS may not be good, i.e., the cost of the assignment as defined in Eq. (2) may be high. Hence, in the Greedy-1 algorithm, we use a greedy heuristics to minimize the total number of clients without QoS in the system. The pseudo-code is shown in Fig. 3. Let lij = Cij be a heuristic measure of the desirability of assigning zone zj to server si. Thus, the smaller the cost Cij is, the higher the desirability lij is. The algorithm iteratively considers all the unassigned zones and pick a zone zj with the maximum difference qj between the largest desirability lijj and the second largest desirability lsj. Then, zj is assigned to a server si
Remark 3.3. The complexity of the Greedy-1 algorithm is O (max{O (mk), O (mn logm), O (n logn)}), where n is the number of zones, m is the number of servers and k is the number of clients in the system. In practice, usually k m and k n, hence the overall complexity of Greedy-1 for most cases can be written as O (mk). 3.4.3. Greedy assignment – algorithm 2 (Greedy-2) The only difference between Greedy-1 and Greedy-2 lies in the cost metric they use. With the cost metric in Eq. (1), Greedy-1 aims to minimize the total number of clients without QoS in the system directly. For the Greedy-2
3559
r's
pe
rs
on
al
co
py
D. N. B. Ta, S. Zhou / Computer Communications 29 (2006) 3553–3566
th o
Fig. 3. Greedy assignment, algorithm 1.
C0ij ¼
ck 2zj
jzj j
Au
algorithm, we propose to use the following metric to measure the cost when assigning a zone zj to a server si P d ck si ð6Þ
C 0 ij measures the average delay from all client ck 2 zj to server si if they are all assigned to si. By minimizing this cost function, it is expected that the selected server si for zj will be near (in terms of network delay) to the center of mass of the client population of zi, thus the number of clients with QoS may become large. 3.4.4. Reference algorithm: lp_solve For comparison, based on the Integer Programming formulation of the CAP, we use the so-called
‘‘branch-and-bound’’ algorithm implemented in the free Mixed Integer Linear Programming (MILP) solver lp_solve [4] to obtain the optimal solutions for the CAP. Note that this approach is only applicable when the system size is small, otherwise the running time of lp_solve will become very long (on the order of several hours), which is clearly impractical for highly interactive applications like DVEs. 3.5. Implementation considerations In this section we discuss some practical considerations on the implementation of the proposed assignment algorithms. The first consideration is the dynamic property of DVEs. During the course of interactions in the virtual world, clients may move from one zone to another, new clients may join, existing clients may also leave the virtual
D. N. B. Ta, S. Zhou / Computer Communications 29 (2006) 3553–3566
co
5. Performance evaluation
py
does not require any special infrastructure to be deployed. As we have already mentioned, placing servers at good network locations is not always achievable, due to several technical and economic constraints. In such scenarios, efficient client assignments are highly desirable to enhance the interactivity of DVEs. Hence, to some extent, we may say that client assignments are of greater importance than server placements, although it would be highly advantageous to combine the effectiveness of both methods.
al
In this section, we compare various server placement approaches and client assignment algorithms. We first study the impacts of server placement and client assignment separately. Then, their combined impacts are investigated. More specifically, while studying the impacts of server placement approaches, we use the basic client assignment mechanism, i.e., the Random assignment algorithm. On the other hand, while studying the impacts of client assignment algorithms, the Random server placement approach is used. 5.1. Simulation models and parameters 5.1.1. Network models Both synthetic topologies generated by the popular topology generator BRITE [2] and real Internet topologies are used in the simulations. Table 1 lists the topologies used in this paper. In our simulations, we used both synthetic topologies generated by the popular topology generator BRITE [2] and real Internet topologies. Table 1 lists the topologies used in this paper. We use BRITE to generate both flat and hierarchical topologies based on the well-known Waxman and Barabasi–Albert model (F-W, F-BA, H-BA/W). The Waxman model [33] considers all pairs of nodes, and then decides whether to add a link between any two nodes with a probability that depends on the distance between these two nodes and the longest distance between any two nodes in the network. The Barabasi–Albert model [8] generates network topologies that exhibit power-laws as observed in the seminal paper by Faloutsos et al. [11] in 1999. To complement the synthetic topologies generated by BRITE, we use a real, flat Internet topology collected from NLANR [5]. For diversity, we also collect a real AS-level
th o
r's
pe
rs
world. An obtained client assignment may not be good after some time. Thus, the proposed algorithms need to be executed from time to time to ensure good client assignments. We should note that dynamic execution of the client assignment algorithms may incur inter-server bandwidth consumption due to the migrations of zones and clients across servers. Nevertheless, we believe this cost is much smaller compared to the bandwidth consumption due to the interactions of users in the DVE. This suggests that the bandwidth cost of re-executing the assignment algorithms is not a big concern. Indeed, a similar assumption is used in the context of the Web replica placement problem [27]. Moreover, in a recent study [19] of the very popular MMOG Lineage II developed by NCsoft, Korea, the authors found that the average value of session durations, i.e., the average time a player keeps on playing the game, is about 3 h, which is rather long. This may indicate that we do not have to execute the assignment algorithms frequently. Another issue is how to obtain the input data for the assignment algorithms. The input data includes the client–server round-trip network delays and the resource requirement of each client on a server. The network delays can be obtained using some scalable network measurement tools such as King [14] or IDMaps [13]. King uses existing recursive DNS queries to accurately estimate the roundtrip network delays between arbitrary Internet end hosts, while IDMaps relies on special end hosts (referred to as the tracers) deployed at some strategic locations in the Internet. Both approaches are scalable and incur little estimation overhead. In this paper, the server resource requirement of each client is measured as the bandwidth requirement. It is well known that the bandwidth requirement in client–server architectures increases quadratically with the total number of clients that are interacting with each other [26]. Thus, we can estimate in advance the bandwidth requirement of each client in a zone based on the number of clients in that zone, as in [26].
on
3560
4. Server placement versus client assignment
Au
In Sections 2 and 3, we have discussed several methods for interactivity enhancement for two different problems, namely server placement and client assignment. In this section, we briefly discuss their relations. First, server placement and client assignment can complement to each other in enhancing the interactivity of DVEs. For example, if all the servers are already placed at some good network locations, then appropriate client assignment algorithms can be used to further enhance the interactivity of the DVE. On the other hand, a good server placement certainly helps to maximize the performance of client assignment algorithms. Second, we believe that client assignment are more flexible than server placement in the sense that the former
Table 1 Network topologies Topology
Nodes
Links
Flat, Waxman (F-W) Flat, Barabasi–Albert (F-BA) Flat, real (F-R) Hierarchical, Barabasi–Albert/Waxman (H-BA/W) Hierarchical, real/real (H-R/R)
3000 3000 3024 3000 3300
6000 5997 5192 6197 13442
D. N. B. Ta, S. Zhou / Computer Communications 29 (2006) 3553–3566
5.1.4. Default settings Unless otherwise stated, the following assumptions and default values are used in the simulations. The clients are uniformly distributed in the physical world as well as in the virtual world. To estimate bandwidth requirement [26], the input sending frequency of each client (frame rate) is set to 25 messages/s, and the size of each input or update is 100 bytes, which are close to real settings [7]. The maximum round-trip delay between any two nodes in the network topology is set to 300 ms. The interactivity requirement, i.e., the DVE delay bound D, is set to 150 ms. The default DVE configuration is 20s-400z5000c-2500cp.
co
py
topology (generated by processing the Border Gateway Protocol (BGP) routing tables) with 110 nodes from [6], and use the DFN (German Research Network) topology [15] with 30 nodes as the router-level topology to construct a realistic hierarchical topology (H-R/R). While these topologies are not complete, they at least partially reflect the ‘‘true’’ topology of the Internet, which may have great impacts on the simulation results. For simplicity, we assume that the round-trip network delay between any two nodes in the network topology is proportional to the number of link-hops between them. This assumption is similar to the one used in most of the previous work [27,24]. In fact, a recent Internet measurement [23] also showed that round-trip delay is well-correlated with network hop counts. Moreover, to obtain more realistic simulations results, we use both shortestpath routing and AS-level hierarchical routing [32] where possible to calculate the network delays. The AS-level hierarchical routing is a more realistic routing strategy for our simulations, since to some extent it may reflect the ‘‘true’’ routing practice in the current Internet.
5.2. Impacts of server placement
on
al
Figs. 4 and 5 show the cumulative distribution function (CDF) of client–server round-trip communication delays using different server placement approaches and network topologies. For flat topologies (Fig. 4), we observe that the Max-router placement performs very well: in terms of pQoS, the performance improvements are 157% (F-W topology), 168% (F-BA topology) and 26% (F-R topology) compared to the Random placement, and 157%, 275% and 92% compared to the Min-router placement. Moreover, from these figures, it is clear that Max-router not only has a high ratio of clients with QoS, but also provide better interactivity for clients that are without QoS. It should be noted that the performance improvement in F-R topology is not as high as that in F-W and F-BA topologies. However, this is not because the Max-router placement performs poorly in F-R topology. As observed in Fig. 4(c), the Random placement also performs quite well in F-R topology. One possible explanation for this result is that the F-R topology, which is a real-world topology collected from [5], has good connectivity, i.e. any node can reach any other node with a small number of hop counts. Indeed, the average round-trip delay between any two nodes in F-R is much smaller than those in F-W and F-BA. For hierarchical topologies (Fig. 5), we obtain similar results to the case of flat topologies, except that the MaxAS/Max-router placement performs the best in this case: its performance improvements in terms of pQoS over the Random placement are 38% (H-BA/W) and 48% (H-R/R).
r's
pe
rs
5.1.2. Workload models Based on existing studies of real online game systems, e.g., [12], we simulated different client distributions (clustered, uniform, etc.) both in the physical and the virtual world. We obtained similar results for all tested network topologies and client distributions. For clarity, we only present here the most representative results. Different DVE configurations are used for performance evaluation. A specific DVE configuration is determined by the number of servers, the number of zones, the number of clients, and the total resource capacity of the system. We use the notation number of servers-number of zones-number of clients-capacity to denote a DVE configuration. For example, the notation 20s-400z-5000c-2500cp means that the DVE has 20 servers, 400 zones, 5000 clients and 2500 Mpbs server bandwidth in total.
3561
b
Random Max-router Min-router 175
200
225
250
delay (msec)
275
300
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 150
c
CDF
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 150
CDF
CDF
a
Au
th o
5.1.3. Performance measure The main performance measure used in the analysis is the percentage of clients with QoS in the system, denoted as pQoS. Results presented here are obtained by averaging the results of 50 simulation runs.
Random Max-router Min-router 175
200
225
250
delay (msec)
275
300
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 150
Random Max-router Min-router 175
200
225
250
delay (msec)
Fig. 4. Impacts of server placement, flat topologies. (a) F-W topology. (b) F-BA topology. (c) F-R topology.
275
300
D. N. B. Ta, S. Zhou / Computer Communications 29 (2006) 3553–3566
b
1
1
0.9
0.9
0.8
0.8 Random Max-router Max-AS/Max-router Min-router Max-AS/Min-router Min-AS/Max-router
0.7 0.6 0.5 150
CDF
CDF
a
175
200
225
250
Random Max-router Max-AS/Max-router Min-router Max-AS/Min-router Min-AS/Max-router
0.7 0.6
275
0.5 150
300
175
200
delay (msec)
225
250
delay (msec)
py
3562
275
300
a
1
c
1
b
0.9
co
Fig. 5. Impacts of server placement, hierarchical topologies. (a) H-BA/W topology. (b) H-R/R topology.
0.95
0.8
0.9
0.9
0.5
175
200 225 250 delay (msec)
Random Greedy-1 Greedy-2 lp_solve
0.75
275
300
0.7 150
0.8 0.75
175
200
225 250 delay (msec)
275
Random Greedy-1 Greedy-2 lp_solve
0.7
0.65
300
on
0.3
0.85 0.8
Random Greedy-1 Greedy-2 lp_solve
0.4
CDF
0.6
0.85
al
CDF
CDF
0.7
0.2 150
1
0.95
0.6 150
175
200
225 250 delay (msec)
275
300
Fig. 6. Impacts of client assignment, small configuration. (a) F-W topology. (b) F-R topology. (c) H-BA/W topology.
5.3. Impacts of client assignment
Au
th o
r's
pe
rs
This can be explained by the fact that Max-AS/Max-router spreads the servers across multiple ASes, while Max-router may place several servers in the same AS. In addition, it is observed that Max-AS/Min-router and Min-AS/Maxrouter both perform worse than Max-AS/Max-router, while Min-AS/Max-router is the worst among the three. This observation suggests that the AS-level node degree seems to be the major factor that affects the performance. However, we should also note that only selecting the ASes with largest AS-level degrees to place the servers is not enough: the router-level node degree within an AS is also an important factor. It is interesting to note that our results here are similar to the observations in [24], although [24] addresses the Web replica placement problem, which is very different from the server placement problem in DVEs. Recall that for a typical DVE, which is partitioned into multiple distinct zones, we do not know in advance which server a client will connect to. Thus, we have to place the servers without knowing clients’ locations, then randomly assign zones to servers and let each client connect to the server hosting its zone.8 In contrast, in the context of Web replica placement problem, a client just simply connects to its closest server. Hence, to some extent, the Web replica placement problem may be ‘‘easier’’ to solve than the DVE server placement problem. Nonetheless, it is clear that the degree-based placement approach is universal. It can be applied into both DVE server placement and Web replica placement with good results.
8
Here, we are using the Random assignment algorithm.
To investigate the proposed client assignment algorithms, we use two configurations, 5s-30z-400c-200cp (small) and 20s-400z-5000c-2500cp (large). The small configuration is only used to enable the comparison between our algorithms and the optimal solution produced by the MILP solver lp_solve [4]. In all other cases, we use the large configuration. Figs. 6 and 7 show the CDF of client–server round-trip communication delays for our algorithms in the small and large configurations. In all cases, we observe that the Greedy-1 and Greedy-2 significantly outperforms the delay-oblivious Random algorithm in terms of pQoS. We also observed that among the three algorithms (Random, Greedy-1 and Greedy-2), Greedy-1 has the best performance in terms of pQoS. This can be explained by the fact that Greedy-1 aims to directly minimize the total number of clients without QoS in the system, while Greedy-2 aims to assign a zone to a server that is close (in terms of network delay) to the center of mass of the client population in that zone. In all results, the performance improvements of Greedy-1 over Random are 63% (F-W), 58% (F-BA), 16% (F-R), 35% (H-BA/W) and 25% (H-R/R) for small configuration, and 116% (F-W), 110% (F-BA), 26% (F-R), 48% (H-BA/W) and 38% (H-R/R) for large configuration.9
9 We note that the performance improvements are better in large configuration, since having more distributed servers means that the greedy algorithms may have more chance to assign clients to good server locations.
D. N. B. Ta, S. Zhou / Computer Communications 29 (2006) 3553–3566
a
b
1 0.9
c
1
1 0.95
0.95
0.8
0.9 0.9
0.6 0.5
0.85
CDF
CDF
0.85
0.4
Random Greedy-1 Greedy-2
0.3 175
200
225 250 delay (msec)
275
0.7 150
0.7
Random Greedy-1 Greedy-2
0.75 300
0.8 0.75
0.8
175
200 225 250 delay (msec)
275
Random Greedy-1 Greedy-2
0.65 0.6 150
py
CDF
0.7
0.2 150
3563
300
175
200
225 250 delay (msec)
275
300
b
1
0.9 0.85
0.8 0.75
0.9
0.85
0.8 0.75
0.7
200
225
250
275
Random Greedy-1 Greedy-2
0.65 300
0.6 150
175
200
225
250
275
300
0.7
delay (msec)
Random Greedy-1 Greedy-2
0.65 0.6 150
on
175
0.8 0.75
0.7
Random Greedy-1 Greedy-2
0.65
1
0.95
CDF
0.9 0.85
0.6 150
c
1 0.95
CDF
CDF
0.95
al
a
co
Fig. 7. Impacts of client assignment, large configuration. (a) F-W topology. (b) F-R topology. (c) H-BA/W topology.
delay (msec)
175
200
225
250
275
300
delay (msec)
rs
Fig. 8. Impacts of imperfect inputs, H-BA/W. (a) Error factor is 1.2. (b) Error factor is 2. (c) Error factor is 4.
r's
pe
We also note that the pQoS values of Greedy-1 are close to the optimal results given by the branch-and-bound algorithm implemented in the lp_solve software. Note that lp_solve can only be applied to small size DVEs, i.e., the small configuration. The average execution time of lp_solve for this configuration is around 1 s. For the large configuration, the results by lp_solve are not shown since the execution time was too long (not finished after more than 10 h), which is clearly impractical for interactive application like DVEs which need timely assignment decisions. In both small and large configurations, all of our proposed algorithms took less than 0.5 s to execute.
Au
th o
5.3.1. Imperfect input data The simulation results obtained above for the client assignment algorithms are based on the assumption that we have perfect information about the network delays between clients and servers. In practice, we usually have rough estimations of network delays rather than perfectly accurate information. To simulate the estimation error, we use the method presented in [27]. More specifically, we apply an error factor e to the perfect input data, i.e., assuming the exact value of the delay is d, then the delay value input to our algorithms is uniformly distributed in the range [de , de]. Note that although the algorithms use inaccurate inputs to make assignment decisions, their resulted performances, i.e., pQoS, are still calculated based on the actual network delays after the assignment is done. As in [27], we use three different values of e: 1.2, 2 and 4, representing the inaccuracies of the popular network delay estimation tools King [14] and IDMaps [13].
Fig. 8 shows a typical simulation result obtained with the inaccurate estimations of network delays. Despite the imperfect knowledge in network delay, the greedy algorithms are still much better than the Random algorithm which does not consider network delays. In particular, for e = 1.2, Greedy-1 is still the best algorithm, and its pQoS only decreases slightly compared to the cases that use perfect information. However, when the estimation error becomes large, i.e., e = 2 or 4, Greedy-2 is the best algorithm. Fig. 8 shows that Greedy-2 exhibits low sensitivity to input data estimation errors, even when the error is very large, i.e., e = 4. This suggests that Greedy-2 is a more suitable choice for the client assignment problem under Internet environments, where perfectly accurate input data for assignment algorithms is usually impossible or very costly to obtain. 5.4. Combined impacts In this section, we investigate the possibility of combining server placement and client assignment to further enhance the interactivity of DVEs. We propose to study two cases. The first one is the case in which the DVE requires a tighter delay bound, e.g., 100 ms, while the second case considers the scenario in which we are forced to place servers at some bad network locations, i.e., the routers with lowest node degrees. The reason for studying the first case is that we would like to know to which extent a combination of good server placement and good client assignment can enhance the interactivity of DVEs. For the second case, our aim is to answer the question whether
D. N. B. Ta, S. Zhou / Computer Communications 29 (2006) 3553–3566
a
b
1
c
1
0.9
CDF
0.6 0.5 0.4 0.3
Max-router + Random Random + Greedy-2 Max-router + Greedy-2
0.2 0.1 100
150
0.8
0.8
CDF
0.7
CDF
0.9
0.9
0.8
0.7
0.5 100
300
0.7 0.6
Max-router + Random Random + Greedy-2 Max-router + Greedy-2
0.6
200 250 delay (msec)
1
150
200 250 delay (msec)
0.5 0.4 100
300
Max-AS/Max-router + Random Random + Greedy-2 Max-AS/Max-router + Greedy-2
py
3564
150
200 250 delay (msec)
300
a
b
1
co
Fig. 9. Combined impacts, tight delay bound. (a) F-W topology. (b) F-R topology. (c) H-BA/W topology.
c
1
0.9 0.9
0.4 0.3 0.1 150
175
200
225 250 delay (msec)
0.7 0.6
Min-router + Random Min-router + Greedy-2
0.2
0.8
CDF
0.5
0.9
275
300
0.58 150
0 175
0.8 0.7
al
0.6
CDF
CDF
0.7
0.6
Min-router + Random Min-router + Greedy-2
on
0.8
1
200 225 250 delay (msec)
275
300
0.5 150
Min-router + Random Min-router + Greedy-2 175
200 225 250 delay (msec)
275
300
rs
Fig. 10. Combined impact, bad server placement. (a) F-W topology. (b) F-R topology. (c) H-BA/W topology.
Au
th o
r's
pe
it is possible for good client assignment algorithms to improve the interactivity level of DVEs when the servers are placed at bad network locations, due to some technical or economic constraints. In the case of tight delay bound (Fig. 9) we use a combination of Max-AS/Max-router (or Max-router)10 and Greedy-2.11 In all cases the combined approach Max-AS/ Max-router (Max-router) + Greedy-2 outperforms the Max-AS/Max-router (Max-router) or Greedy-2 alone. More specifically, the improvements in terms of pQoS of the combined approach over Max-AS/Max-router (Maxrouter) are 144% (F-W), 130% (F-BA), 29% (F-R), 90% (H-BA/W) and 59% (H-R/R). Similarly, the improvements over Greedy-2 are 144% (F-W), 225% (F-BA), 62% (F-R), 34% (H-BA/W) and 33% (H-R/R). This observation suggests that combining good server placement with good client assignment is a very effective approach to enhance the interactivity level of DVEs. In the case of bad server placement (Fig. 10), it is also observed that, despite the fact that servers are already placed at network nodes with lowest degrees, the Greedy2 can still improve the interactivity of the system significantly. More specifically, the improvements of Min-router + Greedy-2 over Min-router + Random in terms of 10
If hierarchical topologies are used, the best server placement approach is Max-AS/Max-router. Otherwise, the best one is Max-router. 11 We select Greedy-2 since it gives the best performance under Internet environments, where input data to client assignment algorithms are usually inaccurate.
pQoS are 150% (F-W), 100% (F-BA), 44% (F-R), 42% (H-BA/W) and 28% (H-R/R). 5.5. Summary of results We have evaluated several degree-based server placement approaches and client assignment algorithms by extensive simulations. Our main findings are as follows. • First, appropriate degree-based server placement approaches perform very well to enhance the interactivity of DVEs. These approaches do not require to know the clients’ locations in advance, thus they do not suffer from the inaccuracies in network delay estimations, and can be applied straightforwardly to DVEs. However, these approaches may not be applicable in every scenarios, due to some inherent technical and economic constraints. • Second, client assignment algorithms use the knowledge of network delays to determine appropriate client-toserver assignment. This approach also yields excellent performance, and is universally applicable, since it does not require any special server positioning. However, the performance of client assignment algorithms seems to inevitably depend on accurate information of the client–server network delays. Nevertheless, our experiment results have shown that a greedy assignment algorithm, Greedy-2, exhibits low sensitivity to delay estimation errors, and performs well even when the error is very large. This property is very useful in Internet
D. N. B. Ta, S. Zhou / Computer Communications 29 (2006) 3553–3566
r's
pe
rs
on
al
co
6. Conclusions Due to the huge resource demands and the real-time requirement of large-scale DVEs, a geographically distributed server architecture is usually needed to support such applications. In this architecture, multiple servers are geographically distributed over the network, and the large virtual world is partitioned into distinct zones to distribute load among the servers. In this paper, we have proposed a network-centric approach to enhancing the interactivity of large-scale DVEs. With this approach, we have considered two key problems, namely server placement and client assignment. Both of them are proved to be NP-hard. Several heuristic placement approaches and assignment algorithms are proposed to address these problems. Finally, extensive simulation studies with realistic models have shown that our approach is very effective in enhancing the interactivity of large-scale DVEs. For the future work, it would be very interesting to further investigate the performance of the degree-based server placements and client assignments in the real deployment of a large-scale DVE, such as an MMOG. In addition, distributed algorithms for client assignment will also be investigated to extend our current work.
[8] A.L. Barabasi, R. Albert, Emergence of scaling in random networks, Science (1999) 509–512. [9] D. Bauer, S. Rooney, P. Scotton, Network infrastructure for massively distributed games, Proc. NetGames (2002). [10] E. Cronin, S. Jamin, C. Jin, A.R. Kurc, D. Raz, Y. Shavitt, Constrained mirror placement on the internet, IEEE J. Selected Areas Commun. (2002). [11] M. Faloutsos, P. Faloutsos, C. Faloutsos, On power-law relationships of the internet topology, ACM SIGCOMM (1999). [12] W.C. Feng, W.C. Feng, On the geographic distribution of online game servers and players, Proc. NetGames (2003). [13] P. Francis, S. Jamin, C. Jin, Y. Jin, D. Raz, Y. Shavitt, L. Zhang, IDMaps: a global internet host distance estimation service, IEEE/ ACM Trans. Networking 9 (5) (2001). [14] K. Gummadi, S. Saroiu, S. Gribble, King: estimating latency between arbitrary internet end hosts, Proc. ACM SIGCOMM IMW (2002). [15] O. Heckmann, M. Piringer, J. Schmitt, R. Steinmetz, Generating realistic ISP-level network topologies, IEEE Commun. Lett. (2003). [16] T. Henderson, S. Bhatti, Networked games: a QoS-sensitive application for QoS-insensitive users, Proc. ACM SIGCOMM (2003). [17] D.S. Hochbaum, Approximation Algorithms for np-hard Problems, PWS Publishing Company, Boston, MA, 1997. [18] B. Patrovsky, J. Mulligan, Developing Online Games: An Insider’s Guide, New Riders Games, 2003. [19] J. Kim, J. Choi, D. Chang, T. Kwon, Y. Choi, E. Yuk, Traffic characteristics of a massively multi-player online role playing game, Proc. NetGames (2005). [20] J. Lui, M. Chan, An efficient partitioning algorithm for distributed virtual environment systems, IEEE Trans. Parall. Distrib. Syst. 13 (3) (2002). [21] M. Mauve, S. Fischer, J. Widmer, A generic proxy system for networked computer games, Proc. NetGames (2002). [22] C.D. Nguyen, F. Safaei, P. Boustead, Comparison of distributed server architectures in providing immersive audio communication to massively multi-player online games, Proc. ATNAC (2004). [23] K. Obraczka, F. Silva, Network latency metrics for server proximity, in: Proceedings of the IEEE Globecom 2000, 2000. [24] R.G.P. Radoslavov, D. Estrin, Topology-informed internet replica placement, in: Proceedings of Web Caching and Content Distribution Workshop, 2001. [25] L. Pantel, L. Wolf, On the impact of delay on real-time multiplayer games, Proc. NOSSDAV (2002) 23–29. [26] J. Pellegrino, C. Dovrolis, Bandwidth requirement and state consistency in three multiplayer game architectures, Proc. ACM NetGames (2003). [27] L. Qiu, V. Padmanabhan, G. Voelker, On the placement of web server replicas, Proc. IEEE INFOCOM (2001). [28] H. Romeijn, D.R. Morales, A class of greedy algorithms for the generalized assignment problem, Discreet Appl. Math. 103 (2000) 209–235. [29] F. Safaei, P. Boustead, C.D. Nguyen, J. Brun, M. Dowlatshahi, Latency driven distribution: infrastructure needs for participatory entertainment applications, IEEE Communication Magazine on Entertainment Everywhere: System and Networking Issues in Emerging Network-Centric Entertainment Systems 43 (5) (2005). [30] S. Singhal, M. Zyda, Networked Virtual Environments: Design and Implementation, Addison-Wesley, Reading MA, 1999. [31] D.N.B. Ta, S. Zhou, A dynamic load sharing algorithm for massively multi-player online games, in: Proceedings of the 11th IEEE International Conference on Networks, 2003. [32] H. Tangmunarunkit, R. Govindan, S. Shenker, D. Estrin, The impact of routing policy on internet paths, Proc. IEEE INFOCOM (2001). [33] B.M. Waxman, Routing of multipoint connections, IEEE J. Select. Areas Commun. 9 (1988) 1617–1622. [34] S. Zhou, W. Cai, B.S. Lee, S.J. Turner, Time-space consistency in large-scale distributed virtual environments, ACM Trans. Model. Comput. Simulation 14 (1) (2004) 31–47.
py
environments, where perfectly accurate input data for assignment algorithms is usually impossible or very costly to obtain. • Third, a combination of good server placement with good client assignment is able to significantly improve the interactivity of the DVE, compared to the cases in which only server placement or client assignment is used. Moreover, a good client assignment can still improve the interactivity of DVEs even if servers are placed at some bad network locations.
Acknowledgements
[1] [2] [3] [4]
Au
References
th o
The authors thank Dr. Gunther Raidl for helping with the NP-hardness of the Generalized Assignment Problem. We also thank the reviewers for their constructive comments and valuable suggestions which help to improve the quality and presentation of this paper.
Akamai. Available from: . BRITE. Available from: . Everquest. Available from: . Mixed Integer Linear Programming solver lp_solve. Available from: . [5] NLANR. Available from: . [6] SSF. Available from: . [7] A. Abdelkhalek, A. Bilas, A. Moshovos, Behavior and Performance of Interactive Multi-player Game Servers, Special Issue of Cluster Computing: the Journal of Networks, Software Tools and Applications, 2002.
3565
D. N. B. Ta, S. Zhou / Computer Communications 29 (2006) 3553–3566 Suiping Zhou is currently an Assistant Professor in the School of Computer Engineering at Nanyang Technological University, Singapore. Previously, he was a Post-doctoral fellow at Weizmann Institute of Science, Israel. He received his B.Eng., M.Eng. and Ph.D in Electrical Engineering from Beijing University of Aeronautics and Astronautics, P.R. China. His current research interests include distributed interactive applications, hybrid control systems and human behavior representation for virtual training systems.
Au
th o
r's
pe
rs
on
al
co
Duong Nguyen Binh Ta is currently a Ph.D student in Computer Science, School of Computer Engineering, Nanyang Technological University, Singapore. He received his B.Eng in Computer Science from the Department of Information Technology, University of Technology, Ho Chi Minh city, Vietnam. His current research interests include Distributed Virtual Environments, Internetworking, Grid computing and optimization.
py
3566
