Large Scale Distributed Virtual Environments on the Grid - Springer Link

Large Scale Distributed Virtual Environments on the Grid: Design, Implementation, and a Case Study Congfeng Jiang, Xianghua Xu, Jian Wan, Wanqing Li, and Xindong You Grid and Service Computing Lab, School of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou 310018, P.R. China {cjiang,xuxhcs,wanjian,liwanqing,youxindong}@hdu.edu.cn

Abstract. Distributed Virtual Environments (DVEs) are geographically distributed computing systems which allow multiple users to interact simultaneously in computer-generated, shared virtual environments. DVEs are widely applied in many fields, such as online role playing games, synthesis military simulations, virtual cities, and virtual shopping malls. To support large-scale DVEs with high fidelity and satisfied interactive performance among various distributed clients, massive computing resources, storage resources, and highly efficient distributed server architectures are needed. In this paper, the key techniques of the system design and implementation for large scale Distributed Virtual Environment on the Grid (DVE-G) are discussed and a hierarchical architecture of DVE-G is proposed. Some aspects of DVE-G, such as resource management, massive data management, scene synchronization, and faulttolerance are also discussed. A work-in-progress DVE-G prototype (e.g., the Virtual Hangzhou Dianzi University tourism system) is provided as a testbed scenario and the present evaluation results show that the architecture of DVE-G is feasible and scales well. Keywords: Grid Computing, Distributed Virtual Environments, Resource Management, Scene Synchronization, Virtual Tourism.

1 Introduction Grid Computing is motivated by the desire to provide transparent access to resources among many organizations to solve large scale distributed cooperative scientific and engineering problems, such as multi-institutional scientific computing and data analysis, and large scale distributed simulation [1,2,3,4]. By integrating resources from multiple domains, a computing grid provides aggregated computation resources to a large number of users. A typical computational grid consists of geographically distributed heterogeneous computing resources, storage resources (hard disks, RAIDs, High Performance Storage System), sensors, and visualization workstations, with uniform resource management system. In grids, the resource management systems provide users consistent, pervasive, inexpensive and secure access to high-end computing capabilities. Nowadays, grids have shown global popularity and harnessed the aggregate computing power to hundreds of TFlops. For example, Mustafee et al. [5] W. Shen et al. (Eds.): CSCWD 2007, LNCS 5236, pp. 384–395, 2008. © Springer-Verlag Berlin Heidelberg 2008

Large Scale DVE-G: Design, Implementation, and a Case Study

385

presented the WinGrid approach to grid enabling existing Windows-based Commercial-Off-The-Shelf (COTS) simulation packages (CSPs). They used non-dedicated, de-centralized, commodity PCs in the Ford Motor Company to achieve a linear speedup when WinGrid is used to harness idle PC computing resources. Distributed Virtual Environments (DVEs) [6] have become more realistic and effective with the advent of computer generated scenarios, visualizations and entity behaviors, as well as cheaper bandwidth and higher performance PCs. However, due to the various requirements in DVEs, e.g., immersiveness, presence, performance, interaction, it is time-consuming and expensive to design and develop a traditional large scale, multi-institutional DVE. Moreover, DVE systems often require huge computing resources and data sets which may be geographically distributed. Fortunately, Grid computing technology is a promising means for alleviating this cost. Furthermore, DVE applications and related applications will benefit from grid computing as it enables collaboration and the use of distributed computing resources, while also facilitating access to geographically distributed data sets. Especially, grid computing based DVEs [4,7,8,9] allows multiple users connected to a networked synthetic environment to interact simultaneously with one another and facilitate interoperability among entities between various institutions. Since traditional networked server architectures can not scale well to support large scale DVE applications, DVE on the Grid (DVE-G) is an ideal and truly scalable solution for this kind of system. In a large scale distributed cooperative virtual environment running over a grid, there are numerous components running concurrently. The interaction jobs may need computing powers from geographically distributed organizations and the availability of computing resources may change during the execution of the specific application. Thus resource management mechanisms, including job scheduling, load balancing, faulttolerance should be addressed in order to meet the high-fidelity, fault-tolerant, and realtime requirements of the applications and the performance requirements demanded. This paper presents the architecture of large scale Distributed Virtual Environment on the Grid (DVE-G), resource management, and the scene synchronization framework of it. Using grid computing technologies, DVE-G can be governed by the grid middleware, and user operations or interactions, including job executions, can easily be integrated into it. Moreover, main research issues involved in applying grid technology to DVEs and the key challenges which need to be solved, such as security aware job scheduling, load balancing, authorization and authentication, can also be incorporated and implemented to make the application fault tolerant and scalable. The rest of the paper is organized as follows: Section 2 presents a brief review of related works on distributed virtual environments over grids. In Section 3, we present a hierarchical architecture and resource management framework of DVE-G. And in section 4, synchronization handling mechanism is presented for DVE-G. We present a work-in-progress prototype and its implementation of DVE-G in Section 5. Finally, we summarize the contributions and make some remarks on further research in Section 6.

2 Related Work Grid technology has gained widespread diffusion within scientific and business communities. It can not only offer sharing and cooperatively using distributed computational resources in a transparent way with respect to their physical locations, but also

386

C. Jiang et al.

present computational and storage resources spread all over the world and managed by different institutions, as they were a single, virtual supercomputer. Thus, parallel tasks and applications could be managed and executed over widespread and heterogeneous platforms. In order to support large scale distributed collaborative simulation, design, or online games, and to cater for the increasing computation demand of it, it is necessary to harness distributed resources over the grid. The High Level Architecture (HLA) for Modeling and Simulation [10] was developed to facilitate interoperability among simulations and promote reuse of simulation components. Using HLA, a large scale distributed simulation can be constructed using a huge number of geographically distributed computing nodes. For example, Chen et al. [11] proposed a Decoupled Federate Architecture, which is concerned with large scale distributed simulation, such as collaborative environments for HLA simulations on the Grid. Lu et al. [12] proposed the design of a Web-based environment to support large scale distributed simulation using Java and HLA. Lees et al. [14] proposed HLA_AGENT, a tool for the distributed simulation of agent-based systems, which integrates the SIM_AGENT agent toolkit and the HLA simulator interoperability framework. In this paper, the prototype we developed was constructed based on Globus Toolkit 4.0[13] and implemented in Java due to its platform-neutrality. The advantages of the HLA are interoperability and reusability. However, the HLA specification only defines a distributed runtime infrastructure (RTI) but not its implementation. Therefore, heterogeneous RTIs can not interoperate. Furthermore, the HLA does not provide any mechanism for managing the resources where the simulation is being executed [15]. In this paper, we have tackled the resources management problem by introducing a hierarchical architecture for DVE-G, which separate the simulation execution and resources management transparently. We also present some preliminary evaluation results, which show that the overhead introduced by the grid resource management extension is acceptable even for high-latency networks with limited computational requirements. In large scale DVEs, latency is critical for providing an acceptable quality of service (QoS), such as immersiveness, presence, performance, and interaction [16, 17]. Ta et al. [18] proposed a Geographically Distributed Server Architecture (GDSA) to provide real-time interactive performance among various clients. They also proposed a network-centric method to provide interactivity enhancement in DVEs, which is achieved through directly minimizing the network latencies in client-server communications [19]. In order to reduce the amount of data communication with acceptable network traffic overhead, and to increase interactive performance and scalability between nodes in DVE-G, suitable Interest Management (IM) mechanism must be integrated. Minson et al. [20] proposed a Distributed Shared Variable abstraction based system which derives interest expressions automatically at the infrastructure level in a 'bottom-up' pattern to improve the precision of the IM system without sacrificing transparency from modelers’ perspective. Pullen et al. [21] proposed a pattern prediction based middleware-layer IM framework to inform the oscillation of the system's protocol for the processing of state updates between two competing modes. To improve the processing performance of Data Distribution Management (DDM) in DVE-G with large traffic volume and loads, Gu et al. [22] presented an Adaptive


387

Dynamic Grid-Based (ADGB) scheme to optimize DDM time through analysis of matching performance. ADGB uses an advertising scheme where information about the target cell involved in the process of matching subscribers to publishers is known in advance.

3 DVE-G Architecture In a grid based large scale DVE, tasks may need computing resources from various institutions. However, the availability of these computing resources may change during the execution of the simulation. In order to meet the real-time requirements demanded by tasks, global resource management mechanisms are required. Moreover, in order to provide support for dynamic balancing of workload, load monitoring, dynamic load balancing, check-pointing and process migration mechanism in a grid are also required. Consequently, we propose a hierarchical architecture of DVE-G in this paper, which separate the job execution and resources management transparently. The DVE-G architecture is shown in Fig.1, which is similar with the OGSA specification. From the highest level, the DVE-G architecture is broken into fabric layer, connectivity layer, resource layer, collective layer, and application layer. The architecture can achieve hardware transparency, interoperability between different processes, and global resource management over various grid sites. We will interpret the proposed architecture from bottom to top detailedly in the following paragraphs. Firstly, it is a fabric layer at the bottom of the architecture, which consists of various hardware and software resources, such as computational resources, communication resources, storage resources, network resources, audio and video equipments, sensors, categories, etc. The resources may be physical infrastructures, such as supercomputers, computer clusters, workstations and PCs, or logical entities, such as a distributed file system. Therefore, the resources provide sharing operations for higher levels to make a physical transparency in front of end users.

Application

Grid Portal

Collective

Scheduling Load balancing Monitoring

Resource

Processes Management

Visualization

Connectivity TCP/UDP

HTTP

Fabric

Grid site 2

Grid site1

Logging

Brokering Authentication Grid site n

Fig. 1. Hierarchical architecture of DVE-G

The second layer of the architecture, namely, connectivity level, provides core communication and authentication protocols required for DVE-G, such as TCP, UDP,

388

C. Jiang et al.

HTTP, etc. The communication protocols enable the exchange of data between grid sites. And the communication protocols can also enable the exchange of messages between user tasks or processes. For example, messages in DVE-G are frequently exchanged between 2-dimensional maps and 3-dimensional virtual scenes for interoperations. Socket communication can serve as the courier between numerous user processes. As for communication between tasks resided in different sites, it is feasible to send it using XML streams, which provides machine-to-machine information exchange between users or avatars in virtual worlds without a human-in-the-loop. For example, an XML stream for answering an entity location query maybe like the following: Additionally, authentication protocols are used to provide security mechanisms for certificating the identity of users and resources in a risky grid. The third layer of the architecture, namely, resource layer, provides services for resource brokering, processes management, and accounting for individual resources. Specifically, it is a higher logical layer with respect to the hardware resources in the first layer. The services include information services and management services. The former can be used to obtain the information of the structure and status of grid sites or resources, such as configuration, current user identity, security policy, etc. The latter enables secure negotiation between shared resources. Summarily, resource layer enables the user jobs to access and control local resources transparently. The collective layer in the architecture contains services that designed for global resources and interactions among virtual organizations. The services include global resource management, job scheduling and load balancing, monitoring, data replication services, service discovering, etc. In large scale DVE-G, there are a large number of objects, users and vast computing resources. Collaborations at different locations are prone to failure. Moreover, the failure of one virtual environment can lead to the crash of the overall application execution. Such risk increases with the scale of DVE-G. Therefore, fault-tolerance is required to support runtime robustness. In order to provide fault tolerance for DVE-G, a Failure Detection and Handling Service (FDHS) framework was proposed to deal with failures, as shown in Fig.2. FDHS enables the detection of both task crashes and user-defined exceptions. A notification mechanism is used which is based on the interpretation of notification


389

messages being delivered from the underlying grid resources, which avoids any unnecessary repeated messages passing when handling failures. The failure recovery module serves as a high-level recovery service, which enables support for multiple failure recovery techniques, the separation of failure handling strategies from the application code, and user-defined exception handlings. The monitoring and reporting mechanism provide correct failure recovery and the overhead for facilitating faulttolerance is minimal.

Failure Recovery

Resource Monitoring

Process and Data Saving

Status of Grid Sites

Status of Process

Fig. 2. Failure detection and handling service framework

A resource monitoring center is always used to provide performance information in aid of failure recovery. The central server and the backup server collect performance information from all other sites periodically and monitor the status of jobs submitted or being run. Every remote host running a job communicates the job execution status to the monitoring center periodically. However, a centralized monitoring may not scale well when the number of grid sites increases. One solution to this problem is to provide monitoring knowledge that correlates alarms and provides fewer, but more meaningful advisory information. Moreover, such a solution could reduce operator load further through automatic corrective action. The top layer, e.g., application layer, comprises the DVE-G applications that run within virtual organization. The applications are constructed by numerous services that will be executed in DVE-G, such as execution control (start, stop, or restart), scene visualization, logging, grid portal, etc. Furthermore, Data Management (DM) is one of the key factors, which provides a flexible and extensible mechanism for data manipulation in the grid and delivering to grid sites based on participating entities' interests of data[23]. DM is of essential importance especially for DVE-G because data management provides the capabilities of the following data services: high performance data transferring or data movement services, data replication management, replica location management and data access management. In DVE-G, massive data, such as large terrains, remote sensing images, texture pictures and other raster images, must be operated efficiently to make a realtime and high-fidelity virtual world.

390

C. Jiang et al.

4 Scene Synchronization To keep the shared virtual environments consistent among a very large number of participating users is the key challenge for a large scale DVE-G system [24,25], which is termed as the synchronization problem. As the number of user grows, more information is generated, which increases network traffic and worsens the network latency. Since the Internet suffers from high network latency, it adds additional challenges to synchronizing the collaboration during virtual world. Scene partitioning, Interest Management and Data Distribution Management are basically sub-solutions for synchronization. To support closely coupled collaborative activity and highly interactive performance around shared users and objects, it is crucial to partition the virtual environment during various users to render individual scenes and to distribute the load among different participating nodes or servers. These requirements include the ability to refer to a specific shared space through scene partitioning for each user to be able to manipulate objects within that space despite of objects out of the current scene. However, it must be noted that there are tradeoffs between presence, fidelity, and interactive performance. In large scale DVEs, there exist more various challenges, such as high Internet latencies, resource demands and Internet security threats. Moreover, to satisfy some specific time constraints or system scalability, quality of service (QoS) assurance is need. The ultimate goal of scene partitioning and Interest Management is to restrict the data transportation only among users and entities who are interested in receiving information about others. However, scene partitioning and the inherent distributed nature of DVEs require more messages passing between sub-regions. The extent to which the amount of the synchronization messages transmitted is dependent on the region granularity. Therefore, an alternative is to filter the data exchange to limit and control the network traffic in DVE applications [26]. However, most of these solutions usually make use of a large number of multicast groups, which send large volume of unnecessary and redundant data. Thus, when to use a minimum number of multicast groups is the key point to accomplish acceptable network traffic and latencies. In this paper, we modified the proposed networked-server architectures approach in [27] to a hybrid adaptive architecture, which releases the servers from updating the location of avatars when synchronizing the state of the system. In our method, the updating messages will not be exchanged by avatars if the avatars are always in a specific scene or virtual space. Specifically, multiple servers are used and an adaptive scene partitioning algorithm is employed to partition the whole virtual environment into regions dynamically. Servers are classified into ordinary servers and super servers, which consists of multiple ordinary servers. All objects in an individual region are managed by one ordinary server. In most cases, when a user or avatar is exploring a region, the corresponding server will be responsible for serving all requests from the viewer. When a viewer is crossing the boundary of two or more regions, the data is exchanged directly between the regions if the regions are belonging to the same super-server. If not, synchronization messages will be exchanged between super-servers first, and then request information will be exchanged between the servers of all the regions involved.


391

The bottleneck of server in [27] is partially avoided in our method. Consequently, the synchronization communications are greatly reduced, as well as the interactive performance and high fidelity are kept well. This method is more robust and scalable both in very large-scale DVEs or mid-scale DVEs. The prototypical implementation shows that the introduced communication overhead is acceptable.

5 A Case Study Based on the proposed framework of DVE-G, we are developing a prototype, e.g. Virtual Hangzhou Dianzi University (VHDU) tourism system, based on Globus Toolkit 4.0 due to its extensive usage of Web Services mechanisms to define the interfaces and structure its components. This prototype was developed to support the serviceoriented distributed applications for DVE-G and addressed the problems such as security, resource access, resource management, data movement, resource discovery, etc. We use Replica Location Service (RLS) [13] as a distributed registry service that records the locations of data copies and allows discovery of replicas. OGSA-DAI (OGSA Data Access and Integration) [13] was used to provide data services to access relational and XML databases, as well as some types of structured file data resources. In DVE-G, a fundamental problem is the mapping of job execution or data transfer onto resources that meet requirements for performance, cost, security, or other quality of service (QoS) metrics. This mapping needs resources discovering services and monitoring services for applications. The complete architecture of the prototype implementation is shown in Fig. 3.

Applications

Visualization

Grid Portal

Info Service

WebMDS

Ganglia

Security

MyProxy

SimpleCA

Data Mgmt

GridFTP

RFT

Res. Mgmt

GRAM

Trust Model

GT 4.0

Red Hat Linux 9

RLS

Java CoG

Grid Nodes + 100Mb LAN

Fig. 3. Architecture of the VHDU prototype

392

C. Jiang et al.

(a) The whole virtual campus

(b) The #1 teaching and research building Fig. 4. VHDU tourism system

Users can login to the VHDU system as himself or his avatar and interact with each other remotely. Objects or entities can be added into VHDU by various distributed users. In VHDU, users can navigate the whole virtual campus through a pre-defined or user-defined path, which can overcome the limitations of realistic campus orientation. VHDU will not only provide various indirect tour experiences to the campus, but also develops a new type of tour system, leading to new worlds which have not been accessible to the public because of various restrictions coming from the current technology, transportation means, and physical limits of human body. We also provide fundamental functions for virtual tourism, including 3-dimensional scene interaction, layer visualization, viewpoint control, collision detection, scene optimization, etc.


393

When a traveling interaction request is submitted, it will be processed automatically and the results will be provided to the user transparently in the sense that the existing navigation code runs unmodified and the users or avatars are unaware that other parts of the system are running remotely and simultaneously. User events and actions in the virtual environment are captured by VRML (Virtual Reality Modeling Language) event handling mechanism and JavaScript handling codes. The multi-users collaborative interaction in a virtual world shows that VRML, with the addition of VRML extension APIs, JavaScript, JSP (Java Server Pages), and the Java 3D API, can be successfully used to create high-performance DVE-G applications. Since we have not finished modeling of traveler’s behaviors, such as chatting with each other in virtual scene, and roaming about the campus, we only present two virtual scenes in Fig.4. There are two cars in Fig.4, which are constructed by different users remotely and can be controlled by distributed users. For example, the user can drive the car in the campus. The whole virtual campus is shown in Fig.4 (a). And the 3-dimensional virtual model of the #1 teaching and research building is illustrated in Fig.4 (b).

6 Conclusions and Future Work We have studied framework design for distributed virtual environment on the grids (DVE-G). We have also developed the data management framework and scene synchronization algorithm for DVE-G. A prototype was provided and the implementation details were discussed. The proposed architecture is feasible and scalable in a real grid with dynamic user participations. The prototypical implementation also provides some interesting guidelines about the way to adopt grid computing technology to DVE to support large-scale DVE applications. In our future work, some implementation issues, such as security, scalability, data and state synchronization will be further addressed. And the VHDU system will be improved to allow users to interact through self-defined motions and gestures within a shared virtual environment via a local network or through the Internet. Acknowledgments. The funding support of this work by State Key Development Program of Basic Research of China (Grant No. 2007CB310900), Science and Technology Research and Development Program of Zhejiang Province, China (Grant No. 2006C13105) are greatly appreciated.

References 1. Xu, Z., Yang, N., Liao, H.: Vega Information Grid for Collaborative Computing. In: Shen, W.-m., Lin, Z., Barthès, J.-P.A., Li, T.-Q. (eds.) CSCWD 2004. LNCS, vol. 3168, pp. 1– 10. Springer, Heidelberg (2005) 2. Pan, W., Shen, Y., Zhang, W.: Grid-based Simulation System of Casting Solidification Process. In: Shen, W.-m., Lin, Z., Barthès, J.-P.A., Li, T.-Q. (eds.) CSCWD 2004. LNCS, vol. 3168, pp. 692–697. Springer, Heidelberg (2005)

394

C. Jiang et al.

3. Li, Y., Wang, G., Yang, S., Shi, M., Xu, J.: Research on Grid-based Cooperative Platform. In: Proceedings of the 9th International Conference on Computer Supported Cooperative Work in Design, Coventry, UK, pp. 378–383 (2005) 4. Liu, X., Jiang, C., Wang, C.: Grid-based Distributed Virtual Environment Simulation. Journal of Computer Engineering and Application 43, 110–112 (2007) (in Chinese with English Abstract) 5. Mustafee, N., Alstad, A., Larsen, B., Taylor, S.J.E., Ladbrook, J.: Grid-enabling FIRST: Speeding up simulation applications using WinGrid. In: Proceedings of the 10th IEEE International Symposium on Distributed Simulation and Real-Time Applications, Malaga, Spain, pp. 157–164 (2006) 6. Morillo, P., Orduna, J.M., Fernandez, M., Duato, J.: Improving the performance of distributed virtual environment systems. IEEE Transactions on Parallel and Distributed Systems 16, 637–649 (2005) 7. Wang, T., Wang, C., Lau, F.C.M.: An architecture to support scalable distributed virtual environment systems on grid. Journal of Supercomputing 36, 249–264 (2006) 8. Taylor, S.J.E., Fox, G., Fujimoto, R.: Revisiting Distributed Simulation and the Grid: A Panel. In: Proceedings of the 9th IEEE International Symposium on Distributed Simulation and Real-Time Applications, Montreal, Canada, p. 161 (2005) 9. Theodoropoulos, G., Zhang, Y., Chen, D., Minson, R., Turner, S.J., Cai, W., Xie, Y., Logan, B.: Large Scale Distributed Simulation on the Grid. In: Proceedings of the 6th IEEE International Symposium on Cluster Computing and the Grid DSGrid Workshop, Singapore, pp. 63–70 (2006) 10. IEEE Standard for Modeling and Simulation (M&S) High Level Architecture (HLA)Framework and Rules, IEEE Standard 1516-2000 (2000) 11. Chen, D., Turner, S.J., Cai, W.: A Framework for Robust HLA-based Distributed Simulations. In: Proceedings of the 20th Workshop on Principles of Advanced and Distributed Simulation, Singapore, pp. 183–192 (2006) 12. Lu, T., Lee, C., Hsia, W.: Supporting Large Scale Distributed Simulation Using HLA. ACM Transactions on Modeling and Computer Simulation 10, 268–294 (2000) 13. Foster, I.: Globus Toolkit Version 4: Software for Service-Oriented Systems. In: Jin, H., Reed, D., Jiang, W. (eds.) NPC 2005. LNCS, vol. 3779, pp. 2–13. Springer, Heidelberg (2005) 14. Lees, M., Logan, B., Theodoropoulos, G.: Distributed simulation of agent-based systems with HLA. ACM Transactions on Modeling and Computer Simulation 3, 1–25 (2007) 15. Cai, W., Turner, S.J., Zhao, H.: A Load Management System for Running HLA-based Distributed Simulations over the Grid. In: Proceedings of the 6th IEEE International Workshop on Distributed Simulation and Real-Time Applications, Fort Worth, TX, USA, pp. 7–14 (2002) 16. Morillo, P., Bierbaum, A., Hartling, P., Fernandez, M., Cruz-Neira, C.: Analyzing the performance of a cluster-based architecture for immersive visualization systems. Journal of Parallel and Distributed Computing 68, 221–234 (2008) 17. Kwok, M., Wong, J.W.: Scalability analysis of the hierarchical architecture for distributed virtual environments. IEEE Transactions on Parallel and Distributed Systems 19, 408–417 (2008) 18. Ta, D.N.B., Zhou, S.: A two-phase approach to interactivity enhancement for large-scale distributed virtual environments. Computer Networks: The International Journal of Computer and Telecommunications Networking 14, 4131–4152 (2007) 19. Ta, D.N.B., Zhou, S.: A network-centric approach to enhancing the interactivity of largescale distributed virtual environments. Computer Communications 29, 3553–3566 (2006)


395

20. Minson, R., Theodoropoulos, G.: An adaptive interest management scheme for distributed virtual environments. In: Proceedings of the 19th Workshop on Principles of Advanced and Distributed Simulation, Monterey, CA, USA, pp. 273–281 (2005) 21. Minson, R., Theodoropoulos, G.: Adaptive interest management via push-pull algorithms. In: Proceedings of the 10th IEEE International Symposium on Distributed Simulation and Real-Time Applications, Malaga, Spain, pp. 119–126 (2006) 22. Gu, Y., Boukerche, A., Araujo, R.B.: Performance analysis of an adaptive dynamic gridbased approach to data distribution management. Journal of Parallel and Distributed Computing 68, 536–547 (2008) 23. Liu, X., Jiang, C., Wang, C.: Massive Data Management Framework for Grid-based Distributed Virtual Environment Simulation. Journal of Computer Engineering and Design 29, 931–933 (2008) (in Chinese with English Abstract) 24. Zhou, S., Cai, W., Lee, B.S., Turner, S.J.: Time-space consistency in large-scale distributed virtual environments. ACM Transactions on Modeling and Computer Simulation 1, 31–47 (2004) 25. Lui, J., Chan, M.: An efficient partitioning algorithm for distributed virtual environment systems. IEEE Transaction on Parallel and Distributed Systems 13, 193–211 (2002) 26. Boukerche, A., McGraw, N.J., Araujo, R.B.: A Grid-Filtered Region-Based approach to support synchronization in large-scale distributed interactive virtual environments. In: Proceedings of the 34th International Conference on Parallel Processing Workshops, Oslo, Norway, pp. 525–530 (2005) 27. Morillo, P., Orduna, J.M., Duato, J.: A scalable synchronization technique for distributed virtual environments based on networked-server architectures. In: Proceedings of the 35th International Conference on Parallel Processing Workshops, Columbus, Ohio, USA, pp. 74–81 (2006)