Extending Wide-Area Replication Support with Mobility and Improved Recovery Hendrik Decker, Luis Ir´un-Briz, Francisco Castro-Compay, F´elix Garc´ıa-Neiva, Francesc D. Mu˜noz-Esco´ı Email: hendrik, lirun, fcastro, fgarcia, fmunyoz @iti.es
Instituto Tecnologico de Informatica Universidad Polit´ecnica de Valencia 46022 Valencia, SPAIN Email:
[email protected]
Abstract. We survey the MADIS architecture. It supports high availability, fault tolerance and seamless error recovery in computer networks for peer-to-peer computing and other distributed web services. MADIS is open for plugging in different protocols for replication, resilience and recuperation. We recapitulate two protocols which ensure fault tolerance and efficient error recovery. We outline ongoing research which aims at improving fault tolerance and error recovery by a dynamic adaptation of protocols plugged into MADIS. Future development of MADIS envisages an extension with functionality supporting distributed webbased computation and communication also for mobile users, possibly including an integration of voice and data traffic over a single internet link.
1 Introduction Peer-to-peer networks as well as web services in general usually reply on one or several underlying databases. Such databases either serve to manage application-specific information, e.g., stored or screened versions of ongoing and previous session data, or more general knowledge provided by third parties, e.g., geographical information systems. Typical examples of such web services are web-based communication and collaboration, digital libraries, web-based information systems, etc. Failure resilience, fault tolerance and seamless error recovery of the databases underlying such web services are key factors for their performance as well as their user acceptance. A tried and tested principle for supporting availability, fault tolerance and transparent error recovery is distributed replication [9] [25]. This principle has been adopted by the middleware MADIS [18]. Traditionally, fault tolerance and transparent error recovery is provided by redundant hardware and a software layer which manages a seamless failover from a failing hardware component to a pre-positioned substitute which then is able to continue to function and perform in place of the defunct component. Hardware redundancy works well in local-area networks, such as ethernets or wireless local-area networks.
This work has been partially supported by the Spanish MCyT grant TIC2003-09420-C02-01.
With MADIS, high availability, fault tolerance and error recovery have been scaled up for communication and computing networks distributed over wider areas, based on the TCP/IP or UDP/IP protocol suites. Stored data items of a given web service are consistently replicated over all network nodes, so that the data and services it provides become more available, more fault tolerant and more efficient with recovering from failures. In section 2, we outline the MADIS architecture. In section 3, we recapitulate several different protocols and outline how a seamless error recovery in data communication networks can be further optimized by a dynamic adaptation of replication strategies. Section 4 addresses future extensions of MADIS with mobility features. In section 5, we conclude.
2 MADIS MADIS [18] is a more efficient, JDBC-based follow-up of the middleware COPLA [16] [7], developed in the GlobData project [8] [24]. A key feature inherited from its antecedent is that MADIS is not only a system for supporting replication and failure resilience of data communication networks, but also a platform for developing such support systems for specific applications. Adaptivity of MADIS is achieved via pluggable protocols. They can be chosen and plugged into the middleware according to service-specific requirements, different network properties and various attributes of terminal devices and user profiles. The use of pluggable protocols in MADIS for web-based information systems and voice/dataintegated collaboration has been described in [2] [1] [6]. Already the predecessor COPLA of MADIS offered a considerable flexibility of adapting its replication management to different degrees of availability [22]. MADIS can be tuned so as to satisfy lower or higher degrees of availability and fault tolerance, which typically is traded off against increasing or relaxing the requirements on the consistency of replicas, and vice-versa. The essential advances of MADIS over COPLA are the following. Firstly, the latter provided mappings to fully object-oriented user interfaces, which resulted in “fat” code and large performance overheads for underlying databases of relational kind. As opposed to that, MADIS is much thinner. It complies with the standard relational database paradigm, using just JDBC, instead of cumbersome combinations of JDBC and CORBA. Secondly, the adaptation of protocols was focused on the tradeoff between consistency and availability. Additionally, MADIS allows to trade off replication consistency also against fault tolerance. Thirdly, MADIS provides much more flexibility for choosing and swapping suitable protocols than COPLA. MADIS offers a range of different pluggable protocols (synchronous, asynchronous, hybrid, with eager or lazy replication, different relaxations of the ACID properties, etc), so as to satisfy lower or higher degrees of availability and fault tolerance, which is traded off against increased or relaxed requirements of replica consistency. By storing complete sets of metadata for different protocols, it is possible
to swap them not only at well-defined break-points (e.g., at the end of sessions), but even while the system is running. Future development of MADIS envisages an extension of supporting the high availability of web services also for mobile users. An extension for wireless communication services, involving an integration of voice and data traffic over a single internet link, is addressed in section 4. 2.1 The MADIS Architecture MADIS is structured by three main layers, as shown in fig. 1. On top, the users layer is the interface for developing and using web services, while the core management layer supports their distributed operation. This middle layer includes the transparent management of replica, precautionary measures to prevent any non-availability of data and services, and the seamless recovery of broken nodes. On the bottom, the storage layer provides access to the underlying database, using triggers via JDBC. Also the interaction between users and core management layer goes through JDBC, using the Java Remote Method Invocation mechanism. Both interfaces are described in [17]. The MADIS core management is described in the following subsections.
Developers
End Users
Users API
Developers Library JDBC + RMI
Core Management Replication Protocol API
Replication Protocol
Core Management Transport Replication Protocol
JDBC + Triggers
Database
Fig. 1. MADIS Layered Architecture
2.2 The Core Management Layer The core management of MADIS has a fixed and an exchangeable part. The fixed part manages transactions, including the delegation of requests to the protocols module,
which constitutes the exchangeable part. It is exchangeable in the sense that protocols for communication, replication and recovery can be plugged in and out, so as to adapt the system’s behaviour to the needs of given applications, network loads and user requirements. Thus, a network administrator can always choose and plug in suitable protocols, in order to adapt the system conveniently. Maintaining consistency of replicated data and restoring it upon recovery, the protocols module communicate with the core management modules at the other network nodes, by means of message passing. Essentially, the duties of these modules are, to – – – – – – –
cater for requests by users and application programs, update local data, disseminate a session’s updates to other managers, monitor group membership, keep track of consistency of stored data, support recovery of broken nodes, decide when to commit or abort transactions.
Application Application
.
Application
CORE MNGR. 1
DB
Protocol
.
Application Application
Protocol CORE MNGR. 2
DB
Protocol
CORE MNGR. .3
Application
DB
Application
Application
Fig. 2. Networked MADIS Core Managers
In MADIS, protocols specifically tailored to the needs of particular applications can be selected at convenience. For instance, eager or lazy protocols, attuned to eager or lazy update policies (cf. [27] [3]) can be used. Examples of protocols have been discussed, e.g., in [23] [26] [15]. The appropriate choice of suitable protocols depends on the applications, their use cases (e.g., casual or business), of given network properties (e.g., mobile or fixed nets, the current load, peculiarities of providers, bearers, backbones and last miles, etc) and user profiles.
2.3 Peer-to-Peer and Client-Server Configurations of MADIS DIRECS can be used either in peer-to-peer or client/server scenarios. “Peer-to-peer” here means that data are replicated transparently at the site of each node. Note that each node can be either client or server to each other node, in dynamically interchangeable roles. Typically, using MADIS in a client/server scenario means that some or all users have access to a virtual database server which does not necessarily belong to the site of any node. Rather, it would be a (public or commercial) third party’s repository. Thus, MADIS users who have subscribed to the services of such databases behave as clients of a virtual server which actually is a network of replicas. Client/server and P2P scenarios can be combined orthogonally. That is, users of one or several external MADIS-supported databases may at the same time use replicas of one or several internal databases that are made highly available and failure resilient by another installation of MADIS. Then, each installation can (and should) be independently tuned according to their own particular requirements. That is, different protocols can be plugged into different installations of MADIS that are running simultaneously.
"INTERNAL" DATABASE USER 2.1
USER 1.1
MADIS
USER 1.2
MADIS Replica I1
Replica I2 USER 2.2
USER 1.3
Replica I3
MADIS USER 3.3
USER 3.1 USER 3.2
MADIS
"EXTERNAL" DATABASE
MADIS
MADIS
Replica E1
Replica E2
Replica E3
Fig. 3. Users as clients and peers of MADIS
For example, in a web-based collaboration service, session participants may consult an external distributed database containing time tables or routes of public transport for arranging a meeting, and they may also seek agreement on which documents to bring to the meeting, by retrieving related data in an internal database. Now, suppose that both the external and the internal database are replicated using MADIS, as illustrated in fig. 3 (the large cylinders suggests that users perceive distributed replicas, represented by small cylinders, as a single system). Then, the external database is probably best served by plugging in a protocol that guarantees a very high degree of consistency, since the provided data are supposed to be accurate at all times. And since it will rarely be written while being read frequently, it will not be a high performance burden to memorize at each node an incremental history of all updates, such that recovery will be very fast after a node’s failure, because hardly any additional net communication will be needed for quickly re-installing the current state. Recovery protocols which pursue such a recovery-preventive strategy are described in section 3. On the other hand, consistency requirements for the internal database can be more relaxed. This is because it neither really matters nor is it possible at all that the latest version of shared collaboration data are always immediately replicated and displayed on each screen at the very same time. Rather, delays of up to a some seconds are usually tolerable in web-based data communication for collaborative dialogues. Hence, protocols used in the internal database may take advantage of such delays by allowing to slightly defer activities for achieving consistency of all replica. Relaxation of consistency can then be conveniently traded off in favour of other virtues of the protocols. For instance, a higher precaution can be taken in terms of failure prevention and seamless error recovery, which improves the availability of shared data as well as their recovery in case of a network breakdown or a node’s failure. Again, we refer to section 3 for details on how protocols may invest in such preventive measures.
3 Speeding up Recovery by Swapping Protocols For web services the replicas of which are distributed over wide areas, error recovery is more complicated than in local-area clusters, due to the geographical distance and the instability of links between nodes. Clearly, instability further increases in mobile networks. Hence, particular care has to be taken for providing a seamless recovery from failures in a wide-area mobile network. In general, the idea of replication is that, in case of a node’s failure or breakdown, some other replica can be charged with the load of the broken node while the latter is recovering. In wide-area networks, however, recovery entails not just local, but additional system-wide costs of data processing and re-transmission of transactions that have been missed by the broken nodes. This is further aggravated in failure-prone mobile networks. For minimising the amount of additional network communication which is necessary for recovery, we have developed a uniform total order communication protocol which uses a reliable multicast algorithm. Uniform means that the protocol ensures that all network nodes (not just one) are always up-to-date with regard to committed transactions. Moreover, it ensures that each node not only receives, but also maintains locally
a record of all delivered messages on hard disc. That way, even after a total breakdown, broken nodes can largely recover themselves from their own memory, such that only the recent incremental changes that have been missed during its downtime need to be re-transmitted for recovery, instead of possibly huge amounts of transactions or even entire database states. Conformant with the general philosophy of protocol adaptiveness in MADIS, this uniform total-order communication protocol can be parametrised with different kinds of recovery protocols. For instance, CLOB (Configurable LOgging for Broadcast protocols)[4] is a suite of broadcast protocols, used as a basis for database replication. CLOB allows for an easy combination of automatic log-based and protocol-specific versionbased recovery techniques. Such a combination provides recovery solutions with optimal performance, but requires some additional storage for maintaining both the log of missed messages and the version-based information needed for recovery. However, version-based information is only a little percentage of the usual log-based one, as has been shown in several examples [11,23], and the additional storage is easily compensated by the obtained performance gains. In [4], we describe in more detail how CLOB aims at managing the logging of missed messages in the broadcast protocol core. It provides automatic recovery in shortterm failures, but discards the log and accordingly notifies the database replication protocol modules in case of long-term outages. This kind of support can be easily combined with version-based recovery protocols, as follows. Once a failure is detected, the replication protocol uses a traditional version-based management strategy for recovery, but it will be discarded if the replica is able to rejoin the system soon. In this case, CLOB automatically propagates the missed update messages to the recovering replica, which then receives and applies them, thus avoiding any delays both in the source and destination replicas. If, on the other hand, the outage period exceeds a given threshold, the reliable broadcast service will notify the replication protocol about that, discard the message logs maintained by CLOB and delegate the recovery management to the upper-layer components. For dealing with longer-term outages, MADIS offers a recovery protocol called FOBr[5], which uses node identifiers for deploying a concept of object ownership. The key of load balancing then is that each node contributes with a subset of the whole set of objects to be recovered, so that it is free to attend other user requests. The use of node identifiers for identifying object versions imposes very low requirements of additional storage space for keeping track of objects missed during outages. Thus, only the last version of an object needss to be migrated, as oppossed to techniques where complete transaction logs are maintained and each object is transferred as many times as updates had been applied to it. This obviously is very advantageous for wide area networks, since network traffic is significantly reduced. Current research is concerned with finding optimal trade-off points for CLOB and FOBr on an axis which quantifies outage intervals. By a complementary combination of runtime experiments and analytical methods[10], we are trying to find out for which intervals the outage time is short enough so that a CLOB protocol should be used, and from which interval length onwards it turns out that using FOBr is more advantageous, for speeding up recovery.
4 Enhancing MADIS with mobility features (an outlook) We are targeting an extension of MADIS for supporting mobility. New protocols will have to be designed and implemented for taking into account an increased volatility of mobile data, a higher risk of network failures, and limited capacities of communication channels and devices. Additionally, we envisage the incorporation of multimedia communication, at a later stage. In the following subsection, we revisit a web-based collaboration project called MOVE, which supports both mobility and multimedia collaboration. Since the architectures of MOVE and MADIS are fairly distinct and orthogonal, an integration of both is conceivable.
4.1 MOVE The main objective of the project MOVE (Mobile Middleware with Voice-Enabled Services)[12] [13] was the integration of voice (real-time data) and multimedia services (real-time and non-real-time data) for mobile GSM-, UMTS- and Web-based collaboration [14]. MOVE is a middleware platform which supports the initiation, maintenance and termination of mobile voice-enabled collaborative multimedia sessions, connecting to each session participant via a single, possibly wireless web link, while simultaneously, a dynamic quality adjustment of integrated voice/data streams is provided. Similar to MADIS, the MOVE middleware also provides location transparency of network nodes. As a demonstration scenario, a collaborative call center application involving webbased VoIP phone conferencing and a hotel-booking service for mobile users, has been developed. It was shown to wide acclaim at the 1999 edition of ITU Telecom in Geneva. That exhibit marked the first time in the history of IP-based telecommunication that a seamless handoff between HSCSD-, GSM- and DECT-based mobile network base stations could be demonstrated in the kbit/s range; handoffs between Wave-LAN base stations even performed in the Mbit/s range. Figure 4 shows that, along with the mobile user and the call centre, third parties such as a web-based information system or any security service may be involved. The icons besides the screens of user and call center agents indicate that MOVE is not only independent of different browser technologies, but that it also adapts to the peculiarities of some of the more prominent web browsers on the market. In MOVE, the streaming of voice, video and webcam data meet established transmission quality standards. Voice and other real-time data streams are transmitted with higher priority, relative to non-real-time data. If, in spite of clever coding and sophisticated compression, the available bandwidth or other resources would otherwise not allow a timely transmission, then visual data are downsized suitably. Typically, attributes of secondary importance are then scaled down. For instance, fine-grained colour palettes may be coarsened, or mapped to gray tone scales, or to black-and-white binaries. Special attention is paid to the monitoring and adaptation of heterogeneous media streams, concerning their adjustment to a wide range of quality-of-service parameters.
Fig. 4. MOVE collaboration scenario
4.2 Adaptivity in MOVE and MADIS Adaptivity in web-based collaboration systems usually refers to one of two distinguished sets of issues. First, adaptivity caters for bridging differences caused by the geographic distance between network nodes and session participants, such as time zone, language, currency, cultural issues, etc. Second, adaptivity refers to a middlewaresupported adjustment to (or overruling of) technical idiosyncrasies such as bandwidth limitations, device properties, mobility, user profiles and locations, transmission delays and jitters, vendor-specific hardware and software properties, as well as different types, capacities, carriers, bearers and backbones of various network services. Neither MOVE nor MADIS is concerned with the first kind of adaptivity, which consequently will not be further addressed. The MOVE adaptivity is of the second kind, as documented in [14] and the project’s final report [19]. In addition to its seamless adaptivity as characterized above, MOVE also offers a patented, dynamically adaptive quality-of-service support [21] [20] for the integration of real-time voice and non-real-time data communication over a single internet link, which responds to essentially the same parameters as those determined by the aforementioned network-, location-, vendor- and user-related technicalities. The MADIS adaptivity does not fall into any of the two categories mentioned above. Although it does offer adaptivity to network load, bandwidth limitations and other static and dynamic properties of web-based collaboration sessions, it does not do so by by means of any web service. Rather, the adaptation of MADIS functionality to the requirements of collaborative sessions is realized, first by choosing, then by monitoring and then suitably exchanging pluggable networking protocols, if the monitoring results advise such an exchange. This is an administrator’s job, which so far has not been automated, nor built into the middleware as an autonomous functionality. In some more detail, the MIDAS adaptivity is recapitulated in the conclusion.
5 Conclusion We have surveyed the middleware MADIS, for supporting an adaptive wide-area replication of distributed data, and thereby the fault tolerance and seamless error recovery of web-based communication networks and services. Moreover, we have outlined an extension of MADIS with functionality as provided by the the middleware MOVE, which supports adaptive voice/data-integrated mobile web-based collaboration services. We have emphasised the adaptivity of both, and contemplated an integration of their respective functionalities into a single web-based collaboration platform which supports mobility. Adaptivity in communication networks usually refers to one of two distinguished sets of issues. First, adaptivity caters for bridging differences caused by the geographic distance between network nodes and session participants, such as time zone, language, currency, cultural issues, etc. Second, adaptivity refers to a middleware-supported adjustment to (or overruling of) technical idiosyncrasies such as bandwidth limitations, device properties, mobility, user profiles and locations, transmission delays and jitters, vendor-specific hardware and software properties, as well as different types, capacities, carriers, bearers and backbones of various network services. However, the MADIS adaptivity does not fall into any of the two categories mentioned above. Although it does offer adaptivity to network load, bandwidth limitations and other static and dynamic properties of web-based collaboration sessions, it does not do so by by means of any web service. Rather, the MADIS adaptivity consists in the possibility to choose among a range of dedicated protocols, which may favour either lazy or eager update propagation, optimistic or pessimistic concurrency control, and may opt for either a primary-copy or an update-everywhere policy. According to the fine-tuning of such parameters, more or less strong requirements of consistency, availability and responsiveness can be guaranteed. Adaptation to the requirements of network communication is realized, first by choosing, then by monitoring and then suitably swapping pluggable networking protocols, if monitoring results advise such a swap. This is an administrator’s job. Its partial automation and integration into the middleware as an autonomous functionality is envisaged in the further development of MADIS. Currently, we are developing new protocols for supporting an even higher availability and failure resilience than what can so far be guaranteed already by MADIS. The price to pay for that are further relaxations of consistency guarantees, which however are feasible, due to the interactive nature of most web services. A higher performance penalty also has to be paid for additional precautionary measures of memorizing transaction data, in order to enable a seamless recovery from network failures. However, these additional costs are justified by the fact that network failures are much more likely to happen in wide-area mobile networks than in hardwired local-area clusters. Thus, the investment made at times when network nodes operate without faults pays off whenever those nodes are recovering from brakdowns without the need of re-transmissions of huge amounts of data. An intended future enhancement of MADIS with mobile integrated voice/data features is expected to be particularly beneficial with respect to the demands of web-based communication with small devices via mobile links, as well as for potentially huge volumes of streaming data. In general, different kinds of multimedia data and devices have
quite different requirements with regard to timeliness, bandwidth demand and suitability for replication. For instance, voice tends to be elusive and ephemeral, and since it is rather “for the moment” than for the record, it is likely not to be stored, let alone replicated. Hence, the flexibility of MADIS to choose well-suited protocols for satisfying particular requirements will be especially welcome.
References 1. J. Bataller, H. Decker, L. Ir´un, F. Mu˜noz: Replication for Web-Based Collaboration. Proc. 4th Int’l Workshop on Web Based Collaboration, 15th DEXA, IEEE Computer Society, 2004. 2. J. Bataller, H. Decker, L. Ir´un, F. Mu˜noz: A Distributed Web Information Systems Platform Supporting High Responsiveness and Fault Tolerance. Proc. CAiSE Workshops, Vol. 1, 7991, Riga Tech. Univ., 2004. 3. Y. Breitbart, R. Komondoor, R. Rastogi, S. Seshadri, A. Silberschatz: Update propagation protocols for replicated databases. SIGMOD Record 28(2):97-108, 1999. 4. F. Castro, J. Esparza, M. Ruiz, L. Ir´un, H. Decker, F. Mu˜noz: CLOB: Communication Support for Efficient Replicated Database Recovery. To appear in Proc. 13th Euromicro Conference on Parallel, Distributed and Network-based Processing, 2005. 5. F. Castro, L. Ir´un, F. Garc´ıa, F. Mu˜noz: FOBr: a version-based recovery protocol for replicated databases. To appear in Proc. 13th Euromicro Conference on Parallel, Distributed and Network-based Processing, 2005. 6. H. Decker, F. Mu˜noz, L. Ir´un, J. Bataller, P. Gald´amez, R. Garc´ıa: Augmenting Web-based Collaboration with Adaptive Replication and Mobility. To appear in M. Matera, S. Comai (eds): Engineering Advanced Web Applications, Rinton Press, Dec. 2004. 7. H. Decker, F. Mu˜noz, L. Ir´un, P. Castro, A. Calero, J. Esparza, J. Bataller, P. Gald´amez, J. Bernab´eu: Enhancing the Availability of Networked Database Services by Replication and Consistency Maintenance. Proc. Parallel and Distributed Databases, 14th DEXA, 531-535. IEEE Computer Society, 2003. 8. http://globdata.iti.es 9. R. Guerraoui, A. Schiper: Fault-Tolerance by Replication in Distributed Systems. Proc. Int’l Conf. Reliable Software Technologies, 38-57. Springer LNCS 1088, 1996. 10. L. Ir´un, F. Castro, H. Decker, F. Mu˜noz: An Analytical Design of a Practical Replication Protocol for Distribute d Systems. Applying Formal Methods: Testing, Performance and M/ECommerce (Pro c. FORTE Workshops), 248-261. Springer LNCS 3236, 2004. 11. L. Ir´un, F. Castro, F. Garc´ıa, A. Calero, F. Mu˜noz: Lazy recovery in a hybrid database replication protocol. Proc. XII Jornadas de Concurrencia y Sistemas Distribuidos, 2004. 12. http://www.cordis.lu/infowin/acts/rus/projects/ac343.htm 13. M. Wallbaum, D. Carrega, M. Krautg¨artner, H. Decker: A Mobile Middleware Component Providing Voice over IP Services to Mobile Users. Proc. ECMAST ’99, 552-563. Springer LNCS Vol. 1629, 1999. 14. D. Carrega, H. Muyal, H. Decker, M. Wallbaum: Integrating voice and data services for mobile internet collaboration with the MOVE middleware architecture. Proc. 1st Int’l Workshop on Web Based Collaboration, 12th DEXA, 532-536. IEEE Computer Society, 2001. 15. L. Ir´un, F. Mu˜noz, J. Bernab´eu: An Improved Optimistic and Fault-Tolerant Replication Protocol. Proc. 3rd Int’l Workshop on Databases in Networked Information Systems, 188200. Springer LNCS 2822, 2003. 16. L. Ir´un, F. Mu˜noz, H. Decker, J. Bernab´eu: COPLA: A Platform for Eager and Lazy Replication in Networked Databases. Proc. 5th ICEIS, Vol. 1, 273-278, 2003.
17. L. Ir´un, R. de Juan-Mar´ın, F. Castro, H. Decker, F. Mu˜noz: Dealing with Readsets and Writesets in the MADIS Architecture. submitted, 2004. 18. http://www.iti.upv.es/madis 19. ftp://ftp.cordis.lu/pub/infowin/docs/fr-343.pdf 20. M. Wallbaum, H. Decker: Enabling Quality-of-Service Management for Voice/DataIntegrated Mobile Communication. Proc. ICDCS 2000 Workshops, section D: Wireless Networks and Mobile Computing, 87-93. IEEE Computer Society, 2000. 21. H. Decker, M. Krautg¨artner: Flexible Quality-of-Service Technology for Supporting Voice/Data-Integrated Nomadic Networking. Flexible Working – New Network Technologies, 215-224. IOS Press, 1999. 22. F. Mu˜noz, L. Ir´un, J. Bataller, P. Gald´amez, J. Bernab´eu, J. Bataller, C. Ba˜nuls, H. Decker: Flexible Management of Consistency and Availability of Networked Data Replications. Proc. FQAS’02, 289-300, Springer LNCS 2522, 2002. 23. F. Mu˜noz, L. Ir´un, P. Gald´amez, J. Bernab´eu, J. Bataller, M.C. Ba˜nuls: GlobData: Consistency Protocols for Replicated Databases. Proc. YUFORIC’01, 97-104. IEEE Computer Society, 2001. 24. F. Mu˜noz, L. Ir´un, P. Gald´amez, H. Decker, J. Bernab´eu, J. Bataller, C. Ba˜nuls: GlobData: A Platform for Supporting Multiple Consistency Modes. Proc. Information Systems and Databases, 104-109, Acta Press, 2002. ¨ 25. T. Ozsu, P. Valduriez: Principles of Distributed Database Systems, 2nd Edition. Prentice Hall, 1999. 26. L. Rodrigues, H. Miranda, R. Almeida, J. Martins, P. Vicente: The GlobData Fault-Tolerant Replicated Distributed Object Database. Proc. EurAsia-ICT, 426-433. Springer LNCS 2510, 2002. 27. M. Wiesmann, A. Schiper, F. Pedone, B. Kemme, G. Alonso: Database replication techniques: A three parameter classification. Proc. 19th Sympos. Reliable Distributed Systems, 206-217. IEEE Computer Society, 2000.
