An Improved Optimistic and Fault-Tolerant ... - Semantic Scholar

An Improved Optimistic and Fault-Tolerant Replication Protocol Luis Ir´un-Briz, Francesc D. Mu˜noz-Esco´ı, Josep M. Bernab´eu-Aub´an Universidad Polit´ecnica de Valencia - 46071 Valencia, SPAIN Email: lirun,fmunyoz,josep @iti.upv.es




Instituto Tecnol´ogico de Inform´atica

Abstract. In this paper, a protocol is proposed that provides the advantages of lazy approaches, forestalling their traditionally found disadvantages. Thus, our approach reduces the abortion rates, and improves the performance of the system. It can also use a dynamic computation of the protocol threshold, approximating its results to the optimal ones. In addition, fault tolerance has been included in the algorithm, using a pseudo-optimistic approach, and avoiding to block any local activity, and minimizing the interference over any node in the system. A complete description of these algorithms is presented here. Finally, and empirical validation is also discused.

Key words:

distributed databases, concurrency control, database replication



This work has been partially supported by the EU grant IST-1999-20997 and the Spanish grant TIC99-0280-C02-01.

1 Introduction Consistency requirements introduced a new parameter in the problem of replication in order to achieve fault tolerance and performance enhancement in distributed systems. In some areas -as distributed databases- this conforms a main problem to solve. The concurrency control can be solved[WSP 00,KR81,EGLT76] using different techniques. Pessimistic techniques use ”locks“ to avoid conflicting transactions to break the consistency, and optimistic techniques are based on timestamps. The main disadvantage of pessimistic techniques is the degradation of the performance, due to the temporal block of the concurrent transactions. Its main advantage is the low abortion rate the pessimistic techniques introduces. Moreover, the optimism principle is also applied in the propagation of the updates in models where a particular transaction is not executed in every node in the system. In such systems, a notification of the updates performed by a transaction must be propagated to the rest of nodes. The pessimistic propagation is commonly known as Eager, and performs the entire diffusion of the updates during the commit phase, making the commit phase more expensive, but ensuring that every replica in the system holds an updated value for its objects. In the other hand, optimistic propagation[Tho79] -described in the literature as Lazy update protocols[BK97,HAA99]-, delays the update propagation over the commit phase. This makes the commit phase faster, but complicates the consistency control, and increases the abortion rate, due to the accesses to outdated objects. The inconvenience of lazy and optimistic approaches can be attenuated if the probability of an object to be outdated is included in a more careful design of the concurrency protocol. Moreover, fault tolerance can also be benefitted from this approach, including it in a modification of the basic algorithm. Many recovery protocols use eager pessimistic approaches, making it necessary to temporally block transactions alive during the recovery process. In the other hand, lazy optimistic approaches for recovery protocols increase the abortion rate in the node being recovered. In the middle point, our proposal avoids to block transactions, and minimizes the abortion rate, using the principle of role migration. We provide in this paper an statistical exploit of the expression for the abortion rate caused by the use of lazy replication, describing a new algorithm to solve the concurrency control in a distributed transactional system (e.g. distributed databases). Moreover, this algorithm makes use of the lazy paradigm, and maintains the optimistic approach in a certain degree. In addition, we provide fault-tolerance to the protocol, making use of the same principle, avoiding to block any transaction in the system during the recovery phase. To this end, the following section describes a basic optimistic lazy protocol for the concurrency control. Section 3 presents a modification of the basic algorithm, introducing a prediction of the probability of outdate in order to improve the behavior of the abortion rate, and an experimental validation of the expression is then shown. In section 4, the modification to provide self-recovery ability to the protocol is detailed. Section 5 compares this protocol with other systems. Finally, section 6 gives some conclusions.

2 An Optimistic Lazy Protocol The basic consistency protocol presented in this section is a variation of one of the consistency protocols currently implemented in the GlobData Project[Ins02] (described in [MIG 02]). In the GlobData project, a software platform is used to support database replication. This platform, is called COPLA, and provides an object-oriented view of a network of relational DBMSs. The COPLA architecture is structured in three different layers, which interact using CORBA interfaces, enabling the placement of each layer in a different machine. Thus, multiple applications (running in different nodes) may access the same database replica using its local ”COPLA Manager”. The COPLA Manager of each node propagates all updates locally applied to other database replicas using its replication management components. One of the problems solved by COPLA is to ensure the consistency among transactions being executed in different database replicas. The solution to this problem depends mainly on the update propagation model: eager or lazy. COPLA is flexible enough to allow multiple update approaches, each one being managed by a dedicated consistency protocol. A particular COPLA component implements the consistency protocol. Multiple implementations of its interfaces are possible, thus allowing that a GlobData system could change its consistency protocol at will. So, the COPLA architecture can be used to experiment with different consistency protocols. The basic optimistic Lazy Protocol implemented in GlobData is named ”Lazy Object Multicast Protocol” (LOMP). It will be described in 2.2; further clarifications are presented in the subsections thereafter. 2.1 Node Roles Considering a given session that tries to commit, the nodes involved in its execution may have four different roles: – Active node. The node where the COPLA Manager that has served the session’s execution is placed. – Owner node. For a concrete object, the node where this object was created. During the consensus process performed at commit time of a session, the owner of an object will be asked to allow this session to complete the commit. Thus, it is the manager for the access confirmation requests sent by the active nodes at commit time. The management of these access confirmation requests is similar to lock management, but at commit time. These requests are detailed in section 2.2. We will denote that a node owns an object with the expression . – Synchronous nodes. If one of our goals is fault tolerance, it becomes necessary a set of nodes that provides guarantees about the last version written for a certain object. So, for each session that is committing, the owner of each written object must multicast this update to a set of synchronous nodes, within the commit phase. We will denote that a node is a synchronous replica of an object with the expression .




 

 


  











– Asynchronous nodes. For an object, all the other nodes that have a COPLA Manager replicating the database. In these nodes, the session updates will be eventually received. We will denote that a node is an asynchronous replica of an object with the expression . Note that:




    

      
     



2.2 Protocol As described above, the GlobData-LOMP consistency protocol multicasts object updates to all synchronous nodes when a session is allowed to commit. Consistency conflicts among sessions are resolved with an optimistic approach, using object versions. To this end, the protocol uses some meta-data tables in the database where the current object versions can be stored. In summary, it processes the following steps: 1. There is no consistency control when a session performs an access to an object. 2. At commit time, the consistency control performs a voting round for every owner node of the accessed objects of the session. 3. If all the owner nodes conforms this commit, the session can continue, and completes the commit. Then, an update message is multicasted to every synchronous replica for each updated object. 4. If any of the owner nodes replies with a denial vote, then the conflicting objects are updated (the adequate value is included in the denial message), and the session is aborted. 5. An asynchronous process is run in the system in order to multicast to every asynchronous replica the updated objects in each node. The full description of the protocol process can be found in [IMDBA03].

3 COLUP: The Cautious Optimistic Lazy Update Protocol



An expression for the probability for an object to cause the abortion of a session has been presented in [IMDBA03]. This expression can be used to predict the convenience for a session to ensure that an object that is asynchronously updated has a recent version in the local database. The complete expression can be written as follows:




    
 
#! "    #$&%('

(1)

Where: – –



is the number of nodes in the system. is the amount of objects maintained by the system (i.e. the distributed database).

–




is the number, in mean, of objects written per write-transaction in the node of the system. – is the number of write-transactions committed per second in the node of the system. – the elapsed time the object has not been updated in the node where the session accessing the object is executed.

 &$ %('
 



"




 $&% '


The expression can be calculated with a few parameters. Only and must be collected in the nodes of the system in order to obtain the expression. Thus, it becomes possible for a node to estimate the convenience for an object to be locally updated before being accessed by a session. This estimation will be performed with a certain deand the node gree of accuracy, depending on the “freshness” of the values of has. In order to apply these results, it becomes necessary to establish a threshold of to consider the object “convenient to be updated”. An adequate value for this threshold should minimize the number of abortions caused by accesses to outdated objects, and keeping low the number of updates for the system. The higher the threshold is, the less number of abortions will occur in the system, but the higher updates will be done, and a higher overhead will be introduced in the system. This threshold can be approximated at run time, dealing with a behavior near to the optimum. To implement this principle a new request message is required. Now, the active node for a session sends “update requests” to the owners of the accesses objects, in order to get the updated versions for such objects. This update request message can be sent along the session execution, in order to update (in a certain degree) the objects accessed by the session. This technique reduces the probability of abortion caused by the accesses to outdated objects within the sessions.




 $&%('



 

3.1 Modification of the LOM Protocol The way the COPLA manager decides to send or not an update request for an object that is about to be accessed is quite simple:



1. During the life of a session, an access to an object is requested by the application. This access is intercepted by the COPLA manager of the active node, and then the probability of being outdated is calculated for the object . If the active node is a synchronous (or owner) node for , this probability is 0. This is because the protocol ensures that incoming updates of the object will abort every transaction conflicting with the updated object. If the active node is an asynchronous node for , then the COPLA manager will use the expression of . To perform this calculation, it is needed : the time elapsed from the last local update of . 2. If this probability exceeds a certain threshold
 , then the COPLA manager sends the update request to the owner of . If the threshold is not reached, the protocol continues as described in section 2.2. This value can be adjusted at run time, in order to obtain results near to the optimum.








 





"

 

3. In other case, after the COPLA manager allows the session to continue, it waits for the update response. This response indicates whether the local version of is updated, or outdated (then, the response contains the newest version for the object). If the local version must be updated, then the update is applied in the local database, and the update time is also written down. 4. Once the COPLA manager has ensured that is updated, the required access to the object can continue.





By forcing to update an object before the session accesses it, the COPLA manager decreases the value of , to the length of the session. Thus, the chance for a session to success the commit phase is higher. This technique implies that every update performed in the asynchronous nodes of an object must include a timestamp of the instant the update is performed at. Note that it is not necessary to use global time, but it is only necessary the local time for each node.

"



3.2 Validation and Accuracy of the Improvement We have validated the expression used in the protocol, by implementing a simulation of the algorithm. In this simulation, we have implemented nodes that concurrently serve sessions, accessing to different objects of a distributed database. A complete description of this simulation and its results can be found in [IMDBA03]. The improvement provided by our approach has been empirically observed, both using a simulation, and a real implementation. The results of these experiments have shown that the COLU protocol provides an increase in the performance of the system, keeping the abortion rate below reasonable levels. The main effect of the overhead of the COLU protocol can be found in the “LocalUpdate” messages. The more accurate these messages are, the more efficient the improvement is. When the expression exceeds the established threshold for an object, and an update request is sent, it is possible for the response of this request to contain the same version for the requested object (e.g. when the object was, in fact, up to date). We name this situation “Inaccurate prediction”. This accuracy depends on the expression used to evaluate the convenience of the LocalUpdate, and on the established threshold in the algorithm. Higher values for the threshold should provide more accurate predictions. In the other hand, higher values of the threshold makes it possible to let outdated values for objects. This is determined by the completeness of the prediction function: the correlation between the prediction and the real world will avoid accesses to outdated objects. In the figure 1(a) we show the correlation between the prediction and the real abortion rate. The proximity of the predicted abortion rate to the experimental observations validates the expression. The figure 1(b) shows the evolution of the inaccuracy of the prediction and the abortion rates. For higher values of the threshold, the number of vain updates is reduced, because the probability for an object to be outdated is also low. In contrast, the abortion

Completeness of the prediction

Inaccuracy and Abortion rate

100

90 Predicted abortions Actual abortions

abortion Eager abortion Lazy abortion PA inaccuracy PA

80 80

70 60

60 %

%

50 40 40 30 20

20

10 0

0 0

0.2

0.4

0.6

0.8

1

0

0.1

0.2

0.3

0.4

PA_outd

(a) Completeness

0.5 Threshold

0.6

0.7

0.8

0.9

1

(b) Accuracy

Fig. 1. Inaccuracy for different thresholds

rate trends to be increased for higher values of the threshold, providing the rates reached with lazy replication. For the simulated scenario, it can be observed that for thresholds around 0,1 the lower number of abortions can be achieved. For this threshold, the 50% of the update requests obtain the same version, and the abortion rate is near to the minimum provided by eager replication.

4 Providing Self-Recovery ability to COLUP In section 3 we have presented a lazy update protocol, with a good behavior in respect to the abortion rate and an empirical mechanism to obtain the optimum value for the threshold of has been also shown. However, the calculus of can be used for other techniques than the evaluation of the convenience for an update. This is the case of the Fault Tolerance. In this section, we will show a modification of the COLU Protocol, in order to provide fault tolerance to the protocol. This modification is based on two principles: role migration, and the calculation of .


  

  


  

4.1 Modification of the COLU Protocol Each node in the system runs a copy of a membership monitor. This monitor is a piece of software that observes a preconfigured set of nodes, and notifies its local node about any change in this set (either additions or eliminations). Each node is labeled with a number, identifying it, and providing an order between every node in the system. The membership monitor used for the LOM Protocol is described in [MGGB00], and can be also used to provide the membership service to our COLU Protocol. The rest of this section shows the differences between the LOM and the COLU protocols.




1. When the membership monitor notices a node failure (let be the failed node), a notification is provided to every surviving node in the system. This notification causes for each receiving node to update a list of alive nodes. The effect of these notifications will be a logical migration of the ownerships of the failed node. Further steps will explain the term logical. 2. During the execution of a session, a number of messages can be sent to the different owners of the objects accessed by this session. If a message must to be sent to a failed owner , then it will be redirected to the new owner for the involved object. This new owner can be assigned in a deterministic way from the set of synchronous replicas of the object (e.g. electing as new owner the node with an identifier immediately higher to the failed one). Let be the new owner for the accessed object. The determinism of the election is important to guarantee that every surviving node redirects its messages to the same node ( ). Note that the messages sent to a node can involve more than one object. This will generate a unique message to the new owner, because every object in the original message had the same owner, and so, will have the same substitute. receives a message considering the node as an 3. When the synchronous replica owner, then the message is processed as if was the original owner. To this end, if the received message was an access confirmation request, then the lock management must be performed by , replying the request as shown in section 2.2. Moreover, if the received message was an update request, then the new owner should reply to the message sending the local version of the object. The update message will be detailed in further steps. This behavior maintains the consistence because the new owner of an object will be always elected from the set of synchronous replicas of the object. This guarantees that the value for the object maintained in the new owner is exactly the same value the failed owner had. is recovered from the failure, every alive 4. Whenever the original owner node node will be notified by its local membership monitor. Then, further messages sent from the nodes to the owner must not be redirected to , because the node has been recovered now. The alive nodes recognizes the recently recovered node by sending a greeting message. A greeting message sent from a node to the recently recovered node contains a list of locks granted to the node by the temporally owner . Using the contents of these lists, the node can generate the structures needed to manage the locks again. Thus, there becomes unnecessary for the to continue managing these locks. Step 8 includes a more detailed description of the contents of the greeting messages. 5. Nevertheless, a recently recovered node will receive request messages concerning owned objects that can have been updated during the failure period. In order to manage this situation, a recovered node must consider every object held in its local database as an “asynchronous replica”. This consideration will be done for an object until either an update reply or access confirmation reply is received from a synchronous replica of the object. These replies will be received in the situations described in steps 6 and 7.





















 








6. If an access confirmation request is received by a recently recovered node , and the involved object has not been already synchronized in the node (i.e. the concerning object has not been already updated from a synchronous replica), then must force the synchronization. This synchronization is performed with an update message sent to a synchronous replica of the object. The reply to this update message will ensure the local database to hold an updated version of the requested object. Once the object is updated in the local database, the access confirmation request can be processed as described for a standard owner node. 7. The recovered node can also process sessions during the synchronization period. These sessions will access to a set of objects. As we see in step 5, every accessed object must be temporally considered as an asynchronously maintained node until the object is synchronized. The treatment for the objects accessed by a local session will depend on the next classification: – Objects with a synchronous maintenance (i.e. either objects owned by the active node, or objects for which the active node is a synchronous replica. – Objects for which the active node is an asynchronous replica. The treatment for the objects with a synchronous maintenance in the active node will be similar to the recovery of the synchrony described in step 6. When the session ask its local COPLA manager about an object originally owned by the node, then an update request is sent to a synchronous replica of the object. The reply to this update message will ensure the database of the active node to hold an updated version of the requested object, and the response to the session can be completed. For the objects maintained asynchronously, the standard treatment can be used, taking into account that the period of outdatedness should include the time the active node was down. to recover the ownership of an object can be found in 8. Another way for a node the greeting messages received by from each alive node. This messages were introduced in step 4. When a node sends a greeting message to , the message not only contains a list of locks obtained by , but it also contains the last version for each locked object . This information is enough for to consider synchronized each object . 9. In order to ensure that a recently-recovered node achieves a correct state for its originally synchronized objects (i.e. the node receives an update message for each object that satisfy ), an asynchronous process becomes necessary to be run. This process, will be executed as a low-priority process, and will send an update request for each object not already synchronized in . Note that the interference of such process in the performance of should be low, because it will only be scheduled during idle periods. 10. The asynchronous process should also include the update, in the local database of the recovered node , of any new object created during the time the node was failed. To perform this update, a simple algorithm may be followed by just at the beginning of its recovery:
















  











– When recovers from a failure, a query is performed to the local database in order to retrieve the identifier for the more recently inserted object owned by every node in the system. This can be done due to the construction of the object identifiers. As a consequence of these requests, knows, for each node, the last inserted object. – Until has received the information from its local database, it will be locked any update in its local database. This ensures that the response of the requests does not includes any update performed after the recovery of . – In addition, and concurrently with these requests, every node in the system sends the greeting message to the recovered node. In this message, explained in 8, additional information can be included. Moreover, a greeting message sent from a node to will include the identifier of the more recently object created by the node . – The comparison of the information contained in the greeting messages, with the values collected from the local database, makes know, for each node , the lost insertions for each node (i.e. the range of objects inserted during the failure). – In addition to these object identifiers, the asynchronous process performs further requests to its local database in order to retrieve a complete list of object identifiers owned by each node in the system, and managed in a synchronous way in . – Objects contained in this list of synchronous identifiers will be considered as asynchronously maintained objects, until an update message will be sent to its owner node, and an up-to-date value is guaranteed in the local database of . Then, the identifier can be removed from the list of synchronous identifier.



















In order to update every object in the local database, the asynchronous process will use the collected information about lost insertions to perform update requests to each owner node about these objects.



The behavior described in this section can be summarized by the rewriting of the expression for presented in (1), and considering as the set of objects with , but not already synchronized (note that
synchronous management in  ):



   

   

   





        
      
  

    

(2)

The expression includes either the access to already synchronized objects, or the access to synchronous objects that have not already been synchronized.

5 Related Work Current work in consistency protocols for replicated databases can be found using either eager [AAES97,WSP 00] or lazy protocols [BK97,HAA99].

Each one has its pros and cons, as described in [GHOS96]. Eager protocols usually hamper the update performance and increase transaction response times but, on the positive side, they can yield serializable execution of multiple transactions without requiring too much effort. On the other hand, lazy protocols may allow a transaction to read outdated versions of objects, hamper the abortion rate, but they can improve transaction response times. Lazy replication is used in the presented approach, and have been implemented in a real environment (COPLA [MIG 02]). A good classification of consistency control protocols was presented in [WSP 00], according to three parameters: server architecture, server interaction and transaction termination. Among the alternatives resulting from combining these three parameters, only two of them seem to lead to a good balance of scalability and efficiency: those based on “update everywhere” and “constant interaction”, that increase the load distribution, and require low communication costs. The use of “locks” in pessimistic consistency control[BSR80] minimizes the number of aborted transactions, but degrades the performance of the system, because the complexity introduced by the management of the locks. On the other hand, the traditional use of “versions” (or “timestamps” [BHG87]) for optimistic consistency control[KR81] provides as an advantage the reduction of the blocking time of the transactions, but increases the abortion rate. Lazy update protocols, introduced in [BK97,HAA99], presented a new approach for the propagation of the updates, in contrast to the traditionally used “eager replication”. Lazy update protocols take advantage of the fact that an object can be written several times for different transactions before another transaction reads it. In respect to recovery protocols, since the first attempts based on dynamic reassignment of the voting roles in pessimistic consistency control, many work has been made in order to provide fault tolerance minimizing the time the system is unaccessible. To this end, approaches like active replication[Sch93] improves the simplicity, at the cost of load distribution, and passive replication[BMST93] increases the load distribution, at the cost of recovery time.

6 Conclusion Lazy update protocols have not been widely exploited due to its excessive abortion rate on scenarios with high probability of access conflicts. Nevertheless, such protocols can provide important improvements in the performance of a distributed system, when the abortion rate can be kept low, and the locality of the accesses is appreciable. We have presented an algorithm based on statistical techniques, that makes use of the prediction of the probability for an accessed object to be out of date ( ), and cause a further abortion of the accessing transaction. This protocol, remains a lazy update behavior, and uses this prediction to reduce its abortion rate, improving thus the performance of the system. In addition, the design of this protocol allows its variation from a pure lazy protocol, to a “paranoid” behavior (in the last case, it is comparable to an eager protocol). In addition, a modification of this protocol is also described, providing self-recovery ability, in order to include fault tolerance in the algorithm. This modification, introduced


    

as a lazy recovery mechanism, allows the re-incorporation of a fault node without interfere in the rest of nodes in the system, and avoiding to block any transaction in the system -even in the recovered node- during the recovery phase. As a result, the complete protocol provides the advantages of lazy approaches, forestalling their traditionally found disadvantages, and including fault tolerance at a very low costs. Thus, our approach reduces the abortion rates, improving the performance of the system, and allowing the recovery phase to avoid blocking of the sessions during the recovery process.

References [AAES97] D. Agrawal, G. Alonso, A. El Abbadi, and I. Stanoi. Exploiting atomic broadcast in replicated databases. Lecture Notes in Computer Science, 1300:496–503, 1997. [BHG87] Philip A. Bernstein, Vassos Hadzilacos, and Nathan Goodman. Concurrency Control and Recovery in Database Systems. Addison Wesley, Reading, MA, EE.UU., 1987. [BK97] Yuri Breitbart and Henry F. Korth. Replication and consistency: being lazy helps sometimes. In Proceedings of the sixteenth ACM SIGACT-SIGMOD-SIGART symposium on Principles of Database Systems, pages 173–184. ACM Press, 1997. [BMST93] Navin Budhiraja, Keith Marzullo, Fred B. Schneider, and Sam Toueg. The primarybackup approach. In S. J. Mullender, editor, Distributed Systems, chapter 8, pages 199–216. ACM Press, Addison-Wesley, Wokingham, Reino Unido, 2 edition, 1993. ISBN 0-201-62427-3. [BSR80] Philip A. Bernstein, David W. Shipman, and James B. Rothnie. Concurrency control in a system for distributed databases (SDD-1). ACM Transactions on Database Systems, 5(1):18–51, March 1980. [EGLT76] K. P. Eswaran, J. Gray, R. A. Lorie, and I. L. Traiger. The notions of consistency and predicate locks in a database syste m. Communications of the ACM, 19(11):624–633, November 1976. [GHOS96] J. Gray, P. Helland, P. O’Neil, and D. Shasha. The dangers of replication and a solution. In Proc. of the 1996 ACM SIGMOD International Conference on Management of Data, pages 173–182, Canada, 1996. [HAA99] JoAnne Holliday, Divyakant Agrawal, and Amr El Abbadi. Database replication: If
 Symposium on Reliable Disyou must be lazy, be consistent. In Proceedings of tributed Systems SRDS’99, pages 304–305. IEEE Computer Society Press, October 1999. [IMDBA03] Luis Ir´un, Francesc Mu˜noz, Hendrik Decker, and Josep M. Bernab´eu-Aub´an. Colup: The cautious optimistic lazy update protocol. In XI Jornadas de Concurrencia, June 2003. [Ins02] Instituto Tecnol´ogico de Inform´atica. GlobData Web Site, 2002. Accessible in URL: http://globdata.iti.es. [KR81] H. T. Kung and John T. Robinson. On optimistic methods for concurrency control. ACM Transactions on Database Systems, 6(2):213–226, June 1981. ´ [MGGB00] Francesc D. Mu˜noz Esco´ı, Oscar Gomis Hilario, Pablo Gald´amez, and Jos´e M. Bernab´eu-Aub´an. HMM: A membership protocol for a multi-computer cluster. In Anexo de las actas de las VIII Jornadas de Concurrencia, Cuenca, Espa˜na, June 2000. [MIG  02] Francesc Mu˜noz, Luis Ir´un, Pablo Gald´amez, Josep Bernab´eu, Jordi Bataller, and Mari-Carmen Ba˜nul. Globdata: A platform for supporting multiple consistency modes. Information Systems and Databases, pages 137–143, September 2002.

[Sch93]

Fred B. Schneider. Replication management using the state-machine approach. In S. J. Mullender, editor, Distributed Systems, chapter 7, pages 166–197. ACM Press, Addison-Wesley, Wokingham, Reino Unido, 2 edition, 1993. ISBN 0-201-624273. [Tho79] Robert H. Thomas. A majority consensus approach to concurrency control for multiple copy databases. ACM Transactions on Database Systems, 4(2):180–209, June 1979. [WSP  00] M. Wiesmann, A. Schiper, F. Pedone, B. Kemme, and G. Alonso. Database replication techniques: A three parameter classification. In Proc. of the 19th IEEE Symposium on Reliable Distributed Systems (SRDS’00), pages 206–217, October 2000.