A Performance Study of Consensus Algorithms in Omission and Crash ...

A Performance Study of Consensus Algorithms in Omission and Crash-Recovery Scenarios Christian Fern´andez-Campusano, Roberto Corti˜nas and Mikel Larrea University of the Basque Country UPV/EHU San Sebasti´an, Spain Email: {christianrobert.fernandez, roberto.cortinas, mikel.larrea}@ehu.es

Abstract—We investigate the performance of the ChandraToueg and Paxos consensus algorithms, using as case study TrustedPals, a smartcard-based framework which allows to implement security policies in distributed systems. The current architecture of TrustedPals uses the Chandra-Toueg consensus algorithm adapted to the omission failure model. In this work, we propose to alternatively use the Paxos consensus algorithm, in order to extend the applicability of the framework to the crash-recovery failure model as well. We have performed simulations to compare these two consensus algorithms under different failure scenarios. Our results show that Paxos is more efficient than Chandra-Toueg if the first process which coordinates or leads a round suffers a failure, while both algorithms perform similarly when there are no failures. Keywords-fault-tolerant agreement; omission and crashrecovery scenarios; performance study;

I. I NTRODUCTION Consensus [1] is one of the most important problems in fault-tolerant distributed computing. In addition to the importance of the problem itself, many agreement problems, e.g., atomic commit [2], group membership [3] or totally ordered broadcast [4], can be solved by reducing them to the consensus problem, i.e., using it as a building block. Informally, in consensus processes propose an initial value and have to agree on one of the proposed values. Although many solutions have been proposed to solve consensus in a synchronous distributed system, Fischer, Lynch, and Paterson [5] showed that it is impossible to solve consensus deterministically in an asynchronous distributed system where at least one process may crash. In order to circumvent the FLP impossibility, Chandra and Toueg [4] proposed the unreliable failure detector abstraction. Roughly speaking, a failure detector is a module that provides information about other processes in the system. Failure detectors offer a modular approach that allows other applications such as consensus to use them as a building block. Additionally, they encapsulate the synchrony assumptions of the system, so applications can be designed as if they run in an asynchronous system. Following the building block approximation of consensus, Corti˜nas et al. [6] proposed a modular architecture to solve Secure Multiparty Computation [7], which is a general security problem that can be used to solve problems such as distributed voting, private bidding and online auctions.

The proposal in [6] combines a consensus algorithm and an Eventually Perfect (denoted 3P) failure detector from [4] adapted to omission scenarios. More precisely, it makes use of a tamper-proof smartcard-based secure platform named TrustedPals [8] in order to solve consensus in a partially synchronous system prone to Byzantine failures, in which the malicious behavior of processes can be reduced to a more benign model of omission failures. Looking for an improvement of that approach, we are now working on how to weaken the system assumptions in order to address more dynamic environments and to extend the applicability of the TrustedPals framework to the crash-recovery failure model as well. As an initial step, we propose to replace the Chandra-Toueg consensus algorithm used in [6] by the Paxos consensus algorithm. Additionally, we are also looking for more efficient solutions in terms of latency and the number of messages exchanged. In this work we present a new architecture for TrustedPals based on the Paxos consensus algorithm and the Omega (denoted Ω) failure detector as an alternative to the one proposed in [6]. We also present a comparison of both proposals based on experimental results from simulations. II. S YSTEM M ODEL The distributed system considered in this work is composed of a static set of processes which can communicate only by exchanging messages. The communication graph is fully connected, i.e., there is a bidirectional link between every pair of processes in the system. Links are reliable, i.e., no message from a non-faulty process is dropped, duplicated, or modified, and no message is generated by the links. Regarding timing assumptions, the bounds on message transmission time and process speeds are not known a priori, although they exist [9]. In this work we initially consider the omission failure model, in which processes may crash by prematurely halting or suffer omissions of messages either on sending (send omission) or on reception (receive omission). We also consider the crash-recovery failure model, in which processes may fail and later recover [10]. We assume a majority of correct processes in the system.

III. B UILDING BLOCKS : C ONSENSUS AND FAILURE D ETECTORS The two consensus algorithms considered in this work are adaptations of two well-known algorithms. On one hand, the rotating coordinator based consensus algorithm proposed by Chandra and Toueg in [4], which has been adapted to the omission failure model in [6]. On the other hand, Lamport’s Paxos [11] consensus algorithm, which relies on a leader process. These two algorithms tolerate up to n2 −1 concurrent failures in a system with n processes. Both algorithms structure their executions into rounds. In each round, a process tries to impose a decision. The algorithms differ in how they choose the coordinator/leader process for the next round: in the Chandra-Toueg consensus algorithm the coordinator role rotates among all processes, whereas in Paxos the leader is directly chosen. In [6], the Chandra-Toueg consensus algorithm is combined with a 3Pom failure detector, a 3P failure detector from [4] adapted to the omission failure model. On the other hand, the leader election mechanism required by Paxos is provided by the Omega failure detector, denoted Ω, proposed by Chandra, Hadzilacos and Toueg [12]. IV. C URRENT VS P ROPOSED A RCHITECTURE The current architecture of TrustedPals is composed of the following modules: the top module presents the user application, Secure Multiparty Comptation in case of [6], which makes use of the consensus service. The core module provides the consensus service. It is composed of a consensus algorithm adapted from [4], combined with a failure detector adapted to the omission failure model (3Pom ), which provides information about well connected processes. Together with these modules, there is a smartcard-based secure platform which allows to reduce Byzantine failures to more benign process crashes or message omissions Finally, the bottom module represents the distributed system in which the application has to be executed. In this work, we propose an architecture that keeps the modular approach of the previous one, but with two main differences: the Chandra-Toueg consensus algorithm is replaced by Paxos, and the failure detector considered now is an adaptation of Ω to the omission failure model (Ωom ). The combination of those two new elements in the architecture provides the system with several interesting features. On the one side, Paxos allows to reach consensus tolerating a high degree of omissive behavior (loss of messages at processes or links). On the other hand, Paxos allows to cope with processes which crash and later recover. V. E XPERIMENTAL R ESULTS We have compared the two architectures of TrustedPals presented in the previous section. To do so, we have implemented both approaches by using JBotSim [13], a tool for the simulation of distributed algorithms.

We are interested in analysing the latency of the consensus algorithms, defined as the time elapsed between the beginning of the algorithm and the instant at which the first process decides. Note that this definition of latency (also known as early latency) is a reasonable performance metric from the point of view of applications relying on TrustedPals, which usually wait until the first reply is returned. A. From 3Pom to Ωom In the architecture proposed in this work, a Ω failure detector is needed for the omission and crash-recovery failure models. As an initial approximation, we implement the Ωom failure detector by using the 3Pom failure detector presented in [6]. We transform 3Pom into Ωom by electing as leader the process with the lowest identifier among the processes considered in-connected and out-connected by 3Pom . This initial approximation on the implementation of Ωom presents two advantages. First, as in Corti˜nas et al. [6], the use of a periodical all-to-all communication pattern, inherent to the implementation of 3Pom , allows to tolerate malicious attacks. Also, from a practical point of view, it allows to make a fair comparison of the two consensus algorithms, since the cost of implementation of the failure detector module in both cases is the same. B. Conducted simulations We have measured the latency in four scenarios: • Failure-free, scenario without failures of processes in the system. • Crash, scenario where the first process which coordinates or leads a round suffers a crash failure and does not recover. • Omission, scenario where the first process which coordinates or leads a round suffers omission failures. • Crash-recovery, scenario where the first process which coordinates or leads a round suffers a crash failure and later recovers. It should be noted that we have measured the latency after the system reaches its steady state, i.e., the failure detector module into each process permanently suspects all faulty processes and no correct process is wrongly suspected. This implies, in the case of Ωom , that every failure detector module has chosen as leader the same correct process, i.e., all correct processes trust the same process as leader. We have conducted simulations consisting in several sequences of 100 consensus executions for a number of processes going from 3 to 24, observing that the algorithms present a good scalability, both for the failure-free and crash scenarios (the omission and crash-recovery scenarios, not shown, have a similar scalability). Therefore, we considered adequate a number of 12 processes for the rest of simulations and results presented in this section. Table I and Figures 1a, 1b, 1c, and 1d present the behavior of the average latency for the consensus algorithms

20

18

18

16

16

14

14 Average latency

Average latency

20

12 10 8 6

12 10 8 6

4

4 CT (Failure-free) Paxos (Failure-free)

2 0

20

40

60

80

CT (p1 Crash) Paxos (p1 Crash)

2 100

0

20

Executions consensus (sequences)

(a) Failure-free scenario.

60

80

100

(b) Crash scenario.

20

20

18

18

16

16

14

14 Average latency

Average latency

40

Executions consensus (sequences)

12 10 8 6

12 10 8 6

4

4 CT (p1 Omission) Paxos (p1 Omission)

2 0

20

40

60

80

CT (p1 Crash-recovery) Paxos (p1 Crash-recovery)

2 100

0

20

Executions consensus (sequences)

40

60

80

100

Executions consensus (sequences)

(c) Omission scenario.

(d) Crash-recovery scenario.

Figure 1: Average Latency of 100 consecutive consensus. Table I: Improvement of Paxos+Ωom vs CT+3Pom Scenario Failure-free Crash Omission Crash-recovery

CT+3Pom 11.02 14.53 16.51 16.58

Paxos+Ωom 11.47 11.66 11.69 11.75

Improvement -4.08% 19.75% 29.19% 29.13%

described above. Observe that the average latency of Paxos is almost steady in a range of 11.6 simulation time units approximately in the four scenarios. On the other side, the behavior of the latency for Chandra-Toueg grows from 11.02 in the failure-free scenario to 16.58 in the crash-recovery scenario. This is due to the fact that Paxos handles more efficiently the leader election mechanism, unlike ChandraToueg where in each new execution of the algorithm the failed process is tried as coordinator of the first round. Furthermore, we can see in Figures 1c and 1d that the omission and crash-recovery scenarios show a similar behavior for both algorithms. This is due to the fact that in the 3Pom failure detector, as well as in Ωom , processes suffering omissions are discarded to be coordinator (or leader). And a crashed process which later recovers can be

Table II: Overhead of CT+3Pom and Paxos+Ωom Scenario Failure-free Crash Omission Crash-recovery

CT+3Pom Latency Overhead 11.02 14.53 31.85% 16.51 49.82% 16.58 50.45%

Paxos+Ωom Latency Overhead 11.47 11.66 1.66% 11.69 1.92% 11.75 2.44%

considered as a process that suffers omissions, so it will be treated as an omissive process by the failure detectors (3Pom and Ωom ). As a consequence, both scenarios present a similar latency. Table I shows the average latency and improvement of Paxos versus Chandra-Toueg. We can see that Paxos has a better performance for the crash (19.75%), omission (29.19%) and crash-recovery (29.13%) scenarios, while it has a slightly worse performance when there is no failure (-4.08%). Additionally, we analysed the overhead of each algorithm regarding the failure-free scenario. The results are presented in Table II. Observe that the results from the current architecture (Chandra-Toueg and 3Pom ) show a performance overhead of 31.85%, 49.82% and 50.45% for

R EFERENCES

24 22

[1] M. C. Pease, R. E. Shostak, and L. Lamport, “Reaching agreement in the presence of faults,” J. ACM, vol. 27, no. 2, pp. 228–234, 1980.

20 Average latency

18 16 14

[2] R. Guerraoui, “Non-blocking atomic commit in asynchronous distributed systems with failure detectors,” Distributed Computing, vol. 15, no. 1, pp. 17–25, 2002.

12 10 8 6 4

[3] G. Chockler, I. Keidar, and R. Vitenberg, “Group communication specifications: a comprehensive study,” ACM Comput. Surv., vol. 33, no. 4, pp. 427–469, 2001.

CT (Crash) Paxos (Crash)

2 0

1

2

3

Failures (nodes)

Figure 2: Comparison for different numbers of failures (Crash scenario) Table III: Overhead of Paxos+Ωom vs CT+3Pom for different numbers of failures Scenario p1 crash p1 and p2 crash p1 , p2 and p3 crash

CT+3Pom 31.85% 71.05% 110.80%

Paxos+Ωom 1.66% 5.41% 8.20%

the crash, omission and crash-recovery scenarios compared to the failure-free scenario. On the other hand, results from the architecture based on Paxos exhibit a slight performance degradation compared with the failure-free case. The reason is that Chandra-Toueg relies on the rotating coordinator paradigm, while Paxos relies on a (more efficient upon failures) leader election mechanism. Lastly, we have performed a comparison for different number of processes suffering failures. Figure 2 shows the behavior of the average latency in the crash scenario for 1, 2 and 3 failures respectively. Also, Table III presents the overhead in each algorithm regarding to the failure-free scenario. Besides, we can compare the overhead suffered between algorithms. The effect of several process failures is very harmful for Chandra-Toueg and directly affects its latency (overhead of 110.80% when the first three processes crash), due to the increased number of rounds required to reach consensus, i.e, with the first three processes crashed it is necessary at least 4 rounds. On the other hand, when a new consensus with a stable leader starts in Paxos then it always succeeds in the first round, and hence the overhead of Paxos will not be substantially affected (overhead of 8.20% when the first three processes crash). ACKNOWLEDGMENT Research supported by the Spanish Research Council, grant TIN2010-17170, the Basque Government, grants IT395-10 and S-PE12UN109, and the University of the Basque Country UPV/EHU, grant UFI11/45.

[4] T. D. Chandra and S. Toueg, “Unreliable failure detectors for reliable distributed systems,” Journal of the ACM, vol. 43, no. 2, pp. 225–267, 1996. [5] M. J. Fischer, N. A. Lynch, and M. Paterson, “Impossibility of distributed consensus with one faulty process,” J. ACM, vol. 32, no. 2, pp. 374–382, 1985. [6] R. Corti˜nas, F. C. Freiling, M. Ghajar-Azadanlou, A. Lafuente, M. Larrea, L. D. Penso, and I. Soraluze, “Secure failure detection and consensus in trustedpals,” IEEE Trans. Dependable Sec. Comput., vol. 9, no. 4, pp. 610–625, 2012. [7] A. C.-C. Yao, “Protocols for secure computations (extended abstract),” in FOCS. IEEE Computer Society, 1982, pp. 160–164. [8] M. Fort, F. C. Freiling, L. D. Penso, Z. Benenson, and D. Kesdogan, “Trustedpals: Secure multiparty computation implemented with smart cards,” in ESORICS, ser. Lecture Notes in Computer Science, D. Gollmann, J. Meier, and A. Sabelfeld, Eds., vol. 4189. Springer, 2006, pp. 34–48. [9] C. Dwork, N. A. Lynch, and L. J. Stockmeyer, “Consensus in the presence of partial synchrony,” J. ACM, vol. 35, no. 2, pp. 288–323, 1988. [10] C. Mart´ın, M. Larrea, and E. Jim´enez, “Implementing the omega failure detector in the crash-recovery failure model,” J. Comput. Syst. Sci., vol. 75, no. 3, pp. 178–189, 2009. [11] L. Lamport, “The part-time parliament,” ACM Trans. Comput. Syst., vol. 16, no. 2, pp. 133–169, 1998. [12] T. D. Chandra, V. Hadzilacos, and S. Toueg, “The weakest failure detector for solving consensus,” Journal of the ACM, vol. 43, no. 4, pp. 685–722, 1996. [13] A. Casteigts, “The abs/1001.1435, 2013.

JBotSim

library,”

CoRR,

vol.