A Composable Algorithm for Clock Synchronization in ...

A Composable Algorithm for Clock Synchronization in Multi-Cluster Real-Time Systems Alexander Hanzlik, Astrit Ademaj 1 Real-Time

Systems Group, Vienna University of Technology, Vienna, Austria {hanzlik,ademaj}@vmars.tuwien.ac.at

Abstract — In a previous work we have shown that by deploying a node with a high-quality oscillator (rate-master node) in each cluster of a real-time system, we can integrate internal and external clock synchronization by a combination of a distributed mechanism for clock state synchronization with a central mechanism for clock rate synchronization. By means of hardware and simulation experiments we have shown that this combination improves the precision of the global time base in single- and multi-cluster systems while reducing the need for high-quality oscillators for non-rate-master nodes. In the original approach all nodes, including the rate-master node, execute a fault-tolerant internal clock synchronization algorithm in course of which all nodes periodically adjust their local clocks by means of clock state correction. In this paper we analyze a derivative of the original algorithm in which the rate-master node does not take part in the internal clock synchronization algorithm. We study and discuss this approach for clock synchronization with regard to cluster precision, cluster drift rate and non-interference with external clock synchronization and multi-cluster clock synchronization.

Key Words: real-time system, global time base, clock synchronization, composability.

1 Introduction The temporal accuracy of information in a distributed real-time system depends on a system-global view of time. Time-triggered distributed systems [1] use a common notion of time to order events and to trigger actions. Each node in a time-triggered system maintains a local clock. All nodes periodically execute a clock synchronization algorithm [2] to bring the local clocks into agreement and to establish a system-wide global time base. A quality parameter for a global time base is the precision, the maximum deviation between any two clocks in the system. The achievable precision mainly depends on message transmission jitter caused by the communication system and and by clock drifts.

HANZLIK, ADEMAJ

The work presented in this paper deals with a clock synchronization algorithm that minimizes the impact of clock drift on the precision of the global time base in time-triggered distributed systems. Local clocks are driven by oscillators that may differ in quality and price. An important quality attribute is the maximum drift rate of the oscillator that is guaranteed by the manufacturer over the life-time of the device. There are several influences on oscillator frequency. A major influence is that of operating over variations in temperature [3] besides other influences like gravity, vibration, electromagnetic interference or aging. Oscillators with a long-term stable drift rate close to the nominal drift rate have a significantly higher price than low-quality oscillators with wider drift-rate margin and inferior long-term stability. The technical challenge lies in the design of a clock synchronization algorithm that will generate a reliable fault-tolerant global time base of high precision and good long-term stability with low-quality oscillators. We propose a combination of a fault-tolerant distributed algorithm for clock state synchronization with a central algorithm for clock rate synchronization. In [4] we have shown that such an algorithm not only improves the precision in single-cluster systems, but can also be deployed in multi-cluster systems and for external clock synchronization with an external reference clock. The algorithm integrates internal and external clock synchronization into one mechanism and establishes a global time base of high precision in multi-cluster systems. In this paper we will present a derivative of the algorithm given in [4] that not only improves the achievable precision of the global time base, but also improves composability with regard to external clock synchronization. The paper is organized as follows: Section 2 elaborates on the system structure that we use for our considerations. Section 3 presents the new algorithm CSCRS N I that is derived from an existing solution that integrates distributed clock state synchronization with central clock rate synchronization. Section 4 investigates the performance of algorithm CSCRSN I by means of simulation experiments. Section 5 concludes the paper with interpretation and discussion of the experimental results.

2 System Structure We assume that a distributed real-time computer system can be built from one or more clusters. A cluster consists of a set of node computers that communicate with each other using a shared communication medium. Figure 1 shows a distributed real-time computer system consisting of two clusters. 2.1 Node Each node contains a host computer that executes the application program and a communication controller for message-based communication with the other nodes. Every node maintains an oscillator that drives a clock local to this node. We call this clock the local clock of the node. 2.2 Communication All nodes communicate by periodic message exchange using an a-priori known global communication schedule available at all nodes. The communication schedule is a periodic

Cluster B (low level cluster)

Cluster A (high level cluster) Node A2 GPS

Node B2 Node B3

Node A1 Gateway

Communication network A

Communication network B

Node A5

Node A3 Node A4

Node B1

Node B4

Figure 1: 2-cluster system and static scheme based on a fault-tolerant global time base. It contains, among other information, which node is allowed to send at which point in global time. One period of communication is referred to as TDMA round. Every node is allowed to transmit a message at least once in each TDMA round. The instant when the message transmission has started is called message send instant. The instant of arrival of the start of a message is called message receive instant. A message sent from a correct sender will arrive correctly at all correct receivers at about the same point in time. 2.3 Oscillators We assume that every oscillator of a node can be characterized by a nominal frequency and a maximum drift rate that defines a permitted drift window around this nominal frequency [5]. Every device that oscillates within this permitted drift window is classified as a correct oscillator, otherwise as a faulty oscillator. The data concerning the nominal frequencies and the permitted drift windows are supplied by the oscillator manufacturer. 2.4 Timer Control Unit We assume that each node has a timer control unit (TCU) in its communication controller that is responsible for the generation of a local view of global time. Each node has its local clock which measures the time with the granularity that we call microtick (µt). The granularity of the node-local view of global time is called a macrotick (MT). To maintain a reasonable global time base, all nodes must be synchronized within a precision of one MT [5]. One macrotick is made from a number of microticks initially derived from the nominal frequency of the oscillator. The number of microticks (µt) per macrotick (MT) is denoted as microtick-macrotick conversion factor (MMCF). By changing the MMCF the frequency divider shown in Figure 2 can be adjusted in order to manipulate the generation rate of the MT, the node-local representation of global time. A clock state synchronization algorithm (executed in each node) periodically calculates a clock state correction term for the local clock (in terms of microticks) and stores the value in the Macrotick Correction Term (MTCT) register (see Figure 2) of the communication controller. A hardware mechanism assures that the value of the MTCT register is added to the microtick counter in order to correct the clock deviations among the nodes in the cluster. The clock state correction algorithm periodically corrects the local clock of the node at so called state synchronization instants contained in the communication schedule (Section

HANZLIK, ADEMAJ

C lock corrections (clock synchronization)

Oscilator Microticks (µT) MTCT

µT Counter

Frequency Divider Macroticks (MT) MT Counter Local View of Global Time

Figure 2: Timer unit of a communication controller 2.2). The time interval between two state synchronization instants is denoted as state synchronization interval. Because of the imperfect manufacturing process of oscillators, each clock has a slightly different drift rate and therefore each node will have a slightly different microtick duration. During a synchronization interval the local clocks of the nodes are running free and may thus deviate from each other due to different clock drifts. These deviations are corrected at the next synchronization instant by the fault-tolerant clock state synchronization algorithm. 2.5 Time difference capturing A node has to know the deviation of its local clock to the local clocks of other nodes to be able to synchronize to these nodes. The process of reading a remote clock is referred to as time difference capturing. The expected message receive instants of all messages are a-priori known to all nodes (Section 2.2). The difference between the expected message receive instant and the actual message receive instant is measured by a time-difference capturing unit of the communication controller. The measured differences in terms of microticks (µt) are used by the clock state synchronization algorithm. These measurements are denoted as time-difference capturing values and represent the difference in the local view of global time between the sender node and the receiver node at the instant when a message is transmitted by the sender node. A positive time-difference capturing value means that the receiver’s clock is running faster than the sender’s clock 1 .

3 Clock Synchronization This section presents a clock synchronization algorithm that combines fault-tolerant clock state synchronization with central clock rate synchronization. The algorithm is based on the system structure described in Section 2. 3.1 Definitions For the considerations in this paper we introduce the following definitions: 1

Consequently, a negative time-difference capturing value means that the receiver’s clock is running slower than the sender’s clock.

Reference clock. A reference clock is a dedicated clock that is used as a time (state) and frequency (rate) standard for a set of clocks under consideration. Clock drift rate. The drift rate of a clock is the deviation of the frequency of this clock to the frequency of the reference clock. Cluster drift rate. The drift rate of a cluster is the deviation of the frequency of the global time of this cluster to the frequency of the reference clock. To assess the quality of a clock synchronization algorithm we use the following parameters: Precision. The precision is the maximum deviation in the state of any two clocks observed during a period of interest. Accuracy. The accuracy is the maximum deviation in the state of any clock and the state of the reference clock observed during a period of interest. 3.2 State and Rate Synchronization Clock state synchronization (CSS). State synchronization is a periodic activity to establish agreement among the states of a set of clocks. Each node periodically adjusts its local clock state according to the following algorithm: – CSS1: Read the states of n other clocks

by means of time difference capturing (Section 2.5).

– CSS2: Calculate a clock state correction term using the fault-tolerant average (FTA) algorithm [6]. • Sort the n clock readings in ascending or descending order. • Remove the k highest and the k lowest readings. • Use the average of the remaining n − 2k readings as the clock state correction term.

– CSS3: Apply the clock state correction term to the local clock

at the state synchronization instant in the communication schedule (Section 2.2).

Algorithm CSS is used for clock state synchronization in the Time-Triggered Architecture TTA [7]. In the TTA, the readings of four clocks are used to establish a fault-tolerant global time base (n = 4). One arbitrarily faulty clock [8] is tolerated (k = 1). A formal verification of the algorithm is given in [9]. Clock rate synchronization (CRS). Rate synchronization is a periodic activity to establish agreement among the rates of a set of clocks. Each node periodically adjusts its local clock rate according to the following algorithm: – CRS1: Read the state of a dedicated reference clock by means of time difference capturing (Section 2.5).

– CRS2: Calculate a clock rate correction term according to the deviation from the reference clock.

HANZLIK, ADEMAJ

– CRS3: Apply the clock rate correction term to the local clock by proper adjustment of the MMCF value of the TCU (Section 2.4). 3.3 The Combined Algorithm To establish and maintain a fault-tolerant global time base of high precision and accuracy, we propose an algorithm that combines distributed clock state synchronization and central clock rate synchronization. The idea behind this algorithm is to add a clock rate synchronization algorithm CRS to algorithm CSS (for which a correctness proof is available) to improve the achievable precision within a cluster. In each cluster, we use at least one node (denoted as rate-master node RM) with a highquality oscillator whose local clock serves as a reference clock for this cluster. The other nodes are denoted as time-keeping nodes TKs and may deploy cheaper oscillators. Clock state and clock rate synchronization (CSCRS). message per TDMA round.

All nodes send at least one

– CSCRS1: The RM executes algorithm CSS. – CSCRS2: The TKs execute algorithm CSS. – CSCRS3: The TKs execute algorithm CRS at the message receive instant of the message from the RM. All nodes execute algorithm CSCRS once per TDMA round 2 . 3.4 Properties of algorithm CSCRS All nodes establish a fault-tolerant global time base by execution of algorithm CSS. The TKs execute algorithm CRS upon reception of a message from the RM. The cluster drift rate is mainly driven by the RM. However, CSCRS1 interferes with – I1: The role of the RM as a reference clock. The execution of algorithm CSS at the RM has an impact on the RM’s local clock state and rate. – I2: External clock synchronization. In case of an externally synchronized cluster (like cluster A in Figure 1 that is synchronized to a GPS [10] reference time server), the RM’s local clock state and rate is intended to follow the state and rate of the external reference time server (e.g. by periodic execution of algorithm CRS at the RM). The execution of algorithm CSCRS at the TKs in cluster A ensures that all nodes in cluster A follow the state and rate of the external reference time server as described in [4]. However, it follows from I1 that execution of algorithm CSS at the RM interferes with external clock synchronization. 2

Algorithm CSCRS is a more systematic description of the approach presented in [4].

3.5 Algorithm CSCRSN I To overcome the problem of interference, we derive algorithm CSCRS N I 3 from algorithm CSCRS by replacing step CSCRS1 with CSCRS1’ and by adding a new step CRS4 to the clock rate synchronization part CRS of algorithm CSCRS. – CSCRS1’: The rate-master node RM runs free. – CRS4: Correct the state of the local clock according to the deviation from the reference clock. Changing to step CSCRS1’ means that the RM does not take part in the clock state synchronization within the cluster. The execution of algorithm CRS at the TKs ensures that the clock rates of the TKs are into agreement with the clock rate of the RM. However, algorithm CSCRS does not correct any state deviations between the local clock of the RM and the local clocks of the TKs. The desired non-interference with the role of the RM as a reference clock forbids any impact of the global time established within the cluster on the local clock of the RM. Therefore, the TKs periodically have to adjust their local clock states to the local clock state of the RM. This is the task of step CRS4. It is easy to see that Algorithm CSCRSN I eliminates I1. If the cluster is externally synchronized, the RM follows the clock state and the clock rate of an external reference time server. Elimination of I1 also eliminates I2, the interference of internal clock synchronization with external clock synchronization.

4 Simulation Experiments We have investigated the performance of algorithm CSCRSN I . The focus of the investigations was the impact of CSCRSN I on precision and cluster drift rate (Section 3.1) compared to the existing solution CSCRS described in [4]. For this reason we have performed simulation experiments using SIDERA, a simulation model for time-triggered distributed systems [11]. 4.1 System configuration Table 1 summarizes the system configuration that we have used for the simulation experiments. It is the same configuration as used the experiments given in [4]. Number of nodes TDMA round length Macrotick duration nominal MMCF Microtick duration (nominal) Clock drift rate window Rate Master

6 12ms (12 × 10 −3 s) 1 µsec (10 −6 s) 20 50 ns (50 × 10 −9 s) 40ppm (4 × 10 −5 s/s) Node 2

Table 1: Cluster configuration 3

Subscript N I stands for Non-Interfering.

HANZLIK, ADEMAJ

The 6 nodes are numbered from 0 to 5 and have been assigned the following clock rates ρi from the permitted drift window (Section 2.3): ρ0 = −2 × 10 −5 s/s, ρ1 = −1 , 2 × 10 −5 s/s, ρ2 = −4 × 10 −6 s/s, ρ3 = 4 × 10 −6 s/s, ρ4 = 1 , 2 × 10 −5 s/s, ρ5 = 2 × 10 −5 s/s. 4.2 Experiments and Results The experiments consist of two simulation runs, each with a duration of 100 TDMA rounds. In Experiment 1, we apply algorithm CSCRS to the cluster configuration presented in Section 4.1 to determine the performance with regard to cluster precision and cluster drift rate (Section 3.1). In Experiment 2, we apply algorithm CSCRS N I to the cluster configuration and compare the results to that obtained in Experiment 1. Cluster precision. Figure 3 shows the achieved cluster precision in terms of nominal microticks (1 microtick = 50 ns). Algorithm CSCRS achieves a precision of 4 microticks (upper window) whereas CSCRSN I achieves a precision of 3 microticks (lower window).

Figure 3: Cluster precision Cluster drift rate. Figure 4 shows the cluster drift rate observed in terms of seconds per second. Algorithm CSCRSN I achieves a cluster drift rate of −4 × 10−6 (lower window). It can be seen from Section 4.1 that this is exactly the drift rate of node 2, the rate-master node of the cluster. Algorithm CSCRS achieves a cluster drift rate of −5, 8 × 10 −6 (upper window). The reason for this deviation of the cluster drift rate from the drift rate of the rate-master node is interference property I1 of algorithm CSCRS, the interference of the internal clock state synchronization with the local clock rate of the rate-master node (Section 3.4). Another effect of I1 is that it takes much longer for algorithm CSCRS (upper window) to reach a stable cluster drift rate than algorithm CSCRSN I (lower window): CSCRS reaches a stable cluster drift rate after 40 TDMA rounds, CSCRSN I needs less than 10 TDMA rounds as shown in Figure 4. Clock rate synchronization. Figure 5 shows the clock rate corrections, i.e. the MMCF value adjustments (Section 2.4) in course of execution of algorithm CRS (Section 3.2).

Figure 4: Cluster drift rate During execution of algorithm CSCRSN I (lower window), there are remarkably less adjustments of the MMCF values necessary at the TKs than during execution of algorithm CSCRS (upper window). Again, the reason for this behaviour is interference property I1 of algorithm CSCRS. Elimination of I1 in algorithm CSCRSN I means more accurate estimates of the rate-master node’s drift rate at the time-keeping nodes. The effect is that with the exception of one node, all TKs never have to readjust their local clock rates after the first adjustment of their MMCF values. The situation is different for algorithm CSCRS (upper window): I1 in algorithm CSCRS causes inaccurate RM clock rate estimates at the TKs. The effect is that the TKs have to perform several readjustments of their MMCF values during simulation time.

Figure 5: Algorithm CRS - MMCF adjustments at the time-keeping nodes

Clock state synchronization. Figure 6 shows the clock state correction terms delivered by the FTA algorithm in course of execution of algorithm CSS (Section 3.2). It can bee seen that algorithm CSCRSN I produces smaller clock state correction terms (1 microtick, lower window) than algorithm CSCRS (2 microticks, upper window). This has two causes:

HANZLIK, ADEMAJ

1. Elimination of I1 in algorithm CSCRSN I provides more accurate rate estimates of the RM’s local clock at the TKs. This brings the clock rate of any single TK into better agreement with the clock rate of the RM. Consequently, this also means better agreement among the clock rates of all nodes in the cluster. 2. Execution of step CRS4 in algorithm CSCRSN I brings the clock states of the TKs into agreement with the clock state of the RM (Figure 7). Consequently, this also means better agreement among the clock states of all nodes in the cluster. Better agreement in clock state and clock rate between all nodes results in smaller clock state deviations at the state synchronization instants (Section 2.4). This in turn means smaller clock state correction terms delivered by the FTA algorithm.

Figure 6: Algorithm CSS - clock state correction terms delivered by the FTA algorithm

Figure 7: Algorithm CRS - clock state corrections at the time-keeping nodes (step CRS4)

5 Conclusion This paper introduces algorithm CSCRSN I , a derivative of algorithm CSCRS presented in a previous work [4]. Both algorithms use a combination of distributed clock state synchronization with central clock rate synchronization using a dedicated rate-master node as a reference clock. Both algorithms integrate internal and external clock synchronization into one mechanism and improve the precision of the global time base in time-triggered distributed real-time systems while reducing the need for high-quality oscillators. We compared the performance of algorithm CSCRSN I with that of the existing solution CSCRS with regard to precision and cluster drift rate by means of simulation experiments.

Compared to its predecessor CSCRS, algorithm CSCRSN I improves the performance and stability of clock synchronization with regard to the following properties: Precision.

The experiments have shown a moderate improvement in cluster precision.

Non-interference. CSCRSN I eliminates an interference between the establishment of the global time base and the role of the rate-master node as a reference clock. In a singlecluster system with no external synchronization, the non-interference property ensures that the cluster drift rate is equal to the drift rate of the rate-master node. For the ratemaster node is intended to deploy a high-quality oscillator with good long-term stability, this means stable cluster drift rate and high accuracy of the global time base. Minimization of correction terms. Another effect of the non-interference property is the minimization of the clock correction terms. The experiments have shown that algorithm CSCRSN I minimizes – the clock state correction terms delivered by the FTA algorithm executed in course of execution of the distributed clock state synchronization algorithm CSS. – the number of clock rate adjustments in course of execution of the central clock rate correction algorithm CRS. Composability. Both algorithms integrate internal and external clock synchronization into one mechanism: in case of an externally synchronized cluster, only the rate-master node is affected by the external synchronization such that it periodically reads external reference time and adjusts its clock state and clock rate to that of the external reference time server. The TKs are not aware of this external synchronization and follow the clock state and clock rate of the RM as shown in [4]. The non-interference property of algorithm CSCRSN I also improves the accuracy of external clock synchronization. In algorithm CSCRS, a change in the drift rate of an externally synchronized rate-master node will interfere with the distributed clock state synchronization algorithm CSS executed at the rate-master node. This is not the case for algorithm CSCRSN I , where the drift rate of the rate-master node is not influenced by the internal clock state synchronization within the cluster. This property improves composability with regard to external clock synchronization as well as multi-cluster clock synchronization, where the global time of a cluster serves as a reference time server for another cluster.

References [1] Hermann Kopetz. The time-triggered model of computation. Proceedings of the 19th IEEE Systems Symposium (RTSS98), Dec. 1998. [2] E. Anceaume and I. Puaut. A Taxonomy of Clock Synchronization Algorithms. Technical Report 1103, Institut de Recherche en Informatique et Syst`emes Al´eatoires (IRISA), July 1997. Available at: http://citeseer.ist.psu.edu/anceaume97taxonomy.html. [3] Hewlett-Packard Company. Fundamentals of Quartz Oscillators. HP application note 200-2, 1997.

HANZLIK, ADEMAJ [4] Hermann Kopetz, Astrit Ademaj, and Alexander Hanzlik. Clock-State and Clock-Rate Correction in Fault-Tolerant Distributed Systems. In The 25th IEEE International Real-Time Systems Symposium, Lisbon, Portugal, Dec. 2004, pages 415–425, December 2004. [5] H. Kopetz. Real-Time Systems: Design Principles for Distributed Embedded Applications. Boston: Kluwer Academic Publishers, 1997. ISBN 0-7923-9894-7. [6] J. Lundelius-Welch and N.A. Lynch. A new Fault-Tolerant Algorithm for Clock Synchronization. Information and Computation, 77:1–36, 1988. [7] H. Kopetz and G. Bauer. The Time-Triggered Architecture. Proceedings of the IEEE, Special Issue on Modeling and Design of Embedded Software, 91(1):112–126, January 2003. [8] M. Pease, R. Shostak, and L. Lamport. Reaching Agreement in the Presence of Faults. Journal of the ACM, 27(2):228–234, 1980. [9] H. Pfeifer, D. Schwier, and F. W. von Henke. Formal Verification for Time-Triggered Clock Synchronization. In Dependable Computing for Critical Applications (DCCA-7), pages 207–226, San Jose, USA, January 1999. [10] P.H.Dana. Global positioning System (GPS) Time Dissemination for Real-Time Applications. RealTime Systems Journal, 12:9–40, January 1997. [11] Alexander Hanzlik. SIDERA - a Simulation Model for Time-Triggered Distributed Systems. Research Report 62/2005, Technische Universit¨at Wien, Institut f¨ur Technische Informatik, Treitlstr. 1-3/182-1, 1040 Vienna, Austria, 2005.