A taxonomy of distributed debuggers based on execution replay

A taxonomy of distributed debuggers based on execution replay Carl Dionne, Alex Informatique, Lachine (Quebec), Canada

[email protected]

Marc Feeley, Universite de Montreal, Montreal (Quebec), Canada

[email protected]

Jocelyn Desbiens, INRS-Telecommmunications, Iles-des-Soeurs (Quebec), Canada

[email protected]

Abstract

This paper proposes a taxonomy of parallel or distributed debuggers based on execution replay. Programming of distributed or parallel systems is a complex task. Many factors contribute to this complexity, but an important one is the non determinism of these systems. Development tools adapted to parallel or distributed processing must must take into account these factors. Execution replay is a technique developped to facilitate the debugging of programs in non deterministic systems. Execution replay has very broad applications. However, not every algorithm can be applied in every situation. This taxonomy aims at clearly describing the context in which a given algorithm may be applied. By describing the context, it is possible to easily compare or chose algorithms. This paper concludes with a review and classi cation of replay debuggers.

1 Introduction It is a well known fact that programming of distributed or parallel applications is a complex task. However, still very few adapted tools are available to support the programmer. In particular, debugging tools are often either not available or not well suited to the problem in many distributed or parallel programming environments. Tools used in a sequential programming environment may not scale appropriatly to distributed or parallel environments. cited four reasons that makes debugging parallel systems more dicult: lack of global time, non-determinism, multiple threads of control and complex patterns of interactions [?]. Innovative strategies have to be developped to provide tools well adapted to these diculties. Various techniques have been proposed to address these problems. Execution replay is one of them, proposed to assist in the debugging of non deterministic programs.

1.1 Related work

This taxonomy aims at providing a terminology to describe with precision a replay algorithm. The need for a terminology in the debugging eld has been identi ed long ago: in conclusion to the 1988 workshop on distributed and parallel debugging, and both concluded that we lacked a theory and a terminology for debugging [?]. In the more restricted eld of replay debugging, important work has been accomplished since then, but we are still far from a general theory and terminology. proposed a taxonomy of race conditions [?], which was extended by and in 1994 [?]. This taxonomy is related to replay since races conditions are central to 1

non determinism. The comprehension of the various types of race conditions lead to algorithms that are optimal in terms of the quantity of logged information [?, ?]. We believe that other aspects of a replay debugger are very important. Our taxonomy widens the scope of these race conditions taxonomies, as it caracterises other aspects of a replay debugger. We cover the characteristics of the replayed system, the integration of the replay debugger into this replayed system, the replay algorithm (where the type of race conditions replayed is included), as well as the characteristics of the replay that results. We believe that a better understanding and formalization of these characteristics is essential to the development of futur replay debuggers, as it allows a clear comparison between existing debuggers, outlines their strengths and weaknesses and shows problems that have not been addressed.

2 Terminology Before we go on with the taxonomy itself, some terms rst have to be de ned.

2.1 What is a bug?

In this discussion, we give the terms failure, error and fault the meaning used by and [?]. The rst indication of an incorrect program is usally a visible symptom, such a wrong value displayed or an error message. This symptom is called failure. This failure is caused by an erroneous state of the system, such as a variable with the wrong value or the execution of the wrong statement. This erronous state is called error. This error may come from another error. The chain of all errors may be followed backward to a point where an instruction of the program causes the initial error. The instruction is called fault. A fault is also called bug.

2.2 Execution Replay

Execution replay aims at providing an eective way to debug non determinist programs. We say that a program P has a non deterministic behavior if executions with the same inputs may dier. With execution replay, information are gathered during an execution E and used to control another execution E , called replay, in such a way that E is identical to E . How can that replay be useful in debugging parallel programs? Let us suppose that execution replay is used to nd the fault that causes a failure observed in E . Information about the execution is required at a certain level of abstraction: we might not wish to debug E at the processor's instruction level, but rather at the source level. The required level of abstraction may varie from a system to another or even from one particular fault to another. We de ne the level of abstraction as a function. Let E be a sequence of states, where the transition from one state to another corresponds to the execution of an instruction at a low level of abstraction. We de ne a function O such that O(E ) is a sequence of more abstract states that describes the execution E at a more abstract point of view. A transition from one state to another in O(E ) corresponds to the execution of an instruction at the chosen point of view.1 It is easy to show that the replay E is not identical to E at all levels: one produces information, one uses it. However, this replay helps to locate the fault if: 0

0

0

 

Both executions are identical at a given level of abstraction. This implies that the fault leads to the same chain of errors and that the same failure is observed. Information may be gathered from the replay to track down the initial fault. This information may be obtained from additional instructions in the code (e.g printf), using standard sequential debuggers or with more sophisticated tools such as visualisation packages [ref].

However, replay systems may prove useful even when these conditions are not meet. si on detecte une erreur, il n'est pas necessaire de continuer! 1

A formal discussion on execution view points in relation with replay may be found in [?].

2.3 Other term related to replay

We have de ned the replay of an execution. We will now introduce additional terms related to execution replay. Execution replay is applied to a particular language or system. We call that language or system the replayed system. In this paper, we distinguish between replay algorithm and replay debugger. Altough the frontier between these terms is not always clear, we say that a debugger is applied to a system, while the algorithm describes a technique that allows to replay a set of instructions. A replay debugger could use more than one algorithm, in order to replay dierent types of instructions or to provide dierent types of replay. A replay debugger is often a component of a more complete debugging system: execution replay is helpfull in debugging if additional information is given during the replay, these information could be gathered by the other components of the debugging system. Of course, these distinctions do not always apply: the distinction between the replay debugger and the replay algorithm is fairly fuzzy when a single algorithm is applied to all instructions of the replayed system, and the debugging system could be made of a single component: the replay debugger.

3 Taxonomy This taxonomy aims at characterising with precision replay debuggers. Some of the systems that will be considered are much closer to a replay algorithm: they deal with a very speci c type of non determinism at try to achieve a speci c goal. Other systems are much closer to a usable debugging system: they are integrated into a complete debugging system consisting of other type of debuggers, may use various replay algorithm to deal with various type of non determinism or to provide various type of replay.

This taxonomy aims at characterising with precision replay debuggers. Characterisation of replay has a wider scope then characterisation of replay algorithms: it has to deal with theoretical aspects of the problem, but also has to deal with the practical aspects. We consider that the integration of the replay debugger into a larger scale debugging system is not a characteristic of the replay debugger, but rather of the debugging system. Thus, aspects such as the interface to visualization packages, user interface, and so on are not relevant in this classi cation. We rather concentrate ourselves on the criterias that are very particular to replay debuggers. These criterias falls into four large families that will now be reviewed. The set of these families allows to clearly describe the replay algorithm: the rst two families of criteria describe in which environment the algorithm is applied, the third family describes which type of algorithm is used and the last family describes the replay resulting from the application of the algorithm. Small scale models of programming may be de ned, but programming, in general, is very hard to de ne with precision. This makes it sometimes dicult to clearly de ne the frontier between the values of the criteria.

3.1 Characteristics of the replayed system

These criteria aims at categorizing the system (or language) for which the replay is provided. Two criterias will be reviewed. Discussion: SISD, MIMD, SIMD. With new threaded systems: distinction is not always clear! Discussion: Shared memory versus distributed memory: With new threaded systems, not always clear Execution unit will call process.

Criteria 1 (Type of instruction) We rst class the replay debuggers according to the type of instructions

they intend to replay. Some debuggers may be acting only on a speci c type of instruction while other may deal with a more complex set of instructions. The instructions of the replayed system may be divided in three classes. Instructions are deterministics, strongly non deterministic or weakly non deterministic. Deterministic instructions always produces the same eect while the eect of non deterministic instructions may change from one execution to the other.

Amoung the non deterministic instructions, we say that one is weakly non deterministic if there exists an ordering of instructions such that it will always produce the same eect. Simple computations are examples of deterministic instructions: log(5) always returns the same value. Access to shared variables are examples of weakly non deterministic instructions: if instructions are correctly ordered, they will have a determinsitic behavior. User inputs or access to the clock are instructions that are strongly non deterministics. Certaines instructions seront faiblement ou fortement non deterministe selon le champs d'action du devermineur. Par exemple, les operations d'entrees/sorties a un peripherique produisent un eet deterministe lorsqu'elles sont totalement ordonnees et sont donc faiblement non deterministes, mais si des acces a ce peripherique sont eectues par une trame qui n'est pas sous le contr^ole du devermineur (i.e. une trame dans une autre application), alors les instructions d'acces au peripherique seront fortement non deterministes. Criteria 2 (Task creation model) The task creation model proposed by the target system in uences the development of the replay debugger. This criteria captures whether the replay debugger requires the task creation to be static, or if it can handle dynamic task creation. Lorsqu'un langage eectue une creation statique des t^aches, toutes les t^aches qui existeront dans le systeme sont creees a l'initialisation, ce qui xe a priori le nombre de t^aches simultanees dans le systeme. Un langage qui eectue plut^ot une creation dynamique des t^aches permet la creation d'une t^ache lors de l'execution. Le nombre de t^ache varie au cours de l'execution, et n'est souvent limite que par les ressources disponibles. Cette caracteristique du langage est importante puisque certains algorithmes ne pourront ^etre utilises qu'avec des langages qui eectuent une creation statique des t^aches.

3.2 Integration into the replayed system

After the replayed system has been considered, we investigate the integration of the replay debugger to this system. This taxonomy, to our knowledge, is the rst eort the formalize this aspect of replay debugger. Descriptions of replay debuggers have often neglected to describe their integration into the replayed system and emphasized rather on the algorithm used to replay the instructions. Two criterias will be reviewed.

During the 1988 workshop on parallel and distributed debugging, a discussion took place as whether debugging of parallel systems diers from debugging of distributed systems. We believe that in the replay eld, there exists a strong dierence. This dierence comes from the importance of type of instructions that often negleted by the replay algorithms, such as le inputs or outputs. Side eects to databases or les. De ne the instrumentation process. Criteria 3 (Integration method) This criteria caracterizes the method used to integrate the replay to the system. This integration is manual when the programmer is involved in the instrumentation of the program and automatic when it is achieved without the intervention of the programmer. While the rst criteria deals with what may be considered the most interesting aspect of the replay debuggers, the second criteria deals with an aspect that often correspond to the instruction that were not considered... The failure may occur in many cases: a mistake occured in the manual instrumentation process, an instruction that is not replayed by the system was called: this include simple cases such as inputs that are not taken care of, but also harder cases, such as memory exhausion that occurs only during the replay. Criteria 4 (Actions taken on failure of the replay) Never detected, sometimes Signaled, always signaled, never occurs. Once again, this is related to the point of view at which the replay is modelled...

3.3 Type of algorithm

The criteria that have been described up to now captures the language or system in which the replay takes place as well as the way that the integration of the replay takes place. We now focus on the technique used to provide the replay itself.

Previous group of criteria were orthogonal. The criteria presented in this group are organised hierachically, and are hortogonal to all others criteria.

Criteria 5 (Data or synchronisation replay) The existing replay algorithms are rst divided along one major axis: whether the replay is based on the data or the synchronisation of the instructions.

Criteria 6 (Checkpoints or instructions based replay) In the case where datas are used to provide

the replay, we divide the replay algorithms one step further down. Checkpoints based algorithms record the complete state of the system at a given time, while instruction based algorithms record the datas used upon the execution of an instruction.

Criteria 7 (View of time) In the case where the synchronisation of instructions is reproduced, we divide

the replay algorithms according to the view of the time they use to provide that synchronisation: a view of time with a single clock (hardware or software synchronised) from which a total ordering of events can be derived, or a view of time based on causal relations[?].

Criteria 8 (Races conditions of which it acts) Most recent replay algorithms record synchronisation based on causal relationship. We divide these algorithme one step further down, according to the type of race conditions they record. The race conditions are classi ed according to the taxonomy proposed by and [?] This criteria is non exclusive: a given debugger could combine many of these techniques. The list of techniques is not exhaustives either: it categorizes the techniques that are currently applied.

3.4 Resulting replay

The last group of criterias categorizes the type of replay obtained with the debugger. A replay occurs in two dierent dimensions: in time and in the processor space. These two dimensions serves to de ne two classi cation criterias.

Criteria 9 (Range of replay in time) This criteria describes whether the replay occurs from a predetermined point in time (static time range) or if the start of the replay may be changed (dynamic time range).

Algorithms that allow dynamically determined arbitrary points may be used as a support for what is sometimes called reverse execution. We simply consider that reverse execution belongs to the interface of the replay algorithm.

Criteria 10 (Range of the replay, in the processor space) This criteria captures whether all processors are involved in the replay or if a subset of the processors initially involved may be replayed.

4 Classi cation of existing replay systems The validation of this taxonomy is done by classifying a set of replay algorithms. The algorithms that follows were choosen either because of their historical importance to the development of the eld either to show the wide range of application of the replay technique. Replay System Instructions Task Replay Alg Races Time Range Processors Instant Replay [?] read/write dynamic sync, causal all static all Optimal Replay, Shared Memory[?] read/write static sync, causal ... static all Optimal Replay, Message Passing [?] send/receive static sync, causal ... static all Replay in an actor language send/receive dynamic sync, causal all static all data, log N/A static subset Replay for CLAP [?] S/R + R/W dynamic sync, causal all static all Real Time Replay RPC? dynamic sync, total N/A static all data, log N/A static subset Reverse Execution send/receive dynamic data, CkkPts N/A dynamic all

5 Conclusion In this paper, we proposed a taxonomy of replay debuggers. This taxonomy covers four major categories of criteria: the characteristics of the replayed system, the integration of the replay debugger to the replayed system, the characteristics of the algorithm as well as the characteristics of the replay itself. A review and classi cation of existing replay debuggers validates the proposed taxonomy. The integration of the replay system to the replayed system is an aspect that has often been neglected. However, from the user's point of view, we believe that it is a very important aspect. In particular, most replay debugger cannot garantee that for any program, in any situation, thay can provide an exact replay. For example, ressources allocation is an aspect that is dealt with at the operating system level, and thus an algorithm that operates at the task level cannot garentee that the ressources will be correctly allocated during the replay. The failure of the replay system has to be planned, for the user wants to rely on it. Replay systems have mostly been proposed to solve parallel computing problems. However, as the usage of clusters of computers increases, eective solutions will also be required to debug non deterministic distributed systems. We believe that these distributed system will require the replay of wider range of instructions than those typically addressed: access to distributed databases, complex interaction with user and so on. Lots of eorts still have to be done before eective replay debuggers allow to nd real bugs in real systems. This taxonomy should help to mesure the work accomplished... and the work still to be accomplished!