Fault-Tolerant Compact Routing based on Reduced ... - CiteSeerX

Fault-Tolerant Compact Routing based on Reduced Structural Information in Wormhole-Switching based Networks* Johan Vounckx, ~G. Deconinck, R. Lauwereins", J. A. Peperstraete Katholieke Universiteit Leuven, ESAT-ACCA, Kardinaal Mercierlaan 94, B-3001 Heverlee, Belgium tel .: +32-(0)16-22 09 31 / fax . : +32-(0)16-22 18 55 ernail : Johan .VounckxDesat.kuleuven .ac.be Key words : reduced structural information, interval routing, fault-tolerance, massively parallel computers Abstract : To satisfy the ever increasing demand for computational power massively parallel computers are mandatory. Such machines require ; as well a compact routing scheme for scalability as an optimal routing algorithm to minimise the communication delay . Combination. of both requirements is reported for regular structures . Yet in a massively parallel machine the probability of failures becomes significant. The regular structure is then corrupted . To overcome this problem this paper describes a compact fault-tolerant routing algorithm for meshes. It is an interval routing scheme which is optimal without failures and nearly-optimal in the presence of failures. To minimise the number of intervals a reduced amount of structural information is used. As a result no extra intervals compared to the fault-free case are needed. We indicate how to make this routing scheme deadlock-free for wormhole routing without extra hardware or buffering . Unfortunately the number of intervals then grows drastically. Therefore we introduce an extension to interval routing, which allows the routing information to remain extremely compact. 1.

Introduction

To meet the ever increasing demand for computational power with the current technology one needs massively parallel computers (MPC) [HPCC93] . An MPC consists of thousands of processors working together on the same problem . Its processors are connected together by a communication network over which they exchange information by sending data (messages). Since it is impossible to connect the processors directly, the messages must be routed through the network to reach their destination. Clearly, the routing algorithm needs information about the network structure to route the messages . This information is stored on each node in routing tables. Classical routing tables give, for each destination node, the outgoing link the messages must follow to reach this node. The amount of routing information is proportional to the number of nodes in the system . This is too much to be useful in an MPC . Compact routing techniques which use only a very limited amount of information must clearly be used. Such techniques are described in ['SaKh85], [FrJa86], [VM87], [FrJa88], [PeUp89], [BaV190], [VoDe93b] . Optimal (shortest-path) compact routing is reported for only few topologies, such as meshes [MaTh90], hypercubes [SuBa771, .... These techniques all rely on the regularity of the topology . For a specific compact routing technique, i.e. ;interval routing, it is mathematically proven [Ruzi88] that only a limited number of topologies can be optimally routed .

' Partly supported by Esprit project 6731 (F-I'MPS), by the FKFO project 2.0104 .94 and by the Belgian Interuniversity Pole of Attraction IUAP-50 " Senior Research Associate of the Belgian National Fund for Scientific: Research

Unfortunately, massively parallel computers suffer from a low overall system reliability. Indeed, the continuing increase of the reliability of individual components is more than neutralised by the device-count of massively parallel machines . Together with executing times for the number-crunching applications on MPC's ranging from some days up to several months, we must expect failures . In the presence of these failures the regular network structure is corrupted . The implicit structural information needed to combine the classical compact routing techniques with optimality is no longer available. To overcome this problem we present a fault-tolerant routing algorithm which uses both actual network information (failures) and (reduced) structural information to route messages . This results in a compact routing scheme which is extremely suited for use in an MPCI A major problem in fault-tolerant routing is to ensure deadlock-freedom when new paths are chosen to overcome failures . Deadlocks can be prevented by using a dedicated buffering mechanism [Giint8l ], [DaSe87], [Shum89], [ChKi92] or by restricting the routing itself [GINi92] . [GINi93] . We will show how to make the routing scheme itself deadlock-free, whithout using dedicated buffering schemes. Currently this is only reported for a limited number of failures without the ;routing being compact [GINi93], [LiHa91 ]. To allow this routing algorithm to use very compact routing information we present a dedicated compact routing scheme . Wormhole switching [DaSe86], [NiMc93], [LeAb92], [ESKn93], [Pars91] and a 2D-mesh topology [Tied91], [EsKn93], [Pars91] are widely used in (massively) parallel systems . Hence we assume that the communication network is based on them.

In the remainder of this section we will clarify wormhole routing, give some preliminaries for a better understanding of this paper and explain some necessary features of multi-interval routing, the compact routing technique we found adecjuate for combining fault-tolerance and compactness . The paper contains two major blocks . In section two we explain the fault-tolerant routing algorithm . No measures are include to make this algorithm itself deadlock-free . Hence an appropriate buffering scheme is needed . In the third section we develop a deadlock-free fault-tolerant routing algorithm . At the end of that section we then give a dedicated compact routing technique to store the information needed for the deadlock-free routing algorithm in a very compact way. Finally we end this paper with a conclusion, stating the work done and enhancements which will be incorporated in the routing scheme in the near future . 1 .1. Wormhole routing Each packet is divided in so-called flits (flow control digit) . A flit is the smallest unit of information that can be accepted or refused (by a link, queue, .. .). Most flits contain data . Header-flits contain routing information. When a packet arrives at a node the header-flit(s) are examined and an output link is chosen . If the channel is already in use, the flits are blocked until it becomes available. Otherwise, flits are directly transmitted on that link. As flits are forwarded, the message becomes spread out across the links between source and destination. It is possible for the first flit to arrive at the destination node, before the last flit of the message has left the source. Becmuse most flits contain no routing information, the flits of a message must remain in contiguous links of the network and cannot be interleaved with the flits of other messages . Once a channel has been acquired by a packet, it is reserved for the whole packet. When the header flit of a message is blocked, all the flits of a message stop advancing. They remain in the network and block the progress of any other message requiring the links they occupy . After the last flit has past, the channel is released. In some wormhole routers only one flit can be buffered at a link . Larger buffers can improve network performance . Wormhole routing does not buffer the messages in the nodes, they stay in the network, i.e. bound to the links they occupy .

In the remainder of this paper we will assume the network to be wormhole switched . Each link is bidirectional with one buffer in each direction . Preliminaries

1.2.

Definition: A graph G = is a set V of vertices or nodes and a set E of edges or links. Definition : A two-dimensional mesh of width A and height 13 is a graph consisting of AxB nodes. Each

node is uniquely defined by a set of coordinates (x,y) or by a number x + yXA . Two nodes (xl,yl) and 1 and xl = x2) . (x2,y2) are connected by an edge iff d:xl - x2l = 1 and yl = y2) or (ly1 - y~ = Definition: The distance d(x, y) between two objects x and y of a graph is defined as the number of edge traversals that are minimally needed to go from one object x to the object y in the fault-free graph. The distance between a node and one of its links is 0. Definition: A routine function fee u(header v) gives the outgoing link a message on node u should follow to go to the node v indicated by the header. A routing function is said to be valid iff it properly routes messages between any two working nodes of the network . The objective of the fault-tolerant routing algorithm is to provide a valid routing .function, despite the presence of link and node failures in the communication network.

13.Multi-Interval Routing As indicated above, compact routing techniques are necessary in an MPC. Interval routing was introduced in [SaKh85] . In MTa87] it was shown that all networks support interval routing . The principle of interval routing is quite simple . In a first phase the processors and llidls have to be numbered . At each node a table of intervals of these numbers is formed . An interval is specified by the number of the first processor belonging to that interval . The next interval then indicates the processor at which the previous interval ends. This is clarified in figure la. One thus becomes a list of intervals which correspond to ranges of processors. To each interval an outgoing link is associated. Messages consist of a header, containing the number of the destination processor, and a data section (figure lb). At each intermediate node is checked to which interval the header belongs . The message is then transmitted over the link corresponding to that interval . processor a processor b processor c processor z

interval l = [alb) over link a interval 2 = [b,c) over link 0 interval 1 = [c,....) over link 7 interval l = [...,z) over link l;

interval 1 message I header

interval 2 interval n

Figure 1 : a) The intervals are uniquely determined by thefirst processor of the interval. b) Principle of interval labelling. The message is~ output on a link, according to the interval the header belongs to.

One can use one interval per outgoing link (called mono-interval routing) or use more intervals per (called multi-interval routing). -Using mom intervals per outgoing link: has the disadvantage of using more routing information, but allows more flexible and optimal routing. Indeed, using only one interval per link results in optimal routing for only a limited number of topologies [Ruzi88]. In [VlTa87J and [FrJa88] algorithms determining the intervals and the numbering of the nodes for arbitrary networks are reported . Both have complexity proportional to the number of nodes in the network . For regular topologies predefined interval tables and numbering schemes, exist [MaTh90] .

2.

Failure cycle routing

In this section we will present an interval routing scheme, which tolerates multiple failures in the anodes as well as links. The number of extra hops and the complexity of the algorithm calcui ., ;, . intervals are proportional to the number of failures in the network. We first explain the fault-free r ,. . . a necessary base for the fault-tolerant routing mechanism . Next step we deal with link and txxic ~., The fault-tolerant routing algorithm described here is very similar to the one used in [ChKiq2 restriction to strict convex regions is alleviated. Besides that a major contribution is the compare of the required routing information . 2 .1 . Fault-Free Routing Scheme Given the two-dimensional mesh topology an optimal routing algorithm can be (figure 2).

Figure 2: Interval table and graphical presentation of the intervalsfor any node (xy) in

the mt-iii

Messages are first routed in the Y-direction and next in the X-dimension . This routing algorithm and deadlock-free [Ma1190]. It will be referred to as the original routing algorithm.

i.

%%

2.2. Fault-Tolerant routing algorithm

Definition: An injured area L of a network is a set

of injured links and nodes which can be constructed recursively . V u,v e 11 :3 path of injured nodes from u to v. A no cle is a strictly cyclic ordered set of nodm ;such that each node can be reached in exactly one hop by the previous node in the set. A failure cycle Cs associated to an injured region I is a, node cycle in an injured network so that all nodes of I are located inside CI and that there is no node lying inside CI which does not belong to I. Examples of a failure cycle and an injured area can be found in figure 3.

Figure 3: An injured area and its ass(xrat(failure cycle. The resulting path between source node s and its destination d is shown.

Faults occurring in the network give raise to, possibly several, failure cycles . The failure cycle algorithm simply adapts the routing function on all nodes belonging to the failure cycles . Mm-'1911 in the fault-free case would be routed into the associated injured region or to the previous proce-ssk)r failure cycle are now sent to the next processor of the failure cycle (figure 3). The former rerouun~ needed to avoid messages getting stuck in failures, the latter derouting is needed to avoid loops .

ti :

Lemma: A message M with destination cl which arrives at a failure cycle will not get stuck in the failure cycle but leave

it at some node u to some node v g CI. This node is called the failure cycle exit node . Proof.- Suppose that a message 14 with destination node d arrives at a failure cycle Cl. The routing function on each node of CI can send M whether to the subsequent node u of the failure cycle, whether to some node v 9 Cl. In the latter case the lemma is trivially proved . The former case occurs when the original routing function would route M into the associated injured area or when the original routing function would also route the message to node u or when the original routing algorithm would send M downwards on the failure cycle. Suppose M continues to be transmitted to subsequent nodes of CI and never leaves the failure cycle. This implies that in the fault-free case, this message M is not able to leave the set of nodes formed by the failure cycle and the injured area. This is impossible since the original routing algorithm is valid and deadlock-free . a

Lemma: The failure cycle routing algorithm is a valid routing algorithm when the injured regions are strict convex l .

a)

b)

Figure 4: a) We distinguish three different situations fax )3 and y) when a message, originating from some node u and destined to a node d, encounters a failure cycle. b) When the source and destination node have the same y-coordinate, thefailure cycl'e's exit node has the same ycoordinate too, but is located closer to the destination node. Proof: We only need to consider the delivery of messages whose path

in the fault-free case crosses a failure cycle. Other messages indeed arrive correctly at their destination since the fault-free routing algorithm is valid . The failure cycle entry node and the failure cycle exit node can coincide. The failures then do not have any effect on the routing. This situation will therefore not explicitly be handled in this proof. For notations we refer to figure 4a. There are three different situations when a message encounters a failure cycle. At first yu = yd (case a), next xu = xd (case and finally xu * xd and A) Yu * Yd (case 'Y). Case a is straightforward (figure 4b) . Due to the routing on the failure cycle, the message will leave this cycle at some exit node e, with Ye Indeed, the message leaves the = :Ydfailure cycle at the point where the original routing would route the messages away from the cycle . Moreover, since the failed area is strict convex, axe - xdl < Ix u - xdJ and the message will ultimately arrive at its destination, even in the presence of multiple failure cycles. Cases P and y can be proven by showing that the messages, when starting at node u and derouted along failure cycles eventually arrive at some node q), with = This is Ydthe case as can easily be derived from figure 5. If the failed area contains some nodes n withy4psign(xd - xn ) * sign(xd - xu), then the failure cycle

1 A strict convex region is a two-dimensional mesh subtopology of the network .

exit node e's x-coordinate equals xd. Otherwise 1xd - xel < 1xd - xul . Hence, the message will fin ;,Ii, its destination node d, even in the presence of multiple failures. a) b)

Figure S: Messages destined to some node d will be! routed towards some node 9 with = yd. From there on, the yp message continues its path towards its destination node . Till now we proved that the failure cycle routing is valid for infinite meshes. Failures at the border ()t meshes cause problems since no complete failure cycle can tie formed . For meshes with wrap-around link.` (i.e. tori), we can form a failure cycle by allowing messages to use the wrap-around links. For meshes without these wrap-around links, the problem can be solved by constructing a `virtually complete" failure cycle by using the existing part of the failure cycle twice (figure 6). Clearly the outgoing link, the message are routed on, depends on the direction in which the messages travel around the part of failure cycle . Hence we need a complete interval table for each incoming link [Inrnos9l] . t !i ~J,Oag::Rw:

u

virtually complete failure cycle r~.Iwrr.,.,.: ..if% f:'tr.`-':

part offailure cycle Clearly the proposed routing algorithm is valid for strict convex injured areas .

Figure 6: Injured areas at borders ofmeshes do not have a complete failure cycle. It ispossible however to construct a virtually complete failure cycle.

C

Lemma : The failure cycle routing algorithm is a valid routing algorithm when the injured regions are near convex 2 (figure 7).

Proof. Near convex areas (figure '7) can be seen as a superposition of strict convex areas . Sequential application of the previous result ,'leads to the same conclusion.

0

2 A near convex sub network C of a itwo dimensional mesh is a scat of nodes satisfying: `d u, v e C I (xu = xv =~ !3 tp e C ~ x(, E [min(xu, xv), max(xu, xv)]) ^ (Yu = Yv =~ !3 q) e C I Y(P 9 [min(xu, yv), max(yu, yv)]) . For reasons of notational convencience strict convex regions and near convex: regions will both be referred to as convex regions .

Figure 7: A convex region can be seen as a superposition of'strict convex regions. Thefailure cycle routing mechanism is valid also here. u = source node, = failure cycle entry node, e = failure cycle exit node, d = destination node .

For non-convex regions the routing scheme is not generally valid. An example is given in figure 7. For any routing sense on the failure cycle one can always find a source/destination pair for which no valid routing exists. A possible solution is to extend The failure cycle to obtain. a convex region. Unfortunately, we now do not longer use all processors which are still functional . This set of this processors is called the implicitly injured area (figure 8). Mostly the loss of these processors is not such an enormous disadvantage, since the available bandwidth to this processors is already extremely reduced . 'Using these processors would slow down the application drastically. Alternatively one could try still to use these processors anyway. Custompaths must then be provided between each pair of processors of the implicitly injured area and between the implicitly injured area and the failure cycle. Since no regular structure is available, it is not possible to give a compact routing scheme for this.

implicitely

injured area injured area

Figure 8. For non-convex injured areas the cycle routing mechanism is not guaranteed to work For any routing sense on the failure cycle one can always find a source/'destination pair (u1, d1 and u2, d2) for which no valid routing exists. The arrows show that the cycle routing is not valid and results in loops. The injured areas are indicated in dark grey, the implicitly injured area is lightly shaded

a)

b)

Figure 10: a) Interval tables according to the allowed turns in case 1 offigure 9. The numbering of the processors is identical to the numbering in section 2'. b) Graphicalpresentation of these intervals. Now we will develop a fault-tolerant deadlock-free routing scheme . Though the mechanism seems to be very similar to the results in [CbYi92], a major difference consists in the abscence of the need for virtual channels to obtain deadlock freedom. A more complex routing algorithm is required however. Definition: The failure rectanele_hf corresponding to a failure cycle CI is the smallest strict convex area which contains all the nodes of the failure cycle. Its border is indicated as BAf. The inner failure rectangle IAf is defined as Af \ Byf The notion of a failure rectangle allows us to organise the network in different areas (figure 11). failure cycle

injured area

0

failure rectangle

inner failure rectangle other areas

Figure 11 : a) Afailure rectangle allows us to define different areas in a network

A={vc-V13ue~f :xv=X U AVUGI,bl' :YV ='Y U} D={-vev~3ueSf :x v =XU AtIueI,bf :Y v YU^xvYUAxv>xu} X3= {v6V~VueIAf :Yva

nodoOrlinks 0, 2, . W

Figure 15: The splitted-header interval routing technique. Each header is splitted into several parts. For each part a series ofpossible links is given by classical interval tables. Combination gives the final outgoing link

link 0 This algorithm seems quite complex at fist sight though a very simple implementation exists. We give a Clike notation in figure 16. The routines INTERVAL ROUTE .1/2 are routines implementing classical interval routing and returning the outgoing link associated with the interval the header belongs to. Since these outgoing links must possibly represent multiple links we choose for the following notation: The links are represented by a word having as many bits as the number of links. Each link is defined by having a 1 in the corresponding bit and 0's elsewhere . A choice between several outgoing links is then logically represented by setting 1's in the corresponding bits and 0's in the bits for links not belonging to the options .

The actual link can then easily be found by determining which bit is set for all header-parts . In fact, if we use the same notation to indicate: the links for a single link, a. simple bit-wise and-operator gives the outgoing link.

ROUTE (header) header- 1 = header >> number of bits-of_pa.rt_2 ; header-2 = header & number of bits of._part_2 ; linkl = INTERVAL ROUTE_1(header -1) ;. link2 = INTERVAL ROUTE_2(header -1) ; return(linkl & link2) ;

Figure 16: Implementation of the splitted-header interval routing mechanism for a header consisting of2 parts. Extension to more parts is straightforward.

We will now apply the splitted-header technique to our deadlock-free routing algorithm. The choice of splitting the header is straightforward . The least-significant part of the header determines the x-position, the most significant part indicates the y-position of the destination node. We then must define two interval tables, a first one for determining possible outgoing links according to the horizontal position of the node and a second one for determining possible outgoing links according to the vertical position of the node. We illustrate this in figure 17 for the fault-free routing algorithm .

I-X j+X

a)

-Y +Y

Figure 17 a) The definition of the links. IM0 0

b)

QT 0

b) The classical intervals translated to two interval tables, onefor the x-coordinate and one for the y-coordinate.

1

If we put the fault-tolerant routing algorithm in this notation we remark that the amount of routing information is very compact . Indeed, the number of intervals is proportional to the number of failure rectangles and hence to the number of failures. A second very interesting feature of this routing scheme is the complexity of the algorithm calculating the intervals . Though the interval tables must be globally changed the complexity of this algorithm remains

proportional to the number of failures . Indeed to set the interval tables at some node it is sufficient to know the position of this node in the system (reduced structural information) and the position relative to the failure rectangles (actual network information) . 4.

Conclusion

In this paper we presented a routing scheme which uses both actual network information and implicit information. This results in a fault-tolerant routing algorithms which remains nearly optimal and is yet very compact due to the interval routing; technique . The complexity of the algorithm calculating the intervals is proportional to the number of failures in the network. This makes this algorithm very well suited for massively parallel computers . Because wormhole switching gains importance in (massively) parallel machines and the buffering mechanism in these networks is mostly, fixed we demonstrated how to adapt the routing to become a deadlock-fioee routing scheme . Unfortunately the number of intervals then becomes proportional to the square root of. the number of processors in the system . Therefore we introduce an extension to interval routing, which allows the routing information to remain extremely compact. Again the complexity of the algorithm calculating this information is proportional to the number of failures in the network .

This routing scheme has been presented for the specific case of a two-dimensional mesh. Yet it can be easily extended to the more general case of k-ary n-cube topologies . In the future we will elaborate the algorithm to support this more general network topology. We will also study the problem of load-balancing in these networks and try to combine it with compact routing information as well as in networks . The relation between adaptive routing and compact routing must be clarified, especially in the presence of failures. References [BaVI90] E.M. Bakker, J. van I-ceuwen, R.B. Tan, Prefix routing schemes in dynamic networks, Technical Report RCrU-CS-90-10, Rijksuniversiteit Utrecht, March 1990

Chien, J.H. Kim, Planar Adaptive Routing: Low-cost Adaptive Networks for Multiprocessors, Pro- 19th ACM Intl . Symp. on Computer Architecture, 1992, pp. 268-277

[ChKi92] A.A.

[DaSe86] W.J. Dally, C.L Seitz, The torus routing chip, Distributed Computing, 1, 1986, pp . 187-196

[DaSe87] W.J. Dally, C.L Seitz, Deadlock-Free Message Routing in Multiprocessor Interconnection Networks, IEEE Traps. on Computers, vol . 36, no. 5, May 1987, pp. 547-553 [EsKn93] R. Esser, R. Knecht,, Intel Paragon XPIS - Architecture and Software Environment, Proceedins?s of Supercomputer 93, Mannheim, June 1993. [FrJa86] [FrJa88] [GINi92] [GINi93] [Giint8l]

G.N. Frederickson, R. Janardan, Separator-Based Strategies for Efficient Message Routing, Proc. of the 27th IEEE annual Symp. on Foundation of Comguter Science, 1986, pp. 428-437

G.N. Frederickson, R. Janardan, Designing Networks with Compact Routing Tables, Algorithmica, 3, 1988, pp. 171-190 C.J. Glass, L.M . Ni, The Turn Model for Adaptive Routing, Proc 19th Intl. Symposium On Computer Architecture., IEEE CS Press, Order No. 2940, 1992, pp. 278-287

C.J . Glass, L.M. Ni, Fault-Tolerant Wormhole Routing in Meshes, Proc. FTCS 23, France, 1993, pp. 240-249 K.D . Gunther, Prevention of Deadlocks in Packet-Switched Data Transport Systems, IEEE Transactions on Communications, vol . 29, no. 4 April 1981, pp. 512-524

Fault-'Tolerance Compact Routing based on Reduced Implicit Information

16

[HPCC93] Grand Challenges 1993 : High Performance Computing and Communications, report by the Federal Coordinating Council for Science, Engineering, and Technology, Committee on Physical, Mathematical, and Engineering Sciences, c/o National Science Foundation, 1993

[Inmos9l] Inmos, The T9000 Tran.sputer - Products overview - manual, INMOS - SGS-Thompson, first edition 1991

[LeAb92] C.E . Leiserson and al ., The Network Architecture of the Connection Machine CM-S, Thinkine Machines Corporation, November 9, 1992 [LiHa91 ]

D.H. Linder, J.C . Harden, .An Adaptive and Fault Tolerant Wormhole Routing Strategy for k-

ary n-cubes, IEEE Trans . on Computers, vol. 40, no . 1, January 1991, pp . 2-12

[MaTh90] D. May, P. Thompson, Transputers and Routers: Components for Concurrent Machines, INMOS Ltd, April 4, 1990

[NiMc93] L.M . Ni, P.K . McKinley, A Survey of Wormhole Routing Techniques in Direct Networks, IEEE Computer, Februairy 1993, pp . 62-76 [Pars91 ]

Parsytec GmbH, Technical Summary Parsytec GC, 'Version 1.0 . Parsytec GmbH, 1991 .

[PeUp89]

D. Peleg, E. Upfal, A Trade-Off between Space and Efficiency for Routing Tables, Journal of the ACM, vol . 36, no . 3, July 1989, pp . 510-530

[Ruzi88]

P. Ru~icka, On Efficiency of Interval Routing Algorithms, in Mathematical Foundations of Computer Science, M.P . Chytil, L. Janiga, V. Koubek (Eds), 1988, vol . 324 of Lecture Notes in Computer Science, pp . 492-500

[SaKh85] N. Santoro, R. Khatib, .Labelling and Implicit Routing in Networks, The Computer Journal, vol. 28, no . 1, 1985, pp . 5-8

[Shum89] M. Shumway, Deadlock-Free Packet Networks, Proc. 2nd Conf. of the North American Transputer Users Group, October 1989, Durham, NC, IOS Press 1990, pp . 139-177 [SuBa.77] H. Sullivan, T. Bashkow, D. Klappholz, A large scale, homogeneous, fully distributed parallel machine, Proceedings 4th Symposium on Computer Architecture, March 1977, pp . 105-124 [Tied91 ]

F. Tiedt, Parsytec GCel Supercomputer, Technical Report, Parsytec GmbH, 1991 .

[VITa87]

J. van Leeuwen, R. B. Tan, Interval Routing, The Computer Journal, vol . 30, no . 4, 1987, pp . 298-307

[VoDe93] J. Vounckx, G. Deconinck, R. Cuyvers, R. Lauwereins, J.A. Peperstraete, Network FaultTolerance with Interval Routing Devices, IASTED Intl . Svmp . Applied Informatics, France, May 1993 . [VoDe93b]J . Vounckx, G. Deconinck, R. Cuyvers, R . Lauwereins, J.A . Peperstraete, Multiprocessor Routing techniques, Delive rable 03 .1 .11L of ESP LIT Project 6731, July 1993