the failure of the Ethernet automation network, e.g. in a production plant, can result in .... network, the MRM sends out special MRP test frames from its ring ports.
Redundancy enhancements for Industrial Ethernet ring protocols Oliver Kleineberg, Markus Rentschler Hirschmann Automation & Control Stuttgarter Straße 45-51 DE-72654 Neckartenzlingen {oliver.kleineberg, markus.rentschler}@belden.com Abstract Ethernet ring redundancy is becoming a commodity in Industrial Ethernet environments. There are many ring redundancy protocols in existence, both standardized and proprietary. They all share the common disadvantage, that they can sustain only one fault in the network structure. A second fault results in parts of the network being unavailable. In this paper, we propose a mechanism which enhances well-known ring redundancy protocols with the capability of tolerating more than a single fault. This is achieved by installing secondary network paths and adding health monitoring to the ring clients. Protocol operation has been tested through simulation and prototypical implementation and is intended to be part of a future major software release of Hirschmann switches. A main advantage of this new protocol is the straightforward network design approach that enables to specifically protect vulnerable ring segments, based on availability calculations.
1. Introduction and Motivation 1.1. Ethernet as mission critical asset With the recent advancement of Ethernet into mission critical application fields, the need for fault-tolerant networks is becoming more and more apparent. Since the failure of the Ethernet automation network, e.g. in a production plant, can result in a halt in production and thus possibly in significant loss of money, increasing the fault tolerance of Ethernet networks is of high importance. Methods to do this have been discussed in [1] and [2]. In the different clauses of [3], several redundancy mechanisms and protocols for different use cases are defined. While [3] specifies media redundancy mechanisms as an international standard, there are many more proprietary protocols in existence. The fact that most major Ethernet equipment manufacturers have developed their own (ring) redundancy protocols is further proof of the markets needs for such mechanisms. While formerly, the support of at least one sophisticated media redundancy protocol has been a unique selling point for some Ethernet equipment manufacturers, this has become more and more a commodity. A modern
Industrial Ethernet switch is more or less expected to support a media redundancy protocol that supports the physical topology of a ring. While the performance concerning failover time and supported number of switches in a ring varies, the basic principle stays the same with all protocols. An overview about Quality of Service (QoS) performance in switched Industrial Ethernet networks with investigation into ring networks is given in [4]. Recent developments, as outlined in [5], also show that the ring topology can be implemented using protocols that achieve zero failover time, meeting even the most demanding applications’ needs today and in the future. But even with further technical development taking place in the field of ring networks, still the restriction applies that only one fault on the physical ring can be tolerated. Therefore, it is of high importance to enhance the ring networks’ resilience towards faults. This paper is organized in the following structure: Section 1 is a general overview and introduction and Section 2 shows the existing drawbacks with state of the art Ethernet ring protocols. In Section 3, we propose a mechanism that can remedy the drawbacks of ring protocols. In Section 4, we show some of the use cases the protocol can be applied to and in Sections 5 and 6, an overview about the implementation and future developments is given. 1.2. Media redundancy with physical rings The physical ring topology is very popular in industrial and power utility automation, because it maps very well to the setup of production shop floors or substations, where e.g. one production cell or substation bay can be covered by a single physical ring. However, the physical ring topology and the redundancy protocols have its limitations. One limitation, for which a solution was proposed in [6], is the fact that the protocol configuration is an elaborate process and the physical rings may have a large diameter with lots of devices to configure. Furthermore, a method to detect possible configuration errors, e.g. in ring protocols, was described in [7]. Another limitation of the single physical ring topology is its limited resilience towards faults: Only a single fault can be compensated.
In this paper, we propose a very flexible mechanism that helps to increase the availability of devices in single Ethernet rings by adding additional redundant paths. A specific characteristic of this mechanism is that its working principle is only dependant on protocol support on the specific devices in the Ethernet ring that need additional protection. The physical arrangement of the secondary network path is completely decoupled from the Ethernet ring and can be flexibly designed to meet the additional redundancy requirements. The system is scalable, enabling either Ethernet ring networks where only specific devices are protected or ring networks implementing additional fault tolerance for all Ethernet ring devices. The secondary network path can be of any type or technology, as long as Ethernet frames can be transported by it.
Figure 1 shows such an exemplary switch line network, where initially m=5 and n=4. Subsequently, when an additional network connection is established between switches no. 1 and no. 5 as indicated and n=m=5, a ring is formed. This means that by introducing one additional media link (n+1), a second network path is introduced, by which all switches in the ring can be reached. This increases the availability of all switches when calculated as a communication path from the management station, as this secondary path is available for all devices inside the ring. Figure 2 shows a star
2. Ethernet Ring Redundancy 2.1. Strengths and limits of single rings The approach behind media redundancy with a physical ring topology is the well-known n+1 redundancy approach. The basic topology for an Ethernet network is the physical line structure. In Ethernet networks that are installed today, the line is achieved by daisy-chaining Ethernet devices, such as switches, as shown in figure 1. A very basic, trivial
Figure 2. Star network and redundant link network, where an additional connection is introduced between the switches no.2 and no.3. When calculated from the management station again, in this case the additional media connection increases the availability just for switches no. 2 and no. 3. So when it comes to increasing the availability of all devices of a network structure with little effort, a ring is superior to a mesh. But a ring, compared to a full mesh, has its limitations in the number of faults it can sustain. When a ring experiences a single fault, it automatically reverts back to a line structure, and all devices in the former ring are still reachable. When the ring that has been degraded to a line experiences a second fault, the different parts of the ring are decoupled. In the case shown in figure 3, the ring is split into a segment that is still connected to the
Figure 1. Ethernet ring redundancy
Figure 3. Ring segments
observation of this installation is, that for m Ethernet switches, at least n=m-1 point to point media connections are necessary for interconnection, e.g. copper or fiber optical cables.
management station and a segment that is no longer reachable by network management. So it is an obvious course of action to further augment ring network availability with additional links. But before this is taken
up, the availability of a single Ethernet ring has to be evaluated further. 2.2. Availability calculations for a single ring In figure 1, Section 2.1, the availability can be calculated as a series of switches and media connections, each with their individual Mean Time to Failure (MTTF). The availability A of a single device can be calculated with the following formula as derived e.g. from [3] Clause 1, as:
A=
MTTF MTTF + MTTR
With the MTTF usually given by the device manufacturer, the Mean Time to Repair (MTTR) is a “soft” term, representing the time it needs to replace or repair a failed component. This is dependant on the repair processes given e.g. by plant management or through service level agreements. To simplify matters for example calculations in this paper, only the switches and the connecting media from switch to switch are taken into account. For a detailed analysis, all components that are part of a communication network need to be considered in sequence, e.g. SFP (Small Formfactor Pluggable) transceivers for fiber optical connections in the switches, as well as the management station and its media connection. In addition, all redundant systems are assumed to be non-repairable. For precise calculations on repairable redundant systems, more elaborate calculation methods need to be used as described in [8]. This simplification does not limit the informative value of the following example calculations. For the following calculations through this paper, always A = 0.9 for both switches and media connections is used. While this is (or at least should) not be a realistic value, it serves well for example purposes. The availability AL of the communication path from switch no.1 that serves as the access device for the management station to switch no. 5 in the first step of figure 1 (5 switches and 4 links) is therefore derived as follows: 9
AL = ∏ Ai = 0.9 9 = 0.387 i =1
If now, as shown in figure 1 in the last step, the line is closed to a ring, two distinct paths from the access switch 1 to switch 5 exist, as shown in figure 4. Both have to be taken into account if we want to know the total availability of the communication path between switch 1 and switch 5. Now as shown in e.g. in [8][9], the total availability of the communication path from the management access switch 1 to switch 5 can be calculated as a parallel structure of the two parallel paths “Path 1” and “Path 2”. The availability of path 1 has already been calculated as
Figure 4. Ring availability structure
AL = AL1 = 0.387. The availability of the second path can be calculated analogously as AL2 = 0.729. Now with the availability of both paths, with the probability of the complementary events, the overall availability Aoverall can be calculated: 2
Aoverall = 1 − ∏ (1 − ALi ) = 0.834 i =1
With respect to the principle of operation of modern ring redundancy protocols, this is as good as it gets for the single Ethernet ring. The only method to increase overall availability - if the availability of each individual component stays constant - is to introduce additional media links. This is not possible with existing Ethernet ring protocols, e.g. MRP (Media Redundancy Protocol), which is described in the following paragraph. 2.3. Existing Protocols and operation overview The internationally standardized Ethernet ring redundancy protocol MRP is used as a generic example for a ring redundancy protocol in this paper. While other, proprietary ring protocols may differ slightly in their mode of operation, most of the statements that are given for the MRP are true for other Ethernet ring redundancy protocols as well. The IEC 62439-2 [3] specifies the Media Redundancy Protocol, which is the further development and international standardization of the Hirschmann HiPER ring and the Siemens OSM ring protocols. The MRP specifies a ring network with one Media Redundancy Master (MRM) and a multitude of Media Redundancy Clients (MRC). The MRM and MRCs use special distinguished ports for individual ring connection. To break the loop that results in the devices physical interconnection into a ring, the MRM blocks one of its ring ports for standard Ethernet traffic. This translates the physical ring structure into a logical line structure for normal Ethernet traffic. To check the state of the network, the MRM sends out special MRP test frames from its ring ports. These test frames are forwarded by the MRCs from one ring port to the other, until they arrive back at the corresponding other ring port of the MRM. Reception of the test frame from one port on the second port tells the MRM that the ring is in good shape. When the ring experiences a fault, the test frames are no longer received by the MRM, which results in the MRM opening its second, formerly blocked ring port for
standard Ethernet traffic. In this way, a line topology and connectivity to all devices is reestablished. Installing additional media connections to circumvent the limitations outlined in Section 2.1 is not possible, because no link arbitration protocol would break the loop structures introduced. Protocols like RSTP (Rapid Spanning Tree Protocol) that are usually used to handle meshed loop structures cannot be used here, as on the ring ports that are operated by the MRP, RSTP may not be enabled. This is due to the fact, that only one link arbitration/redundancy protocol may have control over a physical interface, otherwise the protocols could interfere in each others operation.
3. Enhanced Ring Redundancy Protocol 3.1. Additional health monitoring The basic idea of enhancing the availability of ring networks like MRP is to provide the MRCs with additional media links to an external network. The Enhanced Ring Redundancy Protocol (ERRP) was designed to manage these additional paths. Standard MRP does health monitoring through its test frames only in the MRM. ERRP adds such health monitoring also to the MRCs. Similar to the MRM, the MRCs are additionally supervising the reception of the MRMs test frames they receive and forward on their ring ports. If none are received from either side, ERRP considers the MRC as completely disconnected from the MRM segment of the ring network and opens its additional path to the external network.
prevent loops. This is shown in figure 5 as dashed lines from the MRCs to the external network. As soon as the ring network experiences two simultaneous faults as shown in figure 6, the MRCs on the network segment without the MRM no longer receive the MRP test frames. The formerly blocked link on the MRC that is located on the segment that is cut off from the MRM activates its ERRP port, restoring connectivity through the external network. ERRP ports can be configured on all specific MRCs that –according to e.g. a risk calculation done by the network engineer in respect to the application- need additional availability. In extreme cases, additional redundancy can be configured on all MRCs. In order to be independent from the external, secondary network, the enhanced protocol operation and the arbitration of the additional redundancy links to the secondary network must be done completely on the ring network devices. This makes it possible for the external network to be of any shape and technology, as long as it is capable of transmitting standard Ethernet network traffic.
Figure 6. ERRP operational 3.2. Additional redundancy links The MRM is always fitted with one link to the external network that is always active and transmits/receives any regular network and special protocol traffic. This is shown in figure 5 as a continuous line from the MRM to the external network. Those MRCs that need their availability increased can be fitted with additional links to the external network as well. When the network is in a fault-free state, the ERRP protocol blocks these additional ports (further called ERRP ports) on the MRCs for regular network traffic to
Figure 5. Basic ERRP structure
3.3. Arbitrary external networks The ERRP protocol operation relies completely on the devices inside the MRP ring. This results in great flexibility towards the external network. Depending on the requirements, the external network could i.e. consist of IEEE 802.11 wireless LAN access points, it could also be a second MRP ring or a meshed RSTP network. This is further elaborated in the use cases displayed in Section 4, where we show that the protocol is flexible and can be adopted in different real-world scenarios. 3.4. Handling of segmented networks For the ERRP to work, an MRC has to detect that segmentation, as shown in figure 6, has occurred and that it has been cut off the MRM segment. So for protocol operation, two distinct network segment types can be identified: One segment that includes the MRM and one to n segments that are cut off from the MRM. These segments can be distinguished through the presence of MRP test frames. On the segment where the MRM is present, the MRP protocol test frames are still present and will be received by the MRCs at least on one
port. On segments without the MRM, these frames are no longer present. Based on the test frame presence, the MRC can detect whether it is on the MRM segment or not. If it is on the MRM segment, the MRM keeps the link to the secondary network open, so no action on the MRC is required. If test frames are no longer received, an MRC needs to open the additional redundancy path to the secondary network to reestablish connectivity to the rest of the ring. If the segment without the MRM contains more than one MRC with an additional ERRP redundancy port, these MRCs need to negotiate which device will open the link to the external network. If more than one device would activate the ERRP link simultaneously, this would result in a loop that could disable network operation. 3.5. Segmentation detection and master switch election In order to make sure that only one MRC activates the ERRP link at any time, all devices must elect a segment master switch that activates the link. This segment master switch, in concept, takes over the role of the MRM concerning the ERRP operation on this segment. All other possibly present ERRP ports on other MRCs remain on standby. The election process is done in the style of the RSTP root bridge election: Each MRC that detects that it is on a segment without the MRM starts to publish ERRP protocol frames to its MRP ring ports. These protocol frames are announcing the imminent activation of the ERRP port. While advertising its own ERRP port activation, it listens to possible other annunciation messages from other MRCs. If such an advertisement is received, the Source Media Access Control (MAC) Address included in the received frame is compared to the device’s own MAC address. If the address has a lower numerical value than the devices own MAC address, the advertisement is ignored. If it has a higher numerical value, this device is acknowledged as the segment master switch. The listening device ceases to advertise itself, stops its own ERRP port activation and begins to listen to the annunciation messages of the new segment master. If a device has determined that it is the segment master switch by not receiving an advertisement that has a higher numerical MAC address value, it activates the formerly blocked ERRP port into the external network for standard Ethernet traffic. After opening the ERRP port, it still keeps on sending its annunciation frames to the ring ports, informing other ERRP devices that it is still in operation. These periodically sent annunciation frames now serve as a replacement for the MRP test frames for the other ERRP devices on the same network segment. It also keeps on listening to the network for further ERRP protocol traffic. It also sends out a port opening annunciation frame to the external network to alert other
protocol aware devices in other segments of the additional segmentation that has occurred on the network. Each ERRP protocol aware device on the network that receives such an opening annunciation over the ERRP port flushes its Filtering Database (FDB), so that the newly configured network paths can be learned. If another fault occurs in a network segment without an MRM, the two resulting network segments can be conceptually reduced to the formerly identified segments with and without MRM, with the segment master switch replacing the MRM. Once a ring device stops receiving the segment master switch annunciation frames, it restarts the election process until a new segment master switch has been found. When a fault in a network is repaired, there are two possible cases: Either the segment with the MRM is reconnected to another segment or two segments without MRM are interconnected. 3.6. Reconnection of a segment to the MRM In the first case, after reconnecting the segment without MRM to the segment with MRM, the reception of MRP test frames causes all devices to immediately close the ERRP ports. 3.7. Reconnection of segments without MRM In the second case, after reconnecting, the two segment master switches with open ERRP ports are both on one network segment. They automatically initiate an election process between them, because they both keep on sending their annunciation frames and subsequently receive the information from the other switch. This means that the segment master switch with the lower numerical MAC value will block its ERRP port. This scenario still has a risk of introducing an immediate loop when the two network segments get reconnected and both ERRP ports are still open. This is prevented through normal MRP ring port operation in the case of reconnection. In this case, the MRP ring ports, for a certain time period, are only transparent for special protocol traffic (MRP and ERRP), but are blocked for regular Ethernet traffic. This MRP mechanism gives the annunciation messages from the segment masters ample time to reach the respective other device. 3.8. Loss of MRM The last case not covered until now is the loss of the MRM itself. In this case, the whole ring network reverts to the case of a segment without MRM. This means that election of a segment master switch will occur over all ring devices and one ring device will activate its ERRP link as a replacement for the MRM until the MRM is repaired. 3.9. Loss of link into redundant network The loss of the ERRP link itself while it is active can be compensated in segment master switches. Upon loss
of the ERRP link, the segment master ceases to send annunciation frames. This restarts the election process for the segment master, and if a device with another ERRP link is on the segment, it will take over. The former master switch that has lost its link to the external network will not participate in the election process any more until its ERRP link is repaired. If the MRM loses its link to the external network, the segment with the MRM loses connectivity to the external network. This cannot be compensated, as the MRM cannot cease to send the MRP test frames to its network ring segment. No master switch election will be initiated and no other switch will open its ERRP port and replace the lost connection, unlike on a network segment without MRM, where this is done as described above. 3.10. Availability calculation for an enhanced redundant Ethernet ring Availability calculations for networks enhanced with ERRP are dependant on the availability of the components in the external network. To have a look at the impact of an additional ERRP path on availability, in figure 7, A for the media connections of path 3 and of the switch included in the path needs to be known. A for switch and media connections is again assumed to be A = 0.9. This results in the overall availability of path 3 from switch 1 to switch 5 of AL3 = 0.729.
rd
figure 7. ERRP 3 redundant path Now, all three available paths can be seen as a parallel structure. Therefore, the availability of the path between switch 1 and 5 with ERRP enhancement is the parallel structure of path 1, 2 and 3. This results in AoverallERRP = 0.955. This shows that, in comparison with Aoverall with only the two redundant paths of the ring, the availability with the use of ERRP is significantly increased. 3.11. Protocol operation example The following figure 8 shows how an example network
figure 8. Example network 1-3 will behave, when ERRP is used in conjunction with an MRP redundant ring. In step 1, the network is in good health. The MRM has blocked one of its ports (indicated by the dashed line) for regular traffic. All ERRP ports are also blocked, with the exception of the port on the MRM, which always stays open. In step 2, one media fault occurs on the network, indicated by the lightning bolt. The MRM opens the formerly blocked port to reestablish communication. In step 3, a second media failure occurs. In this case, the ERRP switch on the right is elected as master switch and opens the link the external network. If further segmentation takes place and the left ERRP switch is cut off from the elected segment master, it will also open its port for regular traffic. In Section 4, specific use cases for real world applications are discussed. 3.12. Network recovery time with ERRP There are numerous factors that influence network recovery time with ERRP. On the one hand, the actual implementation in the devices is of importance, on the other hand, the composition of the external network that ERRP uses as additional network path influences recovery times. A detailed description of ERRP, e.g. in state machines, for future possible standardization shall include means to precisely determine network reconfiguration times, together with possible design constraints on the additional network. Simulations and results from actual implementation show recovery times on par with MRP ring recovery times.
4. Use cases 4.1. Wind Park with Ethernet network: Wireless hybrid redundant network An offshore wind park is the use case for the ERRP in a hybrid network configuration. The windmills are interconnected in a redundant ring with fibers lying on the sea ground and to a substation switch on the
mainland. Another switch is placed in each windmills nacelle housing as depict in figure 9 and is connected to the backbone switch in the windmill base. To increase the availability of the communication path to individual windmills, Wireless LAN access points can be installed that are connected to the ERRP port of the ring switch and to the MRM, located in the substation. If a network segment is decoupled, connectivity can be reestablished automatically through the wireless connection. With the use e.g. of a network management system, the damage can now be assessed in more detail and repair action can be planned accordingly. A worst case scenario is the presence of multiple failures without the additional ERRP network connections. In this case, these additional failures are not visible for the management located in the substation. Only the presence of the individual link losses on the ring can be detected. ERRP establishes the connectivity to the lost network segment(s) and helps to reduce valuable time and costs for repairs.
Figure 10. Redundant double MRP ring ring 2. If ERRP was activated on both MRP rings, in certain network failure scenarios, loops can occur. The first, immediate source of a loop can be the ports on the MRMs that connect them to their respective additional networks. If those links are not connected directly to each other, a loop is introduced as shown in figure 11.
Figure 11. Loop introduced by two MRMs Figure 9. Windmills with redundant, wireless network path 4.2. Fully redundant network structure When very high availability with deterministic network reconfiguration time on all switches in a ring is needed, all MRCs in a network can be fitted with an ERRP port. The secondary network is then built as a second MRP ring that mirrors the first ring, as shown in figure 10. The ERRP operates only on the MRP switches in ring 1, while the MRP ring 2 is standard MRP without ERRP. This means that if the MRP ring 2 suffers two subsequent failures, the ERRP on ring 1 will not activate the redundant ports, even if the networks structure could be (manually) rearranged to reconnect the segments of
The same will happen if two simultaneous failures are occurring on both MRP ring 1 and 2 at the same time. The master switch election, as described in Section 3.5, will happen on both MRP ring segments. The ERRP ports opened on both rings may not necessarily be two ports that are directly connected to each other. Future, improved versions of the protocol can be modified in a way that in a double MRP ring configuration, one MRP ring can use the other MRP ring as redundant network and vice versa without the risk of introducing an additional loop. It is, of course, possible to configure and use ERRP in the second MRP ring to connect to an additional, tertiary network segment to further increase the availability of the network. This flexibility is, in real world scenarios, of course limited due to financial feasibility.
5. Protocol implementation
development
and
5.1. Protocol operation requirements As already mentioned, the ERRP protocol basically adds health monitoring functionality in the ring clients (MRCs) that is similar to the watchdog monitoring functionality in the ring master (MRM). Performance requirements on the MRCs will therefore not exceed those on the MRMs. When the MRCs are network switches, the mode of operation of a switch (MRM or MRC) is usually only a matter of device configuration, not of physical or technical limitations. Dedicated (embedded) Ethernet network interfaces with MRP support usually only support the simpler MRC functionality because they have restricted hardware resources. These (embedded) network interfaces are not the typical use case for ERRP, as they usually only support two network interfaces for ring connection. For a multi-port embedded solution, ERRP could be an option. In this case, it is up to the developer to ascertain that ERRP is able to operate without resource problems. 5.2. User Interface and network management ERRP is directly associated with MRC functionality and will therefore be configured at the same time, when the MRC is configured. The user interfaces of Hirschmann switches will allow the configuration of a third ERRP redundancy port either through manual configuration via web or command line interface (CLI). The Hirschmann web interface is based on the Simple Network Management Protocol (SNMP), with a Management Information Base (MIB) providing access to all protocol configuration parameters. In addition to web/SNMP and CLI, it is also possible to configure the protocol automatically through the automatic ring configuration described in [6]. In network management solutions not aware of the protocol, the additional ERRP paths appear as normal physical connections. In protocol aware network management systems, the ERRP MIB information of the individual switches participating can be used to monitor and display the status of the enhanced redundant network in detail.
specific threats to the network structure are known and factored in the estimate of availability. Since ERRP-capable MRCs are fully compatible to other MRP compliant devices, they can be placed flexibly and selectively in the ring network structure to keep overall costs for redundancy to a minimum or to generally upgrade an existing network. This allows maximum flexibility in fault-tolerant design of network infrastructure and gives the possibility to upgrade existing network structures to emerging new requirements with ease. ERRP is a useful extension to MRP and could be suggested as an additional MRP feature for MRP standardization. References [1]
[2]
[3]
[4]
[5]
[6]
[7]
6. Summary ERRP is a powerful method to enhance ring redundancy protocols like MRP. It provides a platform for straightforward design of networks based on the well-known ring structure with additional and scalable risk protection redundancy. Availability calculation is a useful method to assess the risk management and can point the system designer to parts of the network that need additional redundancy, especially when use case
[8] [9]
Prytz, G.; “Redundancy in Industrial Ethernet Networks”, Factory Communication Systems, 2006 IEEE International Workshop on June 27, 2006 Page(s):380 385 Kirrmann, H.; Dzung, D.; “Selecting a Standard Redundancy Method for Highly Available Industrial Networks”, 2006 IEEE International Workshop on Factory Communication Systems, June 27, 2006 Page(s):386 – 390 IEC 62439: “Industrial communication networks: high availability automation networks”, FDIS for Ed.2 approved on 2010-01-29; available at www.iec.ch Thrybom L., Prytz G.: “QoS in Switched Industrial Ethernet”, 14th IEEE Conference on Emerging Technologies and Factory Automation, 2009. ETFA 2009, September 22-26, Palma de Mallorca, Spain Kirrmann H., Kleineberg O. et al.: “HSR: Zero recovery time and low-cost redundancy for Industrial Ethernet”, 14th IEEE Conference on Emerging Technologies and Factory Automation, 2009. ETFA 2009, September 2226, Palma de Mallorca, Spain, Work in Progress Kleineberg O., Rentschler M., Felser M.: “Automatic device configuration for Ethernet ring redundancy protocols”, 14th IEEE Conference on Emerging Technologies and Factory Automation, 2009. ETFA 2009, September 22-26, Palma de Mallorca, Spain Kleineberg O., Felser M.: Network Diagnostics for Industrial Ethernet, 13th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA 2008); September 15-18, 2008, Hamburg, Germany, Work in Progress Kirrmann, H: Fault Tolerant Computing in Industrial Automation, 2nd Edition; available at http://lams.epfl.ch Niemann, Karl-Heinz: Ein Werkzeug zur Verfügbarkeitsberechnung von vermaschten, ethernetbasierenden Automatisierungsnetzwerken (A tool for availability calculations in meshed, ethernetbased automation networks), VDI/VDE Automation 2010 Congress, Baden-Baden, Germany
