Data Distribution Service (DDS) Limitations for Data Dissemination w.r.t. Large-scale Complex Critical Infrastructures (LCCI) Christian Esposito Dipartimento di Informatica e Sistemistica (DIS) Universit´a degli studi di Napoli Federico II via Claudio 21, 80125 - Napoli, Italy
[email protected]
Abstract—The objective of this report is to present the results of the on-going research activity, carried out at the Mobilab research group at the University of Napoli, on evaluating the suitability of the recent OMG specification for pub/sub services, called Data Dissemination Service (DDS) [1], within the context of data dissemination for Large-scale Complex Critical Infrastructures (LCCI).
I. I NTRODUCTION AND M OTIVATING E XAMPLE An LCCI [2] is composed of several systems interconnected by means of wide-area networks, such as Internet; therefore, it represents an example of the Ultra Large Scale (ULS) systems that has been envisioned in a report [3] produced by Software Engineering Institute (SEI) of Carnegie Mellon University in June 2006. Many of the ideas behind LCCIs are increasingly “in the air” in several current projects that aim to develop innovative critical systems. As mentioned, SESAR represents the best concrete example of LCCI, where all the ATM entities, such as Airports Managers, En route/Approach ATC centres, Regional/National/European Airspace Management or even Aircrafts, are interconnected by means of named as System Wide Information Management (SWIM). The data dissemination middleware plays a key role, since it directly affects the achievement of the LCCI mission, and it has to satisfy the following requirements: • Scalability - delivery latency does not have to be strongly affected by the number of interested destinations and the traffic of exchanged data; • Data Delivery Resiliency - The disseminated information has to reach its destinations, despite the occurrence of failures. • Network Heterogeneity - the adopted data distribution middleware needs to offer means to improve the resiliency of the network only when needed, depending on the experienced network behavior. Middleware solutions that adopts a Publish/subscribe interaction model [4], namely pub/sub services, are composed of applications, called publishers, that generate data, and other ones, called subscribers, that consume only the data in which they have expressed their interest. Notification Service is an abstraction provided by the middleware for glueing
together publishers and subscribers. Pub/sub services provide a natural support for scalable data dissemination due to their intrinsic decoupling properties resulted by the mediator role of the Notification Service. In addition, standard-based pub/sub services, such as DDS and JMS [5], have defined several solutions to allow interoperability among different implementations of a given standard, but also among implementations of different standards (e.g., a JMS implementation is able to communicate with a DDS implementation and vice-versa as presented in [6]). Among all the possible implementation of standard-based pub/sub service, DDS appears to be superior since it offers a large set of parameters to tailor the middleware behaviour with respect to the desired QoS properties. For this reason, it represents a good candidate for being used in LCCIs; however, it does not cover all the requirements previouslypresented. In the rest of this report we will analyses the main limitation of DDS to satisfy the requirements of LCCI. II. S CALABILITY L IMITS OF DDS DDS is based on IP Multicast, and its advocates state that this represents one of its strengths since it enables DDS implementations to deliver very low latencies [7]. However, this choice has the weakness of limiting the usability of DDS in large-scale infrastructures due to the known deployability and scalability limits of IP Multicast over Internet [8]. In fact, although all the commercial routers that are currently marketed support IP Multicast, it can be used only in network domains that are managed by a single organization or in local area networks. Internet Service Providers (ISPs) are reluctant to enable data dissemination through IP Multicast in their domains in order to reduce router load and protect against unwanted traffic [9]. In addition, IP Multicast requires substantial changes to the networking infrastructure, e.g., replacement of older devices with new ones that offer a native support to multicast, and costs for such changes are not easily supported by the business model of almost all the ISPs. In addition, IP Multicast forces routers to maintain per-group state [10], e.g., members to the group, that is highly variable over time. This state management clearly violates the “stateless” perspective of the IP protocol, and also introduces strong complexities
and scaling issues in a wide-area scenario such as Internet. Therefore, currently Internet is characterized by few and scattered “islands” where IP Multicast is supported, while the other portions do not provide it. Even if connectivity among routers supporting IP Multicast can be provided using pointto-point IP encapsulated tunnels [11], such solution exhibits severe reliability limitations, i.e., it strongly results vulnerable to the failures of the routers at the end and along the tunnel, and maintainability issues, i.e., the tunnel needs to be manually re-established by human operators every time a failure occur. Even if we neglect the previous issues, IP Multicast has a reputation to impose scalability and reliability penalties [12]. Specifically, IP Multicast exhibits a severe performance impact on routers and NIC hardware and they fails to filter incoming messages beyond a few dozen multicast groups [13]. So, IP Multicast does not scale to large numbers of groups. In addition, IP Multicast has no, or limited, regulation mechanism for the traffic exchanged over the network [14], on the contrary to TCP, which offers rigorous congestion and rate control algorithms. This can cause overloading of the network resources and the consequent increase of message losses experiences by the applications. In addition, the scarce control on its usage, and relative generated traffic, makes IP Multicast vulnerable to sudden traffic “bursts”, which are able to make unstable the entire system. The common solution for the scalability issues of multicast primitives is the adoption of an overlay network [15] for defining the so called Application Level Multicast (ALM) [10]. As the name suggests, ALMs do not use IP Multicast but implement the multicasting service at the application layer:endsystems belonging to a group are interconnected using an overlay network and the duty of replicating a message when it has to be dispatched over several distinct outgoing links is carried out by end-systems rather than routers. Among the existing topologies proposed during the recent years for ALMs, the ones that adopts a peer-to-peer model [16] represent the best way to distribute data in a wide-area network, supporting both availability and scalability. However, although ALMs present neither the deployment issues neither the scaling limitations that affect IP Multicast over wide area networks, they suffer of a worse use of the network resources, by increasing the traffic load and offering lower performances than IP Multicast (as proved by the efficiency analysis presented in [17]). III. R ESILIENCY L IMITS OF DDS The retransmission-based scheme adopted by RTPS is known to have limitations to scale up when the number of destinations grows and the message loss pattern experienced by the network exacerbates [18]. To prove the inability of the ARQ scheme adopted in RTPS to guarantee resilient data delivery in challenging networks such as Internet, we want to use a DDS implementation on a testbed that is representative of Internet. So, we put the following question to ourselves: “Is there such a testbed?” Planetlab [19] is a well-known real wide-area testbed; however, it exhibits uncontrollable network dynamics, such as loss patterns. For a concrete example,
Fig. 1: Comparison of Latencies on a LAN and on Planetlab
we have performed some comparisons between a cluster of computers interconnected by a dedicated LAN, and some nodes in Planetlab. We have run a publisher on one node, and a subscriber on the other node, which exchanges ping pong messages though a DDS implementation. We have measured the latency needed by a message to go to the subscriber and backward, and Figure 1 shows the registered latencies: measures on LAN exhibit lower variability (as demonstrated by an Interquartile Distance of 20), while on Planetlab we have higher fluctuations (as demonstrated by an Interquartile Distance of 37672,5). This allows us to say that Planetlab is not a controllable testbed, so that it is tough to understand if a variation within the obtained measures is due to some unexpected behaviour of the middleware or to uncontrollable phenomena within the testbed. An other solution is to use simulation, although such approach provides us with reproducible measures, we have a scalability issue. Thefore, the only solution is to rely on emulation by using Shunra Virtual Enterprise (VE)1 , which allows users to recreate a specific network behavior according to a defined model. The model adopted in this work is the Gilbert Model [20], one of the most-commonly applied in performance evaluation studies, due to its analytical simplicity and the good results it provides in practical applications on wired IP networks [21]. The Gilbert Model is a 1-st order Markov chain model characterized by two states: state 0, with no losses, and state 1, with losses. There are four transition probabilities: (i) the probability to pass from state 0 to state 1 is called P; (ii) the probability to remain in state 0 is (1 - P); (iii) the probability to pass from state 1 to state 0 is called Q; and (iv) the probability to remain in state 1 is (1 - Q). Given Packet Loss Rate (PLR) and Average Burst Length (ABL), it is possible to compute P and Q as follows: P =
P LR·Q 1−P LR
Q = ABL−1
(1)
To keep our testbed representative of current Internet, we have assumed as PLR and ABL values obtained from a network monitoring campaign performed over PlanetLab [19], [22], a geographically distributed overlay platform designed for deploying and evaluating services over wide-area networks. It has been used for the following reasons: (i) even if many of 1 www.cnrood.nl/PHP/files/telecom
pdf/Shunra Virtual-Enterprise.pdf
TABLE I: Scenarios for the network emulator
Scenario 1 Scenario 2 Scenario 3
Fig. 2: Gilbert Model for Packet losses over a link
PlanetLab nodes are connected to Internet2, or more generally, the global research and education network (GREN) [23], this does not highly influence the observable behavior; (ii) PlanetLab has been designed to subject network services to real-world conditions; and (iii) applying the best practices presented in [24] allow PlanetLab to be suitable for measurements. On the contrary, we decided not to use directly PlanetLab because it has not been designed to perform controlled experiments and to achieve reproducible results [24]. From the observations on PlanetLab, we have defined three network scenarios depicted in Table I, defined in terms of dissemination delay, PLR and ABL (each characterized as median and inter-quartile range (IQR) of 1000 observations of 24-hours monitor traces). Last, tests have been conducted by exchanging messages between the publisher and the subscribers in a round trip manner (only the last contacted subscriber return a copy of the received message to the publisher), and applying a workload extracted from the requirements of SESAR, e.g., publishing rate of 20ms and notification size of 100KB. In Selective Repeat ARQ, message latency of a given message is function of the delivery time of the previous ones. When the notification publishing rate is small, this relation can lead to an instability of the publisher sending queue (i.e., it grows indefinitely), as shown in Figure 3A when using a publishing rate of 20 msec. This instability does not occur if the publishing rate is increased. As shown in Figure 3A, when the publishing rate is too low, performance exhibit a linear trend with the number of exchanged messages, while when using a higher publishing rate of 2 sec, this trend is no more present. In addition, such saturation phenomenon of the sending queue is also related to the experienced network conditions, as illustrated in Figure 3B. In fact, if latency delay increases and losses worsen, then the linear trend of latency (caused by the queue saturation) increases; this is shown by the variation in the latency when passing from scenario 1 (depicted in figure with a blue line) to scenario 2 (depicted with a red line), which has stronger loss characteristics but higher delay. Worsening applied network conditions does not only affect the performance of the middleware, but also its predictability, as clear in the case with no saturation depicted in Figure 3C. In fact, worse network conditions (i.e., higher delay and more losses) imply more fluctuations in the monitored latency: for the case of a publishing rate of 2 sec, when shifting from
Delay (msec) Median IQR 8,9 0,87 27,16 18 43,81 0,78
PLR Median 0,65 1,07 1,77
IQR 1,24 4,86 0,85
ABL Median IQR 1,59 0,28 1,26 0,32 1,44 0,11
scenario 1 to scenario 3 we have registered a variation in the InterQuartile Distance from 80015 msec to 923742,5 msec (with an increase of 843727,5 msec). Last, we have registered an other prove of the publisher being overwhelmed by the ARQ protocol in an hostile network conditions: the publisher discards incoming messages. Specifically, the publisher is able to deliver all the message towards interested subscribers; however, it is not able to deliver to the application the ping message coming from one of the subscribers. This is evident in Figure 3D, and lacks in the red and green lines in Figures 3BC. IV. H ETEROGENEITY L IMITS OF DDS The reliability technique applied by DDS, and in general by a publish/subscribe service, is not adaptive on the heterogeneity of the network. DDS specification states, in the Platform Specific Model (PSM) applied for UDP/IP connections, that ARQ has to be used when the users require a reliable communication, but nothing is said with respect to experienced network conditions. In the typical situation of an LCCI, which is composed by interconnecting different routing domains with assured QoS by means of WAN without any assured QoS, DDS would use ARQ even when not needed since the network is not exhibiting any loss patterns (e.g., for communications within the routing domain of a system). V. C ONCLUSIONS This report has pointed out the main limitations for the applicability of DDS-compliant solutions to large-scale infrastructure such as LCCI. The main solutions would be (i) to adopt a more scalable organization of the notification service by introducing an overlay network for resolving the deployment and scalability issues of IP Multicast, (ii) to include advanced techniques to enhance the reliability offered by the middleware without worsening scalability and timeliness, and (ii) to introduce autonomic capability so that the middleware can adapt to the experienced network conditions by properly tuning the adopted reliability technique or even selecting the best technique for the user requirements. R EFERENCES [1] O. M. Group, “Data Distribution Service (DDS) for Real-Time Systems, v1.2,” OMG Document, 2007. [2] C. Esposito, D. Cotroneo, A. Gokhale and D.C. Schmidt, “Architectural Evolution of Monitor and Control Systems - Issues and Challenges,” introduction paper for the Special Issue on Data Dissemination for Large scale Complex Critical Infrastructures at International Journal of Network Protocols and Algorithms, vol. 2, no. 3, pp. 1–17, 2010. [3] L. N. et al., Ultra-large-scale Systems, The Software Challenge of the Future. http://www.sei.cmu.edu/uls/: Software Engineering Institute, Carnegie Mellon University, 2006.
Fig. 3: Performance of ARQ in different Scenarios and publishing rates: (A) Comparison of performance within the first scenario with two different publishing rates (i.e., 20 msec and 2 sec), (B) performance in the three scenarios with publishing rate of 20 msec, (C) performance in the three scenarios with publishing rate of 2 sec, and (D) losses at the publisher side
[4] P. Eugster, P. A. Felber, R. Guerraoui, and A. Kermarrec, “The many faces of publish/subscribe,” ACM Computing Surveys, vol. 35, no. 2, pp. 114–131, January 2003. [5] S. Microsystems, “Java Message Service, v1.1,” SUN Specification, 2002. [6] R. Warren, “From the Tactical Edge to the Enterprise: Integrating DDS and JMS,” Workshop on Distributed Object Computing for Real-time and Embedded Systems, available at www.omg.org/news/meetings/GOVWS/pr/rte-pres/DDS JMS Presentation.pdf (visited in February 2011), July 2009. [7] A. Corsaro, “The Data Distribution Service for RealTime Systems,” Dr.Dobb’s electronic journal, available at www.drdobbs.com/architecture-and-design/222900238 (visited in February 2011), February 2010. [8] C. Diot, B. Levine, B. Lyles, H. Kassan, and D. Balendiefen, “Deployment numbers for the IP Multicast services and architecture,” IEEE Networks - Special number Multicasting, vol. 14, no. 1, pp. 78–88, 2000. [9] C. K. Yeo, B. S. Lee, and M. H. Er, “A survey of application level multicast techniques,” Computer Communications, vol. 27, no. 15, pp. 1547–1568, September 2004. [10] Y. Chu, S. G. Rao, S. Seshan, and H. Zhang, “A Case for End System Multicast,” IEEE Journal on Selected Areas in Communications (JSAC), vol. 20, no. 8, pp. 1456–1471, October 2002. [11] K. Almeroth, “The Evolution of Multicast: from the MBone to Interdomain Multicast to Internet2 Deployment,” IEEE Network, vol. 14, no. 1, pp. 10–20, January/February 2000. [12] Y. Vigfusson, H. Abu-Libdeh, M. Balakrishnan, K. Birman, and Y. Tock, “Dr. multicast: Rx for data center communication scalability,” Proceedings of the 2nd Workshop on Large-Scale Distributed Systems and Middleware (LADIS), pp. 1–12, September 2008. [13] J.-H. Cui, J. Kim, D. Maggiorini, K. Boussetta, and M. Gerla, “Aggregated Multicast - A Comparative Study,” Cluster Computing, vol. 8, no. 1, pp. 15–26, January 2005. [14] N. Bonmariage and G. Leduc, “A survey of optimal network congestion control for unicast and multicast transmission,” Computer Networks: The International Journal of Computer and Telecommunications Networking, vol. 50, no. 3, pp. 448–468, February 2006. [15] F. Baccelli, A. Chaintreau, Z. Liu, A. Riabov, and S. Sahu, “Scalability
[16] [17]
[18] [19]
[20]
[21] [22]
[23] [24]
of reliable group communication using overlays,” Proceedings of the 23rd Conference of the IEEE Communications Society (INFOCOM 04), March 2004. S. Androutsellis-Theotokis1 and D. Spinellis, “A survey of peer-to-peer content distribution technologies,” ACM Computing Surveys, vol. 36, no. 4, pp. 335–371, December 2004. C. Esposito and D. Cotroneo, “Resilient and Timely Event Dissemination in Publish/Subscribe Middleware,” Inaugural issue of International Journal of Adaptive, Resilient and Autonomic Systems (IJARAS), pp. 1–20, 2010. J. F. D. Rezende and S. Fdida, “Scalability issues on reliable multicast protocol,” Proceedings of COST 237 Workshop, 1999. A. Bavier, M. Bowman, D. Culler, B. Chun, S. Karlin, S. Muir, L. Peterson, T. Roscoe, T. Spalink, and M. Wawrzoniak, “Operating System Support for Planetary-Scale Network Services,” Proceedings of the First Symposium on Networked Systems Design and Implementation (NSDI 04), p. 253266, March 2004. X. Yu, J. W. Modestino, and X. Tian, “The Accuracy of Gilbert Models in Predicting Packet-Loss Statistics for a Single-Multiplexer Network Model,” Proceedings of the 24th Annual Joint Conference of the IEEE Computer and Communications Societies (INFOCOM 05), vol. 4, pp. 2602–2612, March 2005. A. Konrad, B. Zhao, and A. Joseph, “Determining Model Accuracy of Network Traces,” Journal of Computer and System Sciences, vol. 72, no. 7, pp. 1156–1171, 2006. L. Peterson, T. Anderson, D. Culler, and T. Roscoe, “A Blueprint for Introducing Disruptive Technology into the Internet,” ACM SIGCOMM Computer Communication Review, vol. 33, no. 1, pp. 59–64, January 2003. S. Banerjee, T. Griffin, and M. Pias, “The interdomain connectivity of PlanetLab nodes,” Proceedings of Passive & Active Measurement (PAM 04), p. 7382, April 2004. N. Spring, L. Peterson, A. Bavier, and V. Pai, “Using PlanetLab for network research: myths, realities, and best practices,” ACM SIGOPS Operating Systems Review, vol. 40, no. 1, pp. 17–24, January 2006.
