This is the accepted (unformatted) version of the following article: S.-A. Lenas, S. C. Burleigh and V. Tsaoussidis, "Bundle Streaming Service: Design, Implementation and Performance Evaluation", Transactions on Emerging Telecommunications Technologies, Vol. 26, Issue 5, pp.905-917, May 2015. , which has been published in final form at http://onlinelibrary.wiley.com/doi/10.1002/ett.2762/abstract.
Bundle Streaming Service: Design, Implementation and 1 Performance Evaluation Sotirios-Angelos Lenas*, Scott C. Burleigh, and Vassilis Tsaoussidis* *
S.-A. Lenas and V. Tsaoussidis are with the Space Internetworking Center, Democritus University of Thrace, Xanthi, 67100 Greece (e-mail: {slenas,vtsaousi}@ee.duth.gr).
S. Burleigh is with Jet Propulsion Laboratory, California Institute of Technology, Pasadena, California, (e-mail:
[email protected]).
SUMMARY We present Bundle Streaming Service (BSS), a communication framework that allows for reliable data streaming over Delay/Disruptive Tolerant Networks. BSS improves the reception and storage of data streams through the application of sophisticated forwarding tactics and the exploitation of inherent Delay/Disruptive Tolerant Networking (DTN) architecture features. The proposed framework, which targets easy configuration and deployment, comprises two elements: a bundle forwarder and a software library. Both elements were implemented as part of the Interplanetary Overlay Network DTN platform. Using European Space Agency’s (ESA) DTN testbed, we experimentally evaluate BSS performance across a wide range of network scenarios. Based on these emulation results, we discuss the associated performance trade-offs along with potential improvement opportunities and demonstrate BSS’s suitability for both terrestrial and Space environments. Keywords— Data Streaming, Data Forwarding, Delay Tolerant Networking, Interplanetary Networking 1. INTRODUCTION Delay/Disruptive Tolerant Networking is gradually gaining momentum as a promising networking communication architecture for future global internetworking. The growing interest in DTN is further underlined by the continuous development of delay-tolerant networking protocols and associated standards under the auspices of the Consultative Committee for Space Data Systems (CCSDS) and the Internet Research Task Force's DTN research group. Several research studies [1, 2, 3] promote the benefits of DTN [4] and argue for its deployment in disruptive environments, beginning with the use of Bundle Protocol (BP) [5], which is fundamental to the DTN architecture and incorporates most functionalities that an overlay network requires. Delay/Disruptive-Tolerant Networks (DTNs) have already been deployed and tested in a wide variety of diverse networking environments across a large number of research projects. Examples of such networking environments include interplanetary, military, disaster response, underwater, vehicular and some forms of ad-hoc sensor networks. As DTN moves forward into mainstream usage, end user expectations for a functionally-rich communication framework should be met. That is, instead of simply providing basic network services such as routing, transport and network management, DTN also needs to support application services beyond simple bulk data transfers. In Space environments, for example, current and future science mission requirements call for a variety of multimedia service capabilities; a few such application services are presented in [6]. Our work in this study deals specifically with the issue of data streaming, a category of application service that has not yet seen much progress in DTN. Data streaming over DTNs is a challenging task considering jointly the specific characteristics of DTN environments together with the demanding nature of streaming applications. In particular, long signal propagation delays, high Bit Error Rates (BER), frequent disruptions and variable bandwidth are some of the networking factors that act inevitably against the basic application principles of data streaming and call for mechanisms that enhance a smooth viewing experience for end-users. Presently, there are not any standard mechanisms available to support this functionality and typical configurations fail to efficiently transfer data streams. In this context, we have proposed Bundle Streaming Service [7] as a practical approach that addresses many of the networking challenges related to both live and stored data streaming over DTNs. BSS is a framework that enables 1 The research leading to these results has received funding from the European Community's Seventh Framework Programme ([FP7/2007-2013_FP7REGPOT-2010-1, SP4 Capacities, Coordination and Support Actions) under grant agreement n° 264226 (project title: Space Internetworking Center-SPICE). This paper reflects only the authors’ views and the Community is not liable for any use that may be made of the information contained therein. Also, some of the research described in this paper was performed at the Jet Propulsion Laboratory, California Institute of Technology, under a contract with the National Aeronautics and Space Administration.
ETT-13-0301
2
“streaming” data to be conveyed via DTN “bundles” in a manner that supports in-order stream processing with minimal latency. Beyond that, BSS ensures reliable delivery of data to enable ad-hoc “playback” review of recently received information. Potential examples of real-time applications that could exploit the capabilities provided by this framework are one-way voice, video or continuous telemetry streaming. BSS framework was designed with Interplanetary Internet (IPN) [8] and its associated issues in mind. Consequently, it was implemented as part of the Interplanetary Overlay Network platform (ION), a well-known software distribution that implements the DTN architecture as described in Internet RFC 4838 [4,9]. Despite its initial focus on Space environments, our proposal could also fit in terrestrial DTN environments in which network nodes either follow a fixed predetermined route or move freely within a dynamic topology and hence experience occasional disruptions each time they move beyond communication range. Such networks exhibit properties similar to those of Space internetworks, in terms of bandwidth capacity and connection availability, and thus they may be serviced by a common solution framework. In this paper, we continue our work on data streaming over DTNs and provide an extended version of the preliminary performance evaluation presented in [7]. The presentation of the operational and functional requirements of the architecture along with an enhanced description of the BSS framework design and implementation details are among the key additions in the present work. Beyond that, the performance evaluation is based on a much broader set of scenario parameters and experiments including several distinctive but representative terrestrial and Space use cases. In our effort to further assess the efficacy of BSS in real-time applications, we also employed streaming-related metrics such as frame-loss ratio and average Peak Signal-to-Noise Ratio (PSNR). A thorough investigation of the potential impact of different bundle sizes and hop counts on the streaming procedure was also conducted. The rest of the paper is organized as follows. In section 2, we present the related work and highlight the contribution of our proposal. In section 3, we discuss the design principles of the BSS framework, detail its implementation and analyze its operational capabilities. In section 4, we justify the selected scenarios by presenting the use cases from which they were extracted. In section 5, we outline the experimental methodology; then we proceed with the presentation and analysis of the results and elaborate on the involved trade-offs. Based on these results we conclude in section 6, where we highlight BSS framework applicability. 2. RELATED WORK The topic of data streaming in wireless communication environments has been thoroughly investigated, especially during the last decade. Several research studies have been published addressing a wide range of data streaming issues for both terrestrial and Space networking environments. As far as terrestrial environment is concerned, a substantial number of prior proposals are targeting Mobile Adhoc Networks (MANETs). MANETs are closely related to terrestrial DTN in terms of constrained node resources, high-error-rate communication channels, variable capacity wireless links and mobility dynamics, excluding of course the end-to-end routing path requirement. In general, the majority of the efforts focused on MANETs are moving towards two main directions: i) efficiency improvement and ii) redundancy. Among the most popular approaches suggested so far for improving efficiency are: i) the dynamic optimization of data coding, throughout the streaming session, so that the encoding bitrate does not surpass the available bandwidth of the network [10], ii) routing through multiple paths in order to increase delivery probability [11,12], iii) packet prioritization to minimize queuing delay [13] and iv) specifically adapted transport layer mechanisms that aim to reduce delay in the recovery of lost data. Redundancy is usually achieved through the use of Forward Error Correction (FEC) codes or by applying content summarization and error-spreading techniques in order to provide error resilience. A wide range of cross-layer approaches have also been proposed for optimizing video streaming over MANETs [14]. Unfortunately, due to the fact that the characteristics and deployment objectives of each type of DTN may vary significantly, most of the aforementioned approaches cannot be applied efficiently in the context of DTN. In most cases, network functions such as routing, error recovery and congestion are typically located solely at the source or destination, and therefore call for end-to-end connectivity. However end-to-end approaches cannot deal with the disruptive nature of DTN; each DTN node individually should be fully aware of how to handle bundles with frames that belong to some stream, even when the application per se may run at the edges of the network. The use of FEC
ETT-13-0301
3
codes in the bundle layer also has some drawbacks since it might lead to increased demand for network resources, mainly bandwidth, without guaranteeing reliable delivery of frames when the coding rate is not sufficient to replace the loses imposed by the error rate of the communication channel. Finally, reliability should also be taken into consideration, especially for critical applications that handle time-sensitive data [15]. The need for data streaming and especially multimedia streaming is also growing continuously in Space networking environments. This growth comes as a result of the rapid globalization of the telecommunications and entertainment industry, as well as the increased requirements of the new generation of scientific Space missions. A number of streaming-specific satellite communication systems have been proposed using Geosynchronous satellites, Medium Earth Orbit and Low Earth Orbit constellations. From a networking perspective, the operation of these systems relies on streaming-oriented standardized network protocols developed under the auspices of either the European Telecommunications Standards Institute or the International Telecommunication Union (ITU). Typical examples of the deployed network protocols that provide the satellite-dependent features of the lower levels of the network stack (layers 1 and 2) are Digital Video Broadcast over Satellite protocol (DVB-S, versions 1 and 2), the Broadband Satellite Multimedia system and the Asynchronous Transfer Mode protocol suite. At the same time, common Internet network protocols are deployed in the upper levels of the stack (layers 3 and above) [16, 17] in order to provide interoperability at the IP level between terrestrial and Space applications. In Deep Space communications, data streaming is performed using CCSDS standardized protocols. Packet Telemetry/Packet Telecommand or Proximity-1 at the link layer and Space Packet Protocol at the network layer constitute the most common options [18], while a potentially advanced streaming functionality should be supported at the application layer. Overall, given the existing Space protocol stack configurations, it becomes clear that both short and long range Space communication schemes could benefit from the deployment of more advanced streaming mechanisms in the upper layers of the network stack. Yet, to the best of our knowledge, the issue of data streaming in the strict context of DTN has not been studied extensively. In [19], T. Liu and S. Nelakuditi use erasure coding techniques in order to construct a disruptiontolerant video sequence so that in the event of disruption the network may still provide helpful video content to clients by injecting additional “summary frames” to the original stream. Another proposal which is also based on a coding scheme is presented in [20] by P. U. Tournoux et al. They introduce Tetrys, a transport level mechanism based on an on-the-fly coding scheme which provides full reliability provided that the encoding ratio used by Tetrys is higher than the average loss rate. Other well-known approaches employ data replication techniques in order to improve the availability and the consistency of data despite potentially frequent network disruptions. One such approach is presented by N. Kumar and J. Kim [21] who propose a probabilistic data replication strategy, specifically designed for online video streaming on vehicular DTN. Their scheme aims at increasing the availability of video streams among a limited group of mobile nodes whose connection to the main data server is frequently dropped, by locally replicating the most common data. Finally, broadcasting systems that rely on delay-tolerant forwarding of data chunks through mobility of wireless nodes have also been proposed for distributing multimedia content. In [22], G. Karlsson et al. propose such a broadcasting system, which follows a receiver-driven approach and provides unreliable content dissemination to an arbitrarily large group of receivers; under certain circumstances, mainly related to the mobility of nodes, it can also be used for multimedia content distribution. After carefully reviewing the related literature, we noticed that while the majority of the approaches focus on routing, data replication and coding techniques, almost none of them employed any sophisticated forwarding mechanism; instead, the role of the applied forwarding mechanisms is solely confined in handling different packet priorities. Our approach is rooted in this observation and introduces a forwarding technique that employs a combination of transport services in the forwarding procedure. Similar ideas already exist in the literature but are applied in a different context: in [23], N. Panchakarla and J. Ott propose a voice streaming system for terrestrial environments, which operates in a synchronous manner based on RTP/UDP transport services when an end-to-end path exists between the communicating nodes and switches to DTN-based voice messaging when the network is disrupted/disconnected. In [24], L. Keller et al. propose a communication system for cooperative video streaming among a group of smart devices that simultaneously use two network interfaces: cellular 3G/4G to connect to the video server and WiFi for intra-group communication. Unlike existing work, BSS assumes no end-to-end connectivity. It specifically targets DTN, thus its forwarding decisions are made on a hop-by-hop basis. BSS emphasizes the efficient management of available network resources
ETT-13-0301
4
and the improvement of the end user experience in viewing data streams. Given the fact that disruptions are usually localized and experienced only by a few among many receivers, BSS also contributes towards more effective utilization of available bandwidth by significantly reducing retransmission effort. Finally, we also consider the design perspective of BSS to be a significant advantage: the BSS design does not exclude deployments of other sophisticated mechanisms on top of, or below BSS, but instead contributes flexibility that further enhances synergistic application mechanisms. 3. DESIGN AND IMPLEMENTATION DETAILS During the design and implementation phases several operational requirements were identified. As a Spaceoriented service, BSS is mainly aimed at supporting Space mission data-streaming needs ranging from telemetry data-streaming to variable-bit-rate video-streaming. Typical streaming operation in challenged environments is severely affected by extreme communication channel conditions including limited bandwidth availability, long propagation delays and high BERs. Besides the operational requirements, the design of the framework was also based on the streaming-specific network and application functionalities envisioned for BSS. In-order delivery of bundles is considered a crucial property of data streaming applications, so as to provide a smooth viewing experience to the end user. Unfortunately, DTN messaging protocols, such as BP, do not include any inherent mechanism for in-order delivery of bundles. Reliability remains indeed an issue of concern given the potential use of BSS in critical Space applications that are designated to handle time-sensitive data. Considering jointly the design and implementation issues, a fundamental question was the stack layer at which BSS functionality should be implemented. One might argue that implementing BSS solely at the application layer constitutes a flexible solution that totally shifts control over the entire range of BSS features to the application user. Although such an argument might seem reasonable it also presents a number of fundamental drawbacks. The BP specification makes it clear that the application layer is unaware of the supported transport layer services underlying bundle transmission; the bundle layer is responsible for selecting the convergence-layer protocol through which the bundle will be forwarded to the next node. Furthermore, BP specification leaves open the issue of how to select the appropriate convergence-layer protocol. Currently, all major DTN stack implementations follow a one-to-one approach that binds transmissions to specific endpoints to predetermined underlying protocols. Implementing BSS solely at application layer, in line with the aforementioned one-to-one approach, would imply that our BSS application framework must be associated with specific transport services by default. Guaranteeing reliability in such cases can either be achieved by an underlying reliable transport service or by application enhancements that supplement a best-effort transport service; in the latter case, retransmission of lost bundles becomes clearly an application task. The two options entail a trade-off. The additional overhead that characterizes a fine-grained reliable transport protocol has the effect of reducing the volume of data that must be retransmitted; the lack of a reliable transport service may result in retransmission of entire bundles by the application. Moreover, both options have the additional disadvantage of either retarding data delivery until retransmission is completed or else foregoing in-order delivery of data to the user. Therein lies the innovation of BSS: it balances the trade-off between efficiency in initial transmission and efficiency in retransmission, delivering streaming data in transmission (rather than reception) order without increasing delivery latency. This is achieved by performing the streaming functions at the “bundle” layer and employing a pair of transport services at the convergence layer – a best-effort service together with a reliable service. The best-effort service delivers most of the data in transmission order with minimal latency and minimal initial transmission overhead, while the fine-grained reliable transport service ensures eventual delivery of all data lost in transit while minimizing retransmission overhead. Moreover, because the BSS design is neutral with respect to quality of service (QoS) markings defined in the BP specification [5], it naturally supports concurrent transmission of multiple streams at different priorities over the same communication links. All network functionality of BSS is placed within the bundle layer in the form of a forwarding daemon, which is a suitably modified version of the default IPN forwarding daemon included in ION. Application-related functionalities, such as re-ordering of “media” packets received out of transmission order, are preserved at the application layer and are implemented in the form of a software library. In Fig. 1 we depict the architecture of BSS.
ETT-13-0301
Fig. 1. BSS architecture and network layering.
5
Bundle creation time is the criterion on which forwarding decisions are based. The BSS forwarder keeps track of the creation times of the bundles flowing from node X to node Y, at every “hop” along the end-to-end path. Each of the forwarder's neighbors must have two inducts, one for a "best-effort" convergence-layer protocol, such as UDP and another for a reliable convergence-layer protocol, such as TCP. BSS selects the appropriate outduct for forwarding the bundle pending dispatch by applying the following rule: “Each bundle whose creation time is greater than that of all other bundles seen on this stream is forwarded via the ’best-efforts’ outduct. All others are forwarded via the ‘reliable’ outduct.” A prerequisite of BSS is that every bundle sent by a BSS-enabled node has to be custodially acknowledged. If a bundle's custody-accepted signal does not arrive prior to timeout, the bundle is re-forwarded. Since the creation time of a re-forwarded bundle cannot be greater than that of all other bundles previously forwarded in this particular stream, it is instead forwarded to the reliable induct of the next neighbor. In our approach BSS forwarder needs to run in every intermediate node of the end-to-end path. However, the potential absence of a BSS forwarder does not pose any serious problem. The BSS-unaware forwarder simply does not identify the flow of bundles as BSS traffic and treats them as normal traffic; at worst it may impose some additional latency. That is, BSS forwarding has the potential to enrich streaming services but its partial deployment does not really cause any damage. The design of BSS ensures that all bundles in the stream are delivered eventually to their final destination, and all bundles that were never lost or retransmitted along the path are delivered in transmission order with minimum latency. The flow of streaming data never waits for a retransmission to succeed since such an action can degrade the viewing experience of a participant in a data streaming session. In the event that a bundle sent over the non-reliable convergence-layer protocol does not arrive at its next-hop node, that bundle simply will not be included in the flow of streaming data displayed at the destination. Gaps might appear in the display but never any regressions. After a lost bundle does successfully get re-forwarded, it is identified as out-of-stream since its creation time is earlier than that of subsequently forwarded bundles. It eventually ends up at the destination, but because it is out-ofstream, it does not get included in the displayed flow of streaming data; it just goes into the database, along with all of the successfully streamed data that has already been displayed. This enables the user to employ the playback features of the streaming service library to rewind back through the database and replay the data that were missed in sequence with the data that originally were successfully received. Like to the forwarder, the real-time display must not wait for retransmission of lost bundles, because that would either delay the delivery of subsequently transmitted data or else result in the presentation of out-of-order data in the streaming display. The receiver’s application is built using a BSS library. Our intention is to enable applications to pass streaming data received in transmission time order to an application-specific "display" function but also to store all received data (including out-of-order data) in a private database for playback under user control. The reception and real-time display of in-order data is performed by a background thread, leaving the application's main thread free to respond to user commands controlling playback or other application-specific functions. Whenever the background thread receives a bundle, it inserts it into the BSS database in creation-time order for replay on demand. It also checks bundle creation time in order to decide whether to pass the bundle to a callback function for real-time display or to some other function for further stream processing. Meanwhile, the main thread of the application may respond to user commands by calling BSS library functions. For example, library functions may retrieve data from the time-ordered database and allow operations such as forward, backward, fast-forward, freeze etc. The result of the above-described process is unimpeded real-time streaming of all data delivered in transmission order, together with comprehensive replay and review of all the data in the stream.
ETT-13-0301
6
4. USE CASES AND RELATED SCENARIOS Although BSS could be a useful tool for various use cases with notable impact on the user’s viewing experience, the streaming of data from Space to Earth is the most appropriate use case to demonstrate its potential. This is justified primarily by the specific application and networking requirements of Space communications that frequently suffer severe network delays and disruptions. However, BSS could also demonstrate its potential in highly-stressed terrestrial networking environments, such as those where tactical military communications are deployed [25]. We designed a set of flexible scenarios based on realistic use cases that allow us to evaluate BSS performance under various communication patterns. These scenarios were built upon a diverse sample of stressed networking environments, including both terrestrial and Space network topologies. This sample incorporates various communication link properties and a wide range of operational conditions. The Space networking scenarios used in our emulations are classified in three major categories, as shown in Fig. 2. The first and most common scenario involves streaming data between a Space asset in close proximity with Earth and a Mission Operation Center (MOC) on Earth. The topology comprises a rover on Moon, a relay satellite which is orbiting Moon and a ground station on Earth. Given the fact that the round trip communication delay between Earth and Moon is less than three seconds we set the propagation delay for the Moon-relay communication link at 500ms and for the relay-Earth communication link at 1s.
(a) (b) (c) Fig. 2. All three network topologies employed in Space scenarios. a) Moon b) Spacecraft and c) Mars.
In the second category, we consider a Deep Space communication scenario where a distant Spacecraft sends a stream of data to the MOC on Earth through an intermediate Space asset, which could be another Spacecraft or space station. The calculation of the minimum time that a signal needs to travel a Deep Space distance 2 results in a rough value of 7 seconds. In this scenario an arbitrary propagation delay value of 20 seconds per link was set to emulate networking environments with propagation delays well beyond 7 seconds. The last Space scenario reflects the topology and the operations of the ongoing exploration missions on Mars. A rover on Mars streams data to a ground station on Earth via a relay satellite orbiting Mars (Fig. 2 - c). The propagation delay for the long-haul links between Mars and Earth varies from 3 to 20 minutes, since the distance between the two planets varies as they move in their orbits. In order to have a reasonable emulation time interval that allows adequate experiment repetitions to be conducted, we assume for the specific scenario a total average end-toend propagation delay of 5 minutes that remains stable for the total duration of the data streaming session. The BER values used in our experiments were extracted from descriptions of various communication link properties and data found in Space communications literature. More specifically, the most frequently quoted BER values are on the order of 10-5 and 10-6 [27]; therefore, BER values in between this range were applied. In that context, the selection of the proper BER value for each communication link was mainly influenced by the distance that a signal has to travel as well as the specific technological properties of the communication equipment used in each case. At this point, it would be also useful to note that the applied BER values refer to errors after applying error correction codes and hence reflect the BER values experienced by the Bundle layer, the rate at which uncorrectable bit errors are detected. Our ultimate aim was to emulate the properties of actual Space communication topologies with a sufficient level of accuracy while retaining at the same time a high level of flexibility in the scenario development and configuration process. For the first scenario, based on the short distance between the participating objects in terms of Space communication ranges, we set BER at 10-6 for every communication link. The second scenario was more complicated: the low-power antennas of the Spacecraft justify a BER value of 10-5 for the communication link between the two Spacecraft while a 10-6 BER value is more appropriate for the communication link between Earth and the intermediate Spacecraft, given the increased capabilities of the equipment installed on Earth. For example, recall that the Deep Space Network is equipped with massive antenna arrays that are able to enhance the reception of low power/weak signals. Finally in our third scenario, which is related to interplanetary 2
According to the official definition given by ITU, Deep Space starts at a distance of 2,000,000 km from the Earth's surface [26].
ETT-13-0301
7
communications, we set the BER between the rover landed on Mars and the relay satellite at the value of 10-6 which is appropriate given the relatively short distance across these two objects. Due to the long-haul distance between the relay satellite and Earth, we set the value of BER to 10-5 for the communication link connecting the two entities. Our terrestrial scenarios do not capture the wide diversity of parameters which can be encountered in various networking environments. Instead, we couple our extensive Space scenarios with an abstract terrestrial scenario consisting of a generic, symmetric, 3-node string topology (Fig. 3). We derived three distinctive subcases depending on the magnitude of propagation delay alone. Each subcase was named accordingly as: i) Normal, which includes propagation delays of few milliseconds that are typical in LAN topologies, ii) Low, which includes propagation delays in the order of tens of milliseconds, typical in well-interconnected networks and iii) High, which includes propagation delays in the order of hundreds of milliseconds, encountered usually in highly-stressed networking environments such as the ones established in battlefields and civil emergency situations. In each case, we assume that BER remains stable at 10-5 as a result of transmission rate adaptation mechanisms employed by IEEE802.11 protocol to provide a steady minimum transmission rate, sufficient for multimedia streaming.
Fig. 3. All three network properties classes of the topology employed in terrestrial scenarios
Furthermore, in order to evaluate the impact of the number of hops per se on BSS performance, we developed three additional network topologies, each consisting of 5 nodes; the end-to-end propagation delay of the 5-node network topologies remains the same as for the corresponding 3-node topologies while BERs are distributed as depicted in Fig. 4.
(a) (b) (c) Fig. 4. BER and Prop. Delay distribution on each 5-node network topology: a) Spacecraft, b) Mars and c) Terrestrial.
The scenario design phase was completed with the development of a reference scenario for each category. For all categories we used as reference the applicable scenarios with zero BER. This allows us to compare the base performance of the two DTN software architectures (default versus BSS-enabled), as well as the progressive impact of various conditions on the streaming procedure. Furthermore, the reference scenarios have also served as verification tools for the software architectures and the operating system settings. 5. PERFORMANCE EVALUATION The objective of this study was to investigate the benefits of BSS forwarding especially in highly-stressed networking environments. While long communication disruptions due to line-of-sight-blocking, mobility, coverage holes or poor channel conditions are quite frequent in DTN environments, here we focus on the impact of lengthy signal propagation delays as well as transient disruptions. The rationale behind our strategy is straightforward: the objective of BSS is not to provide a universal solution by ensuring successful streaming communication no matter how severe the disruption, which at the limit is impossible, but rather to enhance the streaming data delivery performance under conditions that stop short of permanent connectivity loss. Our experiments were divided into two main performance evaluation sets: i) content-insensitive network performance and ii) multimedia content delivery performance. In both sets, the performance of BSS forwarder (bssfw) is compared with the performance of IPN forwarder (ipnfw); IPN forwarder is ION’s default forwarder for forwarding bundles with endpoint identifiers conforming to the IPN naming scheme. In general, the functionality of all forwarders employed so far by the most commonly adopted DTN platforms (ION, DTN2, IBR-DTN etc.) is identical and limited to simply computing proximate nodes based on the existing routing plan and handling different
ETT-13-0301
8
packet priorities. Evaluating BSS in comparison to other data-streaming approaches that include routing and FEC techniques is beyond the scope of this paper since BSS framework does not constitute an alternative solution; on the contrary, it focuses on enhancing the forwarding procedure while at the same time, due to its design philosophy, promotes the application of complementary mechanisms. Each experiment was repeated as many times as it was deemed appropriate in order for the standard deviation of the results to lie within the 95% confidence interval. A. Testbed Environment and Configuration The emulation of the networking conditions described in section 4 was accomplished through the use of the ESA DTN testbed [28] established in the Space Internetworking Center [29]. The testbed enables emulation of current and future DTN-based terrestrial and Space communication scenarios. It uses the Linux kernel’s embedded Network Emulator tool and the One Way Light Time Transmission Delay Simulator tool provided by ION to emulate, respectively, terrestrial and Space networking conditions. Those tools are able to regulate propagation delay and packet loss and reliably emulate channel conditions. We used equation (1) to calculate packet error rate values (PER) from the BER values presented in section 4 and the bundle sizes. We further adjusted the calculated PER values to reflect the difference in granularity between the two tools. PER =1− (1− BER)s , where s = 8* Bundle Size in bytes
(1)
Equation (1) inherently assumes that independent bit errors and a Gaussian PER distribution model accurately emulate channel noise in this environment. Given the wireless nature of the communication links described in our scenarios, one might argue that the Gilbert-Elliot error model is perhaps more appropriate since it also accommodates burst errors, which are typical in wireless communication environments. The counter-argument would be that in order to emulate Bundle layer noise reliably one must remember that burst error rates at the link layer do not always translate into burst bundle losses. Moreover, our environment includes transient disruptions that produce some of the same effects as bursts of bit errors. In this context, we believe the Gilbert-Elliot model does not reflect well the nature of the environment in which we conducted our experiments. However, this analysis may in the future require a study on each own right. Another issue of concern was the applicability of our PER computations in an environment of asymmetric network links. When a bundle is forwarded through an unreliable transport layer, such as UDP, it is encapsulated in its entirety within a single datagram, while when the same bundle is forwarded through a reliable transport protocol it is fragmented into smaller entities, e.g. TCP segments. In order to evaluate the potential lack of accuracy that would result from applying a uniform PER model to these two distinct cases, we conducted a simple worst-case analysis [30] which showed that the loss in accuracy is minimal. We therefore applied a uniform PER model for our experiments and extended the error margin of our experiments to one RTT (round-trip time). The emulation of the remaining elements, including network protocol stack configurations and streaming applications, was based on the ION platform, version 3.1.1. As an implementation of the DTN architecture, fully conforming to RFCs 5050 and 5326, ION includes implementations of all the necessary components used in the present study: BP, the Licklider Transmission Protocol (LTP [31]) and both BSS and IPN forwarder. Since both terrestrial and Space environments were emulated, two network stack configurations were required. The “terrestrial” stack employs Internet-based network protocols – TCP/IP and UDP/IP – at the convergence layer. The “Space/Deep Space” stack utilizes LTP at the convergence layer: “green” LTP transmission functions as the “best-effort” service while “red” LTP transmission provides the “reliable” service. In Fig. 5 the complete stack of networking protocols used in our experiments is depicted. We also implemented two simple streaming applications in order to emulate the behavior of actual streaming applications. These applications are designed to acquire accurate timing measurements for the dispatch and reception of all “media” bundles. bssStreamingApp simulates the functionality of a media source that produces a stream of 30 frames per second (fps). It wraps the produced stream in “media” bundles of user-defined size and hands them to the bundle layer for transmission. In our experiments, each bundle’s payload is a multiple of 1316 bytes in order to simulate the behavior of the Packetized Elementary Stream mechanism [32], which is used by popular media streaming applications such as VLC [33]. At the other end, the bssRecv simulated media sink is based on the API functions provided by the BSS library. It presents two basic functionalities; it immediately displays any in-order “media” bundles arriving at the destination, and it saves in a specially-designed database all “media” bundles of the received stream, whether received in-order or out-of-order; the bundles are sorted by transmission
ETT-13-0301
9
order within the database. For the second set of experiments, since BSS can be used as a transfer medium for multimedia streaming, we also preliminarily tested its multimedia performance. For that purpose, we modified bssStreamingApp and bssRecv to act as middleware between the well-known media player VLC and ION.
Fig. 5. Representation of the network stack used in both configurations.
Throughout the whole experimental procedure the transmission ratio achieved by bssStreamingApp remained constant at 1Mbps, which is considered adequate to simulate the transmission of a H.264 352×
[email protected] video quality stream. In each case, we assume that for both terrestrial and Space communication links the available bandwidth suffices to cover the transmission rate needs of bssStreamingApp. Finally, the BSS forwarder applies an expiration timer to control the retransmission of lost bundles. This timer is set equal to the RTT value of the communication link plus a small safety margin in order to avoid any spurious timeouts resulting in unnecessary retransmissions that could lead to the under-utilization of the available bandwidth. B. Evaluation Metrics In the following, we detail the evaluation metrics utilized in the paper. Since we are evaluating both the networking and multimedia performance of BSS, we selected performance and quality metrics in each set of experiments that would help us to accurately assess the efficacy of BSS and evaluate the impact of different network properties, bundle size and hop count on BSS performance. In order to evaluate the networking performance of BSS we defined the following metrics: i) Stream Delivery Efficiency (SDE) is defined as the number of bundles received from the final recipient of the stream in a specific period of time, starting from the time that the recipient received its first bundle; ii) Stream Delivery Attenuation (SDA) as the number of out-of-order bundles received from the final recipient of the stream in a specific period of time, starting from the time that the recipient received its first bundle and iii) Stream Delivery Time (SDT) as the time when the last packet of application data is delivered successfully at the receiver. Note that SDT identifies the time when the last missing segment arrives at the destination, which at the same time completes receipt of last missing DTN bundle. The multimedia performance of BSS is evaluated based on three metrics: i) End user’s Display Efficiency (EDE) as the number of successfully displayed video frames, ii) End user’s Display Attenuation (EDA) as the number of wrongly decoded video frames and iii) average PSNR along with actual snapshots of the video display. C. Results and Discussion Network Performance In the first phase of the evaluation we measure network performance, employing the SDE and SDA quality metrics and the SDT time metric. More precisely, SDT is used based on the outcome of the quality metrics in order to further assess the efficacy of the BSS forwarder. In Fig.6 we present the quality results for all three terrestrial scenario subcases presented in section 4. In our effort to assess the impact of different bundle sizes on the performance of the two forwarders, we also included two different payload sizes in our analysis: the commonly used value of 1316 bytes and its double, 2632 bytes. The interval at which the SDE and SDA metrics were monitored was set to 5 minutes, an adequate time period considering the relatively low propagation delay values presented by these network topologies. The source continuously transmits “media” bundles to the destination for an indefinite period of time. As the results suggest, BSS outperforms IPN forwarder in the majority of the cases, managing to deliver a larger number of bundles within the specified timeframe. In typical terrestrial networking environments, represented in our experiments by the Normal subcase, both forwarders achieve equal performance, successfully delivering the expected number of bundles to the final destination. However, in highly-stressed terrestrial networking environments
ETT-13-0301
10
represented in our experiments by the Low and High subcases, the SDE values indicate that the BSS forwarder minimizes TCP performance degradation and gains a noticeable advantage over the IPN forwarder.
(a) (b) Fig. 6. Comparison between BSS and IPN forwarder for the a) 3-node and b) 5-node network topology of all three terrestrial scenarios.
The comparative impact on streaming performance becomes even more significant as the payload size increases. The explanation is straightforward: smaller packets are less susceptible to channel noise than larger packets, so in the large-payload case, where more bundles are getting lost, IPN forwarder expends more for recovery. Unlike IPN forwarder, BSS manages to deliver higher volumes of bundles since it avoids at this stage all unnecessary retransmissions. However, we also have to note that BSS’s SDE improvements come at some cost. IPN forwarder naturally achieves 100% in-order delivery of bundles since it relies wholly on TCP mechanisms; there is never any need for any bundle to be re-forwarded. The BSS forwarder must re-forward all bundles that were originally forwarded via UDP but were lost in transit. The net effect is that the BSS forwarder does less TCP transmission than IPN forwarder but must do additional bundle forwarding, ranging from 6% up to 40% more. This outcome is not really as dramatic as it initially appears; the overall performance impact of this additional transmission is low considering the performance cost savings of initial UDP forwarding compared to TCP transmission. In our view the more important point is that the result is a vast improvement in the total number of delivered bundles. The rest of the SDE and SDA quality metrics results are produced by the three Space scenarios and presented in Fig.7. Although the comparative evaluation results are similarly explained, our Space scenarios incorporate two distinct variations from the terrestrial scenarios. The long-haul propagation delay modeled in most Space scenarios suggested an increased sampling interval (10 minutes) for SDE and SDA metrics. This was deemed necessary in order to allow for sufficient time for the participating nodes to complete at least one communication round. Furthermore, Space links are typically associated with low data rates, such that larger bundle payloads are preferable in order to avoid unnecessary overhead. Therefore the examined payload sizes in our experiments were 2632 bytes and 3948 bytes. From the experimental results (Fig. 7), it is clear that BSS achieves equal or better performance than IPN forwarder in all cases and for both variants of the topology. In particular, an improvement ranging from 20% up to 100% is observed in SDE values for both Spacecraft and Mars scenarios. Beyond the improvement in the delivery efficiency of bundles, the BSS forwarder manages also to increase the quality of the received stream by significantly decreasing the value of SDA in all cases. The difference in magnitude of the SDA values between the Spacecraft and Mars scenarios is due to the asymmetric topology of the Mars scenario, which incorporates a long-haul propagation link at the first hop of transmission and results in fewer bundles reaching the intermediate node, thus decreasing the total number of retransmissions; this results in fewer out-of-order packets received at the final destination.
ETT-13-0301
11
(a) (b) Fig. 7. Comparison between BSS and IPN forwarder for the a) 3-node and b) 5-node network topology of all three Space scenarios
However, fewer bundles were delivered in total than expected in the Spacecraft and Mars tests using 2632-bytes payloads. This appears to be due to the heavy computational load imposed by LTP’s retransmission accounting in these tests: because the bandwidth delay products are large, a large number of small transmission segments must be tracked. We are considering several possible mitigations. SDT was then measured over a stream consisting of 5000 “media” bundles. The streaming session was deemed completed when all 5000 bundles were successfully received from the final node. The purpose of these experiments was to map in time the change in SDE values observed during the first set of experiments.
(a) (b) Fig. 8. Comparison between BSS and IPN forwarder based on SDT metric of a representative sample of cases from both terrestrial and Space scenarios.
BSS forwarder superiority was further confirmed by this second set of results (Fig. 8). Even with the additional error margin incorporated into our results, bssfw still outperforms ipnfw over the entire set of experiments. The reduction in the total requested time for successfully transferring 5000 frames at the destination node is significant for both terrestrial and Space scenarios, with a performance gain ranging from 20% up to 130%. Finally, based on the comparison of the results acquired respectively by the 3-node and 5-node topology variants, we confirmed that the addition of extra nodes in isometric distances along the end-to-end path of a long-haul communication link with stable BER has a positive effect on the performance of both forwarders. Multimedia Performance In our multimedia performance evaluation, the well-known VLC media software was used as the basic media platform for broadcasting and receiving an actual 2-minute video clip. Instead of simply using performance metrics
ETT-13-0301
12
such as EDE and EDA, our intention was to also include relevant multimedia metrics in the evaluation process, such as PSNR and snapshots of the real-time video display; therefore, an appropriate evaluation scenario had to be developed. The basic requirement for this particular scenario was to utilize a network topology with networking conditions that allow for capturing severely distorted real-time display images whose quality can be easily assessed by the human eye. Among the scenarios presented in section 4, the only scenario conforming to this requirement was the Moon scenario, which was selected for this evaluation. Based on the results presented in Fig. 9, it becomes apparent that BSS achieves a notable multimedia performance improvement by delivering a higher number of media bundles in the correct order. Practically, this leads to the successful display of almost 18% more video frames, based on the comparison of EDE values, and at the same time to a significant decrease in the number of wrongly-decoded frames. In general, PSNR values below 30dB suggest very poor image quality. This fact can be attributed to the high BER of the communication channel. Nevertheless, applying BSS resulted in the increase of PSNR value from 12dB to 14dB, a fact that further indicates an improvement in the real-time viewing experience in comparison to that offered by IPN forwarder.
(a) (b) (c) Fig. 9. Comparison between BSS and IPN forwarder based on a) EDE b) EDA and c) PSNR metrics for the 3-node network topology of the Moon Space scenario.
In Fig.10, we present an indicative sample of the actual real-time display images under comparison. Although both images are severely distorted, the quality of the first image enables the viewer to clearly identify the object in focus. Such a difference in image clarity could play a key role in real use cases such as the accurate assessment of a potential emergency situation from the end-user of a monitoring system.
(a) (b) Fig. 10. Video stream snapshots of a) BSS and b) IPN forwarder for the 3-node network topology of the Moon Space scenario.
6. CONCLUSIONS Based on extensive experiments, we assessed the potential contribution of the BSS framework to streaming data in delay tolerant networks. We evaluated both networking and multimedia issues and obtained a series of results that demonstrate a clear-cut advantage for BSS. The range of improvement observed depends on the particular network conditions. More specifically, we concluded that: 1) The BSS framework is most clearly advantageous for long-haul communications, especially between Deep Space and Earth. 2) The duality of the BSS forwarder, with both reliable and unreliable transmission capability, allows for a smoother near real-time streaming viewing experience. 3) The BSS framework is appropriate for use in both terrestrial and Space environments. 4) The BSS framework’s API facilitates the development of data streaming applications in delay tolerant networks. Our future plans include the investigation of BSS adaptability to multicasting communications and its overall
ETT-13-0301
13
evaluation in real Space environments. The results of such additional experiments might call for particular modifications in the BSS framework elements. Towards that direction, BSS is also freely distributed as open-source software [34] to allow easy access to other potentially interested parties. ACKNOWLEDGEMENT We are particularly thankful to the CCSDS – Space Internetworking Services Area (SIS) – DTN working group for their strong interest in testing BSS in actual Space environments using the International Space Station as the basis for the experiments. REFERENCES [1] Farrell S., Cahill V., Geraghty D., Humphreys I. and McDonald P. When TCP Breaks: Delay and Disruption Tolerant Networking. IEEE Internet Computing 2006; pp. 72-78. [2] Fall K. A delay-tolerant network architecture for challenged internets. In Proceedings of the 2003 conference on Applications, technologies, architectures, and protocols for computer communications; pp. 27-34. [3] Burleigh S. et al. Delay-tolerant networking: An approach to interplanetary Internet. IEEE Communications Magazine 2003; 41(6), pp. 128–136. [4] Cerf V. et al. Delay–Tolerant Networking Architecture. Internet RFC 4838. April 2007. [5] Scott K and Burleigh S. Bundle Protocol Specification. Internet RFC 5050. November 2007. [6] Jet Propulsion Laboratory. Strategic Technology Directions 2009. JPL Publication 400-1385. [7] Lenas S.-A., Burleigh S. and Tsaoussidis V. Reliable Data Streaming over Delay Tolerant Networks. In Proceedings of the 10th international conference on Wired/Wireless Internet Communication 2012; pp. 358-365. [8] Cerf V. et al. Interplanetary Internet (IPN): Architectural Definition. Internet Draft. August 2002. [9] Burleigh S. Interplanetary Overlay Network (ION) Design and Operation v1.13. JPL D-48259. Jet Propulsion Laboratory. California Institute of Technology. October 2011. [10] Qin M. and Zimmermann R. Improving Mobile Ad-hoc Streaming Performance through Adaptive Layer Selection With Scalable Video Coding. in Proceedings of 15th ACM Multimedia conference 2007; pp. 717 – 726. [11] Calafate C, Malumbres M., and Manzoni P. Mitigating the impact of mobility on H.264 real-time video streams using multiple paths. Journal of. Communication. and Networks 2004; 6(4), pp. 387–396. [12] Loscri V., De Rango F., and Marano S. Ad hoc on-demand multipath distance vector routing (AOMDV) over a distributed TDMA MAC protocol for QoS support in wireless ad hoc networks: Integration issues and performance evaluation. European Transactions on Telecommunications 2006; 18 (2), pp. 141–156. [13] Kambhatla K. et. Al. Wireless H.264 Video Quality Enhancement Through Optimal Prioritized Packet Fragmentation. IEEE Transactions on Multimedia 2012; 14(5), pp.1480-1495. [14] Mantzouratos S., Gardikis G., Koumaras H. and A. Kourtis. Survey of Cross-layer Proposals for Video Streaming over Mobile Ad hoc Networks (MANETs). In Proceedings of International conference on Telecommunications and Multimedia (TEMU ’12) 2012; pp.101-106. [15] Patra R. and Nedevschi S. DTNLite: A Reliable Data Transfer Architecture for Sensor Networks. Berkeley 2003; Tech. Rep. CS294–1. [16] Matarasso C., Giggenbach D., Hermanns F. and Lutz E. Multimedia Satellite Communications Experiments to International Space Station. International Journal. of Satellite Communications 2002; 20, pp. 333-345. [17] Wang S., Bian D., Jie X. and Liu Hong. The design and implementation of channel simulator for broadband satellite communication. International Conference on Computer Science and Service System (CSSS) 2011; pp.4093-4096. [18] Makovsky A., Ilott P. and Taylor J. Mars Science Laboratory Telecommunications System Design. Nov. 2009. [19] Liu T. and Nelakuditi S. Disruption-tolerant content-aware video streaming. in MULTIMEDIA ’04: Proceedings of the 12th annual ACM international conference on Multimedia 2004; pp. 420–423. [20] Tournoux P., Lochin E., Leguay J., and Lacan J. Robust streaming in delay tolerant networks in Proceedings of. IEEE ICC 2010; pp. 1–5. [21] Kumar N. and Kim J. Probabilistic trust aware data replica placement strategy for online video streaming applications in vehicular delay tolerant networks. Journal of Mathematical and Computer Modeling 2012; 58(12), pp 3–14.
ETT-13-0301
14
[22] Karlsson G., Lenders V. and May M. Delay-tolerant broadcasting. in Proceedings of. 2006 SIGCOMM workshop on Challenged networks 2006; pp. 197–204. [23] Panchakarla N. and Ott J. Delay-tolerant Adaptive Real-time Communication: A Case Study for Voice. in Proceedings of ExtremeCom ’12 2012. [24] Keller L., Le A., Cici B., Seferoglu H., Fragouli C. and Markopoulou A. Microcast: Cooperative Video Streaming on Smartphones. In Proceedings of the 10th international conference on Mobile systems, applications, and services (ACM MobiSys ’12); pp. 57-70. [25] Burbank J. K., Chimento P. F., Haberman B. K. and Kasch W. T. Key challenges of military tactical networking and the elusive promise of MANET technology. IEEE Communications. Magazine 2006; 44 (11), pp. 39–45. [26] International Telecommunications Union. 201, Rev. B: Frequency and Channel Assignments. December 15, 2009. [27] McGowan J. F. Video Technologies for Mars. Proceedings of the Second International Convention of the Mars Society 2000. [28] Samaras C. et al. Extending Internet into Space – ESA DTN Testbed Implementation and Evaluation. 1st Int'l Conference on Mobile Lightweight Wireless Systems, Workshop on Rsrch activities 2009; pp 397-404. [29] Space Internetworking Center. http://spice-center.org [30] Lenas S.-A. Worst-case analysis regarding the uniform application of packet error rate across variable sized bundles. http://utopia.duth.gr/~slenas/documents/BSS-WCA.pdf [31] Ramadas M., Burleigh S., Farrel S. Licklider Transmission Protocol - Specification. Internet RFC 5326. September 2008. [32] Cisco Corporation. Fundamental of Digital Video – Cisco. http://www.cisco.com/en/US/docs/solutions/Enterprise/Video/pktvideoaag.pdf [33] VideoLan Media Player. http://www.videolan.org/vlc/index.html [34] Interplanetary Overlay Network (ION). Jet Propulsion Laboratory. California Institute of Technology. CA. [online at http://ion-dtn.sourceforge.net] Sotirios-Angelos Lenas received his Diploma in Electrical and Computer Engineering from Democritus University of Thrace, Greece 2009 and a M.Sc in Computer Networks and Security in 2011, from the same department. Sotiris has also served as a visiting researcher at NASA’s Jet Propulsion Laboratory, investigating issues related to Delay Tolerant Networking (DTN). His research interests lie mainly in the areas of internetworking in DTN environments, network applications, network queue management and network security. In the past, he has participated in several DTN projects funded by the ESA. Currently, he is a PhD student under the advisory of Prof. Vassilis Tsaoussidis while at the same time, he works as a research engineer at Space Internetworking Center (SPICE). Scott Burleigh has been developing software at NASA’s Jet Propulsion Laboratory since 1986. He was a co-author of the specifications for the Consultative Committee for Space Data Systems (CCSDS) of the File Delivery Protocol (CFDP) and Asynchronous Message Service (AMS), and as a member of the Delay-Tolerant Networking (DTN) Research Group of the Internet Research Task Force he co-authored the specifications for the DTN Bundle Protocol (RFC 5050) and the DTN Licklider Transmission Protocol for delay-tolerant ARQ (RFC 5326). He has developed implementations of these protocols that are designed for deep space mission systems, aiming to enable deployment of a delay-tolerant Solar System Internet. Vassilis Tsaoussidis (M’96-SM’03) received a B.Sc. in Applied Mathematics from Aristotle University, Greece; a Diploma in Statistics and Computer Science from the Hellenic Institute of Statistics; and a Ph.D. in Computer Networks from Humboldt University, Berlin, Germany (1995). He held faculty positions in Rutgers University, New Brunswick, SUNY Stony Brook and Northeastern University, Boston. In May 2003, he joined the Department of Electrical and Computer Engineering of Democritus University, Greece, where he was elected full Professor in 2007. His research interests lie in the area of transport/network protocols, i.e., their design aspects and performance evaluation. He was/is editor for IEEE Transactions on Mobile Computing, the Journal of Computer Networks the Journal of Wireless Communications and Mobile Computing the Journal of Mobile Multimedia and the International Journal of Parallel, Emergent and Distributed Systems. He participated in several Technical Program Committees in his area of expertise, such as INFOCOM, GLOBECOM, ICCN, ISCC, EWCN, WLN, and several others.
