On the Scalability of P2P-Based Push-Driven Live Streaming Systems Cyril Cassagnes
Damien Magoni
Hyunseok Chang
Wenjie Wang
Sugih Jamin
University of Bordeaux University of Bordeaux Bell Labs University of Michigan University of Michigan Bordeaux, France Bordeaux, France Holmdel, NJ, USA Ann Arbor, MI, USA Ann Arbor, MI, USA
[email protected] [email protected] [email protected] [email protected] [email protected]
Abstract—Television transmitted over IP (IPTV) presents numerous opportunities for users as well as service providers, and has attracted significant interest from business and research communities in recent years. Among the emerging IPTV delivery architectures, the peer-to-peer based delivery mechanism is considered attractive due to the relative ease of service deployment. However, the question of how well P2PTV applications would support a growing number of users has not been fully investigated so far. In this paper, we try to address this question by studying scalability and efficiency factors in a typical P2P based live streaming network. Through the use of the data provided by a production P2PTV system, we carry out simulations whose results show that there are still hurdles to overcome before P2P based live streaming could become widely used.
I. I NTRODUCTION With the increasing broadband speed and continued improvement in video compression technologies, Internet-based television services (IPTV) have been experiencing sustained growth lately. When it comes to realizing IPTV services in today’s Internet, peer-to-peer (P2P) based delivery mechanism is considered an attractive option because of the ease of deployment and potential bandwidth savings. In a typical P2P based IPTV network, clients retrieve video streams by connecting to the broadcast server or other existing clients that are already connected to the network. The broadcast server generates packetized video streams by encoding live TV signals captured from satellite. After joining the network, clients can contribute their uplink bandwidth by forwarding the incoming video streams to later joining clients. To allow more efficient utilization of client’s uplink bandwidth, the video streams are typically distributed via the overlay in the unit of “chunks” (e.g., [1]) or “sub-streams” (e.g., [2], [3]). Chunks are time-divided segments of packetized streams, while sub-streams are space-divided subsets of the original streams (e.g., layers in H.264 SVC). The chunks or substreams are either pushed by forwarding clients, or pulled by receiving clients depending on the P2P sharing protocol used. In the pull-driven delivery, clients search and pull individual stream units in an opportunistic way, while in the pushdriven approach, a client establishes a “virtual” connection to a forwarding client, and continues to receive data pushed from the forwarder until either end terminates the connection. Push-driven delivery design was shown to be more efficient than pull-based counterpart in recent work [4].
Compared to traditional P2P data sharing or progressive streaming of video on demand, optimizing end-user experience in the P2P based live streaming environment is a non-trivial task because of its more stringent delay constraint and limited shared buffer space. In addition, upload capacity constraints and inherent churning behavior of participating clients can add to the difficulty in realizing a fully scalable delivery system. Motivated by our earlier study on the operational scalability of P2P based live streaming [5], we explore the impact of uplink bandwidth distribution and peer selection algorithms on the system performance. In our study, we focus on the push-driven, sub-stream based streaming architecture, and perform simulations instantiated with the data contributed by a production system employing such an architecture. The remainder of this paper is organized as follows. Section II outlines the previous work done on P2P based live streaming systems. Section III discusses the scalability of the redistribution in P2P based live streaming networks. Section IV proposes mechanisms for increasing efficiency of the peer selection algorithm in P2P based live streaming networks. Section V presents evaluation results obtained by simulating the architecture of one of the largest production P2PTV providers in Europe. We conclude the paper by presenting future research directions in the last section. II. R ELATED WORK Many multimedia streaming systems have been proposed and evaluated in the research community in recent years [6], [7], [8], [9], [10], [11]. Besides these research prototypes, a number of fully operational P2PTV systems have emerged (e.g., PPLive, PPStream, SopCast, TVAnts, LiveStation, Joost, and Zattoo). With the P2PTV’s growing popularity, a large number of measurement papers have been published on these P2P live streaming systems, tackling workloads [12], topologies [13] or specific systems [14], [15], [16], [17], [18], [19]. Numerous proposals for improving existing P2P-based live streaming systems have also been presented, ranging from feasibility [20] and upload capacity [21] to locality awareness [22] and even stochastic fluid theory for P2P streaming systems [23]. Many recent proposals based on robust incentives [24], altruism [25], contribution awareness [26] and substream trading [27] aim at avoiding free-riders in large scale systems.
III. R EDISTRIBUTION IN P2P LIVE STREAMING OVERLAYS In this section, we study the impact of client’s uplink bandwidth on the scalability of a P2P system (i.e., maximum number of users that can be supported by the system). In this section, we do not consider peer dynamics (e.g., joining/leaving behavior). We will incorporate peer dynamics in later sections. We define the ratio of the incoming stream that can be redistributed to other peers as a redistribution factor, and label it with k throughout this paper. The redistribution factor k can vary between 0 to infinity, and can be fractional, depending on peer’s uplink capacity. If k = 1, it means that the peer can redistribute the full stream. If k = 2, it means that the peer redistributes two copies of the stream. If k = 0.5, it means that the peer redistributes only half of the stream due to its uplink bandwidth constraint. Fractional values are possible because a full stream can be divided into multiple sub-streams, which allows a peer to redistribute only a subset of the stream to other peers. Technically, the redistribution factor of a peer can be estimated as the total upload bytes divided by the total download bytes of the peer. In this paper, we assume that the redistribution factor remains constant over time for any peer. A. Theoretical analysis First, we study the impact of the redistribution factor on system scalability in a theoretical perspective. As the overlay is a directed acyclic graph, we can define the depth of a peer in the overlay as the number of overlay links between itself and peer connected to the source. If we assume that all peers have the same redistribution factor k, we can determine the maximum number of peers in the overlay by using a simple geometric series expressed in k, the graph depth n, and the source capacity C. That is, if Un represents the number of peers at depth n and if U0 = C, then we have Un = k ×Un−1 . The total number of peers is thus equal to: Sn =
n i=0
Ui = C ×
1 − k n+1 1−k
(1)
The maximum number of peers able to connect to the system will depend on the value of k: • If k < 1 then the number of peers will converge to C , and thus the system will not scale in size. Un → 1−k • If k = 1 then the number of peers will linearly diverge; Un → +∞, and thus the system will scale in size. • If k > 1 then the number of peers will exponentially diverge; Un → +∞, and thus the system will scale in size. Considering a more general case where at a given level l, each peer i has an upload (uli ) and download (dli ) capacity rate. Then, the k factor for this level l containing p peers would be: p uli kl = i=1 (2) p i=1 dli We define an idle peer as one not downloading the full stream. We define an hyperactive peer as one having k > 1. In
the more general case, the redistribution factor k is expressed as: k=
Ua × (1 − Ri − Rh ) + Ui × Ri + Uh × Rh Da × (1 − Ri − Rh ) + Di × Ri + Da × Rh
(3)
Da : number of download sub-streams per active peer Ua : number of upload sub-streams per active peer • Di : number of download sub-streams per idle peer • Ui : number of upload sub-streams per idle peer • Uh : number of upload sub-streams per hyperactive peer • Ri : ratio of idle peers vs. total peers • Rh : ratio of hyperactive peers vs. total peers In an ideal world, we would have k ≥ 1 for all the peers and thus have no overlay size limit (although we could have an undesirably deep overlay if k is not much larger than 1). In reality, we have a much diverse set of redistribution factors. • k can be lower than 1 – Freeriders (k = 0) – Security blocked peers (k = 0) – Peers with low uplink capacity (k < 1) • k can be higher than 1 – Subsidiary sources, i.e., super peers (k >> 1) – Peers with high uplink capacity (k > 1) – Idle peers, i.e., peers that do not watch the stream but redistribute a subset of the sub-streams of the stream (k > 1 for some sub-streams) • •
B. Idle peers In order to overcome a low k factor, one solution is to take advantage of idle peers who could redistribute the stream in the background. Not watching the stream actively, idle peers could receive only a fraction of the full stream, and redistribute copies of this fraction, thus acting as a multiplier. Fig. 1 shows the P2P live streaming network capacity as a function of the percentage of idle peers. The number of sub-streams contained in the full stream is set to 16, same as the Zattoo overlay [5]. It is assumed that active peers have k = 0.5, and that idle peers redistribute one sub-stream (1/16th of the stream) four times (thus k = 4). To allow equal availability of different sub-streams in the network, each active/idle peer randomly chooses sub-stream(s) to redistribute. The plots in the figure are obtained by using formula 3. The network capacity is equal to the maximum number of peers in the overlay divided by the maximum number of peers that can connect to the source directly. If the network capacity equals to 1, it means that a peer can only connect to the source, but not to any other peers. If the network capacity equals to 10, it means that the overlay network can support ten times more peers than the capacity of the source. The figure shows both the active network and the idle network capacities. The plot labeled “active” is the number of peers in the network with k = 0.5. The plot labeled “total” is the sum of the number of idle peers and the number of active peers. It shows that idle peers help very little in increasing the total network capacity when idle peer percentage remains realistically low ( 10). Thus the use of idle peers may not help make a system scale in the real world. C. Hyper peers Another solution to overcome a low k factor is to use hyperactive peers for redistributing multiple streams. An hyperactive peer is a peer who is watching the stream and can redistribute multiple copies of the stream, or at least one full stream plus some additional sub-streams. Fig. 2 shows the streaming network capacity as a function of the percentage of hyperactive peers. Similar to the previous case, a full stream consists of 16 sub-streams. It is assumed that regular active peers have a k = 0.5, and that hyperactive peers have a k = 1.5. The plots in figure 2 are also obtained by using formula 3. The plot labeled “hyperactive” is the number of peers in the network with k = 1.5. The plot labeled “total” is the sum of the number of hyperactive peers and the number of active peers. We can see that there is divergence around at 60% of hyperactive peers. If there are less hyperactive peers, then they must have higher k values than 1.5 in order to reach such divergence. IV. E FFICIENCY OF THE PEER SELECTION ALGORITHM IN P2P LIVE STREAMING When a new peer joins an overlay, it performs its own peer selection algorithm to choose the target peers to connect to. If the peer selection is done randomly, the resulting overlay could become quite inefficient in two ways. First, the overlay could become too deep and not wide enough, thus incurring large playback lags. Secondly, the overlay could experience a lot of churns, thus incurring many interruptions for users. A. Effects for the overlay We would like to see whether it is possible to improve the efficiency of a streaming overlay by using peer selection algorithms that incorporate dynamic parameters such as upload capacities, session lengths, distances between peers and
Fig. 2.
Network capacity vs. amount of hyperactive peers.
overlay depth positions. The idea behind this proposal is that by placing more stable and high bandwidth peers closer to the source, it could make the overlay more efficient. The peer selection algorithm is a good place to influence the evolution of the overlay as we can more or less control where the peers will place themselves in the overlay. If a peer selection algorithm manages to put stable peers close to the source, this should reduce the overall churn in the overlay. Also, if this algorithm manages to put high bandwidth peers close to the source, this should increase the capacity of the overlay while keeping a reasonable depth for the overlay. We will study this approach by simulation in Section V.
B. Effects for a peer In reality, peers typically zap through channels and redistribute only a minimum of the stream in order to save their limited upload bandwidth. Such behavior is detrimental to the scalable growth of an overlay. So what can be the incentive for a P2P live streaming peer to behave otherwise? One solution is to employ a tit for tat mechanism with layered quality video as proposed in [28]. If layered quality video is not available, another solution that we propose is to negotiate the depth position in the overlay If the peer has a high uplink capacity and remains in the channel, it will gradually get closer to the source. In the end, the stable and high bandwidth peers would be exposed to less churns in their upstream connectivity and enjoy near real time TV, and therefore would be properly incentivized to stay in the channel. It has been shown in [16] that playback lag in a large scale P2PTV system is typically between 30 to 90 seconds. Thus, the peer position could be one feasible incentive for a stable and powerful peer to stay in the channel. In the simulation results shown in Section V, we use those criteria (e.g., session length and uplink capacity) to see if the overlay would perform better when the peers are selected by looking at such criteria.
V. S IMULATION OF A P2P LIVE STREAMING OVERLAY After having studied some theoretical aspects of a P2P streaming overlay and having proposed enhancement mechanisms, we now present results obtained by simulation and highlight the impact of peer selection algorithms on the performance of a typical P2P live streaming system. A. Simulation parameters and metrics Our simulation code implements Zattoo’s push-driven substream based streaming architecture [3], and was run on top of the network manipulator (nem) software [29]. An Internet map of 4.2k-node was used as the underlying topology [30]. We assume that a given overlay distributes one TV channel, and do not take daily channel size variations into account. Each run lasts for 12 hours and only the last 6 hours are analyzed; after the 6th hour, we are in a steady state regime. Each result value is the average of 30 runs, and the standard deviation values are provided. For all our simulations, we use the same parameters used in our previous work [5], which is based on the analysis of 9.6M sessions recorded by the Zattoo’s P2PTV during a 2-week campaign. According to the session data, the cumulative distribution function (CDF) of the redistribution factor k follows an exponential distribution. 50% of the peers can redistribute less than 50% of the full stream (i.e., k < 0.5). 82% of the peers can redistribute less than the full stream (i.e., k < 1). The uplink capacity of individual peers is assigned so that the resulting distribution becomes the same as the empirical distribution. The NAT type of a peer, which determines its reachability in the overlay, is also taken from the empirical distribution of NAT types reported in [5]. Finally, session’s inter-arrival time and session length are all instantiated from the corresponding exponential distributions reported in [5]. We vary the channel size from scarce mode (close to source capacity) to heavily crowded mode (several times higher than source capacity). Table I shows the remaining input parameters. TABLE I S IMULATION PARAMETERS Parameters Number of sub-streams per stream) Source capacity Search period Maximum search attempts Join timeout period Maximum size of candidate peer list Number of simulation runs per scenario
Values 16 50 clients 2 sec 2 0.25 sec 40 peers 30
In order to assess the performances of the P2P live streaming system, we study the following output metrics: 1) Viewing time ratio (in % = 100 * peer view time / peer life time): average viewing time of the peers that ended during this period (the higher the better).
2) Percentage of kicked out peers (in %): 100 x total number of peers that could not connect to the P2P live streaming overlay network during the given period divided by the number of new peers per period (the lesser the better). 3) Average number of interruptions per peer: total number of video viewing interruptions for all peers in the given period divided by the number of new peers per period (the lesser the better). We analyze these output metrics by varying the number of newly arriving peers per hour, which defines the stress put on the source and its P2P network. B. Simulation results This section presents our simulation results. A peer trying to connect to other peers to get all necessary sub-streams is called an orphan peer. It sends search messages to discover other peers, sends join messages to connect to available peers, and finally obtains the full stream from them. A peer who is able and willing to offer a part of or all requested substreams for an orphan peer is called an adoptive peer. An adoptive peer has positively answered to the search message of a orphan. Once having multiple positive answers from candidate adoptive peers, an orphan has to choose to which peer it should send a join message. We evaluate the following peer selection algorithms described in Section IV: • Random: an orphan peer tries to connect to a randomly selected adoptive peer. • Local: an orphan peer tries to connect to its closest adoptive peer (the distance being measured in hops). • Upload: an orphan peer tries to connect to the adoptive peer proposing the highest upload amount (measured in number of sub-streams). • Uptime: an orphan peer tries to connect to the adoptive peer having the highest session length as it has a higher probability of staying longer in the overlay (i.e. not switching channels). Figure 3 shows the average peer viewing time as a function of the traffic load. We can see that the effects of the various algorithms on the viewing time do not make much difference compared to a random selection. Although there is 7% difference between the worst and the best algorithms at a traffic load of 2000 new peers per hour, and 5% difference at a load of 4000, these values are not tremendous. This somewhat unexpected result implies the relative importance of the user level parameters such as upload capacity and session length over the system parameters such as the peer selection algorithms. Fig. 4 shows the average percentage of peers kicked out as a function of the traffic load. By comparing with the previous results, the decrease in viewing time is mainly due to peers being kicked out of the network. Only a small percentage is caused by the interruptions due to peer de-connections and reconnections. Fig. 5 shows the average number of interruptions per peer as a function of the traffic load. We observe that the average
Fig. 3.
Average viewing time vs. traffic load.
Fig. 5.
Average number of interruptions per peer vs. traffic load.
1.5 minutes. This creates a lot of churns that render the evolution of the overlay hard to control over time. Second, the redistribution factor distribution is heavily lopsided towards small values. 50% of the peers have an upload capacity lower than 50%. Thus, peers with long sessions may have a low upload capacity and not be so useful. All in all, these two factors weigh much more heavily on the viewing time than the various selection algorithms. When traffic load is high, the algorithm that performs better than random selection is the “local” algorithm. VI. C ONCLUSION
Fig. 4.
Average percentage of peers kicked out vs. traffic load.
number of interruptions per peer gradually increases when the number of peers increases until reaching a plateau for 2000 new peers per hour and above. When the number of new peers increases, the overlay grows and the average churn rate become higher, and thus leading to more connections and reconnections. However, when the overlay is getting saturated by the new peers, those new peers cannot manage to join the overlay and are kicked out. Thus, the number of connections and reconnections does not grow as much, because those kicked out peers do not significantly contribute to this number. However, the total number of peers still increases, and thus the ratio does not increase anymore. These results were not expected when we devised these algorithms. Two remarks can explain the difficulties for the alternative algorithms to make a difference compared to the random selection algorithm. First, the sessions are typically short-lived. Roughly 50% of the sessions are shorter than
In this paper we have studied the behavior of a push-driven, sub-stream based live streaming system. We have conducted a theoretical study concerning the impact of the redistribution factor, and have shown that the scalability of the system strongly depends on the available upload capacity of its peers. We also have shown that the use of idle peers as well as hyperactive peers is not a fundamental solution to make the system scalable. We have proposed possible improvement of the peer selection process as the selection process has a direct impact on the structure and efficiency of the overlay. To evaluate its potential impact, we have carried out simulation experiments in order to measure the efficiency of various peer selection algorithms under a heavily loaded live streaming system. We have instantiated our simulation with realistic parameters derived from the data from 9.8M sessions collected by the professional-grade Zattoo P2PTV system. Our results show that the redistribution factor and the session length have a profound effect on the maximum capacity of the P2P overlay, and that the various selection algorithms play a relatively marginal role in improving system scalability. Future work will be aimed at studying other improved peer selection algorithms as well as studying the impact of the buffer size and the peer search parameters on the overall efficiency of a P2P-based live streaming system.
R EFERENCES [1] L. Bracciale, F. L. Piccolo, D. Luzzi, S. Salsano, G. Bianchi, and N. Blefari-Melazzi, “A Push-based Scheduling Algorithm for Large Scale P2P Live Streaming,” in Proceedings of the 4th International Telecommunication Networking Workshop on QoS in Multiservice IP Networks, 2008, pp. 1–7. [2] S. Xie, B. Li, G. Y. Keung, and X. Zhang, “CoolStreaming: Design, Theory, and Practice,” IEEE Trans. on Multimedia, vol. 9, no. 8, December 2007. [3] H. Chang, S. Jamin, and W. Wang, “Live Streaming Performance of the Zattoo Network,” in Proceedings of Internet Measurement Conference, November 2009. [4] M. Zhang, Q. Zhang, and S.-Q. Yang, “Understanding the Power of Pullbased Streaming Protocol: Can We Do Better?” IEEE JSAC, vol. 25, no. 9, pp. 1678–1694, 2007. [5] K. Shami, D. Magoni, H. Chang, W. Wang, and S. Jamin, “Impacts of Peer Characteristics on P2PTV Networks Scalability,” in Proceedings of the 28th IEEE Conference on Computer Communications – MiniConference, April 2009. [6] D. Tran, K. Hua, and T. Do, “ZIGZAG: An Efficient Peer-to-Peer Scheme for Media Streaming,” in Proceedings of the IEEE Infocom, 2003. [7] R. Rejaie and S. Stafford, “A Framework for Architecting Peer-to-Peer Receiver-Driven Overlays,” in Proceedings of the ACM NOSSDAV, 2004, pp. 42–47. [8] X. Liao, H. Jin, Y. Liu, L. Ni, and D. Deng, “Anysee: Peer-to-Peer Live Streaming,” in Proceedings of the IEEE Infocom, 2006. [9] X. Zhang, J. Liu, B. Li, and T.-S. Yum, “CoolStreaming/DONet: A Datadriven Overlay Network for Live Media Streaming,” in Proceedings of the 24th IEEE Infocom, 2005, p. 21022111. [10] F. Pianese, “PULSE: A Flexible P2P Live Streaming System,” in Proceedings of the 9th IEEE Global Internet Symposium, 2006. [11] F. Pianese, D. Perino, J. Keller, and E. Biersack, “PULSE: An Adaptive, Incentive-Based, Unstructured P2P Live Streaming System,” IEEE Transactions on Multimedia, vol. 9, no. 6, 2007. [12] K. Sripanidkulchai, B. Maggs, and H. Zhang, “An Analysis of Live Streaming Workloads on the Internet,” in Proceedings of the ACM IMC, 2004, p. 4154. [13] C. Wu, B. Li, and S. Zhao, “Magellan: Charting Large-Scale Peer-toPeer Live Streaming Topologies,” in Proceedings of the 27th International Conference on Distributed Computing Systems (ICDCS’07), 2007, p. 62. [14] M. Cha, P. Rodriguez, S. Moon, and J. Crowcroft, “On Next-Generation Telco-Managed P2P TV Architectures,” in Proceedings of the 7th International Workshop on Peer-to-Peer Systems, 2008. [15] S. Xie, G. Keung, and B. Li, “A Measurement of a Large-Scale Peer-toPeer Live Video Streaming System,” in Proceedings of the International Conference on Parallel Processing Workshops (ICPPW’07), 2007, p. 57. [16] X. Hei, C. Liang, J. Liang, Y. Liu, and K. Ross, “A measurement study of a large-scale p2p iptv system,” IEEE Transactions on Multimedia, vol. 9, no. 8, 2007. [17] L. Vu, I. Gupta, J. Liang, and K. Nahrstedt, “Measurement and Modeling of a Large-scale Overlay for Multimedia Streaming,” in The Fourth International Conference on Heterogeneous Networking for Quality, Reliability, Security and Robustness, 2007. [18] S. Ali, A. Mathur, and H. Zhang, “Measurement of Commercial Peerto-Peer Live Video Streaming,” in Proceedings of the ICST Workshop on Recent Advances in Peer-to-Peer Streaming, 2006. [19] G. Marfia, G. Pau, P. D. Ricoy, and M. Gerla, “P2P Streaming Systems: A Survey and Experiments,” in Proceedings of the STreaming Day, 2007. [20] K. Sripanidkulchai, A. Ganjam, B. Maggs, and H.Zhang, “The Feasibility of Supporting Large-Scale Live Streaming Applications with Dynamic Application End-points,” in Proceedings of the ACM SIGCOMM, 2004. [21] J. Douceur, J. Lorch, and T. Moscibroda, “Maximizing Total Upload in Latency-Sensitive P2P Applications,” in Proceedings of the 19th ACM SPAA, 2007, pp. 270–279. [22] F. Pianese and D. Perino, “Resource and Locality Awareness in an Incentive-based P2P Live Streaming System,” in Proceedings of the ACM P2P-TV, 2007. [23] R. Kumar, Y. Liu, and K. Ross, “Stochastic Fluid Theory for P2P Streaming Systems,” in Proceedings of the IEEE Infocom, 2007.
[24] M. Feldman, K. Lai, I. Stoica, and J. Chuang, “Robust Incentive Techniques for Peer-to-Peer Networks,” in Proceedings of the 5th ACM Conference on E-Commerce, 2004, pp. 102–111. [25] Y. Chu and H. Zhang, “Considering Altruism in Peer-to-Peer Internet Streaming Broadcast,” in Proceedings of the ACM NOSSDAV, 2004. [26] Y.-W. Sung, M. Bishop, and S. Rao, “Enabling Contribution Awareness in an Overlay Broadcasting System,” in Proceedings of the ACM SIGCOMM, 2006. [27] Z. Liu, Y. Shen, K. Ross, S. Panwar, and Y. Wang, “Substream Trading: Towards an Open P2P Live Streaming System,” in Proceedings of the IEEE International Conference on Network Protocols, 2008, pp. 94–103. [28] Z. Liu, Y. Shen, S. S. Panwar, K. W. Ross, and Y. Wang, “Using Layered Video to Provide Incentives in P2P Live Streaming,” in Proceedings of the ACM Workshop on Peer-to-peer streaming and IP-TV, 2007, pp. 311–316. [29] D. Magoni, “Network Topology Analysis and Internet Modelling with Nem,” International Journal of Computers and Applications, vol. 27, no. 4, pp. 252–259, 2005. [30] D. Magoni and M. Hoerdt, “Internet Core Topology Mapping and Analysis,” Computer Communications, pp. 494–506, 2005.
