SmoothCache 2.0: CDN-quality Adaptive HTTP Live Streaming on Peer-to-Peer Overlays Roberto Roverso
Riccardo Reale
Sameh El-Ansary
Hive Inc. Stockholm, Sweden
Hive Inc. Stockholm, Sweden
Hive Inc. Stockholm, Sweden
[email protected] [email protected] [email protected] Seif Haridi Royal Institute of Technology-KTH Stockholm, Sweden
[email protected] ABSTRACT
at large scale. Work in this area has produced a number of systems such as PPlive [8], CoolStreaming [26], WineStreamer [22], and more recently Bittorrent live[7]. In general, state-of-the art P2P live streaming systems are designed around a set of assumptions derived from the standard RTSP /RTP protocol. RTSP/RTP is based on a push model, where the player requests a certain stream and the streaming server pushes content over UDP to the player. In this setting, the player is a mere renderer of the content, therefore it can be easily manipulated by the P2P system by pausing or skipping, in case of failed delivery by the overlay, or by feeding to the player the quality of the stream (bitrate) that the P2P system is able to deliver. Lately, however, the industry has adopted a new set of streaming protocols, generally classified as adaptive HTTP streaming, that deviate completely from the RTP/RTSP approach. Those make use of HTTP as transport [5], which allows simple and scalable video distribution by enabling CDNs to cache video content in the same way as any other HTTP content. On top of that, adaptive HTTP streaming protocols are designed to dynamically adapt the streaming rate (bitrate) both to the available bandwidth in the network and the capabilities of the playback device that the player is running on. Differently from the RTSP/RTP protocol, adaptive HTTP streaming protocols place the responsibility of managing the delivery of the stream on the player rather than the server. It is in fact the player that is pulling the content from the CDN issuing HTTP requests. The pace and bitrate at which the content is fetched are determined by the player with a complex set of heuristics that take into account many metrics, such as CDN throughput, network delay and rendering capabilities of the host. Due to the complexity of these heuristics and the fact that most of the player implementations are actually closed-source, it is unfeasible for a P2P system to control the delivery of the stream by steering the behavior of the player. As a result, the current state-of-the art approaches, which are designed around RTSP/RTP and therefore rely on the assumption of being able to easily manipulate the player, cannot be applied to adaptive HTTP streaming. However, being an industrial entity, we are concerned with offering a scalable P2P solution for live streaming that not only provides significant cost reductions for content providers,
In recent years, adaptive HTTP streaming protocols have become the de-facto standard in the industry for the distribution of live and video-on-demand content over the Internet. This paper presents SmoothCache 2.0, a distributed cache platform for adaptive HTTP live streaming content based on peer-to-peer (P2P) overlays. The contribution of this work is twofold. From a systems perspective, to the best of our knowledge, it is the only P2P platform which supports recent live streaming protocols based on HTTP as a transport and the concept of adaptive bitrate switching. From an algorithmic perspective, the system describes a novel set of overlay construction and prefetching techniques that realize: i) substantial savings in terms of the bandwidth load on the source of the stream, and ii) CDN-quality user experience in terms of playback latency and the watched bitrate. In order to support our claims, we conduct a methodical evaluation on thousands of real consumer machines.
Categories and Subject Descriptors C.2.2 [Computer-Communication Networks]: Network Protocols; C.2.4 [Computer-Communication Networks]: Distributed Systems
Keywords Content Delivery Network, peer-to-peer, Adaptive bitrate HTTP streaming
1.
INTRODUCTION
Peer-to-peer (P2P) has been extensively evaluated as a way to decrease costs when distributing live video streams Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from
[email protected]. MMSys ’15 March 18 - 20, 2015, Portland, OR, USA Copyright 2015 ACM 978-1-4503-3351-1/15/03...$15.00 http://dx.doi.org/10.1145/2713168.2713182.
61
but that also takes into account the realities of the current mainstream broadcasting ecosystem. For this reason, we settled to develop a P2P solution that is designed around the de-facto standard broadcasting protocol in the industry, that is adaptive HTTP streaming, rather than RTSP/RTP. In addition to that, and in order to be a viable alternative to standard CDNs, we set the goal for our platform to provide the same quality of user experience (QoE) as a CDN would. Therefore, we strive to deliver the same throughput, in terms of bitrate, but also similar playback latency. We define playback latency in this case as the time it takes for the content to be received by a player after it is made available at the source of the stream. In this paper, we detail SmoothCache 2.0, the second iteration of our P2P streaming solution [20] tailored to adaptive HTTP streaming. SmoothCache 2.0 provides a distributed caching platform for video content that is able to significantly lower distribution costs, removing as much as 96% of traffic from the source of the stream, while providing a viewer experience on par with standard Content Delivery Networks. In order to achieve these results, we exploit detailed knowledge of the HTTP live streaming distribution chain. Based on that, we build a distributed caching solution which prefetches content ahead of the live deadline and distributes it to all nodes in the network before the players request it. The distribution of data happens over a self-organizing overlay network that takes into consideration many factors such as upload bandwidth capacity, connectivity constraints, performance history, prefetching point and currently watched bitrate. In order to validate our techniques, we conduct a thorough evaluation of the system on a real network of thousands of consumer machines and show how each technique individually contributes to the performance of the system. Finally, we empirically show that the system provides the same QoE of a standard CDN by comparing bitrate delivered and playback latency of our platform against a commercial CDN. This paper is organised as follows. We start presenting the architecture of the system in Section 2. We then detail how adaptive HTTP live streaming works in Section 3. After that, we delve into the design of our platform, presenting prefetching heuristics and algorithms for overlay construction in Section 4. Section 5 shows our evaluation results. Finally, we present related work in Section 6 and conclude in Section 7.
2.
THE SMOOTHCACHE 2.0 SYSTEM
SmoothCache is a P2P live streaming agent (PLS) that appears as a HTTP proxy for video installed on each of the viewer’s machines. In the presence of SmoothCache, all player requests, are redirected to the local PLS agent instead of going directly to the source of the stream. That is the only change that needs to occur compared to normal operation without our agent. SmoothCache employs a number of helper services which represent our operational centralized infrastructure. It is important to note that our system is designed to be transparent to already-existing streaming infrastructure in order to ensure both vendor neutrality as well as simplified deployment. Therefore, we do not operate our own stream source, but instead we assume that there is an already-published stream, for instance from a Content Distribution Network
62
!""#$%&'(&'$
:;*"&, 0 then upRowCap = upRowCap − I max curRowCap = curRowCap + Oq else r =r+1 upRowCap = curRowCap curRowCap = 0
allowed to download it. That said, all players still request the same fragment t1 during the time window [t1 + µ, t1 + µ + δ], allowing thus a typical delay from the live point of 24 seconds if µ = 20 and δ = 2 seconds. In order to exploit this characteristic of streaming deployments and extend the window of opportunity for peers to download fragments, we design a proactive prefetching heuristic which allows few peers to retrieve fragments directly from the CDN as soon as they appear on it. Figure 7 shows how this heuristic works. P1 proactively prefetches fragment 1 from the CDN and then avails it to P2 which downloads it as soon as the notification arrives because of reactive prefetching. This allows P2 to fetch the fragment from the overlay much earlier than when the player requests it. Specifically, with proactive prefetching, the window of opportunity becomes significantly larger and at most w = µ + 2δ. As mentioned earlier, in proactive prefetching we select a subset of peers in the system, which we call first tier peers, to aggressively prefetch fragments and make them available to the rest of the overlay. The first tier selection process is tracker-assisted and it is tasked to identify the set of peers with best upload capacity. The set is constructed so that the aggregate bandwidth capacity of the peers in it is enough to serve the swarm with fragments. That means that ideally no peer in the swarm (excluding the first tier peers) should download a fragment from the source of the stream when the player asks for it. Instead, the fragment should be already in the local cache of the peer thanks to our prefetching heuristics. In the following section, we describe how the set is constructed at the tracker side.
First Tier Construction. Every peer p reports its upload bandwidth BWp and the bitrate it is watching bp . Periodically, the tracker executes
65
SmoothCache Distributed Caching
First Tier!
t1!
High BW!
Medium BW!
SmoothCache
partner set Ip and the out-partner set Op . Ip contains peers that p has chosen to act as uploaders for fragments. The Peer’s player requestof ! a peer p are the partners that have chosen p out-partners asFirst in-partner and p accepted them as such. Peer p accepts tier peer prefetched fragment! number of out-partners O max depending on its a maximum p Peer prefetched fragment! maximum bandwidth capacity up that is estimated against a bandwidth measurement service.
Low BW!
t1m=t1+"!
t1m+#!
4.3.1
Peer Sampling
By periodically contacting the tracker service, each peer keeps its random sample S of the overlay up-to-date. Samples are also exchanged among peers. We use our NATresilient peer sampling service called WPSS [17]. The service returns a uniform random sample of the nodes in the network, which is added to S. Each peer entry in it contains: a peer identifier, the peer’s NAT-type and the maximum upload capacity. This information is periodically refreshed by the sample service and by the tracker service and used as input in the neighbor selection process.
Figure 8: Timeline of fragment retrieval
•! Bandwidth-ordered overlay determines prefetch order •! Systemp’sstrives to slots added to the curRowCap. Otherwise, a out-partner
new row is created and the current row capacity curRowCap becomes the capacity available for the next row upRowCap. row construction to allocate the •! KeepThe the number of first tieralgorithm peers to continues a minimum peers in F until either all peers from F have been allocated or the number of rows has exceeded the maximum L. Reaching the first case means that F is sufficiently large. In System, since Tools andnot Algorithms for Adaptive HTTP-live on Peer-to-Peer the latterAcase, all peers have beenStreaming allocated, the Overlays4.3.2 In-partners Selection heuristic increases the set of first tier peers in F and retries Let Ipmax be the desirable size of Ip , the set of in-partners until all peers in F have been placed. for a peer p. Every peer periodically updates its Ip set after Besides what we described above, we also augment the a period of Trepair seconds by removing a subset of underheuristic to accommodate for two more factors. First, in performing peers Ipr . A new set of peers Ipa is then selected order to become an eligible first tier candidate, a peer needs from its local sample of the overlay S and added to Ip . This to be behind a NAT that is easily traversable or is an open update happens only if p has not been getting all the content Internet node. Second that peers in F can prefetch multifrom the overlay network in the last period Ts . ple bitrates which affects the calculation of upRowCap to selection process sure that |Ip | = makes a Ther in-partner account for uploading multiple bitrates. Ip \ Ip + Ip = I max . Ipr has size Ipr = min(β∗I max , I max − |Ip |) and β is a replacement factor typically set to β = 0.3. 4.3 Beyond Baseline Caching: Overlay ConEach peer strives to obtain a number of in-partners which struction is equal to I max as a safety measure in order to have readily The overlay construction process is entitled with the task available an alternative sender if connectivity issues, conof creating a set of connections between peers that are used gestion or churn temporarily prevent him from retrieving to exchange video content. Each peer chooses a number of content from one of its partners. in-partners, that are its seeders for content, and a number of out-partners that are leechers and only download content Remove Filters. from him. In our system, the heuristics for the selection We here define a set of removal filters, which are applied of out- and in-partners are designed in a way to create an to the current Ip , and that output the set of peers Ipr to emergent behaviour which organizes the network in a hierremove from Ip . archical and bandwidth-based order. Because of that, peers with higher upload capacity independently place themselves • Prefetching point. This filter identifies and removes higher in the hierarchy and therefore can spread fragments all the peers in Ip that are prefetching fragments later efficiently given their high upload to download ratio. than p. We do so by estimating the prefetch indicaIn Figure 8 we show a simplified example of the overtor pi(i) = tfi − tri for each fragment i, which is the lay created by SmoothCache 2.0 combined with the timeline difference between the time i was prefetched and the of distribution of a fragment. As described in the picture, time i was requested by the player. Note that we use the first tier peers receive the fragment first, as their are tri as a point of reference because we assume that the prefetching from the CDN. Afterwards, the peers with high time between two fragment requests is constant and upload bandwidth, but that are not first tier, also receive corresponds to δ. On top of that, we assume that all the fragment. Then, those provide it to the second group of peers request the same fragment i in a period of time m peers with medium bandwidth capacity, and so on. Given [tm i , ti + δ]. this organization of the network, it is clear that the fragPeers estimate their average request indicator pi over ments are distributed over the overlay in an efficient fashion. the last downloaded fragments and then piggy-back it However, the overlay structure described above is not static to all fragment advertisements. In the filter, we combut rather it is dynamically adjusted according to variations pare p’s average request point with all of its current in bandwidth capacity, churn, connectivity changes and the partners’ and remove partners which have a smaller bitrate peers are watching. pp than peer p. The filter is defined as follows: In the system, all peers run the same partner selection Ip = { q | piq < pip , q ∈ Ip }, process, where each peer p chooses a number of candidates out of a local sample S ⊂ P, provided by a distributed peer where piq is the average prefetch point of peer q and sampling service, that is a subset of all peers in the overlay pip the one of p. P. A peer’s neighborhood is composed of two sets, the in•! Have all peers prefetch fragment ahead of player request
66
• Successful transfers. The filter has the goal of removing a fixed number of under-performing peers in to order to promote discovery of new ones. We define the ordered tuple Q = (q0 , q1 , ..., qn ) containing all peers in Ip0 . The tuple Q is ordered in ascending order according to the number of successful fragment downloads from each peer in the set in the last period of time Trepair . We then select and filter the ones with the fewer number of successful downloads until we remove a minimum of peers from the set, that is |Ipr | = β ∗ Ipmax .
Here B is meant as the largest bitrate of all bitrates watched by peers in S 00 . We then estimate nk , the number of elements to pick from each set Sk in the following way: nk = min(N ∗ αk , |Sk |) for k = b, b − 1, b + 1. At this point, we can build the output of the filter Ipa as:
Ipa .
4.3.3
Add Filters.
• Connectivity. We filter out any peer q which cannot establish bidirectional connectivity with p. S 0 = { q | imp(τ (p), τ (q)), q ∈ S } imp(τ (p), τ (q)) is a predicate that evaluates to true if bi-directional communication between p and q is not possible according to [19], where τ (q) is the type of NAT/firewall/gateway that q is behind. • Bandwidth. This filter removes all peers from S 0 which have lower bandwidth capacity than p. The rationale behind this is to have peers in a certain bandwidth range become partners with peers of similar or higher bandwidth. This filter together with the out-partner selection process described in the next section makes possible to construct an hierarchical overlay structure based on bandwidth capacity of peers. S 00 = { q | up < uq , q ∈ S 0 } • Bitrate. The filter picks a finite subset R of size N from S 0 according to the bitrate peers in S 0 are downloading. The rationale behind the filter is to preferably choose peers from S 0 which are watching the same bitrate b peer p is watching and, additionally, a number of other peers which are watching the bitrates immediately higher and lower bitrate b. We hereby describe how set R is constructed. Let Sk be the set of peers which are downloading bitrate k from S 00 , and let Skn be a set of size nk from Sk without replacement. We define α as the percentage of R to be chosen from Sb (the bitrate p is watching). αb−1 and αb+1 are instead the percentages of the elements to be chosen from Sb+1 and Sb−1 , where b + 1 and b − 1 are the bitrates immediately higher and lower than b, respectively. The values of αb+1 and αb−1 are calculated as: 1−α , 2
1 − α, (
αb−1 =
1−α , 2
1 − α,
j=1..nb
Out-partners selection
The out-partner selection process is responsible for choosing, upon a partnership request from a downloader, if the request should be accepted or not. If that is the case, an existing partner may be evicted from the set of out partners. We hereby describe how this process works. We define Opmax as the maximum number of out-partners for peer p. This is calculated as Opmax = min(up /bp ). As long as |Op | < Opmax , new outgoing partners that chose p as in-partner are accepted. Every newcomer is given a grace period Tgrace during which it is not considered for eviction. When Tgrace is over and the peer has not requested any fragment from p, the peer is added to a set Opr . Consequently, Opr contains at all time the partners which may be considered for eviction. If new applicant q, i.e. a downloader requesting to become partner of p, has a maximum upload bandwidth uq that is larger than any of the peers in Opr , the peer with the smallest maximum upload capacity in Opr is evicted and replaced by q. The bandwidth-based out-partner selection policy makes sure that the best peers in the system, e.g. the first tier peers, accept as out-partners only the best peers in the swarm. In turn every peer will do the same and the overlay will self-organize in an hierarchical structure where peer bandwidth’s utilization is maximized.
Add filters have the purpose of selecting which peers will be added to Ip . Add filters are applied to S and produce a set of best candidates peers Ipa .
αb+1 =
n
If |Ipa | = N then the set Ipa is returned as is. Otherwise, a sample of w = N −nb −nb−1 −n S b+1 values is randomly selected from the set A = Sj \ R and added to
Ipr = { (q0 , q1 , ..., qk ) | k ≤ max{ 0, |Ip0 |−Ipmax (1−β) }}
(
n
n
Ipa = {p ∈ Sb b ∪ Sb b−1 ∪ Sb b+1 }
4.4
Multi-bitrate support
In the previous sections, we have provided a complete description of our system. Here instead, we highlight some aspects of the platform that are relevant to multi bitrate support, that is one of the main features of adaptive HTTP streaming protocols. SmoothCache 2.0 creates a unique overlay network that is used to distribute all bitrates watched by the peers. To make this possible, the first tier selection process is executed multiple times, once for every bitrate. Therefore there will be a number of first tier sets that is equal to the number of bitrates watched in the system. Regarding overlay construction, as described in the inpartner selection heuristics, a peer preferably chooses partners that are downloading (prefetching or watching) the same bitrate that it is watching. On top of that, each peer keeps a small selection of peers in its neighborhood that are downloading bitrates immediately higher or lower to the one it is watching. This as to quickly adapt to bitrate switches without having to fall back to the source of the stream. Note that the invariant of each peer choosing neighbors with higher upload bandwidth is kept even when watching a multi-bitrate stream.
1
