of CDN can be quite expensive. In this paper, we briefly introduce the CoolStreaming system, and report our prelimiary results on large scale broadcasting ...
On Large Scale Peer-To-Peer Live Video Distribution: CoolStreaming and Its Prelimianry Experimental Results Xinyan Zhang 1, Jiangchuan Liu2, and Bo Li1 1
Department of Computer Science, The Hong Kong University of Science and Technology 2 School of Computing Science, Simon Fraser University
Abstract*- It is an attractive idea to spread media data using a peer-to-peer scheme; however it remains unclear whether this technology can really deliver real-time media to a large number of users in global scale. This paper presents an empirical study on CoolStreaming, a Data-driven Overlay Network for live media streaming. The core operations in CoolStreaming are very simple: every node periodically exchanges data availability information with a set of partners, and retrieves unavailable data from one or more partners, or supplies available data to partners. We have extensively evaluated the performance of CoolStreaming over real Internet environment. Our experiments, involving more than 10,000 distinct users, demonstrate that CoolStreaming can achieve quite good streaming quality with globally distributed users, even under formidable network conditions. Meanwhile, we present several interesting observations from these large-scale tests. Finally, we offer some directions toward the future development of p2p media streaming
I. INTRODUCTION It has been around five years since the overlay live streaming concept was first introduced in [2]. It is an attractive idea to spread real time media using a peer-to-peer scheme; however it remains unclear if this method can really deliver real time media to large number of users in global scale. To this end, we have designed CoolStreaming [1], a Data-driven Overlay Network. The core operations in CoolStreaming are very simple: every node periodically exchanges data availability information with a set of partners, and retrieves unavailable data from one or more partners, or supplies available data to partners. There are many related p2p streaming systems, such as NICE [3], ZIGZAG [4] and Narada [2], to name but a few. Most of them are either theoretical frameworks or validated by small scaled experiments only. Using the CDN technology, several large scale real time media broadcasting is performed [5], but the deployment of CDN can be quite expensive. In this paper, we briefly introduce the CoolStreaming system, and report our prelimiary results on large scale broadcasting experiments over this platform. To our knowledge, this might be the first and biggest 24h/7d running overlay broadcasting services. We expect that this report will offer readers a better understanding about this promising and practical solution to live video distribution. II.
COOLSTREAMING : A BRIEF REVIEW
* J. Liu’s work was supported in part by a Canadian NSERC Discovery Grant 288325, an NSERC Research Tools and Instruments (RTI) Grant 613276, and an SFU President’s Research Grant. Bo Li's work was supported in part by grants from Research Grants Council (RGC) under contracts HKUST 6196/02E and HKUST6204/03E, a grant from NSFC/RGC under the contract N_HKUST605/02, and a grant from Microsoft Research under the contract MCCL02/03.EG01.
Figure 1 depicts the system diagram of a CoolStreaming node. There are three key modules: (1) membership manager, which helps the node maintain a partial view of other overlay nodes; (2) partnership manager, which establishes and maintains the partnership with other known nodes; (3) scheduler, which schedules the transmission of video data. For each segment of the video stream, a CoolStreaming node can be either a receiver or a supplier, or both, depending dynamically on the availability information of this segment, which is periodically exchanged between the node and its partners. An exception is the source node, which is always a supplier, and is referred to as the origin node. It could be a dedicated video server, or simply an overlay node that has a live video program to distribute.
Player Membership Manager
Buffer
Buffer Map (BM)
Partnership Manager
Scheduler
Network Interface
Partner
Partner
Partner
Fig. 1. CoolStreaming System.
Membership Management Each CoolStreaming node has a unique identifier, such as its IP address, and maintains a membership cache (mCache) containing a partial list of the identifiers for the active nodes in the CoolStreaming. In a basic node joining algorithm, a newly joined node first contacts the origin node, which randomly selects a deputy node from its mCache and redirects the new node to the deputy. The new node can then obtain a list of partner candidates from the deputy, and contacts these candidates to establish its partners in the overlay. Partnership Management As said, neither the partnerships nor the data transmission directions are fixed in CoolStreaming. More explicitly, a video stream is divided into segments of a uniform length, and the availability of the segments in the buffer of a node can be represented by a Buffer Map (BM). Each node continuously exchange its BMs
with the partners, and then schedules which segment is to be fetched from which partner accordingly. Scheduling Algorithm Given the BMs of a node and its partners, a schedule is to be generated for fetching the expected segments from the partners. For a homogenous and static network, a simple round-robin scheduler may work well, but for a dynamic and heterogeneous network, a more intelligent scheduler is necessary. Specifically, the scheduling algorithm strikes to meet two constraints: the playback deadline for each segment, and the heterogeneous streaming bandwidth from the partners. If the first constraint cannot be satisfied, then the number of segments missing deadlines should be kept minimum, so as to maintain a continuous playback. This problem is a variation of the Parallel machine scheduling, which is known NP-hard. It is thus not easy to find an optimal solution, particularly considering that the algorithm must quickly adapt to the highly dynamic network conditions. Therefore, we resort to a simple heuristic of fast response time [1]. There are several additional modules that enhance the performance of the system or facilitate our experimental study. Failure Recovery and Partnership Refinement In CoolStreaming, a node can depart either gracefully or accidentally due to crash. In either case, the departure can be easily detected after an idle time of TCP or BM exchange, and, as the probability of concurrent departures is rather small, an affected node quickly react through re-scheduling using the BM information of the remaining partners. Furthermore, we let each node periodically establish new partnerships with nodes randomly selected from its mCache. This operation serves two purposes: first, it helps each node maintain a stable number of partners in the presence of node departures; second, it helps each node explore partners of better quality. Command Dispatching and Report Collecting While the monitoring node can maintain a connection to each participating node to dispatch commands and collect reports, such a design is non-scalable. To mitigate this problem, we use another overlay for command dispatching in our experiments. Specifically, each command message contains a unique sequence number; upon receiving a new command, a node simply sends it to a list of known nodes, which can be obtained from the mCache in CoolStreaming module or from a predefined list. As the command messages are limited and sensitive to delay, such flooding is a reasonable choice for broadcasting commands. It also helps with the construction of a reverse path tree for report collection, in which such reports as losses and path lengths can be classified and merged at some junction nodes in the tree and before forwarding back to the monitoring node. Consequently, we can conduct online statistics without overwhelming the monitoring node. III. EVALUATIONS The CoolStreaming system is developed mainly in Python language. It currently streams Real Video and Windows Media formats, but can accommodate other streaming formats as long as they are supported by user-side players. Its implementation is platform-independent, and thus can be used under Unix, Windows, and other operating systems. Since the first release, CoolStreaming system has been running continuously for about a year. Some quick facts about this system are as follows:
Over 1,000,000 clients (in terms of unique IP addresses ) have used CoolStreaming; The clients are from 20 different countries in the world; During peak times, the system has over 50,000 active clients, and each channel has over 25,000 clients. To further understand this system, we examine the logs from 20:26, Mar 10 (GMT), to 2:14 Mar 14 (GMT), 2005. The logs of these 78 hours include the following data: IP addresses of each client Subscribed channel Packet loss rate A.
General Observations
Figure 2 illustrates the performance for a popular event in Mar. 12th, when more than 15,000 users were subscribing a channel to watch a program. The streaming rate is 330Kbps. We can see that at 0:0, less than 500 users were subscribing to the channel, but the population quickly increased at 2:00, when the program started. The figure also shows the corresponding Continuity indices (the number of segments that arrive before or on their playback deadlines over the total number segments [1]). We can see that the continuity indices were generally over 95% in the whole event period, and closely correlated with the client population. Figure 3 depicts the packet loss distribution at the peak time in Figure 2 (around 15,000 users). There are around 13,000 users experiencing 0% packet loss, and only around 1,000 clients experiencing more than 10% of losses. Figure 4 illustrates average continuity index with the channel subscription time during a whole day time for a specific channel. In the initial period when a user subscribed to a channel, the average continuity index was slightly less than 90%; when it stayed in the channel for over 1000 seconds, the average continuity index jumped to 92%, and it further reached 95% after an hour. This observation suggests that the system favors persistent users, possibly owing to the partnership refinement mechanism. B. User Behavior Figure 5 shows the user joining rate for the event in Mar. 12th. At 0:0, we observed less than 20 users joining the system in a minute. At the start time of the event, 2:00, the joining rate reached the peak, around 160 per minute. Then the rate slowed down, but remained at a high level; when the event was going to the end, the joining rate reached another peak (probably because the users were interested in the result of the event). To further evaluate the user behavior, we select two typical periods from the logs: (a) 20:26 Mar. 10 – 0:14 Mar. 12 (GMT) (b) 22:26 Mar. 12 – 2:14 Mar. 14 (GMT) Each period is around 28 hours. Period (a) is a typical normal day with no special events; period (b) is a day with several sports events, and therefore much more users joined the system.
4
2
x 10
100
Number of Users
Continuity Index
98
96 1 94
Average Continuity Index
1.5 95
90
0.5 92
0
0
50
100
150 200 250 300 Minutes from 0:0 GMT, Mar 12
350
90 400
Fig. 2. Number of users and continuity index
85
5
1500
2000 2500 3000 3500 Subscription Time (Seconds)
4000
4500
5000
160 Number of Users Joining the System in a Minute
4
10
Number of Users
1000
Fig. 4. Average continuity index vs. user subscription time
10
3
10
2
10
1
10
0
10
500
0
10
20
30
40 50 60 70 Packet Loss Percentage
80
90
100
Fig. 3. Packet loss distribution
Figures 6 and 7 draw the distributions of the user subscription times for periods (a) and (b), respectively. In period (a), we can see that user subscription times resemble an exponential distribution. This is because there are no attractive events during this period, and thus the users joined and leaved the system like a poison process. However, in period (b), there was a popular program that attracted a large population of users, and kept them in the system for relatively long time.. Further more, Figures 6 and 7 both reveal that there were around 100 users never quit the system. Those users probably always kept the channel subscribed, so that they can watch it instantly at any time. Those persistent users form a stable backbone for the system. C. Geographical Distribution Table 1 illustrates the geographical distribution of the users during periods (a) and (b), respectively. We use the IP addresses to identify the geographical locations of the users.
140 120 100 80 60 40 20 0
0
50
100
150 200 250 300 Minutes from 0:0 GMT, Mar 12
350
400
Fig. 5. User join rate
We can see that the mainland of China and Hong Kong contain the dominate user groups, since most programs broadcasted over CoolStreaming are in Chinese. The rest of users are coming from all around the world, in particular, Europe and North America. IV. CONCLUSIONS This paper presents an empirical study on CoolStreaming, a Data-driven Overlay Network for live media streaming. Apart from the data logs, we have the following observations, which also offer potential research directions: End-to-end bandwidth is a key problem in large scale streaming. In a client/server system, we can support a large number of users in a local area by adding more servers. For a global scaled service, simply adding servers is not enough however. The end-to-end bandwidth may limit the geographical distribution of the users. Hence, CDN is a possible solution, but it remains very expensive and is not readily deployable. On the other hand, P2P enables intelligent path selections that may avoid the above problem. Nonetheless, we also find that, comparing with client-server solution, the overlay solution may introduce additional delay for a user to smoothly playback the video.
CN HK US GB CA FR KR TW IT DE ES FI JP SG NL MY AU SE NO PL NZ MO SI Others Total
4
10
3
Number of Slices
10
2
10
1
10
0
10 2 10
3
4
10
5
10
10
Time (Seconds)
Fig. 6. Number of slices vs. subsciption time in normal day
3
Number of Slices
10
2
10
20550 5125 2458 1410 1210 1069 883 647 518 467 443 403 342 303 232 225 170 123 77 68 62 59 51 274 37169
CN HK US ES KR CA GB TW FR IT SG DE JP FI MY AU NL MO PL SE NO SI NZ Others Total
32217 20725 3290 2989 1834 1707 1326 1213 1088 1059 578 555 519 459 373 252 244 192 181 178 127 91 66 389 71652
1
10
(a) 0
10
3
10
4
10
Time (Seconds)
(b)
Table 1. The geographical distribution of users using the system. (a) 20:26 Mar. 10 – 0:14 Mar. 12 (GMT); (b) 22:26 Mar. 12 – 2:14 Mar. 13 (GMT)
Fig. 7. Number of slices vs. subsciption time in hot event day
Upload bandwidth is a physical limitation. While P2P streaming is more flexible than client/server, its does have certain limitations. The most significant is the demands on the upload bandwidth. For a successful P2P media streaming system, at least we should have average upload bandwidth larger than the streaming rate. Hence, ADSL and other asymmetrical Internet accesses become obstacles. ISP issues and traffic engineering effects. Currently, a large portion of the available bandwidth at network edges and backbone links are occupied by such P2P applications as BitTorrent. Many ISPs thus limit this kind of traffic. For file sharing the limitation may only be annoying (just slow down the download), but for media streaming it can be fatal. Other filtering mechanisms may also create problem to p2p streaming systems. We have already observed such problems in CoolStreaming.
REFERENCE [1]
[2] [3] [4] [5]
X. Zhang, J. Liu, B. Li, and T.-S. P. Yum, “DONet/CoolStreaming: A Data-driven Overlay Network for Live Media Streaming,” in Proc. IEEE INFOCOM'05, Miami, FL, USA, March 2005 Y.-H. Chu, S.G. Rao, and H. Zhang, “A case for end system multicast,” in Proc. SIGMETRICS'00, 2000. S. Banerjee, B. Bhattacharjee, and C. Kommareddy, “Scalable application layer multicast,” in Proc. ACM SIGCOMM'02, Duc A. Tran, Kien A. Hua, and Tai T. Do, “A peer-to-peer architecture for media streaming,” IEEE J. Select.Areas in Comm., vol. 22, Jan. 2004. K. Sripanidkulchai, A. Ganjam, B. Maggs, H. Zhang, “The Feasibility of Peer-to-Peer Architectures for Large-Scale Live Streaming Application”, in Proc. ACM SIGCOMM'04, 2004.
