Comparing Cloud Content Delivery Networks for

Comparing Cloud Content Delivery Networks for Adaptive Video Streaming C H E N WA N G 1 , A N D A L J AYA S E E L A N 2 , H Y ON G K IM 2 T H U R S D AY, J U LY 5 , 2 0 1 8

1IBM

Thomas J. Watson Research Center and Computer Engineering, Carnegie Mellon University

2Electrical

7/10/18

IEEE CLOUD 2018

1

Which Clouds can provide good user experience for video streaming services? Video service provider: which provider to adopt?

Cloud provides content delivery services for video streaming.

Microsoft Azure CDN

Google Cloud CDN

7/10/18

IEEE CLOUD 2018

2

Cloud Content Delivery Networks A geographically distributed network of cache servers or data centers to distribute service or content.

◦ Spatially reduce the distance between servers and users ◦ Improve the service performance.

From top to bottom: ◦ AWS CloudFront ◦ Google Cloud CDN ◦ Azure Verizon CDN

7/10/18

IEEE CLOUD 2018

3

Challenges

1 What are important factors/dimensions to consider when choosing Cloud CDNs for video streaming services?

7/10/18

2 How to evaluate these factors?

IEEE CLOUD 2018

3 How to use the results of comparison given the fact that Cloud providers change their services over time?

4

Related work CloudCmp (AWS, Microsoft, Google, Rackspace) [2] • Compare Infrastructure as a Service for Web applications • Metric: operation response time, intra-cloud/wide-area latencies

CDN comparison (Akamai, LimeLight) [3] • General comparison of CDNs over infrastructures and DNS design • Metric: Geo-locations, DNS response time, cache response time

[2] Li, Ang, et al. "CloudCmp: comparing public cloud providers." Proceedings of the 10th ACM SIGCOMM conference on Internet measurement. ACM, 2010. [3] C. Huang, A. Wang, J. Li, and K. W. Ross, “Measuring and evaluation large-scale CDNs,” in ACM SIGCOMM, 2008.

7/10/18

IEEE CLOUD 2018

5

Important factors for video streaming Video Streaming Service User Quality of Experience Picture Quality 7/10/18

Cost

Smoothness

IEEE CLOUD 2018

6

Data collection

QoE model

[1]

v QoE: Subjective measurement of user experience v QoE model : approximate user QoE v DASH: Dynamic adaptive video streaming over HTTP v Chunk: A segment of video in DASH streaming v QoE range: [0-5]

1 q (t , r ) = qfreezing (t ) × qbit-rate ( r ) 5 Picture Quality

Streaming smoothness

[1] Wang, Chen, Hyong Kim, and Ricardo Morla. "QWatch: Detecting and Locating QoE Anomaly for VoD in the Cloud." 2016 IEEE International Conference on Cloud Computing Technology and Science (CloudCom). 7/10/18

IEEE CLOUD 2018

7

Important aspects of Cloud CDNs Global Performance

Geographical coverage QoE

Outsource infrastructures Own Infrastructure

Azure collaborates with Verizon and Akamai AWS CloudFront, Google Cloud CDN

North America, South America, Europe, Asia, Australia Variance in an hour

Stability Variance in a day Session crashes Scalability QoE drops 8 7/10/18

IEEE CLOUD 2018

Data collection

Measurement System DASH video streaming testbed in the Cloud ◦ Video origin server: Azure/AWS/Google VMs ◦ Content delivery network: Azure/AWS/Google CDN ◦ Users: ◦ 100 PlanetLab nodes ◦ 2 to 1024 emulated clients in AWS VMs

7/10/18

IEEE CLOUD 2018

9

QoE Data Collection LO C AT I O N S O F U S E R S E M U L AT E D I N 1 0 0 P L A N E T L A B S E R V E R S

7/10/18

IEEE CLOUD 2018

10

Global QoE Comparison Comparison Metric • Session QoE for 1 hour video session.

7/10/18

Experiment Setup • Continuous 24 1hour video sessions on 100 emulated users at different locations • Total sessions: 24 x 100 = 2400

IEEE CLOUD 2018

11

Fraction of Emulated Users

Cumulative Distribution of Session QoE Provider

Total PoPs

Azure Verizon

43 edge locations

Amazon CloudFront

82 edge locations 11 regional locations 16 regions

Google CDN 90 Internet Exchanges > 100 interconnected facilities

7/10/18

IEEE CLOUD 2018

12

QoE Comparison across regions

Compare the average and the standard deviation of session QoE for emulated users in each region.

7/10/18

IEEE CLOUD 2018

13

Stability in terms of QoE C O M PA R E H O U R LY Q O E AV E R A G E F O R T WO E M U L AT E D U S E RS AT D I F F E R E N T LO C AT I O N S .

Emulated user in America/New_York

Emulated user in Asia/Tokyo

5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 1 2 3 4 5

UTC Time

UTC Time 7/10/18

15 16 17 18 19 20 21 22 23 24 01 02 03 04 05 6 7 8 9 10 11 12 13 14 15

IEEE CLOUD 2018

14

Hourly QoE variations in one time zone Low average QoE co-occur with large variations. Low average QoE appear in the morning between 10 am to 12 pm. High average QoE co-occur with low variations. High average QoE appear in the afternoon across providers. Azure Verizon CDN

7/10/18

Amazon CloudFront

IEEE CLOUD 2018

Google Cloud CDN

15

Scalability experiment Ø Emulate different scales of burst user demand at one edge location using AWS t2.small instances Ø 2 to 64 clients emulated per instance

Ø Bandwidth is not guaranteed for such instances. Ø Experiments at one scale are run on the same set of instances around during midnight.

Ø 2 to 1024 clients emulated per zone (in one datacenter at the same location)

ØTo prevent the CDN caches from the experiment of previous runs

Ø Run only one scale of demand on one day Ø Run streaming services on 3 cloud CDNs around the same time (one after the other in 3 hours around midnight).

Ø Scalability comparison across regions Ø North America, Europe, Asia

7/10/18

IEEE CLOUD 2018

16

Scalability comparison across regions C O M PA R I S O N O F S E S S I O N C R A S H E S O V E R D I F F E R E N T S C A L E S O F B U R S T U S E R DEMAND.

• • • • •

There were session crashes in 3 cloud providers when the scale >= 256. Cloud vendors varied in scalability in one region. One cloud vendor varies in scalability across regions. Session crashes do not always increases over the scales of user demand. Session crashes are not always the minimum for AWS CloudFront.

7/10/18

IEEE CLOUD 2018

17

Scalability comparison across regions C O M PA R I S O N O F S E S S I O N Q O E O V E R D I F F E R E N T S C A L E S O F B U R S T U S E R D E M A N D .

• In N. America, the user QoE on all cloud CDNs ↓ slightly as the scale of burst users ↑. • In Asia, the QoE ↓ significantly at the scale of 1024 on all cloud CDNs, especially on Azure Verizon CDN. • Many factors can impact the scalability: the amount of available resources at the edges, the dynamic resource usage of edge servers, the caching strategies, etc. • The performance may change due to the sharing of resources among customers1.

1Due

Crashed session QoE = 0

to the resource limit, we cannot re-run exps many times to verify it.

7/10/18

IEEE CLOUD 2018

18

Cost analysis --- Pricing difference • Charge different items Provider

Cache fills

CDN egress traffic

Azure Verizon CDN

✔

✔

Amazon CloudFront ✔

Google Cloud CDN

• Prices vary across regions

7/10/18

$0.17

$0.14

Per GB Cost for Amazon CloudFront

$0.14

✔

✔

✔

Amazon CloudFront Average cost (per GB) $0.25

$0.09 $0.09 $0.11

✔

• Prices drop as usage increases

Amazon CloudFront Regional Data Out (Per GB cost for First 10TB/month) $0.30 $0.25 $0.20 $0.15 $0.10 $0.05 $0.00

HTTP requests

IEEE CLOUD 2018

0.1 0.08 0.06 0.04 0.02 0 0

5000000

10000000

15000000

Total amount of regional data out (GB)

19

Traffic model q User demand (number of monthly subscribers): 𝑵 qWhere is the user? 𝑖# 𝑱

q Total egress traffic: ∑𝒖 ∑𝒋𝒖 𝒓𝒋,𝒖 ×𝟓

qHow long one user watches the video from the website? q ℎ# hours per month q Total number of video chunks: 𝐽# = 3600×

34 5

q Bitrate level of video chunk one user get each time: 𝑟7

q Total HTTP requests: ∑𝒖 𝑱𝒖 q Cache fills

q Cache hit rate: 𝜂 q Cache fills: Total egress traffic × (1 − 𝜂)

7/10/18

IEEE CLOUD 2018

20

Numerical cost analysis v v v v

𝑖# ∈ 𝐼 follows uniform distribution, 𝐼 = {𝑁. 𝐴𝑚𝑒𝑟𝑖𝑐𝑎, 𝐸𝑢𝑟𝑜𝑝𝑒, 𝐴𝑠𝑖𝑎} ℎ# follows Poisson distribution [4] 𝑟7 follows geometric distribution, as users obtain high bitrate more often 0≤𝜂≤1 v 𝜂 = 0, all videos requested are not cached at the edge. v 𝜂 = 1, all videos requested are cached at the edge.

[4] TechCrunch, “Netflix reaches 75% of US streaming service viewers, but YouTube is catching up,” 2017. [Online]. Available: https://goo.gl/dq72Aa 7/10/18

IEEE CLOUD 2018

21

Implications § Video streaming services § should compare Cloud CDNs according to the service locations in advance. § would be better to choose more than one CDN to allow CDN adaptation when one has dynamic performance degradation. § would be better to forecast the burstiness of user demand to determine the CDN adaptation upon prediction / load balancing among multiple CDNs § Should predict the cache hit rate in advance and choose the most economical one.

7/10/18

IEEE CLOUD 2018

22

Questions? Chen Wang: [email protected] Andal Jayaseelan: [email protected]

7/10/18

IEEE CLOUDCOM 2017

23

Numerical cost analysis

7/10/18

IEEE CLOUD 2018

24