IEICE TRANS. COMMUN., VOL.E96–B, NO.3 MARCH 2013
790
PAPER
Optimally Designing ISP-Operated CDN∗ Noriaki KAMIYAMA†a) , Tatsuya MORI† , Ryoichi KAWAHARA† , and Haruhisa HASEGAWA† , Members
SUMMARY Recently, the number of users downloading video content on the Internet has dramatically increased, and it is highly anticipated that downloading huge size, rich content such as movie files will become a popular use of the Internet in the near future. The transmission bandwidth consumed by delivering rich content is enormous, so it is urgent for ISPs to design an efficient delivery system that minimizes the amount of network resources consumed. To deliver web content efficiently, a content delivery network (CDN) is often used. CDN providers collocate a huge number of servers within multiple ISPs without being informed of detailed network information, i.e., network topologies, from ISPs. Minimizing the amount of network resources consumed is difficult because a CDN provider selects a server for each request based on only rough estimates of response time. Therefore, an ordinary CDN is not suited for delivering rich content. P2Pbased delivery systems are becoming popular as scalable delivery systems. However, by using a P2P-based system, we still cannot obtain the ideal delivery pattern that is optimal for ISPs because the server locations depend on users behaving selfishly. To provide rich content to users economically and efficiently, an ISP itself should optimally provide servers with huge storage capacities at a limited number of locations within its network. In this paper, we investigate the content deployment method, the content delivery process, and the server allocation method that are desirable for this ISP-operated CDN. Moreover, we evaluate the effectiveness of the ISPoperated CDN using the actual network topologies of commercial ISPs. key words: content delivery, CDN, ISP
1.
Introduction
Video streaming services for user-generated content (UGC), for example, YouTube [25], are experiencing significant increases in the numbers of their users. These services enable users to easily upload video content, and consequently, downloading video content is becoming a common use of the Internet. UGC is typically short in length and low in video quality, so the impact of delivering UGC on a network is not that great. However, it is highly anticipated that many users will download more attractive video content such as movies or TV episodes provided by dominant content holders such as major filmmakers and TV broadcasting companies on the Internet in the near future. The sizes of these rich content will continue to increase and are expected to reach several gigabytes or more. Therefore, to provide rich content to users economically and efficiently, Manuscript received June 20, 2012. Manuscript revised November 2, 2012. † The authors are with NTT Network Technology Laboratories, NTT Corporation, Musashino-shi, 180-8585 Japan. ∗ This article is based on “ISP-Operated CDN,” by N. Kamiyama, et al., which appeared in IEEE Global Internet Symposium c 2009 2009 IEEE. a) E-mail:
[email protected] DOI: 10.1587/transcom.E96.B.790
ISPs must minimize the amount of network resources consumed by delivering rich content. Content delivery networks (CDNs) have been widely used to deliver small web content efficiently. They were developed when the transmission capacity of network links started to become severely limited. CDN providers collocate as many servers as possible over multiple ISPs to improve the response time. Many CDN providers exist; Akamai Technologies [1] is one well known large-scale CDN provider that has collocated more than 20,000 servers over about 1,000 networks in around 70 countries [1]. CDN providers collocate a huge number of servers over many ISP networks without being informed by the ISPs of detailed network information regarding the network topologies and routing, so it is difficult for them to optimally design the server locations and optimally select the best server for each user request. Although the server collocation, content deployment, and server selection algorithms of many CDN providers are proprietary and private, many CDN providers seem to measure the response time at multiple measurement points in the network and select the estimated best servers [19]. However, measuring the response time for all pairs of servers and users is impossible, so CDN providers have only a limited amount of information. Hence, Akamai s case, for example, the optimality of server selection is strongly degraded for users outside the USA [19]. The typical content delivered on the Internet has primarily been small web content, so the best-effort approach to improve the response time has been widely accepted. However, when delivering large, rich content, it is important to minimize the amount of network resources consumed. Achieving this goal is difficult when using an ordinary CDN, although the response time can be improved [19]. In largescale academic CDN systems, such as CoDeen [6] and Coral CDN [8], it is also difficult to optimize the consumption of network resources because they are fully decentralized systems. Hence, Limelight Networks, a large-scale CDN provider, started a new CDN business in which a limited number of data centers with huge storage capacities are collocated. These data centers are connected by a network with huge transmission capacity operated by Limelight itself [17]. P2P-based content delivery systems such as BitTorrent, in which users download content from other users, are also becoming popular [7]. In BitTorrent, for example, content is divided into multiple blocks, and users downloading blocks
c 2013 The Institute of Electronics, Information and Communication Engineers Copyright
KAMIYAMA et al.: OPTIMALLY DESIGNING ISP-OPERATED CDN
791
also upload in parallel to other users blocks they have already downloaded. Therefore, the number of peers who have the target content increases with the number of users who want to obtain the target content, so a P2P-based delivery system is efficient when distributing popular content to many users. When there are multiple candidate servers that have the target content, downloading from the server with the smallest hop distance is desirable in order to minimize the amount of network resources consumed. However, many P2P-based delivery systems randomly select peers, so a peer with a long hop distance could possibly be selected even though other candidate peers with smaller hop distances exist. Currently, content delivery using P2P-based systems is increasing dramatically, so the waste of network resources caused by P2P-based delivery systems has become a problem [14]. Hence, a new P2P-based system called P4P, which avoids wasting network resources by exchanging information between ISPs and P2P providers, is being investigated [21]. However, the server role still relies on users who behave selfishly, so we need to give users incentives to upload content. We also need to give ISPs and P2P providers strong motivation to share their information. In addition, ISPs cannot design and control the server locations, so server locations are not optimally designed. An approach to reduce the server load of a CDN by combining P2P with the CDN and using P2P in complement has also been investigated [10], [22]. However, the principal goal of this approach is to reduce the server cost, and the amount of network resources consumed is not minimized. Although we cannot change or control the positions of users who are at the receiving ends of delivery flows, we can change or control the positions of servers that are the sending ends of delivery flows through the server allocation and selection policies. Therefore, server locations and server selections are major factors determining the traffic flows within a network, so it is important to optimally design the server locations, manage the positions of all content, and optimally select a server for each request in a centralized way, when delivering rich content. One desirable way for an ISP to achieve this goal is to provide servers with huge capacities, i.e., data centers, at a limited number of locations within its network and operate a CDN service by itself. In this paper, we call this new CDN an ISP-operated CDN or iCDN for short. In the iCDN, ISPs can manage all the server locations as well as the information related to the network infrastructure, such as topology and link capacities, in a centralized way. Hence, unlike in ordinary CDN or P2P-based delivery systems, ISPs can optimally design server locations, optimally deploy content, and optimally select a server for each request. Moreover, by constructing data centers with huge storage capacities at a limited number of locations, managing content location is simplified. AT&T recently announced that it would provide a CDN business by itself [3]. In this paper, we investigate the content deployment method, the delivery process, and the server allocation method that
are desirable for the iCDN† . Moreover, we evaluate the effectiveness of iCDNs using the network topologies of actual ISPs. In Sect. 2, we investigate the content deployment policy and the delivery control process desirable for the iCDN. We then propose the optimum design algorithm of server locations based on a greedy heuristic method in Sect. 3. We show the numerical results using actual ISP network topologies in Sect. 4, and Sect. 5 presents our conclusions. 2.
Mechanism of ISP-Operated CDN
In the iCDN, an ISP operating a large-scale backbone network, such as a Tier 1 ISP, constructs data centers with huge storage capacities at a limited number of locations, e.g., from 1 to 20, within its network. Each data center accommodates a large number of servers using a clustering technique and has huge capacities of storage and processing power to respond to many requests simultaneously. Let N, S , and M denote the number of nodes within the network, the number of data centers, and the number of rich content items, respectively. One centralized control center exists, and it collects the information of stored content from all the data centers and selects a delivery server for each user request. We can improve the reliability of the system by duplicating the centralized control center. Using the iCDN, an ISP can deliver rich content to its users at a minimized cost. Rich content is expected to be delivered efficiently with very low latency, so content providers will have incentives to provide rich content to the iCDN. Although we assume the iCDN within only a single ISP in this paper, we could also consider a case in which multiple ISPs collaborate to operate the iCDN together, and users in one ISP can download content from data centers operated by other ISPs. We leave the detailed investigation of this case for a future study. Moreover, we investigate the delivery system with the target of minimizing the total amount of traffic within the network in this paper. However, we could also consider a delivery system with other optimization targets such as minimizing the maximum link utilization. Such cases are also left for a future investigation. 2.1 Content Deployment Methods We need to determine how to deploy each M rich content item over S data centers. For an ordinary CDN, several content deployment methods have been proposed, and they are classified into two categories: replication methods and caching methods [15]. In the replication methods, the server set is derived for each content item by solving the replica placement algorithm (RPA) that minimizes the optimization function, for example, the average hop length of delivery flows, and copies of each content item are deployed in advance. To efficiently send copies of content to the target servers, deployment methods such as those in [4] have been investigated. In the replication method, we need to estimate †
A shorter version of this manuscript was presented in [11].
IEICE TRANS. COMMUN., VOL.E96–B, NO.3 MARCH 2013
792
the request rate of each content item in advance. Although many methods for RPA have been proposed, e.g., that in [13], a heuristic method based on a simple greedy method is known to achieve the best result [15]. Caching methods do not deploy copies of content to servers in advance; instead, servers autonomously work as caches without the centralized management center. In other words, a server located at the closest position works as a proxy server for each user request. When the closest server does not have the target content, it autonomously downloads the target content from another server, transfers it to the user, and caches it. If the storage space is fully utilized, the target content replaces another content item using a cachereplacing algorithm, such as least frequently used (LFU) or least recently used (LRU). If information about the cached content is shared among cache servers, the target content can be delivered from another cache server having the target content instead of being delivered from the original server. Thus, both the amount of network resources consumed and the response time are reduced. In the caching methods, popular content with larger access counts are more likely to be stored at servers. Therefore, in a practical environment in which the access frequencies of content change dynamically, an ideal caching method that shares the information among servers shows better performance compared with the best replication method [15]. However, if information about the cached content is not shared among servers, many content items will be delivered from the original server. In this case, the performance of the caching methods is degraded compared with that of the replication methods [15]. To reduce the amount of network resources consumed and the response time in a practical environment in which the popularity of content changes dynamically, using the ideal caching method in which information is shared among servers is most desirable. In an ordinary CDN, where a huge number of servers is collocated over multiple networks, use of the ideal caching method has been difficult because the overhead caused by sharing the information among servers is enormous. In the iCDN, on the other hand, the data centers are provided at a limited number of locations, so the centralized control center can easily manage the information on content stored at all the data centers. Therefore, in the iCDN, the ideal caching method is the best choice for the content deployment method. To guarantee content delivery, each content item needs to exist at one or more data centers. Hence, we make one data center among S data centers an original data center, and we assume that the original data center always stores all M content. Other C (= S − 1) data centers work as cache data centers. Hereafter, the original data center and the cache data centers are simply described as the original center and the cache centers, respectively. In the initial state, we assume that all content exists at only the original center. Copies of content are cached at cache centers, and content is deployed over the network automatically as users request it. After starting the iCDN operation, new rich content will be added by content providers. In this case, the content
providers can upload new content only at the original center. 2.2 Delivery Control Sequence The request for rich content m from a user accommodated in node n is transferred to the centralized control center. The control center knows all the cache centers having content m, and it selects one data center including the original center with the shortest hop distance to node n in order to minimize the total amount of traffic within the network. If there are two or more data centers having content m with the shortest hop distance, one data center is randomly selected among them. The control center orders the selected data center to deliver content m to node n. We assume that the Dijkstra shortest hop route is used as the delivery route and that one route is randomly selected when multiple shortest routes exist. LFU and LRU are widely used as cache replacement algorithms, so we assume the use of these policies to update the cached content at each cache center. In the case of LFU, each cache center autonomously records X j , the number of requests for content j during the latest time period W, for all M content. Let Xmin,i denote the minimum value of X j among content cached at the cache center i. In the delivery of content m, content with Xmin,i is replaced by content m at cache center i located on the delivery route, if the following three conditions are satisfied: (i) Xm > Xmin,i , (ii) cache center i does not cache content m, and (iii) the entire storage capacity of cache center i is used. When finding content with Xmin,i , content being delivered to any users is excluded in order to maintain uninterrupted downloading for those users. In the case of LRU, each cache center autonomously records Y j , the time when content j is most recently requested, for all M content. When the storage capacity of the cache center is fully utilized, newly arrived content m replaces content with the oldest Y j . When content sizes are not identical, selecting content to be removed by considering size as well as popularity is effective for improving the cache hit ratio. In this case, Wierzbicki et al. showed that the cache replacement algorithm called LSB (least sent byte), which removes content with the smallest amount of sent data, achieves the highest byte hit ratio (BHR) [20]. The BHR is the total amount of data transferred from the caches divided by the total amount of data transferred. Therefore, although we assume that all M rich content has the same size for simplicity, we can easily extend the iCDN to a case in which content sizes are not identical by using LSB as the cache-replacing algorithm. Content m is simply cached at cache center i located on the delivery route when cache center i does not have content m and the storage capacity of cache center i is not fully utilized. When the cached content is modified, the cache center sends the updated information to the centralized control center. By autonomously updating cache content at each cache center using LFU or LRU, each cache center can always store popular content dynamically. In the case of LFU, how to set the counter update time W is a problem. It seems
KAMIYAMA et al.: OPTIMALLY DESIGNING ISP-OPERATED CDN
793
desirable to set W to a large value and conservatively estimate the content popularity when the popularity rank is stable; on the other hand, it seems desirable to set W to a small value and reflect the popularity trend in a shorter time scale when the popularity rank frequently changes. 3.
Data Center Placement
An ordinary CDN collocates a huge number of servers over multiple networks, so the optimum algorithm designing the server locations has not been well investigated. Many replication methods assume that servers are provided at all nodes and optimally design the deployment positions of each content item [15]. In caching methods, only a few studies, such as the one by Li et al. [16], have investigated the optimum allocation of servers. However, although Li et al. [16] showed an optimum design algorithm based on a dynamic programming approach minimizing the total delivery cost, they assumed a tree-type network topology, and their method cannot be applied to networks with other topologies. Cronin et al. [9] and Qiu et al. [18] proposed some approximation algorithms to optimally place mirror servers among candidate positions on networks with arbitrary topology, and they showed that greedy-based algorithms achieved the best results. However, they investigated the deployment algorithms of mirror servers storing all the content. As far as we know, no optimum design methods have been investigated to place cache servers over networks with arbitrary topology, assuming cache replacement algorithms such as LFU or LRU. Ye et al. [23] recently proposed an optimum method to allocate cache servers to reduce P2P traffic. Their target is closest to the design problem of the iCDN, but their approach is not realistic because they assumed a fixed cache hit ratio without considering cache replacement. The iCDN provides a limited number of data centers with huge storage capacity, so the location of the data centers seriously affects the performance. In this section, we investigate the optimum placement algorithm for data centers. In general, we need O(2N ) calculation time to strictly obtain the optimum allocation pattern in the server placement problem on networks with arbitrary topology. The calculation time exponentially increases with N, so we need to use a heuristic approximation approach except in smallscale networks. In many optimum allocation problems, the greedy algorithms showed the best performance [9], [15], so we investigate a design algorithm based on the greedy method in this paper. As the optimization target, we could consider minimizing the total amount of traffic within the network, i.e., the sum of traffic on all links, or minimizing the maximum link utilization, for example. In this paper, we focus on minimizing the total amount of traffic within the network. This corresponds to minimizing the average hop length of delivery flows, and we can also expect to improve the response time of rich content delivery in addition to minimizing the total amount of network resources consumed.
Let Rn denote the number of requests generated from node n within a long time period T , and define rn as rn = Rn /R, where R is the total number of requests over the network within T . In other words, rn is the demand ratio generated by node n. Moreover, let qn (m) denote the probability that a request generated from node n selects content m. Here, we assume that qn (m) is identical at all nodes and let q(m) denote the selection probability of content m. However, we also investigate how the special locality and time variance in content popularity affect the results in Sects. 4.3.2 and 4.3.3. We also assume that rn and q(m) are estimated for all n and m. The optimality of data center positions will be degraded when actual rn and q(m) deviate from the estimated values. In Sect. 4.3.3, we evaluate the iCDN when these values deviate from the estimated values. Let us investigate the data center placement algorithm for the single original center and C cache centers, separately. 3.1 Placement of Original Center To minimize the average hop length of delivery flows, it is desirable to place the original center at a node located at a central position within the network, i.e., the average hop distance to all other nodes is the smallest. Because all M content is stored at the original center, we define Ho (n), the centrality of node n for the original center, as 1 , i=1 hn,i ri
Ho (n) = N
(1)
where hn,i is the hop length from node n to node i on the shortest hop route and hn,n = 0. We place the original center at the node with the maximum value of Ho (n). Although we define the centrality based on the hop distance to minimize the average hop length, we can also use other criteria, such as the sum of link length or the maximum link utilization on routes, to define the centrality when considering other optimization targets. 3.2 Placement of Cache Centers After placing the original center, we place C cache centers among N −1 candidate nodes. Let V denote the total amount of traffic in the network within the time period T , and we have V = RLh = RL
N
hn rn ,
(2)
n=1
where L is the size of each content, h is the average hop length of delivery flows, and hn is the average hop length of delivery flows to node n. Both R and L are constant even when the locations of cache centers change, so we aim to minimize the average hop length of delivery flows, N hn rn . h = n=1 As mentioned in Sect. 2.1, we assume the caching method to deploy content over data centers, so the set of data
IEICE TRANS. COMMUN., VOL.E96–B, NO.3 MARCH 2013
794
centers having each content m statistically changes. Hence, we cannot strictly derive h when determining the locations of C cache centers. However, as a result of autonomously updating cached content using LFU or LRU at each cache center, popular content is more likely to exist at cache centers. Thus, we estimate h by hˆ =
N hn,min Q(B) + hn,o 1 − Q(B) rn ,
(3)
n=1
where B is the storage capacity of each cache center and up to B content can be stored at each cache center. Let hn,min and hn,o denote the hop distance from the closest data center to node n and from the original center to node n, respectively. Moreover, Q(m) is the cumulative distribution of q(m) and Q(m) = m i=1 q(i). In other words, we assume that all the C cache centers store B most popular content with high probability, and we consider that B most popular content is delivered from the closest data center, whereas the remaining M − B content is delivered from the original center in most cases. Determining how to design C, the number of cache centers, is a problem. When B, the storage capacity of each cache center, is constant, hˆ and the amount of traffic within the network decrease as C increases. At the same time, however, the cost of constructing data centers increases. Hence, we fixedly set the total storage capacity of all cache servers to K content, and we optimally design C as well as the ˆ In other words, we cache center locations minimizing h. set B = K/C. B needs to satisfy B ≥ 1, so the upper limit of C, Cmax , is Cmax = max(N − 1, K). We note that the unit of both B and K is the content count. Although we can also consider the cost of constructing the data center facility, which is independent of the storage capacity, we leave this extension to a future investigation. The greedy method is a heuristic algorithm that places each cache center at a position where we can expect to mostly improve the target function. Let N j denote the node set where j cache centers as well as the original center have been placed. N0 is defined as the node O at which the original center was placed. Let E(n, N j ) denote the reduction effect of hˆ by adding one cache center at node n (n ∈ N \ N j ) for the network with j cache centers at N j , and we have E(n, N j ) =
N
max 0, hi,min, N j − hn,i ri ,
(4)
i=1
where hi,min, N j is the hop distance from node i to the closest data center when j cache centers and one original center are placed at N j . To be exact, we need to multiply the right side of (4) by Q(B). However, the same value is multiplied independently of the position of selected node n, so we omit it. We place the cache centers at nodes with the largest values of E(n, N j ) one by one. Here, we describe the procedure of the optimum placement algorithm of cache centers miniˆ mizing h. (I) Initialize j = 0 and N 0 = O.
(II) For all n of n ∈ N \ N j , derive E(n, N j ) from (4). (III) Place the cache center at node n∗ with the maximum value of E(n, N j ), and update j = j + 1 and N j = N j + n∗ . (IV) Derive hˆ j , the estimated average hop length on the optimum cache center locations when C = j, from (3), and go to (II) if j < Cmax . (V) Select j with the minimum value of hˆ j as the optimum value of C, output the derived cache center locations at the selected j, and complete the procedure. 4.
Performance Evaluation
4.1 Evaluation Conditions In numerical evaluations, we used network topologies of commercial ISP backbone networks publicly available on the CAIDA web page [2]. We evaluated 31 network topologies with a node count N of 10 or more. Table 1 summarizes the node count N, the link count E, the average node degree, the maximum node degree, and H, the average hop length between nodes on the shortest hop routes of these 31 networks. We classify these networks into the following two types. H&S There are several hub nodes with high degree. Other nodes connect to one or more hub nodes, so this type of network is called hub and spokes (H&S). Airline topologies are known to form this structure. A node can reach other nodes with a small hop count through hub nodes, so the hop distance between nodes tends to be small. Here, we define networks with the maximum degree of 10 or more as H&S networks. There are 12 networks of this type. Ladder No hub nodes exist, and the topology structure is a combination of loops. Although the total link cost is reduced, the hop distance between nodes tends to be large. Freeway topologies are known to form this structure. Here, we define networks with a maximum degree under 10 as ladder networks. There are 19 networks of this type. Table 1 also indicates the type of network. The topologies of Savvis Communications and CAIS Internet are examples of each type and are illustrated in Fig. 1. Although both networks are ISPs in the USA, the topological structure is completely different. We generated user requests according to the Poisson process and made rn , the ratio of generation rate at each node n, proportional to the population [5]. Let Λ denote the total generation rate of requests over the network; the request rate from node n was set to rn Λ. We set the number of rich content items as M = 1000. Because we assumed that the sizes of all content were identical, and the unit of B and K was the content count, we did not explicitly set the size of each content item. Yu et al. reported that the access frequency distribution of video content within each continuous 24-hour period
KAMIYAMA et al.: OPTIMALLY DESIGNING ISP-OPERATED CDN
795 Table 1
Basic properties of 31 ISP backbone networks.
Fig. 2
Topology and population of Cable & Wireless network.
Fig. 3 Time series of stored content count at each cache server. (M = 1000, K = 1000, and C = 5)
Fig. 1
Examples of network topologies.
obeyed the Zipf distribution and that the skew parameter θ obeyed the normal distribution with the average of 0.199 and the standard deviation of 0.07, in a VoD service on the Internet [24]. Hence, we assumed the Zipf distribution with θ = 0.199 for q(m) and set q(m)= c/m1−θ , where c is the M q(m) = 1. In the ennormalization constant to make m=1 tire simulation period, the content count M, the popularity ranks, and the time required to deliver each content item, T L , were fixed. In the computer simulation, we set Λ = 1 (/seconds) and T L = 1200 (seconds). Note that the designed locations of data centers as well as the average hop length h are not affected by Λ and T L . 4.2 Content Deployment As mentioned in Sect. 2.1, all content exists at only the original center in the initial state and proliferates over cache cen-
ters as users request content. In this section, we evaluate by computer simulation how many copies of each content item are stored at cache centers using the Cable & Wireless network. We set the total storage capacity of cache centers as K = 1000 and designed the cache center location by the algorithm based on the centrality (hereafter, denoted as the Centrality method) described in Sect. 3. The topology and population of each node in the Cable & Wireless network are shown in Fig. 2. The original center was placed at node 10, from which the average hop distance to nodes with large populations was small. Cache centers were placed at five nodes: 3, 4, 12, 16, and 17. We confirm that cache centers were placed at nodes with large populations, i.e., nodes 3, 4, 16, and 17, or at nodes located among dense quantities of nodes, i.e., node 12. By placing a cache center at node 12, we can greatly reduce the hop length of delivery flows for many destination nodes. The time series of cached content count at each of the five cache centers, when LFU and LRU were used as the cache-replacing algorithm, respectively, are depicted in Figs. 3(a) and (b). When using LFU, we used the accumulated request count observed at each cache center from the beginning of the simulation as X j . Cache centers located at nodes with larger populations, i.e., nodes 4, 16, and 17, received more requests, so the cached content count increased rapidly at these cache centers. Because the storage capacity
IEICE TRANS. COMMUN., VOL.E96–B, NO.3 MARCH 2013
796
Fig. 4 Time series of copy count of each content item. (M = 1000, K = 1000, and C = 5)
of each cache center was B = 1000/5 = 200, the cached content count was constant after reaching 200. No explicit difference was observed between the two cache-replacing algorithms. In a realistic environment, new content is added after the delivery service starts, and the popularity rank changes as time progresses. Thus, we evaluate the influence of the popularity change on the time series of the cached content count. After starting the simulation, we added new content at a time interval obeying an exponential distribution in addition to the M = 1000 content provided at the start of the simulation. In the typical lifecycle of content popularity, a high rank is achieved in the initial stage and then gradually decreases [24]. Hence, we randomly set the initial rank between one and 100 when adding new content and decreased by one the ranks of all existing content with ranks lower than the inserted content rank. To effectively reflect the dynamics of popularity ranks on the cache replacement policy, we reset the observed request counts X j every T r seconds. In evaluation, we set T r = 5 × 104 . Figures 3(c) and (d) plot the time series of cached content count at each of the five cache centers, when LFU and LRU were used as the cache-replacing algorithm, respectively. We also confirm that no explicit difference was observed between the two cache-replacing algorithms. The time series of the copy count including the originals at the original center for the content with popularity ranks of 1, 10, 100, 200, and 1000, respectively, are shown in Fig. 4(a) when LFU was used. We confirmed that content with high popularity always existed at all cache centers, whereas content with low popularity always existed only at the original center. Moreover, the number of copies of content with a popularity rank of around B = 200 slowly changed. In the same way, the results when LRU was used are shown in Fig. 4(b). Because distinguishing the results for multiple content in the same figure is difficult, we show the results only for the content with the popularity rank of 200. We observed that the copy count fluctuated greatly, and we also obtained similar results for content with popularity ranks of 10, 100, and 1000. This is because content was always cached at all cache centers on the delivery route even though their popularity ranks were low when LRU was used. Moreover, the statistical variation of request counts were smoothed for a certain period when LFU was used, whereas the most recent request pattern determined the cached con-
Fig. 5 Copy count of content in popularity rank order. (M = 1000, K = 1000, and C = 5)
Fig. 6 Estimation accuracy of average hop length with static access distribution. (M = 1000)
tent in the case of LRU. As shown in Fig. 3, after about 106 seconds had passed, the content count cached at each cache center reached the cache storage capacity, and we can say that the system went into a steady state. In Fig. 5(a), the copy count including the original of each content item is plotted in descending order of popularity after 106 seconds had passed when LFU was used. We confirmed that content with high popularity existed at many cache centers, whereas many content items with low popularity existed only at the original center. The number of copies of content with a popularity rank of around B was widely scattered. We also depict the results when LRU was used in Fig. 5(b), and we observe that the correlation of the copy count with the popularity rank was weak in that case. Therefore, we confirmed that the content is more efficiently deployed over cache centers according to the popularity rank when LFU is used. 4.3 Estimation Accuracy of Average Hop Length 4.3.1 Case of Stable Popularity Rank and Access Frequency without Spatial Locality In Fig. 6(a), the average hop length of delivery flows obtained by simulation is plotted against C for three cases of K, when we used LFU on the Cable & Wireless topology. The estimation value obtained by (3) is also shown by the bold lines. We designed the locations of cache centers for an arbitrary given value of C by interrupting the process of the Centrality method when C reached the given value. Because the total storage capacity of cache centers K was constant, B decreased as C increased. Hence, the copy count of content with higher popularity had a wider range as C increased. Therefore, the estimation accuracy of the average
KAMIYAMA et al.: OPTIMALLY DESIGNING ISP-OPERATED CDN
797
Fig. 7 Estimation accuracy of average hop length when content access frequency distribution is different among nodes. (M = 1000)
hop length was slightly degraded when C was large. However, in the wide range of K and C, we confirmed that the estimation of the average hop length obtained by (3) was accurate. In the same way, we show the results when LRU was used in Fig. 6(b). As observed in Fig. 5(b), the copy count variance was large when using LRU, so the estimation accuracy of h was degraded† . Hence, we used LFU as the cache replacement policy in the following evaluations. 4.3.2 Case with Spatial Locality in Content Access Frequency Distribution In Sect. 4.3.1, we assumed that qn (m), the access frequency for content m at node n, was identical at all nodes. However, in a realistic environment, the content access frequency is different among nodes. Therefore, we evaluate the estimation accuracy of the average hop length of delivery flows when qn (m) is different among nodes. We set the skew parameter θ of the Zipf function which gave qn (m) at each node according to the normal distribution with the average of 0.199 and the standard deviation of 0.07 [24]. The average hop length of delivery flows observed by simulation as well as the estimated value on the topology of Cable & Wireless are shown in Fig. 7. The estimated values are indicated bold lines, and the actual average hop lengths are indicated by symbols. Instead of (3), we estimated the average hop length by hˆ =
N hn,min Qn (B) + hn,o 1 − Qn (B) rn ,
(5)
n=1
where Qn (m) is the cumulative distribution of qn (m). We confirm that we can accurately estimate the average hop length in the wide range of K and C even when the content access frequency distribution is different among nodes. 4.3.3 Case of Unstable Popularity Rank and Access Frequency In Sect. 4.3.1, we fixed the popularity rank of content as well as the access frequency distribution of content. However, in a realistic environment, new content are added after the delivery service starts, and the popularity rank changes as time progresses. Therefore, we evaluate the influence of a change
Fig. 8 (a) Average hop length versus T r . (b)–(d) Content copy count in descending order. (M = 1000, K = 1000, and C = 5)
in popularity on the estimation accuracy of the average flow hop length. As mentioned in 4.2, after starting the simulation, we added new content at a time interval obeying an exponential distribution in addition to the M = 1000 content provided at the start of the simulation. In Fig. 8(a), the average hop length obtained by simulation on the topology of Cable & Wireless is plotted against T r when we set the average insertion rate of new content, λnc , as 10 or 100 (/day), respectively. The estimated value obtained by (3) is shown by the bold line. Although the estimation accuracy was degraded in the smaller and larger regions of T r , the estimated value agreed with the simulation values in the middle range of T r . For three values of T r , Figs. 8(b), (c), and (d) show the number of copies including the original of each content item in descending order of popularity when the simulation progressed 106 seconds. We set λnc = 100. In the case of T r = 102 seconds, the reset cycle of X j was too short, so the statistical variation of requests strongly affected the cache replacement. In the case of T r = 106 seconds, on the other hand, the old popularity ranks at previous times remained, so the numbers of copies of some popular content added later remained small, whereas the numbers of copies of old content with lowered ranks were still large. When T r = 104 seconds, the content copies were effectively deployed over cache centers according to the dynamics of popularity ranks. Hence, we confirm that we can also accurately estimate the average hop length of delivery flows by suitably setting the LFU reset cycle T r even in a practical environment in which new content is additionally provided and the popularity ranks of content dynamically change. The estimated and actual average hop lengths obtained by simulation when setting T r = 5 × 104 are depicted in Fig. 9(a). The estimated values are indicated bold lines, and the actual average hop lengths are indicated † When K = 10000, B was constantly 1000 in the range of C ≤ 10 because M = 1000. Therefore, content was not replaced in this range.
IEICE TRANS. COMMUN., VOL.E96–B, NO.3 MARCH 2013
798
Fig. 9 Estimation accuracy of average hop length when adding new content or modifying content access distribution. (M = 1000)
by symbols. We confirmed that the average hop length was accurately estimated in the wide range of K and C. Next, we evaluate the estimation accuracy when the distribution of content requests changes dynamically. Here, we changed the skew parameter θ of the Zipf function every hour according to the normal distribution with the average of 0.199 and the standard deviation of 0.07 [24]. The average hop length of delivery flows observed by simulation as well as the estimated values are shown in Fig. 9(b). The obtained results were almost identical to those obtained in Fig. 9(a). We also confirmed that we were able to accurately estimate the average hop length in the wide range of K and C. As a result, we confirmed that we can accurately estimate the average hop length by (3) even when the popularity ranks of content and the request distribution have a spatial locality or they change dynamically. Hence, we evaluate the average hop length using (3) instead of by simulation, hereafter. If the distribution of content requests is much flatter, i.e., the skew parameter θ is much closer to unity, the estimation accuracy of the average hop length might be degraded. We leave the evaluation in this case for future work. 4.4 Comparison of Cache Placement Methods on Cable & Wireless Network Here, we compare some placement algorithms of cache centers on the Cable & Wireless network. We compare the Centrality method with two naive greedy methods. The first algorithm, which we call the Degree method, places cache centers at the highest-degree nodes. The second one, which we call the Population method, places cache centers at nodes with the largest populations. We also compare these three algorithms with the strict optimum allocation (called the Optimum method) in which we check all the possible placement patterns. Many delivery flows take high-degree nodes, and many requests are generated from nodes with large population, so the two naive greedy methods are also expected to reduce the average hop length. The normalized average hop length (NAH) against K for the four cache placement methods is shown in Fig. 10(a). We define the NAH as the average hop length in the iCDN divided by the average hop length when all content is delivered from the original center without using any cache centers. The NAH decreased and the reduction in traffic in-
Fig. 10
Design results in Cable & Wireless network. (M = 1000)
creased as K increased. The reduction effect of the NAH obtained by increasing K was more remarkable in the small-K region. Although the result of the Degree method was larger than that of the Optimum method, the results of the Centrality and Population methods almost agreed with the result of the Optimum method. The designed number of cache centers C for the four methods are plotted in Fig. 10(b). All except for the Degree method placed almost the same number of cache centers. Although the optimum value of C increased as K increased, it was small in the small-K region for all methods except the Degree method. To discover the reason for this, we plotted the NAH when changing B while keeping C constant in Fig. 10(c) and when changing C while keeping B constant in Fig. 10(d), when the Centrality method was used. We confirmed that the NAH was dramatically decreased by increasing B when B was small. On the other hand, the NAH was only slightly decreased by increasing C when B was small. Therefore, when K is small, increasing B is more desirable than increasing C to reduce the NAH, and the optimum value of C is small. We checked the locations where cache centers were placed in the range of 10 ≤ K ≤ 10000. We confirmed that the locations of cache centers deployed by the Centrality method perfectly agreed with the results of the Optimum method, whereas the results of the Population method slightly differed from the results of the Optimum method. 4.5 Comparison on Various Networks Now, we compare the placement methods of cache centers on various networks described in Sect. 4.1. First, we compared the three heuristic algorithms with the Optimum method. Because it is difficult to apply the Optimum method to large-scale networks, we compared the NAH on 13 networks in which N was 22 or less. The NAH values of each of the three heuristic algorithms against the NAH of the Optimum method when setting K = 100 are plotted
KAMIYAMA et al.: OPTIMALLY DESIGNING ISP-OPERATED CDN
799
Fig. 13
Normalized average hop length in 31 networks. (M = 1000)
Fig. 11 Comparisons of NAH in 3 greedy algorithms in 13 networks. (M = 1000, K = 1000 (a)–(c), K = 1000 (d)–(f))
Fig. 14
Designed number of cache servers in 31 networks. (M = 1000)
4.6 Influence of Topological Structure on iCDN Fig. 12 Ratio of average hop length in Centrality and Population methods in 31 networks. (M = 1000)
in Figs. 11(a)–(c). We also show the results when K = 1000 in Figs. 11(d)–(f). We confirmed that we could obtain strictly optimum results by using the Centrality or Population method on all 13 networks. When using the Degree method, however, the NAH increased on many networks. Next, we compare the Centrality method with the Population method for all 31 networks. In Fig. 12, the ratio of average hop length is plotted against the node count N for 31 networks when K was 100, 1000, and 10000, respectively. The ratio of average hop length was the average hop length of the Population method divided by that of the Centrality method. In many networks with N > 22, the ratio of average hop length was greater than unity, so we confirmed that the NAH of the Population method was degraded compared with that of the Centrality method in many large-scale networks. This degradation was much greater in ladder networks compared with that in H&S networks. This is because the hop length of flows tends to be large in ladder networks, and ladder networks are more likely to have nodes located at central positions at which we can effectively reduce the hop length of delivery flows for many users by placing cache centers. In ladder networks, the superiority of the Centrality method against the Population method was more noticeable when N was larger or when K, i.e., C, was larger. When K = 1000, for example, the NAH of the Centrality method was 10 to 30% smaller than that of the Population method in many ladder networks. The calculation time of all three heuristic design methods was O(N 3 ), so it is always desirable to use the Centrality method.
For three values of K, the NAH is plotted against N for 31 networks when the cache centers were placed by the Centrality method in Fig. 13. The iCDN reduced the average hop length of delivery flows in the large range of N. For example, by desirably placing cache centers at a limited number of locations within the network and providing the total storage capacity of cache centers just around the storage capacity of the original center (K = M = 1000), we can reduce the average hop length 40 to 60%. This means that the total amount of network resources consumed is also reduced 40 to 60%, so ISPs can dramatically reduce the cost of delivering rich content to users. The effect of the iCDN is more remarkable in ladder networks in which the average hop length of flows tends to be large. The designed value of C against N for 31 networks when the Centrality method was applied is shown in Fig. 14. The optimum value of C was independent of N, and the topological structure of the networks determined the performance. Ladder networks are more likely to have few nodes at which we can effectively reduce the average hop length by placing cache centers, such as node 12 in the Cable & Wireless network. Hence, the optimum value of C tended to be small in ladder networks† . In H&S networks, on the other hand, there are few hub nodes with high degree, and the original center was placed on one of these hub nodes. Thus, the average hop length tended to be small even when we do not use cache centers. Placing cache centers at nodes with low degree could reduce the average hop length only for few node pairs, so we need to place cache centers at many nodes to further reduce the average hop length. Hence, the optimum value of C was large in H&S networks. In summary, providing the iCDN over ladder-type networks is effective to reduce the average hop length of de† When K = 10000, B took a constant value of 1000 when C ≤ 10, and there was no merit in setting C < 10. Thus, C = 10 in many ladder networks.
IEICE TRANS. COMMUN., VOL.E96–B, NO.3 MARCH 2013
800
livery flows, i.e., the amount of traffic consumed. When we provide the iCDN in ladder networks, the optimum number of cache centers is about five or smaller, independent of the node count N. The iCDN is effective for delivering rich content because it places cache centers at a limited number of locations within the network, instead of providing a huge number of servers, such as in an ordinary CDN. In [12], we also investigated the properties of the topological locations of nodes at which cache placement is effective. 5.
Conclusion
To deliver large size, rich content to users economically and efficiently by suppressing the amount of network resources consumed, the use of an ISP-operated CDN (iCDN), in which an ISP places data centers with huge storage capacity at limited locations, is effective. In this paper, we investigated the content deployment method and the delivery control sequence suited for the iCDN. Moreover, we presented an optimum placement method of original and cache centers that is based on a greedy heuristic algorithm and that provides results close to the strictly optimum allocation pattern. We evaluated the iCDN using 31 topologies of commercial ISP backbone networks. Providing the iCDN over ladder-type networks is effective to reduce the average hop length of delivery flows and the amount of traffic consumed. Although the total link cost is reduced in ladder networks, the average hop length of flows tends to increase. Reducing the hop length is important when delivering rich content in ladder networks. By applying the iCDN on ladder networks, we can effectively use the link resources while suppressing the total cache server cost. References [1] [2] [3] [4]
[5]
[6] [7] [8] [9]
[10] [11]
[12]
Akamai Technologies, http://www.akamai.com CAIDA, http://www.caida.org/tools/visualization/mapnet/Data/ at&t, http://www.att.com L. Cherkasova and J. Lee, “FastReplica: Efficient large file distribution within content delivery networks,” 4th USENIX Symposium on Internet Technologies and Systems (USITS), session of Web Servers and CDNs, March 2003. K. Cho, K. Fukuda, H. Esaki, and A. Kato, “The impact and implications of the growth in residential user-to-user traffic,” ACM SIGCOMM 2006, pp.207–218, Sept. 2006. CoDeen, http://codeen.cs.princeton.edu B. Cohen, “Incentives build robustness in BitTorrent,” 1st WorkShop on Economics of Peer-to-Peer Systems, June 2003. Coral CDN, http://www.coralcdn.org E. Cronin, S. Jamin, C. Jin, A. Kurc, D. Raz, and Y. Shavitt, “Constrained mirror placement on the Internet,” IEEE J. Sel. Areas Commun., vol.20, no.7, pp.1369–1382, Sept. 2002. C. Huang, J. Li, and K. Ross, “Can Internet video-on-demand be profitable?,” ACM SIGCOMM 2007, pp.133–144, Aug. 2007. N. Kamiyama, T. Mori, R. Kawahara, and H. Hasegawa, “ISPoperated CDN,” IEEE Global Internet Symposium 2009, pp.49–54, April 2009. N. Kamiyama, T. Mori, R. Kawahara, S. Harada, and H. Hasegawa, “Analyzing influence of network topology on designing ISP-operated CDN,” Springer Telecommunication Systems, DOI: 10.1007/s11235-011-9605-2, Online First, 2, Sept. 2011.
[13] J. Kangasharju, J. Roberts, and K. Ross, “Object replication strategies in content distribution networks,” Comput. Commun., vol.25, no.4, pp.376–383, March 2002. [14] T. Karagiannis, P. Rodriguez, and K. Papagiannaki, “Should Internet service providers fear peer-assisted content distribution?,” ACM IMC 2005, pp.63–76, Oct. 2005. [15] M. Karlsson and M. Mahalingam, “Do we need replica placement algorithms in content delivery networks?,” WCW 2002, pp.117–128, Aug. 2002. [16] B. Li, M. Golin, G. Italiano, and X. Deng, “On the optimal placement of web proxies in the Internet,” IEEE INFOCOM 1999, pp.1282–1290, March 1999. [17] Limelight Networks, http://www.limelightnetworks.com [18] L. Qiu, V. Padmanabhan, and G. Voelker, “On the placement of web server replicas,” IEEE INFOCOM 2001, pp.1587–1596, April 2001. [19] A. Su, D. Choffnes, A. Kuzmanovic, and F. Bustamante, “Drafting behind Akamai,” ACM SIGCOMM 2006, pp.435–446, Sept. 2006. [20] A. Wierzbicki, N. Leibowitz, M. Ripeanu, and R. Wozniak, “Cache replacement policies revisited: The case of P2P traffic,” CCGRID 2004, pp.182–189, April 2004. [21] H. Xie, Y. Yang, A. Krishnamurthy, Y. Liu, and A. Silberscheatz, “P4P: Provider portal for applications,” ACM SIGCOMM 2008, pp.351–362, Aug. 2008. [22] D. Xu, S. Kulkarni, C. Rosenberg, and H. Chai, “Analysis of a CDNP2P hybrid architecture for cost-effective streaming media distribution,” Multimedia Systems, vol.11, no.4, pp.383–399, April 2006. [23] M. Ye, J. Wu, and K. Xu, “Caching the P2P traffic in ISP network,” IEEE ICC 2008, pp.5876–5880, May 2008. [24] H. Yu, D. Zheng, B. Zhao, and W. Zheng, “Understanding user behavior in large-scale video-on-demand systems,” ACM EuroSys 2006, pp.333–344, April 2006. [25] YouTube, http://www.youtube.com
Noriaki Kamiyama received his M.E. and Ph.D. degrees in communications engineering from Osaka University in 1994 and 1996, respectively. From 1996 to 1997, he was with the University of Southern California as a visiting researcher. He joined NTT Multimedia Network Laboratories in 1997. Since then, he has been engaged in research concerning pricing methods for IP networks, content distribution systems, optical networks, IP traffic measurement, and network design. He is a member of the IEEE and the Operations Research Society of Japan.
Tatsuya Mori is currently a senior researcher at NTT, Tokyo, Japan. He received B.E. and M.E. degrees in applied physics, and a Ph.D. degree in information science from Waseda University, Tokyo, Japan, in 1997, 1999, and 2005, respectively. Since joining NTT Corporation in 1999, he has been researching the management of large-scale networked systems and network security. From Mar 2007 to Mar 2008, he was a visiting researcher at the University of Wisconsin-Madison. He received the Telecom System Technology Award from the Telecommunications Advancement Foundation (TAF) in 2010 and Best Paper Awards from IEICE and IEEE/ACM COMSNETS in 2009 and 2010, respectively. Dr. Mori is a member of ACM and IEEE.
KAMIYAMA et al.: OPTIMALLY DESIGNING ISP-OPERATED CDN
801
Ryoichi Kawahara received a B.E. and M.E. in automatic control, and a Ph.D. in telecommunication engineering from Waseda University, Tokyo, Japan, in 1990, 1992, and 2001, respectively. Since joining NTT in 1992, he has been engaged in research on traffic control for telecommunication networks. He is currently working on teletraffic issues in IP networks in NTT Network Technology Laboratories. He is a member of the Operations Research Society of Japan.
Haruhisa Hasegawa is a Senior Research Engineer, Supervisor, and Group Leader in the Traffic Solution Group working on the Communication Traffic & Service Quality Project at NTT Network Technology Laboratories. He received the B.E., M.E., and Dr. of Information and Computer Science from Waseda University, Tokyo, in 1988, 1990, and 2001, respectively. He joined NTT in 1990 and has been researching traffic engineering and management in telecommunications networks. From 2000 to 2003, he was engaged in network management at the network operation center of NTT East. He is a member of the IEEE Communications Society.
