Provisioning for Bursty Internet Traffic: Implications for Industry and Internet Structure Dr. David Clark MIT Laboratory for Computer Science
[email protected] Dr. William Lehr MIT Internet &Telecoms Convergence Consortium & Columbia University
[email protected] Mr. Ian Liu, MIT Presented at the MIT ITC Workshop on Internet Quality of Service, November, 1999. Abstract1 Internet traffic, as epitomized by Web browsing behavior, is very bursty, or equivalently, the ratio of the peak to average data rate is quite high. To handle the offered traffic, the network must be sized to handle the peak load. Because the peaks of individual users are typically uncorrelated, the network peak load grows much more slowly than the sum of the peak loads of the individual subscribers whose traffic is carried by the network. This implies there are provisioning scale economies associated with aggregating traffic. That is, service providers that are able to aggregate the traffic of a larger number of users are likely to have lower capital and operating costs and there may be a minimum efficient scale of operation for Internet Service Providers (ISPs). In the current environment where users are accustomed to a relatively poor grade of service (i.e., long packet delays are tolerated) and when most users access the Internet via dial-up modems that limit the possible peak to average load ratio, the impact of these scale economies on industry structure are likely to be small. However, with increased quality of service (QoS) expectations (e.g., addition of delay-intolerant real-time services) and the spread of broadband services offering the potential for much higher peak to average load ratios, the provisioning problem may grow in importance. Although models of aggregate traffic flows have been developed [Kelly, 1998] and others have speculated about the nature of interconnection agreements across the Internet hierarchy [Bailey and McKnight, 1997; Lehr, 1998], we are unaware of any work that attempts to argue from traffic characteristics to industry structure.
1
The simulations on which this paper are based were prepared by Ian Liu in the course of completing his Master's Thesis at MIT (see Ian Liu, 1999), but Mr. Liu did not participate in the preparation of this draft. Drs. Clark and Lehr would like to acknowledge the support of the MIT Internet and Telecoms Convergence Consortium (http://itel.mit.edu). --------
Clark & Lehr, Provisioning for Bursty Internet Traffic
Page 1
MIT WISQ, December 1999
This paper offers a first effort at filling this gap. We present the results from a simulation of multiple bursty sources to examine the extent of aggregation efficiencies and the possible implications for minimum efficient scale and industry structure. Although the results presented here are preliminary and based on an overly simple traffic model, they suggest a number of interesting avenues for further research. For example, our results indicate that substantial aggregation economies are realized even with extremely bursty traffic and that these benefits are further enhanced with only minimal buffering. This means that over-provisioning to meet QoS performance goals is likely to be less expensive than might otherwise be expected. The relatively moderate levels of aggregation required suggests that substantial competition in broadband access services may be feasible in densely populated areas but that this may be more difficult to achieve in less populous areas.
1. Introduction The purpose of this paper is to look at two issues in the design of the Internet, and the relationship (possibly contradictory) between them. One issue is the powerful economic advantage that comes from the statistical multiplexing of Internet traffic from many users, and the other is the increasing desire to make explicit service commitments to users, either singly or in aggregate. This is both a technical and an economic question. From a technical perspective, our goal in this paper is to understand if we can make explicit service commitments, and still provide service based on statistical aggregation of traffic. This relates to the optimal level of utilization that is compatible with meeting QoS commitments for minimal delays. From an economic perspective, our goal is to examine the implications of statistical aggregation on the costs of network provisioning and industry structure. These two perspectives are related because the lower the optimal utilization level, the greater the investment in over-provisioning required to sustain any given level of QoS. In Section 2, we explain the provisioning problem posed by bursty Internet traffic and in Section 3 we describe a simulation model we used to examine how this problem is affected by aggregating multiple sources. Section 4 presents the results of this analysis and in Section 5 we speculate about the possible implications of these results for industry structure and the architecture of the Internet. Section 6 offers conclusions and suggestions for further research. 2. Provisioning for Bursty Internet Traffic Traffic aggregation is a key feature of Internet design: traffic from many sources is combined and carried over shared trunks. The capacity of the trunks is shared in a dynamic manner among all of the sources that are sending at any instant. The Internet does not make explicit bandwidth commitments to each source, but depends on the statistics of aggregation to provide a reasonable service to all these flows. A typical traffic source on the Internet does not generate data at a constant rate, but is very bursty in nature. A person cruising the Web, for example, alternates between transferring a page and looking at it. User satisfaction is increased if each active transfer occurs as quickly as --------
Clark & Lehr, Provisioning for Bursty Internet Traffic
Page 2
MIT WISQ, December 1999
possible, so the resulting network load is a series of very high bandwidth bursts intermixed with relatively long periods of silence. Other activities generate bursty traffic patterns—retrieving a succession of mail messages, or interacting in a chat room. It is this bursty nature of traffic that makes statistical aggregation of sources such a powerful feature of Internet operation. If an Internet Service Provider (ISP) had to commit a steady-state bandwidth sufficient to carry the peak transfer rate of the traffic bursts for each user, the resulting trunk utilization would be unacceptably small. If the ISP constrained the user to a rate equal to his long-term average rate, then each transfer would be spread out over such a long period that the service would be very unsatisfactory. But by combining many users whose peak utilization is uncorrelated in time2 and letting them share a common bandwidth pool in a dynamic manner, the peak demands of all these users can be satisfied most of the time. 2.1. The heavy-tailed observation about Internet traffic To characterize the nature of aggregation, it is necessary to understand and model the burstiness of the Internet traffic. The evidence is that the traffic is in fact very bursty. The distribution of burst sizes seems to be heavy-tailed in nature, which means that most of the bursts are small, but there are a small number of bursts that are very large. Intuitively, this distribution of transfer sizes could be produced by visiting a sequence of mostly-textual Web sites, mixed with occasional images and the very occasional download of software. One model proposed for the distribution of burst sizes in Web transfers is provided by the Pareto distribution, which says that the probability that a given burst is larger than x bytes is (1/x)a. If the parameter a is substantially greater than 1, this distribution is fairly well behaved, but as a approaches 1, the tail of the distribution curve becomes heavier, and as a becomes less than 1, the mean of the distribution becomes infinite. (Of course, in practice, other effects bound the length of any single transfer.) One recent measurement of Web transfers found a good match with the Pareto distribution with a near 1.1. {Cite Mor.] 2.2. Dealing with overload The statistical nature of traffic aggregation means that there is no guarantee that there is enough capacity in the Internet to carry all the offered load at any instant. Some level of congestion is to be expected, especially during peak periods of usage. The design of the Internet deals with congestion in a straight-forward manner—when congestion is detected, the sources of traffic are expected to slow down, and when there is no congestion, they are permitted to speed up. (In fact, given this rule, some degree of congestion is the norm, since sources will speed up until congestion slows them down.) The implication of this approach to congestion is that the actual rate that any source can achieve at any instant is unpredictable.
2
That is, as long as the peaks are unlikely to occur simultaneously, the peak of the aggregate load will grow much more slowly than the sum of the peak of individual loads. --------
Clark & Lehr, Provisioning for Bursty Internet Traffic
Page 3
MIT WISQ, December 1999
2.3. Making QoS commitments—does it break statistical multiplexing? Historically, the unpredictable nature of Internet service has been tolerated by the users, but there is an increasing call today for more predictable and higher quality service over the Internet. This demand is driven both by the increased commercial importance of the services supported over the Internet (i.e., the Internet has become a mission-critical resource) and by the desire to support new delay-intolerant services such as IP telephony and streaming audio/video. Service level agreements committing providers to delivering pre-specified levels of service are becoming common both between Internet Access Providers (IAPs)3 and their end-user customers and between backbone ISPs in their interconnection agreements. There is thus a potential collision between the two goals of effective and efficient statistical multiplexing on the one hand, and predictable service contracts on the other. This problem can be considered at several levels. At the edge of the network, individual users are likely to want some sort of contract that describes the service quality they can expect. For these users, it will be necessary to develop a variety of usage profiles that will allow capturing of the benefits of statistical multiplexing, while giving the user some assurance as to what his service will be like. Multiple profiles are likely to be needed to accommodate different types of customer traffic loads and to provide the sales and marketing departments with adequate flexibility in designing and pricing their service offerings. With suitably defined profiles, network planners may use statistical aggregation to economize on capacity requirements while offering QoS commitments that are easily understood by consumers and that will be met with a very high level of probability. The advantage of this approach is that it allows the provider to separate network provisioning from sales and marketing. While the design of these profiles is likely to be an important component in determining the way in which Internet services evolve and future industry structure, these are not the principal focus of this paper. Here we look at another aspect of service contracts—those that occur between service providers where two ISPs or an ISP and IAP interconnect. Consider the prototypical case of a small IAP, serving some number of individual customers, that wants to connect to a larger backbone ISP to obtain wide area connectivity into the rest of the Internet. If that IAP has given some sort of service contracts to its customers, then it needs to purchase a service contract with the backbone ISP that is sufficient to cover the aggregated commitments to its own customers. In other words, to enable this world of service commitments, we must figure out how to aggregate commitments, as well as aggregate actual traffic. This is both a business and a technical problem. For example, the more difficult it is to negotiate market-based servicelevel interconnection contracts with QoS commitments, the more likely it is that the industry will need to be vertically integrated.
3
In this paper we will use the term Internet Service Provider (ISP) to refer both to backbone providers who provide connectivity across the Internet cloud and Internet Access Providers (IAPs) who provide the access connections to end-user locations at the edges of the network. In many cases a single provider owns and operates both backbone and edge-network facilities, but the distinction remains useful in terms of discussing both the architecture of the network and industry structure. --------
Clark & Lehr, Provisioning for Bursty Internet Traffic
Page 4
MIT WISQ, December 1999
2.4. How can providers provision for such traffic? Informal discussions about aggregation of commitments has reflected some skepticism that this can be done effectively. There are several possible things that can go wrong as an IAP aggregates its commitments. One is that the resulting commitment is itself very bursty. This means that the connection between the access network and the backbone network must be sized to carry the peak load, but is on the average underutilized. The other undesirable outcome is that to eliminate this bursty behavior of the aggregated commitment, the access ISP is forced to constrain his individual customers to be less bursty. Both of these outcomes represent a compromise—in the one case the access provider must purchase a link to the backbone that is not fully utilized, even at peak periods, and in the other case the users cannot be given the usage profile that they really want. The fears of this outcome are increased by the presence of the heavy-tailed traffic patterns in the offered load. If the user is allowed to offer this heavy-tailed traffic into the network (in other words, if the user is allowed to have the service profile that actually fits his needs), then this heavy-tailed character may cause the shape of the aggregated commitment to have some sort of heavy-tailed profile itself. If true, this would mean that links both at the edges and in the backbone of the network would need to be very lightly loaded in order to meet QoS requirements. Some researchers [Odlyzko, 1999] argue that we should expect lightly loaded links in the Internet and that this is indeed the best way to address the QoS problem. Proponents of this view argue that approaches that require modification of the protocol suite to support QoS are more expensive than simply putting in excess capacity. They argue that the costs of implementing alternative QoS solutions are likely to require too much overhead to administer and will require expensive changes to the existing infrastructure that will be difficult to coordinate. Whether this perspective is correct or not depends on the cost of carrying excess capacity relative to the costs of implementing alternative solutions. Ceteris paribus, the more excess capacity that is required, the higher the relative costs of over-provisioning. Therefore, a better understanding of how heavy-tailed traffic is likely to aggregate will shed useful light on the debate of whether it is better to over-provision or to implement QoS mechanisms that allow more efficient sharing of constrained capacity. If one accepts the view that even links in the backbone of the network are likely to be lightly loaded, then this may have important implications for how traffic is metered in the network; the types of contracts one should expect to see between service providers; and whether sustainable pricing equilibria are likely to exist among service providers. Let us consider each of these points in turn. First, if one accepts the premise that backbone links will be lightly loaded in order to accommodate a high peak to average traffic ratio, effective periodic traffic sampling is more difficult to implement and metering costs are likely to be higher. There is also less value from
--------
Clark & Lehr, Provisioning for Bursty Internet Traffic
Page 5
MIT WISQ, December 1999
knowing what average usage is because errors in this estimate represent a smaller component of the total capacity and operating costs.4 Second, because metering traffic is a pre-requisite for implementing usage-based pricing, this may make it less likely that usage-sensitive contracts will be widely used in the Internet. There are three basic reasons why usage-sensitive pricing may be desirable: (1) cost recovery; (2) shape customer demand (e.g., congestion pricing a la Mackie-Mason and Varian, 1995); and, (3) facilitate price discrimination.5 In the backbone of the network, only the first two of these motivations are likely to be relevant. From the perspective of cost recovery, relying on usagebased contracts makes less sense if usage cannot be accurately metered;6 and, usage pricing in the backbone to affect customer behavior at the edges is less attractive if the overall strategy for delivering QoS is to over-provision.7 The absence of reliable metering mechanisms also may slow the move to usage-based contracts because of the higher costs associated with monitoring, verification, and enforcement.8 Moreover, if one elects to address the QoS problem by simple over-provisioning, then it will be harder to justify the costs for metering traffic in any case.
4
In general, one would expect total provider costs to be a function of both the peak (X) and the average usage (x), or C(X,x). Letting the peak-to-average ratio be given as Z=X/x, we can re-write total costs as either C(xZ,x) or C(X,X/Z). As Z gets very large, the cost function is likely to be reasonably approximated by C(X,0), or equivalently, knowing the true value of x is not important for estimating costs. In contrast, if Z gets very small, the cost function may be approximated by C(0,x), or equivalently, knowing the true value of x provides a good basis for estimating peak traffic and hence the total cost function. Therefore, the informational value of knowing x is less when Z is high.
, then both of these need to be known to estimate total costs 5
Price discrimination as a motivation for introducing usage pricing is likely to be less important in the backbone of the network where contracts will be between sophisticated businesses and other contracting features may be manipulated to effect the same goals (e.g., term commitments, technical specifications for interconnection, etc.).
6
Consider a standard two-part tariff with a flat monthly component, F, and a per-unit usage charge, a. While it is possible to recover a fixed amount of revenue, C (assume equal to total cost), by either a pure flat rate charge (i.e., F=C) or by a pure usage charge (i.e., a=C/T, where T is total usage), the flat rate approach that is currently used makes more sense if forecasting (and metering) total usage is difficult (i.e., T is unknown).
7
That is, with over-provisioning, there is less congestion for everyone, the aggregate externality is therefore smaller, and the potential efficiency gains from internalizing the externality via congestion pricing is also smaller. Note, also, that if congestion pricing is being used to shape customer demand that flat rate charges that are designed to recover costs must also be adjusted or the carrier will recover more than its costs (excess profits). See Lehr and Weiss (1996) for a fuller discussion of the problems of congestion pricing.
8
Disputes involving usage-based contracts will require that actual usage be verifiable by third parties in order to assure adequate contract enforcement. Higher metering costs imply higher enforcement costs and therefore a higher incidence of moral hazard problems (i.e., incentives to deviate from the agreed contract terms is greater if enforcement is less likely). --------
Clark & Lehr, Provisioning for Bursty Internet Traffic
Page 6
MIT WISQ, December 1999
At the edges of the network, traffic profiles that take account of usage behavior are likely to still prove useful if the peak to average ratio is large. Price discrimination may be needed to facilitate cost recovery and, in any case, is likely to be profit maximizing.9 Also, if usage pricing is being used to shape customer demand, then customers need to face prices that vary with their offered demand. However, even at the periphery, a high peak-to-average ratio is likely to complicate the nature of these usage-sensitive contracts. The above discussion may provide a partial justification for the current practice of not using usage-based contracts widely. In the backbone, carriers typically peer using "bill and keep" wherein carriers agree to terminate each other's traffic for a zero price.10 Similarly, interconnection agreements with non-peering carriers usually involve a flat rate for the connections (no usage component) based on the capacity of the link (i.e., the potential peak throughput rate).11 If the peak significantly exceeds the average level of traffic, then this approach makes sense since it obviates the need to incur the costs of measuring traffic. Third, and finally, a high peak to average ratio that makes over-provisioning the preferred strategy both in the backbone and at the edges of the network may make it more difficult to sustain a pricing equilibrium that will allow carriers to recover the costs of their network facilities. If true, this could have very negative economic consequences. Once “excess” capacity is installed, there is a temptation to sell it, rather than let it sit mostly idle as a cushion for occasional traffic peaks. The more excess capacity, the greater the temptation to sell the capacity and the greater the likelihood that wholesale markets will emerge to facilitate such sales (see Lehr and McKnight, 1998). Because most of the costs of providing the capacity will be regarded as effectively sunk at the time that the carrier seeks to sell available excess capacity, prices could collapse to the very low incremental operating costs of using already-installed capacity. The threat of this sort of destructive price competition from excess capacity can deter investment. Therefore, too much excess capacity in the network may reduce the likelihood of a stable and sustainable economic equilibrium. Therefore, it is important to understand better the characteristics of how Internet traffic is likely to be aggregated and what this means for opportunities to offer QoS commitments over a network that is not fully provisioned (i.e., utilizes statistical multiplexing).
9
With heterogeneous customers, it is likely to be better to offer a menu of (T,a) combinations to allow subscribers to self-select into different revenue classes. This may be needed to facilitate cost recovery if some subscribers would elect not to join the network if T is set at the level that recovers average costs. More generally, using multipart tariffs will facilitate the carriers ability to extract revenue from customers.
10
This approach makes sense as long as the traffic is balanced in both directions or the costs of terminating traffic are sufficiently small (close to zero).
11
Increasingly, Tier 1 carriers are metering usage with lower tier service providers. This metering is usually-based on statistical sampling of the actual traffic. Tier 1 carriers may be pricing in this way to price discriminate or to provide incentives to these lower tier carriers to more accurately forecast their traffic requirements. --------
Clark & Lehr, Provisioning for Bursty Internet Traffic
Page 7
MIT WISQ, December 1999
3. Simulated Heavy-Tailed Traffic Aggregation The approach we take in this paper is a very simple but direct attempt to understand what aggregates of heavy-tailed traffic flows look like. The goal is to look at the overall behavior of aggregates of different sizes, and to relate this to the provisioning decisions necessary to carry this traffic without distorting it—in other words to carry the aggregate in such a way that the individual service commitments are honored.12 In the following sub-section, we first discuss the distribution for offered load that we picked, and our method of modeling the resulting aggregated traffic. We then discuss the resulting aggregate behavior. After this, we speculate on the implications of these results for the access and backbone ISP, and from this draw some general conclusions about industry structure. 3.1. Modeling the bursty Web cruiser as self-similar traffic. For this work, we assumed that all users were identical in their usage pattern, which we picked to resemble the pattern of using the Web. We did not concern ourselves with the issues of heavy and light users, or the broader question of mixing users that are doing very different things. We return to that larger issue later. Our model of offered load was an alternating series of “on” and “off” periods, presumably matching the ”transfer” and “look at” phases of a user exploring the Web. We assumed that both the on and off periods were described by the Pareto distribution as noted above. In this paper, we report results for one particular ratio of on and off times. We assume that the user has a 1% duty cycle—on for 1% of the time. This is the same thing as saying that the transfer rate of the traffic during the on period (the peak rate) is 100 times the average rate. Given this characterization of an individual source, we then generated the resulting behavior of the aggregate of a number of these. We did this by direct computation. We wrote a program that computed a succession of on and off periods for a number of such flows, stepped this program forward through time, looking at successive intervals of about a packet time in duration, and computed the peak aggregate rate that we saw, as this simulation ran for some significant amount of simulated time, about three hours. The program was efficient enough that we could simulate as many as ten thousand or even a hundred thousand sources. We ran a number of experiments, looking both at different levels of aggregation and different values of the Pareto parameter a.
12
We did not concern ourselves in this project with how a service contract to the individual users would be expressed. What we assumed was that there would be some such profile which would allow the presumed individual behavior (heavy-tailed transfers of Web pages) to happen. --------
Clark & Lehr, Provisioning for Bursty Internet Traffic
Page 8
MIT WISQ, December 1999
4. Bursty Traffic Aggregation Results 4.1. Simple aggregation of multiple sources Figure 1 illustrates the results of aggregating different numbers of sources. The horizontal axis represent the number of sources. The vertical axis represents the normalized bandwidth required per source. To understand this representation, consider the extreme cases. For one source, if we are to carry the offered traffic without distortion, we must carry the peak load, which is 100 times the average. So the graph for 1 source indicates that we must provide 100 units of capacity per source. At the other extreme, in the limit as we aggregate more and more traffic, we can provision for the average rate, which is normalized to 1. So, not surprisingly, the curve falls from the peak rate per source of 100 (for this particular case of 1% utilization) toward the average rate of 1. What is interesting is the shape of the curve. We show the curve for a number of values of the Pareto parameter a, and (for comparison) for a situation where the distribution of burst lengths is described by a Poisson distribution, which has been a traditional model for network analysis. The first observation is that until a goes below 1, and the mean of the distribution becomes unbounded, all the curves look rather similar. It does not matter much if the individual sources are Poisson or heavy-tailed. As we aggregate enough of them, the resulting behavior is similar. This is important because it suggests that if we suspect that a is actually likely to be above 1 that the problem of self-similar or heavy-tailed traffic may be far less severe than originally suggested and that traditional approaches to provisioning using standard modeling assumptions may be reasonable. Second, the curve tends toward the asymptote for moderate numbers of active sources. It does not require millions of sources to achieve reasonable aggregation. This suggests that the minimum efficient scale for an IAP is sufficiently small to support substantial competition, at least in more densely populated areas. As an example, consider the curve at an aggregation level of 1000. The capacity value is approximately 3. What this means is that the access provider, to carry this aggregated traffic, must purchase a link of capacity 1000x3 units (where the unit is the average rate of the source). This link will be 1/3, or 33% utilized on the average, and will have just the capacity to carry the peak load that can be expected. When a is less than one, the mean of the distribution becomes unbounded. In this case, the traffic does not aggregate as well. At an aggregation level of 10,000 sources, the link capacity to carry this traffic is about twice what is needed for the more well-behaved traffic. So if traffic on the network really looked like this, it would be necessary either to limit or bound the behavior of the individual sources, or to tolerate quite under utilized links inside the network. However, these values for a have not been reported in practice. As noted above, a = 1.1 has been proposed from one study [Barford and Crovella, 1998]. The peak data rates represented in Figure 1 are very conservative. They represent the peak instantaneous data rates seen from the aggregate of the sources if they are allowed to operate without any constraint. Most networks today do not attempt to carry the absolute peak offered traffic, but provide some degree of buffering to smooth these peaks out. We can expect future service contracts to permit the network to buffer (e.g. smooth or delay) traffic to some degree in order to clip the peaks of the offered traffic. --------
Clark & Lehr, Provisioning for Bursty Internet Traffic
Page 9
MIT WISQ, December 1999
Figure 2 shows what the peak rates would be if the traffic in Figure 1 were subjected to a particular buffering regime, in which we limit the amount of buffering so that no single packet is delayed by more than 100 ms. This number was picked because we assume that most users cannot perceive a distortion in their traffic transfer pattern of a tenth of a second or less. Allowing packets to be delayed by a second would represent a much larger distortion, while a tighter bound would not seem to improve the actual user experience. For comparison, total delays measured across the Internet cross-country today are in the order of 100 ms. As Figure 2 shows, the less heavy-tailed the traffic, the more benefit of a buffer. This is not surprising—the impact of heavy-tailed distributions is to produce occasional very long flows, which cannot be influenced much by adding a 100 ms. buffer. It is the occasional long flow that drives the peak rate up. However, buffering is helpful and improves the aggregated behavior, even for heavy-tailed traffic. For comparison, Figure 3 shows the increasing reduction in peak requirements if the distortion due to buffering is stretched to 250 ms. 4.2. Translating to real numbers The preceding results are expressed in terms of arbitrary flows with a unit average rate and 1% duty cycle. What might they mean for real users of the Web? Take as a rough estimate that the average rate for cruising the Web is 10Kbps. This would allow the downloading of a 12 KByte Web page every 10 seconds. Based on informal discussions with ISPs, this is actually a somewhat high estimate of the needed level of provisioning. Given that rate, however, the peak rate per user is 1 Mb/s, or 100 times the average rate. Not surprisingly, many “broadband” access systems operate in this range, in order to sustain the peak rate for the user in a reasonable way. Using the curve from Figure 1, representing no buffering, and the curve from Figure 2, corresponding to a system with a buffer of no more that 100 ms, we can compute the approximate total amount of capacity needed to carry total loads at different levels of aggregation. # of sources
No buffering
100 ms buffering
100
8.4 Mbps
12%
4.4 Mb/s
23%
1000
30 Mbps
33%
17 Mb/s
59%
10,000
170 Mbps
59%
120 Mb/s
83%
Table 1: Approximate total rates and average link utilization to carry offered load at different levels of aggregation. The numbers in column 1 represent active users. Not all users can be expected to be active at once, even during a busy hour. Again, using a very rough guess that 20% of subscribers might be active during busy periods, we can see that an access network with 500 subscribers could carry the aggregated traffic using a link of capacity between 4.4 and 8.4 Mbps (roughly, --------
Clark & Lehr, Provisioning for Bursty Internet Traffic
Page 10
MIT WISQ, December 1999
somewhat under 10 Mbps) to serve that customer base, while a large access ISP with 50,000 subscribers would require an access link in the range of an OC-3 (155 Mbps) to carry the expected peak load. These numbers are more conservative than what ISPs today would typically provide.13 . This is due in part to the fiercely competitive IAP business in which controlling costs is very important and in which there is little incentive to improve the quality of service offered.There are at least two reasons for this lack of incentive. First, under the current flat rate pricing model (e.g., $20 per month for unlimited usage), there is no way for IAP's to be compensated for offering a higher-level of service quality. Second, because of the fragmented nature of the Internet and the absence of adequate QoS mechanisms, IAPs cannot guarantee end-to-end performance. That is, a high speed access connection to the Internet will only allow the customer to download pages at the fastest rate available along the path from the server to the customer. In this environment, even if an IAP could offer improved performance, it may be difficult to convince customers this is true, or equivalently, there may exist a "Lemons Problem." In this situation, providers would recognize that high-quality offerings could not be distinguished by consumers from low quality offerings that would seek to masquerade as high quality, and hence, the market would fail to support high quality services [Akerlof, 1970]. Carriers and service providers are actively trying to address the lack of QoS mechanisms with new standards and mechanisms (e.g., RSVP, DiffServ, IntServ, caching, excess provisioning, etc.) and innovative service offerings such as service level agreements to end-users. All of the above is subject to the strong assumptions included in our sample example. For example, an ISP today might also provision for a lower actual average rate, compared to the 10 Kbps used here. If the ISP assumed an average of 5 Kbps, it would cut all these numbers in half. We need better data about actual traffic characteristics to more accurately address these issues. 4.3. Discussion of experimental results Each point on the graphs represents the average of 500 runs of an simulation for that level of aggregation and given value of a. Each run is 400,000 data points, representing 10,000 seconds of real time. To illustrate the range of possible error in these simulations, Figure 4 shows the maximum and minimum values seen for each level of aggregation across the 500 runs at that value. 5. Bursty Traffic and Industry Structure In the preceding two sections we discussed why bursty Internet traffic poses a problem for network provisioning that may be partially addressed by aggregating the traffic from multiple sources. Let us now turn to considering the implications of this for industry structure.
13
It is difficult to get real provisioning numbers, because this sort of information is often considered proprietary --------
Clark & Lehr, Provisioning for Bursty Internet Traffic
Page 11
MIT WISQ, December 1999
5.1. The role of the Internet Access Provider As the above data suggests, traffic aggregation is a critical part of Internet economics. If a link is used to carry traffic from only a small number of providers, it must be inefficiently used on the average if the peaks are to be accommodated. Once a certain number of users have been combined, link utilization goes up. One of the key functions of the access network is to perform that traffic aggregation. The IAP connects to individual users on the one hand, and this part of their network must be sized for peak rate. The IAP also connects to backbone ISPs, and that part of their network should ideally represent a high enough level of aggregation that the connection is well utilized, and hence, provisioning can be based on the average aggregate data rate. The costs and benefits of aggregation affect an IAP in two ways—the internal costs of bringing their individual customer’s traffic together inside the network, and the fee paid to the backbone to carry that traffic across the Internet. To minimize the internal costs, an IAP would prefer to aggregate subscribers within a small geographic area, so that the IAP does not have to have long internal paths that are under-utilized. If the market for IAP's is competitive (i.e., there are no important entry barriers), then IAPs will need to minimize costs in order to survive in the long run. The discussion above indicates that there are scale economies associated with aggregating traffic and provides a basis for estimating the minimum efficient scale of operation for an IAP. Focusing only on the aggregation efficiencies, IAPs that are smaller than this minimum efficient scale will have higher capacity costs on average, while IAPs that achieve this level of scale or larger will have approximately similar capacity costs on average. The analysis in Table 1 suggests that the number of customers required to achieve this minimum efficient scale would be between 1,000 and 10,000 active users, which we estimated might result from 5 times that many subscribers. If we take the lower bound of 5,000 subscribers as a target, and we guess that a successful broadband IAP could get 10% market penetration today, this implies that an IAP needs a localized market of 50,000 households to achieve the major benefits of scale (in this one dimension). So a large city can support a number of broadband providers at reasonable scale, but small and medium sized towns may experience either limited competition or somewhat less efficient (and thus more expensive) service. In the U.S., 90% of the counties have less than 61k households so this may be a substantial problem.14 Of course, bandwidth (both internal and in the access path to the backbone ISP) is only part of the cost structure of an access ISP and the analysis above is predicated on strong assumptions; however, it does suggest how aggregation economies may affect the market structure of IAPs. 5.2. Connecting to the backbone ISP and between backbone ISPs The discussion above implied that it would be more efficient if the IAP aggregated sufficient traffic that the link to the backbone ISP was efficiently utilized. The issue is actually broader than just the utilization of the link between the two ISPs. When one ISP connects to the other, that connection is normally subject to a Service Level Agreement (SLA) which represents the
14
There are 3,133 counties in the United States and the average number of households per county is 32k. Data is from the 1996 Population Census for the United States. --------
Clark & Lehr, Provisioning for Bursty Internet Traffic
Page 12
MIT WISQ, December 1999
commitment one ISP makes to the other to carry that other ISP’s traffic. In the vocabulary of this paper, this is the QoS commitment for the aggregated traffic. The backbone ISP charges the IAP a fee that covers not only the costs associated with the link that connects the two, but the costs of transporting the IAP’s traffic across the rest of the Internet. So if the IAP purchases a link, but makes low average use of it, it might still end up paying a fee that corresponds to the maximum traffic the link can carry, because the backbone ISP does not know if the link will be heavily or lightly utilized. The solution to this problem, of course, would be to negotiate a SLA between the access and backbone ISP that itself is bursty in nature. However, there are few models of what such an SLA might look like today, and few recognized methods to police the traffic going across the link to see if it meets the usage profile. A much simpler business relationship arises if the IAP has enough customers that the resulting traffic is reasonably smooth, and the SLA between the IAP and the backbone ISP can just be a continuous transmission rate during the busy hours. That is, as we explained earlier, if the peak-to-average traffic ratio is close to one, then it will be easier to write, monitor, and enforce SLAs between independent backbone providers. If these contracts cannot be written cost-effecitively, then the backbone providers have an incentive to vertically integrate forward with their IAPs to permit end-to-end capacity provisioning. Moreover, when backbone providers interconnect it is more likely that they would choose to do so at a relatively few number of points where traffic is highly aggregated. In addition to facilitating average rate contracts, this reduces routing costs, consolidates the uncertainty associated with exogenous traffic sources, and also reduces monitoring and security costs (e.g., traffic only needs to be monitored at a few places). 5.3. Why is broadband access different? Most IAPs today offer dialup service, not broadband. The aggregation and QoS commitment issues are very different for dialup customers, because the bandwidth limitations of the modem limit the peak rates of the users to such a low level that the resulting traffic is much less bursty. For example, if the average rate of a customer cruising the Web is 10 Kbps, a 33.3 Kbps modem limits the peak rate to about 3.3 times the average, as opposed to the hundred to one ratio in our earlier example. For a peak to mean ratio of 3.3, the per flow allocation drops to 1.25 times the average flow at around 700 active clients, even with no buffering (compared to almost 10,000 clients without buffering in the broadband hundred to one case). So a much smaller pool of customers, measured in the hundreds rather than the thousands, is sufficient to achieve a reasonably smooth aggregate flow. Small dialup access providers with modem pools suffer fewer problems with economy of scale in their trunk provisioning than an equivalent provider using broadband access links. This suggests that as broadband access replaces dial-up access and the minimum efficient scale of operation increases that there ought to be consolidation in the number of access providers. Also, if this scale of operation is difficult to achieve because of the size of the market (i.e., outside of major metropolitan or otherwise densely populated areas) that there will be less competition in broadband services and incentives to vertically integrate may be stronger. Offsetting this potentially pessimistic scenario is the prospect that even at the cost minimizing level of aggregation the heavy-tailedness of traffic at the edges will result in excess capacity in the edge networks. This in turn may encourage more aggressive price competition --------
Clark & Lehr, Provisioning for Bursty Internet Traffic
Page 13
MIT WISQ, December 1999
even if the market is relatively concentrated (i.e., there are a small number of IAPs in most markets). 5.4. Other traffic sources—why model Web traffic? In this paper, we concentrate on one class of traffic: browsing behavior on the World Wide Web. When this traffic is integrated with other types of services such as interactive or streaming video and audio, it is unclear whether the aggregate of this traffic will be more or less bursty than Web traffic. However, it seems likely that future applications are likely to require higher average transfer rates. For example, our guess for an average rate while using the Web was 10 Kbps. Current lowfidelity audio streams require capacities two or three times this. Near CD quality audio streams require about 100 Kbps capacity, 10 times faster than cruising the Web. Medium fidelity television can require 300-500 Kbps, and entertainment quality TV requires several megabits per second. If these higher data-rate sources become popular, they will dominate the traffic, and their characteristics will define the overall traffic mix. We suspect that these other sources are likely to be better behaved than the very bursty web traffic modeled here. So while this prediction about the future must be seen as very speculative, the future may be characterized by a greatly increased demand for absolute bandwidth, but fewer problems deriving from extreme burstiness. 6. Conclusions In principle, the Internet can support interactive, multimedia traffic and the long-heralded convergence of communications infrastructure. To date, realization of this promise has been delayed because of a lack of adequate capacity to support real-time services. This is due to a number of problems, including the lack of appropriate quality of service mechanisms (i.e., ability to support bounded delay guarantees), the need to upgrade customer premise equipment (i.e., multimedia-capable PCs), and need to expand the capacity of local access infrastructure (i.e., broadband, always on local access replaces dial-up access). As these roadblocks disappear, the Internet infrastructure will need to be able to support a much larger load of heterogeneous traffic. In recognition of this need, a number of carriers have been investing heavily in substantial upgrades to the transport capacity of the backbone and access networks. Some analysts have suggested that over provisioning (a single grade of service with enough capacity to assure minimal delays for the most demanding services) is the optimal approach to addressing the quality of service issue, while others believe that capacity will remain scarce (and hence worth economizing on by means of potentially complex QoS mechanisms). This debate boils down to a discussion of optimal investment policies for network infrastructure. The resolution of this debate depends in part on the relative costs of expanding capacity versus implementing mechanisms to utilize capacity more efficiently. Attempts to utilize capacity more efficiently will require usage metering, which if implemented will facilitate usage-based service contracts. Therefore, how one chooses to solve the QoS problem will have important implications for the form of service contracts that will evolve, the physical architecture of the Internet, and Internet industry structure. The attractiveness of over-provisioning as a --------
Clark & Lehr, Provisioning for Bursty Internet Traffic
Page 14
MIT WISQ, December 1999
solution decreases as the quantity of excess capacity required to provide adequate QoS increases (i.e., as the peak-to-average ratio of traffic increases). A second, related-question concerns the optimal utilization level for the network. It is a truism that Internet traffic is more bursty (i.e., peak to average data rate is high) than voice telephony traffic. This suggests that the optimal utilization rate for an Internet is likely to be lower than for a telephone network because of the need to provision to handle peak loads (or else suffer quality of service degradation during peaks). These two questions are related because the lower the optimal utilization level, the greater the investment in over-provisioning required to sustain any given level of quality of service. The greater the cost of using over-provisioning to address the QoS problem, the more attractive are mechanisms that facilitate efficient allocation of scarce capacity. A standard mechanism for addressing the problem of bursty traffic is to aggregate sources in the hope of smoothing peaks (i.e., peaks are uncorrelated). The extent of aggregation required to smooth peaks in Internet traffic may have important implications for how Internet Service Providers (ISPs) interconnect and the sustainability of alternative industry structures (e.g., scale and scope economies and implications for minimum efficient scale of operation). A number of studies have suggested that Internet traffic exhibits an extreme form of burstiness that statistically characterized as self-similar. If true, this may have important implications for the level of efficient traffic aggregation within the Internet and the sorts of interconnection contracts that are used at different locations within the Internet. This paper explores the economic and industry structure implications of provisioning for bursty Internet traffic. We present results from a preliminary study of the aggregation of multiple sources of self-similiar Internet traffic that indicates that relatively moderate levels of aggregation (100 to 1,000 sources) substantially reduce the need to provision for excess capacity to handle peak traffic loads. We consider what this means in terms of the geographic dispersal of ISP competition in light of changing demographics and network architectures. For example, the decline in transport prices in recent years (because of competition and advances in technology such as DWDM) has reduced the premium for over-provisioning and thereby reduced the dollar value of scale economies achievable from traffic aggregation. On the other hand, the potential need to provision to support broadband access traffic means that even relatively small scale economies may be quite valuable to capture. While the overall conclusion we reach is that only moderate levels of aggregation are likely to be needed to achieve minimum scale of operation, even these levels may not be achievable except in relatively densely populated urban areas. If true, prospects for robust competition in these less densely populated areas are threatened. Moreover, the aggregation issue will become more important as broadband access proliferates and the potential peak to average traffic ratio is released from the artificial constraints enforced by the limited capacity of dial-up access connections. The addition of new services such as streaming audio and video may alleviate the problem by reducing the burstiness of the aggregate traffic. The analysis presented here is provocative, but should be regarded as only speculative because it relies on the results of a simple model, extended with some quite strong assumptions. To improve this analysis, it would be desirable to consider alternative models of traffic and to simulate a variety of interconnection topologies. Cost modeling would be useful in helping to --------
Clark & Lehr, Provisioning for Bursty Internet Traffic
Page 15
MIT WISQ, December 1999
assess the relative importance of the aggregation efficiencies we discuss. Finally, additional information on current contracting trends would provide empirical validation for the points we discuss.
--------
Clark & Lehr, Provisioning for Bursty Internet Traffic
Page 16
MIT WISQ, December 1999
References Akerlof, George (1970), "The Market for Lemons: Qualitative Uncertainty and the Market Mechanism," Quarterly Journal of Economics, 84 (1970) 488-500. David Clark (1998), "Internet Cost Allocation and Pricing," In Internet Economics, Lee McKnight and Joesph Bailey, eds., Cambridge, MA: MIT Press, 1998. Bailey, Joseph P. and Lee W. McKnight (1997), "Scalable Internet Interconnection Agreements and Internet Integrated Services," In Coordinating the Internet, Brian Kahin and James Keller, eds., Cambridge, Mass: MIT Press, 1997. Paul Barford and Mark Crovella (1998) "Generating Representative Web Workloads for Network and Server Performance Evaluation," SIGMETRICS '98. Kelly, Frank, "Charging and Accounting for Bursty Connections (1998)," In Internet Economics, Lee McKnight and Joesph Bailey, eds., Cambridge, MA: MIT Press, 1998. Lehr, William (1998), "Understanding Vertical Integration in the Internet," mimeo, MIT Internet and Telecoms Convergence Consortium, February 1998 (available from http://itel.mit.edu). Lehr, William and Lee McKnight (1998), "Next Generation Bandwidth Markets," Communications & Strategies, Number 32, 4th Quarter 1998, 91-106. Lehr, William and Martin Weiss (1996), "The Political Economy of Congestion Charges and Settlements in Packet Networks,” Telecommunications Policy, 20(3), April 1996, pages 219-31. Liu, Ian, "Bandwidth Provisioning for an IP Network using User Profiles," Master of Science in Technology and Policy and Master of Science in Electrical Engineering and Computer Science thesis, Massachusetts Institute of Technology, June 1999. Mackie-Mason, Jeffrey and Hal Varian (1995), "Pricing Congestible Network Resources," IEEE Journal on Selected Areas in Communications, 13 (7) 1141-1149. --------
Clark & Lehr, Provisioning for Bursty Internet Traffic
Page 17
MIT WISQ, December 1999
Odlyzko, Andrew (1999), "Data networks are mostly empty and for good reason," IT Professional 1 (no. 2) (March/April 1999), pp. 67-69.
--------
Clark & Lehr, Provisioning for Bursty Internet Traffic
Page 18
MIT WISQ, December 1999
