iREX: Inter-Domain Resource Exchange Architecture - IEEE Xplore

iREX: Inter-domain Resource Exchange Architecture Ariffin Datuk Yahaya

Tatsuya Suda

School of Information and Computer Science University of California, Irvine [email protected]

School of Information and Computer Science University of California, Irvine [email protected]

Abstract – This paper introduces the inter-domain Resource Exchange (iREX) architecture for the automated deployment of fault tolerant end to end (E2E) inter-domain (ID) quality of service (QoS) policy among resource user and resource provider Internet Service Providers (ISPs). iREX uses economics and fully distributed mechanisms to empower domains to self-manage the deployment of E2E ID QoS policy. In iREX, resource user domains select and reserve ID QoS resources to form E2E ID QoS policy while resource provider domains provide information about available ID QoS resources and support the deployment of policies. iREX promotes accountability by enabling the establishment of bilateral agreements between a resource user ISP and all resource provider ISPs, and promotes congestionavoidance by enabling a distributed resource selection process that selects the least congested ID deployment path. With iREX, domains cooperate to deploy ID QoS policy while maintaining their autonomy at all times. Simulation results show that iREX blocks fewer reservations and causes less congestion than the existing method, and that iREX is able to recover from faults.

The complexity of negotiating for an E2E ID QoS policy increases when the policy deployment traverses multiple intermediate domains on the way to the destination domain because trust must propagate domain hop by domain hop from the source domain, through all the intermediate domains involved in the QoS policy, to the destination domain.

Keywords – inter-domain; QoS policy; resource allocation and management; network control by pricing; economics.*

1. INTRODUCTION

M

a domain’s policy to offer Quality of Service (QoS) to selected network traffic is an important networking research area. Presently, any number of domain Internet Service Providers (ISPs) can create and install policy to selectively support multiple traffic flows with specific QoS requirements within the domain(s) that they control; but because no single ISP controls all the domains in the Internet, deploying end-to-end (E2E) inter-domain (ID) QoS traffic flows must involve negotiating for, and propagating the policies to support the traffic flows’ QoS specification (QoS policy) across ID borders. ANAGING

Since each domain’s network resource is owned and managed by different ISPs, negotiating for a QoS policy across ID borders is complicated by the inter-related nontechnological issues of “ownership” and “trust”. Resource provider domains need to trust resource user domains to compensate them for the resource contributed towards supporting an E2E ID QoS policy, and resource user domains requiring deployment of ID QoS policy need to trust resource provider domains to perform as compensated. This research is supported by the NSF through grants ANI-0083074, ANI9903427 and ANI-0508506, by DARPA through grant MDA972-99-1-0007, by AFOSR through grant MURI F49620-00-1-0330, and by grants from the California MICRO and CoRe programs, Hitachi, Hitachi America, Hitachi CRL, Hitachi SDL, DENSO IT Laboratory, NICT (National Institute of Communication Technology, Japan), Nippon Telegraph and Telephone (NTT), NTT Docomo, NS Solutions Corporation, Fujitsu and Novell.

1.1 CURRENT SOLUTION The current method to deploy ID QoS policy is through the negotiation of a Service Level Agreement (SLA) [1][2]. To use SLAs to deploy ID QoS policy, a business person from the source user domain initiates manual negotiation with his/her counterpart in a neighboring provider domain. During this phase of the negotiation, if the destination domain is not directly connected to the provider domain, the provider domain will act as a proxy and “pretend” to own all network resources necessary to fulfill the source domain’s requirement. In this case, the provider domain will itself become a user domain and proxy for the previous hop user domain to negotiate for the necessary resources with its neighboring provider domain along the ID path to the destination domain. When an SLA has been negotiated between user and provider domains, a technically oriented document called the Service Level Specification (SLS) [3] will be created which will be sent to the network administrator of the provider domain and the QoS policy will be installed by the provider domain’s network administrator. Due to its manual nature, the process of establishing an SLA takes time in the order of days. There are three problems associated with using SLAs: SLAs do not self-manage because they are inherently manual in nature. Slow manual deployment decreases user demand for QoS transport services in the case where users are not capable of forecasting so far into their future needs. User demand for QoS transport services is important to ISPs as it translates directly to revenue. SLAs have limited accountability because of their use of proxies at ISP ID borders. Domains that create SLAs using proxies promise network resources which are not under their direct control. The use of proxies weakens accountability and limits an intermediate domain’s access to the source domain and vice versa – an example of a problem this causes is when an ID resource involved in a QoS policy fails and a new ID path to the destination converges through the normal process of the Border Gateway Protocol (BGP) [4]; in this case, there is no guarantee that ISPs along the newly converged route (that were not along the initial route) will honor the QoS policy negotiated for in the SLA.

1-4244-0222-0/06/$20.00 (c)2006 IEEE This full text paper was peer reviewed at the direction of IEEE Communications Society subject matter experts for publication in the Proceedings IEEE Infocom.

SLAs cannot avoid congestion because their slow manual deployment results in the establishment of essentially static ID routes through the network. Attempting to reroute the ID path to take advantage of current network state information is futile when the reroute cannot be done in a timely manner.

1.2 AUTOMATED POLICIES To make Internet QoS more relevant and useful, the deployment of E2E ID QoS policies must be made faster with domains assigned proper accountability. Specifically, the manual nature of deploying and maintaining E2E ID QoS policies must be revised with automation without the use of proxies. We posit that there are three conceptual requirements that an E2E ID QoS policy automation architecture needs to fulfill: self-management, accountability, and congestionavoidance. Self-management is the ability for the architecture to empower domains that need ID QoS to proactively deploy E2E ID QoS policies across ID borders without using any intermediary, proxy or central entity – while maintaining the autonomy of all domains. Additionally, all mechanisms for self-management should prevail even though some parts of the network should fail. Accountability is the ability for the architecture to properly and transparently assign responsibility, both for providing the resources to support a policy, and for the payment (or stipulation of debt) for such resource usage. Congestion-avoidance is the ability for the architecture to avoid using congested resources and to not add to global network congestion.

1.3 INTRODUCING iREX To fulfill the need for E2E ID QoS policy automation, this paper introduces the inter-domain Resource Exchange (iREX) architecture. iREX uses economics and fully distributed mechanisms to empower domains to self-manage the deployment of fault tolerant E2E ID QoS policy. iREX promotes accountability by enabling the establishment of direct bilateral agreements between a resource user and all resource providers, and promotes congestion-avoidance by enabling a distributed resource selection process that selects the least congested ID deployment path. With iREX, domains cooperate to deploy ID QoS while maintaining their autonomy at all times.

2. iREX OVERVIEW iREX facilitates a domain to domain market for network resources to deploy ID jumbo flows [5], which are aggregated individual flows between two domains. Domains supporting iREX form an “iREX market” by registering offline with a clearinghouse* that acts as a bank to handle settlements for resources bought and sold within the iREX market. iREX markets exist for the sole purpose of trading in ID network resources and members of the iREX market work together to *

1) facilitate ID resource selection by maintaining information about the desirability of resources within the iREX market, and 2) support the deployment of E2E ID QoS policy. In iREX, domains with available ID QoS network resources advertise them for “sale” to their neighboring domains, while domains receiving these advertisements evaluate, select and disseminate advertisements for the most desirable resources to its neighboring domains – in this way, information about the most desirable of globally available ID QoS network resources propagate to all domains within the iREX market. Source domains needing E2E QoS policy deployment choose from advertised ID QoS resources and initiate negotiation for the deployment directly with the providers of the chosen resources. Domains agreeing to support ID QoS policy deployed with iREX actively maintain the deployed policy and participate in its recovery from faults. In the following sub-sections we explain iREX’s concept of resources and how they are used to deploy policy.

2.1 iREX RESOURCES An iREX resource is defined as an abstract network transport service defining ownership and transport responsibility starting from a domain’s ingress border router, going through the domain and ending at a neighboring domain’s ingress border router. When a provider domain sells ID QoS transport services on an iREX resource, that provider domain guarantees that the traffic associated with the sold ID QoS transport service arriving at its ingress border router will be passed on to the neighboring domain ingress border router within some pre-defined QoS specifications. The exact routing within the domain is undefined – left to the provider’s discretion. iREX uses real resource price in the form of “monetary unit per time unit per bundle of resource” to evaluate how valuable an iREX resource is to an iREX market. The real resource price of each ID resource is determined by the provider domain that owns the ID resource according to the domain’s perception of 1) current market prices, and 2) the risk associated with accepting future deployments with regards to impacting currently active deployments. Market prices are dependant on a resource’s location in reference to similar resources (supply) and the distribution of QoS resource users (demand) within the network topology. Risk is dependant on the provider domain’s previous commitments on its network resources. Section 7.2 includes further discussion on the economics that affect real resource price. iREX uses reputation to evaluate an iREX market’s perception of a resource’s ability to conform to an iREX market’s view of good service. Each iREX market defines its own criteria for good service depending on the purpose of the deployments. Reputation is represented by a numeric score which tallies the number of complaints against a resource. The effect of each complaint expires with time. Individual domains decide their own tolerance for particular reputation scores.

Details of this clearinghouse will not be covered in this paper.

This full text paper was peer reviewed at the direction of IEEE Communications Society subject matter experts for publication in the Proceedings IEEE Infocom.

Section 4.3 includes further details on the iREX reputation system. An iREX resource’s desirability is based on real resource price and reputation. An iREX resource is desirable to an iREX market if it is currently the cheapest available reputable resource that is capable of fulfilling the required QoS specification. An iREX resource is the basis for resource trading within iREX. A chain of iREX resources along an ID path can be used to transport ID jumbo flows to uphold deployed E2E ID QoS policy. 2.1.1 iREX PATH VECTORS iREX domains use real resource price and reputation information to evaluate resource when forming chains of ID resources leading to destination domains in the form of path vectors (iREX path vectors). iREX path vectors are similar to BGP, but with two differences. The first difference is that instead of using minimum ID hops as an evaluation metric, iREX path vectors use the current total real resource price (total price) per unit time. The total price per unit time for an ID QoS policy to the destination is determined by adding up all the component real resource prices of the ID resources used in the iREX path vector. The second difference is that since iREX uses source routing, the iREX path vectors are used only as hints and total network convergence for an iREX path vector is not necessary. iREX domains advertise known iREX path vectors to their neighbors. Each domain evaluates and filters received iREX path vectors by first excluding those that use resource with bad reputation scores, and then choosing the path with the cheapest total real resource price to each destination domain. By periodically advertising known real resource prices to neighbors and filtering received advertisements, information about the cheapest reputable ID QoS network resources propagates to all domains within the iREX market. iREX assumes that QoS resource specifications are commoditized (standardized). Each QoS specification as a standardized commodity will have its own set of iREX path vectors. Multiple “markets” of ISPs may exist, each specializing in one or more specific QoS traffic commodity and maintaining the current most desirable ID paths for a particular QoS traffic commodity with a unique set of iREX path vectors. Each domain within an iREX market stores iREX path vectors to facilitate the selection of ID QoS paths.

2.2 iREX POLICY DEPLOYMENT To sell transport services on an iREX resource, a provider domain sets for each iREX resource he owns 1) a minimum price that the provider domain is willing to accept from other domains to reserve the resource, and 2) the maximum duration of time that the provider domain is willing to commit for a sale of services on that particular resource. Every price set by the provider domain is also accompanied by a price version code so that other domains may check the freshness of an advertised price. To buy transport services for deploying an ID QoS policy using iREX, a source domain first identifies the entire ID path

by referring to the current iREX path vector to the destination domain. If the total real resource price per unit time is acceptable to the source domain, it initiates ID QoS policy deployment with the domains along the identified path through reservation request signaling – initiating this reservation request implies a commitment to pay the price that each resource provider has requested for the use of their advertised resource. Responding positively to a reservation request signal implies an agreement by the provider of a resource to 1) deploy network support and 2) actively participate to maintain the policy referred to by the reservation request for the full duration of time that the resources have been reserved.

3. iREX DOMAIN SCENARIO

Figure 1. iREX Domain Scenario iREX’s QoS world view is that each domain has its own retail market dealing with its domain users and a wholesale market which it uses to connect to the Internet through n1 domain neighbors which in turn are also connected to n2 domain neighbors and so on. Figure 1 illustrates this world view and how iREX fits into the classical Policy Based Network Management (PBNM) admission control architecture presented in [6]. Intra-domain QoS policy deployment is done by the PBNM Policy Server (PS) according to the respective domain’s management practices without iREX. If the PS requires ID policies to be deployed, it will communicate the requirements to the iREX ID Policy Server (iIDPS). The iIDPS then uses the iREX protocol to deploy the ID QoS policy. If there are ID QoS policies from another domain that require intra-domain routing either to transit or terminate within the domain the iIDPS will in turn communicate this to the PS. iIDPS in neighboring domains form connections to each other, but any iIDPS may send messages to any other iIDPS. An example of iIDPS to iIDPS communication across several ID borders is the “iREX short circuit process” which is covered in section 4.1. iREX uses TCP [7] for signaling, so inter-domain messages between iIDPSs rely on BGP routing.

This full text paper was peer reviewed at the direction of IEEE Communications Society subject matter experts for publication in the Proceedings IEEE Infocom.

4. iREX ARCHITECTURAL DETAILS In this section we will first explain the iREX short circuit process which is part of how we approach signaling within iREX. We will then explain how iREX does resource reservation, resource reputation and fault tolerance.

traffic specifications. This also allows iREX to use multiple paths between source and destination domain pairs. iREX deployments for the same source destination domain pair may use different paths depending on the current network state.

4.1 iREX SHORT CIRCUIT PROCESS The iREX short circuit process allows intermediate domains to communicate directly with the source domain even where they are several domain hops away from the source domain. This process is useful because network state information local to these intermediate domains is more fresh and reliable coming directly from such intermediate domains. In iREX, the short circuit process is used to update a source domain’s information, which could be as mundane as information about a more current resource price, or as critical as information about a resource being faulty.

4.2 iREX RESOURCE RESERVATION iREX resource component prices quoted and advertised as part of an iREX path vector fluctuate continuously according to network state, and an iREX reservation initiated by a source domain freezes the component prices for the duration of the reservation being deployed. Subsequent reservations for the same source destination domain pair may have different paths and costs depending on the state of the network at that time. To deploy ID QoS policy to a specific destination domain, the source domain will refer to the current iREX path vector to that destination, or if the path does not exist in its set of stored iREX path vectors, the domain can send Path Request messages to its domain neighbors to get a path. Each domain tries to keep the most recent price information within its stored iREX path vectors, but even if a reservation is initiated using an iREX path vector with stale information, the iREX protocol has a feature that allows a resource provider domain to update the source domain with new information during the reservation. When a path with an acceptable price is found, the source domain does reservation signaling similar to the RSVP [8] protocol on an ID level with these conceptual differences: i. In iREX, the source domain is in full control of the whole reservation process. ii. In iREX, intermediate domains can use a “short circuit” process to suggest alternate routes. iii. In iREX, reservations perpetuate based on time, and expire by themselves without the need for tear down. Expiring reservations can be extended at the most current total real resource price for the path. iv. In iREX, domains actively maintain the fault tolerance of deployed reservations within their domain. Within an iREX market, ID QoS policy is uniquely identified by the combination of the source domain’s Autonomous System (AS) number and a source domain generated unique deployment code. Policy so identified will be associated with a source domain selected ID path, and QoS

Figure 2. iREX Reservation Information Passing Figure 2 illustrates how a source domain sends information about a resource reservation in iREX. The source domain supplies a unique deployment code and sufficient ID path information to the next hop domain so that the reservation may be relayed along the source domain chosen path. Each domain checks its own ID resource information for correctness, and if satisfied, deletes it from the reservation message and relays the resultant reservation message to the next hop domain – this continues until the destination domain is reached and an acknowledgment is sent in the reverse direction.

Figure 3. iREX Resource Reservation Signaling Figure 3 illustrates an example of iREX resource reservation scenario where a domain along the path has newer information than the source. Source domain S is trying to set up ID QoS policy to destination domain D on the ID path SIJKD and messages 1, 2, and 3 are Reserve messages. When the Reserve message arrives at domain K, the reservation is supposed to continue with Reserve message 4a to D, but domain K decides to suggest another route through domain L to the source domain by sending Route Update message 4b instead. In this case, the source domain decides to accept the suggestion by K and sends a Reserve Update message in 5, 6, and 7, to inform domains I and J of the change in routing and at domain K, the reservation is continued by Reserve messages 8 and 9. The ID QoS policy is successfully deployed with Reserve Ack messages 10, 11, 12, 13 and 14, and the resultant ID QoS path is SIJKLD.

This full text paper was peer reviewed at the direction of IEEE Communications Society subject matter experts for publication in the Proceedings IEEE Infocom.

4.3 iREX RESOURCE REPUTATION SYSTEM Each iREX market maintains a distributed and redundant database where member domains that find themselves affected by “bad” resource can register complaints. Duplicate complaints where a domain registers complaints repeatedly about the same resource is counted as one complaint. Accumulated complaints about an iREX resource past a threshold count will cause the iREX resource to be excluded from the iREX path vector, and thus preclude the selling of any transport services on that resource. The reputation system is intended only for the selfish selfpreservation of an iREX market and registering complaints are done without malice. Resources may be non-conforming, fall into ill repute, and then be excluded for a variety of reasons including simple system failure or while experiencing a denial of service (DOS) attack. The effects of complaints expire with time.

Figure 4. iREX Reputation Signaling Figure 4 illustrates an example iREX reputation scenario where the current iREX path vector at source domain S indicates that the best route to the destination domain D is using the path SIJKLD. Unfortunately, domain L is experiencing an undefined system failure and can no longer entertain reservations, but domain K is still advertising the resource KL and LD as being desirable. As a result, source domain S is limited to the path SIJKLD but cannot get transport service to domain D because its reservation along the path vector in Reserve messages 1, 2, 3 and 4 are not being processed by domain L. In this case, source domain S first sends Reserve messages as probes like a “traceroute” [9] to each domain along the path and after noting which domain does not respond, finds that domain L is the culprit. The source domain then sends Register Complaint message 6 to the distributed database to register a complaint against the resource LD, and also sends Check Resource message 7 to domain K. Upon receiving a Check Resource message, domain K will send Complaint Query message 8 to the distributed database to get LD’s reputation score, and if enough domains have registered complaints about the resource, domain K will exclude the resource from further iREX path vectors until such time that the resource’s reputation score improves.

4.4 iREX FAULT TOLERANCE iREX domains monitor active deployments and try to recover from any faults either by suggesting to reroute a failed deployment through other available paths or by signaling

a fault towards the direction of the source domain so that other intermediate domains that are involved in the deployment may attempt a recovery. In this way, at a minimum, the source domain will eventually know of a failure and redeploy the requirement.

Figure 5. iREX Fault Recovery Signaling Figure 5 illustrates an example of iREX fault recovery signaling where a resource that is being used by current deployments fails. Source domain S has deployed ID QoS policy to destination domain D on the ID path SIJKLD, but the resource between domains K and L has gone down. Since domain K is closest to the failure, K senses the failure first and initiates recovery by looking for an available path from K to D for the deployment. K finds an alternate path using resource KD and sends the information to the source domain in Route Change message 1. In this example, the source domain agrees with the change, sets a timer for the recovery of the deployment and sends Reserve Change messages 2, 3 and 4 that inform domains I and J of the change and tells K that the source agrees. At K, the Reserve Change message is converted to Reserve message 5 to domain D. When destination domain D replies with Reserve Ack message 6 to domain K, domain K sends Reserve Ack message 7 directly to the source domain to inform it of the successful fault recovery and the ID QoS policy deployment is now changed to path SIJKD. If the source domain does not receive a Reserve Ack message before the timer for the deployment’s recovery expires, then the source domain will cancel the deployment and redeploy using a new reservation on the current iREX path vector to the destination.

5. iREX PROTOCOL The iREX protocol was designed to uphold each domain’s autonomy while providing full deployment control to the initiating source domain; other domains provide information so that the source domain may make informed decisions. This section of the paper will provide an overview of the protocol, more details are available at the iREX website [10]. The iREX protocol set is divided into 5 message groups: The routing group contains messages to advertise and withdraw iREX path vector information, to update a domain that is using stale information for a reservation, to suggest a change in an already deployed reservation, and to request current iREX path vectors from domain neighbors. The reservation group contains messages to deploy a reservation, update information about a reservation being deployed, change information about a deployed reservation,

This full text paper was peer reviewed at the direction of IEEE Communications Society subject matter experts for publication in the Proceedings IEEE Infocom.

cancel a reservation, and also to extend a reservation deployment’s duration. The metering group contains messages that allow domains to monitor the status of reservations that have been deployed. The fault recovery group contains messages that signal faults so that recovery can begin. The reputation group contains messages to complain about an ID resource, inquire about an ID resource’s reputation and to send information about an ID resource’s reputation.

6. iREX ARCHITECTURE OVERHEAD iREX incurs control packet overhead in advertising and signaling. iREX does advertising to propagate ID network resource information to maintain the iREX path vectors, and signaling to communicate between domains. If the information contained in the iREX path vector is stale, domains will signal corrective information back to the source in the form of Route Update messages, so there is a tradeoff between advertisement frequency and corrective signaling overhead.

Domains participate in the maintenance of iREX path vectors through selfish self-interest to ensure that they remain in the path vector, and therefore in the iREX market. Domains support the deployment of E2E ID QoS to earn revenue, and ensure good service to maintain untarnished reputations so that they are not blocked from supplying the iREX market.

7.2 REAL RESOURCE PRICE iREX path vectors are predicated on economics where network resources are treated as a scarce good. Domains are assumed to always act selfishly to balance two priorities, 1) the domain’s responsibility towards its current customers, and 2) the domain’s desire to compete and make money selling ID QoS resources. Additionally, to compete in selling ID QoS resources, each domain also needs to take current market prices into account. When a domain advertises new prices, the distributed iREX path vector selection process will evaluate the new price, and the resource may either be more or less desirable when compared to other competing ID resources advertised by other domains.

iREX incurs additional resource use overhead when routing deployments through multiple paths other than the shortest path for the same source destination domain pair. However, iREX chooses these paths according to the network state, so even when we use more total resource compared to the shortest path, the selected resource are the ones least congested, and by doing so, iREX avoids congested resources. Finally, implementing iREX comes with a database overhead for storage of iREX path vector, reputation, and deployment state information, and may cause a configuration headache for the network administrator in the automated deployment of intra-domain policies to support the ID policies. These are issues that need further investigation through a more detailed design. At this point, we have yet to finalize the database schema or the intra-domain architecture.

7. iREX DESIGN RATIONALE iREX facilitates a domain to domain market for trading in ID QoS network transport resources based on the “Posted Price Competition” economic model where sellers independently choose prices that are publicly communicated to buyers on a take-it-or-leave-it basis [11]. The iREX architecture implements this economic model with the following distributed mechanisms: 1) an information evaluation and dissemination mechanism in the path vector maintenance process, 2) a fault tolerant accountability mechanism in the reservation signaling and reservation maintenance processes, and 3) an ID resource allocation mechanism in the ID resource economic trading processes. These mechanisms are designed to mimic similar mechanisms that permeate everyday human economics. This section explains how and why these mechanisms work in iREX.

7.1 DOMAIN ETIQUETTE In iREX, a domain’s profitability and long-term existence is determined by its performance within an economic market.

Figure 6. iREX Supply Side Economics Each domain decides on a selling price to advertise for the ID QoS resources it wants to sell based on 1) the amount of available resource, and 2) how its current customers are projected to use the available resource. Figure 6 illustrates the economic relationship between the percentage of bandwidth allocated to ID QoS resource trading and the associated risks. The Min Risk/Minimum Allocation line occurs at a point where there is no risk of impacting current customers. When the demand curve moves up the supply curve and the price that the iREX market is willing to pay increases, a domain will allocate bandwidth between the Min Risk and Max Risk point on the x axis. Lower demand for an ID QoS resource will result in an abundance of the resource, which is accompanied by a lowering of the risk of impacting the domain’s customers and a lowering of revenue, therefore the domain will seek to increase the demand of its ID resources by lowering the price of the resource. Higher demand for an ID QoS resource will result in resource scarcity, which is accompanied by the risk that a domain’s current customers will be impacted, so the domain will seek to be compensated for the increased risk by increasing the price of the advertised QoS resources.

This full text paper was peer reviewed at the direction of IEEE Communications Society subject matter experts for publication in the Proceedings IEEE Infocom.

Lower prices accompany lower resource use, and higher prices accompany higher resource use, therefore choosing resources that are cheaper translates directly to choosing resources that are less congested. In this manner, economics is used to dynamically change the iREX path vectors to always choose the cheapest most reputable resources, which also translate into the least congested resources.

the simulator (such as starting price for the flat part, slope for the linear parts, multiplier for the increasing parts, etc.); in this way, every domain within the simulation quotes resource prices differently in response to bandwidth sold within their domains.

7.3 REPUTATION The iREX reputation system holds all resources to a service standard defined by the iREX market. Where the economic system described in the previous section may fail due to a lack of consideration for the quality of a good (i.e. a non performing resource) or for unscrupulous acts (i.e. trying to bias the architecture for unfair advantage), the iREX reputation system fills in. Any resource that has many domains complaining against it will be excluded from the iREX market.

8. SIMULATION DESIGN In order to evaluate the iREX architecture we developed a simulator in 25,000+ lines of Java code with the source code available at [10]. We used this simulator to investigate the performance and capabilities of the iREX architecture, and compare its performance with the current (SLA) method. The iREX simulator implements the iREX protocol and a simplified BGP protocol with Announce and Withdraw. The simulator does packet level simulation for control packets used for iREX and BGP signaling, and flow level simulation for the deployment of QoS flows. Signaling resource advertisements for iREX path vector maintenance is done at 3 minute intervals. The simulator creates a user specified topology of ISP domains with each domain having two main goals: 1) to fulfill a set of generated ID QoS requirements for its local domain users, and 2) to sell available network resources to other domains. To test the iREX architecture’s self-management property, domains within the simulation are totally selfish and have no knowledge outside of their own domain other than those received through BGP and iREX signaling. For the SLA part of the simulation, we assume provisioning with all the ID QoS requirements having already been pre-negotiated along the BGP ID path. Domains therefore have full advance knowledge of all ID requirements originating from, traversing through and terminating in their domain for the entire simulation duration. For the iREX part of the simulation, the domain only has a maximum 1 minute advance knowledge of its own current ID requirements, and reservations need to be initiated ad-hoc with other domains using iREX signaling as previously outlined in this paper. To set the price of resources that a domain owns the simulator randomly assigns each domain one of three price functions – flat then linear, flat then cubed, or tiered with the x axis being the quantity of resources used and the y axis being the price assigned. These price functions further have parameters that are randomly assigned values by

Figure 7. “vBNS” Simulation Topology

Figure 8. “Level 3” Simulation Topology The topology chosen for the simulations are the Very High Performance Backbone Network Service (vBNS) topology and the Level 3 ISP (L3) topology with each point of presence representing an ISP domain. These topologies were chosen because we wanted to simulate iREX using real world topologies. ISP domains are assumed to be connected with OC48 optical fiber links to its neighbors and the length of each link is calculated to be the actual beeline distance between the cities. Figures 7 and 8 illustrate the two chosen topologies. ID reservation requirements within the simulator are viewed as “bundles” of traffic sized at 0.2% of line speed (about 4.8mb/sec) with a 5 minute average reservation duration. The traffic load (total projected bandwidth usage) is determined according to a percentage of each domain’s actual total egress capacity in the vBNS topology from 0% to 100% in 4% steps. To generate requests, we used a simple Poisson arrival model with parameters derived from M/M/∞ analysis. The exact same ID reservation requirements derived for the vBNS topology simulation was then also applied to the L3 topology simulation.

This full text paper was peer reviewed at the direction of IEEE Communications Society subject matter experts for publication in the Proceedings IEEE Infocom.

To show how iREX performs in different situations, in this paper we present three simulation scenarios, a normal operations scenario, a fault scenario and a domain nonconformance scenario.

9. PERFORMANCE EVALUATION 9.1 NORMAL OPERATIONS SCENARIO In the normal operations scenario, the first set of metrics compare deploying E2E ID QoS using iREX against the SLA methodology with exactly the same network and traffic. The metrics we used for the comparison set were blocking probability and congestion while varying the traffic load on the two topologies. Blocking probability is defined as the number of unsuccessful global ID deployments divided by the total number of ID deployments attempted globally. We use this metric to show that iREX blocks less than SLA. To make sure an ID policy installation is successful, for each ID deployment in both the iREX and the SLA simulation scenarios, the simulator checks that all the domains involved in the deployment 1) have an active policy installed to support the deployment, and 2) have actual “real” (non-multiplexed) capacity to deploy the flow.

Figure 10. Links Congestion per Deployment Figure 10 shows link congestion per deployment with the SLA topologies congesting more links as successful deployments increase, but because iREX distributes traffic among many paths, iREX also distributes the congestion. We observe that one interesting difference between provisioning (SLA) and signaling (iREX) for “real” resources is that in provisioning, if an upstream domain is congested and unsuccessful in installing a policy, downstream domains may still have the policy installed – this does not happen in signaling. Another observation from the graph about the iREX topologies is that the congestion curve decreases after a peak; this is because as prices get higher, paths that traverse longer paths through less congested domains are made possible in iREX. The normal operations scenario also includes metrics that measure pareto fairness and multipath probability while varying traffic load, unique paths while varying the number of requests, setup time while varying distance and control packet overhead while varying traffic load for the iREX simulation scenario.

Figure 9. Blocking Probability Figure 9 shows blocking probability with the SLA topologies starting to block at about 10% traffic load and rising sharply while the iREX topologies start to block at about 20% traffic load rising less sharply. At 50% traffic load the SLA topologies are dropping more than 60% while the iREX topologies are dropping less than 35%. iREX drops less requests across all traffic loads because it maximizes network efficiency by taking advantage of multiple routes for the same source destination pair. We define a link to be congested when more than 50% of its capacity is in use, and the comparative metric we use to measure congestion is defined as the total number of links that are congested globally divided by the total number of requests successfully deployed globally. We use the congestion metric to show that iREX causes less congestion per deployment when compared to SLA.

Figure 11. Pareto Fairness Pareto fairness is defined as the percentage of deployments that were successful in the SLA configuration that is also successful in the iREX configuration. We use this metric to show that iREX has minimal impact on deployments

This full text paper was peer reviewed at the direction of IEEE Communications Society subject matter experts for publication in the Proceedings IEEE Infocom.

that would have otherwise been admitted using the SLA method.

iREX’s behavior in seeking out alternate paths for the same source destination domain pair.

Figure 11 shows the iREX architecture’s pareto fairness in both topologies to never be below 80%. Pareto fairness is lower than 100% due to fluctuating path vectors.

Figure 13 shows unique paths between Los Angeles and Boston with the maximum number of paths peaking at 3 paths and 11 paths for vBNS and L3 respectively. The use of multiple paths increase as the number of calls increase, until a peak (global capacity) is reached. It is harder to find alternate paths after this point due to a race condition occurring as the Los Angeles to Boston deployment competes with other global deployments.

Multipath probability is defined as the number of policies that are deployed globally that, at the time of deployment, have more than one path connecting the same source destination domain pair divided by the total number of policies deployed globally. We use this metric to show when iREX employs multiple paths between source destination domain pairs.

Setup time is defined as the time starting when the domain first issues a request for a policy deployment up to the time that the policy deployment is finally successful. Distance is the total distance between the source to the destination domain going through all the intermediate domains according to the deployed path. We use the setup time metric to show time overhead when deploying ID QoS policy using iREX. Setup time for unsuccessful deployments is not tabulated.

Figure 12. Multipath Use Probability Figure 12 shows multipath use probability with both the vBNS and L3 topologies starting to use multiple paths between source destination pairs at about 8% traffic load, and multipath use increases sharply until a plateau is reached as the respective topology’s capacity limit is reached.

Figure 14. Setup Time Figure 14 shows setup time with both the topologies showing a linear trend with some data point exceptions – these exceptions are caused by intermediate domains having to update source domains using stale information because as network capacity is used up by increasing traffic load, prices change faster and information that is far from a domain gets stale faster. We do however observe that the worse case setup time is still under 510 milliseconds at a distance of about 6,000 kilometers.

Figure 13. Unique Paths Unique paths is defined as the number of paths that has at least one component link different from all other paths being used at the time of deployment. The unique paths metric is specific for deployments between Los Angeles and Boston, and measures the number of unique paths that iREX has deployed between these two cities. We use this metric to show

Control packet overhead is defined as the global total number of control packets issued globally for a specific purpose divided by the number of successful global deployments. We tabulate control packets for iREX advertisements, stale information updates, and also the total of all types of control packets. We use the control packet metric to show control packet overhead when deploying ID QoS policy using iREX. Figure 15 shows control packet overhead per deployment with the total control packets less than 10 per deployment for the L3 topology and less than 6 per deployment for the vBNS topology across all traffic loads. We also observe that the

This full text paper was peer reviewed at the direction of IEEE Communications Society subject matter experts for publication in the Proceedings IEEE Infocom.

advertising and advertising control packet overhead is very low at less than 0.2 packets per deployment for both topologies after an initial advertisement peak at time 0.

topology. We note that since we use the traffic generated for the vBNS topology on the L3 topology, there is more residual capacity on the L3 topology configuration to make more fault recoveries, hence the recovery probability never falls below 70%. Recovery time is defined as the time starting when the fault occurs up to the time when the failed deployment is finally restored. Distance is the total distance between the source domain to the domain with the faulted link while going through all the intermediate domains according to the deployed path. We use this metric to show how long it takes iREX to recover from faults.

Figure 15. Control Packet Overhead

9.2 FAULT SCENARIO In the fault scenario, we simulate two kinds of faults, a QoS fault where a link’s QoS goes down but the underlying BGP connectivity is still working, and a Total fault scenario where BGP routing has failed and has not yet converged. The fault that we simulated on both topologies is on the link between San Francisco and Los Angeles. For this scenario, we measure the recovery probability while varying traffic load, recovery time while varying the distance and the control packet overhead while varying the traffic load. Recovery probability is defined as the global total number of failed deployments successfully recovered divided by the global total number of failed deployments. We use this metric to show iREX’s performance when recovering both types of failed deployments.

Figure 17. Recovery Time Figure 17 shows fault recovery time with both the topologies showing a roughly linear trend. We observe definite separate data point clusters between Total and QoS failures because in Total failures, the iREX fault recovery system has to compensate for the possibility of not being able to signal due to BGP failure, so the fault recovery responsibility will propagate upstream towards the source domain. We also observe that the maximum recovery time is under 1,200 milliseconds at a distance of about 5,000 kilometers.

Figure 16. Recovery Probability Figure 16 shows recovery probability in both the Total and QoS scenarios. We observe that we can recover about 100% of the faults in both scenarios as long as we have network capacity – the vBNS topology has less capacity compared to the L3 topology so we do better on the L3

Figure 18. Fault Tolerance Control Packets Control packet overhead is defined as the global total number of fault recovery control packets issued divided by the

This full text paper was peer reviewed at the direction of IEEE Communications Society subject matter experts for publication in the Proceedings IEEE Infocom.

number of successfully recovered deployments globally. We use the control packet metric to show control packet overhead when iREX does fault recovery. Figure 18 shows control packet overhead per recovery with the vBNS topology showing more control packet use than the L3 topology in general – this is caused by the vBNS topology not having as many alternate routes per domain compared to the L3 topology. Domains in the L3 topology experiencing failure on one of its links is able to recover by itself with minimal signaling, hence it has less than 1 control packet per recovery. We observe that in the vBNS topology, the control packet count decreases sharply at about 35% traffic load and continues to decrease – this is because at the 35% point there is also a sharp decrease in the topology’s capacity to support fault recoveries, and fault recoveries continue to deteriorate as observed by figure 16.

factors that make the effectiveness fluctuate. We would like to point out that although we used the same traffic pattern for all simulations on this metric, the reputation system itself changes the traffic pattern once activated because when a domain is being “ignored” by the iREX reputation system, the topology effectively changes, and this introduces error into the metric. Control packet overhead is defined as the global total number of reputation control packets issued per unit time. We use the control packet metric to show control packet overhead for the iREX reputation system.

9.3 DOMAIN NON-CONFORMANCE SCENARIO In the domain non-conformance scenario, we only used the vBNS topology with the traffic load held constant at a 50% traffic load while random domains exhibit non-conformance by advertising prices but not honoring reservations. This situation may happen when a domain cannot respond in a timely manner due to a flood of requests while experiencing a DOS attack. We observe how fast the iREX market reacts to the “bad” domains by measuring reputation effectiveness and control packets over a period of time. Reputation effectiveness is defined as the number of reservation requests that arrive at the “bad” domain while the iREX reputation system is active divided by the number of reservation requests that arrive at the “bad” domain without using the iREX reputation system. We use the reputation effectiveness metric to show how effective iREX’s reputation system is.

Figure 20. Reputation Control Packets Figure 20 shows the control packet overhead of the iREX reputation system. As domains make requests that time out right after t = 0 minutes, domains start sending out probes and then complain. Then as the bad domain is excluded from the iREX path vector at about t = 4 minutes, the domains cease to complain. Notice that less control packets used to complain about a domain’s non-conformance also reflect a lower reputation efficiency.

10. RELATED WORK

Figure 19. Reputation Effectiveness Figure 19 shows time based reputation effectiveness with domain non-conformance initiated at time t = 0 minutes. The graph shows that for all cities simulated, the reputation system starts to be effective at just after 3 minutes. The reputation is effective about 70% on average, with the location of the domain within the topology and the traffic pattern being

iREX uses source routing, and source routing for special purpose flows has been supported by publications [12], [13], [14], [15], and [16]. Applying economics and the concept of pricing within networking has been studied in [17], [18], [19], and [20] but work in ID policy within an economic environment has been sparse. Fankhauser et al. [21] proposed an economics based SLA trading system, Koistenen et al. [22] proposed a protocol for peers to negotiate prices and Wang et al. proposed RNAP [23], but in these systems, policy deployment is done bilaterally among neighboring peers whereas in iREX, the source domain deploys policy bilaterally with all domains involved in the deployment. Ebata et al. proposed a system for ID provisioning and accounting in [24], but they focus on the dissemination of information about services available, whereas iREX focuses on providing information about the desirability of available ID transport services. Choice in ID paths has been commercially offered by [25] and [26], but these are just a start – iREX would offer more than just a choice of the first hop ISP. SLA/SLS automation has been proposed by the CADENUS [27]

This full text paper was peer reviewed at the direction of IEEE Communications Society subject matter experts for publication in the Proceedings IEEE Infocom.

framework, but to solve the inter-domain problem they have a centralized entity called the RMID [28] which has global knowledge while iREX uses fully distributed mechanisms. Bandwidth switching exchanges like Tradingcom Europe [29] and Enron Broadband Services [30] (currently in the midst of restructuring) are centralized services that operate like stock exchanges where ISPs trade excess capacity – iREX is a fully distributed architecture that can be used for similar purposes, but without the use of any centralized element. Preliminary work on iREX concentrating on how economics could be used for E2E ID QoS was previously published in [31].

[1] [2] [3] [4] [5] [6] [7]

11. CONCLUSIONS & FUTURE WORK We have introduced iREX as an architecture that uses economics and fully distributed mechanisms to empower domains to self-manage the deployment of E2E ID QoS policy while maintaining each domain’s autonomy. In conclusion, we revisit the three conceptual requirements we stated for an E2E ID QoS policy automation architecture and outline how the iREX architecture upholds them: Self-management is upheld because the iREX architecture empowers domains that need ID QoS to proactively deploy E2E ID QoS policy across ID borders without using any intermediary, proxy or central entity – while maintaining the autonomy of all domains. Through the fault tolerance and reputation systems, all mechanisms within iREX prevail even when parts of the network fail or are non-conforming. Accountability is upheld because for each deployed E2E ID QoS policy, iREX assures proper assignment of transport responsibility and maintenance accountability in exchange for payment commitment by using reservation signaling to establish a bilateral agreement between the resource user domain and each resource provider domain without the use of proxies, even if they are several domain hops away. Congestion-avoidance is upheld because the iREX path vector selection process ensures that paths that remain within the iREX path vectors after distributed evaluation and filtering are paths that use the least congested resources. We have also presented a multitude of simulation results that demonstrate iREX’s performance. In simulations comparing iREX with the current SLA method, we observed that iREX uses network resources more efficiently which resulted in less reservation blocking, and that iREX caused less congestion per reservation while incurring minimal overhead in terms of signaling. The simulation results also show that iREX has fault tolerant properties that can overcome link failure, and a reputation system that increases resistance to domain non-conformance. Our future work is divided into 1) continuing to prove iREX’s merits by investigating the architecture from the standpoint of economics theory and further simulations, and 2) continuing to build iREX into a real system by mapping the iREX architecture to “real world” intra-domain architectures and designing an iIDPS.

[8] [9] [10] [11] [12] [13] [14]

[15] [16] [17] [18] [19] [20] [21]

[22]

[23] [24] [25] [26] [27] [28] [29] [30] [31]

Frame Relay Forum, “Service level definitions implementation agreement,” FRF.13, August 1998. TeleManagement Forum, “SLA management handbook,” GB917 Eva.Ver.1.5, June 2001. D. Goderis, D. Griffin, C. Jacquenet, G. Pavlou, “Attributes of a service level specification (SLS) template,” Internet-Draft, draft-tequila-sls03.txt, Oct. 2003. Y. Rekhter and T. Li, “A Border Gateway Protocol 4 (BGP-4)”, RFC 1771, Standards, March 1995. W. Fang and L. Peterson, “Inter-AS traffic patterns and their implications,” in Proc. IEEE Global Internet Symposium, December 1999. R. Yavatkar, D. Pendarakis, R. Guerin, “A Framework for Policy-based Admission Control,” RFC 2753, Informational, January 2000. J. Postel, “Transmission Control Protocol,” STD0007 (RFC0793), September 1981. R. Braden, Ed., L. Zhang, S. Berson, S. Herzog, and S. Jamin, “Resource ReSerVation Protocol (RSVP) - Functional Specification”, RFC 2205, Standards, September 1997. G. Malkin, Traceroute Using an IP Option, RFC 1393, January 1993. http://netresearch.ics.uci.edu/iREX/. K. Abbink and J. Brandts,. “Price competition under cost uncertainty: A laboratory analysis,” UFAE and IAE Working Papers 550.02, 2002. L. Breslau and D. Estrin. “Design of inter-administrative domain routing protocols,” Proceedings of SIGCOMM 1990. Sept. 1990, pp. 231-241. D. Clark, “Policy Routing in Internet Protocols”, RFC 1102, May 1989. D. Estrin, M. Steenstrup, and G. Tsudik, “A protocol for route establishment and packet forwarding across multidomain internets,” IEEE/ACM Transactions on Networking, vol. 1, pp. 56--70, February 1993. D. Estrin, T. Li, Y. Rekhter, K. Varadhan, D. Zappala, “Source demand routing: packet format and forwarding specification (Version 1),” RFC 1940, Informational, May 1996. I. Castineyra, N. Chiappa, M. Steenstrup, “The nimrod routing architecture,” RFC 1992, Informational August 1996. J. MacKie-Mason and H. Varian. “Some economics of the internet,” In 10th Michigan Public Utility Conference, Mar, 1993. J. MacKie-Mason and H. Varian, “Pricing the internet,” in Public Access to the Internet, B. Kahin and J. Keller, eds., Prentice-Hall, Englewood Cliffs, NJ, 1995. X.Yang, “NIRA: A New Internet routing architecture,” ACM SIGCOMM FDNA 2003 Workshop, Karlsruhe, Germany, August 2003. S. Shenker, D. Clark, D. Estrin, S. Herzog, “Pricing in computer networks: reshaping the research agenda;” ACM Computer Communication Review, Vol. 26, No. 2, April 1996, pp 19 – 43. G. Fankhauser, D. Schweikert, B. Plattner., “Service level agreement trading for the differentiated services architecture,” Swiss Federal Institute of Technology, Computer Engineering and Networks Lab, Technical Report No. 59. November 1999. J. Koistinen, A. Seetharaman, “Worth-based multi-category quality-ofservice negotiation in distributed object infrastructures,” Hewlett Packard Software Technology Laboratory Technical Report HPL-98-51 (R. 1), July 1998. X. Wang and H. Schulzrinne. “RNAP: a resource negotiation and pricing protocol,” In Proc. NOSSDAV '99, Basking Ridge, NJ, June 1999. T. Ebata, M. Takihiro, S. Miyake, M. Koizumi, F. Hartanto, and G. Carle, “Interdomain QoS Provisioning and Accounting,” Internet-Draft, draft-ebata-interdomain-qos-acct-00.txt, November 1999. InterNap. http://www.internap.com. Routescience. http://www.routescience.com. CADENUS. http://www.cadenus.org. O. Dugeon, A. Diaconescu, “From SLA to SLS up to QoS Control: The CADENUS framework,” XIII World Telecommunications Congress, Paris, France, September 2002. Tradingcom Europe. http://www.tradingcomeurope.com. Enron Broadband Services. http://www.enron.com/corp/products/broadband/. A. Yahaya and T. Suda, “iREX: Inter-domain QoS Automation using Economics,” Proceedings of IEEE CCNC, January 2006.

This full text paper was peer reviewed at the direction of IEEE Communications Society subject matter experts for publication in the Proceedings IEEE Infocom.