A Cross-Layer Cross-Overlay Architecture for Proactive Adaptive ...

1

A Cross-Layer Cross-Overlay Architecture for Proactive Adaptive Processing in Mesh Networks Dilip Krishnaswamy, Hsien-Po Shiang, John Vicente, Vidya Govindan, W. Steven Conner, Sanjay Rungta, Winson Chan, Kai Miao Intel Corporation, 1900 Prairie City Road, Folsom CA 95630, USA Abstract— The dynamic characteristics of wireless mesh network environments make delay-sensitive multimedia applications processing in such networks a very challenging task. Cross-layer optimizations are commonly designed to ensure that application requirements are met. Improved timeliness of delivery and precision of information as acquired by a decision-entity can help in effective management of resources and strategies in the mesh network to achieve better application availability, stability, and performance. In this paper, an information exchange and database architecture is presented to allow regular updates and access of cross-layer parameters through distributed database services over an overlay network. The architecture provides cross-layer cross-overlay information to enable proactive decision making and adaptation with distributed optimization for transmission of delay and bandwidth sensitive application flows in mesh networks. Distributed multi-flow routing optimization over heterogeneous wired + wireless-mesh networks with the suggested information-exchange architecture is also explored.

I. INTRODUCTION In recent years, wireless mesh networks [1] have become increasingly popular and the demands for effective support for end-to-end QoS management of multimedia applications over such networks with dynamically varying conditions have also increased. Centralized cross-layer solutions are becoming less promising due to the limitations of scalability, control information gathering, and implementation complexity. It is essential to have distributed solutions with timely and accurate information exchange for these delay-sensitive applications. Figure 1 shows the general distributed control that can be possible in a mesh network node with cross-layer cross-overlay information exchange in the network. In the distributed architecture, decisions are made based on information available from multiple sources. The information sources could include the following: (a) upper protocol layer requirements, including application Dilip Krishnaswamy is with the Mobility Group at Intel Corporation. Hsien-Po Shiang is a visiting researcher at Intel Corporation and a PhD candidate at the University of California, Los Angeles, CA, USA. John Vicente, Sanjay Rungta, Winson Chan, and Kai Miao are with the Information Technology Research Group at Intel Corporation. John Vicente is a PhD candidate at Columbia University, New York, NY, USA. Vidya Govindan is with the Digital Enterprise Group at Intel Corporation. W. Steven Conner is with the Corporate Technology Group at Intel Corporation

statistics, bandwidth requirements, priority categories, length of data packets, packet arrival rate, impact metric of losing a specific packet, and the deadline for receiving a useful packet, (b) the information from lower layers in the protocol stack that provide parameters that decide the operations of transmission, such as routing decisions, transmission opportunity (TXOP) of contention-less medium access control (MAC), retransmission limits, and physical layer modulation and coding schemes (MCS), and, (c) the network environment that is usually captured by designing a set of good parameters to be monitored and exchanged among network nodes, such as expected delay, auto-regressive bandwidth prediction, or round-time average queue size, and information related to interference and congestion in the network. Based on the information above, intelligent decision-making and control mechanisms can be designed to optimize strategies such as routing paths for flows or transmission opportunities that need to be assigned to different flows. Such optimization techniques can maximize a certain utility function jointly considering the service availability, stability, and application performance. The control mechanisms can operate at different time-scales – some control mechanisms can operate at longer time-scales such as during application layer session management, while other control mechanisms may operate at lower time-scales such as on a per packet basis.

Figure 1. Distributed control with cross-layer cross-overlay information exchange We deploy a lightweight information exchange service on an overlay network to provide support for regular asynchronous infrequent updates about network conditions using Distributed Virtual Machines (DVMs). This has several advantages.

2

The cross-layer cross-overlay architecture will be described in detail, with a discussion of multi-flow joint optimization routing algorithms. The paper is organized as follows. Section II illustrates the cross-layer cross-overlay architecture. Section III describes the database system deployed in the architecture. In Section IV, we present a routing algorithm based on extracted information from the database architecture for cross-layer optimization. In Section V, the architecture is extended to heterogeneous networks and optimization for a multi-hop VoIP application over a heterogeneous network using the cross-layer cross-overlay architecture is discussed. Section VI concludes the paper. II. THE CROSS-LAYER CROSS-OVERLAY ARCHITECTURE The high-level system architecture for the cross-layer cross-overlay information exchange architecture for delay sensitive applications in wireless mesh networks is depicted in Figure 2. The architecture based on the OverMesh research platform architecture that was recently proposed [2] which enables a novel wireless mesh internetworking system and realizes network centric computing. Computational overlays in OverMesh provide the facility to deploy distributed services across mobile mesh nodes. The platform enables development and deployment of concurrent distributed experiments on wireless mesh networks. Based on this research platform, a cross-layer searching algorithm, which combines traditional overlay searching with ad hoc network routing so that a physically short searching route is facilitated, was introduced in [3]. The notion of a Distributed Network Information Base (DNIB) with DNIB-SAs (DNIB-Service Agents) at each node was introduced in [4] for information storage and exchange in a wireless network. This paper provides additional architectural insight into the nature of such databases, the mechanism of information exchange between nodes, and addresses how a problem such as multi-flow routing optimization over heterogenous networks (hybrid wireless networks (WiMAX +

WLAN) with wireless+wired+wireless end-to-end paths) can take advantage of such architectural features. The following are key properties of OverMesh [2]: - Infrastructure-free: a decentralized edge/access system with an aggressive convergence strategy for node computation, network processing and storage. Any physical node may be capable of supporting switching or routing functions, along with client or compute functions; - Network virtualization: based principally on a robust DVM overlay strategy, the use of computational service overlays would enable a computational model for provisioning and managing network structure, services and resources; - Emergent control and manageability: to achieve the same resilience seen in today’s decentralized internetworking systems, it allows for alternative learning and inference techniques to off-load human-dependency on operational management and provisioning to discover, predict and adapt distributed network state; - Cooperative and adaptive end-to-end control: to support a horizontal (end-to-end) and vertical (node-level) systems orientation to scale and adapt wireless communications, It provides for tighter layer integration and automation of application-to-network control and management through cross-layer facilities. OverMesh research is primarily focused on scaling mesh networks, realizing overlay service architectures, resource and resiliency management and end-end optimization challenges through self-organizing and cross layer services, respectively. OpenDHT

VM

VoIP VM

Overlay Applic.

VM

Dist. DB

VM

PlanetLab Prestandard 802.11s

Hardware

LOCAL DB

CROSS LAYER/OVERLAY

(a) It provides more efficient information exchange capability and distributed information availability between nodes. Due to the broadcast nature of the wireless environment, the required information is more easily obtainable from a distributed database rather than to request directly from the sources. (b) The information exchange service is infrequent and lightweight in nature, and therefore the overhead associated with this service is small. (c) It enables scalability with management capabilities for clustering nodes, with a scalable information exchange network service in the application layer. (d) Overall, it provides the ability to perform proactive adaptive processing for multiple flows in the network, based on requirements of the flows and the dynamic conditions in the network, along with optimization at different layers in the protocol stack such as rate adaptation at the application layer, routing optimization at the network layer, or optimization for transmission strategies at the MAC/PHY layers.

Figure 2. System diagram of current cross-layer cross-overlay information exchange architecture The current OverMesh research prototype leverages PlanetLab [5] to provision light weight virtual machines, or VServer, on selected set of OverMesh nodes. PlanetLab is a distributed service platform in the application layer with services in the virtual machines (VMs) such as an OpenDHT distributed hash-table directory lookup service [6], a VoIP application service, and a distributed database service. Each OverMesh node can have multiple virtual machines that participate in different slices [5]. These slices form the different overlay networks the can cooperate and collaborate to provide specific services or perform specific tasks. Current PlanetLab implementation requires an administrator to explicitly select the

3 OverMesh nodes for a given slice at a web console hosted on a central server. This central server also coordinates the allocation of resources and the creation of virtual machines on each OverMesh node. In OverMesh, the challenge is to decentralize this function, and allow the nodes to intelligently form sub-groups of nodes to accomplish tasks over a distributed network.

where glocal(t’) represents the collection of local information gathered at time t’ (where t’≤ t ) directly from the neighboring environment in the vicinity of a node, and hglobal(t’’) represents the collection of global information gathered at time t’’ through the overlay network, where in general t’’ ≤ t’ ≤ t (since global information is exchanged in a slower manner). Information exchange overlay network

Overlay

Overlay VM VM VM VM VM VM VM VM

A pre-standard implementation of IEEE 802.11s [7] is adopted for mesh nodes. The 802.11s mesh implementation provides layer 2.5 routing integrated on the 802.11 MAC and PHY. 802.11s routing uses MAC addresses rather than IP addresses to maintain transparency with the IP layer and existing network infrastructure. The default routing protocol is a variant of the ad-hoc on-demand distance vector (AODV) routing protocol [8] modified to enable route identification based on performance metrics rather than hop-count. The protocol uses an airtime metric similar to ETT, derived from hop-by-hop MAC statistics, that is intended to identify routes that minimize the end-to-end amount of time spent transmitting or retransmitting a message across a multi-hop path, typically resulting in higher throughput routes than when using hop-count as a metric. The information exchange service allows integration of better decision-making in the network. A multi-flow routing optimization problem is solved as an example showing the benefit of such service for delay-sensitive and bandwidthsensitive applications. As shown in Figure 3, the information related to cross-layer parameters are exchanged over the application overlay network as a distributed database service. The information is associated with the local database information to enhance the implemented operations in the 802.11s protocol stacks. It is possible that client nodes with limited capabilities can obtain the global information from the distributed database through information query mechanisms from their more-capable neighbors. For heterogeneous networks such as wireless+wired+wireless networks and/or hybrid WiMAX+WLAN wireless networks, application-layer overlay services can extend across different networks using different wireless protocols. Local information updated may be limited to a specific network-type such as an 802.11s network, while global information stored can span different networks. The current 802.11s standard is targeting relatively small mesh deployments of up to 32 nodes in a network [7]. Overlays can help in information exchange in small-scale networks, and also in larger-scale networks or heterogenous/hybrid networks to enable end-to-end flow optimization and application adaptation in such networks. In summary, with the cross-layer cross-overlay architecture , decisions are made jointly according to the latest local and global information. A general optimization problem that is solved at any given time t can be formulated as: maximize Fapp ( glocal (t '), hglobal (t '') ) , (1)

Overlay

PlanetLab PlanetLab

Overlay VM VM VM VM VM VM VM VM

FLEXMESH FLEXMESH Overlay

Overlay VM VM VM VM VM VM VM VM

PlanetLab PlanetLab

Hardware Hardware

FLEXMESH FLEXMESH

Hardware Hardware

PlanetLab PlanetLab FLEXMESH FLEXMESH

Overlay VM Overlay VM

VM VM VMVM VM VM

Hardware Hardware

PlanetLab PlanetLab FLEXMESH FLEXMESH

Distributed Database Service on PlanetLab

Hardware Hardware FLEXMESH FLEXMESH

Hardware Hardware

Information query Information exchange

FLEXMESH FLEXMESH

Hardware Hardware

Information query

Figure 3. Information exchange using the overlay network. The objective function Fapp may be different for various applications. For example, voice applications are sensitive to the delay metrics, while video applications may need to consider both delay metrics and available bandwidth. In the next section, the database system adopted in the cross-layer cross-overlay architecture is introduced. IIII. A DISTRIBUTED NETWORK INFORMATION BASE FOR THE CROSS-LAYER CROSS OVERLAY ARCHITECTURE In the cross-layer cross overlay architecture, local and global network information can be stored at each node in a Distributed Network Information Base (DNIB) [4] at each node, with query management for retrieval of information. A DNIB Service Agent (DNIB-SA), as illustrated in Figure 5, will be available at each node to respond to requests for information from a neighboring node in the network. The DNIB-SA will respond with information available from the DNIB content locally available at the node, and, if necessary, the agent will forward the request to one or more neighboring node(s), until end-to-end information between two end points involved in the communication is obtained. Some of the tasks that DNIB-SAs can perform (but not limited to) are listed below [4]. DNIB-SAs can (a) respond to requests from neighboring nodes, (b) monitor conditions on active links in active routes and forward information about dynamic variations in link quality to nodes in the local routing table at a node, with information to be propagated quickly to endpoints in a communication path to enable real-time adaptation, (c) change

4 policies that can relate to routing, QoS, or security as needed, and in addition, depending on local, global or end-end situations, the policies could relate to changing flow transport parameters or changing monitoring requirements, (d) change network configuration and enable localized route modifications when required by collaborating with DNIB-SAs in a sub-network to configure an alternate route when conditions on a link degrade, (e) propagate end-to-end information to endpoints for a newly configured route. These services provided by the DNIB-SAs can be enabled through the system overlay architecture.

Figure 4: DNIB and DNIB-SAs distributed configuration The DNIB can consist of two databases – a global database with global information across the overlay infrastructure, and a local database with neighboring information. Although the two databases reside on different protocol layers (see Figure 2), they can exist physically in the same database system at a node (machine). In the current setup, the database system is built using MySQL. The entries of each table are identified with an attribute indicating local/global information. The information exchange mechanisms that update the databases are different for these two kinds of entries. Information exchange mechanisms can be global or local with different update frequencies listed below: (1) Global information update, with frequency freqG . Not all the nodes in the network may have information that pertains to global database information. The information sources for global information exchange are the capable nodes that have the PLANETLAB installed on the overlay networks, which constantly exchange information through the overlay application virtual machines. Information in the global database at a node can be populated by local database information regarding link conditions, or through information messages propagated through the overlay network. Only the entries with a global attribute are exchanged during global information updates. For scalability, one can include architectural options such as proactive propagation of such information, or propagation of such information to only a limited number of hops with further propagation done only on request from a neighboring node. The information source is recorded for each entry in the database to identify the node that provides the information.

(2) Local information updates with frequency freq L . The local database has entries with a local attribute set, and they are updated constantly when the node gathers the neighboring information by sending hello messages to the neighbors. (3) Local –to- Global information sync frequency freq S . The Local-to-Global synchronization frequency is represented as freq S . It should be noted that the synchronization is only meaningful when freqG < freq S ≤ freq L . If one sets freqS = freq L , then if a local information entry is updated, then the corresponding entry in the global database at the collocated node (machine) is updated to keep the global database current with the latest information. For example, local link-layer information could be updated more frequently on the order of 10s of milliseconds (for example every 20ms) while the global information may be transmitted less frequently through the overlay network on the order of a few 100s of milliseconds (e.g. approximately every 300ms for example). Transmission durations for packet data transmission are much smaller time-scales such as a few 100s of microseconds to a few milliseconds (e.g. transmission opportunities for high-data rate video aggregated packet transmissions could last about 2ms over a given link). This ensures that frequencies of updates and network information propagation are infrequent and lightweight, relative to the amount of time taken by application data packets in the network. It should be noted that there is no need for updates to happen at specific times only, with a synchronized clock. There is no need for synchronization for information propagation in the network. Information is propagated whenever possible at an approximate frequency of update for global and local information suggested by freqG , freq S , freq L parameters. Distributed optimization in the network can be performed based on the latest information available. A node performing an optimization uses whatever information it has, which could be current or delayed depending on propagation delays in the network. If there is a possibility that there are multiple sources for the same information arriving over the distributed network, a loosely synchronous value of time can be used to tag messages (such as a network with a loosely synchronized start time and coarse-grain updates to a local time counter (e.g. the counter could increment locally on each node every 100ms)). This can help to keep the latest message regarding some specific information in the database, while ignoring old updates arriving in a delayed fashion from other sources. In the example problem discussed in the paper, distributed dynamic routing optimization is accomplished using the data available through information exchange mechanisms in the cross-layer cross-overlay architecture. Figure 5 illustrates the database designed for the cross-layer cross-overlay architecture. The Flow, Path, Link, Node, and Network-card entities are

5 shown. These entity tables contain the flow-level, path-level, link-level, node-level, and network-card-level information respectively. The connections between the tables are as follows: a. Each flow in the Flow Table can have multiple routes/paths in the path Table, which are composed by several links in the Link table. Each flow in the Flow Table is associated with two nodes in the Node Table indicating the source node and the destination node. This is indicated by the [M:2] cardinality in the relationship between a Flow and Node entities. Multiple flows may exist, and multiple paths are possible for each flow, indicated by the [M:N] cardinality of the relationship between the flow and path tables. Each flow may have different requirements such as End-to-End delay and End-to-End bandwidth. The current effective end-to-end delay and bandwidth that is allocated to each flow is also stored based on latest available information for the flow. b. Each path in the Path table contains the list of the links associated with the path. The Path table provides the path between the source and destination nodes associated with a given flow. Paths can contain one or more links, and each link may be associated with multiple paths, indicated by the [M:N] cardinality on the relationship between the Path and Link tables. c. Each link in the Link Table is associated with MAC addresses associated with the two nodes at the end-points of the link, indicating the transmitter and the receiver of the link connection. Each link may support a different protocol and may operate in a different frequency channel. It is important to keep track of relevant current information about the link such as current MCS scheme, average number of retransmission, and delay, bandwidth, and utilization metrics associated with the link. Different metric values related to specific link between two nodes can be stored for different network protocols such as 802.11n or 802.16e and/or for different channels that may be used for the link communication. d. Each node in the Node Table contains information about the node, including information about multiple network cards with multiple radios, and MAC Addresses associated with them. e. Each network card within a node is represented as a NetworkCard table, and if a network card supports multiple protocols, then the network card entity is replicated for each protocol that is supported. f. The delay metric is expanded further. It is possible that different delay metrics can be associated with each link which can be stored in a Delay Table. A delay value has a type to identify what kind of delay this value represents for, such as ETT, ETX, RTT, or network ping delay, based on different metrics that one may choose to use, and the routing algorithms that one may choose from [8,9,10,11,12,13,14]. Information about an approximate time-stamp associated with the metric and whether the information is obtained local link layer measurements or through the overlay architecture is also stored.

Figure 5: DNIB database design Given a flow with a source-destination pair, for delay-based routing optimization, one can extract the multiple possible routes from the Path Table and sum up the corresponding delays of the links that compose these routes to determine optimal and alternative routes for the flow. IV. MUTLI-FLOW ROUTING OPTIMIZATION The multi-flow routing optimization is achieved by gathering metrics such as ETT/ETX link and airtime estimates [4], delay and bandwidth demand constraints for flows, and information about the interference graph, and channel allocations to links, from the global database and local neighbor information. Additional statistics such as traffic loads and congestion may also be gathered by distributed traffic monitoring and analysis tools such as CoMo [15] which are complimentary to this architecture and provide additional insight into the performance characteristics of different links, nodes, and paths within the network. A joint optimization is performed to meet constraints for multiple flows in the network. Channel allocation is assumed to be fixed prior to solving the routing optimization problem. A source node for a flow performs the joint optimization over all the flows, with its goal being to determine the optimal route (or combination of routes) for flows originating from it. A flow fk is an application with a fixed source and destination pair. Different applications can co-exist in the network with multiple flows. The set of possible selected paths for flow fk is given by : (2) Pathk = { pathk1, pathk2 ,...., pathkn ,..., pathkN } , assuming that there are N paths for flow fk . The selected routing graph (combination of paths of each flow) is given by: (3) r ∈ RouteGraph  { Path1 × Path2 × ... × Pathk } . Let us assume that the throughput demand of flow fk is demandk , and the delay deadline requirement is deadlinek . The ETT metrics for a certain link i maybe different for each flows, since the effective packet length Sizek might be different with applications: S ize k . (4) E T T k ,i = E T X i × t k ,i (T H i )

6 Here TH i represents the throughput (bandwidth) over the link i. and tk ,i is the time portion ([0,1]) of the flow fk being distributed over link i. Let us denote the effective throughput of a path of flow fk pathkn as TH pathkn , which is calculated from the following relationship: 1 path n = max X j k , 1 ≤ j ≤c TH pathkn pathkn

=

where X j

(5)

∑

i ∈ channal j on pathkn

ETTk ,i .

Information about link conditions is taken from the available latest local and global information at a node (global information is delayed relative to the local information). Let βk,n be the portion of the flow fk being distributed over the path pathkn . Let us denote the effective throughput of a flow fk through : multiple paths with a certain routing graph r as TH koverall ,r

THkoverall = ,r

∑ βk,n ⋅ TH path , 0 ≤ βk,n ≤ 1 . n

n k

(6)

The Joint Optimization Problem is: overall 2  1 − THk,r  ∑ demandk  r ∈RouteGraph

minimize

k

subject to

∑

i ∈pathkn

ETTk,i ≤ deadlinek − gapk

∑tk,i

≤1

k

≥ demandk , if fk is a nonscalable application THkoverall ,r pathkn ∈ r, for all fk

(7) As conditions vary, the optimization problem can be solved either in a distributed manner or in a centralized manner, at approximately regular intervals. In a distributed optimization solution, we will assume that each source node for a flow will make independent decisions with regard to its preferred route for the flow, in the process of obtaining a solution to the joint optimization problem over all the flows. The solution to the above problem is solved independently at each source node for a flow based on the latest information available at the source node, and decisions are subsequently taken about the routes that would be used for the flows that a source node is responsible for. Although each source node considers the optimal routes for all other sources as well, it will not force others to follow the choices it might make for other sources. It uses however uses the routing decision it takes for the flows originating from it. The distributed method is based on the latest information from the local and global database of each individual source node, which could be slightly different among source nodes due to the various locations and information update frequencies. For a centralized solution, the multiflow routing joint-optimization problem can be solved at a central node using information collected through overlay mechanisms and local

updates. Subsequently, the selected paths for the flows can be distributed to the participating mesh nodes by information transfer engineered using distributed overlay applications. Nodes will continue to use a previous known solution until an updated solution arrives through an overlap application information transfer mechanism. Flows can be scalable or non-scalable. Non-scalable flows have a fixed minimum requirement on the allocated bandwidth to the flow below which the flow becomes infeasible. Scalable flows (such as video) may have a range of tolerable bandwidth such as bandwidths lower than a maximum demand specified may be tolerable, with some loss in quality. When solving an optimization problem for such scalable applications, such a loss in a metric associated with quality (such as PSNR) should also be considered into the optimization function to minimize the impact to such a metric. For applications such as VoIP that have a strict end-to-end delay constraint but that do not need significant bandwidth requirements, one can focus on path construction through the network and applying the end-to-end delay constraint during path construction. To identify possible routes between a source node and a destination node, one can exhaustively construct all the possible routes that grow from the source. If the accumulated delay of a certain route (by adding the ETT values) exceeds the delay deadline of the application, then one can drop the route from further consideration. Path construction is continued until the destination node is reached, or when the delay deadline is reached. Depending on the average degree of a node in the mesh network, and the average number of hops required to go from a source to a destination, the complexity of such computation can be significant. Consider, for instance, a network with size n (number of nodes), and each node with average degree k (has k neighbors). Assume that the end-to-end delay metric results in an average path length of h. The number of considered routes from a source to a destination in the exhaustive approach could be as high as O(k h ) in the worst case. A sample pseudo code for such path construction is provided in Appendix A. Alternatively, one could use heuristic techniques to identify multiple paths between a source and a destination that meet the delay constraint, by applying Dijkstra algorithm between intermediate nodes, and then extending the path obtained by the Dijkstra algorithm to include the source and the destination in the end-to-end route. Thus, one can intentionally force the selected routes to flow through a set of predetermined nodes, and use the Dijkstra algorithm to find the shortest delay path between two intermediate nodes. We call the set of these predetermined nodes as a Multi-path Middle-node Set (MMS). Note that we can have multiple MMS’s in the network. We give an example using pseudo code for a two-MMS case, one MMS as the neighbors of the source (SourceMMS) and the other MMS as the neighbors of the destination (DestinationMMS). The Dijkstra algorithm is applied to find routes between nodes

7 in the SourceMMS to the nodes in the DestinationMMS. The paths obtained are extended to include the source and the destination to obtain the complete end-to-end route. Routes which exceed the end-to-end delay constraint are dropped during such construction. In this example, assuming that Dijkstra’s algorithm has complexity O((n − 2)2 ) (source and destination nodes are excluded in an n-node graph), the complexity of execution for the path construction for such a two-MMS graph is O(k 2 (n − 2)2 ) (this assumes an average degree of k for the source and destination nodes). Appendix B includes pseudo-code for such a path construction algorithm. When applications are considered that require significant bandwidth demand constraints, one should consider eliminating paths that do not provide the end-to-end throughput that is desired. After possible routes for a flow are obtained, one can consider the possibility of joint optimization of the flows such that the throughput demands and delay constraints of all flows are satisfied, starting with the satisfactory paths that were identified for each flow. Appendix C has pseudo-code for a possible implementation of the multi-flow optimization, by considering the impact to the delays due to the possibility of overlap between the multiple paths selected for each flow..

Figure 6: Routing Optimization example for two flows

Figure 7: Multi-path options for two flows

Figure 6 shows an example problem solved in a matlab simulation environment. Two VoIP application flows are considered and optimized simulatenously – one from SourceA to DestinationA and one flow from SourceB to DestinationB for a mesh network with mesh nodes located as shown in Figure 6. Figure 7 shows multi-path options for the two flows. The simulations were performed to compare the end-to-end flow packet loss rate with traditional layer 2.5 reactive mesh routing protocol and the multi-path routing with proactive adaptation that is suggested in the paper. Traditional reactive mesh routing establishes a route and a new route is chosen only when an existing route fails. Multi-path routing with proactive adaptation selects multiple possible routes from a source to a destination with partial flow allocation to each path, with alternative routes maintained to allow for faster route switching. Using multiple paths for the same flow provides better resiliency. Additionally, faster proactive adaptation with alternate routes maintained based on link conditions helps in reducing the end-to-end flow packet loss rate. The packet error rate over the links was varied with a probability in the range 0.05 – 0.15. Figure 8 shows that the end-to-end flow packet-loss-rate decreases as the end-to-end delay limit is increased for each application flow. For application flow B, there is not much difference in the packet loss rate values between reactive and multi-path proactive adaptation. However, for application A, better results (more graceful degradation in packet loss rate as end-to-end application delay deadline is reduced) are achieved with the multi-path proactive adaptation. Figure 9 shows results from experimentation with worser link condition variations the link packet error rates varying from 0.05 to 0.3. In this case, it can be seen that end-to-end flow packet error rates start increasing for larger end-to-end flow delay limits, as one would expect.

Figure 8: End-to-End Flow Packet Loss Rate vs Application End-to-End Delay (good channel PER variation from 0.05 to 0.15)

Figure 9: End-to-End Flow Packet Loss Rate vs Application Deadline (good channel PER variation from 0.05 to 0.30)

V. EXTENSION OF THE CROSS-LAYER CROSS-OVERLAY ARCHITECTURE TO HETEROGENEOUS MESH NETWORKS In the discussion in the previous section, it should be noted that a pre-standard 802.11s implementation was considered. However, the cross-layer cross-overlay architecture proposed

8 for mesh networks (such as 802.11s) can be extended for end-to-end optimization over heterogeneous networks as well. For example it is conceivable that a node can have an 802.11n radio and an 802.16e radio, and it may receive information over the 802.11n radio interface and forward it over the 802.16e interface. A typical 802.11s mesh network cannot support ranges beyond 100 meters and mobile devices moving at vehicular speed. Multi-hop relay WiMAX mesh networking can offer significant benefits by extending the coverage of a BS, and extending the range of WiMAX "path" to reach a location. WiMAX Relay Stations (RS) can be Fixed, Nomadic or Mobile. Frequency conversion can be supported to support radios/devices operating at different frequencies, which can provide better spectrum utilization. In addition, the use of multi-radio, multi-channel, multi-protocol capabilities with MIMO technologies, and use of techniques such as network coding and co-operative diversity can provide improvements in bandwidths along with the increased range provided by WiMAX. Considering all these factors, the 802.16j standard (this is work in progress) envisons a multi-hop WiMAX network beyond BS-based network and WiFi 802.11s mesh, capable of significantly extending the range and bandwidth, yet seamlessly internetworking with the local mesh (802.11s based) and with a RS resembling a node in the mesh network discussed in this paper.

generalized node architecture for cross-layer cross-overlay information exchange and distributed optimization over heterogeneous networks is shown in Figure 10. Different layers in the protocol stack implementation at a node can exchange information through cross-layer mechanisms using database implementations to store or retrieve data. Information available about link conditions can be propagated from the local link layer database to the DNIB at the application layer. Database updates can proceed asynchronously with producers providing updates when they are available. Consumers of the database information will utilize the latest available information in the database whenever they need the information. The distributed virtual machines can exchange information peer-to-peer using the underlying network infrastructure for message communication in the network. A key objective would be to create lightweight virtual machines so that control, management and optimization functions can be virtualized and implemented in a distributed manner, and easily invoked when required. Additionally, application services can also be invoked using virtual machine implementations. Groups of nodes engaged in a specific distributed processing task can have a distributed virtual machine implementation to support that task. Different groups of nodes may be formed to process different distributed tasks. Every node therefore does not have to provide support for the same VM. Application-layer virtualization capabilities, along with transport layer and network layer services are shown as a single layer – the actual implementation methodology to provide these capabilities may differ across platforms. The virtualization layer has hardware-awareness about the platform hardware resources and constraints, and the multi-comm multi-radio multi-channel communications support available in the device, to make the most optimal utilization about the available communications/networking capabilities and system resource constraints in the device.

WiFi Mesh Hotspot in Beijing Portal Node VoIP peer WiMAX(opt)

Figure 10: Generalized Cross-Layer Cross-Overlay Architecture Over Heterogeneous Networks

WiMAX(opt) Wireline / Fiber Optical Networks

Broadband Access

With different options for mesh networking providing flexibilities in range and performance, there is a need for proactive adaptation over heterogeneous mesh networks with nodes supporting multiple wireless protocols. The proposed architecture can be extended to such heterogeneous networks as well. A generalized architecture can include nodes with multi-radio, multi-channel, and multi-comm-protocol capabilities, and information transmission can be achieved across heterogeneous networks. Additionally, although PlanetLab implementation is used to create distributed virtual machines, the architecture can be general and any available virtualization technology can be used to create such DVMs. A

VoIP peer

Portal Node WiFi Mesh Hotspot in Folsom

Figure 11. Example of using the cross-layer cross-overlay architecture over a heterogeneous network The overlay network can be deployed over a heterogeneous wireless/wireline network as shown in Figure 11. The figure explores the application of the proposed architecture for a VoIP application from Beijing, China to Folsom, CA, USA with flow optimization over heterogeneous networks in the presence of possibly additional other flows in the network. The lower two layers of the cross-layer cross-overlay architecture in Figure 2 can be different for different networks such as 802.11s WiFi

9 Mesh, WiMAX (optional), etc, while the upper layers including and above the PlanetLab implementation layer (or other distributed virtualization service layer) can provide distributed services with virtual machines across the overlay network. The global database provides information for the routing optimization in (7) to obtain multiple paths that satisfy the delay constraint and bandwidth demands. The transmission delay between the broadband access nodes for the wired/optical portion (or optionally satellite links) is assumed to be measurable and available. Portal nodes are pre-determined for each wireless mesh hotspot. The final end-to-end delay that must be within the end-to-end delay limit (such as 180ms) of the delay-sensitive VoIP application is composed by the following segments: (a) the accumulated delay from the VoIP peer to the mesh portal node in the Beijing 802.11s mesh network, (b) the accumulated delay from the 802.11s mesh portal through a WiMAX multi-hop relay mesh (optional) at Beijing, (c) the end-to-end delay through the wired/optical line from Beijing to Folsom, (d) the accumulated delay through the WiMAX multi-hop relay mesh (optional) to the 802.11s mesh portal in Folsom, and, (e) the accumulated delay from the portal node to the VoIP peer in the Folsom 802.11s mesh. All the delay values with their types are exchanged through the distributed services provided on the overlay network across the heterogeneous network path. The routing algorithm optimizations within each network introduced in Section IV can be performed as long as the latest link delay information (such as the ETX, ETT in the pre-standard 802.11s) is available. Reduced delay constraints can be applied across each network, with routing optimization performed in each network, so that the cumulative delay across different networks can meet the end-to-end delay requirements for the application. VI CONCLUSIONS In this paper, a cross-layer cross-overlay architecture was presented that regularly exchanges cross-layer parameters through distributed overlay services in a heterogeneous network architecture. A joint multi-flow routing optimization problem was explored with extraction of useful parameters for optimization from the proposed database, and construction of routes that satisfy the delay constraint and the bandwidth demand. By combining the benefits of decentralization in both the wireless mesh infrastructure and the application layer overlay services, the suggested cross-layer cross-overlay architecture permits a novel peer-to-peer information exchange architecture in the application overlay network (with controllable information exchange overheads) across possibly heterogeneous wired and wireless networks. Future research directions include finding fast routing adaptation methods, under dynamic network conditions, with exploration of optimizations under different network topologies and various application characteristics. ACKNOWLEDGEMENTS The authors thank Lily Yang for feedback.

REFERENCES [1] I. F. Akyildiz, X. Wang, and W. Wang, “Wireless Mesh Networks: A Survey,” Computer Networks Journal, vol. 47, 2005, pp. 445-487. [2] J. Vicente, S. Rungta, G. Ding, D. Krishnaswamy, W. Chan, K. Miao, "OverMesh: Network Centric Computing", Invited Paper, IEEE Communications Magazine, To appear [3] G. Ding, J. Vicente, S. Rungta, D. Krishnaswamy, W. Chan, K. Miao, "Overlays on Wireless Mesh Networks: Implementation and Cross-Layer Searching" , To appear 9th IFIP/IEEE International Conference on Management of Multimedia and Mobile Networks and Services, Dublin, 2006. [4] D. Krishnaswamy, J. Vicente, "Scalable Adaptive Wireless Networks for Multimedia in the Proactive Enterprise", Intel Technology Journal, Q4 2004. [5] http://www.planet-lab.org [6] http://www.opendht.org [7] IEEE 802.11 Task Group s, “Draft Amendment: ESS Mesh Networking” IEEE Std 802.11s draft 0.03, July 2006. [8] Charles E. Perkins and Elizabeth M. Royer. Ad hoc On-Demand Distance Vector Routing. In Proceedings of the 2nd IEEE Workshop on Mobile Computing Systems and Applications, 1999. [9]T. Larsson and N. Hedman, “Routing protocols in wireless ad-hoc networks - a simulation study,” Master’s Thesis, LuLea Tekniska University, 1998. [10] J. Bicket, D. Aguayo, S. Biswas, R. Morris, “Architecture and evaluation of an unplanned 802.11b mesh network,” MobiCom ’05, New York, NY, USA: ACM Press, 2005, pp. 31–42. [11] R. Draves, J. Padhye, and B. Zill, “Routing in multi-radio, multi-hop wireless mesh networks,” in MobiCom ’04. New York, NY, USA: ACM Press, 2004, pp. 114–128. [12] R. Draves, J. Padhye, and B. Zill, “Comparison of routing metrics for static multi-hop wireless networks,” in SIGCOMM ’04. New York, NY, USA: ACM Press, 2004, pp. 133–144. [13] H.P. Shiang, M. van der Schaar, “Multi-user Video Streaming over Multi-hop Wireless Networks: A Distributed, Cross-layer Approach Based on Priority Queuing”, submitted to IEEE JSAC. [14] Changwen Liu, W. Steven Conner, Mark D. Yarvis, and Xingang Guo, "Guaranteed On-Demand Discovery of Node-Disjoint Paths in Ad Hoc Networks," Intel Technical Report IR-TR-2004-261, Intel Corporation, July 2004. [15] Gianluca Iannaccone, “CoMo: An open infrastructure for network monitoring,” Intel Research Technical Report, February 2005. [16] WiMAX Forum White Papers, Technical Documents and Specifications, http://www.wimaxforum.org/news/downloads/, http://www.wimaxforum.org/technology/documents/.

APPENDIX A. Pseudo code for multi-path construction with a delay-constrained exhaustive search For each application with Input: src, dst, deadline, apply the following algorithm: 1. ETT ÅgetETT(Database); 2. N Å size_of_node(ETT); 3. for i = 1 to N 4. Neighbor_i Å CheckConnect(i); 5. end for 6. Put src node in the TakeSet

10 k Å 0; (current route number) route_k Å src; accum_delay_k Å 0; while(dst not in the TakeSet) for all node j that is NOT in the TakeSet but nearest to the TakeSet 12. for all route k that exists in the TakeSet 13. Set i = LastNode(route_k) 14. if( j is in Neighbor_i ) 15. temp_delay Å accum_delay_k + ETT(i, j); 16. if temp_delay < deadline 17. k Å k+1; 18. Add j into route_k; 19. accum_delay_k = temp_delay; 20. end if 21. end if 22. end for 23. Put node j into the TakeSet; 24. end for 25. end while 26. Output route_k that ends with dst and the corresponding accum_delay_k; B. Pseudo code for multi-path construction with a delay-constrained Dijkstra search between neighbors of the source to neighbors of the destination node. For each application with Input: src, dst, deadline, apply the following algorithm: 1. ETT ÅgetETT(Database); 2. N Å size_of_node(ETT); 3. for i = 1 to N 4. Neighbor_i Å CheckConnect(i); 5. end for 6. kÅ 0; 7. for all node i in the Neighbor_dst 8. for all node j in the Neighbor_src 9. k Å k+1; 10. [delay_metric_k, route_k] = Dijkstra_shortest_path(i,j); // Dijkstra algorithm for the shortest path from node i to j 11. end for 12. end for 13. for all route_k 14. if dst or src is in the route_k 15. drop route_k 16. else 17. one row of output_delay_metric Å delay_metric_k+ ETT(src, route_k_start)+ETT(route_k_end, dst); 18. if the delay_metric_k > deadline 19. drop route_k; 20. else 21. update route_k Æ one row of output_route; 22. end if 23. end if 24. end for 25. Output output_route and output_delay_metric; Subroutine for running Dijkstra’s algorithm: [delay_metric_k, route_k] = Dijkstra_shortest_path(src,k) 1. Initialize delay_metric for all nodes Å 0; 2. Put src and its nearest neighbor in the TakeSet; (smallest ETT) 3. update route_nearest_neighbor; 4. while( k not in the TakeSet) 5. index Å 0; for all node i not in the TakeSet 6. for all node j in the TakeSet 7. 8. 9. 10. 11.

7. if i and j are connected 8. Temp_index = delay_metric_j + ETT(i,j); 9. index Å index + 1; 10. end if 11. end for 12. [Value, index]= min(Temp_index); 13. delay_metric_i = Value; 14. put i into TakeSet; 15. update route_i with index; 16. end for 17. end while 18. return delay_metric_k, and route_k C. Pseudo code for proactive multi-flow optimization Input: applications, deadlines, multi-routes for all distinct applications 1. ETT ÅgetETT(Database); 2. N Å size_of_node(ETT); 3. for i = 1 to N 4. Neighbor_i Å CheckConnect(i); 5. end for 6. N_link Å size_of_link(ETT); 7. N_app Å size_of_application(applications); 8. for all the combinations of routes for the applications 9. for j= 1 to N_app 10. N_path_j Å number of path for application j in the combination; 11. end for 12. for i= 1 to N_link 13. for all routes in the combination 14. if i is in the route 15. overlap_count_i ++; 16. end if 17. end for 18. end for 19. for i = 1 to N_link 20. update ETT_shared(i) Å ETT(i)*overlap_count(i) 21. end for 22. for all routes in the combination 23. for all links in the route 24. accum_delay_route += ETT_shared(link); 25. end for 26. end for 27. for j= 1 to N_app 28. delay_metric_j = max(accum_delay_route_j/N_path_j)over paths; 29. if delay_metric_j > deadline_j 30. drop the combination; 31. end if 32. end for 33. end for 34. Output routing combinations, delay_metric;