Optimal Network Selection in Heterogeneous Wireless ... - IEEE Xplore

This full text paper was peer reviewed at the direction of IEEE Communications Society subject matter experts for publication in the IEEE ICC 2009 proceedings

Optimal Network Selection in Heterogeneous Wireless Multimedia Networks Pengbo Si†‡ , F. Richard Yu‡ , Hong Ji† and Victor C.M. Leung§ Key Laboratory of Universal Wireless Communication, Ministry of Education Beijing University of Posts and Telecommunications, Beijing, P.R. China ‡ Department of Systems and Computer Engineering, Carleton University, Ottawa, ON, Canada § Department of Electrical and Computer Engineering, The University of British Columbia, Vancouver, BC, Canada Email: [email protected], richard [email protected], [email protected] and [email protected] †

Abstract— The complementary characteristics of different wireless networks make it attractive to integrate a wide range of radio access technologies. Most of previous work on integrating heterogeneous wireless networks concentrates on network layer quality of service (QoS), such as blocking probability and utilization, as design criteria. However, from a user’s point of view, application layer QoS, such as multimedia distortion, is an important issue. In this paper, we propose an optimal distributed network selection scheme in heterogeneous wireless networks considering multimedia application layer QoS. Specifically, we formulate the integrated network as a restless bandit system. With this stochastic optimization formulation, the optimal network selection policy is indexable, meaning that the network with the lowest index should be selected. The proposed scheme can be applicable to both tight coupling and loose coupling scenarios in the integration of heterogeneous wireless networks. Simulation results are presented to illustrate the performance of the proposed scheme.

I. I NTRODUCTION In recent years, with the rapid growth of wireless communication systems, the complementary characteristics of different wireless networks make it attractive to integrate a wide range of radio access technology (RAT) standards, including wireless wide area networks (WWANs) like cellular networks, wireless metropolitan area networks (WMANs) like WiMAX networks, and wireless local area networks (WLANs) like IEEE 802.11 based networks. Several internetworking architectures between cellular and other RAT systems have been proposed. Loose coupling and tight coupling are the two generic approaches to the integration specified by European Telecommunications Standards Institution (ETSI) [1]. In the loose coupling approach, data flows from different types of networks go to the external IP network directly, and only signaling is required between cellular networks and other complementary networks. In the tight coupling approach, complementary networks communicate with the external network through the cellular networks. In addition, an internetworking architecture is developed by the Third Generation Partnership Project (3GPP) to enable the radio resource reuse between the networks as well as the authentication, authorization and accounting (AAA) [2]. This work was jointly supported by the Hi-Tech Research and Development Program (National 863 Program) under Grant 2009AA01Z211, 2009AA01Z246 and 2007AA01Z211, the National Natural Science Foundation of China under Grant 60832009, and the Scientific Research Foundation of Graduate School of BUPT under Grant No. 6, 2006.

To improve the performance of heterogeneous networks and keep users always best connected (ABC) [3], a number of schemes are proposed to deal with the network integration problems. Analytical hierarchy process (AHP) and grey relational analysis (GRA) are used in [4] to combine multiple network selection criteria and decide the weights of the criteria according to the user preferences and service applications. Authors of [5] propose an architectural framework for network selection and a comprehensive decision making process to rank candidate networks for the users. An optimal joint session admission control scheme is proposed in [6] for integrated cellular/WLAN systems with vertical handoff. Game theory is introduced to heterogeneous networks in [7] for radio resource management including bandwidth allocation and admission control. Authors of [8] propose a Markovian framework for the allocation of multiple services in multiple RATs and a model to embed the evaluation of several RAT selection policies considering different criteria. Although some work has been done to integrate heterogeneous wireless networks, most of the previous work considers network layer quality of service (QoS), such as blocking probability and utilization, as the design criteria. Consequently, application layer QoS, such as distortion for multimedia applications, is largely ignored in the integration of heterogeneous wireless networks. However, multimedia applications, such as video telephony and surveillance, are very promising services that require more radio resource compared to other types of services in heterogeneous wireless networks. From a user’s point of view, QoS at the application layer is more important than that at other layers. Popular video compression standards, such as MPEG-4 and H.264, have the capability to adapt its quality (i.e., distortion) to different network conditions in heterogeneous wireless networks. Therefore, application layer QoS should be taken into account in the design of network selection schemes in heterogeneous wireless networks. To the best of our knowledge, the design of optimal network selection in heterogeneous wireless networks considering multimedia application layer QoS has not been addressed in previous work. In this paper, we present a novel distributed scheme based the stochastic optimization formulation Some distinct features of the proposed scheme are as follows. • To improve the user experience as well as to reduce the cost, we consider multimedia application layer distortion and network access price in the network selection optimization. An application layer parameter, intra-refreshing rate, is adapted in

978-1-4244-3435-0/09/$25.00 ©2009 IEEE


heterogeneous wireless networks. • We formulate the heterogeneous wireless networks as a restless bandit system [9]–[11], which has been successfully applied in operations research and stochastic control problems. With the restless bandit approach, the optimal network selection policy is indexable, meaning that it simply selects the network with the lowest index. • It is a fully distributed and scalable scheme, which can be applicable to both tight coupling and loose coupling scenarios in the integration of heterogeneous wireless networks. The rest of the paper is organized as follows. In Section II, the multimedia transmission in heterogeneous networks are introduced. We formulate the network selection problem as a restless bandit problem in Section III. In Section IV the process of the selection scheme is described. Simulation results are presented in Section V. Finally, we conclude this study in Section VI. II. M ULTIMEDIA T RANSMISSION IN H ETEROGENEOUS W IRELESS N ETWORKS In heterogeneous networks, multiple types of totally N networks cooperate to provide seamless coverage for universal wireless access. In this paper, we consider an area with the coverage of three types of networks: wireless local area networks (WLANs), WiMAX networks and cellular networks. To evaluate the performance of multimedia transmission, distortion is considered as the application layer QoS. A. Heterogeneous Wireless Networks Generally speaking, there are two different ways of integrating heterogeneous wireless networks, defined as tight coupling and loose coupling interworking [6]. In a tightly coupled system, a network is connected to another network in the same manner as other radio access networks. In a loosely coupled system, the heterogeneous wireless networks are not connected directly. Instead, they are connected to the Internet. The proposed scheme is applicable to both tight coupling and loose coupling scenarios.

is the bandwidth required by type l service, and C(n) is the capacity of the network n. And the admissible set of CDMA networks n with matched filter receiver is [14] J : PT ≤ PTM AX , Sn = g(n) ∈ Z+ where PTM AX is the maximum available base station power, and PT is the minimum base station transmission power. C. Distortion Optimization for Multimedia Transmission Intra-refreshing of macroblocks (MBs) is an important approach for rate control and error protection. An intra coded MB does not need information from previous frames that may have already been corrupted by channel errors. This makes intra coded MBs an effective way to mitigate error propagation. Given a data rate in a network, authors in [15] provide a closed form distortion model. The source distortion is given by Ds (Hs , ξ) = Ds (Hs , 0)+ ξ(1 − η + ηξ)[Ds (Hs , 1) − Ds (Hs , 0)], where Hs is the source coding rate, ξ is the intra-refreshing rate, η is a constant based on the multimedia sequence. Ds (Hs , 0) and Ds (Hs , 1) are the time average all interand intra-mode selection frames over the E epochs. T −1 for Yall k Ds (Hs , 0, y), Ds (Hs , 1) = Ds (Hs , 0) = T1 k=0 Y1k y=1 T −1 1 Yk 1 k=0 Yk y=1 Ds (Hs , 1, y), where Yk is the number of T inter/intra frames at epoch tk . According to the rate-distortion model [15], the average channel distortion is given by ψ Ω1 E[Fd (y, y − 1)], Dc (ψ, ξ) = 1 − Ω2 + Ω2 ξ 1−ψ where ψ is the packet loss rate, Ω1 is the energy loss ratio of the encoder filter, Ω2 is a constant based on the motion randomness of the multimedia data, and E[Fd (y, y − 1)] is the average value of the frame difference Fd (y, y − 1). The total distortion is D(Hs , ψ, ξ) = Ds (Hs , ξ)+Dc (ψ, ξ). The optimal ξ ∗ to minimize the total distortion is ξ ∗ = arg min D(Hs , ψ, ξ). ξ

B. Admissible Sets In IEEE 802.11e based WLANs, throughput and delay are important QoS metrics. An optimal operating point is determined in [12]. According to these results, we adopt the following admissible set for wireless LAN n J : B l (n) ≥ T B l (n), E l (n) ≤ T E l (n) , Sn = g(n) ∈ Z+ where B l (n) ≥ T B l (n) is the throughput constraint and E l (n) ≤ T E l (n) is the delay constraint. According to [13], the admissible set of WiMAX network n is L J l l U (n)W (n) ≤ C(n) , Sn = g(n) ∈ Z+ : l=1

where L is the total number of service types and U l (n) is the number of sessions of service type l in network n, W l (n)

(1)

To deal with the time-varying wireless connection states of the networks, we use adaptive intra-refreshing rate ξ ∗ to achieve the minimum distortion. III. R ESTLESS BANDIT F ORMULATION As a special type of stochastic control problem, the restless bandit problem is an extension to the classical multiarmed bandit problem. One project is chosen to be active in a distributed way at each discrete time instant within the N parallel projects with finite state spaces. The active project earns a reward, with the change of its state, while the states of other objects are unchanged. According to the indexable rule of the multiarmed problem, the optimal policy can be found by simply choosing the project with the largest index. Although it is a relatively simple solution to the multiarmed problem, in our network selection problem, it is not realistic to allow only the active network to change state. The restless


bandit problem is proposed to deal with this problem [9], [10]. At each epoch tk , the states of all N networks may change. There is also an indexable rule of the restless bandit problem. Projects are selected to be active according to their indices that are calculated by linear programming (LP) relaxations [10]. A. Decision Epochs and Actions The decision epochs are the session arrival and departure time points. The time intervals between two adjacent arrival epochs and two adjacent departure epochs are both exponential L distributed random variables with the rates, ν = l=1 ν l and L l l l l=1 U (tk )μ , respectively, where U (tk ) is the total number of type l sessions in the networks at epoch tk , ν l and μl are the type l session arrival and departure rate. The action is the network selection decision at the current epoch. At each epoch tk , one of the networks is selected to be active, meaning that it is ready to admit a new arrival session at the next epoch tk+1 if a new session arrives at tk+1 . If network n is active at epoch tk , an (tk ) = 1, else an (tk ) = 0. N The actions satisfy n=1 an (tk ) = 1. B. State Space and Transition Probabilities Assume the number of session with service type l in network n at epoch tk is U l (n, tk ). The state of network n at epoch tk is defined as s(n, tk ) = [U l (n, tk )]l∈{1,2,...,L} , where L is the number of service types. Thus the state space of network n is the admissible set Sn . The state of network n under action a evolves according to a Markov chain with the transition probability pai,j (n) from state si (n) = [uli (n)]l∈{1,2,...,L} to sj (n) = [ulj (n)]l∈{1,2,...,L} . Define the expected interval between two epochs for the state si to be τi = E (tk+1 − tk |si (n, tk )), which is the inverse of the −1 L total traffic rate τi = ν + l=1 Uil (n)μl . Define the transition probability matrix of network n with action a to be P a (n) = [pai,j (n)]S(n)×S(n) , where S(n) is the number of available states s(n) of network n. Denote by (l), 1 ≤ l ≤ L, the L-element row vector of which the lth element is one and the other elements are zero, thus the transition probabilities can be represented as

where D(u) is session u’s distortion, B(u) is the price paid by session u, which is related to the current serving network. c1 ≥ 0, c2 ≥ 0 and c3 are constant coefficients. By adjusting the coefficients, the balance of distortion and price can be achieved. Since sessions of the same service type in the same network have the consistent properties, they have similar distortion. Besides, the costs of these sessions are also the same. (2) can be also written as Z = N Consequently, L T −1 l l l U (n, t k )R (n), where R (n) is the rek=0 n=1 l=1 ward by session of type l in network n. The objective of our problem is to maximize the total reward to achieve Z ∗ = max Z(A). A∈A

(4)

D. Indices and Policies The restless bandit approach has an indexable rule that reduces the computational complexity dramatically. For network n in state in , we denote by the index δn (in ). According to the restless bandit approach, the optimal policy A∗ is a set of optimal actions. Let the element of A∗ in row n and column k be a∗n (tk ), which represents the optimal action for network n at epoch tk . Thus if δn is the smallest in δ1 , δ2 , . . . , δN , a∗n (tk ) = 1, otherwise a∗n (tk ) = 0. Define the set of all available policies to be A = {A}. Thus A∗ = arg maxA∈A Z(A). In our network selection problem, at each epoch, the network with the smallest index δn is set to be active, while other networks are passive. At the next epoch, if a session arrives, the active network will admit the new session; if a session departs, only the corresponding network needs to do the de-association action. To solve the restless bandit problem, a hierarchy of increasingly stronger LP relaxations is developed based on the result of LP formulations of Markov decision chains (MDCs) [10]. Please refer to [10] for details. IV. T HE P ROCESS OF THE O PTIMAL S CHEME

In the proposed optimal scheme, at each epoch, a request from the session is sent to all the networks. If this is an arrival epoch, the new session is to be associated to the current active pai,j (n) = ⎧ if sj (n) = si (n) + (l), network, and an optimal intra-refreshing rate is selected. Each νl ζ(sj (n))aτi , ⎪ ⎪ ⎨ l l if sj (n) = si (n) − (l), network calculates its own index based on the current state, Ui (n)μ τi , l l and shares it with others in a distributed way. The network 1 − νl ζ(sj (n))aτi − Ui (n)μ τi , if sj (n) = si (n), ⎪ ⎪ ⎩ with the lowest index is selected to be the active one for the 0, otherwise. new session association decision at the next epoch. If sj (n) ∈ Sn , ζ(sj (n)) = 1, otherwise ζ(sj (n)) = 0. A. Optimal Network Selection C. System Reward The optimization goal is to maximize the total discounted reward which is defined as Z=

(tk ) T −1 U

β T −k−1 Ru (tk ),

(2)

k=0 u=1

where T is the number of epochs considered, and Ru (tk ) is the reward of session u at epoch tk , which is defined as Ru (D(u), B(u)) = [−c1 lg(D(u)) − c2 B(u) + c3 ] τi ,

(3)

The network selection is in a distributed and cooperative way, which can be divided into the off-line stage and the online stage. The off-line computation is as follow. 1) According to the admissible sets of the networks and the session arrival/departure rate, the state space and transition probability matrices under different actions are determined. 2) For each network n and each possible state in ∈ Sn , input the state transition probability pain jn , the reward Rian , the discount factor β and the initial state probability vector


B. Optimal Intra-Refreshing Rate Given the source-coding bit rate Hs and the packet loss rate ψ for session of type l, the intra-refreshing rate ξ is off-line optimized for different situations according to (1) to minimize the total distortion. Thus the minimized distortion D∗ = D(Hs , ψ, ξ ∗ ) can be calculated as a part of the reward Rn,l . This reward is used for the policy optimization.

10 WLAN WiMAX Network Cellular Network .

Distortion (MSE)

9 8 7 6 5 4 0

0.2

0.4

0.6

0.8

1

Intra−Refreshing Rate

Fig. 1.

The video distortion with different intra-refreshing rate ξ. 1

Optimal Intra−Refreshing Rate

α, then off-line compute the finite set of the indices {δin }. Store these indices and the corresponding pain jn , Rian and α in a table. At epoch tk , the on-line computation is as follows. 1) If it is an arrival epoch and the active network na is capable to admit the new arrival session according to Sna , na admit the session and update the its state sna . If this is an arrival epoch but the active network na is not capable to admit the new session, the new session is to be rejected. If this is a departure epoch, the session leaves from the associated network, and the state of the network is updated. 2) Each network shares its state sn as the initial state probability vector α with the others. 3) With α, each network looks up the index table to find out the corresponding index δin . 4) The networks share their indices δin in a distributed way. 5) Each sender arranges the list of the indices from the lowest to the highest. A network is set to be active if its index is in the first place.

0.8 0.6 0.4 0.2 0

Fig. 2.

WLAN

WiMAX Network Cellular Network

The optimal intra-refreshing rate for different networks.

V. S IMULATION R ESULTS AND D ISCUSSIONS In this section, extensive simulation results are presented to show the performance of the optimal network selection scheme. The area considered is covered by three networks: a WLAN, a WiMAX network and a cellular network. For the cost of network services, we adopt the prices in New York city currently: the price of 3G mobile Internet access provided by AT&T is $60 per month; WiMAX access by Sprint/Nextel is $59.99 per month; WiFi access by AT&T is $19.95 per month. We compare the proposed scheme with the existing scheme, in which no application layer QoS is considered and each individual network is optimized separately. A. Optimal Intra-Refreshing Rate Different types of wireless networks provide different data rates and different link quality to the mobile user. From a user’s point of view, application layer distortion is more important than the QoS at other layers. In Fig. 1, we plot the application layer distortion, which is the mean square error between the original and decoded video frames, with different intra-refreshing rate ξ. To minimize the distortion, the optimal intra-refreshing rate can be optimized for the WLAN, WiMAX network and cellular network, which is shown in Fig. 2. B. Reward along the Time Line At each epoch, a network selection decision is made and the number of associated sessions is updated. To illustrate the dynamics of the system, we plot the session number, which is

an average value of 2000 trials, in Fig. 3. In the initial state, there are two VoIP and one video sessions in the networks. Assume the session departure rate μ = 0.2, video session arrival rate ν1 = 1.6 and VoIP session arrival rate ν2 = 3.2. From Fig. 3, we can see that the number of sessions goes up first and becomes converged after about 60 minutes, when the balance between the expected numbers of sessions departs and arrives is achieved, and the total session number does not change dramatically any more. We can also observe that the WLAN, which provides the highest reward is more likely to be selected when it’s not saturated. After WLAN’s saturation, the WiMAX network whose reward is higher than the cellular network but lower than the WLAN becomes the first choice. C. Reward with Different Traffic Rates In this subsection, we present the affect of traffic rate on the expected reward. We adopt the average value of the reward after 60 minutes in the time line as the expected reward. In Fig. 4, the reward comparison for different ν1 is presented. Assume μ = 0.2 and ν2 = 3.2. Since that the total session number increases as ν1 increases, the total reward also goes up for the existing scheme. However, a larger ν1 also means relatively more video sessions, which consume much more resources. Therefore, in the optimal scheme, resource in the network that provides the highest reward could be used up by video sessions very quickly without obtaining high reward. That’s why the reward in the optimal scheme does not change


300 250

Total

20

WLAN

Expected Reward

Average Session Number

25

WiMAX Network

15

Cellular Network .

10

5

200 150 100 50

Optimal Scheme Existing Scheme

0 0

20

40

60

80

0 2

100

Time (min)

Fig. 3.

2.4

2.8

3.2

3.6

4

VoIP User Arrival Rate

The average number of sessions along the time.

Fig. 5.

The reward comparison with different VoIP session arrival rate ν2 .

R EFERENCES Expected Reward

250

200

150

100

50

Optimal Scheme Existing Scheme

0 0

0.4

0.8

1.2

1.6

2

Video User Arrival Rate

Fig. 4.

The reward comparison with different video session arrival rate ν1 .

dramatically. With different ν1 , the optimal scheme performs much better than the existing scheme. The situation is different in Fig. 5, in which we assume variable ν2 and constant μ = 0.2 and ν1 = 1.6. Increasing ν2 with a fixed ν1 is equivalent to increasing the proportion of VoIP sessions, which consume less resource. Thus the reward increases for both schemes. We can observe from the figures that the optimal scheme improves the reward significantly. VI. C ONCLUSIONS AND F UTURE W ORK In this paper, we have proposed an optimal network selection scheme in heterogeneous wireless networks considering multimedia application layer QoS. An application layer parameter, intra-refreshing rate, is adapted in heterogeneous wireless networks. The integrated network is modeled as a restless bandit system. We have presented an indexable optimal network selection policy. Simulation results were presented to show that application layer QoS has impact on the system performance, and the proposed scheme can improve the performance significantly. Network selection is a complex procedure, in which a number of factors should be considered in practice. In this paper, we consider application layer QoS and price of different networks. It is interesting to consider other factors in our framework to design an indexable network selection schemes.

[1] ETSI, “Requirements and architectures for interworking between HIPERLAN/3 and 3rd Generation cellular systems,” Tech. Rep. ETSI TR 101 957, Aug. 2001. [2] 3GPP TS 23.234, v.6.2.0, “Group services and system aspects; 3GPP systems to wireless local area network (WLAN) interworking; system description (release 6),” Sept. 2004. [3] E. Gustafsson and A. Jonsson, “Always best connected,” IEEE Wireless Commun., vol. 10, pp. 49–55, Feb. 2003. [4] Q. Song and A. Jamalipour, “Network selection in an integrated wireless LAN and UMTS environment using mathematical modeling and computing techniques,” IEEE Wireless Commun., vol. 12, pp. 42–48, June 2005. [5] F. Bari and V. Leung, “Automated network selection in a heterogeneous wireless network environment,” IEEE Netw., vol. 21, no. 1, pp. 34–40, Feb. 2007. [6] F. Yu and V. Krishnamurthy, “Optimal joint session admission control in integrated WLAN and CDMA cellular networks with vertical handoff,” IEEE Trans. Mobile Computing, vol. 6, no. 1, pp. 126–139, Jan. 2007. [7] D. Niyato and E. Hossain, “A noncooperative game-theoretic framework for radio resource management in 4G heterogeneous wireless access networks,” IEEE Trans. Mobile Comput., vol. 7, pp. 332–345, Mar. 2008. [8] X. Gelabert, J. Per´ez-Romero, O. Sallent, and R. Agust´ı, “A markovian approach to radio access technology selection in heterogeneous multiaccess/Multiservice wireless networks,” IEEE Trans. Mobile Comput., vol. 7, no. 10, pp. 1257–1270, Oct. 2008. [9] P. Whittle, “Restless bandits: activity allocation in a changing world,” in A Celebration of Applied Probability (J. Gani, ed.), vol. 25 of J. Appl. Probab., pp. 287–298, Applied Probability Trust, 1988. [10] D. Berstimas and J. Ni˜no-Mora, “Restless bandits, linear programming relaxations, and a primal dual index heuristic,” Operations Research, vol. 48, no. 1, pp. 80–90, 2000. [11] J. L. Ny and E. Feron, “Restless bandits with switching costs: Linear programming relaxations, performance bounds and limited lookahead policies,” in Proc. 2006 American Control Conf., (Minneapolis, Minnesota), pp. 1587–1592, June 2006. [12] H. Zhu and I. Chlamtac, “A call admission and rate control scheme for multimedia support over IEEE 802.11 wireless LANs,” Wireless. Netw., vol. 12, pp. 451–463, July 2006. [13] A. I. Elwalid and D. Mitra, “Effective bandwidth of general Markovian traffic sources and admission control of high speed networks,” IEEE/ACM Trans. Netw., vol. 1, no. 3, pp. 329–343, Jun. 1993. [14] H. Holma and A. Toskala, WCDMA for UMTS: Radio Access for Third Generation Mobile Communications. NY: Wiley, 2004. [15] Z. He, J. Cai, and C. Chen, “Joint source channel rate-distortion analysis for adaptive mode selection and rate control in wireless video coding,” IEEE Trans. Circ. Sys. Video Tech., vol. 12, pp. 511–523, June 2002.