Delay-Aware Resource Provisioning for Cost-Efficient ...

1

Delay-Aware Resource Provisioning for Cost-Efficient Cloud Gaming Mohaddeseh Basiri and Abbas Rasoolzadegan Ferdowsi University of Mashhad Email: [email protected], rasoolzad [email protected]

Abstract—Real-time online gaming via thin clients, which is known as cloud gaming, connects players around the world and let them play high-quality games without need of installing the games or having much processing capacity. Successful deployment of any leader cloud gaming solution requires an ultra low-delay cost-efficient design. Towards this end, it is required to investigate how one can minimize the cloud costs while satisfying the quality of experience (QoE) requirement of users. Here, we aim at presenting a resource allocation framework for cloud centers which benefits from highly virtualized CPU/GPU resources for cost efficiency. Towards this end, we first present an accurate delay model, as the main control parameter for QoE, by considering all sources of delay into account. Then, we present a delay-aware cost-minimizing resource allocation scheme, and follow a tractable approach to derive the performance bounds on blocking probability of the system. The simulation results show that the performance of our proposed schemes outperforms the others in cost efficiency while satisfying the delay requirement of the connected users.

or virtual machine (VM) to run the user requests. So, the game server or VM takes over rendering and streaming encoded game scenes to the user. However, it is very challenging to provide a cost-effective cloud gaming platform with high quality of experience (QoE) perceived by users. Some challenges are as follows: •

Index Terms—Cloud gaming, resource provisioning, performance analysis, blocking probability, queuing theory.

I. I NTRODUCTION New advances in communication networks and cloud technology would make the cloud gaming more real in near future. In recent years, cloud gaming has attracted a lot of attention from both industry and researchers, such that the cloud gaming market is expected to rapidly expand and reach $8 billion sales by 2017 [1]. Several startup companies offer well-known cloud gaming platforms like OnLive [2], Gaikai [3], G-Cluster [4], and StreamMyGame [5]. Cloud gaming is a different type of online gaming services, in which a 3D video game is remotely rendered in possibly far-away data centers and the screen of the game as 2D video is streamed to users in real-time over the Internet infrastructure [6, 7]. Cloud gaming enables users with low processing and rendering ability, e.g. tablets and smart-phones holders, to play high-quality games only by utilizing a good Internet connection link. Gamers are allowed to play the games without downloading and installing any games softwares. Also, there is no need for users to upgrade their hardware constantly. So, they are enabled to play more games by decreasing hardware and software costs [8]. Cloud gaming service providers utilize geographically distributed data centers to present their services to customers. After receiving user requests by cloud gaming platform, the user requests would be dispatched to a specific data center based on the predefined strategy and allocated a game server

•

•

•

Cloud gaming is more interactive and sensitive in comparison with video-on-demand services [9]. Also, online gaming players are pretty impatient and so sensitive to the interaction delay [10]. So, the key requirement for successful migration of gaming to the cloud is providing extremely low-delay services. The inevitable delay in any type of current Internet connection, the massive concurrent gaming actions which must be rendered at the data-centers, and the high volumes of rendered frames which must be transfered in real time to the players, make the delay as the major bottleneck which must be addressed in cloud gaming resource allocation. Different games genres are impacted differently by interaction delay [11], e.g. increasing interaction delay isn’t tolerable in the first-person shooter (FPS) games, while the war strategy games players aren’t so sensitive for the same interaction delay. Thus, the difference among games genres must be considered in the resource allocation scheme of cloud gaming. Cloud gaming services include different remarkable costs, such as coordination cost for keeping consistency among different users of a multi-player game allocated in different physical machines (PMs), bandwidth cost for streaming video game frames, allocated VMs costs, and etc. Hence, to provide good-enough QoE for users and survive in the market competition of cloud gaming, servicing costs must be minimized as much as possible. Resource sharing and management are important issues in cloud gaming context. In existing virtualization approaches, resource sharing is weakly considered, i.e. most cloud gaming service providers run multiple replicas of a game, such that one GPU are allocated for each instance of the game while very little or even no GPU computation resource may be required [12]. In high performance computing domain, virtualization technology can significantly improve the way CPU and GPU resources are used and managed in a cloud data center, and it has been developed dramatically in the past few years [12]. Regarding the need for extremely low-delay cost-efficient design for

2

cloud gaming systems, high level of virtualization is a promising option for cloud gaming platforms to prevent the waste of hardware resources. However, the impact of high level CPU/GPU virtualization and its corresponding resource allocation is not studied in literature and is the focus of this paper.

VI, the blocking probability of the system is investigated. The simulation results and the concluding remarks are respectively presented in section VII and VIII.

As mentioned above, delay is the major bottleneck in deployment of cloud gaming systems. In order to address this problem, one must analyze delay in different stages of cloud gaming, e.g. game-start request arrival, resource allocation to users, game rendering, packet transmission, and etc., which are the focus of this paper. Here, we first present an accurate model for delay performance of cloud gaming systems which considers all sources of delays which might happen in a real world scenario. Then, we present resource allocation schemes which meet the delay demand of the users in single- and multiplayer games with respect to cost-efficiency and perceived good-enough QoE by users. To answer the resource sharing challenge, we utilize a separate sub-model for PMs with a high degree of virtualization to control multi-party gaming operations [13]. Furthermore, we investigate the performance of cloud gaming systems by means of instantaneous and longterm blocking probability. Performance evaluation shows that the proposed resource allocation schemes significantly impact the experienced delay for the players. This work provides insights to delay-aware cost-effective cloud-system design. The main contributions of this work include:

Broadly speaking, the available research works on cloud gaming can be categorized in 3 groups: (i) measurement-based studies; (ii) cloud-gaming quality of experience studies; and (iii) dynamic cloud resource provisioning studies. To better understand the performance of the cloud gaming systems, some measurement works have been focused on the architecture and performance of existing cloud gaming systems. Authors in [14] evaluated the response latency of two cloud gaming platforms, specifically OnLive and StreamMyGame. They determined the response delay by utilizing hooking mechanism in Windows to inject their instrumentation code into the clients. In [15], a systematic analysis of in-market cloud gaming platforms is performed. Also, the framework design of these platforms and the real-world performance of them are investigated in [15]. An empirical network traffic analysis of On-Live and Gaikai cloud gaming platforms is performed in [16]. Also, some researches have been conducted to recognize the impact of different game genres on user experience in the concept of cloud gaming. In [11], a qualitative comparison of distinct genres, including action, puzzle, strategy, and racing games, is investigated based on a common testing methodology that combines both objective and subjective approaches. The analysis results of [11] indicate that interaction games, e.g. action and racing games, are more sensitive to the delay in comparison with puzzle and strategy games. In [17] analyzed network traffic results for various games types to understand the relation among application level parameters, e.g. game genre and input characteristics, generated video characteristics, and network level characteristics. The results in [18] presented that the same latency may have very different impacts on quality of experience based on the game’s real-time strictness. Also, the model developed in [18], simultaneously augmented player’s gaming experience and minimized the operation cost of data centers. To perceive dynamic resource provisioning in cloud gaming services, one must focus on understanding and mitigating the latency problem. [19] mainly focused on maximizing the service provider’s revenue by satisfying pre-defined QoS requirements, and simultaneously cutting down the resource cost in the cloud, but the resource allocation is only considered in a single data center. [20] investigated dynamic resource provisioning problem in massively multi-player online games (MMOGs), but the considered scenario is different from cloud gaming. The purpose of study in [21] is to acquire a strategy for distributing games to the servers to minimize the number of misses and failures and enhance the disk resources usage. In this work, each VM can only host one subset of game replications. VM placement problem in cloud gaming for maximizing the total net profit while maintaining the QoE in good level has been studied in [22]. In this work, the bandwidth allocation problem is not investigated and the operational cost

•

•

•

•

Develop a cloud structure for single- and multi-player games. Introduce two types of VMs in each PM for serving players and performing game-related actions. This model offers a better utilization of the cloud resources, supports high degree of virtualization, and decreases the operational expenses as well as the processing delay. Present a delay-aware cost-effective framework with respect to the perceived QoE by users for resource allocation in cloud gaming systems. Present an accurate Round Trip Time (RTT) delay model for cloud gaming which takes into account different sources of delay, including transmission, propagation, waiting, and processing delays. Furthermore, consider different costs, e.g. coordination cost, bandwidth cost, and VMs running cost, and different games genres in solving resource allocation optimization problem. Follow an analytical and tractable approach for performance evaluation. Investigate the introduced delay in the game start process, i.e. when the resource allocation module allocated the arriving user to the best delay-aware cost-effective data center, and inside the game, when the allocated data center serves the user using its back-end VMs. Derive closed-form expressions which can well predict the blocking probability of the system, and hence, is beneficial for performance evaluation in the design phase.

The reminder of this paper is organized as follows. In the next section, the related works of cloud gaming area is reviewed. In section III, the system model is introduced. The resource allocation problem is provided in section V. In section

II. R ELATED W ORKS

3

Fig. 1: The steps of game handling and servicing on a cloud gaming. Different colors in the left queue present different groups of players.

of cloud gaming service provider is assumed to be proportional to the CPU and GPU utilization of its servers. However, this assumption is only valid for proprietary cloud systems and different pricing strategies must be considered for public cloud systems where the operational cost is dependent on the running time rather than the utilization time [23]. In [9] the authors proposed an online control algorithm for a cloud gaming system over a single data center in which all users seek the same services requirements to cut down the provisioning cost while maintaining user QoE requirements. Later in [7], this concept has been extended to the cloud gaming systems equipped with geographically distributed data centers where each data center keeps a replica of all games and a given user can play its favorite game on any data center. Also, the heterogeneity of the service requirements among users is considered in [7]. However the proposed schemes in [9] and [7] aim at offering a tradeoff between operational costs of a system and the user QoE, the GPU/CPU resource virtualization and its impact on the operational costs of a cloud gaming system as well as the QoE of the users is missing in literature. In [7, 9], it is assumed that after initial game setup, each user is connected to a virtual machine for running its requested games. In this case, we might have several over-utilized and several underutilized (or even idle) VMs, and hence, this heterogeneity of load distributing can significantly impact on the QoE of the users. Then, it is reasonable to assume two types of VMs for running the games, and the other for performing game related processing [13]. Each VM from the former type will be connected to at most one user while each VM from the latter type will be utilized in turn by different users for processing the game actions. III. T HE P ROPOSED S YSTEM M ODEL In this section, we consider a model for servicing and handling cloud gaming users in single- and multi-player games, as shown in Fig. 1. We assume that cloud receives game-start requests from groups of users. In each group, there is a single user or a number of users intending to play a game together. The users in each group, and hence in the cloud, are potentially from

different places around the world who are connected through the Internet infrastructure and are equipped with thin clients, e.g. tablets and smart-phones. Also, the games are run and rendered on a number of data centers around the world and then, the game frames are streamed to the users. In the sequel, dispatching users requests to the cloud gaming platform and serving game actions among different data centers are represented.

A. Serving game-start requests from Cloud Users The game-start requests are submitted to a global infinite queue which processes the requests on the first-in, first-out basis (FIFO). Once a game-start request places at the head of the queue, the resource allocation center (RAC) tries to connect the group users to the available data centers in the cloud. In practice, dispatching game-start requests may happen in a multi-queue fashion, i.e. different types of users are served based on their priorities [24]. For example, the game servers may consider priority lines for logged-in users, requests with a short remaining delay budget, or premium users which have either paid for gaming or collected points in previous game sessions. Also, in some cases the head-of-queue SA, which is to be processed at first in FIFO manner, may be able to tolerate a higher delay than a newly arrived request. In this case, more advanced queue processing schemes can be used for request dispatching, i.e. Shortest-Remaining-DelayBudget-First (SRDBF) protocol [25]. In order to keep the problem tractable and derive closed-form performance results, here we consider the FIFO queuing for request dispatching. One sees in Fig. 1 that each group of users is shown by a different color and is named as a super-arrival (SA). Each super-arrival contains game-start requests from different numbers of users which is independent from other SAs. Let us denote the SA size by n, defined as the the number of game-start requests from different users demanding to play a specific game together. It is clear that n would be 1 in the case of single-player games and a geometrically distributed random variable with parameter p in the case of multi-player

4

User 2

User 1

...

of back-end VMs as servers to serve the front-end VMs and game actions of the users which are attributed to this PM. If RAC decides to connect the ith user ui to the kth physical machine P Mk , a host VM is allocated to the user ui to run the game and game server. Then, ui starts sending control messages to P Mk and waits for screen streaming from it. The virtual connection between this pair will continued to the end of the game without any resource re-allocation. To serve a SA, RAC may decide to connect a user to a PM which has no free capacity but can turn on a new back-end VM. In this case, we need to define a cost model for the cloud by considering different sources of expenditure and delay, and design a resource allocation scheme to minimize the cloud cost. If the RAC can not find a PM that has free capacity or is not able to turn on a new back-end VM, it will block the game-start request.

User N

Front-end VMs

Host VM 1

Host VM 2

...

Host VM N

request scheduler Back-end VMs

VM 1

VM 2

VM 3

VM 4

VM 5

...

VM Y

IV. M ODELING T HE D ELAY AND C OST OF THE C LOUD G AMING Fig. 2: Virtualization of cloud gaming resources sub-model in P Mi

In the following sub-sections, the delay and cost of cloud gaming systems are modeled.

games. Therefore, the probability of having a SA with size i is calculated as

A. Delay Model for Cloud Gaming

Pr (n = i) = (1 − ps )(1 − p)

i−1

p + ps δ,

(1)

where ps is the probability of receiving a single-player gamestart request, δ is the Dirac delta function [26], and p is the success probability in geometric distribution. ps and p are assumed to be a prior known from the statistics of the cloud-gaming system. If a super-arrival waits more than a prespecified time in the global queue, it will be blocked to end the unpleasant delay. The super-arrival which is at the head of the global queue (SAhq ) is processed by the resource allocation center to allocate the available game rendering resources among the SAhq ’s users. By considering different sources of delays in cloud-based systems and costs of the cloud, the RAC searches for a suitable resource among cloud’s physical and virtual machines and finds the best match for each user.

B. Serving Game Actions from Cloud-Gaming Players To make virtualized game engine more efficient, we adopt the proposed virtualized model in [13] for cloud gaming resources. As illustrated in Fig. 2, our model divides virtualization into two levels of front-end and back-end. In each PM, front-end virtual machines are employed to run the game server and directly interact with users, while back-end virtual machines process game-related computing. N users are provisioned with N host virtual machines in the front-end and each host virtual machine runs the game server of the assigned user. Provisioning of host VMs just take place for each user at the beginning of playing. After this assigning, the backend VMs will be employed to serve to the users’ actions. We consider one FIFO queue in each PM with adequate number

In cloud gaming, one of the most important things that must be considered is experienced delay by users. In this paper, we focus on Round Trip Time (RTT) delay which is influenced by some factors, e.g. propagation delay, transmission delay, waiting delay in queue, and processing delay in PM. If PM i serves user j, we formulate the RTT delay for this user as follows: djRT T = duser−cloud + dqueue + dcloud−user + dvmf ,

(2)

where duser−cloud = dutrans + dpropagation dqueue = dwaiting + dprocessing dcloud−user = dctrans + dpropagation Dji dpropagation = Clight Lu Lc dutrans = j ; dctrans = j . Rul Rdl j j In these expressions, Rul and Rdl are the uplink and downlink i data rate of user j, Dj is the distance between user j and PM i, dutrans the transmission delay for user’s data sent to the cloud, dctrans the transmission delay for cloud’s data sent to the user, Lu the user’s control-packet length which is transmitted to cloud, and Lc is the cloud’s screen-packet length which is streamed to user. Also, Clight is the speed of light, and dwaiting and dprocessing are the waiting and processing times for serving a game action at its respective PM. Furthermore, dvmf is a constant and presents processing delay in the frontend VM. Since in cloud gaming, the game is running in the cloud center, not in the user’s computer, the user only sends short payload size control packets which results in negligible transmission delay, dutrans . Motivated by the fact

5

that a longer distance between user and cloud introduces a larger number of intermediate routers in between, one may imagine the case in which the uplink and downlink traffics be routed in different paths, which in turn implies that the experienced delays in the respective communications might be different. This means that the propagation delay which is assumed in (2) to be the same for uplink and downlink communications, might not capture this scenario. However, regarding the low contribution of the propagation delay in the total experienced delay due to the enormous improvement of the wired and wireless communications, and the low impact of the difference between uplink and downlink routs, this difference is usually neglected in literature [27, 28]. To satisfy the Quality of Experience (QoE) for each user, we design resource allocation schemes which guarantee that the derived round trip time delay expression in (2) will be less than the threshold delay.

B. Cost Model for Cloud Gaming In cloud centers, there are two types of resources which serve the users: reserved and on-demand resources [29]. At the beginning of service provisioning, service provider determines a certain number of VMs in each PM as the reserved resources. In the resource allocation process, if the reserved VMs can not satisfy the service request, the on-demand VMs can be instantly allocated by RAC, however, one must note that the service provider charges differently for the on-demand ones [29]. Also, to serve users, each PM needs to transmit game j frames to the users with a satisfactory data rate, e.g. Rdl in (2) for user j. The data rate itself is a function of bandwidth, j Rdl = θWi,j , where Wi,j = F(Wi , vi,f , QoE(i))

(3)

is the bandwidth share of user j at P Mi , θ in bits is a constant, and Wi is the total available bandwidth in P Mi . Also, F shows the bandwidth allocation function, where in the general form, it is a function of total available bandwidth Wi , and the number as well as the QoE requirements of the connected users, i.e. vi,f and QoE(i) respectively. Furthermore, consider a group of users who intend to play together. If we serve these users in different PMs, it introduces a new cost for maintaining consistency among the games of players which requires communications among the centers who render the game. Let refer to this cost as the coordination cost. To optimize the resource allocation problem, we choose the change in the total cost of the cloud (∆C) after resource allocation for SAhq as the objective function to be minimized. Based on the above cost descriptions, one can define the change in the total cost of the cloud as follows: m ∑ ∆C = ∆cost(i) + Cco , (4) i=1

where ∆cost(i) is the change in the cost of P Mi after resource allocation for SAhq . To formulate ∆cost(i), we consider all cost sources in the cloud, e.g. the costs due to the bandwidth

amount and turning on and maintenance of back-end and frontend VMs in each PM. ∆cost(i) =Ww ∆costbw (i)+ Wb ∆costvmb (i) + Wf ∆costvmf (i)

(5)

where ∆costbw (i), ∆costvmb (i), and ∆costvmf (i) represent change in the bandwidth cost of P Mi , change in the back-end VMs costs of P Mi , and change in the front-end VMs costs of P Mi , respectively. Also, Ww , Wb , and Wf represent the corresponding weights, respectively. max Let us assume vi,x shows the maximum number of type x max VMs in P Mi , vr,i,x the maximum number of reserved VMs of type x in P Mi , vi,x the number of active type x VMs before ′ serving SAhq , and vi,x the number type x VMs after serving SAhq , where x ∈ {b, f } demonstrates the type of a VM, and b, f stand for back-end and front-end VMs respectively. Also, ci,x and ci,x + cdi,x are the costs for turning on a reserved and on-demand VM respectively where cdi,x models the penalty cost for utilizing an on-demand VM. Then, one can formulate the change in the cost of back-end or front-end VMs in the P Mi as ∆costvmx (i) =  [( ′ ) ] ) ( max + max + d  ci,x vi,x − vr,i,x − vi,x − vr,i,x ′ +ci,x (vi,x − vi,x )   ∞

(6) ′

max if vi,x ≤ vi,x o.w.,

in which (z)+ means max{z, 0}. Also, the amount of change in the bandwidth cost of P Mi can be denoted as Cbw ∆Wi , where Cbw represents the unit bandwidth cost, and ∆Wi shows the change in the bandwidth utilization of P Mi after serving the SAhq . Furthermore, Cco in (4) is the coordination cost for keeping the game of SA’s users consistent in different PMs. Since Cco is highly application dependent, it is hard to derive a general model for it. Here, we model the coordination cost as a linear function of the number of PMs which are allocated to the users of a SA, N , and coordination cost unit, Ccu , i.e. Cco = Ccu N . The coordination cost unit is also application dependent and varies from a network to another. To let the analysis to be tractable and derive closed-form results, in the following section, we solve the resource allocation optimization problem under three assumptions for the Ccu : (i) high coordination cost, i.e. when the ratio between Ccu and ci,b +cdi,b is very high, (ii) moderate coordination cost, i.e. when the ratio between Ccu and ci,b + cdi,b is close to one, and (iii) low coordination cost, i.e when the ratio between Ccu and ci,b +cdi,b is close to zero. One must note that in the case of cloud systems with single player games, the SA size will be 1 and the resource allocation problem can be solved using the results of resource allocation with high coordination cost. V. D ELAY-AWARE R ESOURCE A LLOCATION S CHEME In this section, we investigate delay-aware resource allocation for minimizing the operational cost of the cloud. Assume the cloud gaming system is in use and SAhq , the request from a group of users which is at the head of the global queue, is

6

ready to be served. There are total m available PMs in the cloud and each of them is currently serving the game actions of a number of players. Here, the problem that must be tackled is determining the optimal PM for serving each user of the SAhq subject to the delay constraint, i.e. the resulted RTT delay must be less than the threshold delay, dth . Also, one must note that the number of reserved and the total number of VMs in each PM are limited and allocate the resources to the users such that the operational costs of the network be minimized. Based on these descriptions and using the delay and cost models of previous section for cloud gaming systems, we can formulate the resource allocation problem for cloud gaming systems as follows: m ∑ min ∆cost(i) + Cco (7) i=1

subject to: ′

max vi,x ≤ vi,x

, ∀i ∈ {1, · · · , m}

djRT T

, ∀j ∈ {1, · · · , n}.

≤ dth

To mathematically tackle this problem, we use the three mentioned models in section IV-B for the coordination cost, i.e. high, moderate, and low coordination cost models. In the sequel, the solution to the optimization problem in (7) is presented based on the coordination costs assumptions in three subsections. A. High Coordination Cost Model The first solution for the optimization problem in (7) comes from high coordination cost assumption, i.e. Cco is much higher than the costs of reserved and on-demand back-end VMs. In this case, if the SA users are allocated to different PMs, the users’ actions would be sent to different locations which results in much more complexity to keep consistency in the game. To eliminate the high cooperation cost, it is meaningful to allocate all SA’s users to one PM to force the Cco term in (7) to zero. This assumption puts an integrity constraint on the resource allocation problem, i.e. a PM either will have enough capacity to accept and serve all the users of a SA or it will reject the SA completely. As mentioned, the size of SAhq is denoted by n which means it conveys the game-start requests from n users. Let us denote λu as the arrival rate of the game actions for each user at cloud gaming system. For P Mi from m available physical machines, the total game-action arrival rate before resource allocation for the SAhq , service rate, and the number of servers are denoted by λi , µi , and vi,b , respectively. Denote the index of the optimal PM for serving SAhq by i∗ . Our aim here is to find which PM can serve the SAhq with the lowest cost while the threshold delay is satisfied, i.e., i∗ = arg min ∆cost(i)

(8)

i=1:m

s.t.: djRT T ≤ dth

, ∀j.

To solve this problem, first we need to find the change in the cost of each PM in the case of serving SAhq . Then, we must find how many back-end and front-end VMs must be turned on

to keep djRT T less than dth in each PM. Based on our model, we must turn on one front-end VM for each user of SAhq . The main problem that one can be faced is the number of back-end VMs to serve the SAhq . To mathematically tackle this problem, we must model the queuing delay, i.e. dqueue in (2). The part of PM which is responsible for processing game related actions can be modeled as an M/M/s queuing system. When we model the processing time with an exponential distribution, the mean processing time for a game action in the backend virtual machines is required. For deployed cloud systems, one may use packet tracking softwares like WireShark1 to track the transmitted and received game actions/responses and estimate such parameter, or one may use the techniques provided in [30], to investigate the average processing time. One must note that our main aim in the paper is to provide a deep insight to the system performance in the design phase, and hence, the average processing time is regarded as a tuning parameter in the present work. In light of above discussion, dqueue can be derived from M/M/s queuing [31] as follows: dqueue =

Pq (s, µλii ) sµi − λi

+

1 , µi

(9)

where Pq is the probability that an arriving request is forced to join the queue, i.e. all servers are busy, and is derived as Pq (s,

λi 1 )= ∑s−1 s! µi 1 + (1 − ρi )( (sρi )s ) k=0

(sρi )k k!

,

(10)

λi where ρi is the server utilization, i.e. ρi = sµ . Then, to find i the minimum required back-end VMs, which must be turned on to satisfy the threshold delay constraint, the following equation must be solved for the set of critical nodes, i.e. G, ′

dth =

u Pq (vi,b , λi +nλ ) µi

Dgi 1 Lu +2 + g (11) Clight Rul vi,b µi − (λi + nλu ) µi Lc + + dvmf , θF(Wi , n + vi,f , {QoE(i), QoE(SAhq )}) ′

+

where g ∈ G. We see that solving the above equation needs knowledge about the bandwidth allocation functions, i.e. F. One must note that bandwidth is a major source of operational costs in a network, and hence, sophisticated bandwidth allocation schemes might be implemented in order to satisfy the QoE of the connected users with a limited set of resources. The interested reader may refer to [32, 33], where cloud bandwidth and bitrate allocation functions have been proposed to minimize the total amount of utilized cloud resources, while satisfying the QoE of the users. In order to see how the required number of virtual machines must be scaled with the number of arriving users, one may simplify the equation in (11) by utilizing an equal bandwidth allocation scheme, i.e. by using: F(Wi , n + vi,f , {QoE(i), QoE(SAhq )}) =

Wi , n + vi,f

which in turn results in an interesting equation that shows the scalability of the whole system, as follows: 1 https://www.wireshark.org

7

Lc y = Z− θ Wi

1 ∑x−1 ( µii )k λi x! k=0 k! ) xµi )( ( λi )x ) λ

(x µi − λi )(1 + (1 −

,

µi

(12) where Z is a constant and is defined as: Z = dth −

B. Moderate Coordination Cost Model For the second solution of the optimization problem (7), we assume that the coordination cost is comparable with the costs of the reserved and on-demand back-end VMs. Here, we model the coordination cost as a linear function of the number of PMs which are allocated to the users of a SA, N , and coordination cost unit, Ccu , i.e. Cco = Ccu N . With this assumption, each PM serves k (0 ≤ k ≤ n) users of the SAhq . In the following, the problem mentioned in previous sub-section is solved by considering new assumption. To solve the problem, first we need to find the minimum required back-end VMs for each PM, which must be turned on to satisfy the threshold delay constraint and serve k (0 ≤ k ≤ n) users of the SA. As mentioned in the previous subsection, the following equation must be solved for the set of critical nodes, i.e. G: dth =

′

+

Dgi 1 Lu +2 + g µi Clight Rul

vi,b,k µi − (λi + kλu ) Lc + + dvmf , θF(Wi , vi,f , {QoE(i), QoE(SAhq )}) (13) ′

where g ∈ G, and vi,b,k is the required back-end VMs to serve k users of the SAhq . Then, the change in the cost of back-end VMs in P Mi to serve k users of the SAhq is calculated: ∆costvmb (i, k) =  [( ′ ( ) ) ] d max + max +  − vi,b − vr,i,b ci,b vi,b,k − vr,i,b ′ +ci,b (vi,b,k − vi,b )   ∞

m ∑

( ) ∆cost i, R(i) + Ccu I(R)

(16)

subject to:

In (12), x denotes the number of active back-end VMs and y denotes the number of connected users, where the latter is also equal to the number of active front-end VMs. By solving equation (11), we can find the minimum required ′ back-end VMs, i.e. vi,b , for P Mi that can be equal or more than vi,b . Then, the change in the cost of P Mi to serve the SAhq can be calculated by (5). Finally, the best PM to assign to SAhq is derived from (8).

′

R∗ = arg min

i=1

Dgi 1 Lu −2 − g − dvmf . µi Clight Rul

u Pq (vi,b,k , λi +kλ ) µi

users of the SAhq . Vector R has m rows where the entry in the ith row represents the number of users from the SAhq allocated to the P Mi . Then, the problem is transformed to find the optimized vector R:

(14) ′

m ∑ ( ) R(i) = n i=1

R(i) ∈ N, where R(i) represents ith element of vector R and I(R) shows the number of non-zero elements in vector R. Therefore, the best PMs would be assigned to the users of the SAhq . C. Low Coordination Cost Model The third solution for the optimization problem (7), comes from low coordination cost assumption, i.e. we assume that the coordination cost is much lower than the costs of the reserved and on-demand back-end VMs. In this case, one can consider each user of the SAhq as an independent SA that must be allocated to a suitable PM. Therefore, the resource allocation problem reduces to the resource allocation with high coordination cost assumption when the SA size is set to one, i.e. n = 1. Then, we solve the resource allocation with high coordination cost for each user of the SA separately, to accommodate SA user’s requests in available PMs. VI. B LOCKING P ROBABILITY F OR C LOUD G AMING To design a cloud system, we need the performance expectation of the system under different types of deployments and resource provisioning. In this section, we calculate the blocking probability of our cloud gaming system model. Blocking is occurred in our system under two conditions: (i) blocking due to the unpleasant delay for game-start request which may be occurred at the global queue, or (ii) blocking due to the insufficient capacity in the PMs. Therefore, the blocking probability is formulated as follows: Pbl = BPq + (1 − BPq )BPr ,

(17)

where BPq is the blocking probability due to the unpleasant delay for game-start request and BPr is the blocking probability due to the insufficient resources and PMs. In the next step, we will calculate each of BPq and BPr , separately.

max if vi,b,k ≤ vi,b A. Blocking Probability Due To The Unpleasant Delay o.w. To determine the BPq , the RAC is modeled as an M/G/1 queuing system, which the SA arrival rate, service rate, and Then, the change in the cost of P Mi to serve k users of the size are denoted by λ, µ, and n, respectively. As mentioned SAhq is considered as follows: before, the SA size, n, has geometric distribution with expected value p1 , in which p is the success probability. If the ∆cost(i, k) =Ww ∆costbw (i)+ service time to serve a user is considered by a very short Wb ∆costvmb (i, k) + Wf ∆costvmf (i). (15) time, t, the service time would be a geometrically distributed Finally, to solve the optimization problem, we consider a random variable with expected value pt . If t is assumed very vector Rm×1 to determine that each PM serves how many short, discrete time geometric distribution can be approximated

8

with continuous time exponential distribution with parameter 1 t µ = p [34]. Therefore, the RAC can be modeled as an M/M/1 queuing system with SA arrival rate λ and service rate µ = pt . Finally, the BPq is the probability that the delay of the M/M/1 queuing system is more than the predefined waiting time to start the game. By supposing z as the delay of global queue, Dus as the maximum delay accepted to start the game, and by regarding to the calculations of [31] for computing the probability density function (PDF) of delay of M/M/1 queue, BPq can be formulated as follows:

where M is a subset of PMs that includes P Mi if Npi ≥ n holds. Also, En {.} shows expectation over n, n is the SA size, and (a) is due the fact that n follows a geometric distribution with parameter p. Now, we must calculate Npi and the PDF i i i of Nth , i.e. prob(Nth = j). As we know, Nth = Npi − Nai , i i where Np and Na are the maximum number of users that can be served by queue i and the number of currently active users in queue i respectively. Thus, the following holds:

∫∞

By assuming homogeneous users with request rate of λu , Npi is calculated as

BPq = prob(z > Dus ) =

f (z)dz = e

−( pt −λ)Dus

, (18)

i prob(Nth = j) = prob(Nai = Npi − j).

max , Nimax }, Npi = min{vi,f

Dus

where f (z) = (µ − λ)e−(µ−λ)z , λ, and µ = pt are respectively the PDF of delay, total arrival rate, and service rate.

where Nimax is found by solving the following equation: dth =

B. Blocking Probability Due To The Insufficient Resources To calculate BPr , we must assume three scenarios for the coordination cost, i.e high, moderate, and low coordination cost. In the following, we only investigate the case of high coordination cost because deriving closed-form analytical performance expressions for low- and moderate coordination cost cases is highly complicated, if not impossible. One must note that the high coordination cost case also covers the singleplayer games, e.g. angry birds 2 , which constitute a big portion of all available games. As mentioned in section V-A, by considering the high coordination cost assumption, all users of the SAhq must be allocated to one PM to force the Cco term in (7) to the zero. Therefore, the blocking probability of SAhq ’s users requests is equal to the probability that there are no capacities in PMs to serve all users of the SAhq entirely. In other words, the capacity of a PM must be sufficient to serve and accept either all users of the SAhq or reject them totally. i To calculate BPr , let us assume that Nth is the extra amount of users that the ith PM can tolerate without exceeding its delay from the threshold delay, and Npi is the maximum number of users that PM i can serve simultaneously. One i sees that Npi and Nth are deterministic and random variables respectively. Now, BPr can be defined as: BPr = prob(max{Npi } < n) i

+

prob(max{Npi } i

≥ n)En

{( ∏

i prob(Nth

)} < n) ,

= prob(max{Npi } < n) + prob(max{Npi } ≥ n) i

j=1

∏

i prob(Nth

) } < j) prob(n = j) ,

i∈M

( )(maxi {Npi }) ( )(maxi {Npi }) = 1− 1−p + 1−p maxi {Npi } {( ) } ∑ ∏ i j prob(Nth < j) (1 − p) p ,

(a)

j=1

i∈M

(19) 2 https://www.angrybirds.com/

+

1 + µi

vi,r µi − ˜ D Lu Lc 2 + + Wi + dvmf , ˜ ul Clight R N max Nimax λu

(21)

i

= k) =

c ∑

prob(Nai = j)

j=0

i

∑

max Pq (vi,r , Nimax λµui )

˜ ul and D ˜ are the worse case average uplink data where R rate and communication distance over the set of potential connected users. After deriving Npi , one can find the PDF i of Nth , by computing the PDF of Nai . To this end, we need to derive the probability of having k simultaneous users connected to the queue, and hence, the game length must be known. Generally speaking, the game length can be modeled as a random variable with general distribution. The exponential distributions family are usually used for modeling events’ lifetimes [35]. For example, in the case of Angry Birds game, where the game termination depends on satisfaction of the user or his failure, the game length can be modeled as an exponentially distributed random variable. In some games, the game length is a constant value, e.g. in the case of time-constrained games like Counter-Strike game 3 . So, by assuming that the game length is a constant value T , the users presence (not users requests) in PM can be modeled using M/D/c queuing system, where the active users are served in parallel by c servers, and c ≫ Npi . Now, the probability of simultaneous presence of k users in the queue can be found using the results of [36], i.e. by solving the following system of equations: prob(Nai

i∈M

maxi {Npi } {(

(20)

c ∑

(λip T )k −λi T e p , k!

prob(Nai = j) = 1.

(22) (23)

j=0 i is found as follows: Now, by using (22), the PDF of Nth i prob(Nth = j) = prob(Nai = Npi − j).

(24)

Also, by assuming an exponential distribution with parameter gl as the PDF of game length, the users’ presence in PMs can be modeled using M/M/c queuing system, where the active 3 http://www.counter-strike.com/

9

users are served in parallel by c servers, and c ≫ Npi . Now, the probability of simultaneous presence of k users in the queue can be found using the results of [37]: prob(Nai = k) =

1 c−1 ∑ j=0

(cρ)j j!

+

(cρ)c 1 c! 1−ρ

(cρ)k , k!

(25)

TABLE I: Simulation Parameters (Configuration 1)

General parameters

Value

Simulation area

Square with sides of length 2000 km 4 data center located on the edges of the simulation area each equipped with 1 PM Distributed according to a SPPP in the area 5

Data centers

λip gl

where ρ = c . Again, by using (24) one can find the PDF i of Nth from the PDF of Nai in (25). Finally, by having the i PDF of Nth one can compute the BPr from (19). VII. P ERFORMANCE E VALUATION In this section, we evaluate the system performance. To this end, serving cloud users which are randomly distributed according to a spatial Poisson point process (SPPP) in a square area with sides of length 2000 km with 4 data centers at the edges each equipped with 1 PM is simulated using MATLAB. The implemented cloud system follows the presented structure in Fig. 1. One must note that if the cloud system is regarded as a global system distributed over the earth, i.e. the data centers are distributed over the countries and users from all places try to connect, and for each new arriving user do the optimized resource allocation, the complexity of the proposed algorithm will be high. In practice, the request dispatching and resource allocation is done for a cluster of servers, which are placed in a geographically close region, e.g. in a country. This issue is similar to the new Cloud Radio Access Network (CRAN) paradigm in 5th generation of cellular networks [38], where it is proposed to do the access management and device scheduling in a unified cloud-based platform, to optimize the performance. However, as it introduces complexity issues, and the corresponding required computations is not manageable in real-time, clustering for nearby networks is considered, i.e. several nearby base-stations cooperate with each other. In our paper, we also consider the same trend, and consider a limited set of close data centers to be clustered and perform cloud-based access management together. In this case, we believe that the complexity is manageable in real time. To make the simulation model more realistic, the workload model obtained from the real online game traces in [39] is used in our simulations for modeling the game length. To this end, we assume that the game length follows a truncated Pearson distribution of type 5 with α = 0.76, β = 7.57, and maximum gaming length of 1000 sec. Also, our pricing strategies for bandwidth and virtual machines follows the same trends as in [7, 9]. The detailed simulation parameters can be found in Table I. As a benchmark, the performance of the proposed delay/cost-aware resource allocation (DCRA) scheme with high coordination cost assumption is compared against a proximity-aware resource allocation (PRA) protocol [9] which aims at assigning the arriving SA to the closest possible datacenter, and hence, minimizing the data transmission delay; and a load-aware resource allocation (LRA) protocol [9] which aims at assigning the arriving SA to the PM with the lowest traffic load, and hence, minimizing the queuing delay. Fig. 3 compares the delay performances of the proposed resource allocation schemes using the empirical cumulative

Users Number of reserved back-end VMs at each PM Total number of available backend VMs at each PM Number of reserved and available front-end VMs at each PM Available bandwidth at each data center Maximum tolerable average end2end delay for gaming Average processing time at each back-end VM, i.e. 1/µ

10 40 100 Mbps 200 msec 100 msec

Traffic parameters

Value

SA arrival rate Action arrival rate from each connected user Mean SA size Average size of transmitted packet from player to the cloud Average size of transmitted packet from player to the cloud Game session length

5 per minute 0.5 per sec

Uplink bandwidth of users

2 10 Bytes 50 KBytes Follows truncated Pearson (α = 0.76, β = 7.57, max length =1000 sec) Follows uniform distribution in [5,10] Kbps

Cost parameters

Value

Reserved back- and front-end VM cost

Follows uniform distribution in [0.05,0.07] $ per hour 0.02 $ per hour Follows uniform distribution in [0.05,0.07] $ per gigabyte per sec

Extra cost for on demand VM Bandwidth cost

distribution function (CDF) of the delay. From Table I one sees that the average service-time and the threshold average delay are 0.1 and 0.2 seconds respectively. In Fig. 3, one sees that the gaming delay is distributed between 0.1 and 0.2 seconds, i.e. the average delay for all served users will be less than 0.2 second. Also, one sees that the delay performance of the load-

10

1

0.6

0.4

0.2

delay bound = 0.2 sec

mean service− time = 0.1 sec

0 0.1

0.12

0.14

0.16 Time (sec)

0.18

Emprical CDF of delay

0.8 Emprical CDF of delay

1

proposed delay/cost−aware load−aware proximity−aware

0.8 0.6 0.4 proposed delay/cost-aware load-aware proximity-aware

0.2

0.2

0 0.12

Fig. 3: Gaming delay analysis for configuration 1

0.14

0.16

0.18

0.2

Time (sec)

Fig. 5: Gaming delay analysis for configuration 2

0.8

0.6

proposed delay/cost−aware load−aware proximity−aware

0.4

0.2

0 0

1

2

3

4 Cost ($)

5

6

7 −3

x 10

Fig. 4: Cloud cost analysis for configuration 1

aware scheme, which aims at minimizing the queuing delay, and proximity-aware scheme, which aims at minimizing the data transmission delay, are almost the same. Furthermore, one sees that the proposed DCRA scheme introduces a higher delay than the benchmark schemes, however, the resulting delay is less than the required threshold delay for gaming. This is due the fact that our proposed delay-aware cost-optimized scheme aims at bounding the delay by the maximum tolerable delay in order to minimize the network cost. Fig. 4 compares the cost performance of the proposed RA schemes using the empirical CDF of the cloud cost per second per PM. Ones sees that the cloud cost using the proximity- and load-aware schemes is approximately one times higher than the proposed DCRA schemes. The DCRA scheme outperforms the load- and proximity-aware schemes because the former tries to assign the arriving users to the data center with the cheapest VM and bandwidth prices that satisfies the average delay requirements of the users. Now, we extend our simulation testbed from 1-tier, denoted by configuration 1, to 3-tier, denoted by configuration 2. In the new configuration, there will be three tiers of data centers in the previous simulation area. Then, instead of 4 data centers, 12 data centers will serve the gaming traffic. The simulation results for this configuration are depicted in Fig. 5 and Fig. 6. One sees that by increasing in the number of data centers in the region, the experienced delay by users has been decreased. This is due to the fact that the distance between users and data centers has been decreased. Also, the queuing time at the data

Emprical CDF of cost per sec per PM

Emprical CDF of cost per sec per PM

1

1 0.8 0.6 0.4 proposed delay/cost-aware

0.2

load-aware proximity-aware

0 0

1

2

3

4

Cost ($)

5

6

7 ×10 -3

Fig. 6: Cloud cost analysis for configuration 2

centers has been decreased in the second configuration due to the decrease in the number of connected users to each data center. Also, one sees that the average operational expenses of the network has been increased due to the increase in the number of active physical machines in the simulation area. Fig. 7 presents the overall performance comparison for the two investigated configurations. One sees while the delay is bounded to the maximum allowed delay, the proposed scheme is able to provide a huge cost reduction. Also, it is evident that the gap between operational costs of the proposed and benchmark schemes increases as the number of deployed data centers increases. Finally, we investigate the system performance in a largescale, where 50 data centers have been deployed based on a Poisson point process in a square area centered at origin with diameter 4243 km. The results for this configuration, denoted by configuration 3, have been plotted in Fig. 8-Fig. 9. One sees that these figures follow the same trend as in Fig. 7, i.e. by increase in the number of data centers, our proposed scheme can contribute significantly in decreasing the network costs, while the QoE of the users is satisfied. For example, taking the average of the networks costs from Fig. 9 shows that the average cost per second per PM in the proposed scheme is 4

11

10 0

0.35

0.25

delay (config. 1) cost (config. 1) delay (config. 2) cost (config. 2)

Blocking Probability

Average cost ($) Average delay (sec)

0.3

0.2 0.15 0.1

10 -1

10 -2 Simulation Analytical

10 -3

0.05 10 -4

0 Proposed

Load-aware

Proximity-aware

Emprical CDF of delay

1

0.6 proposed delay/cost-aware load-aware proximity-aware

0.12

0.14

0.16

0.18

Emprical CDF of cost per sec per PM

1 0.8 0.6

proposed delay/cost-aware load-aware proximity-aware

0 0

0.5

1

Cost ($)

7

8

9

10

A. Directions for future works

Fig. 8: Gaming delay analysis for configuration 3

0.2

6

0.2

Time (sec)

0.4

5

fact that the gaming length in the simulations is assumed to follow a Pearson distribution, however, in (25) we model the gaming length by an exponential distribution with the same mean value as the mean value of the Pearson distribution. As the developed performance expression in (19) can well predict the behaviors of the system, one can use this closedform expression for resource provisioning in data centers to achieve a guaranteed level of quality of service.

0.8

0 0.1

4

Fig. 10: Blocking probability: comparison between analytical and simulation results

Fig. 7: Comparison of configuration 1 and 2

0.2

3

1/p = average SA size

Cost and delay analysis

0.4

2

1.5

2 ×10 -3

Fig. 9: Cloud cost analysis for configuration 3

times lower than the other schemes. In the sequel, the derived analytical results in section VI are validated using quantitative analysis. To this end, the blocking probability of the arriving SAs versus average SA size is depicted in Fig. 10. In this figure, the red-colored curve refers to the closed-form blocking-probability expression in (19) and (25). Also, the blue-colored curve refers to the resulting blocking probability from the simulation results (configuration 1). One sees that derived blocking probability expression in (19) follows the same trend as the simulation results with a negligible performance gap. We see that the performance gap at the highest level is 3 percent. This gap can be due to the

Here, we investigate research directions in which the present work can be extended. This work has been devoted to the investigation of the impact of the proposed framework on the cost and delay performance of the system. In recent years, green and battery lifetime-aware network design, where the former aims at decreasing the energy consumption of the server-side in order to reduce the operational expense of the network, and the latter aims at reducing the energy consumption in the client-side in order to increase the battery lifetime and user experience, have attracted lots of attentions [40–44]. Then, one may also investigate the impact of the resource allocations and cloud structure on the energy consumption of the network and the users to figure out the tradeoffs among (i) experienced delay in data transmission, (ii) operational and deployment expenses, (iii) energy consumption of the network, and finally (iv) energy consumption of users. VIII. C ONCLUSION Cloud gaming enables users with low computing capacity equipments to play high-quality games. While most aspects of computing are nowadays migrated to the clouds, gaming industry has not yet completely utilized the potential strengths of the clouds for offering high quality games to anywhere anytime. The key requirement for successful deployment of any cloud gaming system is guaranteeing the quality of experience for end users. The quality of experience itself is a complicated function of different parameters where the most important parameter is the Round Trip Time delay, i.e. the time span between action of the player and presentation of

12

the refreshed screen for the user. One may overcome the delay problem by placing gaming data centers anywhere, and by equipping them with powerful virtual machines, however, game providers need cost efficient cloud deployment strategies to minimize the deployment and operational costs of the cloud gaming while guaranteeing the QoE for users. Towards this end, delay-aware cost-minimizing resource provisioning and allocation for cloud gaming systems is investigated in this work. To maximize the efficiency of cloud gaming systems, we develop a cloud structure which benefits from two types of VMs for serving players and performing game-related actions. Then, the optimal allocation of arriving users to different data centers and switching on enough computing resources at data centers is investigated. The simulation results show that the derived analytical performance indicators can well predict the performance of the system. Also, promising cloud cost results are presented which confirm that the cloud cost can be significantly reduced while keeping the delay performance lower than a threshold. R EFERENCES [1] “Distribution and monetization strategies to increase revenues from cloud gaming,” http://www.cgconfusa. com/report/documents/Content5minCloudGamingReportHighlights.pdf, accessed: 2016-03-03. [2] https://www.onlive.com/, accessed: 2016-03-03. [3] https://www.gaikai.com, accessed: 2016-03-03. [4] http://www.g-cluster.com/eng, accessed: 2016-03-03. [5] https://streammygame. com/smg/index.php, accessed: 2016-03-03. [6] S. Shi, C.-H. Hsu, K. Nahrstedt, and R. Campbell, “Using graphics rendering contexts to enhance the real-time video coding for mobile cloud gaming,” in Proceedings of the 19th ACM international conference on Multimedia. ACM, 2011, pp. 103–112. [7] H. Tian, D. Wu, J. He, Y. Xu, and M. Chen, “On achieving costeffective adaptive cloud gaming in geo-distributed data centers,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 25, no. 12, pp. 2064–2077, 2015. [8] C.-Y. Huang, C.-H. Hsu, Y.-C. Chang, and K.-T. Chen, “Gaminganywhere: an open cloud gaming system,” in Proceedings of the 4th ACM multimedia systems conference. ACM, 2013, pp. 36–47. [9] D. Wu, Z. Xue, and J. He, “icloudaccess: Cost-effective streaming of video games from the cloud with low latency,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 24, no. 8, pp. 1405– 1416, 2014. [10] C. Chambers, W.-C. Feng, S. Sahu, D. Saha, and D. Brandt, “Characterizing online games,” IEEE/ACM Transactions on Networking (TON), vol. 18, no. 3, pp. 899–910, 2010. [11] P. Quax, A. Beznosyk, W. Vanmontfort, R. Marx, and W. Lamotte, “An evaluation of the impact of game genre on user experience in cloud gaming,” in Proceedings of IEEE International Conference on Games Innovation (IGIC). IEEE, 2013, pp. 216–221. [12] Z. Qi, J. Yao, C. Zhang, M. Yu, Z. Yang, and H. Guan, “Vgris: virtualized gpu resource isolation and scheduling in cloud gaming,” ACM Transactions on Architecture and Code Optimization (TACO), vol. 11, no. 2, p. 17, 2014. [13] Z. Zhao, K. Hwang, and J. Villeta, “Game cloud design with virtualized cpu/gpu servers and initial performance results,” in Proceedings of the 3rd workshop on Scientific Cloud Computing Date. ACM, 2012, pp. 23–30. [14] K.-T. Chen, Y.-C. Chang, P.-H. Tseng, C.-Y. Huang, and C.-L. Lei, “Measuring the latency of cloud gaming systems,” in Proceedings of the 19th ACM international conference on Multimedia. ACM, 2011, pp. 1269–1272. [15] R. Shea, J. Liu, E.-H. Ngai, and Y. Cui, “Cloud gaming: architecture and performance,” IEEE transactions on Network, vol. 27, no. 4, pp. 16–21, July 2013. [16] M. Manzano, J. Hernandez, M. Uruena, and E. Calle, “An empirical study of cloud gaming,” in Proceedings of the 11th Annual Workshop on Network and Systems Support for Games (NetGames), Nov 2012, pp. 1–2.

[17] M. Suznjevic, J. Beyer, L. Skorin-Kapov, S. Moller, and N. Sorsa, “Towards understanding the relationship between game type and network traffic for cloud gaming,” in Proceedings of the IEEE International Conference on Multimedia and Expo Workshops (ICMEW). IEEE, 2014, pp. 1–6. [18] Y.-T. Lee, K.-T. Chen, H.-I. Su, and C.-L. Lei, “Are all games equally cloud-gaming-friendly? an electromyographic approach,” in Proceedings of the 11th Annual Workshop on Network and Systems Support for Games (NetGames). IEEE, 2012, pp. 1–6. [19] T. N. B. Duong, X. Li, R. S. M. Goh, X. Tang, and W. Cai, “Qosaware revenue-cost optimization for latency-sensitive services in iaas clouds,” in Proceedings of the IEEE/ACM 16th International Symposium on Distributed Simulation and Real Time Applications (DS-RT). IEEE, 2012, pp. 11–18. [20] M. Marzolla, S. Ferretti, and G. D’angelo, “Dynamic resource provisioning for cloud-based gaming infrastructures,” Computers in Entertainment (CIE), vol. 10, no. 1, p. 4, 2012. [21] D. Finkel, M. Claypool, S. Jaffe, T. Nguyen, and B. Stephen, “Assignment of games to servers in the onlive cloud game system,” in Proceedings of the 13th Annual Workshop on Network and Systems Support for Games (NetGames). IEEE, 2014, pp. 1–3. [22] H.-J. Hong, D.-Y. Chen, C.-Y. Huang, K.-T. Chen, and C.-H. Hsu, “Placing virtual machines to optimize cloud gaming experience,” IEEE Transactions on Cloud Computing, vol. 3, no. 1, pp. 42–53, 2015. [23] Y. Li, X. Tang, and W. Cai, “Play request dispatching for efficient virtual machine usage in cloud gaming,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 25, no. 12, pp. 2052–2063, 2015. [24] I. Dimitriou, “A mixed priority retrial queue with negative arrivals, unreliable server and multiple vacations,” Applied Mathematical Modelling, vol. 37, no. 3, pp. 1295–1309, 2013. [25] M. Kalil et al., “Low-complexity power-efficient schedulers for LTE uplink with delay-sensitive traffic,” IEEE Transactions on Vehicular Technology,, vol. 64, no. 10, pp. 4551–4564, 2015. [26] J. S. Tanton, Encyclopedia of mathematics. Facts On File, 2005. [27] S. Abolfazli, Z. Sanaei, M. Alizadeh, A. Gani, and F. Xia, “An experimental analysis on cloud-based mobile augmentation in mobile cloud computing,” IEEE Transactions on Consumer Electronics, vol. 60, no. 1, pp. 146–154, 2014. [28] W. Gong, Z. Chen, J. Yan, and S. Qianjun, “An optimal vm resource allocation for near-client-datacenter for multimedia cloud,” in 2014 Sixth International Conference on Ubiquitous and Future Networks (ICUFN). IEEE, 2014, pp. 249–254. [29] S. Shen, K. Deng, A. Iosup, and D. Epema, “Scheduling jobs in the cloud using on-demand and reserved instances,” in Euro-Par Parallel Processing. Springer, 2013, pp. 242–254. [30] H. Zhu, S. Goddard, and M. B. Dwyer, “Response time analysis of hierarchical scheduling: The synchronized deferrable servers approach,” in IEEE 32nd Real-Time Systems Symposium (RTSS), 2011, pp. 239– 248. [31] W. J. Stewart, Probability, Markov chains, queues, and simulation: the mathematical basis of performance modeling. Princeton University Press, 2009. [32] D. Niu, H. Xu, B. Li, and S. Zhao, “Quality-assured cloud bandwidth auto-scaling for video-on-demand applications,” in INFOCOM, 2012 Proceedings IEEE. IEEE, 2012, pp. 460–468. [33] H. Ahmadi, S. Z. Tootaghaj, M. R. Hashemi, and S. Shirmohammadi, “A game attention model for efficient bit rate allocation in cloud gaming,” Multimedia Systems, vol. 20, no. 5, pp. 485–501, 2014. [34] T. Bonald and M. Feuillet, Network Performance Analysis, ser. ISTE. Wiley, 2013. [Online]. Available: https://books.google.com/books?id=4rxWS-5OJ60C [35] A. K. Gupta, W.-B. Zeng, and Y. Wu, Probability and statistical models: foundations for problems in reliability and financial mathematics. Springer Science & Business Media, 2010. [36] G. J. Franx, “A simple solution for the m/d/c waiting time distribution,” Elsevier transactions on Operations Research Letters, vol. 29, no. 5, pp. 221–229, 2001. [37] G. Bolch, S. Greiner, H. de Meer, and K. S. Trivedi, “Single station queuing systems,” Queuing Networks and Markov Chains: Modeling and Performance Evaluation with Computer Science Applications, pp. 209–262, 1998. [38] A. Checko, H. L. Christiansen, Y. Yan, L. Scolari, G. Kardaras, M. S. Berger, and L. Dittmann, “Cloud ran for mobile networksa technology overview,” IEEE Communications surveys and tutorials, vol. 17, no. 1, pp. 405–426, 2015. [39] M. Kwok and G. Yeung, “Characterization of user behavior in a multiplayer online game,” in Proceedings of the ACM SIGCHI International

13

[40] [41]

[42] [43] [44]

Conference on Advances in computer entertainment technology. ACM, 2005, pp. 69–74. H. Guan, J. Yao, Z. Qi, and R. Wang, “Energy-efficient SLA guarantees for virtualized GPU in cloud gaming,” IEEE Transactions on Parallel and Distributed Systems, vol. 26, no. 9, pp. 2434–2443, Sept 2015. I. Ghergulescu, A. N. Moldovan, and C. H. Muntean, “Energy consumption analysis of cloud-based video games streaming to mobile devices,” in IEEE International Symposium on Broadband Multimedia Systems and Broadcasting, June 2015, pp. 1–6. A. Azari, “Energy-efficient scheduling and grouping for machine-type communications over cellular networks,” Ad Hoc Networks, vol. 43, pp. 16–29, 2016. G. Miao and G. Song, Energy and Spectrum Efficient Wireless Network Design. Cambridge University Press, 2014. A. Azari and G. Miao, “Energy efficient MAC for cellular-based M2M communications,” in 2014 IEEE Global Conference on Signal and Information Processing, 2014, pp. 128–132.