A Reinforcement Learning based, Priority-aware Computational Offloading for Vehicular Networks Rico Valentino Ajou University, Suwon Republic of Korea
[email protected]
Dohyung Kim
Young-Bae Ko
Ajou University, Suwon Republic of Korea
Ajou University, Suwon Republic of Korea
[email protected]
[email protected]
ABSTRACT
2
Building an optimal offloading decision engine for vehicular network could be a hard challenge because of the environment complexity and variety types of vehicular applications. Different types of applications may need a different priority too. Road safety application should have a higher priority than the infotainment application because it contains a bigger loss potential to human being when it is not finished on time. To address the problem, we propose a priority-aware computational offloading system on vehicular network to find the best offloading decision strategy. We adopt Reinforcement Learning (RL) technique to obtain the optimal computational offloading strategies with the goal of minimizing the delay cost or response time while satisfying the application Quality of Service (QoS). The experimental result shows that the proposed scheme has a shorter response time and less violated priority cases compared to the previous no priority-aware offloading scheme.
In [2], the authors proposed a delay constrained offloading framework for vehicular edge computing network by adopting a Stackelberg game theoretic approach. In [3], an opportunistic WiFi offloading scheme was proposed for vehicular environment by optimizing the trade-off downloading cost and downloading delay. However, there were no consideration of different QoS needs in the decision making engine of the former. Therefore, whenever different applications with different QoS come to the processing queue, the offloading decision engine does not aware of the priority criteria, resulting in the applications’ QoS dissatisfaction. For example, let us assume that safety applications for detecting accidents are running together with games application. At the same time, because the car computational power is already fully occupied, both applications need to do offloading scheme. When this situation occurs, the previous scheme does not have any consideration to execute which application first, it could be either of them. If it choose the games application to be offloaded first, then the accident detector application will be delayed for some time. In this kind of situation, the QoS of the safety applications is not satisfied and could be dangerous if an accident really occurs in front of our car but we could not detect it in time. This scenario shows how important to consider employing the QoS aware computational offloading system in vehicular network environments.
1
INTRODUCTION
Computational offloading is one possible solution to complete resource-hungry applications in time by offloading some tasks to the server. Vehicular networking application can be differed into three categories based on their usage and requirements: road safety application, cooperative traffic efficiency application, and infotainment application [1]. The difference of these applications criteria lead to a need of different priority in term of executing these applications. Active road safety applications tend to have a higher priority to be executed first due to their impact to people safety rather than game applications. So, intuitively active road safety applications are preferred to be executed in the local processing unit of vehicle in order to meet the delay constraint compared to the other applications. However, some safety applications need a very high computational resource power such as for object detection or face recognition; in this case it is better to offload this task to the server given that the communication delay is relatively small compared to its computation. It could be a challenge for an offloading decision engine to decide which applications to be offloaded first and to be locally processed while satisfying their QoS. In this work, we propose a more intelligent offloading decision engine by adopted RL technique. In addition, our scheme considers a queue of applications rather than a singular application.
3
MOTIVATION
PROPOSED WORK
In general, the high priority job will be executed locally if the computing power of vehicle is enough to complete it within the maximum tolerable delay. However, even though the computational capacity is enough, the next job could have a higher priority. To avoid that kind of situation, in this work we consider finding the best combination of offloading decision for each application in queue window. We determine which job to be offloaded to MEC server and which job being executed locally. We adopt RL technique to optimize the goal function of finding the best offloading decision combination to achieve the fastest response time of the system while satisfying the job QoS. We consider M Road Side Units (RSUs) located along a road which will deliver the offloaded data to the MEC server. Each RSUi has its transmission range on segment Ri. We consider there will be a W size of computation-intensive applications queue instead of
CoNEXT’18, December 4-7, 2018, Heraklion/Crete, Greece
R. Valentino et al. Reinforcement learning element for the given system model: state: s = ( ) with is the total response time of the vehicle offloading system action: it consists of vector reward: the agent will receive a reward on each step , with Tcloc is the response time of
Figure 1: Priority-aware Computational Offloading Decision Framework singular application task on each vehicle (Fig. 1). Applications are denoted as An (In, Cn, Dn, πn), where In is input size of the computational data, Cn is computational cycle needed to complete the application, Dn is the maximum tolerable delay, and πn is the application QoS (safety>efficiency>infotainment). We denote αn element {0,1} as the computation offloading decision of application n; 0 if a vehicle executes application by doing local computing, otherwise 1 for offloading to MEC server. Local computing delay ( ) can be computed as: (1)
executing all tasks locally and is the current state response time. Specifically, we use Q-learning algorithm to solve the problem, by looking at the maximum Q value, the RL agent can decide the best combination of to give the optimal response time while satisfying the QoS of applications. can be given as: (5) Where is the current state and action, is the next state and action. value shows how the agent treats the future reward.
4
EVALUATION AND RESULTS
We used IEEE 802.11p Wi-Fi protocol with 10MHz bandwidth for our ns-3 simulation. The computation capacity of vehicle and MEC server are 1GHz/sec and 5GHz/sec, respectively. Job queue window size W is set to 5 and QoS type is set to 3 different priority. is set into 0.8, input size In is set randomly from 100 to 800 kB, and computational cycle Cn is set between 1000 to 9000 Megacycles.
Where is the computation capacity of vehicle (i.e., CPU cycles per second). Offloading computing delay ( ) can be obtained as: (2) and
are the communication delay of uploading the input data
and downloading the result data, respectively. and are the instantaneous data rate when uploading and downloading the result back, respectively.
is the computation delay on the server
with , the server computing power. Then the total response time can be expressed as : ∑
(3)
With a limitation of maximum tolerable delay and computation capacity, the problem can be formulated as: ∑
s.t. C1:
{
}
C2: (4) In (4), is the offloading decision vector. represents each application is executed either locally or in the server. guarantees the total response time should not exceed the maximum tolerable delay. To solve the optimization problem, we propose reinforcement learning methods to find the optimal . We introduce β as the priority weight of application on the reward state. β will be small if the agent decide to offload the highest QoS job so that the reward will be big, otherwise β is big. By giving such a reward scheme, the agent will learn to choose the highest priority job first to be offloaded, if there are more than one application needs to be offloaded.
Figure 2: a) Response time of different schemes with variety of total application input size b) Total number of violated priority cases Total application size means the sum of input parameter size from all applications in the queue. No priority-aware scheme uses FIFO style scheduling to execute the application queue and have no information of applications’ QoS. Full local means all applications are executed locally in the vehicle. Figure 2a) shows the proposed scheme have a better performance by 20% and 25%, compared to no priority-aware and full local scheme’s, respectively. It happens because the agent learned not only from a single application at a time, but also from a queue of applications so that it can avoid suboptimal offloading decision. Figure 2b) shows the total of violated priority cases. Violated priority case means the situation when the decision engine offloads a lower QoS application first even though there is a higher QoS application to offload too. Our priority-aware scheme has a total of 5.1% violated cases which is better than no priority-aware scheme with 49.6% of violated cases.
A Reinforcement Learning based, Priority-aware Computational Offloading for Vehicular Networks
REFERENCES [1] E. Uhlemann, "Introducing Connected Vehicles [Connected Vehicles]," in IEEE Vehicular Technology Magazine, vol. 10, no. 1, pp. 23-31, March 2015. [2] K. Zhang, Y. Mao, S. Leng, S. Maharjan and Y. Zhang, "Optimal delay constrained offloading for vehicular edge computing networks," 2017 IEEE International Conference on Communications (ICC), Paris, 2017, pp. 1-6. [3] N. Wang and J. Wu, "Opportunistic WiFi offloading in a vehicular environment: Waiting or downloading now?," IEEE INFOCOM 2016 - The 35th Annual IEEE International Conference on Computer Communications, San Francisco, CA, 2016, pp. 1-9. [4] C. Zhang, B. Gu, Z. Liu, K. Yamori and Y. Tanaka, "A reinforcement learning approach for cost- and energy-aware mobile data offloading," 2016 18th AsiaPacific Network Operations and Management Symposium (APNOMS), Kanazawa, 2016, pp. 1-6. [5] X. Wang, Z. Ning and L. Wang, "Offloading in Internet of Vehicles: A Fogenabled Real-time Traffic Management System," in IEEE Transactions on Industrial Informatics (2018). https://doi.org/10.1109/TII.2018.2816590 [6] X. Chen, L. Jiao, W. Li and X. Fu, "Efficient Multi-User Computation Offloading for Mobile-Edge Cloud Computing," in IEEE/ACM Transactions on Networking, vol. 24, no. 5, pp. 2795-2808, October 2016.
CoNEXT’18, December 4-7, 2018, Heraklion/Crete, Greece
