in fog using content awareness and weighted graph. Service reconstruction mechanism is devised to dynamically recycle modularized resources after mapping ...
Software-Defined Efficient Service Reconstruction in Fog Using Content Awareness and Weighted Graph Shan He 1, Mianxiong Dong 2, Kaoru Ota 2, Jun Wu 1, Jianhua Li 1, and Gaolei Li 1 1 School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Shanghai, China 2 Department of Information and Electronic Engineering, Muroran Institute of Technology, Muroran, Japan {hszoe1995, junwuhn, lijh888, gaolei_li }@sjtu.edu.cn, {mx.dong, ota}@csse.muroran-it.ac.jp Abstract—Fog computing, shifting intelligence and resources from remote cloud to edge networks, has the potential of providing low-latency for the end-to-end communication from data sources to users. However, it’s hard to enhance resourceefficiency in the existing relatively static and proprietary framework of fog nodes due to the diversity of service requirements. With the growing deployment of fog computing, the overall resource consumption in fog will be huge without considering the efficient service provisioning in each fog node. On one hand, different fog users require diverse local services policies which are carried out within the fog nodes. Moreover, for one user, the requirements on services are time-varying. On the other hand, the processing strategies on different types of content (e.g. video, audio, etc.) are also distinct. These dynamic features impose the need for user-driven and content-based service reconstruction in order to achieve the high recycling utilization of resources of fog system. To this end, we propose a software-defined efficient service reconstruction (SDSR) scheme in fog using content awareness and weighted graph. Service reconstruction mechanism is devised to dynamically recycle modularized resources after mapping different contents to relevant operations. Weighted graph is introduced to schedule and optimize the services reconstruction in terms of resource saving during content-driven controlling. User-defined interfaces are designed to enable fog users to reconfigure the recyclable resource modules. Simulation results demonstrate that the service cost of each fog nodes is reduced significantly, thus promote efficient service provisioning for the whole fog system. Index Terms—Fog computing; software-defined efficient services reconstruction(SDSR); content-driven; weighted graph
I. INTRODUCTION Fog computing, acting as an intermediary layer between fog users and underlying networks, which brings the intelligence and resources near the end devices, is regarded to have the capability of saving energy in cloud computing by [1, 2]. Massive data collected by the large-scale end devices (i.e. smart sensors, road side units and radio frequency identification systems) are directly sent to the local fog nodes within their region to get timely and high-rate services instead of transmitted to the remote cloud. Finally, the processed contents are uploaded to the upper layer fog users for further global analytics as well as long-term storage. Therefore, fog computing can relive the burden of the data center [3], reduce the end-to-end delay [4] under different circumstances. It has gained increasing attention for supporting various time-critical applications especially on wireless sensor network [5], 5G [6], smart grid [7] as well as e-health care [8].
However, with the larger and larger deployment scale of fog nodes, great attention should be given to the provisioning of efficient services of fog system. Since the whole fog system is composed of a great number of fog nodes, the key problem then focuses on the optimization of resource consumption within each single fog node. However, in real scenarios, the services policies carried out in fog nodes are distinct and dynamic mainly from two perspectives. For one thing, fog users have diverse demands on data processing services provided by local fog nodes. When the application scenarios transform or fog users’ requirements change, tedious efforts must be made to rebuild new services in fog nodes. This variability and dynamic features make the service on contents differ in some core parameters setting, such as number of data sets, accuracy, range of values, deviation, derived value and data compression ratio etc. The selection of certain processing algorithms, dictionary, data format, mathematical formulas, fusion regulation, compression methods, storage format may also vary. For another, contents of different types are needed to be handled under diversified processing strategies. These factors lead to the resourceinefficiency and labor-consuming rebuilding and redeployment on services in the existing relatively static and proprietary framework of fog nodes. Nevertheless, it’s worth noting that although the services required by different fog users and contents are diverse, they have many overlapping parts in component operations. This characteristic offers a promising way to recycle these basic units. To this end, we put forward a software-defined efficient service reconstruction (SDSR) scheme for fog system based on content awareness and weighted graph. Three mechanisms are designed in this novel fog nodes architecture to facilitate the efficient services provisioning by recycling resources. A dynamic service reconstruction mechanism is presented, in which the services are resolved into multiple service operations. An atomic operation is further encapsulated into multiple recyclable modules, implemented with different algorithms. Due to the resource-constrained feature of end devices, the contents are usually delivered to fog nodes without encryption. Hence, the fog nodes are capable of actually ‘seeing’ the content, which means the content awareness is made possible [9, 10]. Utilizing this ability, a content-based policy making scheme is also devised to assist in the resource-efficient service reconstruction. Content-Based Labelling (CBL) technology [11] can be used here to identify them with specific labels. Then different contents are automatically processed under corresponding policies by constructing recyclable modules
978-1-5090-5019-2/17/$31.00 ©2017 IEEE
We introduce the software-defined concept into fog system by designing a novel hierarchical architecture for fog nodes to realize an efficient service provision scheme through resources recycling and service reconstruction in fog system.
Making the most of the content-aware capability of fog system, we devise a content-driven policy making mechanism, mapping different types of contents to diverse processing service policies accordingly. Weighted graph is introduced to optimize the overall resource consumption in fog system.
A modularized recyclable service model is presented, in which basic operations are abstracted and encapsulated to realize the service reconstruction.
To obtain the high-level flexibility, we also propose a user-defined reconfiguration paradigm by enabling fog users to control and customize the service by simply operating modules or specifying particular requirements on their service policy. Based on the real-time network states, parameters of the modules can also be self-adaptively adjusted to make the services agile enough to the changing conditions. We implement and evaluate the proposed softwaredefined efficient services for fog system and simulation results demonstrate that the consumption of services resource is decreased significantly.
The rest of this paper is organized as follows. We investigate the related works about application scenarios and resource-efficiency of existing fog system in Section II. The general overview and design principle of the proposed architecture are given respectively in Section III and IV. Simulation experiments and the preliminary results are provided in Section V. Finally, the paper is concluded in Section VI. II. RELATED WORKS Since being proposed by Cisco in 2012, fog computing has been applied in wide range of fields [2]. By exploiting the concept of fog computing into E-health scenarios, the real-time monitoring and low-latency response can be realized easily [11]. In [12], a hierarchical fog computing architecture is presented to support ephemeral storage and local data mining in Smart Cities. The authors integrate FOG-engine into wireless sensor networks to offload the data analytic tasks from remote cloud in [13]. These different scenarios impose diverse services requirements for fog nodes, which lead to the resource inefficiency in services redeployment and rebuilding. Hence, it is important to optimize the resource consumption for the whole fog system. An election-based routing protocol are proposed to save energy, thus prolong the lifetime of the fogsupported wireless sensor networks [14]. With the purpose of
Yet few of these proposed approaches considered the efficient service provisioning. An efficient scheduling mechanism was provided to realize the energy-awareness in cloud scenario [17]. But very few literatures focus on the efficient service designation in fog system, especially in terms of the recycling and reconstruction of resources. To facilitate the flexible reconfiguration of service resources, the idea of “software-defined everything” is noticed. Researches on integrating Software-Defined Networking (SDN) controller into fog computing has been attracting more and more attention [18, 19]. However, these solutions only combined these two paradigm at network level rather than the dynamic reconstruction performing within fog computing nodes. To the best of our knowledge, the design of providing softwaredefined efficient service for fog system is still nearly blank. III. SOFTWARE-DEFINED EFFICIENT SERVICE With the goal of designing a resource-efficient service reconstruction paradigm for fog system, we propose a software-defined efficient service provisioning scheme for fog nodes as shown in Fig.1.
User 1
User 2
User 3 Fog users
Service requirements
minimizing the task consumption, Lingjun Pu, et al. designed a novel mobile task offloading paradigm for the device-to-device (D2D) Fogging by facilitating the collaboration among mobile users [15]. In [16], a mobile caching scheme was presented for the energy-efficient fog nodes while maintaining the Quality of Service (QoS) provisioning.
Real-time network state
properly. By the designation of user-defined reconfiguration interfaces, fog users are enabled to customize the service by simply programming the service modules and specifying their particular requirements without rebuilding the whole service. The contributions of this paper are the following:
User-defined reconfiguration Processed content uploading User-defined policy
Software-defined requirements receiving
Network state
Network state monitoring
Management proxy Network state detector
Requirements repository
Content-driven control Service policy making Content-driven policy making engine Requirements resolver
Content-operation mapping table
User requirements resolving
Service logic order
Service module management
Content-based policy repository
Service reconstruction Service scheduling Service reconstructor Content-aware unit Module manager
Service module pool
Module recycling
Content temporary storage
Content analyzing
Service resource repository
Fog node Data collection
Fig. 1. Hierarchical architecture of proposed SDGC
In this section, we present the architecture of softwaredefined fog nodes and how the three newly designed mechanisms coordinate with each other. A. Basic Architecture The proposed hierarchical architecture is composed of three mechanisms logically, namely, service reconstruction, contentdriven control and user-defined reconfiguration. The service reconstruction involves content awareness and reconstruction of required modules based on the service policy issued by the upper layer control logic. The weighted graph is introduced to assist in the service policy making and optimizing. Fog users are enabled to program and customize services performing in the underlying fog nodes by reconfiguring particular demands or modifying service modules via the exposed interfaces instead of creating new services to adapt to their changing requirements. As shown in Fig. 1, the software-defined architecture of fog nodes is enriched with several extra functional components, respectively named from bottom to up as follows: (1) service reconstruction: Content-aware Unit, Service Reconstructor and Module Manager, (2) content-driven control: Content-driven Policy Making Engine and Requirements Resolver, (3) userdefined reconfiguration: Management Proxy and Network State Detector. Moreover, three special storage space, namely, (1) Service Resource Repository (composed of service module pool and content temporary storage), (2) Content-based Policy Library (composed of content-operation mapping table and processing logic order), and (3) Requirements Repository (composed of user-defined policy and network state), are needed to support its normal functioning. More details about three mechanisms are described as below. B. Service Reconstruction As the service execution and enforcement layer of the proposed software-defined fog nodes, the service reconstruction is essentially the same as the efficient service actuator of fog nodes. As depicted in Fig. 2, it is primarily responsible for invoking and initializing the required modules from the service module pool according to the policy, scheduling them in the reasonable execution order to realize the recycling utilization of service resources. Content awareness is also performed in this layer. Content-operation mapping ralation
Users’ service policy Real-time network state
Content-based policy library
Requirements repository
Requirements resolver
Service logic order
Content-driven policy making engine
Intermediate result
Module 1
Module 2
……
Module n Processed data
Raw data 1
The service reconstruction involves invoking and initializing the required modules and scheduling them in the reasonable order. These recyclable service modules are stored in the service module pool in the Service Resource Repository, while the execution logic order is maintained in the Content-based Policy Library. These are done by the Service Reconstructor. Since modules are designed loosely coupled, the Service Reconstructor is introduced as a middleware to allow multiple independent modules to be "plugged in" to it [14]. Information like intermediate results are not transmitted from module to module. Instead, they are delivered via the reconstructor. The reconstructor should also achieve a unified business interoperability and compatibility of different modules. Modules connected to it are designed to communicate using standard format messages, without needing to know each other's specific order in the sequential policy graph and other properties. The required module ID is issued by the Content-driven Policy Making Engine. Content awareness is also achieved in this layer. As explained in section I, fog nodes are capable of extracting and analyzing the payload and key attributes of content collected. The Content Aware Unit assigns each content with specific label based on their types and features which decide how to process them using the CBL technology. C. Content-Driven Control The content-driven control layer is the core of the designation of resource-efficient efficient service provisioning in the software-defined fog nodes. The main components of this layer are Content-driven Policy Making Engine and Requirements Resolver. The Content-driven Policy Making Engine works equivalent to the brain of the software-defined fog nodes, deciding how to reconstruct the recyclable resources. It determines which modules to call, how to organize the selected modules in right order and how to integrate the fog user’s preference into the service reconstruction.
Service reconstructor Initparameter
The modularization of basic service operation is important for the recycling and reusability of resources. Services are decomposed into multiple basic essential operations. An atomic operation (such as data compression, data filtering, data averaging computing, etc.) is further abstracted and encapsulated into modules, each of which is implemented by different algorithms. All the service modules in the charge of Module Manager are stored in the service module pool. They can be reused and easily migrated when the application scenario transforms or fog users’ requirements change. The scalability is realized by extending external third-party module library into service module pool. It is noteworthy that the initial parameters of each module are defined by fog administrators at the start of processing and then can be modified when the network states monitored by the detector fluctuates exceeding the threshold.
Service module pool
Fig. 2. Content-driven and user-defined service reconstruction.
The contents collected from end devices are assigned with different labels using the CBL technology. These labels are used mainly for two purpose. Firstly, they assist in deciding on accordance service operations based on the content-operation mapping table stored in the Content-based Policy Repository. Secondly, the label tagged on each content represents its
current state which is useful when mapping these contents into the policy graph. The state of a content means its key attributes parameters relevant to the processing. For example, the size of content, resolution ratio, the security-related pattern (whether it is a plaintext or ciphertext). The orchestration logic order is also an important part in a service policy. Inspired by the ‘polymorphism’ concept in object-oriented ideas, which defines common member functions in base class and its derived classes but using different policies to implement them since objects usually have individual differences in terms of detailed implementation, we assume that contents with different labels should follow the same processing logic. That is to say, the execution order should be only dependent on the required service operations. As long as the service modules for a certain content are determined, the execution order of these modules is also established according to the pre-defined processing flow. The processing flow is defined by fog administrators during the deployment and service configuration of fog nodes. Once the required operations are determined, the reconstructor just simply map them to the predefined logic order flow. Although the required service operations are decided, the selection of specific modules is still unresolved. The weighted graph and its relevant adjacent matrix are introduced to assist in service modules selection and minimization of the overall resources consumption. The initial parameters passed to required modules are configured by fog administrators, but some parameters vary with the changes in network state to dynamically adapt to the current network performance. More details about the weighted graph based policy making scheme are illustrated in section IV. It is also worth mentioning that when makes a policy for a specific service providing for a certain type of content, the demands associated with this service are retrieved from the Requirements Repository. The current network state information is also considered to adjust relevant parameters of some service modules to make the service adapting to the changing network states. D. User-defined Reconfiguration With the resource-efficient service reconstruction scheme implementing in the underlying fog nodes, fog users are enabled to reconfigure and specify their different demands on service reconstruction via the proposed Management Proxy. During the process of service policy making, the fog user’ requirements and preferences are taken into consideration in priority. Considering the fluctuant network states, a time-varying parameter adjustment scheme is also presented to adapt the underlying service modules accordingly. The interaction flow is shown in Fig. 3. Two main components in this layer are Management Proxy and Network State Detector. In order to design and realize a finer-grained control on the service reconstruction, service modules stored in the Service Resource Pool can be customized through managing operations (such as adding, deleting, modifying etc.) by Module Manager. In this mechanism, fog users can reconfigure service resources at a highly abstract lever in a software-defined manner. It is
obvious that the addition or deletion of modules will not affect other modules. But, weight edges standing for new service modules may need to be added in the policy graph and the logical order may also change under this circumstances. The Network State Detector keeps an eye on the real-time conditions of the changing network, such as network speed and transmission delay. Once the state deviation exceeds the predefined threshold value, this message will be reported to the Service Reconstructor to help in the adjustment of relevant parameters. For example, the data compression ratio can be set to a higher value when the network speed is low to ensure the transmission efficiency. Network state detector
Fog users
Requirements resolver
Content-driven policy making engine
Service reconstructor
Users’ resource-recycling requirements Real-time network state Receiving feedback Detector rate adaption
Users’ resource-recycling requirements Parameter adaption Execution Order Required modules Reconstruction feedback
Processed content
Fig. 3. Reconstruction sequence of service resources.
IV. DESIGN PRINCIPLES A. Modularization of Data Service operations In the modularization of service, recyclable modules belonging to a certain operation are implemented using distinct algorithms with different computation cost. The definition and relationship between them can be expressed as: ={
,
, ……,
}
(1)
where DPOi represents i-th service operation implemented in in different modules. Which module to choose depends on the analysis of users’ requirements and the optimization of overall resource consuming realized in the content-driven control layer. Each module is defined by four types of interfaces represented as: ={
.o,
.in,
.out,
.parm }
(2)
where:
.o defines the details about how to deal with the data using the algorithm embedded in the module.
.in = { . , . , ……, inputs of the service module.
.out = { . , . , ……, set of outputs of the service module.
. parm is the relevant parameters of each module. It is specified in the format of numbers or strings by
.
} is the set of .
} is the
the service reconstructor at first, for example, the data compression ratio, filtering conditions, data size threshold etc. But with changes of network state or the demands of users, it can be adjusted automatically. B. Content Service Policy Making Model The service policy is used to instruct the recyclable service modules to realize the resource-efficient service reconstruction. It can be represented by: DP = {mod, para, ord},
(3)
where mod represents required service modules, para means initial parameters and ord is the order of execution. Content with specific label can be mapped to a certain set of service modules. The initial parameters and execution order are given by the fog nodes in advance and may change. Graphs and its adjacent matrix are used to model the service policy. The content-based service policy is modeled as a directed =( ) representing global logic weighted acyclic graph = { n1, n2, …, nS } is a infinite non-empty set of policy, where S logic nodes representing the possible intermediate states of content in the data processing and = { } (i, j [1, S] ) representing specific service which is a 2-element subsets of modules. An edge eij connecting ni to nj indicates that a certain service module has to be called in order to move from one state ni to its adjacent vertex representing the next state nj. Generally speaking, more than one edge exists between two adjacent nodes since different algorithms can be used to implement one service operation. A certain path Pc = (nl, nk) in the generated sub-graph is a possible way to orchestrate a set of service modules. The service policy is obtained as long as the path is chosen. Based on its required operations, the starting node nl and the terminal node nk are decided. So the Content-driven Policy Making Engine firstly searches for a sub-graph that actually is a policy set which satisfies the demands of the content in the given global graph. Next, all we need to do is to = ( nl, nk ) in the choose a certain path (service policy) generated sub-graph E = ( , ) for the content Cg according to the resource each path consumes and to optimize the policy making, where = { nl, nl+1, …, nk }. C. The Resource-Efficient Policy Optimization Module With the goal of optimizing the service policy making in terms of the overall resource consumed, the adjacent matrix and weight matrix are introduced. Let G be the number of input packets(content) in certain time window. The g-th content , where g = 1, 2, …, G, is defined to identify the content to be processed. Denote H as the number of required operations of content and K as the number of different algorithms(modules) that can implement the certain represents the k-th module of the h-th operation. Thus, operation. It’s noteworthy that H = S – 1, where S represents the total number of content states during the service. We use E = ( eij ), where i, j [1, S]，to denote the adjacency matrix of the global logical policy graph = ( , ), in which: =
(4)
= 0, if i = j, i, j.
and by definition
We also define a matrix Boolean values
=(
) established by G K
to represent the sub-graph, where: (5)
= and also by definition
= 0, if i = j,
i, j.
is used to denote whether the edge { } in the Matrix global policy graph is chosen, which, in other words, whether the h-th service operation is needed for the content . In order to estimate the resource consumed by each policy, a ), where each matrix element is weight matrix W = ( the computation cost of the k-th module for the h-th service operation. The optimization target of the service policy making is to choose the one with relatively least resource consumed. Hence, the parameter Res is introduced to represent the total resource the policies consumed for all the content, where: Res =
,
i, j
(6)
In most cases, we tend to choose the policies with the least resource consumed for the service provisioning. But sometimes users may have special requirements. For instance, they may demand one of the component operations performing on certain content implemented with a specific module(algorithm). This customization can be realized by setting preferences on relevant items via the interfaces provided by user-defined reconfiguration layer in the label-operation mapping table stored in the Content-based Policy Library to enforce the execution of the required service modules. Under these circumstances, the Requirements Analyzer checks the constrains placed on the certain labels to guarantee the needed modules involved to achieve the user-defined service reconstruction. V. SIMULATION AND ANALYSIS In this section, we present the validity of proposed softwaredefined efficient service reconstruction (SDSR) scheme by simulation on delay time cost and computing cost of fog nodes. The main objectives of the simulation are to test the feasibility of the weighted graph based service policy making and observe the influence of performance factors on the fog systems. We consider a fog node with 100 available service modules and the contents’ requests are random. In this scenario, we assume that the performance of content-based services reconstruction algorithm is direct proportional to its complexity. The comparison of delay time costs in different architectures is shown in Fig. 4, where the number of service modules is varying in the interval of 0 to 30. As seen, for content requests with little service modules, the delay time cost is fluctuant when the number of service modules belongs to [0, 15]. For a common fog node, the delay time cost is largest; for fog node with QoS services, the delay cost is also larger than that of proposed content based reconfiguration in SDGC scheme.
ACKNOWLEDGE This work was supported in part by the National Natural Science Foundation of China under Grant 61401273, 61431008 ， 61571300 and partially supported by the JSPS KAKENHI Grant Number JP16K00117, JP15K15976, KDDI Foundation. REFERENCES [1]
[2] Fig. 4 Comparison of delay time costs in different architectures.
[3]
[4] [5]
[6] [7]
[8]
Fig.5 The value of performance/complexity of proposed services scheduling for mobile and static devices.
Fig. 5 shows the computing cost of proposed SDGC scheme for mobile and static devices, respectively. In this case, we exploit the ratio value of performance/complexity as the optimal parameter to evaluate the influence of performance factors on the fog systems. A higher ratio value of performance/complexity represents higher quality of data processing services. The simulation results show that by reconstructing the fog nodes, the proposed SDGC scheme supports mobile and intermittent connections, fog providers can classify the mussy requests, schedule the data processing services and gives accurate responses dynamically. The proposed scheme can be utilized to improve the users’ quality of experience. VI. CONCLUSION With the goal of reducing the overall resource consumption for fog system, we focused on the efficient service reconstruction of the fog node. To address the challenges that the existing solidified architecture of fog nodes brings and save resource, we proposed a software-defined service provisioning scheme based on content awareness and weighted graph by the designation of service reconstruction, content-driven policy control and user-defined reconfiguration. The proposed SDGC scheme can obviously reduce the resource consumptions by recycling service resources, meanwhile meeting fog users’ diverse service requirements on contents. Based on SDGC, the fog system can interplay with the upper applications or the cloud data centers in a flexible and efficient way.
[9]
[10]
[11]
[12]
[13]
[14] [15]
[16]
[17]
[18]
[19]
F. Jalali, K. Hinto, R. Ayre, T. Alpcan, R. S. Tucker, “Fog Computing May Help to Save Energy in Cloud Computing,” IEEE Journal on Selected Areas in Communications, vol. 34, no. 5, pp. 1728-1739, 2016. F. Bonomi, R. Milito, J. Zhu, and S. Addepalli, “Fog Computing and its Role in the Internet of Things,” ACM Edition of the Mcc Workshop on Mobile Cloud Computing, pp. 13-16, 2012. H. Zhang, Y. Xiao, et .al, “Fog computing in Multi-tier Data Center Networks: A Hierarchical Game Approach,” IEEE International Conference on Communications (ICC), pp. 1-6, 2016. Y. Y. Shih, et al, “Enabling Low-Latency Applications in Fog-Radio Access Networks,” IEEE network, vol. 31, no.1, pp. 52-58, 2017. M. Dong, K. Ota, A. Liu, “Preserving Source-Location Privacy through Redundant Fog Loop for Wireless Sensor Networks,” CIT/IUCC/ DASC/PICom, pp. 1835-1842, 2015. M. Tao, K. Ota, M. Dong, “Foud: Integrating Fog and Cloud for 5GEnabled V2G Networks,”IEEE Network, vol. 31, no. 2, pp. 8-13, 2017. A. Y. Okay, S. Ozdemir, “Fog Computing based Smart Grid Model,” 2016 International Symposium on Networks, Computers and Communications (ISNCC), pp. 1-6, 2016. S. Chakraborty, S. Bhowmick, P. Talaga, D. P. Agrawal, “Fog Networks in Healthcare Application,” IEEE 13th International Conference on Mobile Ad Hoc and Sensor Systems (MASS), pp. 386-387, 2016. G. Li, M. Dong, K. Ota, J. Wu, J. Li, T. Ye, “Deep Packet Inspection Based Application-Aware Traffic Control for Software Defined Networks,” GLOBECOM, pp. 1-6, 2016. L. Huang, G. Li, et .al, “Software-defined QoS Provisioning for Fog Computing Advanced Wireless Sensor Networks,” IEEE SENSORS, 2016. K. Wrona, S. Oudkerk, A. Armando, “Assisted Content-based Labelling and Classification of Documents,” International Conference on Military Communications and Information, pp. 1-7, 2016. T. N. Gia, M. Jiang, et .al, “Fog Computing in Healthcare Internet of Things: A Case Study on ECG Feature Extraction,” IEEE Conference on Computer and Information Technology, pp. 356-363, 2015 B. Tang, Z. Chen, H. Gerald, T. Wei, H. He, Q. Yang, “A Hierarchical Distributed Fog Computing Architecture for Big Data Analysis in Smart Cities,” ACM ASE Bigdata & Socialinformatics, pp. 1-6, 2015. Mehdipour, Farhad, Javadi, Bahman, Mahanti, Aniket, “FOG-engine: Towards Big Data Analytics in the Fog,” IEEE 14th Intl Conf on Dependable, Autonomic and Secure Computing, pp. 640-646, 2016. P. G. V. Naranjo, M. Shojafar, et .al, “A new Stable Election-based Routing Algorithm to Preserve Aliveness and Energy in Fog-Supported Wireless Sensor Networks,” IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 2413-2418, 2016. L. Pu, X. Chen, J. Xu, X. Fu, “D2D Fogging: An Energy-Efficient and Incentive-Aware Task Offloading Framework via Network-assisted D2D Collaboration,” IEEE Journal on Selected Areas in Communications, vol. 34, no. 12, pp. 3387-3901, 2016. S. Wang, X. Huang, et .al, “CachinMobile: An Energy-Efficient Users Caching Scheme for Fog Computing,” IEEE/CIC International Conference on Communications in China (ICCC), pp. 1-6, 2016. L. Zhou, B. Zheng,et .al, “Toward Efficient Service in Cloud: From the Perspective of Scheduling,” International Conference on Computing, Networking and Communications (ICNC), pp. 939-943, 2012. T. Slavice, Y. Kenji, et .al, “Software-Defined Fog Network Architecture for IoT,” Wireless Personal Communications, vol. 92, no. 1, pp. 181-196, 2017.
