sensors Article
Collaborative Working Architecture for IoT-Based Applications† Higinio Mora 1, * 1 2 3 4
* †
ID
, María Teresa Signes-Pont 1 , David Gil 1
ID
and Magnus Johnsson 2,3,4
ID
Department of Computer Science Technology and Computation, University of Alicante, 03690 Alicante, Spain;
[email protected] (M.T.S.-P.);
[email protected] (D.G.) Department of Intelligent Cybernetic Systems, NRNU MEPhI, 115409 Moscow, Russia;
[email protected] Department of Philosophy, Lund University Cognitive Science, 22362 Lund, Sweden Magnus Johnsson AI Research AB, 24334 Höör, Sweden Correspondence:
[email protected]; Tel.: +34-96590-3400 This paper is an extended version of our paper published in Mora, H.; Signes-Pont, M.T.; Gil-Méndez, D.; Ferrández-Pastor, F.J. A Proposal for a Distributed Computational Framework in IoT Context. In proceedings of the conference on the Ubiquitous Computing and Ambient Intelligence. Springer: Cham, Germany, 2017.
Received: 14 April 2018; Accepted: 21 May 2018; Published: 23 May 2018
Abstract: The new sensing applications need enhanced computing capabilities to handle the requirements of complex and huge data processing. The Internet of Things (IoT) concept brings processing and communication features to devices. In addition, the Cloud Computing paradigm provides resources and infrastructures for performing the computations and outsourcing the work from the IoT devices. This scenario opens new opportunities for designing advanced IoT-based applications, however, there is still much research to be done to properly gear all the systems for working together. This work proposes a collaborative model and an architecture to take advantage of the available computing resources. The resulting architecture involves a novel network design with different levels which combines sensing and processing capabilities based on the Mobile Cloud Computing (MCC) paradigm. An experiment is included to demonstrate that this approach can be used in diverse real applications. The results show the flexibility of the architecture to perform complex computational tasks of advanced applications. Keywords: embedded systems; internet of things; mobile cloud computing; computer modelling; sensor processing modeling
1. Introduction The new era of Knowledge Society has brought advanced services for improving the quality of life of citizens and making better use of resources. These services are based on modern paradigms of Information and Communication Technologies (ICT) such as the Internet of Things (IoT) and the Cloud Computing paradigms. In this way, new concepts have been created which apply IoT to benefit different areas of society and industry. For instance: ambient assisted living [1], smart cities [2] or smart logistics [3]. In these scenarios, new data management issues arise for integrating environmental sensor data efficiently and handling data from different sources [4]. Recent applications have been developed around the aforementioned concepts where sensing and processing capabilities of the devices play an important role. These devices are usually embedded systems and/or mobile devices such as smart phones, wearables, laptops, tablet PCs, etc. To deploy complex artificial intelligence applications in IoT environments provides a powerful driver for
Sensors 2018, 18, 1676; doi:10.3390/s18061676
www.mdpi.com/journal/sensors
Sensors 2018, 18, 1676
2 of 18
increased edge computing capabilities. Real-world use cases of artificial intelligence combining with the Internet of Things [5]. This evolution promotes a digital transformation of the society by providing the citizens and professionals with advanced applications for sensing and analyzing data on the ground. Due to the recent successes and the anticipated breakthroughs in different fields, it has now become one of the most promising research areas [6]. Indeed, this fact is largely accelerated by new smartphones and communication capabilities. However, the design of advanced IoT-based applications remains a challenge [7]. Handling simultaneous data flows, data processing and/or complex mathematical function execution could overflow the computing capabilities of the embedded systems and mobile devices. One approach to overcome this drawback consists in designing a distributed system where the sensor devices are the distributed part to acquire the data and a centralized infrastructure that performs the hard processing. The classical client/server architecture has been designed for that purpose. Currently, this centralized infrastructure is usually deployed in the Cloud [8]. However, this shift introduces several new risks, and some bottlenecks and delays may result from the communications among the devices and the centralized system. In particular, the latter drawback is strongest for multimedia data, for example, in applications that use video and image acquisition devices. For that reason, it is difficult to implement a centralized multimedia analysis system in the cloud [9,10]. To overcome these bottlenecks and delays, this work extends our recent proposal of a distributed architecture [11] to perform collaborative work for IoT-based environments and sharing the application workload among the available devices. This improved architecture takes into account the different network layers and their computing platforms involved, from the remote Cloud servers to connected smart sensors and “things”. The approach aims at optimizing the use of computational resources of an IoT environment while providing a framework able to obtain data from sensors, perform complex computational tasks and run advanced applications. Our working hypothesis is that the conception and development of processing models based on schemes of collaborative working and cloud computing can supply the necessary processing power for running applications when they run on embedded devices with limited performance. The auxiliary use of cloud computing infrastructure on demand will provide flexibility to perform the necessary tasks as well as mechanisms to support the service quality maintenance. This must be accomplished even with devices and sensors with low computing capability. The main contribution of the paper is the proposal of a distributed architecture that combines sensing and processing at different levels of the network to perform a collaborative work based on the Mobile Cloud Computing (MCC) paradigm. This approach can be used in a diversity of real applications running in different environments under different conditions where a set of computing systems are available. To validate the proposal, this research includes a set of experiments with an open dataset split into several subsets to enable parallel testing. The main goal of this splitting is to prove that the collaborative work does affect neither the final accuracy nor the rules and knowledge obtained. The rest of the paper is organized as follows: Section 2 describes the basic background of distributed computing for IoT; Section 3 defines the collaborative working architecture; Section 4 describes the validating experiments; and finally, Section 5 draws the relevant conclusions and presents future work. 2. Background of IoT Distributed Computing Nowadays it is very common that, due to the high performance of the servers, there is not a strong need of taking the time to design a good plan where all the resources are used. However, it is a reality that smart devices come continuously with greater computing capabilities, and moreover, other conditions can exist that prevents the use of external cloud servers. Distributed computing is a field of computer science studying distributed systems. A distributed system can be of a variety of possible configurations, and its components can be of different size and performance such as servers, workstations, personal computers, mobile computers, smart devices and
Sensors 2018, 18, 1676
3 of 18
smart sensors. The goal of distributed computing is to make such a network work in a collaborative way, that is, run as a single system. In this way, the components of a distributed system are located on networked computers, communicating and coordinating their actions by means of exchanging messages to meet a common goal [12,13]. In addition to the existing variety of embedded systems, the mobile devices, such as smartphones, tablets, laptops, wearables, etc., play an important role in the development of IoT solutions. Mobile devices have evolved considerably in recent years as a result of improvements in mobile technology, mobile networking and mobile computing. Examples of such improvements are increased processing power of mobile devices, novel forms of user interaction and new connectivity protocols. Hardware improvements have motivated the explosive growth of mobile applications, especially those that can perform off-screen processing, reducing battery consumption. These processing features are commonly adopted by mobile applications or ‘Apps’ such as music players and activity trackers that are widespread in the smartphone user community. Nowadays, the rapid expansion of smartphone ownership across the world reaches 80–90% of penetration rate in the most advanced markets [14]. This trend places in the hands of users a powerful tool for accessing to the Cloud services and IoT resources to run sophisticated applications. In this way, significant IoT opportunities are starting to emerge for citizens. The IoT environments are highly distributed architectures and they can be considered as very weakly-coupled computer systems. Usually, the connected devices are heterogeneous and present a dynamic behaviour. In the view of the recent works on this topic, there are four major research lines in distributing application systems for IoT environments that are transversal areas in the design of many advanced applications such as Artificial Intelligence applications, games, e-business and e-commerce, etc. These lines are the following: (i) framework design for distributed computing, (ii) integration with Cloud Computing resources or MCC concept, (iii) security, and (iv) distributed applications design. There are a lot of research focused on all these topics. The next subsections describe the recent works on these issues. Table 1 summarizes some relevant and representative contributions. 2.1. Framework Design for Distributed Computing The framework is basically related to modelling the distributed system, the communication protocols, access and discovery services and the design of the scheduling method of the tasks along the whole system. This issue may include the specification of the devices involved and the requirements of the system operation such as the Quality of Service (QoS) [15,16], the definition of task and the application constraints [17,18]. As a representative example, a distributed framework based on IoT is proposed in [19] for monitoring human biomedical signals in activities involving physical exertion. The major advantages and novelties of this scheme is the flexibility in computing the health application by using resources from available devices inside the body area network of the user. Other approaches develop the concept of Virtual Sensor as an abstraction of real sensors, reproducing their logical behaviour and enriching their functionalities with programmable operators [20,21]. The use of virtual sensors has increased continuously for translating the passive sensors to smart things and to simplify creating and configuring complex applications. This evolution enables better sensor management capability and facilitates the distribution of data and computations among them. New middlewares for distributed environments aim to improve the integrated management of heterogeneous data, resources and events as well as to provide resource discovery and interoperability within the diverse applications and services. Recently, there have been a number of proposals for middlewares tackling with several aspects of IoT distributed systems [22]. Finally, recent trends are pushing the processing capabilities to the edge of network where data analytics and knowledge generation are performed by heterogeneous things [23]. This approach
Sensors 2018, 18, 1676
4 of 18
achieves a dramatic reduction in latency and ensures the security of locality information [24]. Regarding to this issue, intensive research is carried out to adapt the job-based scheduling methods to embedded devices and types of applications involved in the IoT environments to fully utilize the nodes and to achieve performance improvements [15,25–28]. 2.2. IoT and Cloud Computing Combination The cloud computing is considered as resource provisioning paradigm in distributed systems. In this way, the cloud infrastructure can centralize most part of the processing cost. The combination of IoT and Cloud Computing generates synergy for both paradigms and makes the objects smarter [29]. Cloud-based intelligence provides value to IoT systems since datacenters can compute the complex machine learning and big data processing methods to infer meaning from the raw data [5]. This integration called CloudIoT allows to provide powerful applications to the users in several real-world domains [30–33]. However, a system can only handle so much data before the system slows down and latency grows to an unacceptable point for the application. Recent approaches try to leverage the computing power of the devices and to outsource the workload to the cloud only when necessary. This trend is called Mobile Cloud Computing (MCC). This paradigm is a promising way to improve the performance as well as reducing the power consumption of a “thing” by executing some parts of the application on a remote server. This combines cloud computing, mobile computing and wireless networks to bring rich computational resources to mobile users, network operators, as well as cloud computing providers [34,35]. In this way, the distribution of the processing between devices of the IoT and cloud computing resources is able to increase the capabilities of the IoT system and to achieve greater overall performance in application execution. There are several techniques to improve the efficiency and effectiveness of the offloading process, such as multi-criteria decision analysis [36], stochastic analysis [37], or application-oriented [38]. The efficient dynamic allocation of tasks is a very important and difficult topic on IoT environments. Sometimes synchronization between the server-side layer and the devices or between devices is needed. In this regard, push notifications are the delivery of information from a software application to a computing device without a specific request from the device. They originate from the server-side layer and they are used to receive processing alerts in the device. An important advantage of push notifications in MCC is that the technology does not require that device’s Apps are open in order to receive these alerts [39]. This allows smartphones and wearables to receive and display text message alerts even when their screens are locked and the app that receives the push notification is closed. 2.3. Security Presently, the security is one of the most restrictive concerns for technological development in areas such as IoT or Cyber-Physical Systems [40]. This issue gains even more importance when the processing load is shared with cloud resources [41] and MCC paradigm [42]. The most important aspects in these fields are guaranteeing user privacy, data confidentiality and the provision of application security that uses cloud resources. These requirements play an essential role as nowadays IoT is characterized by heterogeneous technologies, which concur to the provisioning of innovative services in various application domains [43]. Due to this complexity of the nature of distributed IoT paradigm it is very challenging to design fully secure methods and protocols to detects and prevents security vulnerabilities and attacks [44]. In addition, the new European General Data Protection Regulation (GDPR) [45] imposes restrictive requirements to data processing, especially when the data come from human monitoring. GDPR compliance is particularly challenging in IoT environments, because it can be difficult to gain the consent needed to process the personal data. Therefore, a GDPR awareness stage is needed in the IoT system design process to analyze its compliance taking into account the technologies involved
Sensors 2018, 18, 1676
5 of 18
and the processing made [46]. User privacy challenges arise even when generating information or inferences about the user behaviour [47]. However, security and privacy can also be improved with IoT distributed systems. The GDPR prohibits the transfer of data to countries that don’t have an adequate level of data protection [48]. Currently, cloud providers can deploy their datacenters infrastructure out of ‘safe’ regions. In this way, the collaborative work among the objects helps protect the sensitive data by keeping it at the source rather than sending that information to the cloud [49]. In addition, the distributed nodes can be used for enhancing the overall security of the network. Therefore, there are works that propose to distribute the security checking workload over the network and/or a cloud computing server. There exist multiple options in relation to what network nodes should be responsible for the execution of the security processes [50]. 2.4. Distributed Applications Design The applications for IoT are essentially distributed systems by nature. In this way, IoT is becoming an emerging Internet-based information architecture to allow data and information flows from the real world to advanced users and industrial applications. Application design relies heavily on thing programming close to the operating system level, which reduces efficiency and reliability of the IoT application [51]. The design complexity is reduced significantly, when using web services [52]. There are several relevant works to update knowledge and current challenges. For example, the research presented by [53] introduces a cloud-based platform for the deployment of distributed IoT applications. The main characteristics of this platform are that each object is an autonomous social agent; the platform as a service (PaaS) model is fully exploited; reusability at different layers is considered; the data is under control of the users. There are also commercial frameworks and platforms designed for developing and running Internet of Things applications [54]. The importance of this platforms makes it possible to develop standards-based endpoints and data warehousing, that will enable secure interoperation of modularized and distributed applications. These solutions can also support big data processing and runtime autonomic management [55]. The feature of reusability is very significant as it allows the programmers to generate templates of things and their services. Connected things may use different types of protocols and connectivity patterns. In this way, design patterns add an abstract layer and facilitate to make designs robust and reusable solutions [56]. Distributed applications classically record a numerous set of events, messages and log files, making these platforms an excellent data source for process mining tools [57]. Nevertheless, understanding the behaviour of all these applications, very often with heterogeneous devices, from their event logs can be a complex as well as error prone task. In this area, the rapid advances in interoperability and information integration methods have driven massive growth in the use of integrated information systems for IoT applications [58]. Finally, centralized approaches have been the prevailing choice for providing intelligence based on the acquired data. But, beyond response times and network loads, some distributed applications need a decentralized intelligence approach in order to better meet the environment and application requirements. There are proposals to move intelligence to the edge in order to offer low-level intelligence for IoT applications [59]. In this line, decentralized multiagent systems provides ways of dealing with autonomy and heterogeneity [60]. In addition, this approach enables discovery and selection of IoT devices and data resources.
Sensors 2018, 18, 1676
6 of 18
Table 1. Recent advances on IoT distributed computing. Research Line
Main Contribution Area
(i) Framework design for distributed computing Managing the quality of experience in the multimedia IoT [16] QoS-Aware scheduling of services-oriented IoT [17] Distributed computational model for shared processing [18] IoT-Based Computational Framework [19] A scalable IoT framework using virtual sensor [20] Middleware for Internet of Things [22] MinT: Middleware for Cooperative Interaction of Things [23] Integration of Edge, IoT and the Cloud [24] Scheduling internet of things Apps in cloud computing [25] Payload-size and deadline-aware scheduling [26] Task Requirement Aware Pre-processing and Scheduling [27] Flexible framework for real-time embedded systems [28]
Quality of Experience Scheduling method Distributed computing model Distributed computing model Virtual sensor framework Middleware Middleware Edge of Things Scheduling method Scheduling method Scheduling method Scheduling method
(ii) Integration with Cloud Computing resources IoT and Cloud Computing [29] Machine learning for IoT [5] Model of Internet of Things and Cloud (IoT-Cloud) [30] A study on cloud-based Internet of Things: CloudIoT [31] Integration of Cloud computing and IoT [32] Cloud Computing and Internet of Things Integration [33] Framework for computation offloading [34] MCC for computation offloading [35] Multi-Criteria Decision Analysis Methods [36] Stochastic Analysis of Delayed Mobile Offloading [37] Application-oriented offloading [38] Mobile Cloud Services [39]
General analysis Cloud-based Intelligence Mobile cloud computing General analysis Survey General analysis Mobile Cloud Computing Mobile Cloud Computing Offloading process analysis Offloading process analysis Offloading process analysis Mobile Cloud Services
(iii) Security Trust computation models for service management in IoT [40] Secure integration of IoT and Cloud Computing [41] Security and privacy challenges in MCC [42] Security, privacy and trust in IoT [43] Cyber security framework for IoT-based Energy Internet [44] Fog computing security [49] Distributed intrusion detection system [50] GDPR and the Internet of Things [46] Normative challenges of identification [47]
Survey IoT-Cloud security MCC security Survey Intelligent Security System Fog computing security Distributed system security GDPR GDPR
(iv) Distributed applications design Design flow for web service applications [51] The web of things [52] Cloud-based platform for distributed IoT applications [53] Commercial frameworks for the IoT [54] A Self-Managing Containerized IoT Platform [55] IoT Design Patterns [56] Data Mining proposal of distributed applications events [57] Open IoT Ecosystem [58] Future Internet of Things Controller [59] IoT and Multiagent Systems [60]
Model-based design Web service -based design Deployment platform Survey of design platforms Design platform Design patterns Data Mining Deployment platform Decentralized Intelligence Decentralized Intelligence
2.5. Findings After reviewing the representative proposals in this field, some findings can be identified that justify and summarize our contributions to previous works:
•
The number of connected things is increasing significantly. This increases the possibilities of designing advanced applications that take advantage of their ubiquitous sensing and computing possibilities.
Sensors 2018, 18, 1676
•
•
7 of 18
The computing resources of the whole network can be used for enhancing the performance of IoT-based applications by sharing the processing load among the available platforms, and a way to leverage more intensively the deployed infrastructure. Despite the progress achieved by recent research, the proper distribution of the application workload remains a challenge. There is a lack of formalization and commonly agreed mechanisms to implement real collaborative applications for IoT environments.
Sensors 2018, 18, x FOR PEER REVIEW
7 of 18
7 of 18 This work fits perfectly with Framework design for distributed computing. In this area, Sensors 2018, 18, x FOR PEER REVIEW 7 of 18 the aforementioned findings lead the us to advance continuously development of new and architectures. Basically, contributions presented in inthe Table 2 summarize the frameworks research line in Sensors 2018, 18, FOR PEER REVIEW architectures. contributions inxTable 2 summarize the research line in e the research lineand in different topics.Basically, This is a the challenging issue,presented and therefore, progresses are slowly made. Most of the and2018, architectures. Basically, the contributions presented in Table 2 summarize the research line Sensors 18, x FOR PEER REVIEW of 18 in topics. is a challenging progresses slowly made. Most of7 edge the ly made. Most of different the proposals areThis focused on a specificissue, topic and suchtherefore, as modelling [18,19], are quality of experience [16], of different topics. This is a challenging issue,and andarchitectures. therefore, progresses made. Most of the Basically,are theslowly contributions presented inPEER Table 2 su Sensors 2018, of 18, x FOR REVIEW proposals are focused on a specific topic such as modelling [18,19], quality of experience [16], edge perience [16], edge of things [24], middleware [22,23], and scheduling method [17,25–28]. proposals are focused on a specific topic such as modelling [18,19], quality of experience [16], edge of different topics. This is a challenging issue, and therefore, progresses a and architectures. Basically, the contributions presented in Table 2 summarize the research line in things [24], middleware [22,23], and scheduling method [17,25–28]. In this line, the research presented represents a step forward in designing collaborative schemes things topics. [24], middleware [22,23], andissue, scheduling method [17,25–28]. proposals are progresses focused on are a specific such as modelling [18,19], qualth different This is a challenging and therefore, slowlytopic made. Most of the and architectures. Basically, In this the research represents a step forward in designing schemes ollaborative schemes of IoT line, applications by presented sharing the application’s workload between collaborative the IoT devices of the In this line, theon research presented represents a step forwardquality in[22,23], designing schemes things [24], middleware andcollaborative scheduling method proposals are focused a specific topic such as modelling [18,19], of experience [16], edge of [17,25–28]. different topics. This is a chall applicationsAbynew sharing the application’s workload IoT devices of the environment. IoT devices of of theIoT environment. application formalization and between schedulerthe module is introduced to handle the of IoT applications by sharing the application’s workload between the IoT devices of the In this line, the research presented represents a step forward in things [24], middleware [22,23], and scheduling method [17,25–28]. proposals are focused on a des spe application formalization scheduler module introduced to handle the working The duced to handle A thenew working collaboration amongand heterogeneous thingsis and other networked resources. environment. A new application formalization scheduler is introduced to schemes handle the of IoT applications by sharing the application’s workload betwe In this line, the research presented represents aand step forward inmodule designing collaborative things [24], middleware [22,23 collaboration among and other networked resources. The components of the ked resources. The components of theheterogeneous scheduler are things detailed and a practical use case is described. collaboration amongthe heterogeneous things other networked resources. Thethe research environment. A and new application andthe scheduler module of working IoT applications by sharing application’s workload between the formalization IoT devices In of this line, p scheduler are detailed and a practical use case is described. components of the scheduler are detailed and a practical use case is described. working collaboration among heterogeneous things and other environment. A new application formalization and scheduler module is introduced of to handle the IoT applications by shar Table 2. Time estimation of fatigue analysis application. of theapplication. scheduler are detailed and a practical is des working collaboration Table among heterogeneous things and other networked resources. The Ause environment. newcase applicat 2. Time estimationcomponents of fatigue analysis Table 2. Time estimation of fatigue analysis application. Computing Platform Frame Computing Cost Threshold = 5 components of the scheduler are detailed and a practical use case is described. working collaboration amon 11 Table 2. Time of fatigue analysis applicaa d=5 Classroom 25 s ~2 min Computing PlatformMobile PCFrame Computing Cost Threshold =estimation 5 components of the scheduler Computing Platform Frame Computing Cost Threshold = 5 1 1 Table of fatigue analysis application. ~2 n 1 Time estimation Classroom Tablet PC 50 s ~4 min 25 s 25 Classroom Mobile PC2. Frame Computing Cost T Classroom Mobile PC 1 s Computing Platform ~2min min 11 1 Table n Classroom Smartphone 50 s ~4min min 1 50 s ~4 Classroom Tablet PC 1 ClassroomPlatform Tablet PC Frame Computing 50 s Classroom ~4 min Mobile 25 s Computing Cost Threshold =PC 5 1 2 2 50 s ~4 min Classroom Smartphone n School Workstation 5 min 25 min 1 Classroom Smartphone ~4 min Tablet PC 50 s Computing Classroom Mobile PC 1 11 1 25 50 s s Classroom~2 min 2 5 min 25 School Workstation n Classroom1resources 13 s ~1min min 1 2 1 School Workstation 5 min 25 min Classroom Smartphone 50 s Classroom M Classroom Tablet PC33 ~4 ~1 min 13 s50 s min Classroom resources n Cloud Server 1 1 25 s + 5 s 2.5 min 2 3 Classroom resources ~1 min School Workstation 5 min Classroom T 25 s + 50 5 s 13 min Cloud Server Smartphone Classroom s s ~42.5 min 11 Total time for 25 students. 22 Total time for 12 classrooms of 25 students. 33 Total time for 12 n 32 1 1 Total time for 25 students. 2 Total time 3 Server 25 s + 5 s 2.5 min Classroom resources 13 sClassroom Sm SchoolCloud Workstation 5 min 25 min for 12 classrooms of 25 students. Total time for 12 classrooms of 25 students classrooms of 25 students plus communications delay. Total time for 12 plus communications delay. 3 1 Total time for 2 3 1 Server 25 s + 5 sSchool Work 25 students. classrooms ofCloud 25 students. Classroom resourcesTotal time for 12 13 s ~1 min Total time for 12 2 Total time for 12 classrooms of 25 stu classrooms of 25 students plus 3communications1 delay. Total Cloud Server 25 s + 5time s for 25 students. 2.5 min Classroom re 3. Distributed Computational Architecture 3. Distributed Computational Architecture 1 Total time for 25 students. 2 Total time for 12classrooms 3 of 25 students plus communications delay. Cloud Se classrooms of 25 students. Total time for 12 3. classrooms Distributed Computational Architecture delay. 1 Total time for 25 student of 25 students plus communications 3.1. 3.1. General Scheme General Scheme 3. Distributed Computational Architecture classrooms of 25 students pl 3.1. Scheme TheGeneral primary objective of the Architecture proposed distributed architecture is to take advantage the deployed The primary objective of the proposed distributed architecture is to takeofadvantage of the 3. Distributed Computational 3.1. General Scheme ke advantage of infrastructure the deployed infrastructure of things andofthe cloud resources to reduce to thereduce computing costs and things andcomputing the cloud computing resources the 3. computing costs Distributed Computational The primary objective of the proposed distributed architecture is to take advantage of the the computing costs improve the Scheme overall The main to share the application’s workload between and improve the performance. overall performance. The idea mainisidea is to share the application’s workload between 3.1.deployed General infrastructure of things and the cloudThe computing the computing costs architecture primaryresources objectivetoofreduce the proposed distributed ’s workload between the server-side and and the rest of things withwith computing capabilities suchsuch as smartphones, wearables, the server-side the rest of things computing capabilities as smartphones, wearables, 3.1. General Scheme resources to and improve the overall performance. The main idea is to share the application’s workload between deployed infrastructure of things and the cloud computing The primary objective of the proposed distributed architecture is to take advantage of the rtphones, wearables, tablets, smartsmart sensors, and other devices. devices. This workload-sharing among theamong things enables tablets, sensors, and embedded other embedded This workload-sharing the things the server-side and the of things computing as smartphones, wearables, and improvecapabilities the overall such performance. The main idea is to share the app deployed infrastructure of rest things and thewith cloud computing resources to reduce the computing costs The primary objective o g among the things a horizontal to mitigate costs,torather than resort remote servers. Thus, accordance withThus, our in enables scaling a horizontal scaling mitigate costs, to rather than resort to in remote servers. smart sensors, and other embedded devices. This workload-sharing among the infrastructure things the idea server-side andthe theapplication’s rest of things with computing capabilitiesofsuch andtablets, improve the overall performance. The main is to share workload between deployed thin te servers. Thus,proposal, in accordance these kinds devices perform more processing tasksperform than the more server-side layer. In addition, with of our proposal, these kinds of devices processing tasks than the a horizontal scaling to mitigate costs, rather than resort remote servers. Thus, the inThis tablets, smart sensors, and embedded devices. workload theenables server-side and the rest of things with computing capabilities such astoother smartphones, andwearables, improve overall perfo ssing tasks than cloud the server-side layer. In addition, cloud is available to use onlycase as aoflast if needed. In computing is available to use onlycomputing as a last resort if needed. In the theresort asynchronous accordance ourand proposal, these kinds of devices more processing tasks than theand enables a horizontal scaling to mitigate costs, rather than tablets, smart with sensors, other embedded devices. Thisperform workload-sharing among the things the server-side theresort rest oft t resort if needed.synchronization In the case of needs the asynchronous between cloud server computing and the between cloudsynchronization server computingneeds and the different devices, our system develops server-side layer. Inscaling addition, computing is available useproposal, only as athese lastservers. resort ifThus, needed. accordance with our kinds of devices performand moro enables a horizontal to cloud mitigate costs, rather than to resort to remote in In tablets, smart sensors, computing and athe different devices, our approach. system develops a push notification-based approach. push notification-based the case with of theour asynchronous synchronization needslayer. between cloudprocessing server computing the to use In addition, cloud computing isand only accordance proposal, these kinds server-side of devices perform more tasks than the enables aavailable horizontal scaling In this section, a model of computation suitable for applications IoT applications is defined according to that In this section, a model of computation suitable for IoT is defined according to that different layer. devices, system develops a push approach. theisnotification-based case of theto asynchronous synchronization needs between cloud server-side In our addition, cloud computing available use only as a last resortaccordance if needed. In with our proposa ned according to that architecture. proposal focuses distributed applicationsthat thatcan canbe berepresented representedby byaa graph graph 𝔸𝔸 = architecture. TheThe proposal focuses onon distributed applications this section, a model ofsynchronization computationdifferent suitable for IoT applications is defined that In addition, ourcloud system develops a according push notification-based appro the caseInof the asynchronous needsdevices, between server computing and to the server-side layer. sented by a graph 𝔸𝔸 == {𝕌𝕌, where: { , 𝔽𝔽}} where: architecture. The proposal ona distributed can be a graph 𝔸𝔸 IoT = asynchronou Inapplications this section, that a model of represented computation suitable application different devices, our system focuses develops push notification-based approach. theby case of for the 𝔽𝔽}this architecture. The proposal focuses on distributed can d b • {𝕌𝕌, represents theof execution unitsunits of theofapplication. Therefore, IoTthe application can be can ourthat • In 𝕌𝕌where: represents the execution the application. Therefore, IoTdifferent application section, a model computation suitable for IoT applications isthe defined according toapplications that devices, system {𝕌𝕌, 𝔽𝔽} where: e IoT application can broken down into a list of execution units: = {u , u , . . . , u }. be broken down into a list of units:0𝕌𝕌 = {u00, uthat un−1 }. represented by a graph n−1 22,n…, architecture. The proposal focuses onexecution distributed be =can a model of In this𝔸𝔸section, −1can • 𝕌𝕌 represents the execution units applications of the 2application. Therefore, the IoT application 𝔽𝔽 represents the data flows exchanged between the execution units. The data flows set proposal focu where: architecture. The •{𝕌𝕌,•𝔽𝔽} represents the data flows exchanged between the execution units. The data flows set be broken down into a list of execution• units: 𝕌𝕌 = {u0, urepresents 𝕌𝕌 2, …, un−1}. the execution units of the application. There s. The data flows set the precedence the precedence between the execution units and the volume of exchanged data. F(i,j) ∈where: 𝔽𝔽 defines {𝕌𝕌, 𝔽𝔽} between the execution units and the volume of exchanged data. F(i,j) ∈ defines be broken down into a list the ofunits. execution units: 𝕌𝕌can = set {u0, u2, …, un−1}. 𝔽𝔽 represents representsthe theexecution data flows exchanged between the execution data flows • • 𝕌𝕌 units of the application. Therefore, IoT The application data. F(i,j) ∈ 𝔽𝔽 defines the volume the volume of data exchanged between the execution unit j. of data exchanged between the•units execution unitrepresents i andi and j.of exchanged precedence between theexecution execution data.exchanged 𝔽𝔽 defines between the execut be the broken down into a list of units: 𝔽𝔽 𝕌𝕌and = {uthe •F(i,j)𝕌𝕌∈ represents the }. the data flows 0, uvolume 2, …, un−1 the volume of data exchanged between the execution unit i and j. the precedence between the execution units and the volume of exch The execution units of an application are related to its capacity of processing data and tasks in be broken • 𝔽𝔽 represents the data flows exchanged between the execution units. The data flows set down into a lis thethe volume of data exchanged between the execution unit i and ing data and tasks in parallel. It is a very important feature forunits modern machine learning and bigdata. data approaches on IoT theThe precedence between the execution and volume of exchanged F(i,j) ∈ 𝔽𝔽 defines • represents thej. d execution units of an application are related to its capacity of processing data and tasks in ta approaches on IoT applications, since theexchanged edge things can increase significantly the performance and costs ofthe theprecedence system the volume of data between the execution unit i and j. between parallel. It is a very important feature for modern learning and data approaches on IoT Themachine execution units of anbig application are related to its capacity oft d costs of the system without having to send the data to the server for a centralized processing. For example, Figure 1 shows of data excha volume applications, since the of edge can increase significantly theimportant performance anddata costs ofthe the system It very feature for modern machine learning and The execution units an things application areparallel. related to isitsa capacity of processing and tasks in mple, Figure 1 shows
buted architecture is to take advantage of the and architectures. Basically, the contributions presented in Table 2 puting resources to reduce the computing costs different topics. This is a challenging issue, and therefore, progresse a is to share the application’s workload between proposals are focused on a specific topic such as modelling [18,19], q ng capabilities such as smartphones, wearables, things [24], middleware [22,23], and scheduling method [17,25–28]. ces. This workload-sharing among the things Sensors 2018, 18, 1676 8 of 18a step forward in d In this line, the research presented represents ther than resort to remote servers. Thus, in of IoT applications by sharing the application’s workload bet vices perform more processing tasks than the environment. A new application formalization and scheduler mod vailable to use only as The a lastexecution resort if needed. In application are related to its capacity of processing data and tasks in units of an working collaboration among heterogeneous things and othe eds between cloud serverItcomputing and the feature for modern machine learning and big data approaches on IoT parallel. is a very important components of the scheduler are detailed and a practical use case is cation-based approach. applications, since the edge things can increase significantly the performance and costs of the system for IoT applications is defined according to that without having to send the data to the server for a centralized processing. ForTable example, Figure 1 shows 2. Time estimation of fatigue analysis app plications that canthree be represented by a graph(𝔸𝔸i=) modeled according to this principle where the fragmentation feature of cases of applications Computingamong Platform Frame Computing Cost data generates execution units and opens more processing opportunities the things. Sensors 2018, 18, xmore FOR PEER REVIEW 8 of 18 1 Classroom Mobile PC 25 s e application. Therefore, the IoT application can Classroom Tablet PC 1 50 s ··· u3n-4 50 s = {u0, u2, …, un−1}. 𝔸𝔸1 ≡ u10 u32 ··· u1n-1 u11 u12 1 Classroom Smartphone between the execution units. The data flows set School Workstation 2 5 min the volume of exchanged data. F(i,j) ∈ 𝔽𝔽 defines u33 1 ··· u3n-3 𝔸𝔸3 ≡ u30 Classroom u31 resources 13 s cution unit i and j. 3 ··· u2n-2 Cloud Server 25 s + 5 s u22 1 Total time for 25 students. 2 Total time for 12 classrooms of 25 d to its capacity of processing data and tasks in ··· u3n-2 u34 u21 𝔸𝔸2 ≡ u20 achine learning and big data approaches on IoT classrooms of 25 students plus communications delay. ficantly the performance and costs of the system u ··· u2n-1 PEER REVIEW 8 of 18 23 ··· u3n-1 u35 3. Distributed Computational Architecture tralized processing. For example, Figure 1 shows
this principle where the fragmentation feature ofof different flow possibilities in our proposed distributed system. Figure Figure1.1.Diagram Diagram of different possibilities our proposed distributed system. ···General uin3n-4 3.1. Scheme ··· u u u Sensors 2018,flow 18,ux32FOR PEER REVIEW processing opportunities among the things. 11 12 1n-1
The primary objective of the proposed distributed architectu accordancewith with this this model, model, the the devices devices ⅅ involved involved in IoT are as InInaccordance in the the IoT applications applications aredefined defined ··· u u 𝔸𝔸 ≡ u u deployed infrastructure of things and the cloud computing resource 33 3n-3 u32 3 30 ··· u1n-1 u11 u12 𝔸𝔸131≡ u10 asfollows: follows: ··· u2n-2 u22 and improve the overall performance. The main idea is to share the • Let LetS Sbebethe theset setofofsensor sensordevices. devices.These Thesedevices devices donot not havecomputing computing capabilities themselves. • themselves. server-side the restcapabilities of things with computing capabilities s u34 the···do u3n-2have and u21 u33 𝔸𝔸 ≡ u31 u30 3 Their work consists sensing and communicating the data to other devices the cloud. Their work consists inin sensing and communicating data to other devices oror the cloud. tablets,the smart sensors, and other embedded devices. This workl uThis u2n-2 the things that have 22 • Let Let P the be set theofset of available computing platforms. set···the includes •··· P be available computing platforms. This set includes things that have processing enables a horizontal scaling to mitigate costs, rather than resor u23 u2n-1 ··· u u 35 P u 3n-1 u34 m processing The capabilities. The of the set alsodata acquire the datait. andthese process it. of devices perform 𝔸𝔸 ≡P setu20 21 can the 2devices capabilities. devices of the can also acquire and accordance with ourprocess proposal, kinds LetC be C the be the of cloud computing resources. In the thisremote set the remote servers where isthe server-side layer. In addition, cloud computing available to use o • • Let set ofset cloud computing resources. In this set servers where the processing 1. Diagram of different flowprocessing possibilitiesload in our proposed distributed system. ··· u u is outsourced are located. 23 asynchronous 2n-1 the case of the synchronization needs between u35 clo load is outsourced are located. That is: ⅅ = {S}S⋃ {P}S⋃ {C} different devices, our system develops a push notification-based app with this model, the devices That is: ⅅ =involved {S} {P}in the {C} IoT applications are defined as In thisofsection, a flow model of computation suitable for IoT applicat Figure 1. Diagram different possibilities innetwork. our proposed system The elements of those sets are interconnected creating the IoT communication The distributed architecture. The proposal focuses on distributed applications that ca The elements of those sets are interconnected creating the IoT communication network. architecture model distributes execution units {𝕌𝕌} across the available devices ⅅ according to their t of sensor devices.The These devices do not have computing capabilities themselves. {𝕌𝕌, 𝔽𝔽} where: In accordance with this model, the devices ⅅ involved in the IoT applications a architecture model distributes execution units { } across the available devices according to their particular capabilities and the application constraints. Figure 2 illustrates this idea. nsists in sensing and communicating the data to other devices or the cloud. follows: constraints. Figure 2 illustrates this idea. particular capabilities and the application • 𝕌𝕌 represents the execution units of the application. Th set of available computing platforms. This set includes the things that have be broken down into adevices list of execution units: 𝕌𝕌 = {u0,capabilitie u2, …, un−1 • Let S be the set of sensor devices. These do not have computing pabilities. The devices of the P set can also acquire the data and process it. C1theexchanged • 𝔽𝔽 in represents the data flows between Their work consists sensing and communicating data to other devicesthe orexe the C platforms set of cloud computing resources. In this set the remote servers where the theavailable precedence between the execution units the volume of e • Let P be the set of computing platforms. This setand includes the thin d is outsourced are located. the volume of data between the execution and processing capabilities. The devices ofexchanged the P set can also acquire the dataunit andiproc ⋃ {P} ⋃ {C} • Letcloud C be the set of cloud computing resources. In this set the remote serve The execution units of an application are related to its capacity of those sets are interconnected creating the IoT communication network. The are located. processing load is outsourced cloud parallel. It is a very important feature for modern machine learning distributes execution units {𝕌𝕌} across the available devices That is:ⅅⅅaccording = {S} ⋃ {P}to⋃their {C} ··· this idea. applications, since the edge things can increase significantly the perfo ies and the application constraints. Figure 2 illustrates to send the data to the serverthe for IoT a centralized process The elementswithout of thosehaving sets are interconnected creating communication cases ofexecution applications (𝔸𝔸i{𝕌𝕌} architecture modelthree distributes units across the available devices ⅅ acco ) modeled according to this principle wh P platforms data generates more execution units and opens more processing oppo particular capabilities and the application constraints. Figure 2 illustrates this idea. C1
···
cloud S platforms
···
P2
P1
P3
C1
C platforms
···
cloud
S1
(a)
cloud
S2
(b)
Figure 2. IoT communication network. (a) General scheme; (b) Example case. Figure 2. IoT communication network. (a) General scheme; (b) Example case.
cloud
···
···
2 P3 new things can appear and disappear. The IoT environments since P1 have a dynamicPbehaviour Therefore, a discovery service Pofplatforms things is needed to conduct the registration of a new device and unregistration when it is not available. This service builds the set ⅅ and, therefore, it plays a P significant role in design of IoT applications as they can allow clients and applications to1 access ···
P2
u33used ···to process u3n-3 an he “Response_time” of a computer is defined proportionally execution timeportion required 𝔸𝔸3 ≡ ofuby The “Transfer_rate” isthe defined the availableubut bandwidth communicate through the network, so it doestonot take intoasaccount network latencies, 31 only local 30 cted benchmarkresources, execution mostly unit in being the computer (0 and 1 are mapped to situations where the ··· u u execution unit. This portion the processor. 22 includes the 2n-2input and the output data of the execution unit. The ··· u3n-4 u ···benchmark urespectively). uavailable u12portion 𝔸𝔸1 ≡ method u10 busy, 11 as the 1n-1 ssor is practically idle unacceptably The task does for quantifying “Transfer_rate” willbandwidth take not intoused account the 32u characteristics Theand “Transfer_rate” is defined of to process an ··· u3n-2of the u21 𝔸𝔸2 ≡ u20 34 unicate throughexecution the network, it does not take intonetwork. account network but only local interconnecting The current rate can be for each unit.soThis portion includes the input and latencies, thetransfer output data of theknown execution unit.device The by monitoring the u33operation u3n-3 rces, mostly being the processor. Sensors 2018, 1676 network.“Transfer_rate” 9the of 18··· conditions 𝔸𝔸3 data ≡ the u31 with the uin30accordance Other methods can to ··· useinto historical in method for18,quantifying will characteristics of u23betake u2n-1account ··· u u3n-1 35 he “Transfer_rate” is defined network. as each the portion ofsituation, available used tofor process an ···averaged uor22bandwidth case to usecan data retrieved on regular checks. Inthe many cases, the set of interconnecting Theand current transfer rate beu2n-2 known each device by monitoring tion unit. This portion the output data ofain the execution unit. The devices Pand andto S21use are connected to Wireless Localwith Areathe Network (WLAN), therefore, the latency network.includes Other methods can be historical data accordance operation conditions in ··· u3n-2 uand uthe uinput 34 system. 2≡ 20 The IoT𝔸𝔸environments have a Diagram dynamicofbehaviour since new things can appeardistributed and disappear. Figure 1. different flow possibilities in our proposed d for quantifying “Transfer_rate” will take into account the characteristics of the is reduced and the transfer rate of the network is very often cases, higherthe than connections through each case and situation, or to use averaged data retrieved on regular checks. In many set of Therefore, a discovery service of things is needed to conduct the registration of a new device and onnecting network. The Pcurrent transfer ratewith can beWireless known for each device by monitoring the Internet cloud servers. devices and S are connected to athe Local Area Network (WLAN), and therefore, the latency ···builds uservice 23model, 2n-1 ··· defined withThis this the udevices involved in the IoT applications are unregistration whenInit accordance is not available. the set ⅅ and, therefore, it plays a significant u35 u3n-1 as rk. Other methods can be toand use the historical data in accordance with operation conditions in connections Other components can be the defined in order to identify different performance behaviour. For is reduced transfer rate of the network is very often higher than through follows: role in design of IoT applications as they can allow clients and applications to access available resources ase and situation, or to use averaged data retrieved on regular checks.ofInthe many cases, the set Sensors of 2018, PEER REVIEW example, ‘Power_consumption’ things, ‘Usage_pricing’ of 18, thex FOR resources, ‘Security_risk’ of Internet with the cloud servers. and data provided by things. This discovery service flow can be centralized [61]proposed and/or decentralized [62]. Figure 1. Diagram of different possibilities in our distributed system. es P and S are connected to acomponents Wireless Local Area Network therefore, the • sharing Let S bethe thedata, set of sensor devices. These devices dolatency not have computing capabilities themselves. etc. Other can be defined in(WLAN), order to and identify different performance behaviour. For There2018, are18, several proposals in the literature focused on this service [63,64]. In thisarchitectures. work, it is supposed Sensors x FOR PEER REVIEW 9Basically, ofor 18the cloud. and the contributio uced and the transfer rate of the network is very often higher than connections through Their work consists in sensing and communicating the data to other devices Every device the IoT environment,of dthe i ∈ resources, ⅅ, has a‘Security_risk’ performance of level, which can be example, ‘Power_consumption’ of the of things, ‘Usage_pricing’ In accordance with this model, the devices involved indifferent the IoTtopics. applications are defined as issue, that a proper discovery service is running to maintain the set ⅅ updated. This is a challenging et with the cloud servers. Let P be thebyset of available Thisexpression: set includes the things that have characterized a function “Perf”computing defined byplatforms. the following sharing the data,• etc. follows: distribute resourcesininananefficient efficientway wayand andmeet meetthe theapplication applicationrequirements, requirements, the resource ToTo distribute resources the resource proposals are focused on a specific Other components can be defined order to identify different performance For capabilities. Thedidevices theabehaviour. Pperformance set can also acquire data and be process it. topic suc Every device inofprocessing the IoT environment, ∈ ⅅ, of has level, the which can Perf: ⅅproposed ⟶ 𝕍𝕍. (2) utilization must be properly characterized. Forreason, this reason, a 𝕍𝕍vector is proposed defined utilization must be properly characterized. For this a vector is defined as a vectoras things [24], middleware [22,23], and schedu ple, ‘Power_consumption’ of the things, ‘Usage_pricing’ of the resources, ‘Security_risk’ of • Let C be the set of cloud computing resources. In this set the remote servers where the • Let S be the set of sensor devices. These devices do not have computing capabilities themselves. characterized by a function “Perf” defined by the following expression: a vectorofdomain of features relevant modelling features modelling the behaviour of the computing load on each device. domain relevant the behaviour of the computing load on each device. For this line, theor research presented g the data, etc. processing load isinoutsourced arecommunicating located. Their work consists and the data other devices the The evaluation ofsensing the function Perf can be known priorto toIn the execution of thecloud. application orrepr can Perf:on, ⅅ on, ⟶could 𝕍𝕍.could (2) For applications such asones the ones this work focuses be defined as a domain of vectors with applications such as the this work focuses 𝕍𝕍 be defined as a domain of vectors with of IoT applications by the app very device of the IoT environment, dis: i the ∈ ⅅ, a⋃ performance which device can be ⅅ =over {S}has ⋃time. {C}example, • Let P be set of{P} available computing platforms. This set includes things that have beThat dynamic For alevel, smartphone will show high the values for sharing Response_time if twocomponents componentsininthe therange range[0,[0,1],1],with withthe thefollowing followingsemantics: semantics: two environment. A new application formaliza cterized by a function “Perf” defined by the following expression: processing capabilities. ofcase, thetoP setperformance can also acquire the data it.periodically to itofisthe busy with user In thisprior the mustand be updated The evaluation function Perf activity. canThe be devices known the execution of function the application orprocess can The elements of those sets are interconnected creating the IoT communication network. The collaboration heterogen • time. Let C the cloudOther computing resources. Instable thisworking set remote servers the make the rightset decisions. devices could performance levels due towhere their dedicated be dynamic over Forbe example, a=of smartphone device will showhave high values forthe Response_time if among 𝕍𝕍 Response_time × Transfer_rate. (1) Perf: ⅅ ⟶ 𝕍𝕍.distributes (2)available devices ⅅ according = Response_time × Transfer_rate. (1) architecture model execution units {𝕌𝕌} across the to their components of theinscheduler areTable detailed an processing load outsourced are located. use or their capacity for new function tasks. The static can beperiodically stored a Look-Up (LUT) it is busy with user activity. In thisiscase, theabsorbing performance must bedata updated to particular capabilities andisthe application constraints. Figure 2 illustrates this idea.by The “Response_time” of a computer defined proportionally to the execution time required he evaluation ofmake the function Perf can be known prior to the execution of the application or can That is: Other ⅅevaluation =of {S}devices ⋃ {P}of⋃could {C}function. for fast the the decisions. have stable performanceto levels due to their dedicated Theright “Response_time” aunit computer defined proportionally the to execution time required aFor selected benchmark execution inshow the is computer (0 and 1 are mapped situations where the Table 2. Time estima namic over time.use example, a smartphone device will high values forcan Response_time if Once the computational load is characterized with normalized and relative values, each device or their capacity for absorbing new tasks. The static data be stored in a Look-Up Table (LUT) by a selected benchmark execution unit in the computer (0 and 1 are mapped to situations where The elements those sets arebusy, interconnected creating the IoT communication processor is practically idle andofunacceptably respectively). The benchmark does notnetwork. The usy with user activity. In this case,of the performance functionshows must be updated periodically to units task of the IoT environment its ability to run execution (u) in 𝕌𝕌 for the application in a forthe fast evaluation the function. Computing Platform processor is practically idle distributes and unacceptably busy, respectively). Theavailable benchmark task does not architecture model execution units {𝕌𝕌} across the devices ⅅ according to their communicate through the network, so it does not levels take into account network latencies, but only local C1 C platforms the right decisions. Other devices could have stable performance due to their dedicated Sensors Once 2018, 18, x FOR PEER REVIEWload and 8device of 18 homogeneous comparable way. the computational is characterized normalized and relative values, but eachonly communicate through the processor. network, so it does notwith takeconstraints. into accountFigure network latencies, local Mobile PC 1 Classroom particular capabilities and the application 2 illustrates this idea. resources, mostly beingThe the their capacity for absorbing new tasks. static data candescribes be stored in a Look-Up Table (LUT) The next section how to perform the scheduling of the execution among ofresources, the IoT environment shows its ability to run execution units (u) in 𝕌𝕌 for the application in a units mostly being the Classroom Tablet PC 1 the The “Transfer_rate” is processor. defined as the portion of available bandwidth used to process an t evaluation of the function. available computing things andofremote servers. homogeneous and comparable way. as the The “Transfer_rate” is u defined available bandwidth execution Smartphone 1 ··· anunit. uexecution uClassroom ··· the u11 portion uportion 𝔸𝔸 uload 32process 3n-4 The execution This includes input and the values, output datadevice ofused the to 12 1n-1 1≡ 10 unit. Once the computational is section characterized with normalized andthe relative each cloud The next describes how to perform scheduling of the execution units among the 2 unit. This portion includes the input and will the output into data of the execution unit. The method School method for quantifying “Transfer_rate” account the C1forWorkstation C platforms cloud IoT environment showscomputing its ability to runScheduler execution units (u) in take 𝕌𝕌 for the application in acharacteristics of the 3.2. IoT available things and remote servers. quantifying “Transfer_rate” will take into account the be characteristics of the interconnecting Classroom resources 1 interconnecting forueach device the ··· unetwork. u33by monitoring uknown 3n-3 geneous and comparable way. network. The current transfer rate𝔸𝔸can 3≡ 31 30 The current transfer rate canproposed be to known for each device monitoring the the network. Other methods can workload Cloud Server 3 across The needs an IoT scheduler to operation distribute the application ···execution network. Other methods can use historical inby accordance with conditions in ···architecture ube u2n-2 he next section 3.2. describes how to perform the of thedata units among the 1 IoT Scheduler 22 scheduling 2 Total in be to useand historical dataavailable into accordance with the conditions each and situation, orexecute to use Total time 25of students. time the devices ofdata theoperation environment. The in aim of case this module is the tofor the process a each case situation, or use averaged retrieved on regular checks. In many cases, set ble computing things and remote servers. cloud ··· u u u 𝔸𝔸2averaged ≡ The u20proposed 34 3n-2 21 P platforms data retrieved on regular checks. In many cases, the set of devices P and S are connected to classrooms of 25 students plus communica collaborative way taking into account the limitations of and the devices and their performance rates. an IoTLocal scheduler distribute the application workload across devices P and S arearchitecture connected toneeds a Wireless Area to Network (WLAN), therefore, the latency cloud areduced Wireless Local Area (WLAN), and the latency isisreduced and the transfer rate Usually, the scheduler module of an operating system is a the time-consuming the availableand devices ofNetwork the environment. Thetherefore, aimisofvery this module to execute process in Paofprocess, especially is the transfer rate of the network often higher than connections through T Scheduler 2 P3 P1 ···workload uaccount u 23 2n-1···is presented Distributed Computational Architecture ··· best the network is very often than connections through Internet with the3. cloud umake u3n-1 when a higher complex and isand needed to the choices. In this case, the collaborative taking into the limitations of the devices their performance rates. 35 servers. Internet with way the cloud servers. he proposed architecture needs IoT scheduler to defined distribute application across Other components be inthe order to identify different behaviour. proposed module designed to feasible decisions inperformance theprocess, least possible amount of time, even if Usually, theanscheduler module ofisan system isworkload a time-consuming especially Other components cancan be defined inoperating order tomake identify different performance behaviour. For 3.1. General Scheme vailable devices when of the environment. The aim of this module is to execute the process in a P platforms For example, ‘Power_consumption’ of the things, ‘Usage_pricing’ of the resources, ‘Security_risk’ of they are not optimal. The components of this module are shown in Figure 3. a complex workload is presented and is needed to make the best choices. In this case, the Figure 1. Diagram of different flow possibilities in our proposed distributed system. example, ‘Power_consumption’ of the things, ‘Usage_pricing’ of the resources, ‘Security_risk’ of orative way taking into account the limitations the devices their performance rates. amount sharing the data, proposed module isetc. designed toofmake feasibleand decisions in the least possible time, even sharing the data, etc. The of primary objective of theP propose P2 if P1 S platforms ··· 3 S Usually, the scheduler module of an operating system is a time-consuming process, especially In accordance with this model, the devices ⅅ involved in the IoT applications are defined Every device of the IoT environment, dimodule ∈di , ∈hasare a performance level,3.which can1which be characterized ··· S2 they are not device optimal. of this shown Figure Every ofThe thecomponents IoT environment, ⅅ, has a in performance level, can asbe of things deployed infrastructure and the c a complex workload presented and is needed to following makebythe best choices. In this case, the follows: by a is function “Perf ” defined by the expression: and improve the overall characterized by a function “Perf” defined the following expression: (a) (b) performance. The m sed module is designed to make feasible decisions in the least possible amount of time, eventhe if server-side and the rest of things with • Let S be the set of sensor devices. These devices have computing capabilities themselves. Perf: ⅅdo ⟶not 𝕍𝕍..network. (2)(2) 2. IoTin communication (a) General scheme; (b) Example case. re not optimal. The components of this module Figure are shown Figure 3. tablets, smart sensors, and other embedd Their workSconsists in sensing and communicating the data to other devices or the cloud. platforms ··· S1application S2 The function Perf bebe known prior toto the execution ofthe ororcan enables a horizontal scaling to mitigate The evaluation ofthe the function Perfcan can known prior the execution ofthe the application can • Let P evaluation be the setof of available computing platforms. This set includes things have The IoT environments have a dynamic behaviour since new things that can appear and disappear. bebedynamic over time. For example, a smartphone device will show high values for Response_time if accordance with our proposal, these kind dynamic over time. For example, a smartphone device will show high values for Response_time if (a) processing capabilities. devices service of the P setthings can also acquire to theconduct data and process it. (b) of a new device Therefore, aThe discovery of is needed the registration and it is busy with user activity. In this case, the performance function must be updated periodically to server-side layer. Inthe addition, cloud compu is busy with user this case, the performance function must be updated periodically to • it Let C be the set activity. of cloudIn computing resources. In This this set the remote servers unregistration when it is not available. service builds the setExample ⅅwhere and,case. therefore, it plays a Figure 2. IoT communication network. (a) General scheme; (b) make the decisions. Other could tototheir dedicated thedue case oftheir the asynchronous synchroniz make theright right decisions. Otherdevices devices couldhave havestable stableperformance performancelevels levels due dedicated processing load is outsourced located. significant role inare design of IoT applications as they can allow clients and applications to access use or their capacity for absorbing new tasks. The static data can be stored in a Look-Up Table (LUT) different devices, our system develops a pu use or their capacity for absorbing new tasks. The static data can be stored in a Look-Up Table (LUT) That is: ⅅ = {S} ⋃ {P} ⋃resources {C} available and data things. This service can beand centralized [61] The IoT environments have provided a dynamicby behaviour sincediscovery new things can appear disappear. for fast evaluation of the function. In this section, a model of computation for fast evaluation of the function. and/or decentralized [62]. There are several proposals in the literature focused on this service [63,64]. Therefore, a discovery service of things is needed to conduct the registration of a new device and The elements of those setsload areisinterconnected creating the IoT and communication network. The Once the values, each device architecture. The proposal Once thecomputational computational load ischaracterized characterizedwith withnormalized normalized andrelative relative values, each device focuses on distrib In distributes this work, when itexecution is supposed a across properThis discovery isthe running maintain the setitⅅplays updated. unregistration it is units notthat available. serviceservice builds set ⅅtoand, therefore, a architecture model {𝕌𝕌} the available devices ⅅ according to their ofofthe itsits ability to to runrun execution units (u) (u) in 𝕌𝕌 for for thewhere: application in ain 𝔽𝔽} theIoT IoTenvironment environmentshows shows ability execution units in {𝕌𝕌, the application significant role in design of IoT applications as they can allow clients and applications to access particular capabilities and the application homogeneous andand comparable way. a homogeneous comparable way. constraints. Figure 2 illustrates this idea. • 𝕌𝕌service the execution un available resources and data provided by things. This discovery canrepresents be centralized [61] The Thenext nextsection sectiondescribes describeshow howtotoperform performthe thescheduling schedulingofofthe theexecution executionunits unitsamong amongthe the be focused broken down a list[63,64]. of execution and/or decentralized [62]. There are several proposals in the literature on thisinto service available availablecomputing computingthings thingsand andremote remoteservers. servers. • 𝔽𝔽 C represents data flows exc In this work, it is supposed that a proper discovery service is running to maintain the set the ⅅ updated. 1 C platforms the precedence between the execution 3.2. IoT Scheduler the volume of data exchanged between The proposed architecture needs an IoT scheduler to distribute the application workload across Thethe execution cloud the available devices of the environment. The aim of this module is to execute process units in a of an application a parallel. It is a very important feature for m cloud collaborative way taking into account the limitations of the devices and their performance rates. applications, since the edge things can incre Usually, the scheduler module of an operating system is a time-consuming process, especially ··· without In having to send when a complex workload is presented and is needed to make the best choices. this case, thethe data to the server three cases of applications (𝔸𝔸i) modeled acco Pproposed platforms module is designed to make feasible decisions in the least possible amount of time, even if data generates more execution units and ope they are not optimal. The components of this module are shown in Figure 3. P1
P2
P3
Sensors 2018, 18, 1676
10 of 18
3.2. IoT Scheduler The proposed architecture needs an IoT scheduler to distribute the application workload across the available devices of the environment. The aim of this module is to execute the process in a collaborative way taking into account the limitations of the devices and their performance rates. Usually, the scheduler module of an operating system is a time-consuming process, especially when a complex workload is presented and is needed to make the best choices. In this case, the proposed module is designed to make feasible decisions in the least possible amount of time, even if they are not optimal. The components of this module are shown in Figure 3.
Sensors 2018, 18, x FOR PEER REVIEW
𝔸𝔸1 ≡
u10
u11
u12
Figure 3. IoT scheduler design.
u22
𝔸𝔸 ≡ u
··· ···
u
u32
u1n-1
u2n-2
𝔸𝔸3 ≡ u30
21 20 When a new execution unit 2(ui ) arrives, it must be evaluated where it should be executed according to the described model. The execution units can be created on the device itself. In any case, ··· u2n-1 u23 the scheduler process is the same. As shown in the previous figure, the IoT scheduler consists of four main components:
u31
u33 u34 u35
Figure 1. Diagram of different flow possibilities in our proposed distributed system
(1)
(2)
The ‘Candidate_devices’ component obtains a list of the available devices that can process the In accordance with thisdevices model,inthe execution unit (ui ). This list comes from the discovered thedevices system ⅅ. involved Accordingintothe theIoT applications a follows: execution unit features, this function selects the feasible devices than can process it. Here, a first device filter is introduced, setting up the best chances for processing. The high • Let S be the set of sensor devices. These devices do not have computing capabilitie heterogeneity of IoT resources needs an information model to represent it containing Their work consists in sensing and communicating the data to other devices or th unambiguous and machine-interpretable descriptions of the available resources. In this way, • Let P be the set of available computing platforms. This set includes the thin the computing platforms discovered should be described in terms of metadata such as resource processing capabilities. The devices of the P set can also acquire the data and proc type, computing power, memory, location, etc. as well as information to reach its exposed services. • Let C be the set of cloud computing resources. In this set the remote serve If there is a new type of device, then it needs to be registered in the system by cataloguing its processing load is outsourced are located. features by type of execution unit. Then, when a new execution unit arrives (ui ), the system That is: ⅅ = {S} ⋃ {P} ⋃ {C} knows the set of devices that it is able to compute. Based on this information, this function selects the set of candidate devices. The elements of those sets are interconnected creating the IoT communication architecture model distributes units across the available When only one device is available for this execution unit, execution it is selected for {𝕌𝕌} processing it. If more devices ⅅ acco particular capabilities and the application constraints. Figure 2 illustrates this idea. than one is listed, it is necessary to decide which to select. The lists of candidate devices are established by default for each type of execution unit according to their capabilities and features. The ‘Performance_function’ component calculates the Perf function for that list of devices. This function determines the best option. For devices with static rates the function can read C platforms
C1
cloud cloud
Sensors 2018, 18, 1676
(3)
(4)
11 of 18
the data from Perf-LUT. This is a fast operation. Next, if the static data is not available, then it is necessary to estimate or evaluate it for each device by means of an evaluation function. These evaluations can be processed in parallel. Finally, once the performance data is ready, the system chooses the best option. If there are no available devices, the task must to be processed on the device itself or an error flag must be raised. The ‘Performance_evaluation’ component evaluates or estimates the performance of the selected devices on_the_fly. The delay at this stage depends on the method used. For example, in order to evaluate network performance, a number of multiplatform tools are available. One simple solution is the Iperf (http://iperf.fr) tool. It allows target nodes to be set by running an Iperf process on each platform. As a result, the response time and the transfer rate can be obtained for each available platform. The delay of this calculation can be constrained and then a suboptimal decision is preferable in order to meet the deadline. At this point, other strategies and policies could be addressed, mainly by adapting the successful results from previous and future research. As a result of our previous research on distributed and mobile systems, in [28] a proposal is introduced that combines imprecise computing strategies with cloud computing, which can be used to design a real-time component. The ‘Perf-LUT’ component stores the static performance data of the devices. It consists of a memory located near the Perf function calculation module with precalculated results for each device and each instance of the vector. The size of this module depends on the knowledge stored about the devices performance. When the list of available devices is fixed and the network performance is stable, for example in a controlled environment, the job of the scheduler module is considerably simplified since it can work as a priority list stored in memory. That is, the devices are ordered by features and performance, and then they are selected according to their availability.
4. Case Study In this section, we present a simple case study where the proposed distributed method comes into play to handle an application where sensing and complex computations take place. The aim is to validate our proposal and show the benefits of the architecture in providing flexibility for sharing the processing of the application. The experiments emphasize the advantages of the proposed distributed architecture model, which is more innovative and useful in an IoT environment compared to a traditional centralized model. The application used as example consists of a method for analyzing attention degree of students in classroom. This information is monitored by the instructor while he/she is giving lessons to the students. In this way, the teacher can know who and when the students are tired or have lost the attention, in a collective and individual way. Please, note that this is only an illustrative example of the advantages of moving the processing load to the edge and promoting the collaborative work among the things around. The added value of this approach mainly falls into emphasize the advantages of the distributed method, being able to test anywhere by professionals, with mobile, tablet or other IoT device. This problem is largely studied in the field of driving security since the degree of attention of the driver is a key feature to prevent car crashes [65–67]. The driver fatigue detectors are including as premium features of high quality cars. These detectors are usually based on analysis of driver facial features such as eyes and yawning. Due to this reason, the typically installed sensor is a digital camera. Translating this problem to the student attention issue, Figure 4 shows a simple scheme of the application.
Sensors 2018, 18, 1676
In this the research presented represents a step forward i different topics. This is line, a challenging issue, and therefore, progresses are slow offocused IoT applications sharing application’s proposals are on a specificby topic such asthe modelling [18,19], workload quality of eb environment. A new application formalization and scheduler m things [24], middleware [22,23], and scheduling method [17,25–28]. working collaboration among heterogeneous things and o In this line, the research presented represents a step forward in designing 12 of 18 components of the scheduler are detailed and a practical use of IoT applications by sharing the application’s workload betweencase the environment. A new application formalization and scheduler module is intr Table 2. Time estimation of fatigue working collaboration among heterogeneous things and otheranalysis netwoa components of the scheduler are detailed and a practicalFrame use case is described. Computing Platform Computing Co
Classroom Mobile PC 1 25 s Classroom Tablet PC 50 s 1 Computing Platform Smartphone Frame Computing Cost Classroom 50 s Thresho 2 1 School Workstation 5 min ~2 m Classroom Mobile PC 25 s 1 1 Classroom 13 s Classroom Tablet PC resources 50 s ~4 m Cloud 1Server 3 25 s + 5 s ~4 m Classroom Smartphone 50 s 2 1 School 2 Total time Workstation 5 minfor 12 classrooms25ofm2 Total time for 25 students. 1 Classroomofresources 13 s delay. ~1 m classrooms 25 students plus communications Cloud Server 3 25 s + 5 s 2.5 m 3. Distributed Computational Architecture 2 Total time Total time for 25 students. for 12 classrooms of 25 students. 3 Table 2. Time estimation of fatigue analysis application. 1
1
classrooms of 25 students plus communications delay.
3.1. General Scheme
3. Distributed Computational The primary Architecture objective of the proposed distributed architec deployed infrastructure of things and the cloud computing resour 3.1. General Scheme and improve the overall performance. The main idea is to share th the server-side the proposed rest of things with computing capabilities The primary objective and of the distributed architecture is to t tablets, smart sensors, and wor deployed infrastructure of things and the other cloud embedded computing devices. resourcesThis to reduc Figure 4. Application scheme: (a) Application context inside classroom; (b) Application steps.is to enables athehorizontal scaling to mitigate costs, than re and improve the overall performance. The main idea sharerather the applicatio accordance withofour proposal, these kinds of devices perform the server-side and the rest things with computing capabilities such as sm server-side In addition, cloud computing available to use Regarding privacy compliance of this application there arelayer. two other issues that must bedevices. met: firstly, tablets, smart sensors, and embedded Thisis workload-sharin the case the asynchronous synchronization consent has to be given since user data (face) is monitored andofscaling processed; student enables a horizontal to secondly, mitigate the costs, ratherfaces thanneeds resortbetween to rem ourthese system develops a push notification-based accordance with ourdevices, proposal, kinds of devices perform more proca must be processed in an anonymous way. That is, different the system can infer when the attention degree In section, a model ofascomputation suitable foronly IoT as applic server-side layer. Inthis addition, computing is available a la of a student is low and raise a flag indicating his/her position in the cloud classroom shown Figure 4.to use architecture. focusesdata. onneeds distributed applications that thestudent, case of nor therelate asynchronous synchronization between cloud serve (red dot), but it cannot identify who is this thisThe withproposal any academic {𝕌𝕌, has 𝔽𝔽}our where: different devices, system develops push notification-based approach. For this application, the list of execution units two dimensions: It aconsists of the list of this section, a model computation suitableafor IoT applications is def application steps depicted by Figure 4b, andInfor each student in theofclassroom. Typically, school • 𝕌𝕌 represents the execution units of the application. classroom can have around 25 students.architecture. The proposal focuses on distributed applications that can be repre be broken down into a list of execution units: 𝕌𝕌 = {u0, u2, …, u {𝕌𝕌, 𝔽𝔽} exchanged where: On the other hand, the data flows between the execution units are frames, • 𝔽𝔽 represents the data flows exchanged between the e frames windows (when the face is located), and data templates. In addition, this application allows the precedence between units the execution units and the volume th o • 𝕌𝕌 represents the execution of the application. Therefore, a high parallel processing since the eyes and mouth can be processed in parallel, and each student can the volume of data exchanged between execution unit i an be broken down into a list of execution units: 𝕌𝕌 = {uthe , u , …, u }. 0 2 n−1 also be analyzed in parallel. This high parallelism facilitates the collaboration work among different • 𝔽𝔽 represents the data flows exchanged between the execution un The execution units of an application are related to its capaci computing platforms. the precedence between the execution units and the volume of exchanged parallel. It iscan a very important feature for modern The available computing platforms for this application be deployed at several layers, machine learnin the volume of data exchanged between the execution unit i and j. applications, since the edgethe things can increase for example, each classroom is equipped with a mobile PC, and, eventually, teacher can havesignificantly the pe without having the to the server centralized proce execution units of to ansend application are related to for its acapacity of proces a tablet PC and a smartphone that might beThe involved in the calculations. Indata addition, the school cases of applications (𝔸𝔸i)modern modeled according to this and principle parallel. It isthree a very important feature for machine big d could have a specific workstation to assist the processing of all classrooms at time, and, in the case oflearning data generates more execution units and opens more processing opa applications, since the edge things can increase significantly the performance overhead, computing services are hired temporarily from an outside cloud server. A high-resolution without having to send the datafor to the server forfeatures a centralized camera is employed to provide frames where detecting and searching several face of all processing. For ex cases aofscheme applications according to this principle where the fr i) modeled students at once. The next figure (Figurethree 5) depicts of the(𝔸𝔸 whole system. This architecture is data generates more execution units and opens more processing opportunities a concretized to the general scheme of the IoT communication network shown in Figure 2a. The value added of the proposed architecture is the collaborative work that takes place under this approach. So that, it does not matter what are the specific distribution of nodes, which can be networked sensors, computing platforms, or cloud servers. The scheme depicted in Figure 5 shows a possible distribution of the devices where each classroom has a particular set of available devices. For example, Classroom1 has three local computing platforms and Classroomm has only two.
Sensors 2018, 18, 1676
13 of 18
Figure 5. IoT application environment.
Formally, the IoT environment consists of the following devices:
• • •
Sensors set {S} = {si : High resolution camera of classroom i} Computing platforms {P} = {pij : computing platform i of classroom j, p: school workstation} Cloud server {C} = {c0 }
The devices of {S} and {P} are in the IoT communications network connected to the school WLAN. The cloud server is accessed through Internet. Based on the experimentation data of previous works [65], the processing time for the face search in the whole image for a single user is around 0.5 s in a standard workstation. It is supposed the same time for mouth analysis in a frame. Table 2 shows a comparative of the estimated calculation time of a frame for each type of computing platform according to standard hardware configuration of each type of device. It is supposed a classroom with 25 students, and a little school with 12 classrooms. In addition, as shown by Figure 4b, fatigue detection of a student needs a positive search in several consecutive frames. For example, for ‘threshold’ = 5 the costs should be five times longer. Communication time between platforms within the WLAN are not taken into account in this comparative. For Cloud computing processing an extra delay should be included to transmit the frames (or frame windows). Another drawback arises concerning the security issue when using Cloud resources. In order to meet the new GDPR requirement regarding user data processing, the cloud infrastructure has to be deployed in safe regions that provide an adequate level of data protection [48]. In this way, to keep the processing load of frames inside the WLAN prevents security risks in processing this student behaviour. The estimative delay times shown by Table 2 give a more precise picture of the situation. The centralized workstation cannot be adopted in many cases due to the long period to detect fatigue in a student. The school could deploy more resource hardware to speed up the calculations. Instead, the proposed IoT architecture can be used to make the most of the existing classroom resources. The combined use of the three computing platforms of the classroom in a collaborative way can drastically reduce the application time. This collaborative work obtains a performance for fatigue analysis application in around 1 min by using only local resources, that is, the classroom mobile PC,
Sensors 2018, 18, 1676
14 of 18
Table PC and smartphone by processing all students in parallel. In addition, the whole school can be processed in about the same time by putting all available devices to work together. The school workstation can be used to assist classrooms, centralize all data and infer statistical information about the interests of each lesson. The execution units involved in this application will be responsible for computing a portion of the dataset, for example, the portion of the frame where each student is placed. Figure 6 shows a scheme of this collaborative work.
Figure 6. Collaborative work example.
The scheduler of the IoT architecture can be computed on the classroom mobile PC. Based on the previous features, a possible implementation of the scheduler components for properly handling this IoT application is the following:
•
•
•
•
Candidate devices: The instructor can have tablet and/or smartphone able to take part of this application. Thus, this module obtains the list of platforms present on the classroom (classroom mobile PC, table PC, smartphone), the school workstation and the cloud server. Performance evaluation module: The performance of the involved devices in this case study is known, and additional devices can not join this application ecosystem in each classroom. Therefore, this function simply obtains the available capacity of every device according to their current workload. Perf-LUT: The smartphone and tablet of the instructors are evaluated offline to update this module. At high level, an example of the content of this module is shown in Table 2. More deeply, this LUT stores the computing time of each stage (execution unit) of the method for analyzing the attention degrees of students. The parallelism feature of this application allows the handling of the portion of frame of each student as an execution unit. Performance function (Perf): The behaviour of this function can be set to allow the processing as close as possible to where the data are obtained. Therefore, the devices are sorted by proximity and the result retrieved from the performance evaluation module. In general terms, the classroom mobile PC should be selected in the first place, next the tablet PC, and next the smartphone. When these devices are busy, the workload can be outsourced to the centralized workstation and cloud server.
This example shows the possibilities of the proposed architecture to deploy and compute applications in a collaborative way. As shown, the architecture leverages the computing capabilities of available IoT devices to speed up the computations and achieve better performance than in a centralized way. In addition, this alternative could not be available due to network congestions, or privacy issues.
Sensors 2018, 18, 1676
15 of 18
For an accurate deployment, a calibration step is needed of the available objects that could take part in the processing, as well as a further application analysis for building the graph of executions units and data flows. This information helps to configure the Performance Evaluation Module of the scheduler. In addition, the application requirements and the working environment constraints play an important role in designing the collaborative network and proper profiling the scheduler. The implementation of the execution units would also have to take into account limited-resource target devices such as wearables, sensing devices and other mobile computers. In these cases, some key features would be considered such as the enhanced communication capabilities and battery consumption (if applicable). 5. Conclusions In this paper, a wide review of the state of the art of distributed computing for IoT systems is described. It is a common issue that computing requirements for monitoring and advanced analysing of the data acquired by IoT environments usually exceed the capabilities of the sensors and even the mobile computers. In this work, a distributed architecture that combines sensing and processing at different levels of the network to share the computing load among the available devices has been proposed to address this challenge. The IoT environments composed of wearables and other biosensors may benefit from this architecture by allowing the processing of advanced applications with real-time constraints in a collaborative way. Besides summarizing the relevant and representative contributions, as well as the background of IoT distributed computing, the research presented in this paper represents a further step for designing collaborative schemes of IoT applications. The main advantage of the proposed system is that it enables real-time monitoring and analysis of all the acquired data by networked devices. In addition, it provides flexibility in executing the application by using resources from cloud computing services, but also from other available computers that are recommendable for their cost reduction or better availability. The key novelties are the application formalization and its decomposition in execution units, and the new scheduler module. This is the core component of the architecture to distribute the execution units and dispatch them according to the computing capabilities of each device. A simplified case study is included to demonstrate that our approach is possible to implement in any area. In this case, we have used an educational environment as IoT is drastically changing the education around the world. We would like to emphasize that the aim of this case study is just to present our general model and particularized to the IoT. Further work must be invested in building a proper predictive model of the available resources. System resilience is another interesting issue that should be further developed. The proposed framework could support a fallback policy, to be activated when central control is not available or accessible. This point could be expanded by adding a service discovery mechanism, allowing computers to autonomously take suboptimal decisions based on local information coming from neighbouring devices. Currently, a larger amount of data is being gathered, including open data, in order to construct a more challenging experimental scenario. Author Contributions: All authors were involved in the foundation items. All authors wrote the paper and read and approved the final manuscript. Acknowledgments: This work has been funded by the Conselleria de Educación, Investigación, Cultura y Deporte, of the Community of Valencia, Spain, within the program of support for research under project AICO/2017/134. Conflicts of Interest: The authors declare no conflict of interest.
References 1.
Calvaresi, D.; Cesarini, D.; Sernani, P.; Marinoni, M.; Dragoni, A.F.; Sturm, A. Exploring the ambient assisted living domain: A systematic review. J. Ambient Intell. Hum. Comput. 2017, 8, 239. [CrossRef]
Sensors 2018, 18, 1676
2. 3. 4. 5. 6. 7. 8. 9. 10.
11.
12. 13. 14. 15. 16. 17. 18. 19. 20.
21. 22. 23. 24.
25.
16 of 18
Mora, H.; Gilart-Iglesias, V.; Pérez-del Hoyo, R.; Andújar-Montoya, M.D. A Comprehensive System for Monitoring Urban Accessibility in Smart Cities. Sensors 2017, 17, 1834. [CrossRef] [PubMed] Sun, K.; Ryoo, I. A Smart Sensor Data Transmission Technique for Logistics and Intelligent Transportation Systems. Informatics 2018, 5, 15. [CrossRef] Mora, H.; Gilart-Iglesias, V.; Pérez-delHoyo, R.; Andújar-Montoya, M.D.; Gabucio, H.C. Interactive cloud system for the analysis of accessibility in smart cities. Int. J. Des. Nat. Ecodyn. 2016, 11, 447–458. [CrossRef] Mahdavinejad, M.S.; Rezvan, M.; Barekatain, M.; Adibi, P.; Barnaghi, P.; Sheth, A.P. Machine learning for Internet of Things data analysis: A survey. Dig. Commun. Netw. 2017. [CrossRef] Klumpp, M. Innovation Potentials and Pathways Merging AI, CPS, and IoT. Appl. Syst. Innov. 2018, 1, 5. [CrossRef] Shi, F.; Li, Q.; Zhu, T.; Ning, H. A Survey of Data Semantization in Internet of Things. Sensors 2018, 18, 313. [CrossRef] [PubMed] Ruan, J.; Chan, F.; Zhu, F.; Wang, X.; Yang, J. A Visualization Review of Cloud Computing Algorithms in the Last Decade. Sustainability 2016, 8, 1008. [CrossRef] Shukla, A.; Simmhan, Y. Model-driven scheduling for distributed stream processing systems. J. Parallel Distrib. Comput. 2018, 117, 98–114. [CrossRef] Chien, S.-Y.; Chan, W.-K.; Tseng, Y.-H.; Lee, C.-H.; Somayazulu, V.S.; Chen, Y.-K. Distributed computing in IoT: System-on-a-chip for smart cameras as an example. In Proceedings of the Asia and South Pacific Design Automation Conference (ASP-DAC), Chiba, Japan, 9–22 January 2015. Mora, H.; Signes-Pont, M.T.; Gil-Méndez, D.; Ferrández-Pastor, F.J. A Proposal for a Distributed Computational Framework in IoT Context. In Proceedings of the International Conference on Ubiquitous Computing and Ambient Intelligence, UCAmI 2017, Philadelphia, PA, USA, 10–15 July 2015; pp. 194–200. [CrossRef] Cao, Y.; Long, C.; Jiang, T.; Mao, S. Share communication and computation resources on mobile devices: A social awareness perspective. IEEE Wirel. Commun. 2016, 23, 52–59. [CrossRef] Kim, S.; Kim, D.-Y.; Park, J.-H. Traffic management in the mobile edge cloud to improve the quality of experience of mobile video. Comput. Commun. 2018, 118, 40–49. [CrossRef] Zenith’s Mobile Advertising Forecast Report, Smartphone Penetration to Reach 66% in 2018. Available online: https://www.zenithmedia.com/smartphone-penetration-reach-66-2018/ (accessed on 12 April 2018). Li, L.; Li, S.; Zhao, S. QoS-Aware Scheduling of Services-Oriented Internet of Things. IEEE Trans. Ind. Inform. 2014, 10, 1497–1505. [CrossRef] Floris, A.; Atzori, L. Managing the Quality of Experience in the Multimedia Internet of Things: A Layered-Based Approach. Sensors 2016, 16, 2057. [CrossRef] [PubMed] Colom, J.F.; Mora, H.; Gil, D.; Signes-Pont, M.T. Collaborative building of behavioural models based on internet of things. Comput. Electr. Eng. 2016, 58, 385–396. [CrossRef] Mora, H.; Colom, J.F.; Gil, D.; Jimeno-Morenilla, A. Distributed computational model for shared processing on Cyber-Physical System environments. Comput. Commun. 2017, 111, 68–83. [CrossRef] Mora, H.; Gil, D.; Terol, R.M.; Azorín, J.; Szymanski, J. An IoT-Based Computational Framework for Healthcare Monitoring in Mobile Environments. Sensors 2017, 17, 2302. [CrossRef] [PubMed] Kim-Hung, L.; Datta, S.K.; Bonnet, C.; Hamon, F.; Boudonne, A. A scalable IoT framework to design logical data flow using virtual sensor. In Proceedings of the Wireless and Mobile Computing, Networking and Communications (WiMob), Rome, Italy, 9–11 October 2017. Martínez-Rodríguez, M.C.; Prada-Delgado, M.A.; Brox, P.; Baturone, I. VLSI Design of Trusted Virtual Sensors. Sensors 2018, 18, 347. [CrossRef] [PubMed] Razzaque, M.A.; Milojevic-Jevric, M.; Palade, A.; Clarke, S. Middleware for Internet of Things: A Survey. IEEE Internet Things J. 2016, 3, 2327–4662. [CrossRef] Jeon, S.; Jung, I. MinT: Middleware for Cooperative Interaction of Things. Sensors 2017, 17, 1452. [CrossRef] [PubMed] El-Sayed, H.; Sankar, S.; Prasad, M.; Puthal, D.; Gupta, A.; Mohanty, M.; Lin, C.-L. Edge of Things: The Big Picture on the Integration of Edge, IoT and the Cloud in a Distributed Computing Environment. IEEE Access 2017, 6, 1706–1717. [CrossRef] Narman, H.S.; Hossain, M.S.; Atiquzzaman, M.; Shen, H. Scheduling internet of things applications in cloud computing. Ann. Telecommun. 2017, 72, 79. [CrossRef]
Sensors 2018, 18, 1676
26. 27. 28. 29.
30. 31.
32. 33. 34.
35. 36. 37. 38. 39. 40. 41. 42. 43. 44. 45. 46. 47. 48. 49. 50.
17 of 18
Haferkamp, M.; Sliwa, B.; Ide, C.; Wietfeld, C. Payload-Size and Deadline-Aware scheduling for time-critical Cyber Physical Systems. In Proceedings of the Wireless Days, Porto, Portugal, 29–31 March 2017. [CrossRef] Bharti, S.; Pattanaik, K.K. Task Requirement Aware Pre-processing and Scheduling for IoT Sensory Environments. Ad Hoc Netw. 2016. [CrossRef] Mora, H.; Gil Mendez, D.; Colom López, J.F.; Signes Pont, M.T. Flexible framework for real-time embedded systems based on mobile cloud computing paradigm. Mob. Inf. Syst. 2015. [CrossRef] Cavalcante, E.; Pereira, J.; Alves, M.P.; Maia, P.; Moura, R.; Batista, T.; Delicato, F.C.; Pires, P.F. On the interplay of Internet of Things and Cloud Computing: A systematic mapping study. Comput. Commun. 2016, 89–90, 17–33. [CrossRef] Dinh, T.; Kim, Y.; Lee, H. A Location-Based Interactive Model of Internet of Things and Cloud (IoT-Cloud) for Mobile Cloud Computing Applications. Sensors 2017, 17, 489. [CrossRef] [PubMed] Babu, S.M.; Lakshmi, A.J.; Rao, B.T. A study on cloud based Internet of Things: CloudIoT. In Proceedings of the Global Conference on Communication Technologies (GCCT), Kanya Kumari District, India, 23–24 April 2015; pp. 60–65. [CrossRef] Botta, A.; Donato, W.; Persico, V.; Pescapé, A. Integration of Cloud computing and Internet of Things: A survey. Future Gener. Comput. Syst. 2016, 56, 684–700. [CrossRef] Malik, A.; Om, H. Cloud Computing and Internet of Things Integration: Architecture, Applications, Issues, and Challenges. Sustain. Cloud Energy Serv. 2018. [CrossRef] Chen, X.; Chen, S.; Zeng, X.; Zheng, X.; Zhang, Y.; Rong, C. Framework for context-aware computation offloading in mobile cloud computing, Journal of Cloud Computing Advances. Syst. Appl. 2017, 6, 1. [CrossRef] Akherfi, K.; Gerndt, M.; Harroud, H. Mobile cloud computing for computation offloading: Issues and challenges. Appl. Comput. Inform. 2018, 14, 1–16. [CrossRef] Bangui, H.; Ge, M.; Buhnova, B.; Rakrak, S.; Raghay, S.; Pitner, T. Multi-Criteria Decision Analysis Methods in the Mobile Cloud Offloading Paradigm. J. Sens. Actuator Netw. 2017, 6, 25. [CrossRef] Wu, H.; Wolter, K. Stochastic Analysis of Delayed Mobile Offloading in Heterogeneous Networks. IEEE Trans. Mob. Comput. 2018, 17, 461–471. [CrossRef] Tseng, F.; Cho, H.; Chang, K.; Li, J.; Shih, T.K. Application-oriented offloading in heterogeneous networks for mobile cloud computing. Enterp. Inf. Syst. 2017, 1–16. [CrossRef] Gheith, A.; Rajamony, R.; Bohrer, P.; Agarwal, K.; Kistler, M.; Eagle, B.L.W.; Hambridge, C.A.; Carter, J.B.; Kaplinger, T. IBM Bluemix Mobile Cloud Services. IBM J. Res. Dev. 2016, 60, 1–12. [CrossRef] Guo, J.; Chen, I.-R.; Tsai Jeffrey, J.P. A survey of trust computation models for service management in internet of things systems. Comput. Commun. 2017, 97, 1–14. [CrossRef] Stergiou, C.; Psannis, K.E.; Kim, B.-G.; Gupta, B. Secure integration of IoT and Cloud Computing. Future Gener. Comput. Syst. 2018, 78, 964–975. [CrossRef] Mollah, M.B.; Azad, M.A.K.; Vasilakos, A. Security and privacy challenges in mobile cloud computing: Survey and way ahead. J. Netw. Comput. Appl. 2017, 84, 38–54. [CrossRef] Sicari, S.; Rizzardi, A.; Grieco, L.A.; Coen-Porisini, A. Security, privacy and trust in Internet of Things: The road ahead. Comput. Netw. 2015, 76, 146–164. [CrossRef] Sani, A.S.; Yuan, D.; Jin, J.; Gao, L.; Yu, S.; Dong, Z. Cyber security framework for Internet of Things-based Energy Internet. Future Gener. Comput. Syst. 2018. [CrossRef] EU GDPR Portal: Site Overview. Available online: https://www.eugdpr.org/ (accessed on 8 May 2018). Wachter, S. GDPR and the Internet of Things: Guidelines to Protect Users’ Identity and Privacy. SSRN 2018. [CrossRef] Wachter, S. Normative challenges of identification in the Internet of Things: Privacy, profiling, discrimination, and the GDPR. Comput. Law Secur. Rev. 2018. [CrossRef] Sanz, R.M.C. Your Guide to the GDPR, IEEE Spectrum. Available online: https://spectrum.ieee.org/ telecom/internet/your-guide-to-the-gdpr (accessed on 8 May 2018). Khan, S.; Parkinson, S.; Qin, Y. Fog computing security: A review of current applications and security solutions. J. Cloud Comput. 2017, 6, 19. [CrossRef] Colom, J.F.; Gil, D.; Mora, H.; Volckaert, B.; Jimeno, A. Scheduling framework for distributed intrusion detection systems over heterogeneous network architectures. J. Netw. Comput. Appl. 2018, 108, 76–86. [CrossRef]
Sensors 2018, 18, 1676
51. 52. 53. 54.
55.
56.
57.
58. 59. 60.
61.
62.
63.
64.
65. 66. 67.
18 of 18
Lekidis, A.; Stachtiari, E.; Katsaros, P.; Bozga, M.; Georgiadis, C.K. Model-based design of IoT systems with the BIP component framework. J. Softw. Pract. Exp. 2018. [CrossRef] Zeng, D.; Guo, S.; Cheng, Z. The web of things: A survey. J. Commun. 2011, 6, 424–438. [CrossRef] Girau, R.; Martis, S.; Atzori, L. Lysis: A platform for IoT distributed applications over socially connected objects. IEEE Internet Things J. 2017, 4, 40–51. [CrossRef] Derhamy, H.; Eliasson, J.; Delsing, J.; Priller, P. A survey of commercial frameworks for the internet of things. In Proceedings of the IEEE 20th Conference on Emerging Technologies & Factory Automation (ETFA), Luxembourg, 8–11 September 2015; pp. 1–8. Khazaei, H.; Bannazadeh, H.; Leon-Garcia, A. SAVI-IoT: A Self-Managing Containerized IoT Platform. In Proceedings of the 5th International Conference on Future Internet of Things and Cloud (FiCloud), Prague, Czech Republic, 21–23 August 2017. Qanbari, S.; Pezeshki, S.; Raisi, R.; Mahdizadeh, S.; Rahimzadeh, R.; Behinaein, N.; Mahmoudi, F. IoT Design Patterns: Computational Constructs to Design, Build and Engineer Edge Applications. In Proceedings of the IEEE First International Conference on Internet-of-Things Design and Implementation, Berlin, Germany, 4–8 April 2016. Muthusamy, V.; Slominski, A.; Ishakian, V.; Khalaf, R.; Reason, J.; Rozsnyai, S. Lessons learned using a process mining approach to analyze events from distributed applications. In Proceedings of the 10th ACM International Conference on Distributed and Event-based Systems, Irvine, CA, USA, 20–24 June 2016; pp. 199–204. Robert, J.; Kubler, S.; Kolbe, N.; Cerioni, A.; Gastaud, E.; Främling, K. Open IoT Ecosystem for Enhanced Interoperability in Smart Cities—Example of Métropole De Lyon. Sensors 2017, 17, 2849. [CrossRef] [PubMed] Rahman, H.; Rahmani, R. Enabling distributed intelligence assisted Future Internet of Things Controller (FITC). Appl. Comput. Inform. 2018, 14, 73–87. [CrossRef] Singh, M.P.; Chopra, A.K. The Internet of Things and Multiagent Systems: Decentralized Intelligence in Distributed Computing. In Proceedings of the IEEE 37th International Conference on Distributed Computing Systems (ICDCS), Atlanta, GA, USA, 5–8 June 2017. Fortino, G.; Lackovic, M.; Russo, W.; Trunfio, P. A discovery service for smart objects over an agent-based middleware. In Proceedings of the International Conference on Internet and Distributed Computing Systems, Hangzhou, China, 28–30 October 2013; pp. 281–293. Gomes, P.; Cavalcante, E.; Rodrigues, T.; Batista, T.; Delicato, F.C.; Pires, P.F. A Federated Discovery Service for the Internet of Things. In Proceedings of the Workshop on Middleware for Context-Aware Applications in the IoT, Vancouver, BC, Canada, 7–11 December 2015; pp. 25–30. Datta, S.K.; Da Costa, R.P.F.; Bonnet, C. Resource discovery in Internet of Things: Current trends and future standardization aspects. In Proceedings of the IEEE 2nd World Forum on Internet of Things (WF-IoT), Milan, Italy, 14–16 December 2015. Ccori, P.C.; De Biase, L.C.C.; Knorich Zuffo, M.; da Silva, F.S.C. Device discovery strategies for the IoT. In Proceedings of the IEEE International Symposium on Consumer Electronics (ISCE), Sao Paulo, Brazil, 28–30 September 2016. D’Orazio, T.; Leo, M.; Guaragnella, C.; Distante, A. A visual approach for driver inattention detection. Pattern Recognit. 2007, 40, 2341–2355. [CrossRef] Sun, W.; Zhang, X.; Peeta, S.; He, X.; Li, Y. A Real-Time Fatigue Driving Recognition Method Incorporating Contextual Features and Two Fusion Levels. IEEE Trans. Intell. Transp. Syst. 2017, 18, 3408–3420. [CrossRef] Hemadri, V.B.; Kulkarni, U.P. Detection of Drowsiness Using Fusion of Yawning and Eyelid Movements. In Advances in Computing, Communication, and Control; Communications in Computer and Information Science; Springer: Berlin/Heidelberg, Germany, 2013; Volume 361. © 2018 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
