Components of Fog Computing in an Industrial

Fog Networking for 5G and IoT Workshop 2015 (IEEE SECON Workshop)

Components of Fog Computing in an Industrial Internet of Things Context Vangelis Gazis∗ , Alessandro Leonardi∗ , Kostas Mathioudakis∗ , Konstantinos Sasloglou∗ , Panayotis Kikiras∗ , Raghuram Sudhaakar† ∗ AGT

Group (R&D) GmbH, Hilpertstrasse 35, 64295 Darmstadt, Germany {vgazis, aleonardi, kmathioudakis, ksasloglou, pkikiras}@agtinternational.com † Cisco Systems Inc., USA, {rsudhaak}@cisco.com

devices to 50 billion by 2020 and estimate the total value of the IoT market at $14.4 trillion. The impact of the IoT market on the global GDP has been estimated by McKinsey as between $2.7 trillion and $6.2 trillion annually by year 2025 [8]. IDC has further elaborated the distribution of the respective technical impacts across the IoT value chain [9].

Abstract—The “Internet of Things” (IoT) represents an embodiment of the continuous convergence between the physical facet of human activities and its reflection on the information world. In the context of “Industrial Internet”, where the operation of complex physical machinery is integrated with networked sensors and software applications, IoT is understood as a technological enabler of significant advances in the efficiency of industrial processes. In this paper, we elaborate on the main challenges brought on by the Industrial Internet and identify its key enabling technologies relevant to the Fog Computing paradigm. Further on, we propose the introduction of an Adaptive Operations Platform (AOP) to provide end-toend manageability of the enabling Fog Computing infrastructure according to the operational requirements of industrial process. We present our prototype setup for AOP and elaborate on its performance evaluation in a typical industrial scenario. Index Terms—Fog Computing, Internet of Things, Sensors, Actuators, Industrial Internet, SCADA, Control, Processes.

A. Definitions for the Internet of Things The European Commission (EC), in its research agenda, has defined IoT as an integrated part of the Future Internet (FI) where “things having identities and virtual personalities operating in smart spaces using intelligent interfaces to connect and communicate within social, environmental, and user contexts” [10]. The term Web of Things refers to the use of standard technologies in the World Wide Web (WWW) to instrument IoT [11]. The Telecommunication Standardization Sector of the International Telecommunication Union (i.e., ITU-T) defines IoT as a global infrastructure for the information society, enabling advanced services by interconnecting (physical and virtual) things based on, existing and evolving, interoperable information and communication technologies [12]. It becomes thus apparent that the application scope of IoT exceeds that of the World Wide Web by several more application domains (e.g., smart grid, industrial automation, etc.). The rest of the paper is structured as follows: Section II introduces the industrial domain, highlighting the main properties and major shortcomings of infrastructures therein, and, presents the Fog Computing paradigm. Section III provides the motivation for our work and describes an Adaptive Operations Platform (AOP) to address key limitations that an industrial infrastructure poses to data-intensive IoT applications. Section IV presents a deployment case for AOP, followed by evaluation scenarios for two use cases. Finally, Section V concludes the paper and provides directions for future work.

I. I NTRODUCTION After a decade of increasing interest, the Internet of Things (IoT) finds itself at the spotlight of innovation interest as a major disruptive technology [1]. Originally, the term Internet of Things envisioned a world where computers would relieve mankind from the Sisyphean burden of data entry by automatically recording, storing and processing all information relevance to human activities [2]. Recently, additional interpretations of what IoT is about have emerged [3]–[5]. A significant part of IoT concerns the integration between device-oriented sensor networks and data-oriented applications. This covers a diverse set of capabilities (e.g., sensor registration management, sensor performance management, device management, data collection, data aggregation, etc.). Through the IoT infrastructure, state information that reflects significant aspects of daily life can be captured from distributed geographically sensory artifacts and conveyed to applications which drive online processes serving human needs, on the individual, group and social level. The amount of interest in IoT has been recently significantly raised, with Cisco [6] and Ericsson [7] repeatedly acknowledging the importance of IoT and its related technologies. The projections of Cisco set the expected number of interconnected

II. C HALLENGES IN I NDUSTRIAL I OT Recently, the application of IoT in the industrial domain has been gaining momentum. In this direction, the Industrial Internet Consortium was founded in March 2014 by AT&T, CISCO, General Electric, Intel and IBM. Its objective is to foster and promote innovation-driven and industrial oriented

c 2015 IEEE 978-1-4673-7392-0/15/$31.00

978-1-4673-7392-0/15/$31.00 ©2015 IEEE

1

Fog Networking for 5G and IoT Workshop 2015 (IEEE SECON Workshop)

use cases which, in turn, will drive the definition and development of a reference architecture for the Industrial IoT [13].

maximize the elimination of redundant information but do not address the communication impact coming from the data exchanges required in order to assemble and build the global knowledge. The latter concern is more efficiently addressed by data filtering techniques that are distributed throughout the communication infrastructure, ideally being applied on data in transit. This approach may be inherently coarsegrained (i.e., thus not achieving global maxima in eliminating information redundancies) but enables significant benefits on the operational level. For instance, event management in the context of facility operations can leverage the data filtering capacity of network nodes to detect significant events and annotate them with key local information (e.g., network load, etc.), thus enabling a more efficient treatment at the operational level. In the case where network nodes have in-situ capacities for data filtering and data processing, the amount of data transported over the network infrastructure may be significantly reduced. Assuming all the information required to fully define a particular event is available at the topological neighborhood of the network node where that event occurs, this approach could also significantly relieve the administrative burden of the infrastructure’s operational stuff. Ideally, the interaction with the underlying communication infrastructure should be largely independent of the particular applied approach (e.g., by adopting a technology-independent API [15]). This raises the need for a generic framework to inspect and manipulate IoT data while in transit. While the enforcing mechanisms of such a framework must be collocated with other elements of the user plane within networking devices, the respective controlling agent can be distributed independently of the user plane elements.

A. Key features of Industrial IoT Industrial applications are typically founded on a centralized model of data processing. Data generated by industrial devices (which may be geographically distributed over a wide area) are conveyed over the network infrastructure to a central storage facility. The latter is typically embodied as a data center where intensive data processing tasks are carried out [14].

Fig. 1: Centralized (legacy) model of industrial applications.

B. Fog Computing

It is straightforward to understand that, under the centralized model, the communication infrastructure of IoT can become increasingly challenged, as the number of connected things increases. Increases in the volume and the velocity of IoT data exchanged will consume communication resources, in proportion to the number of communicating peers and the diameter of the connecting network topology. Particular types of IoT data (e.g., video streams, realtime measurements) will further stress the communication infrastructure, possibly to the point where resource starvation occurs. Therefore, it is crucial that efficient mechanisms to minimize the traffic load injected into the communication infrastructure and apportion it among its elements according to its particular capabilities and its currently available resources. However, the criteria driving this optimization often require global knowledge of the infrastructure’s conditions and, there, fall within the scope of the platform supporting IoT applications end-to-end. To this end, techniques to reduce the data footprint (e.g., dimensionality reduction, etc.) are crucial in filtering out the noise and removing redundancy. With multiple alternative techniques available, an intelligent approach is required to select the most suitable one according to the particular application. For instance, dimensionality reduction methods that rely on global knowledge may

A technology independent API for data collection at the edge is only possible if we can deploy these small compute modules at the edges of the network where the data sources are present. A study of the network components that are used today reveal that there is a significant amount of spare compute that is left over on the networking devices [16]. It is also worthy to note that the spare resources are in a niche class between the cloud/data center scale and edge devices (e.g. sensors, sensor gateways etc.). This class of compute nodes are called Fog nodes and in general consist the backbone of the Fog Computing Architecture [17]. Fog Computing is a new paradigm that lends itself to distribute first order operations (e.g., filtering, aggregation and translation) on raw data generated by the sources. Firstly, it allows use to detect events of interest and eliminates the need for storing raw data in the cloud. Secondly, it allows for the operational size of the network to increase many fold by reducing the total traffic. Large scale IoT deployments can benefit from leveraging the Fog Computing paradigm. To fully realize the potential of Fog Computing we need to be able to handle data from the highly dissimilar data sources. The challenge here is to handle different protocols (ranging from simple broadcast to stageful mechanisms), data

2

Fog Networking for 5G and IoT Workshop 2015 (IEEE SECON Workshop)

eliminates the needs for implementing protocol drivers thereby significantly reducing the size of the code base. This allows us to operate on devices that have comparatively lower resources, such as the Fog nodes. Secondly, the data can be implicitly normalized by simply applying a format specifier to the query output. Again, this can be done on the fly without the necessity to change code. Thirdly, it offers end users the capability to modify the queries on the fly thereby creating the equivalent of a control loop between the cloud application and the fog node. This capability is very useful in managing data from large scale deployment across diverse geographical areas as it minimized the need to access the device to periodically calibrate them. Instead the cloud application can correct for the calibration changes by modifying the query. In general, generic protocol payload handling is a an area of interest that has significant benefits in managing IoT data. An implementation of this technique called Data in Motion (DMo) [20] is installed on Cisco edge routers, such as the C819 and CGR1000 [21], [22]. The following section develops an industrial solution in the context of predictive maintenance applications, using the Fog computing paradigm and the DMo technology. III. A F OG -E NABLED P LATFORM FOR I NDUSTRIAL I OT A. Motivation In developing a particular product, equipment manufacturers collect the performance data required to understand the statistical properties of the malfunctions it may present within its expected lifetime. This data supports the development of a failure model particular to each product which can be used to estimate the probability of malfunctions and/or failures. Unavoidably, such a failure model is couched upon a set of assumptions regarding the operational context of the respective product. These assumptions are reflected in the product’s blueprint and the operational envelope included in its specifications. However, the actual operational conditions of a product cannot be accurately determined at the time its failure model is developed. On the other hand, operation of a production under conditions that exceed those defined in its specifications, increases the probability of a malfunction and/or failure before a scheduled maintenance. As a result, application of the failure model alone cannot account for the effect of the operational setting. Consideration of additional information about the operational context would be required to remedy this discrepancy and provide the most accurate information regarding future malfunctions and failures. However, such information is under the authority of the operational administration of each particular instance of the product; hence it is not readily available to the equipment manufacturer. By combining the predictive envelope of the manufacturer’s failure model with the detailed operational record of the administrator’s infrastructure, a better modelling fit can be achieved. The significance of operational data is further underlined by the increasing variance in customer requirements placed upon modern automation systems. The latter, originally designed to operate under central control for the production of large

Fig. 2: The Fog Computing paradigm.

formats (simple byte representations to complex schema based formats). As aforementioned, this is difficult problem and when handled in an intelligent manner allows for distributed event detection in the Fog Layer [18]. The following section proposes methods that aim to handle IoT data in a universal manner. Data acquisition and normalization has typically been accomplished by implementing a set of protocol drivers and converters at the gateway devices. Examples of such implementations abound in database integration and data modeling/presentation applications in the cloud. However, this approach does not scale well when the number of protocols that have to be handled is ever changing and numerous. Further, the lower resource availability at the Fog Layer (as compared to the cloud) makes this approach infeasible for practical large scale deployments. We propose a novel technique that treats packet payload as a schema-less database record. As and when packets arrive they are inserted into the local schema-less database [19]. Once inserted into the database and indexed the payloads offer themselves to be queried at the level of complexity offered by the database. The complexity of the query can be adjusted based on the use case at hand or as per the end users requirements. This technique offers multiple advantages over the current state of the art mechanisms that use a driver/translator for each protocol to apply rules on the incoming payloads. Firstly, it

3

Fog Networking for 5G and IoT Workshop 2015 (IEEE SECON Workshop)

quantities of relatively few product types, are now being increasingly tasked to the small scale production of customized items.

types found in the industrial site along with dynamic data collected during the latter’s operation. The former data are accessed from the failure model database (depicted as Model DB in Fig. 3) which provides an interface to equipment manufacturers for registering and updating this information. The latter data are accessed from the equipment performance database (depicted as Performance DB in Fig. 3) whose data are populated with equipment performance data under the control of the industrial infrastructure’s administration. Typically, containing operational data (e.g., measurements of operating conditions, etc.) for the equipment comprised in the infrastructure is commonly found as part of the SCADA historian database. In combining data from the Model DB and the Performance DB, the MB will produce a so-called fused model for each equipment type. The fused model can be understood as refined version of the failure model for the particular industrial infrastructure (i.e., a version of it that better fits the operational context). The Rule Mapper (RM) functional element tasked with mapping the fused model to a set of traffic handling rules understood by the Software Defined Networking (SDN) infrastructure. For each particular equipment type, the RM interacts with the MB over the Iz interface (Fig. 3) to access the proper version of the associated fused model. Traffic handling rules (i.e., DMo rules) persist in the rules database (depicted as Rules DB in Fig. 3) under control of the RM element. The Rule Deployer (RD) functional element which, given a set of traffic handling rules and a description of the capabilities found in the SDN infrastructure, computes the deployment plan to apply this set of rules on the appropriate elements. A database (depicted as Infrastructure DB in Fig. 3) contains descriptions of the devices included in the infrastructure, their deployment layout and interconnections thereof (e.g., networking topology, etc.), along with the associated configuration and provisioning settings. Traffic handling rules are accessed through the RM element over the Iz interface shown in Fig. 3.

B. APO overview We propose the introduction of a so-called Adaptive Operations Platform (AOP), to improve the effectiveness of applying equipment failure models by taking into consideration the deployment and configuration of each particular infrastructure. From a value chain perspective, the platform interfaces to equipment manufacturers and the operational administrations of Supervisory Control and Data Acquisition (SCADA) infrastructures, offering the following capabilities: Enabling the equipment manufacturer to: • Manage the publication (i.e., publish, update and withdraw) of the failure model associated to each particular type of equipment. • Manage the access rights of each managed (e.g., published) failure model. • Record the access and usage of each managed (e.g., published) failure model. Enabling the administration of the SCADA infrastructure to: • Manage the subscription (i.e., subscribe and unsubscribe) to the failure model associated to each particular type of equipment. • Query database of failure models for those matching a given set of properties.

IV. D EPLOYMENT AND E VALUATION Illustrating the advantages gained from our proposed architecture, we consider an exemplary scenario that models the real behaviour of a machinery. Our scenario consist of N sensors monitoring the status of the deployed machinery. Each sensor reports regularly the captured values through a router to a central server, as depicted in Fig. 4. The measurements are stored in a database and are illustrated to an operator. The objective of deploying the sensors and aggregating the measurements is to investigate potential anomalies in the monitored machinery. The investigated scenario is tested using from the one hand a common router (see Fig. 4) and from the other a router/gateway supporting DMo (see Fig. 5), with the aim to compare the centralized scheme to the APO approach. Specifically, we have used a Cisco ISR 819 router supporting the DMo middleware. Regarding the traffic generated by the sensors, we have considered an inter-packet time modeled

Fig. 3: Architecture of the Adaptive Operations Platform (AOP). As depicted in Fig. 3, AOP is structured upon the service capabilities of the following layers: • The Fog Infrastructure that includes networking equipment with particular Fog capabilities and provides for end-to-end communication services. • The Operational Support System (OSS) that leverage the Fog Infrastructure to provide the customary asset management and business support functions (e.g., inventory, maintenance, provisioning, etc.). To efficiently leverage key features of the Fog Infrastructure, AOP includes several functional elements. The Model Building (MB) functional element combines static information about the failure models of the equipment

4

Fog Networking for 5G and IoT Workshop 2015 (IEEE SECON Workshop)

Fig. 4: Scenario using a normal router. Fig. 6: Distribution of measurement values.

a Machine Learning (ML) algorithm (e.g., k-Means) that is trained to understand the normal behaviour of the appliance. Then, using the Rule Mapper (RM), it sets a rule to receive only data outside of the normal range in order to identify anomalies. These data will be sent as events to the operator. In our case, the ML indicates that the average of the received values is close to 60, which corresponds to the normal behaviour, and sends to DMo a new rule that specifies to the router to forward to the server only values that represent anomalies, e.g., in the range [0, 50] and [70, 100]. Obviously, the range can be set differently according to thresholds. The results obtained using this approach are illustrated in Fig. 7. As expected, we obtain a huge decrease in the number of transmitted packets. Fig. 7, also depicts the CPU utilization of the router. We observe that the use of DMo claims only 8% of the CPU for a 1/3 reduction in the received traffic.

Fig. 5: Scenario using a router supporting DMo.

according to a random Poisson process. This allows us to take into account both network delay and possible network congestions. In the following we investigate two use cases; one related to the reduction of received data, and one dealing with filtering the data traffic according to the sources. A. Use Case 1 In this scenario we show how we managed to reduce the amount of data received by the operator (and stored in the Historian) using a router supporting DMo. Let us consider one of the N sensors and suppose that this sensor is monitoring some operational behaviour of a machine (e.g. its operational temperature). Looking at the traffic generated in a time interval equal to 120s, it is possible to see that the measurements are distributed as a Gaussian distribution (see Fig. 6). Specifically, most of the values are in the range [50, 70] which corresponds to the normal behaviour of the specific monitored machine in our scenario. The values out of that range correspond to potential anomalies of the machine. Using a common router (Fig. 4), all the measurements gathered by the sensor are received by the server and stored, filling in most of the cases, the database with redundant information. This also leads to increased bandwidth usage for transferring the data but also it consumes resources and space in the database. Moreover, to detect the anomalies, a process has to take into consideration a large amount of collected data due to the large number of sensors. In order to reduce the amount of data stored, it is possible to apply the AOP approach. In this case, the server runs

Fig. 7: Router CPU utilization and total received packets. B. Use Case 2 The aim of this use case is to demonstrate the capability of the proposed system to change dynamically the rules when needed. Specifically, we want to measure the temperature of N = 6 appliances inside a rack and create an alert to the operator only when the temperature of some appliances is above a certain threshold (anomaly). We need also to identify the appliance that has the anomaly. We use 6 sensors, one for each appliance, and one sensor which measures the temperature inside the rack. As in Use Case 1, the normal behaviour of the appliance is when the

5

Fog Networking for 5G and IoT Workshop 2015 (IEEE SECON Workshop)

a good potential as a key enabling technology to address this challenge. Plans of future work involve the application of Fog Computing in more use cases from the industrial domain. R EFERENCES [1] N. I. C. NIC, “Disruptive Civil Technologies: Six Technologies with Potential Impacts on US Interests out to 2025,” Tech. Rep., Apr. 2008. [Online]. Available: http://www.dni.gov/nic/confreports_disruptive_tech. html [2] “The Internet of Things,” Tech. Rep., 2005. [3] L. Atzori, A. Iera, and G. Morabito, “The Internet of Things: A survey,” Computer Networks, vol. 54, no. 15, pp. 2787 – 2805, 2010. [Online]. Available: http://www.sciencedirect.com/science/article/ pii/S1389128610001568 [4] J. Gubbi, R. Buyya, S. Marusic, and M. Palaniswami, “Internet of things (IoT): A vision, architectural elements, and future directions,” Future Generation Computer Systems, no. 0, pp. –, 2013. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0167739X13000241 [5] D. Miorandi, S. Sicari, F. D. Pellegrini, and I. Chlamtac, “Internet of things: Vision, applications and research challenges,” Ad Hoc Networks, vol. 10, no. 7, pp. 1497 – 1516, 2012. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S1570870512000674 [6] Cisco, “The Internet of Things,” online, Tech. Rep., 2013. [Online]. Available: http://share.cisco.com/internet-of-things.html [7] Ericsson, “More than 50 billion connected devices,” online, Tech. Rep., 2013. [Online]. Available: http://www.ericsson.com/res/docs/ whitepapers/wp-50-billions.pdf [8] McKinsey, “Disruptive Technologies,” online, Tech. Rep., 2013. [Online]. Available: http://www.mckinsey.com/insights/business_ technology/disruptive_technologies [9] Worldwide internet of things predictions for 2015. IDC. [Online]. Available: http://www.idc.com/getdoc.jsp?containerId=prUS25291514 [10] D. G. INFSO, “Internet of Things in 2020: A Roadmap for the Future, INFSO D.4 Networked Enterprise & RFID and INFSO G.2 Micro & Nanosystems in co-operation with RFID Working Group of the European Technology Platform on Smart Systems Integration (EPOSS),” Tech. Rep., 2008. [Online]. Available: http://www.iot-visitthefuture.eu/fileadmin/documents/ researchforeurope/270808_IoT_in_2020_Workshop_Report_V1-1.pdf [11] D. Guinard, V. Trifa, F. Mattern, and E. Wilde, “From the Internet of Things to the Web of Things: Resource-oriented Architecture and Best Practices,” in Architecting the Internet of Things, D. Uckelmann, M. Harrison, and F. Michahelles, Eds. Springer Berlin Heidelberg, 2011, pp. 97–129. [Online]. Available: http://dx.doi.org/10.1007/978-3-642-19157-2_5 [12] ITU-T, Recommendation ITU-T Y.2060, Overview of the Internet of Things, ITU-T Std. Y.2060, June 2012. [13] I. I. Consortium. [Online]. Available: http://www.iiconsortium.org [14] J. Bruner, Industrial Internet. O’Reilly, March 2013. [Online]. Available: http://www.oreilly.com/data/free/industrial-internet.csp [15] oneM2M, oneM2M Functional Architecture Baseline Draft, oneM2M Std., August 2014. [Online]. Available: http://www.onem2m.org/images/files/deliverables/ TS-0001-oneM2M-Functional-Architecture-V-2014-08.pdf [16] M. Hajibaba and S. Gorgin, “A review on modern distributed computing paradigms: Cloud computing, jungle computing and fog computing,” CIT. Journal of Computing and Information Technology, vol. 22, no. 2, pp. 69–84, 2014. [17] F. Bonomi, R. Milito, J. Zhu, and S. Addepalli, “Fog computing and its role in the internet of things,” in Proceedings of the first edition of the MCC workshop on Mobile cloud computing. ACM, 2012, pp. 13–16. [18] M. Khatoon, “Fog computing and its role in internet,” 2014. [19] P. Shvaiko and J. Euzenat, “A survey of schema-based matching approaches,” in Journal on Data Semantics IV. Springer, 2005, pp. 146– 171. [20] “Data in motion (dmo),” https://developer.cisco.com/site/data-in-motion/ index.gsp. [21] “Cisco 819 integration services router,” http://www.cisco.com/c/en/us/ td/docs/routers/access/800/819/software/configuration/Guide/819_SCG/ 1_overview.html. [22] “Cisco connected grid router,” http://www.cisco.com/c/en/us/products/ routers/1000-series-connected-grid-routers/index.html.

Fig. 8: Temeprature values collected from a rack of devices.

average value calculated in 120s is equal to 60. In order to save bandwidth and resources in the server we have created a rule in the DMo router to send to the server only measurements from the rack sensor. Looking at the Fig. 9, in normal conditions we have an amount of packets received equal to 200. Let us consider that after a while (at time 720s), the temperature monitored by the sensor rack increases (from 60 to 80). This provides an estimate that an abnormal event occurs inside the rack without designating which appliance has an anomaly. In this case, the server receives values above the threshold (set to 60) and using the RM it sends a new rule to the DMo for the router to forward the data coming from all the 6 sensors. The result will lead to a temporary increase in the number of received packets, in order to identify the appliance with the anomaly. At this point, the ML identifies the sensor which is sending data above the threshold and sets a new rule which instructs the DMo router to send data only about this sensor (i.e., in addition to the rack sensor). The new rule applies until the situation in the rack goes back to the normal conditions. After that, the operator will receive again the measurements of the rack sensor only.

Fig. 9: Packets received and measured values vs. time from different sources. V. C ONCLUSIONS With IoT applications emerging with an increasing rate in multiple domains, several considerable technical challenges are being raised. Introducing more intelligence in the control of industrial installations so as to improve operational efficiencies while ensuring a good utilization of the communication and computing resources of the infrastructure is a major such challenge. The Fog Computing paradigm demonstrates

6