Distributed Osmotic Computing Approach to

2018 IEEE First International Conference on Artificial Intelligence and Knowledge Engineering (AIKE)

Distributed Osmotic Computing Approach to Implementation of Explainable Predictive Deep Learning at Industrial IoT Network Edges with Real-Time Adaptive Wavelet Graphs Emmanuel Oyekanlu Electrical and Computer Engineering Department Drexel University, Philadelphia Pennsylvania, USA [email protected] with distributed computing, computing tasks are shared among participating devices. In addition, with distributed computing, the network is more secured and resilient to attack, since failure of one node may not lead to collapse of entire network. Common resource-constrained IoT devices will also be able to contribute to reliable IoT distributed edge computing since tasks can be allocated to them based on their respective capacities using an osmotic microservice approach [3], [4]. In osmotic computing, applications are divided into microservices using software defined osmotic layers through which adjustable services are provided based on available software, hardware capacity, and based on needs of different layers of the system [5]. Benefits of osmotic computing for IoT systems includes the provision of service modularity among different IoT systems layers [6]. Osmotic computing also prevents architecture erosion in applications design in addition to enhanced load balancing and system components capacity utilization. By using a distributed osmotic computing approach, highly compute intensive applications, such as deep learning will be progressively shifted from the cloud to the edge and fog layers of IoT. Presently, well-known deep learning solutions such as the Apple Siri and Google Now are still primarily cloud based [7]. In state of the art deep learning application, big data must be transferred from network edge to cloud layer for deep learning implementation. Deep learning approach based on data transmission from edge to cloud layer suffers from communication network bandwidth shortage and there is also persistent concern regarding data security and safety issues, especially when healthcare data sets are considered [7]. Also, to implement deep learning applications, specially designed and costly hardware are always needed [8] – [10]. This approach, delays time to market, increase development cost, and there exist the inherent system integration challenges associated with that approach. To ameliorate these problems, our contribution in this paper, include using a widely available, low-cost, fixed-point hardware, the Texas Instruments (TI) C28x DSP in an osmotic distributed computing framework for deep learning application at the edge and fog layer of IIoT networks. The C28x is selected for our purpose since it is widely deployed in numerous IoT devices and systems such as hybrid electric vehicles, surveillance cameras, microwave ovens, Tablets, Fridges, Bicycles and PCs [11], [12]. It is also used extensively in wearable medical devices, such as Pulse Oximeters, portable Electroencephalograms (EEGs), Ventilators, Continuous Positive Airway Pressure (CPAP) machines, Ultrasound machines, Electrocardiogram (ECG) machines, CT Scanners

Abstract— Challenges associated with developing analytics solutions at the edge of large scale Industrial Internet of Things (IIoT) networks close to where data is being generated in most cases involves developing analytics solutions from ground up. However, this approach increases IoT development costs and system complexities, delay time to market, and ultimately lowers competitive advantages associated with delivering next-generation IoT designs. To overcome these challenges, existing, widely available, hardware can be utilized to successfully participate in distributed edge computing for IIoT systems. In this paper, an osmotic computing approach is used to illustrate how distributed osmotic computing and existing low-cost hardware may be utilized to solve complex, compute-intensive Explainable Artificial Intelligence (XAI) deep learning problem from the edge, through the fog, to the network cloud layer of IIoT systems. At the edge layer, the C28x digital signal processor (DSP), an existing low-cost, embedded, real-time DSP that has very wide deployment and integration in several IoT industries is used as a case study for constructing real-time graph-based Coiflet wavelets that could be used for several analytic applications including deep learning preprocessing applications at the edge and fog layers of IIoT networks. Our implementation is the first known application of the fixed-point C28x DSP to construct Coiflet wavelets. Coiflet Wavelets are constructed in the form of an osmotic microservice, using embedded low-level machine language to program the C28x at the network edge. With the graph-based approach, it is shown that an entire Coiflet wavelet distribution could be generated from only one wavelet stored in the C28x based edge device, and this could lead to significant savings in memory at the edge of IoT networks. Pearson correlation coefficient is used to select an edge generated Coiflet wavelet and the selected wavelet is used at the fog layer for pre-processing and denoising IIoT data to improve data quality for fog layer based deep learning application. Parameters for implementing deep learning at the fog layer using LSTM networks have been determined in the cloud. For XAI, communication network noise is shown to have significant impact on results of predictive deep learning at IIoT network fog layer. Keywords— explainable AI, real-time, embedded C, deep learning, data quality, distributed computing, industrial IoT, wavelet

I. INTRODUCTION Most common edge devices in IoT are always resourceconstrained. They always have low memory, and many are battery powered and located in places that are difficult to access [1], [2]. Due to these challenges, IoT devices can benefit immensely from distributed computing at network edges, since 978-1-5386-9555-5/18/$31.00 ©2018 IEEE DOI 10.1109/AIKE.2018.00042

179

(for imaging applications), and in Magnetic Resonance Imaging (MRI) applications [13]. In addition, it has wide application in numerous large-scale engineering projects, industrial control, surveillance drones, aircraft cabin equipment, powerline communication (PLC), and smart grid (SG) DC motor applications [14], [15], [16], [17]. Our distributed osmotic computing framework for deep learning application is as illustrated in Fig. 1. In Fig. 1, the C28x DSP hardware use for osmotic distributed computing for deep learning application is situated at the network edge where rotating machines in the case of IIoT, SG and other Cyber Physical Systems (CPS) exists. It is worthy to note however that the network edge could be anywhere such as a hospital, monitored human, moving car, etc., where data production using edge devices takes place. The C28x is deployed in this paper to produce Coiflet 2, Coiflet 3, Coiflet 4 and Coiflet 5 wavelets in form of a microservice at the edge layer of the IIoT network. Coiflet wavelets are constructed for the C28x using a combination of embedded system’s low-level computing technology and advanced signal processing approach. Our design, to the best of our knowledge is the first known application of the low-cost ($1.5 - $20 based on type and variant) C28x to Coiflet wavelet construction [18]. Coiflet wavelets are the focal point of this paper due to their numerous applications in several industries such as for improving IoT data quality in data and signal denoising, data and image compression [19], [20], [21], [22], fault detection in IIoT rotating machines [23], numerical analysis, and wavelet based statistical machine learning inferencing [24], [25], [26]. Our Coiflet wavelet design approach will lead to significant memory savings by providing Coiflet wavelets as a microservice at network edges since only one Coiflet wavelet is constructed and stored in the C28x DSP lookup table (LUT). Other Coiflet wavelets from Coiflet 2 to Coiflet 5 are adaptively produced from the initially constructed and stored Coiflet wavelet, by exploiting the similarity in nature of Coiflet wavelets. Constructed wavelets are then utilized in denoising application at the fog layer of the IIoT network to improve the quality of data for deep learning application. Even though Coiflet wavelets have been in existence before, and the wavelets are available in standard numerical software (such as Matlab and R) toolboxes; however, as stated in [27] and [28], using such toolboxes for IoT edge analytics always incur significant computation and performance related penalties, since there must be a conversion of data and code from high level software language to low level machine codes. In our

design in this paper, low-level Embedded C have been used for Coiflet wavelets construction. Since most hardware in IoT edge analytics are resourceconstrained, it will be a useful contribution to IoT edge analytics if needed wavelets are produced right at the edge in a distributed osmotic computing approach, using low-level machine codes that will incur lesser computational overheads. There have been prior attempts in literature in designing hardware based wavelets. In [28], authors reported the design of a CMOS based wavelet generator. In [29] and [30], authors discussed the design of a 60 GHz and 20 GHz wavelet generator for radio applications using gated tunnel diodes. In [31], authors reported the design of an InP HEMTs hardware wavelet generator. In all the cases reported however, wavelets were obtained using very complicated hardware structure that are not suitable for IoT edge analytics. Our approach in this paper uses low-level embedded C and a widely available, low-cost DSP which have been proven to work well in several IoT industries. The remaining parts of this paper is as follows. Section II is a brief review of the application of long short-term memory (LSTM) networks to deep learning. Section III is a discussion of the method used to construct Coiflet wavelets for C28x DSP and how constructed wavelets satisfy important wavelets criteria. Section IV is about a memory-saving adaptive graphical method of generating any needed wavelet from Coiflet 2 to Coiflet 5 in the C28x LUT. Section V discusses how constructed wavelet is used to improve IIoT machine data quality at the fog layer, while Section VII shows our XAI deep learning implementation. Section VIII is conclusion of the paper. II. BRIEF REVIEW ON THE APPLICATION OF LSTM NETWORK TO DEEP LEARNING In deep learning applications, feedforward neural networks (FNN) are always applied to approximate a function ∗ [32]. In classification tasks, the goal of the neural network is to map an input data or signal x to a defined category y using the approximation function ∗ . The classification approach is defined as = ∗ ()

To accomplish the classification task, the FNN must learn the parameter that result to the best mapping of y for = (; )

(2)

When FNNs are equipped with feedback connections, they are referred to as recurrent neural networks (RNNs). RNNs have

Osmotic computing: microservices Edge Layer

(1)

Osmotic distributed computing: network training in the cloud using big data generated at network edge

Fog Layer

Cloud Layer

To aid distributed IoT cloud computing: Adaptive graphical Coiflet Wavelet generation as a microservice, using embedded systems and other low-cost technology

To aid distributed IoT cloud computing: Data pre-processing, noise removal, data quality improvement. Use wavelets and other tools provided by edge microservice devices. Use machine learning parameters available from cloud network. Deep learning is also possible here, based on available computing power

To aid distributed IoT cloud computing: machine learning training using big data. Deep neural network and other machine learning technology also possible here if not possible to be accomplished at the fog layer

Fig. 1: Distributed osmotic computing approach for implementing deep learning application at the edge-fog layer of IIoT networks

180

2

where is filtered to be between -1 and 1 using a tanh gate. The final output of the LSTM RNN network is given as

been successfully applied to many deep learning problems including natural language modelling, image recognition and captioning, speech recognition, etc. However, RNNs have been known to suffer from the problem of long term dependencies since, as RNN network grows, it becomes increasingly impossible to relate past information to the present task [33]. To overcome problem of long term dependencies, RNNs that utilizes LSTMs have proven to be quite effective [34]. A generic RNN network containing repeating modules of LSTMs is shown in Fig. 2. The cell state in LSTM based RNN networks enable past information to flow along the RNN network unchanged. Generally, the RNN in Fig. 2 will output a new value ℎ based on old cell value ℎ and new signal input . To prevent information overflow, gates, which are made of sigmoid functions are used. Sigmoid functions assign weights that varies between 0 and 1 to information. A value of 1 will allow the concerned information to flow through the gate un-impeded while an information with a value of 0 will be stopped by the gate. The forget gate layer serves the purpose of maintaining or removing information from the cell state. Suppose the old cell state in Fig. 2 is described by , the forget gate sigmoid function will examine both last cell value ℎ and new input and output a value between 0 and 1 that will determine whether to retain or flush out an information from the cell state. The value can be described [33] as = ( [ℎ , ] + )

= tanh ( [ℎ , ] + )

(5)

= ∗ [ + ] ∗

(6)

ℎ = * tanh ( )

(8)

III. EXPERIMENTAL SYSTEM SETUP AND METHOD OF COIFLET WAVELET CONSTRUCTION FOR THE C28X DSP AT NETWORK EDGES The C28x DSP is manufactured by TI [14]. To program the C28x DSP for generating Coiflet wavelets, the Code Composer Studio (CCS), a programming interface for real-time systems, also manufactured by TI is used. The C28x DSP used in this study exist as part of an IIoT edge node (a modem), i.e., the TMS320C2000 powerline communication (PLC) modem manufactured by TI. The TMS320C2000 is always used for communication over the low voltage end of the powerline for industries that implements IIoT such as the SG, and other CPS. The TMS320C2000 modem is attached to a computer that have the CCS real-time programming interface. Through the CCS, the C28x DSP (embedded in the TMS320C2000 modem) could be programmed using the embedded C programming language. The setup is as shown in Fig. 3 In designing the Coiflet wavelets, an advanced signal processing method that involves signal design, signal clipping and signal concatenation is used [35], [36], [37]. Mathematical equation corresponding to sine waves are written using fixedpoint embedded C programming language that is compatible with the C28x DSP in CCS. The resulting sinusoids are stored in the C28x LUT. Relevant portions of designed sinusoids

To determine the next values that will be stored in the cell state, a scale , which is a value that determines how much we update the new state values will be multiplied with vector of new candidates , with weight matrix W, where , , and the new cell state are given as (4)

(7)

There are many variants of the LSTM RNN networks that are implemented for deep learning applications. As discussed in [34] however, performances of existing variants are similar, and none of the existing variants including those with peepholes have significant performance improvements over each other. In this paper, an XAI deep learning application based on LSTM RNN network for analyzing and predicting future machine data of an IIoT machine situated at the network edge is accomplished at the network fog layer using the Coiflet wavelet produced as a microservice at the edge layer by the low-cost C28x DSP.

(3)

= ( [ℎ , ] + )

= ( [ℎ , ] + )

Finally, the new output ℎ of the LSTM RNN network is determined by the part of the cell state that will be outputted

Cell State

tan

Input Gate

tan

ℎ

tan

tan

tan

tan

Forget Gate

Fig. 2: The Long Short-Term Memory (LSTM) based Recurrent Neural Networks have been very successful in various deep learning applications

181

TMS320C2000 PLC modem

Communication network

PC with Texas Instruments CCS

PC with Texas Instruments CCS

IIoT Edge Layer (Coiflet Wavelet) generation as a microservice using TMS320C2000 C28x DSP

IIoT Fog Layer Deep Learning implementation using the Coiflet wavelet generated at fog layer to improve data quality

IIoT Cloud Layer Deep Learning parameter specification

Fig. 3: Experimental setup for distributed deep learning implementation using wavelet to improve data quality at the edge fog layer of IIoT network using generic hardware %

E = ∫ % |(!)|" #! < ∞

are then clipped and concatenated together to produce Coiflet 2 wavelet. The method of generating the Coiflet 2 wavelet is as shown in Fig. 4. In Fig. 4, the portion of each sine wave that are in the dashed red blocks are the clipped part from each sine wave. Note that clipped sine waves are varied in amplitudes to reflect different parts of the Coiflet 2 wavelets that have different amplitudes. Resulting Coiflet 2 waveform is shown in the extreme right of Fig. 4. Fig. 5 is the graph of the generated Coiflet 2 waveform shown in CCS when the CCS interface is running the C28x DSP in real-time. Using similar method and shifting the waveforms on the sample axis (x - axis), Coiflet 3, Coiflet 4 and Coiflet 5 wavelets are generated and shown in Fig. 6, Fig. 7 and Fig. 8 respectively. Magnitude of each sinusoid and their respectively clipped parts are conditioned on the allowed digital to analog converter (DAC) voltage level of the C28x DSP. From Fig. 4, it is noticeable that each designed Coiflet wavelet from Fig. 5 to Fig. 8 have five clipped sinusoid parts that are sequenced, concatenated and joined together. Amplitudes (A1 to A5) of each sequenced clipped waveform is shown in Table I. To ensure that designed Coiflet wavelets satisfies important wavelet criteria, the wavelet compactness, zero-mean, admissibility and finite energy criteria are used. The finite energy criteria stipulate that a wavelet must have finite energy [38] E given as

Where (!) in (9) is the mother wavelet, i.e., Coiflet wavelets in our case. The wavelet admissibility condition [38] also stipulates that a wavelet must have zero average (zero-mean condition) based on its symmetricity. % |'()|*

& = ∫-

0.0750

0.950

0.450

0.950

0.02

Coiflet 3

0.0725

0.925

0.435

0.925

0.04

Coiflet 4

0.0715

0.920

0.425

0.920

0.06

Coiflet 5

0.0705

0.900

0.415

0.900

0.07

+

||

#! < ∞

(10)

The admissibility condition in (10) is based on the Fourier transform of the designed wavelets. For the wavelet to be admissible as a true wavelet, then it must have real valued FFT and the FFT must exist only at positive value [39] region of the sample axis. Without loss of generality, one of the constructed wavelets, i.e., Coiflet 2 is examined and the wavelet is found to satisfy admissibility condition since the FFT is real valued and it exist only at positive regions of the sample axis. The plot is shown in Fig. 9. The compactness criterion means that the wavelet must be non- zero only on a limited portion of its domain. It can also be interpreted that the wavelet amplitude must start at zero, increases, and decrease back to zero [38] on its sample axis. The compactness criterion can be verified by observing that the designed Coiflet wavelets from Fig. 5 to Fig. 8 satisfy the compactness criteria. To examine the zero-mean and finite energy criteria, 96-points sample values of each designed Coiflet wavelets were exported from CCS to Matlab in Excel csv file format and the mean of the sample values yielded values that are quite close to zero as shown in Table 2. That the Coiflet wavelets mean are not exactly zero is attributable to the fact that Coiflet wavelets are not quite completely symmetrical in shapes [38]. The finite energy condition is also satisfied when the Coiflet wavelet samples are examined with Matlab. The energy in the respective Coiflet wavelets are also shown in Table 2. Further comparison with Coiflet wavelets generated using Python, obtained from [40], and shown in Fig. 10 confirms that the Coiflet wavelet designed using C28x real-time DSP are comparable with those generated using wavelet toolboxes from computing software toolboxes.

Table I: Amplitude of each clipped and sequenced Coiflet wavelet parts Coiflet A1 A2 A3 A4 A5 Wavelet Coiflet 2

(9)

+

+

+

=

Fig. 4. An advanced signal processing method involving clipping, sequencing and concatenation of sinusoids is used to design Coiflet wavelets for generic low-cost C28x DSP existing at IIoT network edges. Embedded C programming language is used in this design.

182

Fig. 5. Coiflet 2 wavelet designed using generic low-cost C28x real-time DSP and open source embedded C codes at the edge of IIoT network


Table 2: Coiflet wavelet criteria Wavelet Compactness Zero Mean Coiflet 2 Coiflet 3 Coiflet 4 Coiflet 5

Yes Yes Yes Yes

0.000016099 0.000005221 0.000012490 0.000012260


The adaptive graphical method involves exploiting the similarity in the nature and construction of the Coiflet wavelets to generate one wavelet from another. By showing that any needed Coiflet wavelet (between Coiflet 2 and Coiflet 5) could be generated from the last used Coiflet wavelet in the C28x DSP LUT, then there is not a pressing need to store all the wavelets in the C28x DSP LUT. In order words, any needed wavelet is reachable from the last used wavelet using a directed graph approach. A directed graph G = (V, E), having set of edges E and set of vertex V will have a reachability relation based on the transitive closure of E. The transitive closure is based on the set of all ordered pairs of vertices in V for which there exists a connectivity [43]. From Coiflet 2 wavelet, Coiflet 3 could be constructed by varying the amplitudes of the clipped signal 1, clipped signal 2, clipped signal 3 and clipped signal 4 in Fig. 4. The only signal part that needs significant change is clipped signal 5 which will have to be changed from a sine wave to a cosine wave and clipped to generate Coiflet 3 from Coiflet 2. Also, Coiflet 3 could be produced from Coiflet 2 by clipping

Finite Energy 1.8162 1.6686 1.8213 1.8229

IV. ADAPTIVE GRAPHICAL METHOD FOR GENERATING COIFLET WAVELETS AT EDGE OF IIOT NETWORKS USING C28X REALTIME DSP

Since most edge analytic devices are resource constrained, it is needful to find methods by which memory could be conserved at IoT network edges. In this paper, an adaptive graphical method is introduced for the generation of any needed Coiflet wavelet [41], [42] from Coiflet 2 to Coiflet 5 when only one of the wavelet is stored in the LUT of the C28x DSP. The size of each constructed wavelet measured with Matlab from their samples in Excel file is 1.19 kB.

183

Coiflet 3

Coiflet 4


Coiflet 2 Fig. 9: Wavelets constructed for the C28x DSP using embedded C at the edge of IIoT network satisfies wavelet admissibility criteria as shown by wavelet’s FFT.

(a)

(b)

(c)

Coiflet 5

Fig. 11. Graph transitive closure showing how Coiflet 2, 3, 4 and Coiflet 5 could be generated at the network edge using only one stored Coiflet wavelet and real-time C28x DSP hardware

generated from Coiflet 2 samples in the C28x LUT by shifting Coiflet 2 on the sample axis, and by varying the amplitudes of the relevant parts of Coiflet 2. From Fig. 11, it is noticeable that Coiflet 2 can also be generated from Coiflet 3 by varying the amplitudes of the relevant clipped sinusoid parts. So, this method of wavelet generation using the graph-based method is adaptive and all wavelets (Coiflet 2 to Coiflet 5) can be generated by following the graph paths and vertices. Transitive closure illustrating adaptive reachability of each Coiflet wavelets from one another is shown in Table 3.

(d)

Fig. 10: (a) Coiflet 2 (b) Coiflet 3 (c) Coiflet 4(d) Coiflet 5 (from Python software) [40]

the sine wave part that is having a crest instead of a trough in the sine wave. This method of using former Coiflet wavelets to produce a new wavelet mimics a graph-based approach in which the reachability of one wavelet from a prior wavelet indicates that there exists a valid transitive closure from the prior wavelet to the new wavelet. This graph-based wavelet processing approach is shown in Fig. 11. In Fig. 11, the vertices are the Coiflet wavelets while the edges are the clipped sinusoid parts that are needed to generate a new wavelet from a former wavelet. Essential parts that needed to be varied to construct one wavelet from another are shown in the red rings in Fig. 11. Fig. 11 also displays essential sinusoid parts that needs to be clipped to construct new Coiflet wavelets in red-dash-blocks. To be able to construct Coiflet 3 from Coiflet 2, it is essential that the location of the wavelet on the sample axis in the CCS be shifted by a factor of about five samples. It is worthy to note that significant savings in computation can be achieved if Coiflet wavelets are derived from one another in sequential order by following the graph. For example, Coiflet 3 can be

Table 3: Transitive closure of the wavelet graph in Fig 11. Wavelets Coiflet 2 Coiflet 3 Coiflet 4 Coiflet 5 Coiflet 2 1 1 1 1 Coiflet 3 1 1 1 1 Coiflet 4 1 1 1 1 Coiflet 5 1 1 1 1 V. IMPROVING DATA QUALITY FOR XAI DEEP LEARNING APPLICATION AT THE FOG LAYER USING COIFLET WAVELET CONSTRUCTED AT THE EDGE LAYER OF IIOT NETWORK Further evaluation of Coiflet wavelets generated as a microservice at the IIoT network edge is achieved by using Coiflet wavelets constructed for the generic C28x DSP to improve data quality of IIoT machine time-series data for XAI deep learning application at the fog layer. The time-series data used is SG electricity consumption data. The rate of variation of electricity consumption is always related to the response and

184

rotation per minute (RPM) of the SG machine [44], [45] situated at the IIoT network edge. The data is obtained from [46], and it is assumed in this paper that the time-series data have been temporarily stored in C28x device memory at the IIoT network edge for a few minutes. However, without loss of generality, for the fact that deep learning applications always need huge data sets, the time-series data used have been collected for up to 4000 mins and it is shown in Fig. 12. An additive white gaussian noise (AWGN), of amplitude 0.02 have been added to the time-series to simulate a time-series data set that have been corrupted with communication network noise when the data is transmitted from the edge to the IIoT network fog layer. AWGN is a type of noise that is commonly found in both wireless and IIoT PLC networks [47]. Coiflet wavelet have been constructed at the edge by C28x DSP that exist in the modem of TMS320C2000. The modem has further been used to transmit the machine time-series from edge to the IIoT fog layer (Fig. 3). The desired task at the fog layer in this paper is to use LSTM RNN network for predictive XAI deep learning application. Predicting future electricity energy consumption, will determine future IIoT machine on-time and demand response based on available machine time-series data. As a preprocessing step for the predictive XAI deep learning application, Coiflet wavelets produced as osmotic microservice wavelets at edge layer are used to denoise the received timeseries data set at the fog layer before predictive deep learning. It is assumed in this paper that the fog layer device has more computing power than edge layer device [48], and a more compute-intensive software such Matlab can be used at the fog layer. This is different from the edge layer in which low-level Embedded C is used to create Coiflet wavelet, and this Embedded C/Matlab combination at the IIoT edge/fog layer very well illustrate a software defined osmotic collaborative approach to computing for large scale CPS such as IIoT networks. Excel.csv samples of Coiflet 2, Coiflet 3, Coiflet 4 and Coiflet 5 wavelets are obtained from CCS and individually convolved [49], [56] with the electricity time-series data set. Convolution results are shown in Fig. 13 and Fig. 14 for Coiflet 2 and Coiflet 3 wavelets. Coiflet 4 and Coiflet 5 produces similar convolution results and are not shown in this paper. Pearson Correlation coefficient (PCC) is used to verify the wavelet that produces a denoised electricity data set that has the closest similarity to the original underlying electricity dataset. PCC values can range from 1 to -1. A value of 0 signify lack of association between two examined time-series while a positive value that is tending towards 1 signifies close association (correlation) between two examined time-series. A negative value indicates negative correlation, i.e., as the value of one time-series decreases, the other time-series increases [50], [51]. Result of the PCC for each wavelet is shown in the bar graph of Fig. 15. From Coiflet 2 to Coiflet 5, PCC values are 0.8799, 0.8138, 0.8354 and 0.8071 respectively. This result signify that Coiflet 2 wavelet have the best PCC performance for denoising the electricity time-series data set at the fog layer. Denoised electricity consumption data resulting from pre-processing with Coiflet wavelet 2 is shown in Fig. 16, with a clear set of timeseries data, with improved quality when compared with the original noisy time-series data of Fig. 12. Due to improved quality of data due to significant noise reduction, cleansed electricity consumption time-series produced by the convolution of Coiflet 2 wavelet with the original noised time-series is

Fig. 12. Noisy IIoT electricity consumption data used to further evaluate the osmotic distributed computing approach for deep learning XAI implementation at the edge and fog layer of IIoT network in this paper

Fig. 13. Distributed software defined osmotic computing approach using different devices having varying capacities. Coiflet 2 wavelet produced by generic C28x DSP at edge layer is used in convolution to improve data quality before deep learning at IIoT fog layer

Fig. 14. Distributed software defined osmotic computing approach using different devices having varying capacities. Coiflet 3 wavelet produced by generic C28x DSP at edge layer is used in convolution to improve data quality before deep learning at IIoT fog layer

selected for use in predictive XAI deep learning implementation at the fog layer in this paper. VI. LONG SHORT TERM MEMORY PREDICTIVE DEEP LEARNING IMPLEMENATION AT THE IIOT NETWORK FOG LAYER USING THE ADAM OPTIMIZATION ALGORITHM After preprocessing the electricity consumption data to denoise it and improve its quality, the denoised data and the original noised data set are then used to examine the effect of

185

Table 4: Deep Learning parameters used in this paper Max. Epoch

Deep Learning Hidden Units

Drop Period

300

200

150

Rate Schedule

Piecewise

Gradient Threshold

Initial Learning Rate

1

0.009

results are used to train the regression LSTM network to predict future electricity generation and consumption based on available electricity consumption time-series. Initial 90% (about first 3,600 sample of the time-series data of Fig. 16) of the denoised data set is used as the training data. The network is first initialized by predicting on the training data. Predictions are obtained while the LSTM network learn to predict the value of the next time step based on the present input sequence. The network state is updated after each prediction, and the previous prediction is used as input function to the network. The remaining 10% test data set are used to verify the trueness of the deep learning network prediction using root mean square error value. For comparison and XIA, original noisy data sets (Fig. 12) are also separately used for deep learning prediction so that impact of noise on deep learning network performance can be observed. Results are shown in Fig. 17 to Fig. 20. In Fig. 17, it could be observed that when denoised data set are used, predicted time-series (in red circle) have similar trend to the actual time-series (also shown in red circle in Fig. 16). RMSE between deep learning prediction and actual data set have a low value of 1.6552 as shown in Fig. 18. When noisy data set are used however, deep learning LSTM network has poor performance as shown in the red circled prediction trend shown in Fig. 19 when compared to the actual true observation shown in red circle of Fig. 16. RMSE value is also high with a high value of 24.2384 when noisy data set are used as shown in Fig. 20. From the foregoing, for explanation of the performance (XIA) of deep learning at the fog layer, it could be observed that performance of deep learning implemented in this paper is affected to a large degree by communication channel noise when data is exchanged between the edge and fog layer of IIoT network osmotic collaborative computing that produced Coiflet wavelet (from the edge to the fog layer) that could be used for pre-processing and denoising before the actual deep learning implementation at the fog layer is thus a valuable approach for optimization of deep learning prediction.

Fig. 15. Coiflet 2 wavelet have the highest Pearson correlation coefficient. Coiflet 2 wavelet produces data set with most noise reduction

collaborative computing communication channel noise on the performance of predictive deep learning network that is implemented as a sequence-to-sequence regression LSTM system in this paper. At the fog layer of IIoT implemented with a PC with Matlab, the LSTM deep learning network is configured with Matlab to have 200 hidden units [52]. Other deep learning implementation parameters that have proven to work well for similar time-series data set are obtained in literature and used to configure a single-input deep learning network in Matlab. An initial learning rate of 0.009 is used since authors in [53] assert that most deep learning toolboxes such as Keras, Tensorflow, Blocks, Lasagne, Caffe, MxNet and Torch uses an initial learning rate of 0.009. Other 1-D time-series deep learning configuration parameters are obtained from [52] and [53], but with slight variations such as an epoch value of 300 and drop period of 150; used in our implementation. These varied values are used to improve speed of our deep-learning implementation since we have a longer time-series than the time-series available in [53]. Parameters used in our LSTM RNN XIA deep learning implementation in this paper are shown in Table 4. The Adam gradient descent algorithm is selected as the optimization solver due to its numerous reported benefits such as ease of implementation, computational efficiency, low memory requirement, and because of its requiring parameters that needs lesser amount of tuning [53], [54], [55]. The denoised electricity consumption data is divided into ratio 9:1 for training and testing data respectively. Parameters of Table 4 which could also be assumed to have been obtained from cloud layer deep learning training

Fig. 17. Performance of deep learning network using denoised data shows the positive effect of Coiflet wavelet-based pre-processing on the performance of deep learning prediction.

Fig. 16. Denoised data set with significantly reduced noise and improved data quality when compared with initial noised data set of Fig. 12

186

advanced signal processing approach. Our contribution in this paper to the best of our knowledge, is the first known case of Coiflet wavelet generation by the fixed-point C28x DSP using embedded C programming language. The wavelets are found to satisfy important wavelet criteria. Software defined osmotic distributed computing approach is used by which different types of software based on network needs and hardware capacities are applied at different layers of the IIoT network. Embedded C, a low-level machine language that is always suitable for use by low-power devices is used to construct needed Coiflet wavelets for the C28x at the edge layer of the IIoT network. An adaptive wavelet graph approach is used to construct other needed wavelets based on the original Coiflet wavelets stored in the C28x edge device. Pearson correlation coefficient is used to show that Coiflet 2 wavelet have better denoising performance when compared with other wavelets at the fog layer. For collaborative computing, Coiflet 2 wavelet is then shared with the fog layer PC, and it is used to denoise and improve the data quality of machine time-series data as a pre-processing step for deep learning implementation at the network fog layer using Matlab. Result of LSTM RNN based deep learning prediction indicate that time-series data with improved data quality yielded better prediction than noised data time-series data. This indicate that the osmotic distributed computing approach to deep learning prediction used in this paper achieves desired predictive results, and the approach will contribute to the development of XAI for deep learning applications. Further work in this domain will include an approach by which complete machine learning application will be achieved at the IIoT network edge layer using generic hardware such as the C28x DSP.

Fig. 18. RMSE value is low between deep learning predicted timeseries and the actual observed time-series when denoised data set is used for deep learning prediction

Fig. 19. Performance of deep learning network using data corrupted with communication channel noise shows poor deep learning prediction when compared to actual observed data

REFERENCES [1]

P. Sethi, S. Sarangi, “Internet of Things: Architectures, Protocols, and Applications”, Journal of Electrical & Computer Engineering, Vol. 2017, Article ID 9324035, 2017 [2] Varghese etal, ”Challenges and Opportunities in Edge Computing”, arXiv:1609.01967v1 [3] D. Datla et al, “Wireless Distributed Computing: A Survey of Research Challenges”, IEEE Comm. Mag., Vol. 50, Issue. 1, January 2012 [4] “On Decentralized and Distributed IoT Edge Computing”, Digital Incubation and Growth GmbH, online, available, www.thingforward.io, July 2017 [5] A. Souza et al, “Osmotic Monitoring of Microservices between the Edge and Cloud” In: Int’l Conf. on HPCC, 2018, Exeter: IEEE” [6] M. Villari, M. Fazio, S. Dustdar, O. Rana, R. Ranjan, “Osmotic Computing: A New Paradigm for Edge/Cloud Integration”, IEEE Cloud Computing, IEEE, 2016 [7] A. Morshed, P.P. Jayaraman, T. Sellis, D. Georgakopoulos, M. Villari, and R. Ranjan, “Deep Osmosis: Holistic Distributed Deep Learning in Osmotic Computing” IEEE Cloud Computing, Blue Skies, 2017 [8] V. Sze, Y. H. Chen, T. Yang, J. S. Emer, “Efficient Processing of Deep Neural Networks: A Tutorial and Survey”, in proc. IEEE, vol. 105, no. 12, December 2017 [9] T. Dettmers, “Deep Learning Hardware Limbo”, online, available, www.timdettmers.com/2017/12/21deep-learning-hardware-limbo, 2017 [10] AnandTech, “Hardware for Machine Learning”, online, available, https://blog.algorithmia.com/hardware-for-machine-learning, March, 2018 [11] Texas Instruments, “Hybrid and Electric Vehicle Solutions Guide”, Texas Instruements, 2013

Fig. 20. RMSE value is high between deep learning predicted timeseries and the actual observed time-series when noisy data set is used for deep learning prediction

VII. CONCLUSION In this paper, a distributed and collaborative computing approach by which already existing and widely available hardware such as the C28x DSP could be made more scalable, by applying them to new uses is discussed. Wavelets that are needed at the fog layer of IIoT networks are constructed for the low-cost C28x DSP available as part of the TMS320C2000 modem situate at the edge layer of SG IIoT network using an

187

[37] S. S. Prasad, and S. K. Sanyal, “Design of Arbitrary Waveform Generator based on Direct Digital Synthesis Technique using Code [38] M. V. Berkel, “Wavelets for Feature Detection: Theoretical Background”, CST 2010.009 [39] Matlab, “Inverse Continuous Wavelet Transforms”, Matlab, 2017. [40] Pywavelets “Wavelet Browser”, online, available, http://wavelets.pybytes.com/wavelet/coif2/ , February, 2018 [41] Matan Gavish, Boaz Nadler, and Ronald R. Coifman, “Multiscale wavelets on trees, graphs and high dimensional data: Theory and applications to semi supervised learning”, In Johannes Frankranz and Thorsten Joachims, editors, ICML, pages 367–374, 2010. [42] Raif M. Rustamov and Leonidas Guibas. Wavelets on graphs via deep learning. In NIPS, 2013. [43] E. Cohen, “Size Estimation Framework with Application to Closures and Reachability”, Journal of Computer and System Sciences, 55, 441-453, 1997 [44] A. Havens, “The Difference Between Electricity Demand and Electricity Consumption”, online, available, www.lucidconnects.com, June 2017 [45] USA Dept. of Energy, “Determining Electricity Motor Load Efficiency”, White Paper, Motor Challenge, online, available, www.energy.gov/10097157, July, 2018 [46] Matlab, “Wavelet Data Compression”, Matlab R2018a, 2018 [47] Z. Li, Q. Liang, J. Mu, W. Wang, and B. Zhang, “The Hybrid Architecture of Wireless and Power Line Communications in HANs for Smart Grid Security”, Journal of Security & Comm. Networks, vol. 8, pp. 2404-2410, 2015 [48] N. Mohan and J. Kangasharju, "Edge-Fog cloud: A distributed cloud for Internet of Things computations," 2016 Cloudification of the Internet of Things (CIoT), Paris, 2016, pp. 1-6. F. Petzschner, “Wavelet and Wavelet Convolution and Brain Music”, Translational Neuromodeling Unit, 2015 [49] W. H. Press, William et. al, "Fourier and Spectral Applications", Sec. 13, Numerical Recipes in C, 2nd Ed., Cambridge University Press, 1992. [50] W. H. Press, et al "Linear Correlation", Sec. 14.5, Numerical Recipes in C, 2nd Ed., Cambridge University Press, 1992 [51] Laerd Statistics, “Pearson Product-Moment Correlation”, online, available, https://statistics.laerd.com/statistical-guides/pearsoncorrelation-coefficient-statistical-guide.php, July 2018 [52] Matlab, “Time-series Forecasting Using Deep Learning”, online, available, https://www.mathworks.com/help/nnet/examples/time-seriesforecasting, July 2018 [53] J. Brownlee, “Gentel Introduction to the Adam Optimiation Algorithm for Deep Learning”, online, available, https://machinelearningmastery.com/adam-optimization-algorithm-fordeep-learning/, July 2018 [54] D. Kingma, J.L. Ba, “ADAM: A Method for Stochastic Optmization”, Int’l Conf. on Learning Representations, 2015 [55] S. Ruder, “An Overview of Gradient Descent Optimization Algorithm”, arXiv:1609.04747v2[cs.LG], January 2017 [56] E. Oyekanlu, K. Scoles, “Real-Time Distributed Computing at Network Edges for Large Scale Industrial IoT Networks”, IEEE Int’l Congress on Internet of Things, San Francisco, California, USA, July 2018

[12] Texas Instruments, “TMS320F2812 Digital Signal Processors: Implementation Tutorial”, Texas Instruments, May 2002. [13] Texas Instruments, “TI HealthTech Engineering Components for the Life”, Slyb207a, Texas Instruments, 2013. [14] Texas Instruments, “TMS320F2809, TMS320F2808, TMS320F2806, TMS320F2802, TMS320F2801, TMS320C2802, TMS320C2801, TMS320F28016, TMS320F28015 Digital Signal Processors Data Manual”, Texas Instruments, May 2012. [15] J. Shandle, “Field-Oriented Control of Small DC Motors Put Drones on a Rising Flight Path”, Electronic Products, online, available, www.digikey.com/en/articles/techzone/2015/feb/field-oriented [16] Texas Instruments, “Digital Control Developer Forum – Digital Control Systems Group”, Texas Instruments, 2000 [17] Texas Instruments, “Motor Drive and Control Solutions”, Texas Instruments, 2017 [18] Texas Instruments, “Software Frameworks for C2000 Piccolo 32-bit MCU Family”, online, available, www.ti.com/ww/mx/multimedia/webcasts/TI_webinar_28-04-2010.pdf, Texas Instruments, 2010 [19] I. M. Johnstone, B. W. Silverman, “Boundary Coiflets for Wavelet Shrinkage in Function Estimation”, J. Appl. Probability, Vol. 41A, Issue (2004), 81-98 [20] M. R. Islam, F. Bulbul, and S. S. Shanta, “Performance Analysis of Coiflet-Type Wavelets for a Fingerprint Image Compression by Using Wavelet and Wavelet Packet Transform”, IJCSES, Vol. 3, No. 2, April 2012. [21] S. Gupta, R. C. Chauham, S. C. Sexana, “Wavelet Based Statistical Approach for Speckle Reduction in Medical Ultrasound Images”, Med. Biol. Comput., 2004, 42, 189-192 [22] M. V. Berkel, “Wavelets for Feature Detection: Theoretical Background”, CST 2010.009 [23] S. Huang, C. T. Hsieh, “Coiflet wavelet Transform Applied to Inspect Power System Disturbance Generated Signals”, IEEE Trans. on Aerospace & Electronics Systems, Vol. 38, No. 1, Jan 2002 [24] P. Muller, B. Vidakovic, “Bayesian Inference in Wavelet-Based Models”, Springer Science+Bus. Media, NY, 1999 [25] F. Abramocich, T. C. Bailey, T. Sapatinas, “Wavelet Analysis and Statistical Application”, Journal of the Royal Statistical Society: Series D (The Statistician); Vol. 49, Issue 1. April 2000. [26] A. Antoniadis, “Wavelets in Statistics: A Review”, Journal of the Italian Statistical Society (1997) 6:97 [27] M. Nikolic, B. Chandramouli, J. Goldstein, “Enabling Signal Processing over Data Streams”, SIGMOID, Chicago, Illinois, USA, 2017 [28] A. Bautista-Castillo, J.M. Rocha-Perez, A. Diaz-Sanchez, J. Lemuz Lopez, L.A. Sanchez-Gaspariano, “A CMOS Morlet Wavelet Generator”, Radioengineering, Vol. 26, No. 1, April 2017 [29] M. Egard et al, “60 GHz Wavelet Generator for Impulse Radio Applications”, EuMA, Rome, Italy, 2009 [30] M. Egard, M. Arlelid, E. Lind, G. Astromkas, and L.E. Wernersson, “20 GHz Wavelet Generator Using a Gated Tunnel Diode”, IEEE Microwave & Wireless Components Letter, Vol. 19, June, 2009. [31] Y. Nakasha et al, “A W-band Wavelet Generator Using 0.13 µm InP HEMTs for Multi-Gigabit Communications Based on Ultra-Wideband Impulse Radio”, IEEE 2008 [32] I. Goodfellow, Y. Bengio, and A. Courville, “Deep Learning”, MIT Press, 2016 [33] Colah’s Blog, “Understanding LSTM Networks”, online, available, http://colah.github.io.posts/2015-08-Understanding-LSTMs/, 2015 [34] K. Greff, R. K. Srivastava, J. Koutnik, B.R. Steunebrink, J. Schmidhuber, “LSTM: A Search Space Odyssey”, IEEE Transaction on Neural Networks and Learning Systems, October 2017 [35] Hioki, “Waveform Generation System, Solution Catalog Arbitrary Waveform Generation Unit U8793 and Memory HiCorder Combining Generator and Memory Function”, U879 Es-6YB, 2016, pp. 4 [36] National Instruments Korea, “Modular Instruments 200 MS/s 16 Bit Arbitrary Waveform Generation”, online, available, www,ni,com/korea, 2018, pp.9

188