Abstract-This paper discusses three learning algorithms to train R.ecrirrenl, Neural Networks for identification of non-
Identification Of Nonlinear Dynamical Systems Using Recurrent Neural Networks Laxmidhar Behera, Swagat Kumar and Subhas Chandra Das Department of Electrical Engineering Indian Institute of Technology,Kanpur Kanpur, 208016 INDIA 05 12-259-7198
[email protected],
[email protected],
[email protected]
Abstract-This paper discusses three learning algorithms to train R.ecrirrenl, Neural Networks for identification of non-linear dynamical systems. We select Memory Neural Networks(MNN) topology for the recurrent network in our work. MNNs are themselves dynamical systems t,hat have internal memory ohtained by adding trainalile temporal eleinerits to feed-forward networks. Three leariling procedures nemcly Back-Propagation Through Tiriie(BPTT), Real Time Recurrent Learning(RTRL) and Ext,ended Kalman Filtering(EKF) are used for adjusting the weights in &INN to train such networks to identify the plant. T h e relative effectiveness of different learning algorithms have been discussed by comparing the mean square error associated with them and corresponding computut,ional requirements. T h e simulation results show- that HTRL algorithm is efficient for trainiug MNNs to model noiilinear dynamical systems by considering both compntat.ional complexity and modelling accnracy. Eventhough. the accuracy of system identification is best with EKF, but it has the drawback of heing computationally int,ensive.
1, INTRODUCTION
A recurrent network model with internal memory is best suited for ideni:ification of systems for which incomplete or no knowledge about, its dynamics exists. Iu this sense, Memory Neuron Xetworks(MNN)[Y] offer truly dynamic niodcls for identificat,ioii of nonlinear dyriamic systems. T h e special feature of these networks is t h a t they have internal trainable memory and can hence directly model dynamical syst,ems without having t o b e explicitly fed with past inputs and outputs. Thus, they can identify systems whose order is unknown or systems with unknown deltry. Here each unit of neuron has, associated with it, a memory neuron whose single scalar. output siimmarizes the history of past activations of that unit. The weights of coirnectiori into memory neuron involve feedback loops: the overall network is now a recurrent one. The primary nini"of this paper is to analyse. the different learning algorithnis on the basis of modeling accuracy and conrputational intensity.The weight coefficients of MNN nro adjusted using a B P T T update al0-7803.765 I-WO3/SI7.00 o?\m CEEE
la
Figure 1. Structure of Memory Neuron Model
gorithm. To increase t h e modeling accuracy two other algorithms namely RTRL and EKF have been proposed. It is concluded t h a t RTRL identifies the system more efficiently when modeling accuracy as well as computational intensity are taken into account. T h e rest of the paper is organized as follows. T h e following section describes the architecturr arid dynamics of MNNs. In section 111, the different learning algorithms namely BPTT, RTRLII] and EKF[5] have been elaborately discussed. The simulations carried out and the results are presented in section IV. T h e final conclusion have been given in section V.
2 . MEMORY NEURAL NETWORK In this section, the structure ofthe network is descrihed. The network used is similar to the one that is descrihed in [3]. T h e architecture of a MNN is shown in Figure 1. The memory neuron takes its input from the corresponding network neuron and it also has a self feedback. This leads to storage of past values of the network neuron in the memory neuron. In t h e output layer, each network neuron can have'a cascade of memory neurons and each of them send their output, to that network neuron in the output layer. Dynamics of the network
The following notations are used to describe the functioning of the network.
Control Systems and Applications / 1121 L is the number of layers of the network with layer 1 as the input layer and layer L as the output 1ayer.Nl is the number of network neurons in layer 1.z:(k) is the net input t,o the jth network neuron of layer 1 a t time k. s i ( k ) is the output of the j t h network neuron of layer1 a t time k. ,u:(k) is the output of t,he memory neuron of the jth network nenron of layer1 at time k,l < L . w t j ( k )is the connecting weight from the ith network neuron of layer 1 to the jth network neuron of layer 1 + 1 at time k. f i j ( k ) is the connecting weight from the meniory neuron of the ith network neuron of layer 1 to the j t h network neuron of layer 1 1 at time k. a : ( k ) is the connecting weight from the jth netwowrk neuron to its corresponding memory neuron. a f j( k ) is the connecting weight from the (j-1)th memory neuron t o the j t h memory neuron of the ith network neuron in the output layer at time k. u:(k) is the output of t h e j t h memory neuron of the ith network neuron in the output layer at time k. @ ( k ) is the connecting weight from the j t h memory neuron of the ith network neuron in the output layer at time.k. Mj is the numher of memory neurons asociated with the jth network neuron of the output layer. g(.) is the activation function of the network neurons. T h e net input to the jth network neurons of layer 1.1
