Levenberg-Marquardt Learning Algorithm for ... - Semantic Scholar

Neural Information Processing - Letters and Reviews

Vol.9, No.2, November 2005

LETTER

Levenberg-Marquardt Learning Algorithm for Integrate-and-Fire Neuron Model Deepak Mishra, Abhishek Yadav, Sudipta Ray, and Prem K. Kalra Department of Electrical Engineering Indian Institute of Technology, Kanpur, India E-mail: [email protected], [email protected] (Submitted on July 22, 2006) Abstract— In this paper, Levenberg-Marquardt (LM) learning algorithm for a single Integrate-and-Fire Neuron (IFN) is proposed and tested for various applications in which a neural network based on multilayer perceptron is conventionally used. It is found that a single IFN is sufficient for the applications that require a number of neurons in different hidden layers of a conventional neural network. Several benchmark and real-life problems of classification and function-approximation have been illustrated. It is observed that the inclusion of robust algorithm and more biological phenomenon in an artificial neural network can make it more powerful. Keywords—Levenberg-Marquardt learning algorithm, Integrate-and-Fire neuron model, Multilayer Perceptron, Classification, Function approximation.

1. Introduction Various researchers have proposed many neuron models for artificial neural networks. Although all of these models were primarily inspired from the biological neuron, there is still a gap between the philosophies used in neuron models for neuroscience studies and neuron models used for artificial neural networks. Some of these models exhibit a close correspondence with their biological counterparts while others do not. Author in [1] has pointed out that while brains and neural networks share certain structural features such as massive parallelism, biological networks solve complex problems easily and creatively, and existing neural networks do not. He discussed the issues related to the similarities and dissimilarities between biological and artificial neural systems of present days. The main focus in the development of a neuron model for artificial neural networks is not its ability to represent biological activities with its maximum intricacy, but some mathematical properties, e.g., its capability as a universal function approximator. However, it can be advantageous for artificial neural networks if we can bridge the gap between biology and mathematics by investigating the learning capabilities of biological neuron models for use in the applications of classification, time-series prediction, function approximation etc. In this work, we used the simplest biological neuron model i.e. integrate and fire model for this purpose. The first artificial neuron model was proposed by McCulloch and Pitts [2] in 1943. They developed this neuron model based on the fact that the output of the neuron is 1 if the weighted sum of its inputs is greater than a threshold value and 0 otherwise. Later on a learning rule was proposed in [3] that became initiative for artificial neural networks. He postulated that the brain learns by changing its connectivity patterns. Authors in [4] presented the most analyzed and most applied learning rule. It was called the least mean square learning rule. Later it was found that this rule converges in the mean square to the solution that corresponds to least mean square output error if all the input patterns are of the same length [5]. A single neuron of all the above and many other neuron models proposed by several scientists and researchers are capable of linear classification [6]. In [7], authors have incorporated various aggregation and activation functions to model the nonlinear input-output relationships. In [8], the chaotic behavior in neural networks that represent biological activities in terms of firing rates has been investigated. Author in [16] discussed biologically inspired artificial neurons and authors in [15] introduced neuronal models with current inputs. Training the integrate-and-fire model with the Informax 41

D. Mishra, A. Yadav, S. Ray, and P. K. Kalra

Learning Algorithm for Integrate-and-Fire Neuron Model

principle was discussed in [13] and [14]. All the previous approaches are based on the backpropagation learning algorithm, the only drawback of backpropagation learning algorithm is slow convergence and in order to accelerate the convergence of the algorithm we have proposed Levenberg-Marquardt [20] Learning Algorithm for Integrate-and-Fire Neuron Model. This paper describes implementation of training algorithm for Integrate-Fire neuron (IFN) model based on the Levenberg-Marquardt (LM) algorithm. In Section 2, a brief discussion on the biological neuron is presented and formation of Integrate-and-Fire neuron model from Hodgkin-Huxley model is described. Inspired from the relationship between injected current and interspike interval for integrate-and-fire neuron (IFN) model for the learning is proposed in Section 3. The comparison of the proposed model with classical multi layer perceptron (MLP) is described in Section 4. In Section 5, we concluded our work with a brief discussion.

2. Biological Neurons 2.1 Biological Neuron A neuron is the fundamental building block of the biological neural networks. A typical neuron has three major regions: the soma, the axon and the dendrites. Dendrites form a dendritic tree which is a very fine bush of thin fibers around the neuron’s body. Dendrites receive information from neurons through axons, i.e., long fibers that serve as transmission lines. An axon is along cylindrical connection that carries impulses from the neuron. The end part of an axon splits into a fine arborization which terminates in a small end-bulb almost touching the dendrites of neighboring neurons. The axon-dendrite contact organ is called synapse. Details of the biological neuron can be found in [11].

2.2 Integrate-and-Fire Neuron Models The integrate-and-Fire (IF) neuron is the simplest of the threshold-fire neuron models. It is based on the membrane potential equations of HH model [21] with the membrane potential currents omitted. Although the integrate and-fire model is a very simple, it captures almost all of the important properties of the cortical neuron. Figure 1 shows the basic circuit of this model consists of a capacitor C in parallel with a resistor R driven by a current IEXT. The driving current can be split into two components, IEXT = IR(t) + IC(t). The first component is the resistive current which passes through the linear resistor R and the second component charges the capacitor C. Thus

I EXT =

v dv +C R dt

(1)

where, v(t) is the membrane potential. A spike occurs when v(t) reaches a threshold VTH. After the occurrence of a spike, next spike cannot occur during a refractory period TREF. This model divides the dynamics of the neuron into two regimes: the subthreshold and the suprathreshold. The Hodgkin-Huxley equations [21] show that in subthreshold regime, sodium and potassium active channels are almost closed. Therefore, the corresponding terms can be neglected in the voltage equation of the HodgkinHuxley model. This gives a first order linear differential equation similar to Eq.(1). In case of the suprathreshold region, if the voltage hits the threshold at time t0, a spike at time t0 will be registered and the membrane potential will be reset to VRESET. The system will remain there for a refractory period TREF. Figure 2 shows the response of an integrate-and-fire model. The solution of the first-order differential equation in Eq.(2) describing the dynamics of this model in subthreshold region can be found analytically. With v(0) = VREST, the solution of Eq.(2) is given as Eq.(3). C

dv = GL (VL − v) + I EXT dt

− I v(t ) = EXT (1 − e GL

Here GL is the leakage conductance. Let us assume that v(t) hits VTH at t=TTH. Thus

42

tG L c

) + (VREST − VL )e

(2) −

tG L c

+ VL

(3)

Neural Information Processing - Letters and Reviews

Figure 1. Circuit diagram of an integrate-and-fire Neuron model

VTH =

I EXT GL

Vol.9, No.2, November 2005

Figure 2.Response of an integrate-and-fire neuron model

TTH G L ⎛ ⎜1 − e − C ⎜ ⎝

TTH G L ⎞ ⎟ + (VREST − VL )e − C + VL ⎟ ⎠

(4)

Therefore TTH can be written as TTH =

C ⎛ I EXT − GL (VREST − VL ) ⎞ ⎟ ln⎜ GL ⎜⎝ I EXT − GL (VTH − VL ) ⎟⎠

(5)

Interspike interval TISI is the summation of TTH and TREF. Thus TISI = TTH + TREF

(6)

Therefore TISI =

C ⎛ I EXT − GL (VREST − VL ) ⎞ ⎟ + TREF ln⎜ GL ⎜⎝ I EXT − GL (VTH − VL ) ⎟⎠

(7)

Frequency f is reciprocal of interspike interval and hence is given by f =

1 C ⎛ I EXT − GL (VREST − VL ) ⎞ ⎟ + TREF ln⎜ GL ⎜⎝ I EXT − GL (VTH − VL ) ⎟⎠

(8)

3. The Proposed Model Inspired from the relationship between injected current and interspike interval for integrate-and-fire neuron in Eq.(7), following aggregation function is assumed instead of the weighted sum of a conventional neuron: n

net k = Π (ai log(bi x ki ) + d i ) i =1

for k = 1,2,3...P.

(9)

where n is the number of inputs and P is the number of patterns. Eq.(9) corresponds to the first part of integrateand-fire model which is represented by an RC circuit in Figure 1. In Eq.(9), the net input to the activation function of the neuron is considered analogous to the interspike interval and the input x to the neuron is considered to be analogous to a function of the injected current IEXT. Weight b is assumed to be associated with this input to represent the temporal summation when inputs from other synapses are also present. x=

I EXT − GL (VREST − VL ) I EXT − GL (VTH − VL )

(10)

43

D. Mishra, A. Yadav, S. Ray, and P. K. Kalra

Learning Algorithm for Integrate-and-Fire Neuron Model

Figure 3. f-I relationship of an integrate-and-fire neuron model In view of evidence in support of the presence of multiplicative-like operations in the nervous system, multiplication of net inputs to the activation function is considered. In biological neural systems, this operation depends on the timings of various spikes. Aggregation of the exponential waveforms with different time-delays has been approximated by considering a weight associated with the input to the aggregation function. The second part of the integrate-and-fire neuron model is represented in terms of a threshold type of nonlinear block. In this paper, we considered sigmoid function to represent the activity in this block. yk =

1 1 + e − net k

(11)

3.1 Biological Significance of the Proposed Model This model is inspired from the fact that the actual shape of the action potential does not contain any neuronal information. It is the timing of spikes that matters. As the firing frequency is directly related to the injected current, we considered the f-I characteristic of integrate-and-fire neuron as the backbone of our model. All artificial neuron models have two functions associated with them, i.e., aggregation and activation. In case of the integrate-and-fire model there are two parts in its circuit representation, i.e., RC-circuit and threshold-type nonlinearity. While aggregation is inspired from the f-I relation derived from the response of the RC circuit, nonlinearity is introduced in terms of sigmoid activation function. This activation function is continuous and differentiable; therefore it can easily be incorporated in learning. As its output f(x) approaches zero when input x approaches a large value, and is always greater than 0.9933 for x > 5.0, it can be considered to represent an approximation of threshold-type nonlinearity to some extent. A substantial body of evidence supports the presence of multiplicative-like operations in the nervous system [10]. Physiological and behavioral data strongly suggest that the optomotor response of insects to moving stimuli is mediated by a correlation-like operation [11]. Another instance of a multiplication-like operation in the nervous system is the modulation of the receptive field location of neurons in the posterior parietal cortex by the eye and head positions of the monkey [11]. Multiplication operation is used for aggregation of inputs to the artificial neuron in many research papers including [7]. In our work, we incorporated this multiplication operation while aggregating inputs to the activation function.

3.2 Development of the Training Algorithm Pattern classification with neural classifier basically involves understanding the class boundaries by the classifier. To attain this capability, the classifier has to undergo a training phase. The same is achieved with the help of a training algorithm. In this paper we have focused on the application of Levenberg-Marquardt algorithm for the learning of single IFN model. In paper [9], author presented a simple steepest descent method to minimize the following error function: 44

Neural Information Processing - Letters and Reviews

Vol.9, No.2, November 2005

e=

P

1

∑ (tk − yk ) 2

(12)

k =1 2

where t is the target and y is the actual output of the neuron and e is the function of parameters ai, bi and di; i = 1, 2, ...n. Therefore, the parameter update rule (weight update rule) can be expressed in terms of the following equations: ∂e (13) ainew = aiold − η ∂ai binew = biold − η

∂e ∂bi

(14)

∂e , for i = 1, 2, 3, ...n. (15) ∂ci Partial derivatives of e with respect to parameters ai, bi and di (i = 1, 2, ..., n) can be given by the following equations: cinew = ciold − η

P ⎛ ⎞ ∂e log(bi xki ) ⎟ = ∑ (tk − yk ) yk (1 − yk )netk ⎜⎜ ⎟ ∂ai k =1 + a log( b x ) d i ki i ⎠ ⎝ i P ⎛ 1 ∂e = ∑ (tk − yk ) yk (1 − yk )netk ⎜⎜ ∂bi k =1 ⎝ ai log(bi xki ) + di

⎞⎛ ai ⎞ ⎟⎜ ⎟ ⎟⎜ b ⎟ ⎠⎝ i ⎠

(16)

(17)

P ⎛ ∂e 1 = ∑ (tk − yk ) yk (1 − yk )netk ⎜⎜ ∂di k =1 a log( b i xki ) + d i ⎝ i

⎞ ⎟. (18) ⎟ ⎠ In [9], it is found that the performance of IFN model is quite comparable to the performance of classical MLP when simple gradient descent algorithm is used for the training of the neural network. We incorporated the LM method for the training of proposed IFN model and compared with classical feedforward neural network by solving several classification and function approximation problems.

3.3 The Levenberg-Marquardt Algorithm Gradient-based training algorithms, like backpropagation, are most commonly used by researchers. They are not efficient due the fact that the gradient vanishes at the solution. Hessian-based algorithms allow the network to learn more subtle features of a complicated mapping. The training process converges quickly as the solution is approached, because the Hessian does not vanish at the solution. To benefit from the advantages of Hessian based training, we focused on the Levenberg-Marquardt Algorithm. The LM algorithm is basically a Hessian-based algorithm for nonlinear least square optimization [20]. For neural network training the objective function is the error function of the type r P 1 e = ∑ (tk − yk ) 2 , (19) k =1 2 where yk is the actual output for the k-th pattern and tk is desired output. P is the total number of training patterns. z represents the weights and biases of the network. The steps involved in training a neural network using LM algorithm are as follows: 1) Present all inputs to the network and compute the corresponding network outputs and errors. Compute the mean square error over all inputs as in Eq.(19). 2) Compute the Jacobian matrix, J(z) where z represents the weights and biases of the network. 3) Solve the Lavenberg-Marquardt weight update equation to obtain ∆z. 4) Recompute the error using z +∆z. If this new error is smaller than that computed in step 1, then reduce the training parameter µ by µ−, let z = z +∆z, and go back the step 1. If the error is not reduced, then increase µ by µ+ and go back step 3. The µ− and µ+ are defined by user. 5) The algorithm is assumed to have converged when the norm of the gradient is less than some predetermined value, or when the error has been reduced to some error goal. The weight update vector ∆z is calculated as 45

D. Mishra, A. Yadav, S. Ray, and P. K. Kalra

Learning Algorithm for Integrate-and-Fire Neuron Model

[

∆z = J T ( z ) J ( z ) + µI

]

−1

J T ( z) E ,

(20)

where E is a vector of size P calculated as E = [t1 − y1 t2 − y2 ... t P − y P ]T .

(21)

T

Here J (z)J(z) is referred as the Hessian matrix. I is the identity matrix, µ is the learning parameter. For µ = 0 the algorithm becomes Gauss-Newton method. For very large µ the LM algorithm becomes steepest decent or the error backpropagation algorithm. The parameter is automatically adjusted at each iteration in order to secure convergence. The LM algorithm requires computation of the Jacobian J(z) matrix at each iteration step and the inversion of JT (z)J(z) square matrix.

4. Illustrative Examples 4.1 Classification Problems XOR Problem: The XOR problem, as compared with other logic operations (NAND, NOR, AND and OR), is probably one of the best and most used nonlinearly separable pattern associator, and consequently provides one of the most common examples of artificial neural systems for input remapping. We compared the performance of integrate-and-fire neuron (IFN) with that of multilayer perceptron (MLP). For this purpose, we considered an MLP with 3 hidden units. Figure 4 shows the mean-square-error (MSE) vs. number of epochs curves for training with MLP and IFN while dealing with the XOR-problem. It is clear from this figure that the proposed model takes only 10 iterations while MLP takes 23 iterations for training to achieve an MSE of the order of 0.0001. Table 1 and Figure 5 exhibit the comparison between MLP and IFN in terms of the deviation of actual outputs from corresponding targets. It can be seen here that the performance of IFN is almost same as compared with MLP. It means that a single IFN is capable to learn XOR relationship almost 2 times faster than an MLP with 3 hidden units. Table 2 shows the comparison of training and testing performance with MLP and IFN while solving the XOR-problem. 3-bit Parity Problem: The 3-input XOR has been a very popular benchmark classification problem among the researchers of ANN. The problem deals with the mapping of 3-bit wide binary numbers into its parity. If the input pattern consists the odd numbers of 1’s then the parity is 1, otherwise it is 0. This is considered as a difficult problem because the patterns that are close in the sample space, i.e. the numbers that differ in only one bit, require their classes to be different. For comparison of the performance with IFN and MLP in case of 3-bit Parity Problem, we considered MLP with 5 hidden units. Figure 6 shows the comparison of MSE vs. number of epochs curves with conventional multilayer perceptron and the proposed single IFN model while training the artificial neural systems for 3-bit Parity Problem. It is clear from this figure that the proposed model takes only 45 iterations as compared to 70 iterations taken by MLP for training to achieve an MSE of the order of 0.00001.

Figure 4. Learning profiles for XOR-problem

46

Figure 5. Target vs. Actual Output with MLP and IFN for XOR-problem

Neural Information Processing - Letters and Reviews

Vol.9, No.2, November 2005

Table 1. Outputs of IFN and MLP for XOR-Problem Input 0.1 0.1 0.1 0.9 0.9 0.1 0.9 0.9

Target 0.1 0.9 0.9 0.1

Output with MLP 0.1 0.9 0.9 0.1

Output with IFN 0.1 0.9 0.9 0.1

Table 2. Comparison of Training and Testing Performance for XOR-Problem S.No. 1 2 3 4 5 6 7 8

Parameter Training Goal in terms of MSE Iterations Needed Training Time in seconds Testing Time in seconds MSE for Testing Data Correlation Coefficient Percentage Misclassification Number of Parameters

MLP 0.00015 23 0.312 0.001 0.0 1.0 0% 11

IFN 0.00015 10 0.094 0.001 0.0 1.0 0% 6

Table 3. Outputs of IFN and MLP for 3-bit Problem Input 0.1 0.1 0.1 0.1 0.9 0.9 0.9 0.9

0.1 0.1 0.9 0.9 0.1 0.1 0.9 0.9

Target 0.1 0.9 0.1 0.9 0.1 0.9 0.1 0.9

0.1 0.9 0.9 0.1 0.9 0.1 0.1 0.9

Output with MLP 0.1 0.9 0.9 0.1 0.9 0.1 0.1 0.9

Output with IFN 0.1 0.9 0.9 0.1 0.9 0.1 0.1 0.9

Table 3 and Figure 7 exhibit the comparison between MLP and IFN in terms of the deviation of actual outputs from corresponding targets. It can be observed that the performance of IFN is almost same as compared with MLP but IFN is capable to learn this relationship almost 1.5 times faster than that in case of an MLP with 5 hidden neurons. Table 4 shows the comparison of training and testing performance with MLP and IFN while solving the 3-bit Parity problem.

Figure 6. Learning profiles for 3-bit Parity problem

Figure 7. Target vs. Actual Output with MLP and IFN for 3-bit parity problem 47

D. Mishra, A. Yadav, S. Ray, and P. K. Kalra

Learning Algorithm for Integrate-and-Fire Neuron Model

Table 4. Comparison of Training and Testing Performance for 3-bit Problem S.No. 1 2 3 4 5 6 7 8

Parameter Training Goal in terms of MSE Iterations Needed Training Time in seconds Testing Time in seconds MSE for Testing Data Correlation Coefficient Percentage Misclassification Number of Parameters

MLP 0.00003 70 0.312 0.0002 0.0 1.0 0% 22

IFN 0.00003 45 0.094 0.0002 0.0 1.0 0% 9

4.2 Function Approximation Problems Internet Traffic Data: Short term internet traffic data was supplied by HCL Infinet Ltd. (a leading Indian ISP). This data represents weekly internet traffic (in kbps) with a 30-minute average. Four measurements y(t − 1), y(t − 2), y(t − 4) and y(t−8) were used to predict y(t). For comparison of the performance with IFN and MLP with Internet Traffic Data, we considered MLP with 6 hidden neurons. Figure 8 shows the comparison of MSE vs. number of epochs curves while training MLP and IFN artificial neural systems for Internet Traffic Data. This figure shows that the proposed model exhibits faster training on this data. Figure 9 shows the comparison between MLP and IFN in terms of the deviation of actual outputs from corresponding targets. Data till 250 sampling instants was used for training and rest of the data was used for testing. It can be observed that the performance of IFN for training data is almost same as compared with MLP while its performance is better for testing data, i.e., after 250 sampling instants. Table 5 shows the comparison of training and testing performance with MLP and IFN for the Internet Traffic Data.

Table 5. Comparison of Training and Testing Performance for Internet Traffic Data S.No. 1 2 3 4 5 6 7

Parameter Training Goal in terms of MSE Iterations Needed Training Time in seconds Testing Time in seconds MSE for Testing Data Correlation Coefficient Number of Parameters

Figure 8. Learning profiles for Internet Traffic Data

48

MLP 0.005 30 8.9 0.032 0.00053 0.911 32

IFN 0.005 30 0.79 0.031 0.00066 0.902 12

Figure 9. Target and Actual Output with MLP and IFN for Internet-Traffic Data

Neural Information Processing - Letters and Reviews

Figure 10. Learning profiles for EEG Data

Vol.9, No.2, November 2005

Figure 11. Target and Actual Output with MLP and IFN for EEG Data

Table 6. Comparison of Training and Testing Performance for EEG Data S.No. 1 2 3 4 5 6 7

Parameter Training Goal in terms of MSE Iterations Needed Training Time in seconds Testing Time in seconds MSE for Testing Data Correlation Coefficient Number of Parameters

MLP 0.01 25 8.9 0.032 0.00814 0.829 32

IFN 0.01 5 0.79 0.031 0.00913 0.812 12

Electroencephalogram Data: Electroencephalogram (EEG) data used in this work was taken from [19]. Presence of randomness and chaos [8] in this data makes it interesting for neural network related research. In this problem also, four measurements y(t−1), y(t−2), y(t−4) and y(t−8) were used to predict y(t). We considered MLP with 5 hidden neurons for the comparison purposes. Figure 10 shows the comparison of MSE vs. number of epochs curves with conventional multilayer perceptron and the proposed single IFN model while training the artificial neural systems for EEG data. It is clear from this figure that the proposed model exhibits faster training on this data. Figure 11 shows the comparison between MLP and IFN in terms of the deviation of actual outputs from corresponding targets. Data till 200 sampling instants was used for training. It can be seen here that the performance of IFN for training as well as testing data is much better than that of MLP. It means that a single IFN is capable to learn this relationship faster than that in case of an MLP with 5 neurons and its performance on seen as well as unseen data is significantly better. Table 6 shows the comparison of training and testing performance with MLP and IFN while applying for the EEG Data.

5. Conclusions The training and testing results with different benchmark and real-life problems show that the proposed artificial neural system with a single neuron inspired from the integrate-and-fire neuron model is capable of performing classification and function approximation tasks as efficiently as a multilayer perceptron with many neurons and in some cases its learning is even better than that of a multilayer perceptron. It is also observed that training and testing times in case of IFN are significantly less as compared with MLP. The LM algorithm is employed for the training and the results indicate that the application of LM algorithm is very efficient for the training. Future scope of this work includes incorporation of these neurons in a network and analytical investigation of its learning capabilities (e.g., as universal functions approximator). Acknowledgment: The authors would like to thank HCL Infinet Ltd. (India) for providing Internet-Traffic Data.

49

Learning Algorithm for Integrate-and-Fire Neuron Model

D. Mishra, A. Yadav, S. Ray, and P. K. Kalra

References [1] W.J. Freeman, “Why neural networks don’t yet fly: inquiry into the neurodynamics of biological intelligence,” IEEE International Conference on Neural Networks, 24-27 July 1988 pp.1-7, vol.2, 1988. [2] W. McCulloch and W. Pitts, “A logical calculus of the ideas immanent in nervous activity,” Bulletin of Mathematical Biophysics, vol.5, pp. 115-133, 1943. [3] D. Hebb, “Organization of behavior,” John Weiley and Sons, New York, 1949. [4] B. Widrow and M. E. Hoff, “Adaptive switching circuits,” IREWESCON Connection Recors, IRS, New York, 1960. [5] B. Widrow and S. Steams, “Adaptive signal processing,” Prentice-Hall, Englewood Cliffs, NJ., 1985. [6] M. Sinha, D.K. Chaturvedi and P.K. Kalra, “Development of flexible neural network.” Journal of IE(I), vol.83, 2002. [7] R. N. Yadav, V. Singh and P. K. Kalra, “Classification using single neuron,” Proceedings of IEEE International Conference on Industrial Informatics, 2003, pp.124-129, 21-24 Aug. 2003, Canada. [8] D. Mishra, A. Yadav and P. K. Kalra, “Chaotic Behavior in Neural Networks and FitzHugh-Nagumo Neuronal Model,” Proceedings of ICONIP-2004, LNCS 3316, pp.868-873, Dec. 2004, India. [9] A. Yadav, D. Mishra, R.N. Yadav, S. Ray, and P. K. Kalra, “Learning with Single Integrate-and-Fire Neuron,” IEEE International Joint Conference on Neural Network, IJCNN-2005, Montreal (Canada), 2005. [10] C. Koch and T. Poggio, “Multiplying with synapses and neurons,” Single Neuron Computation, Academic Press: Boston, Massachusetts, pp.315-315, 1992. [11] C. Koch, “Biophysics of Computation: Information Processing in Single Neurons,” Oxford University Press, 1999. [12] P. Chandra and Y. Singh, “Feedforward sigmoidal networks - equicontinuity and fault-tolerance properties,” IEEE Transactions on Neural Networks, vol.15, pp.1350-1366, Nov. 2004. [13] J. Feng, H. Buxton and Y. C. Deng, “Training the integrate-and-fire model with the Informax principle I,” J. Phys. A, vol. 35, pp. 23792394, 2002. [14] J. Feng, Y. Sun, H. Buxton and G. Wei, “Training integrate-and-fire neurons with the Informax principle II,” IEEE Transactions on Neural Networks, vol.14, pp. 326-336, March 2003. [15] J. Feng and G. Li, “Neuronal models with current inputs,” J. Phys. A, vol. 24, pp. 1649-1664, 2001. [16] M. Scholles, B. J. Hosticka, M. Kesper, P. Richert and M. Schwarz, “Biologically-inspired artificial neurons: modeling and applications,” Proceedings of 1993 International Joint Conference on Neural Networks, IJCNN ’93-Nagoya, vol.3, 25-29 Oct. 1993, pp.2300-2303, vol.3. [17] N. Iannella and A. Back, “A spiking neural network architecture for nonlinear function approximation,” Proceedings of the 1999 IEEE Signal Processing Society Workshop - Neural Networks for Signal Processing IX, 23-25 Aug. 1999, pp.139-146. [18] S. C. Liu and R. Douglas, “Temporal coding in a silicon network of integrate-and-fire neurons,” IEEE Transactions on Neural Networks, pp.1305-1314, vol.15, Sept. 2004. [19] www.cs.colostate.edu/eeg/ [20] M.T. Hagan, M.B. Mehnaj, “Training Feedforward Networks with the Marquardt Algorithm,” IEEE Transaction on Neural Networks, Vol. 5, No. 6, 1994. [21] A.L. Hodgkin and A.F. Huxley, “A quantitative description of membrane current and its application to conduction and excitation in nerve,” Journal of Physiology, 117:500–544, 1952.

50

Neural Information Processing - Letters and Reviews

Vol.9, No.2, November 2005

Deepak Mishra was born in Seoni (Madhya Pradesh, India) on 18th July, 1978. He is pursuing his Ph. D. in Electrical Engineering in Indian Institute of Technology Kanpur, India. His major field of study is Neural Networks and Computational Neuroscience. Mr. Mishra is a student member of IEEE society. (Home page: http://home.iitk.ac.in/ ~dkmishra)

Abhishek Yadav was born in Mainpuri (Uttar Pradesh, India) on 21st October, 1976. He is pursuing his M. Tech. in Electrical Engineering in Indian Institute of Technology Kanpur, India. His major field of study is Computational Neuroscience. He is working as ASSISTANT PROFESSOR in the Department of Electrical Engineering, College of Technology, G. B. Pant University of Agriculture and Technology, Pantnagar, Uttaranchal, India. He is currently on leave to pursue his higher studies. Mr. Yadav is a student member of IEEE society.

Sudipta Ray was born in Rishra (West Bengal, India) on 3rd September, 1980. He is pursuing his M. Tech. in Electrical Engineering in Indian Institute of Technology Kanpur, India. His major field of study is Computational Neuroscience.

Prem K. Kalra was born in Agra (Uttar Pradesh, India) on 25th October , 1956. He received his BSc (Engg.) degree from DEI Agra, India in 1978, M.Tech degree from Indian Institute of Technology, Kanpur, India in 1982 and Ph.D. degree from Manitoba University, Canada in 1987. He worked as assistant professor in the Department of Electrical Engineering, Montana State University Bozeman, MT, USA from January 1987 to June 1988. In July-August 1988 he was the visiting assistant professor in the Department of Electrical Engineering, University of Washington Seattle, WA, USA. Since September 1988 he is with Department of Electrical Engineering, Indian Institute of Technology Kanpur, India where he is a Professor. Dr. Kalra is a member of IEEE, fellow of IETE and Life member of IE(I), India. He has published over 150 papers in reputed National and International journals and conferences. His research interests are Expert Systems applications, Fuzzy Logic, Neural Networks and Power Systems.

51