Network response time for a general class of WTA - CiteSeerX

Network response time for a general class of WTA Peter K.S. Tam1 , John P.F. Sum2 , C.S. Leung2 and L.W. Chan2 Dept. of Electronic Engg., Hong Kong Polytechnic University, Hung Hom, Kowloon. Dept. of Computer Science and Engg., Chinese University of Hong Kong, Shatin, N.T., Hong Kong [email protected];pfsum,csleung,[email protected] 1

2

Abstract| In [1] we proposed a simple circuit of Winner-Take-All neural network and

derived an analytical equation for its network response time [2]. In this paper, we explore this analytical equation for a more general class of winner-take-all circuits. We show that this equation for network response time is indeed an upper bound for a general class of WTA.

1 Introduction Since the beginning of neural network research, the Winner-Take-All (WTA) network has played a very important role in the design of most of the unsupervised learning neural networks [3] such as competitive learning and Hamming network. Lippman rst proposed a discrete-time algorithm called Maxnet to realize the Hamming network [4]. Recently, Dempsey and McVey designed an alternative one called peak detector neural network (PDNN) based on the Hop eld network topology [7], and provided a hardware implementation for it [6]. Seiler and Nossek [8] independently proposed an inputless WTA cellular neural network based on Chua's CNN [9]. Amongst most of the models, their dynamical equations are governed by many parameters and so the design of such networks are complicated. Only a few of them provided analysis on the network response time [6]. In accordance with these diculties, we proposed in [1] a simple analog circuit for WTA with its dynamical equation being governed by just one parameter. Therefore, the design of the network is relatively simple and the analysis on the network response time becomes feasible. In [2], an analytic equation for a simple WTA circuit is derived, under mild assumptions. Intensive computer simulation con rms that the derived equation is a close approximation to the true network response time. In this paper, we further demonstrate that this analytic equation for network response time is indeed an upper bound on a more general class of WTA networks. This paper is organized into six sections. The next section will introduce the model proposed in [1] and the general WTA model. Then some properties governing the derivation of the analytical equation for network response time will be stated in section three. Section four presents the derivation of this analytical equation. Section ve will present those simulation results and con rms that the analytical equation can be treated as an upper bound for the general WTA model. The conclusion is presented in section six.

2 Network Model We consider an N -neurons fully connected inputless WTA neural network. For the ith neuron, i = 1; : : :; N , the state potential (state variable) and the output of the neuron are denoted by vi (t) and hi respectively, where hi is a piecewise linear function ( 1of vifi , vi.e.> 1 i hi = h(vi ) = vi if 0  vi  1 (1) 0 if vi < 0. Simple WTA model | In our proposed model [1], the output of each neuron is connected to all the other neurons and itself, in the same way as Maxnet. The connection is excitatory if the output is selffedback. It is inhibitory when the connection is interneuron. The network dynamics can be described as follows: N dvi (t) = h(v (t)) ?  X h(vk (t)); (2) i dt k =1 for all i = 1; : : : N and 12 <  < 1. The condition on  is used to assure that dvdti < 0 if the ith neuron is not the winning neuron for all time and dvdtN > 0 when non-winning neurons have reached zero [1]. General model | For some models such as Seiler-Nossek [8], a decay term ?vi(t) is usually involved in the dynamical equation: N dvi (t) = ? v (t) + h(v (t)) ?  X h(vk (t)); (3) i i dt k =1 where 0 < . In this case, even the winner, its state 1 potential will also decay to zero as t ! 1. This

general WTA model has been proposed for a long time. However, the bound on its response time has not been studied.

3 Properties of Simple WTA model For the ease of discussion, it is assumed that the initial state potentials can be arranged in a strictly ascending order, i.e. v1 (0) < v2 (0) < : : : < vN (0), for a suitable index set f1; : : :; N g. Now, let us present some properties of the simple WTA model (2) which are useful for the later discussion. The proofs are obmitted here but they can be found in [1]. Theorem 1 If, v1 (0) < v2 (0) < : : : < vN (0), then v1 (t) < v2 (t) < : : : < vN (t), for all t > 0. Theorem 2 If v1 (0) < v2 (0) < : : : < vN (0), then there exists T1 < 1, such that 0 = v1 (T1 ) < v2 (T1 ) < : : : < vN (T1 ). Theorem 3 If v1 (0) < v2 (0) < : : : < vN (0), then there exists T1 < 1, such that 0+ = h1 (T1 ) < h2 (T1 ) < : : : < hN (T1 ). Theorem 4 If v1 (0) < v2 (0) < : : : < vN (0), then there exists 0 < T1 < T2 < : : : < TN ?1 < 1 such that

hi (t) = 0 8t  Ti : Theorem 5 If v1 (0) < v2 (0) < : : : < vN (0), then there exists TN < 1 such that 8 t > TN ,  = N hi (t) = 01 ifif ii 6= N , where i = 1; 2; : : :; N .

4 Network Response Time of the Simple WTA Model We can proceed to see what will happen immediately after T1 . Once t  T1 of Theorems 2{4, h1 (t) = 0; dhdt1 (t) = 0 and 2 _ 3 2 3 2 h (t) 3 h2 (t) 1 ?  ?  : : : ?  66 h_ 3 (t) 77 6 ? 1 ?  : : : ? 7 6 h23 (t) 7 4 ::: 5 = 4 ::: ::: ::: ::: 54 ::: 5: ? ? : : : 1 ?  hN (t) h_ N (t) The output dynamic is now governed by an N ? 1 dimension rst order dierential equation. Let us denote h^ N (t) = (h1 (t); h2 (t); : : :; hN (t))0 for all 0 < t < T1 and ^hN ?1 (t) = (h2 (t); : : :; hN (t))0 ; when t is just greater than T1 , where 0 denotes transpose. Note that N is the index of the neuron for which the initial state potential is the largest. Therefore, when 0 < t < T1 , d^ ^ (4) dt hN (t) = AN hN (t) and when t is just greater than T1 , d^ ^ (5) dt hN ?1 (t) = AN ?1 hN ?1 (t); where 2 1 ?  ? : : : ? 3  1 ?  : : : ? 7 Ak = 64 ? ::: ::: ::: ::: 5 ; ? ? : : : 1 ?  kk for k = N ? 1; N . Just after t = T1 , the network dynamical equation may be changed from (4) to (5) which represents a reduced-dimension system. Hence T1 can be evaluated using the technique of eigenvalue-eigenvector analysis on AN [1]. For all t 2 fs  0j0 < hi (s) < 1; i = 1; 2 : : :; N g, # " " PN PN v (0) # (0) v  k t (1?N )t k =1 + e vi (0) ? k=1N k (6) hi (t) = e N

for all i = 1; 2; : : :; N . Obviously, the output of the 1 neuron will be the rst one reaching zero since hi < hj if i < j . Hence, T1 can be evaluated by setting h1 (t) = 0. 3 2 PN v (0) k k=1 (0)) ? v 1  1 N 5: PN v (0) (7) T1 = ? N log 4 =1

k

N

k

Substituting T1 into Equation (6), we can readily show that ?1 3 N 2 PN v (0) k k=1 v1 (0)) 5 (v (0) ? v (0)); N PN v ?(0) hi (T1 ) = 4 1 i =1

k

N

k

for all i = 2; 3; : : :; N . Note that hN may reach 1 within the time period 0 < t < T1 . However, extensive simulations indicated that the case when vN reaches one earlier than vN ?1 reaches zero is scarce. So we can make the following assumption. Assumption 1 The Nth neuron reaches one later than the Nth?1 neuron reaching zero. Using the above assumption and Equation (6), we can readily deduce that # " PN (0)) (0) ? v ( v 1   2 ; (8) T2 ? T1 = ? (N ? 1) log PkN=2 k (0) ? v ( v   1 (0)) k k =2 # " PN 1 k =3 (vk (0) ? v3 (0)) T3 ? T2 = ? (N ? 2) log PN (9) k =3 (vk (0) ? v2 (0)) ::: and " PN # 1 k =N ?1 (vk (0) ? vN ?1 (0)) TN ?1 ? TN ?2 = ? 2 log PN : (10) k =N ?1 (vk (0) ? vN ?2 (0)) Hence, the network response time Trt = TN ?1 , can be written explicitly as follows: # " PN N ?1 1 X k =N +1?j (vk (0) ? vN ?j (0)) Trt = j log PNk=N +1?j (vk (0) ? vN +1?j (0)) j =2 # " PN 1 (0) v  k + N log PN k=1 : (11) k =1 (vk (0) ? v1 (0)) It is interesting to note that the network response time is dependent solely on  and the initial conditions of the neurons only.

5 Simulation Veri cation Equation (11) indicates that the network response time relies on two factors: the initial conditions of the neurons' state potentials and the parameter  for a xed size of the network. But, it may be queried about the consistency of Equation (11) and the actual network response time because an assumption has been made prior to the derivation. In [2], we have demonstrated that Equation (11) is indeed a close approximation to the actual network response time. The results are listed here in the rst two columns of Table 1. For the cases when > 0, they are depicted from the third column to the sixth column. For each speci ed pair of  and , the computer simulates the Equation (3) for N = f20; 24; 28; : ::; 100g and get 21 sampled network response times for such (; ). Then we repeat this run for additional 49 times with dierent initial conditions. Therefore, for each (; ), we collect altogether 21  50, i.e. 1050, sampled network response times. The average of these 1050 samples constitutes one entry listed on the table1 Figure 1 shows the simulation results. The two solid lines correspond to the network response time of the simple WTA model while the four dash lines correspond to the network response time of the general WTA circuit. In accordance with Table 1 and Figure 1, it is found that the simulated network response times of the general WTA model are well below to that of the simple model and also the response time evaluated by the analytical equation (11). Furthermore, it is also found that the decreasing trends of those response time curves (dash lines) are the same. We therefore suggest that the analytical equation (11) derived for the simple WTA model can be treated as an upper bound for those general WTAs. 1

For the details of obtaining the rst column, please refer to [2].

 = 0 (eva.) 0.6 2.5521 0.7 2.1875 0.8 1.9141 0.9 1.7014

= 0 = 0:05 = 0:10 = 0:15 = 0:20 2.4870 2.4374 2.3084 2.1796 2.0508 2.1263 2.0838 1.9734 1.8628 1.7522 1.8559 1.7849 1.6898 1.5948 1.4999 1.6454 1.5605 1.4773 1.4134 1.3106

Table 1: The network response time P of WTA with dierent decay values . The system is described as dvi (t) follows: dt = ? vi (t)+h(vi (t))? Nk=1 h(vk (t)). The rst column corresponds to the case when = 0 and the results are evaluated using the analytic equation. The second to the sixth column correspond to the results obtained from intensive computer simulation. 3

2.5

2

1.5

1

0.5

0 0.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

0.95

1

Figure 1: The average response time of the network for dierent values of  and dierent values of . The horizontal axis corresponds to the value of  while the vertical axis corresponds to the response time. Solid lines are the cases when = 0: solid-x is the evaluated and solid-+ is the actual. Dot-dash lines correspond to the cases when > 0: = 0:05 (dot-dash-x), = 0:10 (dot-dash-+), = 0:15 (dot-dash-o) and = 0:20 (dot-dash-*).

6 Conclusion In summary, we have reviewed a simple WTA model proposed in [1] and some of its properties. An analytical equation for its response time is presented { Equation (11). In accordance with extensive computer simulations, it is demonstrated that this equation can be treated as an upper bound for a more general class of WTA models. Hence, it can be treated as a cue for the design of those general WTA models.

References

[1] J. Sum and P. Tam, Design and analysis of a simple circuit for Winner-Take-All neural network, submitted to IEEE Transaction on Circuit and System, Part I.

[2] J. Sum and P. Tam, Network response time of a simple circuit for WTA, submitted to IEEE Transaction on Circuit and System, Part I.

[3] Y. Pao, Adaptive Pattern Recognition and Neural Networks. Addison-Wesley, 1989. [4] R. Lippman, An introduction to computing with neural nets, IEEE ASSP Magazine, Vol. 4, pp.4-22, 1987. [5] J. Lazzaro et.al., Winner-Take-All network of O(N ) complexity, NIPS'89. pp.703-711, 1989. [6] G.L. Dempsey and E.S. McVey, Circuit implementation of a peak detector neural network, IEEE Transactions on Circuits and Systems-II, Vol.40, No.9, September, pp.585-591, 1993. [7] J.J. Hop eld, Neurons with graded response have collective computational properties like those of two-state neurons. Proceedings of National Academy of Sciences, Vol.81, pp.3088-3092, 1984. [8] G. Seiler and J. Nossek, Winner-Take-All cellular neural networks, IEEE Transactions on Circuits and Systems-II, Vol.40, No.3, March, pp.184-190, 1993. [9] L. Chua and L. Yang, Cellular neural networks: theory, IEEE Transactions on Circuits and Systems, Vol.35, No.10, October, pp.1257-1272, 1988. [10] A. Yuille and N. Grzywacz, A Winner-Take-All mechanism based on presynaptic inhibition feedback, Neural Computation, 1, 334-347, 1989.