cyclopedia of Electrical and Electronics Engineering, J. G.. Webster, Ed. New York: Wiley Sons, 1999, pp. 396â405. [2] K. Chen and D. Wang, âA dynamically ...
LOGIC COMPUTATION USING COUPLED NEURAL OSCILLATORS Dongming Xu, Jose C. Principe and John G. Harris Computational NeuroEngineering Laboratory Department of Electrical and Computer Engineering University of Florida, Gainesville, 32611 USA {dmxu, principe, harris}@cnel.ufl.edu ABSTRACT This paper presents our recent work in using neural oscillators inspired by olfactory cortex models for boolean logic computation. Two such coupled oscillators are sufficient to implement all the boolean primitives (including AND, OR and XOR). Although the same computational model is being studied as a content addressable memory, these results indicate that additional computational power exists in this class of oscillatory neural networks. Detailed configuration of the proposed structure, experimental results and potential applications are presented. 1. INTRODUCTION Oscillatory networks have been studied as information processors by many researchers because they can be constructed from realistic nonlinear dynamical systems and are biologically plausible. Examples such as relaxation networks [1, 2] that are generalized from natural physical systems have demonstrated their computational ability in image processing and as associative memories. Different structures of oscillatory networks such as Cellular Neural Networks (CNNs) [3, 4, 5], phase locked loop neural networks [6] and multi-resolution neural networks [7] have also been proposed for the purpose of pattern recognition and image segmentation. The dynamical behavior of the individual components in most of the networks are either single-frequency oscillations or limit cycles. In its most general structure, Freeman’s computational model of the olfactory cortex [8] is a chaotic network. There is a significant difference between oscillatory networks and the classical Hopfield-type dynamical neural networks. Processed information is represented by either a limit cycle or nonconvergent dynamics [9], rather than by fixed-point solutions. As for the information processing, in most oscillatory networks the initial conditions are given as the inputs and the networks evolve trough time, where phase information are used to memorize and recall patterns. Freeman’s network receives external input that is mixed with the states, and can work as either a dynamic autoassociative memory, feature extractor or a classifier [10, 11]. In this paper, we propose a different application of Freeman’s network. Instead of using the network as a dynamical content addressable memory, we use the same coupled oscillatory components to perform logic computation. In [12], we studied the synchronization behavior of coupled neural oscillators in terms of coupling strength. With fixed coupling coefficients, a combination of external inputs will lead the coupled oscillators to two different states: synchronization and desynchronization. It turns out that with only two oscillators, by determining the proper values
;,(((
of the coupling coefficients, we can build functional blocks to implement two-input boolean logic gates including AND, OR, XOR, NAND, NOR, XNOR and NOT. The oscillator used here is named the reduced KII (RKII) set that is the fundamental building block in Freeman’s model. Freeman’s model is a biologically realistic computational model of the olfactory cortex [8, 9] that has been used in various pattern-recognition applications [10, 11]. It is a locally stable but globally chaotic system with very high dimensionality. The complexity of the whole system is expressed by a hierarchical embedding of simpler structures. Four different levels are included as K0, KI, KII and KIII, where K stands for Katchalsky. The most basic processing element (PE) in the system is a K0 set which is a nonlinear second order dynamical system as shown in Fig. 1.
Inputs
+
Summing node
F(s) 2nd order dynamics
Q(x)
Output
Asymmetric nonlinear function
Fig. 1. An excitatory K0 set. An inhibitory K0 set has negative sign of nonlinear gain in the 3rd block. The K0 has three major blocks: a summing node, second order dynamics and an asymmetric nonlinear block. Two kinds of K0 sets are distinguished by their signs of the nonlinear gain. The second level KI has interconnected K0 sets with only positive (excitatory) or negative (inhibitory) connections. A KIII network that consists of all the previous levels describes the behavior of the olfactory cortex [8]. A KII set is composed of four K0 sets. Two of them are excitatory while the other two are inhibitory. Basically, the KII set is an oscillator whose behavior is controlled by an external input. A KII network is a network of coupled oscillators (KII), in which different KII sets are interconnected through excitatory cells and inhibitory cells. The RKII set discussed in this paper is a simplified version but plays the same role as that of a KII set when the number of channels in the KII network increases [8, 13]. In Section 2, the basic dynamics of single and coupled RKII sets will be introduced. Based on the conclusions in Section 2, we propose in Section 3 the methodology to use coupled RKII sets as logic gates as well as basic structure in potential applications of pattern recognition. Section 4 presents the simulation results obtained through ODE solvers in MATLAB. Finally, in Section
9
,6&$6
P(t)
P(t)
M
M1
G1
G Fig. 2. A reduced KII set consists of one mitral cell and one granule cell that are coupled through Kmg (> 0) and Kgm (< 0).
5, we will give conclusions and discuss possible structures to use RKII networks with more computational power. 2. DYNAMICS OF SINGLE AND COUPLED REDUCED KII SETS An RKII set is an input controlled oscillator [14, 15] that is described by the following ordinary differential equations (ODEs) ⎧ 2 d m(t) dm(t) 1 ⎪ ⎪ ⎪ + (a + b) · · + m(t) ⎪ ⎪ a·b dt2 dt ⎪ ⎪ ⎪ ⎨ = Kgm Q(g) + P, Kgm < 0 2 d g(t) dg(t) ⎪ 1 ⎪ ⎪ + (a + b) · · + g(t) ⎪ ⎪ a·b dt2 dt ⎪ ⎪ ⎪ ⎩= Kmg Q(m), Kmg > 0
(1)
where 1/a and 1/b are time constants of the 2nd -order dynamics. They are given experimentally as a = 220/s and b = 720/s [8]. P is a time-invariant input. Q(x) is the nonlinear function that models the spatio-temporal integration of spikes into mesoscopic waves measured in the cortex [8] and is defined by the following equation: Q(x) =
Qm · (1 − e −1
x
− eQ−1 m
)
x > x0 else
(2a) (2b)
where x0 = ln(1 − Qm · ln(1 + 1/Qm )). Qm is an adjustable parameter that controls the ratio between positive and negative saturation values of Q(x). As discussed in [15], when Eq. (3) is satisfied, the behavior of a RKII set is determined solely by the input.
Q
(a + b)2 (a + b)2 1 · < |Kmg · Kgm | < (3) ∗ a·b a·b · Q (g )
(m∗ )
In Eq. (3), m∗ and g ∗ are the fixed point solutions of Eq. (1), which are also determined by the external input P (t). Under this condition, positive input will cause a RKII set to oscillate while with zero input, it will stay at equilibrium. In order to help understand the more complicated dynamical behavior of Freeman’s KII and KIII networks, in [12], a synchronization analysis is performed for two identical coupled RKII sets
P(t) M2
Kgm
Kmg
Kgm
Kmg
KMM
Kgm
Kmg KGG
G2
Fig. 3. Two coupled identical reduced KII sets. Kmm and Kgg are the interconnection weights among excitatory and inhibitory cells respectively.
(Fig. 3). The configuration of coupling in terms of the first system is defined as ⎧ (1) m ¨ + (a + b) · m ˙ (1) + m(1) ⎪ ⎪ ⎨ = ab · (Kgm · Q(g (1) ) + P (t) + Kmm · Q(m(2) )) . (4) ⎪ g¨(1) + (a + b) · g˙ (1) + g (1) ⎪ ⎩ (1) (2) = ab · (Kmg · Q(m ) + Kgg · Q(g )) where Kmm > 0 and Kgg < 0 are the interconnection weights. m(2) and g (2) are coupled from the second system. The term identical here means that the two RKII sets have exactly the same internal coupling weights (Kmg and Kgm ) as well as external input P (t). Analytical solutions are given in [12] on the interconnection weights Kmm and Kgg so that synchronized behavior could be achieved between the outputs of M1 and M2 of the coupled RKII sets. Given a specific set of external inputs and internal couplings, the synchronization boundary can be computed in the parameter space of Kmm and Kgg . Apparently, there are two typical behaviors in the state space of M1 and M2 : identical synchronization (IS) and desynchonization (which may include 180 degree phase difference as well as quasi-periodic motion). Note that IS indicates a synchronization of both amplitude and phase. 3. COUPLED NEURAL OSCILLATORS AS LOGIC GATES 3.1. Design Methodology and Measure of Synchronization From previous section, we know that there are two basic behaviors for two coupled RKII sets. Based on the analysis, if we denote synchronization as state 1 and desynchronization as state 0, the coupled system is a logic function when defined as f (I) = S(m(1) , m(2) , θ)
(5)
where I = {0, 1} × {0, 1} is the input space and S = {0, 1} is any function that determines whether or not the outputs of the two systems m(1) and m(2) are synchronized. θ is the threshold used in S. Note that I is not limited to binary values of 0 and 1. Basically, any two values that represent higher and lower level inputs can form the input space. Function S is just a symbolic system that you can assign either 0 or 1 to any of the two states. This means that, if f can implement one set of the basic logic computations (for example, AND, OR, XOR), it should also be able to realize their complements with a different assignment of states in the actual measure of synchronization. The NOT function can also be implemented with many options. One example is to
9
short the inputs of NAND gate or fix one of its inputs to 1. A two-input logic gate has a total of 16 boolean functions. In this paper, we only conisder the case that the coupled oscillators have a symmetric structure. So it cannot distinguish between the inputs (0,1) and (1,0). However, this could be solved by using weighted inputs. The weights then become the parameters in addition to Kmm and Kgg that need to be designed. By considering only the symmetric structure, without combination of different gates, the coupled oscillators have the ability to implement 8 out of the 16 logic functions that include all the boolean primitives. They cover most of the universal sets (for example, {AND, OR, NOT} and {NAND}) that can express any logic functions. Synchronization can be measured using either correlations between the outputs of the two sets or direct detection of phase difference. Equation (6) calculates the correlations between the time averaging between the two state variables . C(m1 , m2 ) < m1 · m2 > − < m1 >< m2 > = (< m21 > − < m1 >2 )(< m22 > − < m2 >2 )
(6)
Notice that when the two sets are synchronized, the phase plot is a straight line. When they are desynchronized, there may be either limit cycle or quasi-periodic motions in the phase plane. So another way is to plot the bifurcation diagram, so that in the case of desyncronization, multiple period could be detected.
However, we have not yet devised an automated rule. It is also interesting to mention that a single layer of coupled RKIIs yields non convex regions that implement the XOR and XNOR, unlike topologies of static nonlinearities (the MLP requires a hidden layer to solve the XOR problem). 4. SIMULATIONS AND DISCUSSIONS In the simulations, we fix Kmg = |Kgm | = 3 and use {0,1} as the input space. Different values of Kmm and Kgg are searched to see if all the six basic logic gates could be realized. Given the simple structure of just two coupled oscillators, the results are very promising. Using a synchronization criterion that assigns state 1 to synchronization and state 0 to desynchronization, the coupled RKII sets can implement AND, NOR and XNOR functions. That is, by assigning 1 to desynchronization and 0 to synchronization, we have the complementary functions, i.e. all the six logic functions. Figures 4 and 5 show the implemented XNOR function in the time domain and phase plane respectively. Transient behaviors are discarded to guarantee a steady-state of the outputs. In the case when inputs are [1 1] and [0 0], the two reduced KII sets are perfectly synchronized. When inputs are [1 0] and [0 1], the two oscillators are desynchronized and even present chaotic behaviors.
Input: [0 0]
5
3.2. Determining the Coupling Coefficients
[Λ1 ] < 0,
(7)
where [Λ1 ] is the largest real part of the eigenvalues of the Jacobian matrix computed from the dynamical system in terms of the differences between the two oscillators. It turns out that in our particular case, most of the synchronization boundaries are actually monotonically increasing functions with respect to Kmm and Kgg . So, for example, there exists a bifurcation point in the coupling space that once crossed for larger coupling produces synchronized outputs, while with a smaller coupling we will see desynchronized behavior. A rule of thumb is sgn(δK) = D − S, where D is the desired state of synchronization and S is the measured state.
4
3
3
2
2
1
1
0 Ŧ1
0 3.1
3.15
3.2
3.25
Input: [1 0]
5
Amplitude
The interconnection coefficients determine when the two oscillators will be synchronized. Given a desired logic function, how can we decide the values of Kmm and Kgg ? There are two basic approaches depending on actual applications. If our purpose is to build each of the six gates given specified input values, we can simply search the parameter space to get the appropriate set of coupling strengths for every logic gate. After we have all the settings for the required functions, the couplings will never be changed. A second alternative calls for the design of learning rules, as done in neural networks. The RKII can be thought as a learning machine implementing discriminant functions (DF) (such as the multiplayer perceptron), where the output space has only binary values and the coupled oscillators are actually dividing the input space in a particular way by means of DF. The purpose of setting the coupling coefficients is to move the DF around and to look for the best way of mapping the input space to the desired output space configuration. In such cases, an hypothetical learning algorithm should be able to train the interconnection weights based on different problem settings. From [12], the criterion to build synchronized oscillators is:
Ŧ1
3.1
4
3
3
2
2
1
1
0
3.15
3.2
3.25
Input: [1 1]
5
4
Ŧ1
Input: [0 1]
5
4
0 3.1
3.15
3.2
3.25
Ŧ1
3.1
3.15
3.2
3.25
Fig. 4. Time domain response when implementing coupled RKII sets as XNOR gate. (Solid line shows the output from M1 while dashed line shows the output from M2 .) Table 1 gives the parameter values to build the logic gates. In the second column of the table, the label (0 or 1) defined for synchronized behaviors in S is indicated. Table 1. Configurations for different logic functions Function Sync. Kmm Kgg Kmg Kgm AND (NAND) 1(0) 1.75 -2.0 3 -3 NOR (OR) 1(0) 1.2 -0.4 3 -3 XNOR (XOR) 1(0) 1.5 -0.4 3 -3 One important question is how to extend this result to a network of oscillators. There are basically two structures. The first one is to implement the logic gates just as in digital design. That
9
Input: [0 0]
Input: [0 1]
4
6. REFERENCES
2.5 2
3
[1] D. Wang, “Relaxation oscillators and networks,” in Wiley Encyclopedia of Electrical and Electronics Engineering, J. G. Webster, Ed. New York: Wiley Sons, 1999, pp. 396–405.
1.5 1
1
0.5
M2
2
0 0 Ŧ1 Ŧ1
0
2
4
Ŧ1
0
Input: [1 0]
2
4
6
Input: [1 1]
6
5
5
4
4 M2
[2] K. Chen and D. Wang, “A dynamically coupled neural oscillator network for image segmentation,” Neural Networks, vol. 15, no. 3, pp. 423–439, Apr. 2002.
Ŧ0.5
[3] L. Chua, CNN: A Paradigm for Complexity. World Scientific Pub Co, 1998.
3
[4] L. Chua and L. Yang, “Cellular neural networks: Theory and applications,” IEEE Trans. Circuits Syst., vol. 35, no. 10, pp. 1257–1290, Oct. 1988.
3 2 2 1
1
0
0 Ŧ1 Ŧ1
New York:
0
1 M1
2
3
Ŧ1
0
2 M1
[5] L. Chua and T. Roska, “The cnn paradigm,” IEEE Trans. Circuits Syst., vol. 40, no. 3, pp. 147–156, 1993.
4
[6] F. Hoppenstadt and E. Izhikevich, “Pattern recognition via synchronization in phase-locked loop neural networks,” IEEE Trans. Neural Networks, vol. 11, no. 3, pp. 734–738, May 2000.
Fig. 5. Phase plot in the state space of M1 and M2 when implementing coupled RKII sets as XNOR gate.
is, outputs from one set of coupled oscillators will be the input of the next one. And combinations of different gates will implement more complicated logic functions. The other way is to fully connect N RKII sets. In this network, if we consider all the combinations of the synchronization between any subset of the N oscillators, the output space will be of the size 2N . This shows the great potential of this network in terms of capacity. We already successfully showed that the areas in the parameter space exist to get all desired logic functionalities. However learning is still an open question because we still need to show that the coupling coefficients are trainable regarding both computational efficiency and ability of convergence. Also, although the potential capacity of this network is very large, the controllability of all the 2N attractors in the output space is not guaranteed. Theses are some of the questions that need to be further investigated. 5. CONCLUSIONS Based on the synchronized behaviors of two coupled neural oscillators, we proposed a novel method to build basic logic gates from this structure. Experiments showed that the intrinsic structure of the coupled sets provide us with very flexible configurations regarding the coupling strengths so that all six logic gates can be implemented. The results and discussions demonstrate additional computational power that this particular oscillatory network has.
Acknowledgement This work was partially supported by ONR N00014-1-1-0405.
[7] R. D. Henkel, “Segmentation with synchronizing neural oscillators,” Center for Cognitive Science, Bremen, Tech. Rep. ZKW-Report, Apr. 1994. [8] W. Freeman, Mass Action in the Nervous System. New York: Academic, 1975. [9] J. Principe, V. Tavares, J. Harris, and W. Freeman, “Design and implementation of a biologically realistic olfactory cortex in analog VLSI,” Proc. IEEE, vol. 89, no. 7, pp. 569–571, July 2001. [10] R. Kozma and W. Freeman, “Chaotic resonance-methods and applications for robust classification of noisy and variable patterns,” Neural Networks, vol. 4, no. 1, pp. 103–121, 2001. [11] Y. Yao, W. Freeman, B. Burke, and Q. Yang, “Pattern recognition by a distributed neural network: An industrial application,” International Journal of Bifurcation and Chaos, vol. 11, no. 6, pp. 1607–1629, 1991. [12] D. Xu, J. Gao, and J. Principe, “Synchrnoization analsysis of neural oscillators in the olfactory system,” to be submitted to Phys. Rev. E. [13] V. Tavares, “Design and implementation of a biologically realistic olfactory cortex model,” Ph.D. dissertation, Univ. of Florida, Gainesville, May 2001. [14] D. Xu, L. Deng, J. Harris, and J. Principe, “Design of a reduced kii set and network in analog VLSI,” in Proc. IEEE International Symposium on Circuits and Systems (ISCAS’03), Bangkok, Tailand, May 2003, pp. 837 –840. [15] D. Xu and J. Principe, “Dynamical analysis of neural oscillators in an olfactory cortex model,” accepted by IEEE Trans. Neural Networks.
9
