information processes in the brain [6]-[8], experimental analysis, and mathematical modeling of biological subsystems p.erforming information processing (in ...
E E E TRANSACTIONS ON NEURAL NETWORKS, VOL. 7, NO. 2, MARCH 1996
290
formation rocessing Using Networks I
ic
Yuri V. Andreyev, Yuri L. Belsky, Alexander S. Dmitriev, and Dmitrij A. Kuminov
piecewise-linear one-dimensional (1-D) maps of an interval into itself. This concept differs substantially from the notion of a neural network as a “black box.” We shall show below that the existence of a mathematical model which can be realized by means of a neural network of this kind leads to striking computational efficiency. Another important property of such networks is the limited number of interconnections, making the I. INTRODUCTION proposed architecture suitable for very large scale integration UE to recent investigations of the nature of electroen- (VLSI). cephalograms (EEG’s) of human and animal brains In Section I1 we give a brief summary of earlier work on [ 11-[5], the development of a qualitative understanding of the problem of storing and retrieving information using 1-D information processes in the brain [6]-[8], experimental maps. Here, an example illustrating the main principles of analysis, and mathematical modeling of biological subsystems storing sequences containing repeated fragments is given. p.erforming information processing (in particular, in the In Section 111, we discuss a realization of 1-D map dynamics rabbit olfactory system [9]-[ lo]), the fundamental role of by means of a recurrent neural network with a single hidden deterministic chaos, and complex dynamics in information layer. We show that the algorithm of the network training processing by biological systems has becoming clear. €or production of the input-output map is reduced to simple These results lead naturally to the exciting idea of using formulas, and illustrate the method with two examples. complex dynamical phenomena in deterministic systems to Section IV is devoted to storing and retrieving pictures using develop new information technologies. Underlying our ap- recurrent neural networks with chaos. Here, the representation proach to information processing using deterministic chaos is of binary and color pictures as strings is considered. We the concept of an attractor in the phase space of a system; it is presented examples of preliminary encoding of the strings, the properties and bifurcations of an attractor which determine forming maps in which the strings are related to stable limit the information processing capabilities of a system. cycles, and examples of how the map’s dynamics may be The possibility of using chaos and complicated dynamics in realized using recurrent neural networks. The absence of reneural networks to solve problems associated with information strictions on the type of the stored sequences is demonstrated. processing has already been discussed in the literature. In In particular, even strings which differ only slightly from each particular, questions related to the storage and retrieval of other can be stored. We then give an example of storing and temporal sequences in neural networks were discussed in [ 111 retrieving pictures, consider the computational efficiency of and [12]. Learning a sequential structure in simple recurrent the process for map design, demonstrate training of a neural networks was studied in [13]. A wide range of problems con- network with up to 20 000 elements, and illustrate information nected with chaos in natural and artificial neural networks has retrieval from the network. been investigated by Freeman ([ 141 and references therein). In Section V additional opportunities related to the transition It should be noted, however, that in most papers connected from information storage on stable limit cycles to storage on with the use of chaos and complex dynamics in neural net- unstable limit cycles are discussed, along with the realization works, the neural network itself plays the role of a “black of a global chaotic regime in the system. In particular, the box” which is trained according to certain rules to obtain a possibility of using neural networks with such dynamics for desired response t o a prescribed input signal. image recognition is demonstrated. An example is given. The goal of this research is to demonstrate the application Finally, we summarize the results of this paper and draw some of complex dynamics and chaos to storing and processing conclusions. information in recurrent neural networks, which provide a hardware realization of simple mathematical models of a
Abstract-In this work, we study information processing applications of complex dynamics and chaos in neural networks. We discuss mathematical models based on piecewise-linear maps which enable us to realize the basic functions of information processing using complex dynamics and chaos. Realizations of these models using recurrent neural-like systems are presented.
Manuscript received February 23,1993; revised April 20, 1994 and October
3, 1995. This work was supported in part by the Russian Foundation for Fundamental InvestigationsGrant N 93-012-730 and by INTAS Grant INTAS-
11. STORING AND b T R I E V I N G INFORMATION USING LIMITCYCLES OF I-D MAPS
94-2899. The authors are with the Institute of Radio Engineering and Electronics, Russian Acadeinv of Sciences. 103907 Moscow. Russia. Publisher Item’ Identifier S 1045-9227(96)01253-2
The procedure for storing and retrieving information on the basis of limit cycles in 1-D dynamical systems was introduced in [15] and [16] and further developed in [17]-[20]. Here is a
1045-9227/96$05.00 0 1996 IEEE
29 1
ANDREYEV et al.: INFORMATION PROCESSING USING DYNAMICAL CHAOS
brief description of the method. Let ala2
. . . a,
Output Layer
(1)
be a given sequence of symbols (called an information block), each element a, of which belongs to an alphabet composed of N symbols. A 1-D map of an interval into itself is designed for this sequence. The map is chosen such that it possesses a stable limit cycle with period n, the iterates of which are in mutual unambiguous correspondence with the elements of the sequence (1). In the simplest case, each element of the alphabet is related to its own value of the mapping variable and to an interval of thle mapping variable of the length 1/N [15], [16]. Retrieval of the information block is performed by setting the initial condition within one of the segments on the unit interval [0, 11 corresponding to the symbols of the information block, and by further transforming the sequence of numbers generated by iterates of the map into the sequence of symbols. If the initial condition is arbitrary, convergence of the system trajectory to the limit cycle is preceded by a transient process. Similarly, a map with a few stable limit cycles can be formed, whose elements are unambiguously related to the elements of the corresponding stored sequences. The information storage capacity of this system is very limited. For example, two strings which contain at most two equal symbols cannot be stored simultaneously. To overcome this limitation the concept of storage level is introduced, and a map is designed in which each value of the mapping variable corresponds not to a single element of the sequence (1) but to q consecutive elements [ 151, [ 161. Retrieval of information in this case is performed by setting the initial condition within one of the sjegments on the unit interval [0, 11 corresponding to a fragment of an information block which is q symbols in length, iterating the map from this initial condition, and by then transforming the sequence of numbers back into a sequence of symbols. Such a generalization of the map design procedure enables one to store arbitrary symbol sequences in which there are no coinciding fragments containing q or more common elements. Otherwise, storage at this level is impossible. A natural way to overcome this limitation is to use a higher storage level, but this way leads to other restrictions. Actually, as the storage level increases, the length of the information intervals in the map decreases, thus forcing us to proceed from ordinary precision calculations to special high-precision calculations with all the corresponding problems (deceleration of calculations, increased memory requirements, etc.). The problem of storing arbitrary sequences can be solved by means of a special coding procedure [17]-[20] which we describe briefly below. Let the sequences be stored at a qth level, and let them contain a few identical fragments of length q. A new generalized element representing this fragment is added to the alphabet and all inclusions of this fragment in the sequences are replaced by the new element of the alphabet. If, upon the next presentation of the sequences (using the newly added element of the alphabet), identical fragments of the length q are once again found, they are replaced by another new element of the alphabet. Alphabet extension is continued
Hidden Layer
Input
Layer
Fig. 1. Neural network structure
until the stored information sequences contain no repeating fragments of the length greater than or equal to q. This encoding procedure allows arbitrary information blocks to be stored at any level beginning with the second. Example I : Let us consider in detail the procedure for storing two sequences “abc...dxyw...” and “cde...kxyz. ..” made up of symbols from the Latin alphabet. Let the storage level be q =2. Let us take the first fragment of the first information block “ab” and look through the sequences. If there is no other inclusion of this fragment, take the next fragment “bc” and repeat the search. Let a fragment “xy” which occurs at ith step be encountered again in the sequences. Then a new element p = “xy” is added to the initial alphabet, and every “xy” in the sequences is replaced by p. The sequences now look like: “abc...dpw...” and “cde...kpz ....” Then the search process for identical fragments is repeated with the fragment “pw.” If other fragments are found to repeat in the sequences then a new element is added to the alphabet for each such occurrence, and corresponding substitutions are made. The process finishes when the last fragment of the second sequence is reached. As a result, a new extended alphabet is formed. The original sequences expressed in this alphabet now contain no repeating fragments of the length greater than or equal to q. The number of computer operations necessary to form the new alphabet and to represent (to encode) the original sequences with it, is proportional to the square of the amount of the stored information, i.e., for an amount L of stored information, the number of operations is -L2.
111. REALIZATION OF THIS METHODFOR STORAGE AND RETRIEVAL OF INFORMATION USING RECURRENT NEURALNETWORKS Storage and retrieval of information based on limit cycles in 1-D maps can be implemented by means of recurrent neural networks. Let us consider a network composed of three layers of neurons (called the input layer, the hidden layer, and the output layer) and a feedback loop from the output layer to the input layer (Fig. 1). The three layers of the neural network are used to emulate a 1-D map function, and the feedback loop with unit delay is to formulate the system as a discrete-time dynamical system. The input and output layers contain one element each. There are no couplings between the elements of the hidden layer. A signal corresponding to an initial condition is applied to the input element. The signal from the output of this element is sent to the neurons of the hidden layer. The sum of the signals from the hidden layer neurons is applied as input to the output
IEEE TRANSACTIONS ON NEURAL. NETWORKS, VOL. 7, NO. 2, MARCH 1996
292
layer neuron. The coefficients of the couplings between the layers of the neural network are determined by the function of the information storage map. The signal at the network output represents the value of the next iteration of this map. All the neural-like elements have the same structure and the same piecewise-linear characteristics. The threshold values for the elements of the input and output layers are zero, the threshold values for the elements of the hidden layer are determined by the form of the function which is realized in the network. It is known [21], [22] that an arbitrary mapping of R”into Rz(R” and Rz being real m- and I-dimensional spaces, respectively), can be approximated to any arbitrary degree of accuracy by a multilayer feedfonvard neural network composed of elements with sigmoid characteristics. It is easy to see that an arbitrary 1-D piecewise-linear map M ( z ) defined at the unit interval 1 = [0,1]
M
:
[O, 11 -+ [O, 1](1+I)
(2)
can be precisely realized as a neural network, composed of elements with identical piecewise-linear characteristics. Indeed, let each element of the hidden layer have characteristics
f(.) =
{
0, IC,
xl0,
0
1,
< x 5 1,
(3)
2>1
and let the 1-D piecewise-linear map M ( x ) be defined by the set of points
, Yl), (x27
%!)7
‘’
’
( z P + l , YP+1)
(4)
where P is the number of linear segments in the 1-D map; (x,1 Yz) are the coordinates Of the left and right endpoints Of the linear segments. Then the map function M ( x ) may be represented as N
N
0.0
0.5
x
(b) Fig 2 Dynamcal system with information block 174 stored at the first level. (a) the map function and (b) resolution of the map by neural element functions.
Example 3: The same information block as in Example 2 is now stored at the second level. We want to create a neural a=1 a=l network which realizes the dynamics of the corresponding 1-D where a , = x,+1 - x, , pt = ya+l - ya. T,’ = l / a a are the map. The required mapping function is presented in Fig. 3. The couplings from the input layer to the hidden layer, T,” = ,Bz structure of the neural network which emulates the dynamics are the couplings from the hidden layer to the output layer, of this map is the same as in the previous example. The and 0, are the thresholds of the elements in the hidden layer. corresponding coupling coefficients are given in Table 11. It Example 2: Let there be an alphabet consisting of the 10 should be noted that because of the decreased lengths of the digits 0, 1, 2,. . ., 9. Assume that we wish to construct a neural information storage regions due to the information block being network which realizes the dynamics of a 1-D map with the stored at the second level, specificationsfor the precision of the information block “174” stored at the first level. This map characteristics of the neurons and on the values of the coupling has the form presented in Fig. 2(a). The neural network which coefficients are stricter than for storage at the first level. emulates the dynamics of this map contains seven (2 . 3 1) elements corresponding to the three “information regions” of I v . STORING AND RETRIEVING PICTURES the map. The coefficients of the couplings between the neurons are calculated according to expression (5) and are shown in Let us consider an application of our method and its neural Table I. The contribution of each neuron to the formation of network implementation in the case of two-dimensional (2the map function is shown pictorially in Fig. 2(b). As can be D) pictures. The following procedure is used. We first divide seen from the figure, each neuron element is responsible for the a 2-D picture into elements, then choose an alphabet from creation of one linear region in the map. The neural system these elements, and introduce a rule for scanning the elements is started by applying an initial condition to the input layer of the picture. Each of these steps may be done in a different element with the feedback loop disconnected. The feedback way. Here we demonstrate one which seems natural and rather loop is then closed. efficient.
+
ANDREYEV et a/.: INFORMATION PROCESSING USING DYNAMICAL CHAOS
293
TABLE I PARAMETERS OF A NEURAL NETWORK REALIZING A MAP WITH INFORMATION BLOCK174 STORED AT THE FIRSTLEVEL
N
T;
ON
TABLE I1 PARAMETERS OF A NEURAL NETWORK REALIZING A MAPWITH INFORMATION BLOCK174 STOKED AT THE SECOND LEVEL
1
10
0.745
0
5.88
0.743
2
10
0.01
1
100
0.005
17
3
5
-0.61
1
3
4.34
-0.57
0.78
4
10
0.01
4
4
100
0.005
41
5
5
0.29
2.5
5
3.13
0.24
1.31
6
10
0.01
7
6
100
0.005
74
4
7
4
-0.42
3
1
7
5
I
I
-0.455
xm+i
0.5
This sequence is the information block in this case. It can be related to the limit cycle of a 1-D map in the same way as for the examples from Section 111. Note that this cycle corresponds also to all cyclic permutations of the information block (7), i.e., to any information block of the type
-
at^ ‘ azn‘ ’ amy ‘ a m n a l l aina2l’. a2n’ ‘ a ( z - 1 ) ~ ’ a(z-1)n.
X
0.5
0.0
m
Fig. 3. Map with information block 174 stored at the second level.
Assume that we have a 2-D color (in the general case) picture. The first step consists of spatial digitization of the image, translating the initial picture into a pattern of m x n cells. Each of these cells is painted in some color from a chosen palette, a color nearest to that dominating in the cell. A palette with N, colors forms the basis of the alphabet, and the procedure far painting the cells with a limited number of colors corresponds to “color digitization,” or resolving the picture into color ellements of the alphabet. The pattern corresponding to the picture is then treated as a 2-D matrix
A = lluzJ1l, i = l , . . . , m , j = 1 , ” . , n
(6)
where azJis an element of the alphabet. The next step is to construct an information block. This could be done in a variety of ways as long as the transformation from a pattern into an information block and its inverse transformation are unambiguous. Consider the noninterlaced “TV-type” transformation. In this case, the pattem (matrix A) is scanned line by line from top to bottom. As a result, a 1-D sequence of length m x n is obtained U11
. . . UlnU21 . *
u2n
.
Um1
* *
. umn.
(7)
(8) This causes a problem for pattern retrieval by iterating the dynamical system because there is no information about which element of the cycle corresponds to the beginning of the pattem. To overcome this problem, a special symbol is added to the alphabet and to the sequence: a label that is placed at the beginning of the information block. When the label appears, the first element following the label is considered the first element of the picture and hence any possible ambiguity in pattern retrieval can be avoided. Thus, in the case of storing color pictures the alphabet consists of N , 1elements (number of colors N, plus one new element for the label). The main problem in storing pictures by this method arises from the possible presence of long identical fragments. For example, the maximum length of coinciding fragments for the photograph in Fig. 5(a) is 40 (16 gray-scale colors are used as the basis of the alphabet). To store a picture in this case, preliminary coding is used. Example 4: Let us assume that we wish to store a 8 x 8 pattern representing a chessboard [Fig. 4(a)]. The initial alphabet consists of just three symbols: a white cell (W), a black cell ( B ) ,and a label ( L ) . Now form the information block. It is represented by a 1-D sequence of symbols beginning with the label and 64 alternating symbols corresponding to black and white cells. The second storage level is used. An information block can be stored at that level if there is no repeating pieces of length greater than or equal to two. Obviously this condition is broken for the given information block. Actually, a pair of symbols BW, corresponding to black and white cells, is repeated many times in the block [Fig. 4(a)]. Let us extend the alphabet by adding a new (fourth) element a2, corresponding to the pair BW. Then make the necessary substitutions in the
+
[EEE TRANSACTIONS 0” NEURAL NETWORKS, VOL. I, NO. 2, MARCH 1996
294
information block. After all substitutions, the block looks like
L~u~u~u~u~W .. .U ~ U ~ U ~ B I where a fragment between the brackets is repeated four times. The number of block elements has been reduced from 65 to 37, but there are now four identical pieces of nine elements. Adding an element a4 = azaz, we reduce the length of the information block from 37 to 25 elements
Lla4a4Wa4azB1. But this is also insufficient to fulfill the storage condition. By adding as = a404 we obtain
L~a~Wa4a2B~~asWa4azB~ . ... Then introducing c1 = a s W , we obtain
L~c~a4a2B~~cla4a2B~~~f. The next new element is tation becomes
c2
= cla4 and the block represen-
L C ~BQc ~ u ~ B c ~ u ~ B c ~ c L ~ B with
c3 = cza2,
the block representation is
L cB~ cB~ cB~ cB~ adding
e4
= c ~ Bthe , block becomes
Le4 c4 c4 c4 and so on, until the final substitution e5 = c4c4 gives
Lcg c5. Thus, in the process of encoding, the alphabet was extended from three symbols ( B , W , and L) to nine, and the length of the information block was reduced from 65 elements (in the initial alphabet) to three elements (in the final alphabet). In the given example the length of the information block is reduced almost by a factor of two for each new element of the alphabet, i.e., the information block length reduction is nearly exponential. Equivalently, one can say that the alphabet length increases as the logarithm of the pattern length if the alternating black-and-white structure is retained. We will show below that in real multicolored pictures compression ratio is not so large, and the length of the final alphabet scales approximately linearly with total length of all patterns (the total amount of information stored). To visualize the process of image “orthogonalization” by encoding, consider the following example. Example 5: Let two similar images be stored in the same map: a chessboard and a chessboard with a “defect” [Fig. 4(b)]. In this case, as in Example 4, the initial alphabet consists of the same three elements: a white cell, a black cell, and a label. Two information blocks are formed. The first is a sequence of symbols beginning with the label, and containing 64 other alternating symbols corresponding to the black and white cells. The second block looks like
LIBWBWBWBWIBBWBWBWBIBW . . . W B .
(b)
Fig. 4. A picture of a chessboard: (a) original picture and (b) a chessboard with a “defect.”
Let us extend the alphabet by adding the fourth element d l , corresponding to the pair LB. Then make the necessary substitutions in both information blocks. As a result, we obtain
and
ANDREYEV et al.: INFORMATION PROCESSING USING DYNAMICAL CHAOS
295
The element d l W is twice encountered in these blocks, so we replace it ,with the element d2 = d l W and obtain
d2IBWBWBWIWBWBWBWIBW * .. d 2 ) .. . J B B . . . Then we introduce the elements d3 = dzB, d4 = d3Wl dg = d4B1de; = dSW, d7 = deB, d8 = d7W. As a result we obtain
dsWBWB... d8BBWB.e.. Thus, the identical pieces at the beginning of the blocks were compressed into a single element. The pair of elements dsW is unique for both blocks. The next pair W B is repeated many times. Let us designate it as 12. The information blocks now look like
d s ( ~ z ~ ~ ~ 2 ~ z. .). (12121212 B~~~z~zW~ ds)BB121212). . . 112121212. Further encoding through the succession of substitutions = 1212,18 = 1 4 1 4 , f i = b B , f 2 = f i b , f 3 = f3W, f 8 = f 4 f 4 , hl = B14, ha = h112, leads to
14
f212,f4
=
.f f
d8 8 418
dsBh2hznrf8k and finally (h3 = &d8) to h3fsf4
h,Bh2h2Wfg. Finally, the first block consists of three elements and the second of six. The final extended alphabet consists of 22 (b) elements, only six of which are used in the compressed Fig. 5. A picture of a girl: (a) original picture 320 x 200, 16 gray colors information blocks. The map contains two cycles of length and (b) transient process, observed by picture retrieval. 3 and 6, respectively. The neural network emulating dynamics the neural network representation is 15 times greater than of the map contains 19 neurons in the hidden layer. that by iterating the map directly. In case of a hardware Example 6. Let us discuss now the capabilities and features of the method for storing and retrieving 2-D patterns (pictures) implementation of the neural network, however, we suggest for a more complicated example: storage and retrieval of that the situation will be the opposite, because only one step photographs. A gray-scale picture of a girl is presented in will be needed to retrieve a symbol (and the corresponding Fig. S(a). To store it as a cycle in a dynamical system, the group of pixels) of the final generalized alphabet. The efficiency of the method which we have outlined for picture was presented as a pattem of 320 x 200 pixels, with storing and retrieving information, and its neural network a palette of 16 gray-scale colors as the basis of the initial implementation, is closely connected with the rate at which alphabet. The length of the initial information block was the final alphabet increases compared to that of the stored 64001 elements. To store this picture, encoding for storage information. If the alphabet increases in a highly superlinear at the second level was performed. manner, it seriously restricts the maximum amount of stored We performed numerical experiments to study the dynami- information. Preliminary estimates of the rate are hard to make, cal properties of the map with a stored picture. Fig. 5(b) shows but Example 7, given below, indicates that the length of the a typical picture which appears for arbitrary initial conditions. final alphabet for typical human face pictures increases linearly Several “noisy” (chaotic) lines at the top of the pattern window with the amount of information stored. correspond to transient chaos, preceding the moment when the Example 7: In Fig. 6 nine pictures are presented. Each phase trajectlory enters the vicinity of the cycle. A “fall” onto pattern is 100 x 100 and each has 16 levels of brightness. the cycle can occur at any point of the cycle. After this a piece Various numbers of patterns were stored in a map. The results of the stored picture appears at the screen, the beginning of are presented in Fig. 7. The plot indicates practically linear the piece corresponding to the point of the cycle at which the dependence of the length of the alphabet on the quantity of trajectory had “entered.” When the point corresponding to the stored information. label is reached the noisy part of the picture is redrawn and a V. INFORMATION PROCESSING USINGRECURRENT NEURAL screen with the “clear” picture appears, as in Fig. 5(a). NETWORKS REALIZING THE DYNAMICS OF 1-D MAPS, Simulations were performed not only of direct iterations of WITH INFORMATION STORED AS UNSTABLE CYCLES the I-D map, but also of the neural network whose behavior Storing information as unstable cycles of a l-D map is emulates the map. The hidden layer of the network contains 18 5 11 elements. The time for information retrieval using performed in the same way as storing using stable cycles,
296
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL I, NO 2, MARCH 1996
0-
2
rt:
6
8
M
Fig 7. Dependence iof the extended alphabet length on the amount of information stored.
Fig. 6. Graphic information stored in the dynamical system: Nine pattems 100x100, 16 gray colors.
the only difference being that the product of the slopes of the information regions of the map, those through which the cycle passes, is greater in magnitude than one in this case [17]-[20]. There are no stable limit cycles in such a map. Hence for arbitrary initial conditions a trajectory of the system visits most parts of the phase space. Along with the cycles corresponding to information blocks many other unstable cycles can exist in the map. Recognition of information can be organized by reducing the slope of information regions corresponding to knowledge of the information presented as input. Let M information blocks be stored as unstable cycles in a 1-D map, and a let a symbol string of length L be presented for recognition. The question is whether this sequence of L symbols corresponds to any of the stored information blocks. We will treat this sequence as an input signal which controls the state of the system. The rules for the effect of this sequence on the system are as follows: 1) If the signal contains a fragment corresponding to an information region of the map function, then the slope k of this region is set less than one, llcl < l . 2) In other cases the signal does not affect the map function. If an input signal corresponds to one of the stored images, information regions which are inclined at an angle less than n/4 appear in the map. The total product of the slopes of the information regions of the cycle, corresponding to the input signal, becomes less than unity. A stable cycle then appears in the phase space of the system, and attracts trajectories for a certain set of initial conditions. Convergence of the trajectory to this stable cycle may be treated as recognition information. It is important that in this case a single attracting cycle exists in the phase space of the system. Therefore, if a strange attractor with a continuous measure, intersecting information regions of the map, existed before the change of the slopes of the information regions, then after the change the trajectory will
inevitably reach one of these information regions and converge to the corresponding cycle. This means that a “hole” appears in the chaotic set, through which the trajectory transfers from the chaotic to the regular regime. This situation represents loss of stability for a chaotic set and a transition from stable to metastable chaos. ‘The appearance of this transition (crisis of a strange attractor) adlows us to realize information recognition. Note also, that due to the uniqueness of the stable limit cycle appearing because of the crisis of the strange attractor, the recognition process is practically independent of the initial conditions for the phase trajectory. The choice of initial conditions determines only the duration of the transient process from metastable clhaotic set to the stable limit cycle, i.e., the recognition time. The dynamic properties of the system with information stored on unstable cycles are illustrated by means of an example with threle stored information blocks. Example 9: Information blocks 97583, 14568, and 123 are stored at the second level. The alphabet consists of 10 symbols, namely the digits G 9 . It is important that the phase trajectory in the initial map with unstable cycles visits all information regions of the map [Fig. 8(a)]. This property is expected to remain if the perturbations of the map introduced by the change of slopes are small. The sequences 1231, 135681, and 946839 were used as input signals: the stored information block without erroirs in the first case, a block with one error in the second case, and a block with two errors in the third case. In all three cases, a stable limit cycle appeared related to the corresponding information block, and after a transient process, coupled with the trajectory wandering over the metastable chaotic set, the trajlectory “fell” onto the corresponding stable limit cycle [Fig. 8(b)]. The structure of a neural network corresponding to the described map with dnven slopes in the information regions is presented in Fig. 9. This is a four-layer network. It contains 40 neural-like elements in the hidden layers, one element in the input layer, and one element in the output layer. By contrast witlh the neural net depicted in Fig. 1, there are two hidden layers in this network. The presence of two layers allows us to control the slope of the information regions. The first hidden layer is composed of neurons related to the
ANDREYEV er al.. INFORMATION PROCESSING USING DYNAMICAL CHAOS
291
v r
A
0.5
0.0 I 0
I
I
t
Fig. 9. Neural network emulating a map with driven slopes.
layer, shown in black, are all the same. As a result, the network depicted in Fig. 9 becomes equivalent to that shown in Fig. 1. Hence, it realizes a map with the initial slopes in the information regions. Assume now that the slope of ith information region should be changed to Icl. The leftmost endpoint of the noninformation region preceding this ith information region, and the rightmost endpoint of the noninformation region following one must remain undisturbed. Simple calculations show that this restriction is fulfilled if the parameter pcL,is given by
X
0.5
pcL,=l-h ko
0
300
t
(b)
Fig. 8. System with three information blocks 97583, 14568, and 123 stored as unstable cycles: (a) temporal behavior and (b) the process of recognizing information block 123.
information regions of the map. The second layer consists of neurons related to noninformation regions of the map, with extra neurons shown in Fig. 9 in black. With the help of the latter, the slopes of the information regions of the map are controlled. For example, let the initial values of the slopes of the information regions be ko. If the values of the slopes remain constant, the value of parameter p is set to zero. In this case, there are no couplings between the elements of the second
hidden layer, and the couplings between the elements of the first hidden llayer and the elements of the second hidden layer are unity. Since the output signals from the neurons in the first hidden layer lie within the unit interval [0, 11, the signals at the input and output of each neuron in the second hidden
'
Now we describe the process of image recognition in the neural system. Let M information blocks be stored in the network and a string of length L be presented to the system. Then the network functions as follows: Step 1) The feedback loop is opened. Step 2 ) If the first q elements ( q is the storage level) of the input string are related to an information region of the map, the network is modified by changing the corresponding coupling parameter p. The problem of searching for the corresponding information interval is solved as follows. We want to find a neural-like element which corresponds to an information region related to the fragment a, . . . . U ~ + ~ - I .Let us apply a signal corresponding to this fragment to the input of the neural network. Then all the neurons in the first hidden layer located to the left of the element related to this fragment will have unity at their output, while all the neurons located to the right of this element will have zero value at their output. The value of the signal at the output of the required element will be in the range from zero to one.
If the first q elements of the input string are not related to any information region of the map, the network is not modified. Steps 3) to (L - q + 1) The procedure of Step 2) for network modification is successively applied (L - q - 2)
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. I, NO. 2, MARCH 1996
298
times to other substrings with q elements from the second until the end of the input string. Step (L - q ) The feedback loop is closed and the process of iteration begins in the neural network; this can lead to convergence of the trajectory to a stable limit cycle, which appears in the system as a result of modification of the network. The appearance of a stable limit cycle indicates that the input signal is related to an information block stored in the system. Convergence to this cycle can therefore be treated as “recalling” or “recognition” of the input signal.
VI.
CONCLUSIONS
The models studied in this paper allow us to realize a set of basic functions in information processing using chaos and complex dynamics, such as: Storage and retrieval of information blocks by means of stable and unstable limit cycles of neural networks. Realization of associative memory, allowing the search for and identification of an information block from its fragment. Image recognition based on the use of storage using unstable cycles, introduction of an extemal signal and formation of a “metastable chaos-stable limit cycle” pair. Novelty filter. If an information signal which has not been stored in the system is applied to the input of a neural network with information blocks stored as unstable cycles, no stable limit cycle appears in the network after the input signal has been processed. Rather, the trajectory continues to wander chaotically over the phase space. Thus, the state “I do not know” is realized and the system works as a novelty filter. The discussed models should be noted for their considerable information capacity and high computational efficiency together with a limited number of interconnections. REFERENCES S. Osovetz, A. Ginsburg, V. Gurfinkel, L. Zenliov, L. Latash, V. Malkin, P. Melnichuk, and E. Pastemak, “Electrical activity of brain: Mechanisms and interpretation,” Uspekhi Fizicheskikh Nauk, vol. 141, no. 1, pp. 103-150, 1983 (in Russian). P. E. Rapp, I. D. Zimmerman, A. M. Alhano, G. C. Deguzman, and N. N. Greenbaum, “Dynamics of spontaneous neural activity in the simian motor cortex: The dimension of chaotic neurons,” Phys. Lett. A , vol. 110, pp. 335-338, 1985. A. Bahloyantz, G. Nicolis, and M. Salazar, “Evidence of chaotic dynamics of brain activity during the sleep cycle,” Phys. Lett. A , vol. 111, pp. 152-156, 1985. A. Bahloyantz and A. Destexhe, “Low-dimensional chaos in an instance of epilepsy,” in Proc. Nut. Acad. Sci. USA, vol. 83,1986, pp. 3513-3517. A. Destexhe, J. A. Sepulchre, and A. Bahloyantz, “A comparative study of the experimental quantification of deterministic chaos,” Phys. Lett. A, vol. 132, pp. 101-106, 1988. J. S. Nicolis, “Chaotic dynamics as applied to information processing,” Rep. Prog. Phys. vol. 49, pp. 1109-1187, 1986. -, Dynamics of Hierarchical Systems: An Evolutionary Approach. Berlin: Springer-Verlag. 1986. ~, Chaos and Information Processing: A Heuristic Outline. Singapore: World Scientific., 1990.
C. A. Skarda and W. J. Freeman, “How brain make chaos in order to make sense of the world,” Behavioral Brain Sci., vol. 10, pp. 161-165, 1987. Y. Yong and W. J. Freeman, “Model of biological pattern recognition with spatially ch.aotic dynamics,” Neural Nethiorks, vol. 3, no. 2, pp. 153-170, 1990. -, I.Guyon, L. Personnaz, J. P. Nadal, and G. Dreyfus, “Storage and retrieval of complex sequences in neural networks,” Phys. Rev. A, vol. 38, no. 12, pp. 6’365-6372, 1988. S. Dehaene, I. P. Changeux, and J. P. Nadal, ”Neural networks that learn temporal sequences by selection,” in Proc.. Nut. Acad. Sci. USA, vol. 84, 1987, PE). 2727-2731. D. Servan-Schreiber, A. Cluremans, and J. L. McCleland, ”Learning sequential structure in simple recurrent networks,” in Advances in Neural Information Processing Systems I, D. S. Tourelzky, Ed. Palo Alto, CA: Morgan Kauffman, 1989. W. J. Freeman, ‘Tutorial on neurobiology: From single neurons to brain chaos,” Inr. J. Bijiurcation Chaos, vol. 2, no. 3, pp. 451482, 1992. A. S. Dmitriev, “Storing and recognition information in one-dimensional dynamical systems,” Radiotekhnika i Elektronika, vol. 36, no. 1, pp. 101-108, 1991 (in Russian). A. S. Dmitriev, A. I. Panas, and S. O.’Starkov, “Storing and recognition information based on stable cycles of one-dimensional map,” Phys. Lett. A, vol. 155, no. 8,9, pp. 494499, 1991. Y. V. Andreyev, Y . L. Belsky, and A. S. Dmitriev, “Information processing in nonlinear systems with dynamic chaos,” in Proc. Int. Seminar Nonlinear Circuits Syst., Moscow, vol. 1, 1992, pp. 51i60. Y. V. Andreyev, A. S. Dmitriev, L. 0. Chua, and C. W. Wu, “Associative and random access memory using one-dimensional maps,” Inr. J. Bifurcation Chaos, vol. 2, no. 3, pp. 483-504, 1992. A. S. Dmitriev, “Chaos and information processing in dynamical systems,” Radiotekhizika i Elektronika, vol. 38, no. 1, pp. 1-24, 1993 (in Russian). Y. V. Andreyev, A. S. Dmitriev, and S. 0. Starkov, “Pattern processing using one-dimensional dynamic systems,” Software Package “Inform Chaos.” Moscow Institute of Radioengineering and Electronics, Russian Academy of Sciences. Preprint No. 2, p. 584, 1993 (in Russian). K. H o d , M. Stinchcombe, and H. White, ”Multilayer feedforward networks are universal approximators,” Neural Networks, vol. 2, pp. 359-366, 1989. -, “Universal approximation of an unknown mapping and it derivatives using multilayer feedforward networks,” Neural Networks, vol. 3, pp. 551-560, 1990. ,
Yuri V. Andreyev was bom in Ufa, USSR, in 1960. He received higher education in the Moscow Institute of Physics and Technology, graduating in 1983 He received the I’h D degree from the Institute of Radio Engineering and Electronics (IRE), IJSSR Academy of Sciences, Moscow, in 1993 Since then he has worked in the Institute of Radio Engineering and Electromcs (IRE), USSR Academy of Sciences, in the field of microwave sohd-state oscillators and nonlinear chaotic dynamics He is currently worlung in the area of information processing using chaotic dynamics
Yuri L. Belsky was born in Moscow, USSR, on May 26, 1968 He received higher education in the Moscow Institute of Physics and Technology, graduating in 1991, and received the Ph.D. degree from the Institute of Radio Engineering and Electronics, USSR Academy of Sciences, Moscow, in 1994 His research interests include information processing in nonlinear systems, complex dynamics, and chaos.
ANDREYEV et al.: INFORMATION PROCESSING USING DYNAMICAL CHAOS
299
Alexander S. Dmitriev was born in Kuibyshev, USSR, in 1948. He graduated from the Moscow Institute of Physics and Technology in 1971 and received the Ph.D. degree in 1974 from the same institute. He also received the D.Sc. degree from the Institute of Radio Engineenng and Electronics, USSR Academy of Sciences, Moscow, in 1988. He IS presently the Chief of the Nonlinear Dynamics Group in the Institute of Radio Engineering. His research interests include nonlinear dynamics, bifurcation phenomena, chaos, and neural networks.
Dmitrij A. Kuminov was born in Donetzk, USSR, on August 14, 1967. He received higher education in the Moscow Institute of Physics and Technology, graduating in 1991, and received the Ph.D. degree from the same institute in 1994. His research interests include nonlinear systems, complex dynamics and chaos, and neural networks.
