Topology learning solved by extended objects: a neural network model Csaba Szepesvaril, Laszlo Balazsl and Andras Lorincz Department of Photophysics, Institute of Isotopes of the Hungarian Academy of Sciences Budapest , P.O. Box 77, Hungary, H-1525
Abstract It
is
shown
that local. extended objects of a metrical topological space shape the
receptive fields of competitive neurons to local filters.
Self-orl!,anized topolol!,Y
learninl!, is then solved with the help of Hebbian learninl!, tOl!,ether with extended objecttl that provide unique illfonnaLion about neighborhood relaLiolltl.
A topo
graphic,al Illap if) dcduc,(xl and it; used to Dpeed up further adaptation in a changing
environment with the help of Kohonen type learning that teaches the neighbors of \vlnning neuro n s
as
well.
Introduction
Self-organized learning is a most attractive feature of certain artificial neural network paradigms (Grossberg, 1976, Kohonen, 198'1, Carpenter and Grossberg, 1987, Fiildiiik 1990). It is considered as a means of solving problems in an unknown environment. The architecture of the neural network, however, is in general 'inherited', in other words is prewired and may severely limit the adaptation possibilities of the net. An example is the Kohonen-lype lopographical lIlap that has a builL-in neighborhood relation. Various alLelIlpLs have been made 10 resolve this problem such as the self-building model of J:irit�ke (J:irit�ke, 19(1) and lhe neural gas network of Martinelz and Schulten (:\1arlinelz and SclmlLen, 1(91). The closest 10 lhe presenl work is lhe neural gas nelwork. Ie is based on lhe idea thaI lopology can be learnl on the basis of joint similarity. Ie determines adjacent neurons based on the distance measured in the metric of the input vector space. Consider, however, the example of a maze embedded in a two-dimensional Euclidean space. The two sides of one of the walls in the maze have very close input vectors in the metric of the Euclidean space though they may be very far from each other in the metric space brought about by the topology of the maze. The question then arises if and how and under what conditions a given network is capable of determining the topology the external world. The model we present here relies on the extended nature of objects in the external world. This method can take advantage of the same idea of joint similarity and provides a 1 Permanent
address: Janos Bolyai Institute of lVlathematics, C niversity of Szeged, Szeged, Hungary,
II-6720. E-mails:
[email protected].
1
simple route for exploring the neighborhood relations as the objects themselves bring the information. The receptive fields of lhe neurons are local fillers and lhe neural neLwork may be considered as a dimensional reducing sysLem thaI provides posiLion infoIlllation and neglects object delails. In order 10 take advanlage of neighboring relalions and lhe possibility of Kohonen type neighbor lraining a dislance function may be established wiLh lhe help of IIebbian leaming: exlended objects may overlap wilh receptive fields of diJrerent neurons and may excite more t.han one ne111'0n at. t.he same time. These ne11rons then aSS11me activ ities different. from 7,ero, and Hebbian learning may be 11sed t.o set 11p connectivity t.hat mat.ches t.he topology. The st.rengt.h of t h e connedion may be relat.ed t.o t.he dist.ance in f11rther processing. The greater t.he strength , t.he smaller t.he distance. In t.his way a t.opographi cal map is est.ablished and I(ohonen-t.ype t.raining becomes a feasi ble means of speeding up further adaptation in a changing environment. Dimensionality reduction with spatial filters
firs\' we define locaL exlended objects. LeI us assume thaI the extemal world is a metrical Lopological space equipped wilh a measure and is embedded in a bounded region of Euclidean space. Let us then consider a mapping from Lhe subsels of Lhe bounded region of the Euclidean space into a finite dimensional vector space. This mapping could, for example, be defined by choosing two vectors of the subseLs randomly. Another Lype of mapping may, for example, spalially digitize Lhe exLemal world, and form a digital image. Hereinafter we shall,]se t.his mapping and cal l t.he element.s of t h e digit.i7-ed image a" pixels. A vector of t.he vector space wi l l be considered a local extended objed if Ii) t. here exi sts a conneded open set of t.he met.rical topologica l space t.hat. after mapping i s identical wit h t.he said vec.t or, Iii) i f t h e meaS1l1'e of t.hat. open set is not. 7-e1'O, and (iii) if t.he open set.'s convex h1111 t.aken in t.he vector space is in the topological space. Let us further assume that our inputs are locaL extended objects and our task is to provide the approximate position of the corresponding real object with no reference to its form. In order to be more concrete, let us take the example of a three dimensional object mapped onto two two-dimensional retinas, i.e. to a many dimensional vector space. The vector, that describes the digitized image on the retinas is the extended object. The task is 10 deteIllline the posiLion of lhe originaL real object, with no reference Lo ils form and with no a priori knowledge of the dimensionalily, nor even of the topology of lhe exlernal world. This problem wili noL be considered here, however lhese lools are general enough to solve it. In lhe following we restrict our invesligalions to the case of a single retina. Vie may say, for example, Lhat an object is 'in lhe middle' or 'in the upper lefL corner' or that it is 'down and right'. This task may be considered as a dimensionality reduction problem since if the image is given in the form of n X n pixels, having continuous grey scale intensities, then one maps an n X n input matrix having elements on the [0, 1] real interval into the world of m expressions that denote the possible different positions. Let us assume that the spatial filters that correspond to our position expressions already exist and let us list the expressions and organize the filters in a way, that the it' filter corresponds to the ith expression. For example, in the case of a two-dimensional image the expression 'middle' would correspond to a spatial filter that transforms the image by causing no change in pixel intensities around the middle of the image but the farther the pixels are from the center the more the filter decreases the pixel intensities. As a demonstration Fig. 1 shows spatial filters of a nine-expression set. These are Gaussian
.J
filters but other filters could serve just a,s well. The filters are digitized by replacing the center value of every pixel by the closest digitized grey scale value. LeL GCi) denote the i'h digitized GauSbian spatial filler, and leL S denote an inpuL (image) vecLor. Let q}�, ,spq(l S p, q S n ) denote the values in pixels (p, q) of the digitized ith Gauss filLer and the input vector, respeclively. Now, the searched position estimation may be given in the following fashion: first, leL the input veclor paSb all of the digitized GauSb filLers. Let us denote the ouLput of thei'h iilter by Gi - examples wi l l be given later - and denote the mapping by d: G' T'l -
d(Gli) S) , I
I
(1 "
- 1 2� -
•••
I
, In)
(1)
The mapping Ii is to be engineered in SIlch a way that it can be considered as a 'distance' function R"xn R+ U {O} providing distance-like quantities between the pattern inputting the network and the Caussian filters. YVith such a d function one might choose the smallest Gi value. If that has index j, then we say: the position of the object is the j'" expression. There are various possibilities for function d; here we list three of them: --+
•
Conventional fillering is defined by muiLiplying the input values by the filler values and then integrating over the input space. In order to fulfil our requirements for the 'distance' function d, let us define it in the following fashion: n
d, (X, Y)
=
1-
(2)
(lin') L XijYij ,;,j=1
from here onwards it is assumed that 0 S "'ii, Yij S 1. This 'distance' function has the form 1 - (1/n')X . Y where X· Y is the inner product or spherical distance. Since this 'distance' definition is normalized, we might define an 'input-to-filler similarity measure' , or measure of similarity, S in short, as S = 1 - d, where d is the 'distance'. The smaller the distance , the larger the similarity between input and filter vectors. -
•
One might try to use the usual Euclidean distance in
d,(X, Y) •
Another form t h at
is
not
a
Rnxn,
n
=
(1/n2) L (Xij - Yi:il' i,j=1
that is
(3)
metri c b1lt may be 1lsed here is
rl3 ( X, Y)
=
n (1In2) L .fij erij - Yij)'
i,j=l
(1 )
Forming spatial filters by competitive learning
Special competitive networks may form equiprobabilistic digitization of the input space according to the density distribution of input vectors (Desieno 1988). Networks of pure compeLiLivity are known to solve the problem of equiprobabilistic digitization for uniform distributions (Kohonen 1984).
Let
'"S
define a competitive neural network of m neurons. The external world provides
inpub 10 inpuL nodes. Every input node is connected Lo every neuron o[ lhe neLwork (see Pig.2). Every neuron sLores a vector o[ Lhe inpuL space. The sLored vedor o[ Lhe i'h neuron is denoLed by Wi. Training modifies Lhe sLored vectors. The procedure o[ modification is ali follows: •
An input is presented to the network in accordance with the density distribution. Input nodes forward inputs to the neurons.
•
Neurons process their inputs and develop activities in accordance with the equation
Di where
x
mcaSllfC •
=
d(x, Wi)
=
1
d
is
a
is the forwarded inpnt. vect.or, of simil arity.
-
(.J)
S;
'distance' fnndion and Si
1S
t.he
CompeLiLion slaIls. The winner of Lhe compelilion is Lhe neuron whose sLored vedor is the 'closest' 10 lhe inpul vector, i.e. lhe neuron having lhe smallesL
'distance' (or largest similarity). •
The sLored veclor o[ lhe wlIliling neuron updaLe rule:
i is
lhen modified wilh Lhe help of Lhe
(Ei)
where n is lhe so called learning rale; 0 < n S 1. Here we apply a conslanl learning raLe during lhe whole lraining. It was found Lhal the Lime dependenL learning rale did noL improve lraining resuILs. lhe advanLage thaI it keeps adapLivily.
Time independenL learning raLe has -
In the numerical simulations, we presented two-dimensional objects of a two-dimensional space to the network. T he input space and one of the input vectors that was presented arc illustrated in Fig.:3. Input vectors were derived by computing the overlap of the local, extended, randomly positioned objects and the pixels of digitization. Two different objects were used in these runs, an X shaped object (shown in Fig.3) and an a like object (not shown). In the first set of runs a single object was presented to the network at random positions. In other runs two or three objects were presented simultaneously to the network at random positions. The training procedure resulted in spatial filters for 'distance' functions d" d" and do defined in the previous section. We tried single objects for 'distance' functions d, and do. For spherical distance d, up to three objects were presented simultaneously. First, we tried the Euclidean distance function d,. Judging from our experience the network was able to learn only if the stored vectors were set close to zero prior to t.raining. The noise resistance of t.he network eqnipped with t.he Enclidean di st.ance was rat.her smal l . The hellfistic reasoning for t.his tinding is gi ven in the !\ppendix. The term
L (wi,??
(k,I)EI
of Eq. (12) in the Appendix - the indices correspond to the digitization of the two dimensional space and I denotes the set of zero elements of input vector x of a given extended object - is responsible for the poor performance of the Euclidean distance
function. This term ma,nifests itself in large distance valiles for inputs with a, fair amount of !lOise. In l his way one sin gle neuron thal has small wi�) componenls can always win the compeLition. Training - as the anal ysis in the Appendix shows - tends to lead lo this alLracLor. The simplest way of eliminating lhat lerm is to mo dify the Euclidean distance func tion Lo 1/(1 + (I - (I - O)2t)W(0))
(17)
is fulfilled, then it wins again. Inequality (17) shows that the first neuron's chance of x we have winning keeps growing. Taking the limit of t ---;
I I I I'? >1/(1 + 10(0))
(18)
and
this expression is independent of
o.
This
gives
rise to an upper limit of
W(O) .
Above
that limit, i.e. [or sIllall objects, the Euclidean distance [uncLion cannol solve Lhe problem or, al least, one may say thal the probability o[ having only one winning neuron is larger than zero: according lo our experience it is close to 1. Having more lhan one neuron, however, does not Illean Lhat more lhan one spatial filter will be formed. The question is how to [orm separale fillers. As iL is shown in the paper Lhe ' spherical disLance' is an appropriate solulion o[ lhis problem.
l-.. eurCll vUlllPUl,ClblUll U t 'Yj pp.
"±"±L-"±00,
L ';J';J"±
iL
References
Carpenter, G.A., and Grossberg, S. 1 987. A massively parallel architedme for a self organizing neural pattern recognition machine. Camp. Vision, Graphics and Image Pmc. 37 0/1- 115.
Desieno, D. 1988. Adding conscience to competitive leMning. Neural Netwo'rb, IEEE Press, New York, pp. 117-124.
Proc.
Int.
Conf.
Fiildi;ik, P. 1 990. Forming sparse representation hy local anti-Hehhian learning. Cybem. 64 165-170.
on
Riol.
self organizing feature map s with problem dependent cell structure. A rtificial Neur'al Networks, T. Kohonen, M. M iiki sara, O. Simula and .T. Kangas, eds., Elsevier Science Publishers B . V . , Amsterdam Vol . 1 pp. 40:3-408.
Fritzke, B. 1991. Let it grow
Gross b erg, S. 1976. Adapt i ve pattern classification and universal recoding. 1 &; 11. Cybem. 23 121-134. i(ohonen , '1'. 1 98 1 .
8e11- O"ganizat;on and A 8sodative Alf m orll.
13iol.
Springer-Verlag, Herlin
lVIartind7" '1'., and Schll lten , K . 1 991 . ;\ "nellral-ga,s" ndwork karn s topologies. Art i ficial NC1J.m.i Nctwo1'A,s, '1'. Kohonen , IVI . lVI ii kisara, O. Sinl1lia and .J . Kangas, cds., �:Isevicr Science Pll blishers H . V . , i\ m stcrdmn Vol. 1 PI'. 397 ·1 02. S7,epesv;i,ri , Cs. 1 992. Competitive nemal networks with l ateral connections: collective learning and topology representation . TDK Thes;8 in Hllngarian . ,
l-.. eurCll vUlllPUl,ClblUll U t 'Yj pp.
"±"±L-"±00,
L ';J';J"±
i0
Figure captions •
Figure 1 .
Digitized spatial Gaussian jilters
Gau",ian filler" of a llVo dimenbional box correbponding 10 expre""ion" : left ' , 'upper middle' 'upper right', eLc. The figure
Willi
drawn
by
upper
generaling pixel"
randomly wilh probabililie" lhal corre"poml Lo Lhe gray "cale value". •
F igure 2.
Archileclul'e of lhe arli{icial nt uml nelwork
ANK has a seL of inpul node". Inpul" connecLed 10 Lhe nelwork are denoLed i = 1, 2, . . . , n. Every inpul node is connected 10 every neuron of Lhe neLwork. Every neuron stores a vector of the input space: neuron j stores (w;j), w�:i) , . . . w�) ) . Neurons develop another set of connecti on s, the (qkl) topology connecti ons , an internal representation of the topology of the external world.
The by
•
"' i ,
Figure
:3 .
l '�pical input
The hox on the ldt hand side shows an ohjed at. a given posit.ion t.hat. was t.rans mit.ted t.o t.he inp11t. nodes. The middle hox shows the o11t.pnts of t.he inp11t. nodes developed . Tnpnt nodes developed adivities according t.o their overlaps with t.he inp11t.ted ohj ed. The npper and the lower hoxes on t.he right han d si de show t.he ontpnt.s ni' ) of t.he ne11rons and the st.ored vedor of t.he winning ne11ron , respec tively. •
F igure 1. Training 7'esults on self-organized filter for'mation dur'ing training The numbers show the t r aining steps. Filters arc formed during the fir st 5000 steps. At later steps the configuration undergoes minor modifications in accordance with the random object generation, but stays stable. The figure was drawn by generating pixels randomly with probabilities that correspond to the gray scale values.
•
Figure 6.
Learnt one and two dimensional topologies
Connection thicknesses show the strengths of topology connections qk/. In the left hand side figure objects were genera.ted everywhere in the two dimensional box. No line means approximately zero strength connections. In the right hand side figure objects were generated along three horizontal strips in the two dimensional box with Mbitrary ordina.tes. No line means zero strength connections. •
Figure 7.
;110notonicity of topological connection strengths
Strength of topology connection qkl a.s a function of overlap of filters. The overlap of the k'h and II" filters is defined as Li i a{�) w1? .
•
Figure 8.
.
Adaptivity of ANN's
After a sudden change in the external world or, here, the average object size. networks try to adapt. Adaptation means a change of filter size and creation or death of filters. The graph on the left hand side shows the evolution of filter sizes for the competitive network. The graph on the right hand side shows the evolution of filLer "i�e" for a compeliLi ve nelwork lhaL developed Lopology connections and Kohonen lype neighbor lraining. The graph, depict poinl" from 100,000 leal'lling sleps prior Lo and 100,000 leal'lling "lep" aILer lhe sudden ch an ge. SLep number
zero is the time of the sudden ch ange The region of the first 20,000 steps after the .
change is enlarged and enclosed wilh dashed lines. Size is defined as •
Figure 5.
Li,i(U{�) )2
Training n:sults of the three-object case
Receptive filters are formed in the case of training with by showing three randomly positioned objects simultaneously. The noisy background is the result of the pres ence of more lhan one object al lhe same Lime. The noise increases Lhe avarage filLer size resul t ing in Lhe slrung decrease of lhe recepli ve field of one neuron in Lhe lhree objecL case.
I' purR.
'-�OInpIiUluon O( .)) jJp. 441 q..YI. 1 'J,N
1 .00
0.50
0.00
Figure 1: Digili:,;ed �paLial Cam�iall iilleb
r" eural '-'OlnpU�a�lOn til ,}) pp,
'H1-'E)Il, 1 "''''4
10
fL--
Ym
Figure 2: Architecture of the a,rtificial nellfal network
r" eural '-'OnlpU�a�lOn til ,}) pp,
'H1-'F>Il, 1 "''''4
11
. 1 1 1 1 1 1 1 1 1
*
-
:J U I I I
Figure :3: A captured screen
l� eural vOlnpm;al;lOll D ( J) pp. LFH-qO(5, l CJCJq
0.80
0.40
0.50
I
0.25
!lJ '"
1 (5
I
!lJ '"
�
�
0.00
0.00 Learning Step: 0
Learning Step: 2000
Learning Step: 5000
Learning Step: 1 0000
� :: I �
�
��� ,-----c--,
0.00 Learning Step: 20000
Figure 4: Training results on self-organized filter formation during training
l� eural vOlnpm;al;lOll D ( J) pp. LFH-qO(5, l CJCJq
0.45
Learning Step: 20000
D Learning Step: 20000
Figure .5 : Training results of the two- and three-object cases
r" eural '-'OlnpU�a�lOn til ,}) pp,
'H1-'E)Il, 1 "''''4
G S E) G S E) G S E)
Figure 6: Learnt one and two dimensional topologies
r" eural '-'OlnpU�a�lOn til ,}) pp,
0.45 0.4 0.35 0.3 0.25 0.2 0.15 0. 1 0.05 o
o
'H1-'E)Il, 1 "''''4
0.05 0. 1 0. 1 5 0.2 0.25 Overlap
Figllfe 7: Monotonity of topological connection strengths
l'l eUral vOIUpU"La"LlOn D( ')j pp.
l :J �n
LL
Size 2.50
Size 2.50 2.00
'l 'l l -'W I'i ,
2.00
(4) ���
1 .50
1 .50
1 .00
1 .00
0.50
0.50
(4)
(1)
(7)
(4)
(1) - "--. � ....
j ��(5 ) 0.00 + ������������ - 1 00
-50
a
Learning Steps (in thousands)
100
0.00
- 1 00
-50
a
10
,,�_
� __
2 0 50
Learning Steps (in thousands)
Figure 8 : Adaptivity of the network
(1)
100
