Information processing in synchronous neural networks - CiteSeerX

11 downloads 0 Views 995KB Size Report
façon à accommoder des cycles de longueur deux dans le cadre de la mécanique statistique. Utilisant la méthode des répliques, nous déterminons le diagramme de phase incluant un ... Jij these patterns are fixed points of the dynamics at.
J.

Phys.

France 49

(1988)

Classification Physics Abstracts 87.30G 64.60C -

-

Information

13-23

75.10H

-

JANVIER

1988,

13

89.70

processing

in

synchronous

neural networks

J. F. Fontanari and R. Köberle Instituto de Fisica SP, Brasil

(Requ

le 16

e

Química de São Carlos, Universidade de São Paulo, Caixa Postal 369, 13560 São Carlos,

juin 1987, accepté le

16

septembre 1987)

Résumé. Nous obtenons le diagramme de phase du modèle de Little quand le nombre p d’échantillons mémorisés croit comme 03C1 03B1N, où N est le nombre de neurones. Nous dédoublons l’espace de phase de façon à accommoder des cycles de longueur deux dans le cadre de la mécanique statistique. Utilisant la méthode des répliques, nous déterminons le diagramme de phase incluant un paramètre J0 pour contrôler l’apparition des cycles. La transition de phase entre les phases para- et ferromagnétiques passe du second ordre au premier ordre au point tricritique. La région de recouvrement de l’information est un peu plus grande que dans le modèle de Hopfield. Nous trouvons également une phase paramagnétique à basse température qui a des propriétés physiquement inacceptables. 2014

=

The phase diagram of Little’s model is determined when the number of stored patterns p grows as 03B1N, where N is the number of neurons. We duplicate phase space in order to accomodate cycles of length within the framework of equilibrium statistical mechanics. Using the replica symmetry scheme we

Abstract. 03C1

=

two

2014

determine the phase diagram including a parameter J0 able to control the occurrence of cycles. The second order transition between the paramagnetic and ferromagnetic phase becomes first order at a tricritical point. The retrieval region is some what larger than in Hopfield’s model. We also find a low temperature paramagnetic phase with unphysical properties.

Recently methods developed in the study of equilibrium statistical mechanics of spin glasses [1, 2, 3, 4], have been applied to investigate information processing and retrieval in neural networks. Although there is a long way to go, if reasonably realistic biological systems are to be described, the models we are able to control, do exhibit a number of interesting features, which makes their study a worthwhile endeavour. Properties such as fault tolerance to errors, information storage and retrieval due to implementation of auto-associative memories, etc. have been shown to arise as a consequence of the existence of an infinite number of ground states in a spin glass interacting via long ranged forces. study Little’s model [5 based on synchronous update in the limit, when an infinite In this paper

symmetric couplings has limit cycles oft length points. The question arises then as to how can equilibrium statistical mechanics describe a system with cycles [2]. As we will see, since in our case the cycle’s length is two, we have to duplicate the phase space [7]. The concomitant proliferation of order parameters allows the adequate description of this more complicated behaviour.

with

1. Introduction.

we

number of patterns is to be stored

In contrast to assynchronous dynamics (such as Monte Carlo update in Hopfield’s model), when all states change simultaneously in parallel, the system may exhibit cycles. It is indeed easy to show that Little’s model

[6].

two in addition to fixed

In section 2 we calculate the free energy and derive the equations for the order parameters, which are discussed and solved for special cases in section 3. A discussion is presented in section 4. 2. Free energy and order parameters.

Little’s model consists of a network of N neurons N. The transidescribed by spins S, = ± 1, i 1, tion probability W (J/I ) from state I {5,} at time t to state J at time t + 1 is given by

Article published online by EDP Sciences and available at http://dx.doi.org/10.1051/jphys:0198800490101300

=

...,

=

14

where

For

symmetric couplings Jij Jji, this leads stationary distribution of states given by e- H, =

to a

with

Here f3 is a measure of the noise level in the system and the couplings are

where aj = ± 1, j N are a set of duplicate 1, variables. Ising Following Amit et al. [3] we now compute the quenched free energy under the assumption that a =

finite number s of the

...,

overlaps n

"

remains finite

quenched

free energy per

spin

is

as

N --+

oo.

The

given by the replica

method :

The p patterns (prototypes) {ç r ; i = 1, N} =1, ..., p are quenched, independent random variables taking the values ± 1. Due to the coupling Jij these patterns are fixed points of the dynamics at T=0, poo. ...,

J.L

In order to evaluate the trace Tr e- OR W

we use

where

the

Introducing

identity

..., n we

the

replicated

write (Z")

variables

sf, (T f,

p

=

1,

as

where

with Z° Let

If

=

us

we

Z v for cp * s and Z° Z w for cp > s. first evaluate Z’. Performing the average =

expand

Inserting

this

the

log cosh, keeping

expression

over

terms of order

into Z’ and

performing

the

tiN

the

g¡,

we

integral

we

obtain the factor

get

over

m:, n: and f p 0

we

obtain

finally

15

where

P and R

are

symmetric n

where p pu and r pa

are

defined

Q,

equation (2.12). Now we proceed to thermodynamic limit as

x n

only

matrices

given by

for p =1= (T and

evaluate Z v3. First

we

--AL-

have omitted trivial

multiplicative

rescale m, n and f in order to obtain

a

constants in

well defined

Thus

Now

to

use

self-averaging

in the

identity

replace Ni 1 E byand

(2.12)

for Z’

we

obtain

use

the result in

equation (2.15).

With this

expression

for Z v and

equation

finally

where

with the notation

m.

In the limit N - oo the integrand is dominated by its saddle point, furnishing the following free energy per spin

with

In this paper we mainly discuss the replica symmetric theory, in which we use the following parametrization

16

With this ansatz the n - 0 limit of the terms in equation (2.19) may be explicitly performed. We get

[8] Now we linearize the quadratic terms in the spin variables by Gaussian transformations, to obtain after insertion into equation (2.23)

where where

Finally

Using

we

have to compute

the ansatz

(2.21),

we

rearrange

HI;

into

stands for the average over the 6,’s and three Gaussian variables Zi, i 1, 2, 3 with mean zero and unit variance. Putting everything together we get for the free energy the final expression

and over

=

Minimizing f with respect to the parameters f, m, n, p, p, r, r, ql, ql, qo, 40 yields their interpretation in original variables Si, (T and );’ together with the order parameter equations. We thus obtain within the replica symmetric theory : i) the macroscopic overlaps between equilibrium states and prototypes : terms of the

ii) the Edwards-Anderson

iii)

the total

mean

order parameters

square random

overlaps

with p-s patterns

17

where > The 8

s.

+

3s

coupled equations

for the order parameters

where

are :

Since qo =1

dynamics,

as

are at a fixed point of the opposed to a limit cycle. we

In order to evaluate the following identities

3. Solution of order parameter

P -+

oo

limit

we use

the

equations and phases.

In general it is impossible to solve explicitly the coupled equations for the order parameters, but in special limiting cases this can be done. In the following we will do this in order to obtain relevant sections of the phase diagram and discuss the nature of possible solutions.

T = 0. The sign of Q plays an important role, if we want to take the limit J3 --+ oo. Accordingly we distinguish three cases.

3.1 PHASE

for

AT

-

It is

now

straightforward

to

get

equations (2.28) :

These are the same equations as obtained by Amit et al. [3], for the T = 0 limit of Hopfield’s model. This is to be expected, since at a fixed point equation (2.2) may be rewritten at T 0 as =

where

and

The free energy at T

=

0 becomes

18

Which differs from Hopfield’s model only Jo and Amit et al.’s analysis for this case taken over with a slight change in notation. term

Q

3.1.2

0.

The

-

equations

become

by

the be

p

=

0, implying p

=

0,

c --+ oo

with energy

can

now :

which is always less than the spin-glass energy (3.9). Whether fixed point or cycles are realized depends now on the value of Jo, selecting the lower of two

energies equation (3.5)

versus

(3.10).

case Q = 0 gives identical results to The resulting phase diagram is shown in 1, where all transitions are first order.

3.1.3 The

Q > 0.

figure

-

where

and

Since qo = - 1, we are at a complete cycle, all spins changing sign at each update. Evaluating the limits in equations (3.6) we get

The

only

solution of

equation (3.7)

for m ° is

m’ = 0, yielding

Phase diagram in the a - Jo plane. abounds the region where retrieval states appear, which become stable inside the F region. a g separates the p and SG phases. At Jo 0, we get a, - 0.138 and a F == 0.051 as in Hopfield’s model.

Fig.

with the energy

being given by

Since c is positive semidefinite, this spin glass exists solution only for a > 2/ 1T and 1T the last 7() (a /2 )1/1, inequality resulting from

Q0. If c

we

oo, we

relax the implicit assumption (3.8) of also encounter a paramagnetic phase with

with the order parameter

from which

we

easily

equations

obtain

1.

-

=

3.2 PHASES

FOR a

equation (2.26) yields

=

0.

-

The a --+ 0 limit of

19

Since the lefthand side is > 0, whereas the right hand side is 0 we see that the only solution is

In the following we will the Mattis type only :

n now

study retrieval solutions of

satisfies

Expanding this equation in powers of n, for the n # 0 solution

we

obtain

vv

Fig. 2. - a = 0 plot of phase diagram. TCP indicates the tricritical end-point at T 2/3, Jo = - 1/3 log 2, marking change from a first order (continuous line) to a second order transition.

where

=

n2 vanishes by

as

long

at the critical

temperature Tc defined 3.3 SYMMETRIC AND ANTISYMMETRIC SOLUTIONS. In the cases studied up to now we only found the following types of solutions :

i) symmetric

as

This 2nd order transition between a ferromagnetic and a paramagnetic phase changes to first order, when condition (3.19) ceases to be verified. This happens at the tricritical point

where a new solution of free energy appears.

equation (3.16)

with lower

solutions :

ii) antisymmetric

solutions :

We believe these to be the only ones, because solutions with n v =1= e v or p =1= r would correspond to breaking spontaneously the symmetry S - a and since we have failed to encounter this at T 0, we conclude that it should not occurr at all. =

The first order phase boundary between the P and F phases may be obtained numerically by equating the free energies of the two phases. The result is shown in figure 2, where the point T 0, Jo 0.5 at which the first order line touches the T = 0 axis can easily be obtained analytically. We also show the curve labeled T, limiting metastable F solutions. =

In order to obtain information about compute the parameter qo

=

-

cycles

As for the antisymmetric solutions, they only exist 0. This can easily be seen using the relation = r ql/2 in the expression for E 1,

at T

p

=

=

-

equations (2.28)

we

Which would yield complex order parameters. This doesn’t happen at T 0, because the offending terms vanish in this limit. Thus for T> 0 only symmetric solutions exist and the order parameter equations for this case follow from equations (2.28). Restricting ourselves to retrieval states they are =

In the P

phase

yielding cycles

this

for

gives

Jo

0.

20

b ) Transition spin-glass

to

paramagnetic.

0, all other parameters are finite. equation for the 2nd order transition surface Tg (a, Jo) between P and SG phase is obtained expanding equations (3.25b), (3.25c) in powers of

Only

n

=

The

qu

which go the

yields together with the equations for qo following behaviour for Tg ( ex , J 0) for a and Jo = 0:

Since

and 1

know that at T 0 this transition is first that there should exist a tricritical conclude order, line (TCL) on the surface (3.31) separating these two different critical behaviours. The TCL may be obtained numerically imposing the existence of two different solutions of equations (3.25a), (3.25c), 0. which coincide at the TCL, both having ql 3 of shows the the TCL onto the Figure projections planes T 0 and Jo 0. In agreement with section 3B, the line reaches the a 0 plane at the point T 2/3 and Jo = - 1/3 log 2. We furthermore observe that it exists only in the half-space Jo 0. The first-order transition SG-P is obtained equating the free energies of these two phases. we

=

we

where

=

=

=

=

=

and

From these equations and transitions :

we

obtain the

following

phases

a) Paramagnetic phase. Here all order parameters which satisfy

vanish, except qo and

ilo,

For

high temperatures

this

gives

Whereas for low temperatures equation (3.27a ) that qf -+ 1 as emay write

we

/ T,

from that we

see so

Projections of the tricritical line (TCL) onto the 0 (continuous line). T 0 (broken line) and Jo The broken line exists only in the region Jo 0. Fig. 3. planes

c) implying

=

=

Retrieval

phase.

we obtain the surface T R ( a , 10) below which retrieval states become metastable. The results is shown in figure 4. For Jo 0 we obtain a phase diagram very similar the one of Hopfield’s model, except that for a > 3.74 we find a paramagnetic phase with qo 0,

Solving numerically equations (3.25)

this last result agreeing with our discussion about the 0 in the symmetric occurrence of cycles at T =

phase.

-

=

21

therefore associate with the existence of in model. In the traditional highLittle’s cycles P temperature phase we have qo > 0.

which

we

Proceeding

as

in reference

[3],

we

obtain

The behaviour of TF (a, 0) near T = 1 is obtained equating the free energies of the F and SG phases,

yielding

This

together

with

equation (3.34) gives

where we have included the results for Hopfield’s model in parentheses for comparison. Thus the F and retrieval regions are slightly larger in Little’s model.

e)

The limit

In this limit

4.

Fig.

-

Jo = 0 plot of phase diagram. Tf is the

transition

temperature between F and SG phases. The continuous lines are first order transitions, while broken ones are second order. The inset shows the appearence of the P at T 0 and large values of a.

phase

Jo we

00.

obtain from

equation (3.25)

with

=

The equations reduce to the corresponding ones of Hopfield’s model provided we rescale T Little 2 7"Hopfic)d’ This result is reasonable considering that large values of 10 suppress cycles and tend to align 0’i with Si. We also see that the synchronous dynamics is much more stable to noise than the asynchronous one for the same retrieval capacity, provided that neurons have a sufficiently large selfinteraction 1(). =

d)

The

phase-diagram around

T

=

1

for Jo

=

0.

The aim of this section is to obtain the behaviour of the curves TF ( a ), limiting the stable and metastable retrieval regions, in the vicinity of the point T 1 and a 0, where we have discontinuous first order transitions but with small order parameters, so that the equation (3.25) may be expanded in powers of n and t -1 - T. We obtain =

=

4.

Replica symmetry breaking.

It is well known, that replica symmetry breaking (RSB) is necessary to stabilize the spin-glass phase at low temperatures. The effects of RSB in the F and SG phase are similar to the ones in Hopfield’s model and thus small. For example, the entropy at T = 0, J" = 0 in the FM phase is S = -1.4 x 10- , at a = a, 0.138 and vanishing exponentially =

.

From these

with

we

get

In the SG

phase

at

The analog of the Almeida-Thouless line can be computed as in reference [3]. The sign of the replicon mode changes when

22

where

how be discarded. This point is being reserved for a future study. Although this P phase is unimportant for retrieval purposes, for which the replica scheme seems to provide results in agreement with simulations, we do not see how these diseases may be cured within the replica scheme. We have studied this phase by simulations at T 0. In figure 5 we show the result of measuring the fraction q n/N, where n is the number of spins belonging to cycles, against N. We interpret these results saying that qo (= 1 - 2 q ) never reaches the value qo = -1, as predicted by equation (3.6), but rather that qn - + 1 with only a finite number of spins oscillating. This would exclude =

=

where

we

tested for instabilities

only

in directions

q a¡3’ ij a¡3 with a =1= f3 . The entropy of the low temperature P phase is also negative, going to - oo as T -+ 0 exactly as in the Sherrington-Kirkpatrick (SK) model [9]. Since RSB has nothing to say about this unphysical situation, we leave this matter for a discussion in the next section.

T

=

0, Jo ± 0 solutions with

Q

thus

stigmatize this P phase as replica symmetric calculation.

0 of section 3 and an

artefact of the

5. Discussion.

In this paper we have analised the phase diagram and storage properties of a synchronous model for associative memory. Our results are qualitatively similar to the ones of model, albeit with the following differences :

Hopfield’s

a)

there is

a new

parameter Jo, which may be used

to control the occurrence of

cycles b) a tricritical line appears at the surface separating the P and F phases c) for or 0 we find a P phase even at T 0, which is not connected to the usual high temperature P phase. This phase has negative entropy, whose =

value goes to -

oo

at T

=

0

as

in the SK model.

RSB is expected to cure the unphysical aspects of the SG and F phases, but cannot exorcise problems in the low temperature P phase, since only qo and go are nonzero here. Breaking the symmetry along the diagonal in q a¡3 and ij a¡3 is apparently useless, since we found this phase to be stable in the directions qa a and 4,,aLet us note that this P phase is our only option in the region Jo -1, where a solution with qo 0 is to be expected. The SG phase with qo 0 doesn’t

exist for

a

2 or

as can

be

seen

from

equation (3.8)

and comments there after. RSB may extend the size of the SG solution, so that the P phase may some

Fig. 5. Histograms of the fraction q of spins belonging to cycles for N 100, 200 and 300. The parameters « 0.5 and Jo 0 are kept fixed and we average over -

=

=

=

initial states and realizations of the

( );’) .

Acknowledgments. The research of R.K. is partially supported by CNPq and J.F.F. holds a FAPESP fellowship. Simulations were carried out on a VAX 780/11 computer, partially maintained by funds from CNPq.

23

References J. J., Proc. Natl. Acad. Sci. USA 79 (1982) 2554 ; ibid 81 (1984) 3088. [2] PERETTO, P., Biol. Cybern. 50 (1984) 51. [3] AMIT, D. J., GUTFREUND, H. and SOMPOLINSKY, H., Ann. Phvs. 173 (1987).

[1] HOPFIELD,

[4] [5] [6]

Parallel Models of Associative Memory, Eds G. E. Hinton and J. A. Anderson (Lawrence Erebaum Ass.) 1984. LITTLE, W. A., Math. Biosci. 19 (1974) 101. FONTANARI, J. F. and KÖBERLE, R., Phys. Rev. A 36 (1987) 2475.

[7]

This

which arises very naturally, when solves this model, has also been used by the authors of reference [3] and J. L. van Hemmen, Phys. Rev. A 34 (1986) 3435. If the cycle length is l, we have to introduce l copies of phase

duplication, one

space.

[8]

See

appendix

A of reference

[3]

for

a

similar cal-

culation.

[9] KIRKPATRICK, S. B 17 (1978)

and SHERRINGTON, D., 4384.

Phys.

Rev.