façon à accommoder des cycles de longueur deux dans le cadre de la mécanique statistique. Utilisant la méthode des répliques, nous déterminons le diagramme de phase incluant un ... Jij these patterns are fixed points of the dynamics at.
J.
Phys.
France 49
(1988)
Classification Physics Abstracts 87.30G 64.60C -
-
Information
13-23
75.10H
-
JANVIER
1988,
13
89.70
processing
in
synchronous
neural networks
J. F. Fontanari and R. Köberle Instituto de Fisica SP, Brasil
(Requ
le 16
e
Química de São Carlos, Universidade de São Paulo, Caixa Postal 369, 13560 São Carlos,
juin 1987, accepté le
16
septembre 1987)
Résumé. Nous obtenons le diagramme de phase du modèle de Little quand le nombre p d’échantillons mémorisés croit comme 03C1 03B1N, où N est le nombre de neurones. Nous dédoublons l’espace de phase de façon à accommoder des cycles de longueur deux dans le cadre de la mécanique statistique. Utilisant la méthode des répliques, nous déterminons le diagramme de phase incluant un paramètre J0 pour contrôler l’apparition des cycles. La transition de phase entre les phases para- et ferromagnétiques passe du second ordre au premier ordre au point tricritique. La région de recouvrement de l’information est un peu plus grande que dans le modèle de Hopfield. Nous trouvons également une phase paramagnétique à basse température qui a des propriétés physiquement inacceptables. 2014
=
The phase diagram of Little’s model is determined when the number of stored patterns p grows as 03B1N, where N is the number of neurons. We duplicate phase space in order to accomodate cycles of length within the framework of equilibrium statistical mechanics. Using the replica symmetry scheme we
Abstract. 03C1
=
two
2014
determine the phase diagram including a parameter J0 able to control the occurrence of cycles. The second order transition between the paramagnetic and ferromagnetic phase becomes first order at a tricritical point. The retrieval region is some what larger than in Hopfield’s model. We also find a low temperature paramagnetic phase with unphysical properties.
Recently methods developed in the study of equilibrium statistical mechanics of spin glasses [1, 2, 3, 4], have been applied to investigate information processing and retrieval in neural networks. Although there is a long way to go, if reasonably realistic biological systems are to be described, the models we are able to control, do exhibit a number of interesting features, which makes their study a worthwhile endeavour. Properties such as fault tolerance to errors, information storage and retrieval due to implementation of auto-associative memories, etc. have been shown to arise as a consequence of the existence of an infinite number of ground states in a spin glass interacting via long ranged forces. study Little’s model [5 based on synchronous update in the limit, when an infinite In this paper
symmetric couplings has limit cycles oft length points. The question arises then as to how can equilibrium statistical mechanics describe a system with cycles [2]. As we will see, since in our case the cycle’s length is two, we have to duplicate the phase space [7]. The concomitant proliferation of order parameters allows the adequate description of this more complicated behaviour.
with
1. Introduction.
we
number of patterns is to be stored
In contrast to assynchronous dynamics (such as Monte Carlo update in Hopfield’s model), when all states change simultaneously in parallel, the system may exhibit cycles. It is indeed easy to show that Little’s model
[6].
two in addition to fixed
In section 2 we calculate the free energy and derive the equations for the order parameters, which are discussed and solved for special cases in section 3. A discussion is presented in section 4. 2. Free energy and order parameters.
Little’s model consists of a network of N neurons N. The transidescribed by spins S, = ± 1, i 1, tion probability W (J/I ) from state I {5,} at time t to state J at time t + 1 is given by
Article published online by EDP Sciences and available at http://dx.doi.org/10.1051/jphys:0198800490101300
=
...,
=
14
where
For
symmetric couplings Jij Jji, this leads stationary distribution of states given by e- H, =
to a
with
Here f3 is a measure of the noise level in the system and the couplings are
where aj = ± 1, j N are a set of duplicate 1, variables. Ising Following Amit et al. [3] we now compute the quenched free energy under the assumption that a =
finite number s of the
...,
overlaps n
"
remains finite
quenched
free energy per
spin
is
as
N --+
oo.
The
given by the replica
method :
The p patterns (prototypes) {ç r ; i = 1, N} =1, ..., p are quenched, independent random variables taking the values ± 1. Due to the coupling Jij these patterns are fixed points of the dynamics at T=0, poo. ...,
J.L
In order to evaluate the trace Tr e- OR W
we use
where
the
Introducing
identity
..., n we
the
replicated
write (Z")
variables
sf, (T f,
p
=
1,
as
where
with Z° Let
If
=
us
we
Z v for cp * s and Z° Z w for cp > s. first evaluate Z’. Performing the average =
expand
Inserting
this
the
log cosh, keeping
expression
over
terms of order
into Z’ and
performing
the
tiN
the
g¡,
we
integral
we
obtain the factor
get
over
m:, n: and f p 0
we
obtain
finally
15
where
P and R
are
symmetric n
where p pu and r pa
are
defined
Q,
equation (2.12). Now we proceed to thermodynamic limit as
x n
only
matrices
given by
for p =1= (T and
evaluate Z v3. First
we
--AL-
have omitted trivial
multiplicative
rescale m, n and f in order to obtain
a
constants in
well defined
Thus
Now
to
use
self-averaging
in the
identity
replace Ni 1 E byand
(2.12)
for Z’
we
obtain
use
the result in
equation (2.15).
With this
expression
for Z v and
equation
finally
where
with the notation
m.
In the limit N - oo the integrand is dominated by its saddle point, furnishing the following free energy per spin
with
In this paper we mainly discuss the replica symmetric theory, in which we use the following parametrization
16
With this ansatz the n - 0 limit of the terms in equation (2.19) may be explicitly performed. We get
[8] Now we linearize the quadratic terms in the spin variables by Gaussian transformations, to obtain after insertion into equation (2.23)
where where
Finally
Using
we
have to compute
the ansatz
(2.21),
we
rearrange
HI;
into
stands for the average over the 6,’s and three Gaussian variables Zi, i 1, 2, 3 with mean zero and unit variance. Putting everything together we get for the free energy the final expression
and over
=
Minimizing f with respect to the parameters f, m, n, p, p, r, r, ql, ql, qo, 40 yields their interpretation in original variables Si, (T and );’ together with the order parameter equations. We thus obtain within the replica symmetric theory : i) the macroscopic overlaps between equilibrium states and prototypes : terms of the
ii) the Edwards-Anderson
iii)
the total
mean
order parameters
square random
overlaps
with p-s patterns
17
where > The 8
s.
+
3s
coupled equations
for the order parameters
where
are :
Since qo =1
dynamics,
as
are at a fixed point of the opposed to a limit cycle. we
In order to evaluate the following identities
3. Solution of order parameter
P -+
oo
limit
we use
the
equations and phases.
In general it is impossible to solve explicitly the coupled equations for the order parameters, but in special limiting cases this can be done. In the following we will do this in order to obtain relevant sections of the phase diagram and discuss the nature of possible solutions.
T = 0. The sign of Q plays an important role, if we want to take the limit J3 --+ oo. Accordingly we distinguish three cases.
3.1 PHASE
for
AT
-
It is
now
straightforward
to
get
equations (2.28) :
These are the same equations as obtained by Amit et al. [3], for the T = 0 limit of Hopfield’s model. This is to be expected, since at a fixed point equation (2.2) may be rewritten at T 0 as =
where
and
The free energy at T
=
0 becomes
18
Which differs from Hopfield’s model only Jo and Amit et al.’s analysis for this case taken over with a slight change in notation. term
Q
3.1.2
0.
The
-
equations
become
by
the be
p
=
0, implying p
=
0,
c --+ oo
with energy
can
now :
which is always less than the spin-glass energy (3.9). Whether fixed point or cycles are realized depends now on the value of Jo, selecting the lower of two
energies equation (3.5)
versus
(3.10).
case Q = 0 gives identical results to The resulting phase diagram is shown in 1, where all transitions are first order.
3.1.3 The
Q > 0.
figure
-
where
and
Since qo = - 1, we are at a complete cycle, all spins changing sign at each update. Evaluating the limits in equations (3.6) we get
The
only
solution of
equation (3.7)
for m ° is
m’ = 0, yielding
Phase diagram in the a - Jo plane. abounds the region where retrieval states appear, which become stable inside the F region. a g separates the p and SG phases. At Jo 0, we get a, - 0.138 and a F == 0.051 as in Hopfield’s model.
Fig.
with the energy
being given by
Since c is positive semidefinite, this spin glass exists solution only for a > 2/ 1T and 1T the last 7() (a /2 )1/1, inequality resulting from
Q0. If c
we
oo, we
relax the implicit assumption (3.8) of also encounter a paramagnetic phase with
with the order parameter
from which
we
easily
equations
obtain
1.
-
=
3.2 PHASES
FOR a
equation (2.26) yields
=
0.
-
The a --+ 0 limit of
19
Since the lefthand side is > 0, whereas the right hand side is 0 we see that the only solution is
In the following we will the Mattis type only :
n now
study retrieval solutions of
satisfies
Expanding this equation in powers of n, for the n # 0 solution
we
obtain
vv
Fig. 2. - a = 0 plot of phase diagram. TCP indicates the tricritical end-point at T 2/3, Jo = - 1/3 log 2, marking change from a first order (continuous line) to a second order transition.
where
=
n2 vanishes by
as
long
at the critical
temperature Tc defined 3.3 SYMMETRIC AND ANTISYMMETRIC SOLUTIONS. In the cases studied up to now we only found the following types of solutions :
i) symmetric
as
This 2nd order transition between a ferromagnetic and a paramagnetic phase changes to first order, when condition (3.19) ceases to be verified. This happens at the tricritical point
where a new solution of free energy appears.
equation (3.16)
with lower
solutions :
ii) antisymmetric
solutions :
We believe these to be the only ones, because solutions with n v =1= e v or p =1= r would correspond to breaking spontaneously the symmetry S - a and since we have failed to encounter this at T 0, we conclude that it should not occurr at all. =
The first order phase boundary between the P and F phases may be obtained numerically by equating the free energies of the two phases. The result is shown in figure 2, where the point T 0, Jo 0.5 at which the first order line touches the T = 0 axis can easily be obtained analytically. We also show the curve labeled T, limiting metastable F solutions. =
In order to obtain information about compute the parameter qo
=
-
cycles
As for the antisymmetric solutions, they only exist 0. This can easily be seen using the relation = r ql/2 in the expression for E 1,
at T
p
=
=
-
equations (2.28)
we
Which would yield complex order parameters. This doesn’t happen at T 0, because the offending terms vanish in this limit. Thus for T> 0 only symmetric solutions exist and the order parameter equations for this case follow from equations (2.28). Restricting ourselves to retrieval states they are =
In the P
phase
yielding cycles
this
for
gives
Jo
0.
20
b ) Transition spin-glass
to
paramagnetic.
0, all other parameters are finite. equation for the 2nd order transition surface Tg (a, Jo) between P and SG phase is obtained expanding equations (3.25b), (3.25c) in powers of
Only
n
=
The
qu
which go the
yields together with the equations for qo following behaviour for Tg ( ex , J 0) for a and Jo = 0:
Since
and 1
know that at T 0 this transition is first that there should exist a tricritical conclude order, line (TCL) on the surface (3.31) separating these two different critical behaviours. The TCL may be obtained numerically imposing the existence of two different solutions of equations (3.25a), (3.25c), 0. which coincide at the TCL, both having ql 3 of shows the the TCL onto the Figure projections planes T 0 and Jo 0. In agreement with section 3B, the line reaches the a 0 plane at the point T 2/3 and Jo = - 1/3 log 2. We furthermore observe that it exists only in the half-space Jo 0. The first-order transition SG-P is obtained equating the free energies of these two phases. we
=
we
where
=
=
=
=
=
and
From these equations and transitions :
we
obtain the
following
phases
a) Paramagnetic phase. Here all order parameters which satisfy
vanish, except qo and
ilo,
For
high temperatures
this
gives
Whereas for low temperatures equation (3.27a ) that qf -+ 1 as emay write
we
/ T,
from that we
see so
Projections of the tricritical line (TCL) onto the 0 (continuous line). T 0 (broken line) and Jo The broken line exists only in the region Jo 0. Fig. 3. planes
c) implying
=
=
Retrieval
phase.
we obtain the surface T R ( a , 10) below which retrieval states become metastable. The results is shown in figure 4. For Jo 0 we obtain a phase diagram very similar the one of Hopfield’s model, except that for a > 3.74 we find a paramagnetic phase with qo 0,
Solving numerically equations (3.25)
this last result agreeing with our discussion about the 0 in the symmetric occurrence of cycles at T =
phase.
-
=
21
therefore associate with the existence of in model. In the traditional highLittle’s cycles P temperature phase we have qo > 0.
which
we
Proceeding
as
in reference
[3],
we
obtain
The behaviour of TF (a, 0) near T = 1 is obtained equating the free energies of the F and SG phases,
yielding
This
together
with
equation (3.34) gives
where we have included the results for Hopfield’s model in parentheses for comparison. Thus the F and retrieval regions are slightly larger in Little’s model.
e)
The limit
In this limit
4.
Fig.
-
Jo = 0 plot of phase diagram. Tf is the
transition
temperature between F and SG phases. The continuous lines are first order transitions, while broken ones are second order. The inset shows the appearence of the P at T 0 and large values of a.
phase
Jo we
00.
obtain from
equation (3.25)
with
=
The equations reduce to the corresponding ones of Hopfield’s model provided we rescale T Little 2 7"Hopfic)d’ This result is reasonable considering that large values of 10 suppress cycles and tend to align 0’i with Si. We also see that the synchronous dynamics is much more stable to noise than the asynchronous one for the same retrieval capacity, provided that neurons have a sufficiently large selfinteraction 1(). =
d)
The
phase-diagram around
T
=
1
for Jo
=
0.
The aim of this section is to obtain the behaviour of the curves TF ( a ), limiting the stable and metastable retrieval regions, in the vicinity of the point T 1 and a 0, where we have discontinuous first order transitions but with small order parameters, so that the equation (3.25) may be expanded in powers of n and t -1 - T. We obtain =
=
4.
Replica symmetry breaking.
It is well known, that replica symmetry breaking (RSB) is necessary to stabilize the spin-glass phase at low temperatures. The effects of RSB in the F and SG phase are similar to the ones in Hopfield’s model and thus small. For example, the entropy at T = 0, J" = 0 in the FM phase is S = -1.4 x 10- , at a = a, 0.138 and vanishing exponentially =
.
From these
with
we
get
In the SG
phase
at
The analog of the Almeida-Thouless line can be computed as in reference [3]. The sign of the replicon mode changes when
22
where
how be discarded. This point is being reserved for a future study. Although this P phase is unimportant for retrieval purposes, for which the replica scheme seems to provide results in agreement with simulations, we do not see how these diseases may be cured within the replica scheme. We have studied this phase by simulations at T 0. In figure 5 we show the result of measuring the fraction q n/N, where n is the number of spins belonging to cycles, against N. We interpret these results saying that qo (= 1 - 2 q ) never reaches the value qo = -1, as predicted by equation (3.6), but rather that qn - + 1 with only a finite number of spins oscillating. This would exclude =
=
where
we
tested for instabilities
only
in directions
q a¡3’ ij a¡3 with a =1= f3 . The entropy of the low temperature P phase is also negative, going to - oo as T -+ 0 exactly as in the Sherrington-Kirkpatrick (SK) model [9]. Since RSB has nothing to say about this unphysical situation, we leave this matter for a discussion in the next section.
T
=
0, Jo ± 0 solutions with
Q
thus
stigmatize this P phase as replica symmetric calculation.
0 of section 3 and an
artefact of the
5. Discussion.
In this paper we have analised the phase diagram and storage properties of a synchronous model for associative memory. Our results are qualitatively similar to the ones of model, albeit with the following differences :
Hopfield’s
a)
there is
a new
parameter Jo, which may be used
to control the occurrence of
cycles b) a tricritical line appears at the surface separating the P and F phases c) for or 0 we find a P phase even at T 0, which is not connected to the usual high temperature P phase. This phase has negative entropy, whose =
value goes to -
oo
at T
=
0
as
in the SK model.
RSB is expected to cure the unphysical aspects of the SG and F phases, but cannot exorcise problems in the low temperature P phase, since only qo and go are nonzero here. Breaking the symmetry along the diagonal in q a¡3 and ij a¡3 is apparently useless, since we found this phase to be stable in the directions qa a and 4,,aLet us note that this P phase is our only option in the region Jo -1, where a solution with qo 0 is to be expected. The SG phase with qo 0 doesn’t
exist for
a
2 or
as can
be
seen
from
equation (3.8)
and comments there after. RSB may extend the size of the SG solution, so that the P phase may some
Fig. 5. Histograms of the fraction q of spins belonging to cycles for N 100, 200 and 300. The parameters « 0.5 and Jo 0 are kept fixed and we average over -
=
=
=
initial states and realizations of the
( );’) .
Acknowledgments. The research of R.K. is partially supported by CNPq and J.F.F. holds a FAPESP fellowship. Simulations were carried out on a VAX 780/11 computer, partially maintained by funds from CNPq.
23
References J. J., Proc. Natl. Acad. Sci. USA 79 (1982) 2554 ; ibid 81 (1984) 3088. [2] PERETTO, P., Biol. Cybern. 50 (1984) 51. [3] AMIT, D. J., GUTFREUND, H. and SOMPOLINSKY, H., Ann. Phvs. 173 (1987).
[1] HOPFIELD,
[4] [5] [6]
Parallel Models of Associative Memory, Eds G. E. Hinton and J. A. Anderson (Lawrence Erebaum Ass.) 1984. LITTLE, W. A., Math. Biosci. 19 (1974) 101. FONTANARI, J. F. and KÖBERLE, R., Phys. Rev. A 36 (1987) 2475.
[7]
This
which arises very naturally, when solves this model, has also been used by the authors of reference [3] and J. L. van Hemmen, Phys. Rev. A 34 (1986) 3435. If the cycle length is l, we have to introduce l copies of phase
duplication, one
space.
[8]
See
appendix
A of reference
[3]
for
a
similar cal-
culation.
[9] KIRKPATRICK, S. B 17 (1978)
and SHERRINGTON, D., 4384.
Phys.
Rev.
