ITERATIVE RETRIEVAL IN ASSOCIATIVE MEMORIES BY ... - CiteSeerX

ITERATIVE RETRIEVAL IN ASSOCIATIVE MEMORIES BY THRESHOLD CONTROL OF DIFFERENT NEURAL MODELS THOMAS WENNEKERS, FRIEDRICH T. SOMMER and GU NTHER PALM Department of Neural Information Processing University of Ulm, 89069 Ulm, Germany We investigate the retrieval properties of Hebbian auto-associative memories in the limit of sparse coding. Appropriately chosen threshold control strategies in nite size associative memories increase the completion capacity for iterative retrieval (and even for the very fast two-step retrieval) above the asymptotic capacity for extremely large networks. We relate these results to a biologically motivated network consisting of excitatorily coupled cells which are controlled by a globally acting inhibitory interneuron. Choosing a homogenous coupling matrix and dierent excitatory single neuron types, we nd in an explicit numerical comparison, that the global behavior of spiking neurons in general is dierent from rate-function and probabilistic binary units. We also show that a network of spiking neurons with Hebbian coupling matrix is able to complete and segregate several distorted patterns that are simultaneously presented at the input. In this case the global dynamics falls into rhythmic activity and processes one input pattern per period, a behavior that might be related to rhythmic cortical activity as found, for example, in the visual cortices of cats and monkeys.

1. Introduction

Donald Hebb's idea of cell assemblies as a reasonable internal representation of events, concepts and situations in the cerebral cortex has been elaborated in considerable detail.1 2 3 4 From the engineering point-of-view it is closely related to auto-associative memory, where stored patterns are retrieved by (iterative) pattern completion.5 6 Both ideas need the mechanism of threshold control, i.e. a dynamical adjustment of a global activation or desactivation parameter to control the total activity level.2 3 7 This paper is concerned with the analysis of this process of threshold control. We start with a simple observation concerning the level of total activity that can be stabilized in an equilibrium of excitation and inhibition. Let us consider simple threshold neurons. The neuron's output is binary and it is one, if the weighted sum of its inputs exceeds a threshold value : ;

;

;

;

;

;

 In

: H.J.Herrmann, D.E.Wolf, E.Poppel (eds) Supercomputing in Brain Research: From Tomography to Neural Networks, pp 301-319, World Scienti c, Singapore, 1995. 1

2 Iterative Retrieval in Associative Memories by Threshold Control of Dierent Neural Models

z = i



1 if 0 else

P j

C i 

(1)

ij j

Now we take a statistical approch: in a large randomly connected network an average neuron gets n excitatory aerents of strenth E and n inhibitory aerents of strength I. These aerents or inputs i are modelled by binary random variables X and Y , respectively, with p[X ] = p[Y ] = p. What is the probability Xi Xe p[E
X ? I
Y   ] ? e

i

j

j

j

j

j

n

n

j

j

j

=1

j

=1

The answer can be calculated by means of elementary probability and is plotted in gure 1. With realistic values for the numbers n and n of excitatory and inhibitory inputs, stable activity is possible only at low total activity.7 e

i

Fig. 1. Iteration curves for the percentage of active neurons in a randomly connected neural network. One neuron receives on average 8000 excitatory and 2000 inhibitory connections. The strength of an inhibitory connection is 12 times that of an excitatory one. The activation threshold is varied between 14 and 21 (in units corresponding to the strength of one excitatory connection).

This analysis is based on extremely simpli ed model neurons and random connectivity. In the next section we will consider structured connectivity matrices C = (C ) that are built from auto-associative memory. In this model we will compare essentially two dierent strategies for dynamical threshold control to optimize the amount of retrieved information. The full dynamical stability problem of threshold control can only be studied with more realistic, spiking neural models that also include low-pass ltering time-constants and time delays in the control loops. The remaining sections of the paper contribute to the analysis of this problem. ij

Iterative Retrieval in Associative Memories by Threshold Control of Dierent Neural Models 3

2. Iterative pattern completion

We investigate iterative pattern completion in an optimally lled large auto-associative memory. Figure 2 shows the retrieval capacity and the 'optimal sparseness', represented by the number of one-components in 'optimal' patterns that achieve maximal completion capacity for varying memory size. 0.2 bits/synapse 0.19  0.18  0.17   0.16 0.15   0.14 0.13 1000

l

16 14 12 10 8 6 4 1000

         10000

10000

100000 no. of neurons n

100000 no. of neurons n

1e+06

1e+06

Fig. 2. a) Optimal capacities where M, k and l have been optimized. In this case k2l. The curves are obtained from the theory of one-step and two-step retrieval. The simulation results are plotted for one-step (), two-step () and iterative retrieval (). The asymptotic capacity value approximately 17.32% for one-step, two-step, and iterative retrieval is shown by the horizontal line. b) The optimal length of the address pattern l corresponding to the optimal capacity C achieved by two-step retrieval with strategy Tk+.

In the following we compare dierent threshold control strategies in one optimized situation: We always have n = 1900 and between M=2000 and M=15000 patterns stored in an auto-associative n-neuron feedback network. Each pattern has k = 13 one-components (and n ? k zero-components). We start the retrieval

4 Iterative Retrieval in Associative Memories by Threshold Control of Dierent Neural Models

process by a pattern of l ones; in most cases l < k and the l ones of the start pattern form a proper subset of the k ones de ning the stored pattern to be retrieved. For a comparison of proper subset completion and correction of additional wrong ones, see gure 3. 0.2 bits/synapse   0.15      @

@

@

@

@

@

@

0.1

@

 @

@

 @

0.05 @

 e

e



( (

             











J

!

0

!



6

8

10 12 14 16 length of address l J



18

20

Fig. 3. The capacity of one-step (), two-step (), and iterative retrieval () (8th step) with the threshold control strategy Tk+ for dierent address patterns; the solid line shows the completion capacity for error free retrieval (M=10,000).

Fig. 4. Proper choice of the threshold given the number of ones in the input pattern. Greyscale represents a numerical estimate for the probability that strategy CA chooses a certain threshold for input-patterns of xed activity l.

Our rst threshold control strategy (abbreviated as Tk) simply chooses  = min(l; k). The idea with this strategy is that after the rst retrieval step the correct k ones should be among the active neurons and the threshold of k should then help to 'weed out' the additional wrong ones. This strategy can be improved technically, if one forms the intersection of the next pattern in this retrieval process with the previous pattern, thus preventing further growth of the pattern (strategy

Iterative Retrieval in Associative Memories by Threshold Control of Dierent Neural Models 5

Tk+). The technically motivated strategy Tk+ could even be realized biologically by means of a specially adjusted low-pass behavior of the neurons.

iterations

5 4 3



 

 

                1     2

0 2000

4000

6000 8000 10000 12000 14000 no. of stored patterns

0.2 bits/synapse 0.15

               0.1        0.05          0 2000

4000

6000 8000 10000 12000 14000 no. of stored patterns

Fig. 5. a) The mean iteration time of the incremental and binary Hebbian learning rule, with matrix size n=1900, k=13 and l=6. The eective storage range is 2000