Luc Rodet & Guy Tiberghien .... The network must be able to self-organize such that one group of neurons ... This kind of network can be said to "self-organize".
A DYNAMIC MODEL OF ASSOCIATIVE SEMANTIC MEMORY
Luc Rodet & Guy Tiberghien Laboratory of Experimental Psychology,University of Grenoble, France
Abstract When we analyse a particular stimulus, a number of neurons in thebrain are specifically activated. The recognition of a visual pattern can be interpreted as the activation of a particular group of such neurons, in a particular way. Some recent results (Freeman, 1990) have shown that a group of neurons may respond to a stimulus with a coupled activity. This notion of coupled neural activity can be used to understand the multiple levels of parallelism in the brain. It is known that, in response to a single stimulus, many mental representations are activated in parallel (Anderson, 1983). Our objective is to examine how coupled activity of groups of neurons can explain this parallelism of concept activation in the brain. With a mathematical model of neural activity, the response of a single neuron or groups of neurons can be simulated while, with different simulations using simple neural architectures, the synchronisation of neural activities can be demonstrated.
1. Introduction When the term "memory" is used by computer scientists, it generally refers to the address or location of stored information. Most operations performed by a computer program are directed at changing or retrieving information stored in memory. In all standard digital computers, the structure of memory is the same. The problem faced by a computer engineer is to make efficient use of memory. How should the data organized to make the access of informaion maximally efficient? For cognitive scientists, memory is not only the hardware used for storing information, but is also a "device" used by humans in many complex activities. Many studies have shown that the structure of human memory is not as simple as that of computers. Several dichotomies have been proposed to separate human memory systems according to the kinds of information and processes they support: short-term memory and long-term memory, episodic and semantic memory, procedural and declarative memory (e.g., Anderson, 1983). It is well known that access to information in human memory can occur in a variety of ways and with a variety of different access "keys". Recognition and retrieval, for example, depend on the spatial and temporal context from which they are accessed. Additionally, a variety of parameters can change the modality of memory access. For example, the degree of repetition and and the strength of associations between mental representations can change the end result of recognition or retrieval. In some ways, the structure and processes of human memory seem to operate on principles nearly antagonistic to those of computers. For example, if we wish to model human memory with classical computer systems, we must accept a fixed structure for memory that is organized in terms of addresses and bytes (Hintzmann, 1986). We must also reduce the accessibility of memory to the mere problem of retrieving information. This approach does not seem particularly appropriate for understanding human memory. To model brain activity with computers, it is necessary to adapt them to the particularities of human memory, and not the opposite. In this paper, we present a model of memory based on some psychological hypotheses. More precisely, our objective is to build a computer model of semantic memory. Semantic memory links all of the general information
about objects that are known to a subject. We assume that semantic memory is organized with associations between mental representations of patterns that have been learned. In addition, we think that connectionnist networks are well-suited to model associations in semantic memory. In this paper we present a general schematic for implementing neural network/associative memory models of semantic nets.
2. Static and dynamic connectionist systems The simulation of the semantic structure of memory by connectionist models is a difficult enterprise because there is a strong apparent contradiction between the notion of symbolic localized representations and the logic of a distributed memory. Fodor and Pylyshyn (1988) have claimed that connectionist models are unable to simulate the compositionality rule of a syntactical semantic structure. Moreover, from an empirical point of view, a complete distribution of information is probably an unrealistic postulate because we have neurophysiological evidence for a large degree of brain localization. Additionally, there is neuropsychological evidence of several functional dissociations (Caramazza, 1992 ; Mishkin & Appenzeller, 1987). In fact, there are two different main strategies for attacking this problem. First, it is possible to build hybrid models in which a connectionnist system operates on a symbolic system of semantic representations Our own choice is different and we prefer to build a genuine neuromimetic model for simulating the interactions in a semantic memory. Building a computational model of human memory imposes an adaptation of these models within psychological constraints. Connectionist models can be considered as an attempt to make more efficient models of a brain system that are more adapted to psychological and neurophysiological theories. Connectionist models consist of groups of interconnected units (i.e., cells, neurons). At any given time, each cell has a particular state: activated or not activated. This activity state is computed in the same way for each neuron. The state of the entire network is the set of activities of all neurons in the network. Most neural networks operate in two steps: a learning and test phase. "Learning" in a neural network is the process by which the connections between neurons of the net are modified in response to a set of input stimuli. Neural networks can be characterized by the particular learning rule used to adjust weights in response to input stimuli. In the test phase, the network computes an output in response to a particular pattern, which may or may not have been learned. In most connectionist models, when a (learned or new) pattern is presented, the system converges into a stable state, which can be taken as the response of the system to the pattern. It is in this sense that neural networks can be said to simulate some aspects of human memory. They can generate a response to a particular memory key, as a function of the collective states of neurons that have been trained to respond in particular ways during the learning process. However, connectionist models do not take into account one of the most important characteristics of human memory. When we look an object, like a lamp, our memory does not generate one and only one stable output, but generates a complex sequence of outputs (cite Tulving?). With a single image of a lamp, we activate many different mental representations (e.g., the bulb, switch, entire lamp and others associated facts that may be necessary to light the lamp). Each time we see the lamp, we will activate other mental representations associated to the lamp and to each new context. In addition, our memory, in response to the image of the lamp, does not stabilize to one particular state. The many concepts associated to the lamp are successively, and perhaps simultaneously, activated in our memory. For example, if we are in our bedroom, looking at the lamp can make us think about a book we want to read tonight, and, a few seconds later, we start to search for our reading glasses. Most neural networks can be considered as static models of human memory. Specifically, neurons in a network stabilize to a single output state. This response of the net is deterministic (i.e., always the same, if the input is the same). Neural networks are associative systems : they associate inputs with outputs. In their original form, perceptrons, Kohonen maps, Hopfield models, and
backpropagation nets are static models of memory (Hopfield, 1982 ; Kohonen, 1984 ; Rumelhart & McClelland, 1986). To take the temporal aspects of human memory into account, it is necessary to consider dynamic models (ref. other temporal neural nets). Some neural networks, which learn and generate temporal sequences, can be classified as dynamic nets. In response to an input, the network does not stabilize to a single state, but generates a particular sequence of previously learned outputs. In these dynamic networks, the sequence generated is associated to the pattern presented to the network. If the same stimulus is used as input, the response will be the same. However, to consider effects of spatio-temporal context, it is necessary to develop new dynamic connectionist models of human memory.
3. Different levels of parallelism To simulate effects of context, an associative and dynamic model of semantic memory must have a complex structure of associations between mental representations. One pattern can activate many concepts in parallel during a given time period (cf., Figure 1). Each concept can have different activation levels at any time. This fact leads to several constraints for models : a) in response to one input, the network activates several learned patterns at the same time ; b) sequences of patterns can be generated by the model ; and, c) the output of the model is not only dependent on the presented pattern, but also on the previous and next inputs and on previous states of network. _______________________________________________ Insert Figure 1 about here _______________________________________________ As previously noted, classical connectionist models do not permit many simultaneuousely activated patterns in the system. Rather, the current network state is a combination of the states of all neurons. With such an approach, it is not possible to have many patterns activated at the same time. To solve this problem, we can separate neurons of the network into groups associated with each learned pattern. With many groups of neurons in the same network, it is be possible to store a learning base and to generate outputs containing several activated patterns. In neural networks, neurons are connected with other neurons. All units of the net compute their states simultaneously. (synchronous asyncronous updating?). This first level of parallelism occurs in all connectionist models. If we introduce subsets of neurons that can be activated at the same time, we can consider a second level of parallelism. Further, we can imagine, by extension, many levels of parallelism in neural networks. These levels could allow us to built more realistic and complex models of brain structure, that are dynamic group networks (refs?).
4. A model of dynamic group networks Neural networks can be see as a single group of neurons. Thus, dividing this set of cells into subsets is not only a problem of grouping. How should the groups of neurons be distinguished? Two answers are possible for this question : neural networks with limited or complete learning. Limited learning refers to the situation where groups of neurons are separated without learning. Each unit is attached or belongs to one group, independently of the particular patterns learned. As such, there are as many groups of neurons as patterns. Learning is limited to modifying connections between predefined groups. For example, many neural networks that can learn sequences have as many layers as patterns. A group of neurons consists of a layer of a network. At the opposite
extreme, complete learning refers to a case where there is a selection of the group of neurons based on a particular pattern (Edelman, 1987). The network must be able to self-organize such that one group of neurons becomes specific for each pattern that is presented to the net. This means that the learning process must specify which neurons become specific to which patterns and must also specify the number of neurons for each group. Connections between these groups are also determined by the learning of patterns. The first approach imposes many constraints on the system. First, it requires that the number of patterns to be learned is known. Second, it assumes that the process of learning a pattern is independent of the previous state of the network : i.e., the separation between groups is defined before learning. Finally, the number of neurons that code a particular pattern is not function of the learning of the pattern. On the contrary, learning groups of neurons allows for building models of memory with less constraints. The coding of patterns only depends on these patterns. When a stimulus is presented to the network, the latter determines a group of neurons, according to the previous state of the net. With this conception of memory, the system can modify old groups when it creates another one. This kind of network can be said to "self-organize". In these two views of learning, however, there are two types of connections in the network. Connections between neurons of the same group do not play the same role as connections between neurons of different groups. The first kind of connection, defines the neural synchronization between the units corresponding to a pattern. As a neuron computes its state according to the value of the connections between itself and all associated neurons, high weights of intra-connections generate coupled activities of the neurons. The second type of connection between two groups has a temporal associative role. It determines the temporal correlation of the activity of all groups. Then, to build a dynamic groups network, it is necessary to define two kinds of connections. However, we can assume that in real neural nets there is one type of connection (even if they are all different). Differentiating two kinds of connections can be made without defining different functions for connections. If we consider high weights for intra-connections and low weights for inter-connections, and more intra-connections than inter-connections, it seems possible to assign two roles to the connections. Some simulations showed that high connections allow for synchronization of the activities of neurons for a group. Low weights define temporal associations between activities of the groups. The architecture of the model groups three parts : neurons, dendrites and synapses. _______________________________________________ Insert Figure 2 about here _______________________________________________ Neurons are connected to others through dendrites and synapses and each neuron is characterised by (Figure 2A): - many inputs - a somatic potential - an output : spikes The output Va is computed as a function of somatic potential in three parts (Figure¬3)¬: Part 1 : Va increases up to a maximum : Vl. τm is the temporal constant that controls it. Part 2 : Va is equal to the maximum of the spike : Vam. Part 3 : Va is equal to the minimum of the spike : Vk. _______________________________________________ Insert Figure 3 about here _______________________________________________
Each dendrite is characterised by (Figure 2B) : - an input potential (the output of a neuron) - many parameters : space constant, temporal constant, number of connexions between two neurons,¬... - an output potential (a part of the somatic potential of a neuron) The output potential is function of the input and of all parameters in two phases (Figure¬4). Part 1 : the somatic potential (V) increases up to the maximum (V0e-µL). A temporal constant (α) controls it. Part 2 : the somatic potential (V) decreases. A temporal constant (k) controls it. V0 is the post-synaptic potential V0e-µL is the maximum value of the somatic potential. µ is the space constant : it controls the reducing of the post-synaptic potential along the dendrite. L/W is the time corresponding to the maximum of the somatic potential. It depends on L (the length of the dendrit) and W (a parameter : the weight). _______________________________________________ Insert Figure 4 about here _______________________________________________ Synapse is simulated as a box of neurotransmitters, characterised by two parameters : an in rate and an out rate (Figure 2C). When a spike is coming from a neuron through the synapse, the neurotransmitters leave the box, and the post-synaptic potential depends on the neurotransmitters' percentage. The entire system can be illustrated as in the figure 2D. Each neuron (A or B) creates spikes with particular frequencies. Spikes changed into dendritic potentials that are grouped as input of an other neuron (C). This neuron is responding with a frequency function of the input. In first simulations, each group (A, B, C) contained six connected neurons (with weights equal to 1) and was connected to the others as follows : A to B and A to C. The average weights of connections between groups were 0.5 (A to B) and 0.7 (A to C). We simulated spikes for each neuron and we looked the activity of the net over time (Figure 2). With these simulations, we wanted to examine the synchronisation of neural activities in one group of neurons. The implementation of the mathematical model can show that many neurons with strong associations have coupled activities. _______________________________________________ Insert Figure 5 about here _______________________________________________ We tested the influence of all parameters in small architectures. We made a lot of tests to examine the comportment of one group of neurons with differents groups of parameters and with different architectures. In particular, we looked at the possibility for the system to change stable states with time. With the simulation of synapses and neurotransmitters' box, a group of neurons stays a little time in a stable state like limit cycle, and goes into a state of null activity. Then, these simulations can show the possibility of temporary stable states in groups of neurons. Another issue to consider with a dynamic groups network is that it is possible to have many levels of neuron groups. We can imagine that some groups are subdivided into subsets and that these are divided into parts, and so on. As the previous simulation showed, the unity of a group of neurons can be defined as the number and the weights of intra-connections. Thus, a tree of groups can be learned in affecting a different level of unity to each group. For example ( of "dognition"!),
the network learns a image of a dog. A group of neurons codes this image. At a second time, a part of the dog (the head) is presented to the net. In the first group, some neurons that codes the head, will be more connected. Then, there will be a group of neurons for the head and for the dog. The first will be included in the second. If, in the test phase, we present the head, the network will recognize the head before activating the dog's group (the head is an object that is a part of the dog). If we present the dog, all the neurons of the group dog will be activated : the system recognize the dog, and will not activate the head alone (the head is a part of the dog, like all parts). Then, semantic nets can be implementied in neural networks with groups of neurons and hierarchical relations and decompositionality is compatible with connectionist models.
5. Conclusion Memory is one of the most complex basic system of the brain. If we want to understand how it works, we must have a multidisciplinary approach. Connectionist models can help to simulate some psychological processes. But, to build a realistic model of semantic memory, it is necessary to consider new neural networks that take dynamism of brain activity into account. Some recent theories shows the possibility of modeling complex devices in neural networks. Groups of neurons (Edelman, 1987) seem to be a solution to code semantic nets with connectionist models, and to build dynamic system of human memory. Synchronization of the activity of neurons into chaotic attractors is a hypothesis that can be implemented in neural networks (Freeman, 1990) and may be taken in relation with groups of neurons. A dynamic model of semantic memory could help us to understand associations between concepts. The development of these models can be made in parallel with development of more complex experimental paradigms in priming. It is necessary to have data about dynamic associations in human memory to model the brain devices. Psychological experiments and connectionist theory must be strongly connected in the aim to study human memory.
References Anderson, J.R. (1983). The architecture of cognition. Cambridge, M.A.: Harvard University Press. Caramazza, R. (1992)., Is cognitive neuropsychology possible? Journal of Cognitive Neuroscience, 4, 80-95. Fodor, J.A., & Pylyshyn, Z.W. (1988). Connectionism and cognitive architecture: a critical analysis. ,Cognition, 28, 3-71. Kohonen, T. (1984), Self-Organization and Associative Memory.Springer Series in Information Sciences, Springer-Verlag, Berlin Heidelberg New-York Tokyo. Hopfield, J.J. (1982). Neural networks and physical systems with emergent selective computational abilities.Proc.Natl.Acad.Sci., USA, 79, 2554. Hintzman, D.L. (1986). “Schema abstraction” in multiple-trace memory model.Psychological Review, 93, 411-428. Edelman, G.M. (1987). Neural Darwinism : The theory of Neuronal Group Selection.New York : Basic Books. Freeman, W. (1990). Model of Biological Pattern Recognition with Spatially Chaotic Dynamics.Neural Networks, 3, 153-170. Mishkin, M., & Appenzellzer, T. (1987). The anatomy of memory. Scientific American, 256, 8089. Rumelhart, D.E., & McClelland, J.L. (1986). Parallel distributed processin:Explorations in the microstructure of cognition., Vol. 1. Cambridge (MA):MIT Press.
activation's rate
activation's rate
activation's rate
0
1
2
pattern 1 pattern 2 pattern 3 pattern 4
time
Figure n°1 : Example of the output of an associationnist and dynamic model of semantic memory : Four patterns are activated, with different levels.
somatic spikes
potential
inputs
spikes
post-synaptic potential
somatic potential
neuron (A)
in rate dendrit (B) neurotransmitters out rate
synapse (C)
spikes
A spikes
C spikes
B general description (D)
Figure 2 : Model of neural activities
2
spike (Va )
Vl E time (t)
1
3
δt (V a (t) − V l (t)) + V a (t) τm
1
V a (t + δt) = −
2
V a (t + δt) = V am
3
V a (t + δt) = V k
when Va > E
E is a thresholds
Va
time
high frequence of spikes
Va
time
small frequence of spikes
Figure 3 : Output of neurons
post-synaptic spikes potential
somatic potential
V0
V L
somatic potential V
V0 e-uL
time t
L/W
1
2 −µL
( e αt − 1)
1
V(t ) = V 0 e
2
V(t ) = V 0 e −µL ( e −kL (t −L / W ) )
Figure 4 : Somatic potential
time
A
B
means emission of a spike
C
Figure n°5 : Activities of three associated groups (A, B and C) of six neurons
