Segmenting the objects before learning them. If the objects can be segmented from each other, learning becomes easier. However, some a priori knowledge ...
AB
Selective attention improves learning Antti Yli-Krekola, Jaakko Särelä and Harri Valpola Helsinki University of Technology http://www.lce.hut.fi/~aylikrek/
Helsinki University of Technology
Real world data, such as a robot’s video signal, or a speech recognizer’s audio signal, are cluttered with distractors. This makes learning a model for the data extremely difficult. The brain uses selective attention to ease the learning problem. We show how artificial systems can do it as well.
Distractors
The model
Since real-world data consist of many distinct objects, events, dynamics etc., a model of the world would also consist of distinct submodels. Learning one submodel for one object could be easy, but in the real world all the objects are present simultaneously. This causes interferences which slow down learning.
In the brain, learning happens mostly with those objects that get into the focus of selective attention [1]. We suggest that this happens for guiding representational capacity and for speeding up learning. When attention focuses on a limited amount of objects at a time, it solves the interference problem.
Object A
Object B
Object A
Object B
Selective attention has been suggested to emerge from competitions between neurons representing different features [2]. According to this biased-competition model (Figure 5), neighbouring neural areas give contextual bias to each other. The result is that the features that get selected at each time will form coherent objects. Attention segments the objects. For allowing attention to change its focus, we use neuronal habituation. The active neurons get tired, which causes them to lose the competition after a while.
Features
Figure 1. Model consists of implicit submodels
Features
Figure 2. Interference during learning causes wrong associations.
The interference problem occurs in a variety of models. It occurs in supervised learning when distinct objects in the inputs map to distinct target signals. Even learning of linear mappings becomes more difficult because of the interference. Additionally, learning a generative model for the data suffers from the interference.
Segmenting the objects before learning them If the objects can be segmented from each other, learning becomes easier. However, some a priori knowledge about the objects is required for the segmentation. Visual objects can be segmented based on local Gestalt principles (continuity, closure, proximity, etc.). Different objects share the same local rules. Figure 4 illustrates how a feature map can segment an object using Gestalt principles. Neurons are connected together based on how well their features obey the Gestalt rules.
Figure 5. Competition in local ar- Figure 6. Habituation causes the focus of attention to change eas with contextual bias We have previously used a model where biased-competition and unsupervised learning are combined and operate simultaneously [3]. For this study, we used a simplified model, where adaptive segmenting improves learning of independent component analysis (ICA).
The model uses the following steps: 1. Learn Gestalt-like rules for a set of features. We set the weight between two feature neurons to be the covariance of their activities.
Feature map / /
Activated / silent neuron, and their connections
2. Use the learned weights to segment the objects. We used biased-competition for
the segmentation. A habituation mechanism keeps attention jumping between all the objects in the input.
3. Learn ICA from segmented data.
Compared to existing models, our model is novel in the way that adaptive attention improves perceptual learning. The targets of attention are not predefined proto-objects. Instead, they are learned from the data.
The Gestalt rules can be learned from the data. For instance, the local continuity rules of visual objects can be learned from correlations between edge detectors. This strategy is not restricted to visual modality, but Gestalt rules can be learned across modalities.
The experiment ICA model is learned for 500-dimensional data. The input space is divided into five 100-dimensional areas. The Gestalt rules are learned between these areas, and competition occurs within the areas.
Sensory Input
Figure 3. A feature on the left is con- Figure 4. Using learned Gestalt principles nected to the features on the right with to segment an object strength (gray value) depending on feature correlations.
Success of ICA was measured with Amari-index [4]. Zero Amaris means perfect separation. On Figure 7, the success of FastICA, with and without our segmentation algorithm, is plotted as a function of training dataset size.
In this experiment, segmentation reduces the needed number of samples by a factor of 50.
References [1] Ahissar, M., Hochstein, S.: Attentional control of early perceptual learning. PNAS 90, 5718–5722 (1993) [2] Desimone, R., Duncan, J.: Neural mechanisms of selective visual attention. Ann. Rev. Neurosci. 18, 193–222 (1995) [3] Yli-Krekola, A.: A bio-inspired computational model of covert attention and learning. Master’s thesis, (2007) [4] Amari, S., Cichocki, A., Yang, H.H.: A new learning algorithm for blind source separation. Proc. NIPS 1995, 757–763. (1996)
Separation error [Amari]
We made artificial random continuity rules between the areas, and then made artificial ‘‘objects’’, which obey these rules. The Gestalt rules were learned from objects that do not belong in the ICA dataset, which consisted of 20 objects plus noise. 0
10
−1
10
0
10
segmented non−segmented 1
10
2
10 Samples
3
10
Figure 7. Separation success
4
10
