A Review on Feature Binding Theory and Its Functions Observed in

Cogn Comput DOI 10.1007/s12559-016-9446-0

A Review on Feature Binding Theory and Its Functions Observed in Perceptual Process Shifei Ding 1,2

&

Lingheng Meng 1,2 & Youzhen Han 1,2 & Yu Xue 3

Received: 2 February 2015 / Accepted: 14 December 2016 # Springer Science+Business Media New York 2016

Abstract Binding problem, which is also called feature binding, is primarily about integrating distributed information scattered on different cortical areas in a reasonable way. As a key problem in cognitive science and neuroscience, this concept is increasingly becoming a focus of consciousness study. This paper first introduced the concept, characteristics, and biological basis of feature binding. Then, this paper illustrated three feature binding theories namely feature integration theory, synchronous neural activation theory, and neural network model of feature binding, and then reviewed the advantages and disadvantages of these three feature binding theories. To demonstrate why feature binding indeed exists, we reviewed works on the functions of feature binding observed in perceptual learning. Conclusions were reached that feature binding exists in many processes of perception. This paper also suggested future research in this area should focus on systematic study of bundled brain mechanisms.

Keywords Feature binding . Perceptual learning . Synchronous neural oscillations . γ-band oscillations . Cognitive neuroscience

* Shifei Ding [email protected]

1

School of Computer Science and Technology, China University of Mining and Technology, Xuzhou 221116, China

2

Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China

3

School of Computer and Software, Nanjing University of Information Science & Technology, Nanjing 210044, China

Introduction A great deal of physiological and anatomical evidence shows that different areas of the brain take charge of different sensory information and even the same type of representational information is perceived by various parts [1–3] of the brain. For example, Friedman-Hill et al. [4] found that color, shape, and motion features of one particular object were obtained by different visual perception processes. Among these features, color and shape were characterized by level-connected area which is located on ventral visual path from occipital lobe to temporal lobe of the brain. By contrast, features, for example motion, with space-time trajectory, were characterized by path from occipital lobe to parietal lobe of the brain [5]. These studies [1–3] also found that the lack of perception among patients with brain injury is selective. For instance, these studies indicated that color blindness was deficient in color vision, but shape and motion perception abilities are not affected. Similarly, motion-disabled patients only lose their motion perceptual ability, but their other perceptions are normal [6]. Study on ordinary people’s PET and functional magnetic resonance imaging (fMRI) brain images [7] suggested that when responders were asked to respond to different features of the same stimulus on a display, local activation will transfer between different brain regions. To perceive an object as a unified whole as well as avoid mistakenly binding features, such as shape, color, motion, size, and distance, the brain possesses the ability to reasonably organize all information scattered in different cortical areas together [8–10]. This is the philosophy of “Binding Problem/Feature Binding.” While some review work has been done in [10], in this paper, we will provide more details on binding problem. Binding problem is not very intuitive and obvious for human, so people are not always aware of the necessity of

Cogn Comput

solving it. Nevertheless, neuroscientific and psychological research suggests that this problem indeed exists in the human brain and has a close connection with brain functions. In some literatures, for example [11], binding problem is referred as an area concerning how brain solves problem on integrating dispersive characteristics. The concept of binding problem was initially proposed in research on feature integration and perception segmentation [12, 13], which also plays an important role in understanding the neural mechanism of cognitive and psychological process. Now, binding problem has also been introduced into studies on object perception, memory, sensorimotor integration, language processing, logical reasoning, and consciousness awakening’s producing and maintaining. For instance, Von Der Malsburg [14] pointed out that the main defect of cognitive process’s neural model was binding problem not being reasonably treated, which pointed out the importance of binding problem. Currently, binding problem is one of the intensively researched areas of brain science [15] and also has become a focus for consciousness debate [16]. In addition, the philosophy of feature binding has affected the development of other domains. For example, Yu J. et al. exploited the philosophy of binding in terms of binding different image features in order to solve image-ranking problems. Their work achieved state-ofthe-art results from a perspective of engineering and further promoted widely applying feature binding to other research fields [17–20]. Feature binding was also applied to image segmentation [21–23], pattern recognition [24–26], feature abstraction [27–29] and information retrieving [30–32], and so on. Other parts of this paper are organized as follows. In the second section, biological basis of feature binding: Gammaband is introduced. The third section introduces three main proposed theories, i.e., feature integration theory, synchronous neural oscillations theory, and neural network model of binding problem. The forth section primarily demonstrates the importance of binding mechanism in cognition processing by showing close relationships among binding and perception segmentation, perceptual information selection, memory, and awakening of consciousness. Finally, combined with our research experience, this paper gives prospects of feature binding mechanism.

Biological Basis of Feature Binding: Gamma-Band Gamma-band oscillation (γ-band) ranges from 30 to 100 Hz. It can be recorded in many brain areas such as somatosensory cortex, hippocampus, and thalamus. Gamma-band oscillation is also associated with sensory and cognitive functions, feature binding, selective attention and, in particular, memory processing [33], which forms the biological basis of feature binding. Currently, great progress on feature binding has been made, which is shown in study on gamma neural oscillations.

Moreover, gamma-band oscillation can be detected at different levels namely microscopic (spikes) level, mesoscopic (local field potentials, LFP) level, and macroscopic (electroencephalogram, EEG) level. The results from these levels all suggest that γ-band is not only a basic neural activity of a single neuron but also a neural activity of a neuron group. The functions of gamma oscillation mainly include perception and consciousness [34], arousal [35], behavior [36], and attention and memory [37]. Among these functions, attention and memory have got much deeper research. Experiments employing EEG, intracranial EEG (iEEG), and magnetoencephalography (MEG) to assess animal and human brain activities indicate that gamma-synchronized neural oscillations associate with various of feelings, including vision [38], audition [39], and somatesthesia [40]. To form these feelings, one primary function of gamma oscillation is binding multiple features. Concrete evidence obtained by different techniques is reported by scientists. For instance, Kreiter and Singer [41] used moving grating as visual stimulus to record firing pattern of two cats’ visual cortex neurons which prefer different grating directions. Their study found that the two neurons synchronously fired at gamma frequency when the directions of grating were in these two neurons’ common preference, while they asynchronously fired when the directions of grating were in different preferences. This result reveals gamma oscillation can bind different features of the same sort of stimuli. Kaiser and Buhler [42] used EEG to assess their experiment in where 16 adults were presented four types of stimuli namely a real triangle (Fig. 1 top left), an illusory triangle (Fig. 1 top right), a no-triangle stimulus (Fig. 1 bottom left) with rotated inducer disks, and a curved illusory triangle (Fig. 1 bottom right) serving as targets that subjects had to respond to (Fig. 1) at equal probabilities. In their study, illusory triangle led to an increase of gamma synchronization. This result suggests that gamma oscillation binds triangle features. Gross, Schnitzler, and Timmermann et al. [43] used MEG to record normal, healthy subjects’ responses to nociceptive stimuli. And their finding was that the amplitude of gamma oscillation induced by perceived nociceptive stimuli was significantly increased than that induced by unperceived nociceptive stimuli. This also demonstrates the significance of gamma oscillation in sensory perception. Gamma oscillation not only can produce feeling by binding features induced by one stimulus, but also can bind multimodal stimuli of one object. In [44], when researchers showed pictures or sounds from an animal to a responder, gamma spectrum experienced a significant increase, while the increase was much slight when they showed pictures or sounds from different animals to the subject. This is consistent with gamma’s capability to promote exchanging and integrating of information from different neurons. In other words, gamma oscillation can integrate activities of neurons that are used to encode different stimuli and then form a unified feeling of one object.

Cogn Comput

1)

2)

3)

4)

Fig. 1 Four types of stimuli. 1) A real triangle, 2) an illusory triangle, 3) a no-triangle stimulus, 4) a curved illusory triangle

Gamma oscillation is closely related to memory task as well. Normally, memory task can be divided into two phases: learning memory and recalling memory. Works in [45–47] suggested that when binding occurred in sensory cortex or other more advanced cognitive centers, selective attention could be achieved by regulating reactivity of downstream neurons. Work in [48] also suggested that gamma oscillation participated in both short-term and long-term memory. All of the aforementioned experiments provide evidence for feature binding from a biological perspective. The key principle of these experiments is when perception needs binding multiple features, gamma oscillation will occur, which indicates feature binding indeed exists in perception process.

location of a view, and the others contain some implicit information about the spatial arrangement of features. Detectors in every feature map connect with units on primary position map. In order to bind “what” with “where”, an attention window moves on the position map and selects features related to the current attention position. At the same time, this attention window will temporarily put other object features out of current perceptual level. As a result, selected features become the representation of current activated objects. And the structural relationship between them has also been analyzed at this moment. Meanwhile, binding error can be avoided as well. After unified object representation has been established, it can be further used to match with stored template and to recognize object. And finally, related behavior can also be completed. The aforementioned feature integration model is illustrated as Fig. 2. Feature integration theory has been supported by a large number of behavioral and cognitive neuroscience evidence. Behavioral evidence is mainly from perceptual reports and conjunction search under the condition of attention distribution. Most of the work on this theory is from study about vision system such as [50, 51]. By contrast, neuroscience evidence mainly comes from research on bilateral parietal lobe damage [5], monkey brain’s single cell recording [52], PET imaging [53], and cranial magnetic stimulation across [54]. With the advancement of research, more and more scholars point out that there are mainly three deficiencies existed in traditional feature integration theory: (1) the traditional feature binding is measured by conscious reports, so there is no using

The Main Theories on Binding Mechanism There are plenty of binding mechanism theories, but some of the most representative ones are feature integration theory in cognitive psychology, influential synchronous neural oscillations in neuroscience (also called temporal synchronization), and neural network model of binding which is proposed in recent years. The Feature Integration Theory Feature integration theory (also called spatially selective attention) is deposited and developed by Treisman in [11, 49]. As the name implies, this theory tries to employ selective attention mechanism, i.e., attention window on visual space, to explain feature binding problem. This theory holds the notion that from the early stage of parallel processing to the later integration, feature binding is achieved by spatial attention model. This model consists of a position primary map and a set of independent feature maps. And the position primary map is used to register a location of one object, but it cannot access features of that corresponding location. The feature maps mainly contain two types of information: one is “flag” which is used to indicate whether a feature exists in the

Fig. 2 Feature integration model

Cogn Comput

of unconscious implicit indicators; (2) all of these reports are inspected according to results from immediate reports and there are no discoveries based on memory reports; (3) involved information only contains characteristics and location of a stimulus and do not involve binding between behavior and perception. This conclusion is significantly inspiring in revealing general binding mechanism. In order to overcome these drawbacks, researchers exploit negative priming paradigm [55] and other indirect test methods to investigate implicit bindings. One notion proposed by Treisman [37] suggests that research on implicit and explicit binding illustrates that binding problem might be closely linked to the nature of consciousness. However, this kind of link needs to be further studied. Synchronous Neural Activation Theory Synchronous neural activation theory (also called temporal synchronization or temporal correlational hypothesis) is currently considered as another influential, yet more neurological, binding theory. This theory holds that feature binding is based on synchronous activation neurons [13, 56]. Specifically, neurons characterizing one particular object or trajectory will be synchronously activated corresponding behavior and this phenomenon can normally be accurate to millisecond. However, this kind of synchronous activation will not happen among cells in charge of different characters. This type of instant synchronous activation is the main idea of synchronous neural activation theory. Synchronous activation mechanism can selectively mark responses of neurons responsible for encoding and distinguish these responses from others activated by different objects. This highly selective and temporal structure allows different cell populations to be activated synchronously and at the same time, to be different from each other. In addition, binding based on time synchronization also has a function of selecting next processing object. The existence of accurate synchronous activation makes it possible that specific content becomes a desired activation result and is then perceived by other brain neurons. After this, these selectively activated neurons will be bound in different brain regions to ensure further processing can be achieved. Synchronous neural activation theory holds that binding is accomplished through synchronous activation of neural activity. Time synchronization’s neural basis is connective cells which simultaneously react to stimulating features. This kind of connection can make features to be bound quickly by spatially parallel manners under a temporal synchronization which is accurate to millisecond. Therefore, brain’s activation processes can be seen as a series of “micro state.” And each state is determined by simultaneous activities of a set of neurons. Broadly speaking, neurons oscillated simultaneously are considered to perceive the same thing. To illustrate this

hypothesis, herein, we provide a classic example as shown in Fig. 3, “the example of green suitcase next to blue hat” [57]. Hypothesizing the box and the hat in this figure has three attributes1: position, contour, and color, respectively. Different parts of the brain contain a large number of neurons. And these neurons can detect various properties of each object. According to this assumption, as the moving of time axis represented by dotted gray line on the right, the brain first perceives uniform and coherent images that have different features by detecting neurons oscillation’s synchronization and then combine these features into an integral suitcase (N1, N2, and N3) or a hat (N4, N5, and N6) rather than randomly combining colors and lines. Time consistency among multiple brain regions is achieved by synchronously releasing neurons. As shown in Fig. 4, synchronous release usually takes the form of synchronous oscillation with frequencies between 40 and 70 Hz. PLL (phaselocked) 40 Hz oscillation may be the best answer to explain binding problem happened in the brain from a perspective of synchronous neural activation. Phase-locked neural oscillations achieve feature binding by marking released neurons associated with certain visual stimulus. Many literatures [34, 58] have reported evidence of this synchronous neuronal release. In fact, a large amount of study, for example [16], suggests that synchronous neuronal release has a striking universality in different species and cortical system. Typically, these observed synchronous neural release has the following four characteristics: (1) it is very precise as its coincidence window is about 10 ms; (2) it reflects topological properties of feature space and depends on the closeness of receptive fields and on the similarity of neural identity deviation; (3) simultaneous release can be endogenous or exogenous. To be more specific, when stimuli lack clear time structure or engage in selfgenerated activity, endogenous simultaneous release will occur. And this release is accomplished by inherent interconnectedness. By contrast, when being requested to react to rapidly disappearing stimuli, exogenous simultaneous release will occur, and this is achieved by synchronously inputted perceptual signals; (4) simultaneous release usually associates with gamma-band/γ-band (30 to 60 Hz). Gamma-band is a sort of neural oscillation pattern happening in the brain with frequencies between 25 and 100 Hz [59, 60], though 40 Hz is a typical frequency. Besides, there is also study suggests gamma waves may be implicated in creating the unity of conscious perception (i.e., the binding problem) [56]. Binding mechanism based on temporal synchronous release has many great advantages such as low demand for functional structure, high flexibility, and wide adaptability. Malsburg [14] pointed out that even though only a small part of binding problems could be explained by time signal 1 Here, we suppose the box and the hat have only three attributes just to make our demonstration simpler.

Cogn Comput Fig. 3 The example of green suitcase next to blue hat

synchronicity of the nervous system, it still plays a central role in study on brain function. This argument provides a research basis to explain a large number of problems which has plagued people for a long time. Engel et al. [16] also admitted that even though time binding is very speculative, currently, there is no better theory to replace it in order to explain various issues related to cognitive functions of binding problem. The Neural Network Model of Binding Neural network model of binding (also is referred to as the competitive layer model [61]) was proposed by Watanabe, Nakanishi, and Aihara [15]. This model is composed of a primary map and two higher modules, which is illustrated in Fig. 5 “A schematic representation of the primary map and the higher modules” [15]. These two modules are responsible for two different c h a r a c t e r i s t i c s o f p r i m a r y m a p , r e s p e c t i v e l y. Meanwhile, each module is divided into three different levels and every two levels are double-way connected. External characteristics of an object are characterized by “overall dynamic cell collection” which are located in primary map and two higher modules. To be more specific, details are encoded by the primary map and rough complex features are characterized by the two higher modules. Finally, feature binding is accomplished by “overall dynamic cell clusters.” The foundations of this model are “functional connectivity” and are accomplished by double-way linked neural networks. Functional link means using coupled sensor to encode dynamic link between specific neurons

which change with temporal spike. This kind of link is different from traditional synaptic connections, mainly because its value will dynamically change with the exciting level of neural network and its intensity is determined by time consistency level between occasional action potentials and neurons. Meanwhile, it is directly impacted by the dynamic change between space and time. Cells linked by functional links are called “dynamic cell clusters.” This model uses double-way link, and it even can explain time synchronous release phenomena without introducing neural oscillation units

Fig. 4 Temporal synchronization of neuronal activity

Cogn Comput Fig. 5 A schematic representation of the primary map and the higher modules

which are essential in time synchronization theory as mentioned before. How to Deal with These Opposing Theories The theory posited by Treisman [14] referring to feature binding as a result of spatially selective attention requires a prerequisite that the concept of “attention” is a well-defined one. In practice, attention windows are chosen empirically, especially in computer simulation of variable attention windows used in computer vision. The concept of “attention,” however, cannot be attributed to certain neurological facts, since “attention” itself is somewhat of an abstract theory. In this sense, Treisman’s feature binding theory is a good one insomuch as it gives a theoretical framework in which to fit data, thereby setting experimental data in a significant direction. However, since theories are somewhat flexible in the sense that data can be differently interpreted to be consistent with the theory, in order for a theory to be good it must be able to be disproved by specific characteristics of the data. Since “attention” itself is an abstractly defined term, it might be the case that it would be very difficult to disprove a theory whose central element was attention. However, the notion of expanding and shrinking receptive fields of visual neurons was referred to by Treisman [11] and could serve as the concrete definition of “attention,” thus making the theory more disprovable and therefore valid. A major problem with theories as a guideline for validity of empirical data is that since there is so much experimental data to explain; differing theories can claim to explain different aspects of the data. Therefore, we cannot judge which theory is more feasible. For example, a supposedly contrasting theory to Treisman’s feature binding is the aforementioned temporal correlation hypothesis, in which features are bound together in an object representation due to the synchronous firing of specific groups

of neurons [56]. However, Treisman [11] claims that the temporal correlation hypothesis does not provide answers to the same binding problem that her theory tries to tackle. Treisman [11] claims that neural synchrony can explain how features remain bound, but does not explain how they are originally bound together. From a point of view of computing, feature integration and neural network model are easier to be simulated by computer. Synchronous neural activation model is hard to be simulated, though it reflects a valid or even sounder phenomenon. Thus, it is clear that the binding problem is not only a real one, but a problem that has many different facets that are able to be explained by both psychological and neurological data. Table 1 compared the differences between the three aforementioned theories from the point of view of the methodology and whether or not computer simulation is possible.

Functions of Binding Mechanism in Perceptual Processes In this section, we will present the functions of binding mechanism happening in perceptual processes. To data, there is still not a unified opinion on binding mechanism’s concrete functions in perceptual processes, so herein, we just refer to several popular opinions held by researchers. As Friedman-Hill [4] found in their report on perception, right parietal lobe was related to visual features binding, while Ashbridge [62] found that right parietal lobe had no direct relationship with binding in visual search task but only related to the selection of spatial position, and ventral temporal cortex was actually responsible for feature binding. Besides visual feature binding, there was a study [63] shown that hippocampus related to feature binding happened in memory. Prabhakaran et al. [64] observed that when working memory stored integrated features, prefrontal

Cogn Comput Table 1 Comparison between three theories

Theories

Methodology Simulated by computer

Feature integration theory

Synchronous neural activation

Neural network model

Psychological

Neurophysiological

Neurological

Yes

No

Yes

lobe was activated. Llinas et al. [65] found that nonspecific thalamic circuits were responsible for binding during perception awakening. In addition, in some specific binding task, left middle temporal lobe, cerebellum, left frontal lobe, parietal lobe, left medial frontal gyrus, right anterior cingulate, right frontal gyrus, right prefrontal, anterior cingulate cortex, left center back, and other brain regions could all be activated. It is obvious that in different cognitive tasks, brain’s binding mechanism can be extremely complex. However, this does not hinder researchers from applying proposed binding mechanism, even though not perfect or accurate, to artificial intelligence-related tasks. In pattern recognition, feature binding principles are employed by researchers to recognize images [66, 67] and sounds [24]. These reported results show that using feature binding principle to perform pattern recognition tasks can often get better performance than other methods. Actually, these great results obtained in other field, in turn, prove the significance of feature binding in cognitive processes. In addition to applications in computer science, there are researches indicating that emotion could also be impacted by the binding of visual and motional features [68]. Definitely, these efforts are helpful to build emotion recognition system in order to regulate people’s emotions as indicated in [69, 70]. These explorations on binding different kinds of information are significantly meaningful for research on whole cognitive mechanism, and more details will be given in the following sections.

Binding and Perception Distinction At neurophysiological level, it is found that binding is highly related to perception distinction of situation and external sense input is usually characterized by binding [71]. It has also been suggested that synchronous neural release depends on the presentation of stimulation. For instance, only when spatially separated cells response to the same object, does they show strong synchronization. If two cell populations react to two independent objects, then activations of these two cell populations almost have no time dependence. This phenomenon is observed in [71] and suggests that each object has its own corresponding mode of synchronous activation, and these modes of synchronous activation normally tend to be different. Therefore, perception distinction between different objects can be achieved by binding mechanism based on synchronous neural release. Through study on EEG, Tallon-

Baudry [72] found that the perception of consistency of objects was associated with activities of some specific γ-band. Other related work from Sato et al. [73] employed multi-scale perception to recognize extremely similar faces. This hierarchical computational model first abstracted low-level and high-level features, respectively, and then bound different levels of features together to perceive hybrid images. Although this work was not implemented in a strict sense of feature binding, it did successfully bind different levels of features and achieve state-of-the-art results, which means feature binding is helpful to explain human’s perception process at least from a perspective of engineering.

Binding and Selection of Perception Information Binding must be realized selectively, if not, miss binding will happen and lead to misconception or disorder. Concrete evidence is provided by scientists in Psychology, Neuroscience and Physiology. For example, Fries et al. [74] recorded neural responses in a cat’s visual cortex under condition of binocular rivalry. In their experiment, they simultaneously gave two stimuli moving to different directions to an awake cat’s left and right eye. Then, they used raster’s deviation to indicate the direction of eye movement and inferred perception advantage (i.e., the result of perception) according to eyes’ moving direction. Their experiment found that neuronal activities used to characterize perception of stimulus have a high degree of synchronization, while other neurons used to characterize ignored stimulus only have weak time dependence. Meanwhile, simultaneous release of these recording points accompanied advantaged γ oscillations. Interestingly, the intensity of γband of neurons characterizing selected stimulus increased, while the intensity of γ-band of neurons characterizing ignored stimulus decreased. This result shows that at least in the early stages of visual perception, dynamic selection and inhibition of sensory information are associated with synchronous adjustment of neural activities. This also proves that binding and selection of perception information are closely related to each other, since it occurs simultaneously with various perception processes. The relationship between binding and perception selection has been further confirmed by study on midbrain superior colliculus’ injury. As Brecht’s work [75] showing, potential target of directive behavior is characterized by synchronously activated cells on superior colliculus. Many other researches

Cogn Comput

attempted to directly verify the function of binding for selecting target on superior colliculus. These researches primarily study how eye movement triggered by electrical stimulus affects time dependence between micro-stimulations of different recording points on superior colliculus. And results show that eye movement is significantly different from synchronous stimulation as long as there is a slight time interval between two stimuli. In experiments implemented on human subjects, Brown [76] and Srinivasan [77] used “frequency tagging” method to investigate mechanisms of binocular rivalry’s nerve electricity and nerve magnetism. Here, “Frequency tagging” means the stimuli presented to two eyes twinkle with different frequencies. Both of their study found that cortical response was dominated by frequencies of two kinds of stimuli, and band intensity of left and right eye was enhanced when two eyes act on perception objects. Furthermore, the perception of one particular stimulus accompanies the enhancement of consistency of nerve magnetic signals generated between two hemispheres or on one single brain hemisphere. This result illustrates that the synchronization of neurons can be seen as a function of perceived state. To conclude, these researches all suggest that time-based binding mechanism plays an important role in selecting which signals to enter consciousness. Binding and Memory Fig. 6 Memory system

Binding and Working Memory Research [78, 79] suggests that the function of working memory (also called short-term memory) depends on temporal coordination among nerve cells (temporal coordination). Memory system illustrated in Fig. 6 includes three distinct levels namely sensory memory, short-term memory, and long-term memory. Tallon-Baudry et al. [37] studied the relationship between synchronization of γ-band and working memory. They found that in visual delay matching task, the change of synchronous signal occurred only in frequencies ranging from 25 to 60 Hz. This means that accurate synchronization between ventral occipital and frontal area was strengthened during matching process. And a study directed by Sarnthein et al. [80] suggested that in visual-spatial working memory task, time consistency between electrodes of forehead and posterior parietal parts had also been strengthened. And this kind of increase occurred on γ-band, while on a lower frequency band (for example θ frequency), this kind of increase did not occur. This shows that time binding mechanisms also plays an important role in working memory. Researches on aging cognition [81] found that compared with the young, the elderly had an obvious shortage on memory about binding characteristics, while there was no difference between them on memorizing single feature. This shows that binding mechanism has a closely relationship with multi-

feature combined information’s coding, saving, and extracting. This conclusion has also been supported by researches on brain imaging. Mitchell et al. [63] utilized fMRI technology to examine neural activities activated when elderly or young people trying to complete working memory tasks. Examined working memory tasks included both tasks requiring and not requiring feature binding, respectively. Their results showed that young people’s activation level in left anterior hippocampus (which is in charge of feature binding in working memory) was significantly higher when recalling binding features than recalling a single feature. By contrast, the elderly people’s activation level in this brain area was noticeably lower than single feature binding condition. This suggests that for responders who have normal binding capacity, working memory tasks requiring feature binding can lead to greater hippocampus activation. In addition to hippocampus, left anterior cingulate cortex (BA32/24) and left central back (i.e., the premotor cortex BA6) also belong to “bundling zone.” This study not only proves that binding problems exist in working memory, but also makes great contribution to revealing the neural basis of binding. Memory is a key for feature binding in obtaining coherent object representations [82]. An influential conception of visual working memory is each of a small number of discrete

Cogn Comput

memory “slots” stores an integrated representation of a single visual object and includes all its component features as well. If a scene contains more objects than slots there have, then visual attention will control which objects to gain access to memory. In [83], Bays et al. thought that a key prediction of such model was that the absolute error in recalling multiple correlated features of an object, because features belonging to an attended object were all stored and bound together. They tested participants’ ability to reproduce information from memory about an object’s color and orientation, which is shown in Fig. 7: (a) Subjects were presented with an array of colored, oriented bars followed by a pattern mask. After a blank retention interval, a probe appeared and then subjects utilized two response dials to adjust its color and orientation to match the item at corresponding location in a memory array (the target). (b, c). Turning each dial cycled probe through a circular parameter space of possible colors or orientations. Some examples of orientations (b) and colors (c) are shown corresponding to different points in each response space. (d) Recalling precision as a function of memory load: one object (low) versus six objects (high). Precision here is defined as the reciprocal of error’s standard deviation in subjects’ responses. Specifically, zero indicates chance performance. Their results support shared-resource model of working memory. In this model, increasing memory loading gradually degrades the storage of visual information and reduces the fidelity of both object’s features and feature bindings (as shown in Fig. 7d).

Lukeu et al. [84] pointed out that episodic memory depends on the ability of individuals to tie up an event’s central

information with contextual information. Central information refers to the theme of an event (such as main content of a conversation). Contextual information is composed of spatial and temporal characteristics of events, encoding mode (such as visual or auditory), physical characteristics (such as pitch or color), and emotional status. Contextual information can act as clues to extracting center information. In order to form memory traces which are used to complete and relevant to one particular event, it is not enough to just code these two kinds of information independently. Bindings between central features and contextual features must be established, because the possibility of an event being extracted depends on the number and the strength of these bindings. It is found that there are two forms of binding between central features and contextual features: intentional binding and unintentional binding. These two forms of binding have different effects on binding different characteristics. For example, Chalfonte [85] had found that for binding object with location, unintentional binding was better than intentional binding, while for binding objects with colors, intentional binding was much better. Lukeu et al. [84] further investigated the effects of these two bindings on word recognition and word-color combination recognition, and examined their corresponding brain mechanisms as well. Their results showed that in unintentional binding, left middle temporal lobe and cerebellum were both activated in word recognition task, whereas left frontal and parietal areas were activated in combining recognition task. By contrast, in intentional binding, left middle frontal gyrus, right anterior cingulate and right frontal gyrus were all activated in word recognition task, whereas only right prefrontal was activated in combining recognition task.

Fig. 7 Storage and binding of object features. a Subjects were presented with an array of colored, oriented bars followed by a pattern mask. After a blank retention interval, a probe appeared and then subjects utilized two response dials to adjust its color and orientation to match the item at

corresponding location in a memory array (the target). b, c Turning each dial cycled probe through a circular parameter space of possible colors or orientations. d Recalling precision as a function of memory load: one object (low) versus six objects (high)

Binding and Episodic Memory

Cogn Comput

Fig. 8 The activities of three sleep stages

Binding and Consciousness Awakening Binding and Perception Awakening Some study has found that γ-band appears in awake state as well as rapid eye movement (REM) sleeping state (Fig. 8). However, in deep sleeping state, such frequency band disappears from electroencephalogram (EEG) or magnetoencephalographic (MEG). The similarity between these two states in high-frequency activities suggests that, at least in these two states, synchronization of neural activities’ frequency bands is associated with awake state’s generating process. Besides, other study discovered that in a deep anesthesia state, perception awakening’s high-frequency components also disappeared just like that in deep sleeping state. This is because activities of γ-band are associated with neurons’ synchronous activation. From the above results we can infer that binding mechanisms participate in the transition from deep anesthesia state or deep sleeping state to awake state. Binding and Consciousness At present, a main notion held by consciousness domain is that sense should be regarded as an integrated function [86] generated from a group of interactive systems mainly including sensing system, memory structure, execution control center, emotion system, and motivation system. In general, consciousness’s concrete functions mainly contain the following two aspects: (1) cross-system coordination function, i.e., an ability to integrate and to harmoniously and consistently organize these processes on a distributed neural activity model; (2) dynamic selection function which dynamically selects certain content in a given time, as only a small part of

information can enter consciousness at that moment. From the meaning of binding, these two functions are closely linked to binding mechanism. In fact, the previously mentioned perception distinction, information selection, working memory, awakening, and other mental processes are all main contents of consciousness. Relationships among these processes and binding mechanism further confirm the importance of binding in consciousness. Apart from that, researchers found that binding also played an important role in natural language processing and logical reasoning [14]. Henderson [87] pointed out that the syntax of natural language needed flexible binding mechanism to handle them. For example, connectionist model was not very successful in interpreting the issue of how a word and its grammatical function combined. To solve this problem, Henderson proposed a new model based on flexible time binding mechanism, and this model was evaluated as an important step to establish a neural model for processing natural language. Besides, Ajjanagadde et al. [88] discussed the role of binding in logical reasoning. From these reports, we can see that binding problem may exist in any consciousness process that requires integrating multiple features or multi-level abstraction into an integral conception of a material or abstractive object in our lives.

Conclusions and Prospects As a review, this paper introduced major research on feature binding mechanism. Combined with the latest research at home and abroad, it gave a systematical introduction to current theoretical research on feature binding which were examined in perceptual learning. It can be seen from this paper that binding problem has come in for considerable scrutiny lately. Moreover, it has become an important field in cognitive science, neuroscience, and brain science. However, current research is still in its infancy, because there is no uniform theory can perfectly explain binding problem. For example, for most influential theories like neural synchronous activation, the hypothesis about synchronous oscillation at 40 Hz, their achievements are undeniable and they help us understand the principle of consciousness as well, but they are still very speculative and do not solve all problems about feature binding. For instance, if two or more kinds of neuron synchronous oscillation modes exist simultaneously, then binding would need a mode selection mechanism. However, this is not well answered by neural synchronous activation theory yet. In addition, brain areas in charge of different forms of binding may have different working mechanisms when they handle different cognitive tasks. Therefore, research on brain’s binding mechanism is far from over. On the one hand, it requires deeper research to figure out which brain region is in charge of one particular

Cogn Comput

binding. On the other hand, we need to enhance systematical study on integrating brain function, which is responsible for general binding problem, and also to combine a variety of scattered research on brain function location in order to integrate these theories into a systematical framework. Binding mechanism plays a vital role in a wide range of cognitive functions, including perception, memory, language, reasoning, and even consciousness. With the development of cognitive neuroscience, research on the relationship between binding problem and cognitive function has been studied deeply. And in the future, related works may more focus on refined study about binding mechanisms happening in different cognitive processes, and focus on systematical research on brain’s general binding mechanism. Study on how do brain binds different cognitive features also needs to be refined. This is because that research on how the brain images and sound features are bound together and how these external perceived features are bound to emotional features by brain are vital to get a whole perception of external word. To achieve these goals, researchers engaging in binding’s theoretical research need to collaborate with biologist, psychologist, and computer scientist. Scholars in biology could get inspiration from methods proposed by researchers in artificial intelligence as well. At the same time, it is also crucial to understand robust feature extracting and binding and the reason for false binding. Currently, research on binding mechanism are mainly limited to the human brain or cortex of mammals, so trying to explore whether creatures who have no complex cortex have feature binding ability is also meaningful.

References 1. 2.

3.

4.

5.

6. 7.

8.

9. 10. 11. 12. 13. 14. 15.

16. 17.

Acknowledgements This work is supported by the National Natural Science Foundation of China (Nos.61379101 and 61672522), the National Key Basic Research Program of China (No.2013CB329502), the Priority Academic Program Development of Jiangsu Higer Education Institutions, and Jiangsu Collaborative Innovation Center on Atmospheric Environment and Equipment Technology.

18.

19. 20.

Compliance with Ethical Standards 21. Conflict of Interest The authors declare that they have no conflict of interest. Informed Consent All procedures followed were in accordance with the ethical standards of the responsible committee on human experimentation (institutional and national) and with the Helsinki Declaration of 1975, as revised in 2008 (5). Additional informed consent was obtained from all patients for which identifying information is included in this article.

22.

23.

24. Human and Animal Rights This article does not contain any studies with human or animal subjects performed by any of the authors.

Damasio AR. The brain binds entities and events by multiregional activation from convergence zones. Neural Comput. 1989;1:123–32. Livingstone M, Hubel D. Segregation of form, color, movement, and depth: anatomy, physiology, and perception. Science. 1988;240:740–9. Wang X, Shi Z, Zhiwei S, Shi Z. A feature-based model for the integration of object recognition theory. J. Grad. Univ. Chinese Acad Sci. 2012;29:399–405. Friedman-Hill SR, Robertson LC, Desimone R, Ungerleider LG. Posterior parietal cortex and the filtering of distractors. Proc Natl Acad Sci U S A. 2003;100:4263–8. Friedman-Hill SR, Robertson LC, Treisman AM. Parietal contributions to visual feature binding: evidence from a patient with bilateral lesions. Science. 1995;269:853–5. Zeki S. Cerebral akinetopsia (visual motion blindness). Brain. 1991;114:811–24. Sereno MI, Dale AM, Reppas JB, Kwong et al. KK. Borders of multiple visual areas in humans revealed by functional magnetic resonance imaging. Science 1995;268:889–893. Zmigrod S, de Sonneville LMJ, Colzato LS, Swaab H, Hommel B. Cognitive control of feature bindings: evidence from children with autistic spectrum disorder. Psychol Res. 2013;77:147–54. Van Der Helm PA. Cognitive architecture of perceptual organization: from neurons to gnosons. Cogn Process. 2012;13:13–40. Han Y, Ding S. Research and application analysis of feature binding. Intell Inf Process VII. 2014. p. 133–40. Treisman A. Feature binding, attention and object perception. Philos Trans R Soc London. 1998;353:1295–306. Treisman AM, Gelade GA. Feature-integration theory of attention. Cogn Psychol. 1980;12:97–136. Von Der Malsburg C. The correlation theory of brain function. Model Neural Networks II. 1994;1–26. von der Malsburg C. Binding in models of perception and brain function. Curr Opin Neurobiol. 1995;5:520–6. Watanabe M, Nakanishi K, Aihara K. Solving the binding problem of the brain with bi-directional functional connectivity. Neural Netw. 2001;14:395–406. Engel AK, Singer W. Temporal binding and the neural correlates of sensory awareness. Trends Cogn Sci. 2001;5:16–25. Yu J, Rui Y, Tao D. Click prediction for web image reranking using multimodal sparse coding. IEEE Trans Image Process. 2014;23: 2019–32. Yu J, Rui Y, Tang YY, Tao D. High-order distance-based multiview stochastic learning in image classification. IEEE Trans Cybern. 2014;44:2431–42. Yu J, Hong R, Wang M, You J. Image clustering based on sparse patch alignment framework. Pattern Recogn. 2014;47:3512–9. Yu J, Tao D, Member S, Wang M. Learning to rank using user clicks and visual features for image retrieval. 2015;45:767–79. Jin J, Xu B, Wang Y. A face detection and location method based on feature binding. Signal Process Image Commun [Internet] Elsevier. 2015;36:179–89. Zheng Y, Jeon B, Xu D, QMJ W, Zhang H. Image segmentation by generalized hierarchical fuzzy C-means algorithm. J Intell Fuzzy Syst. 2015;28:961–73. van der Spoel E, Rozing MP, Houwing-Duistermaat JJ, Eline Slagboom P, Beekman M, de Craen AJM, et al. Association analysis of insulin-like growth factor-1 axis parameters with survival and functional status in nonagenarians of the Leiden Longevity Study. Aging (Albany NY). 2015;7:956–63. Zhuo G, Yu X. Real-world audititory perception based on auditory feature binding. Proc Int Conf Comput Asp Soc Netw CASoN’10. 2010;351–4.

Cogn Comput 25. 26. 27. 28.

29.

30.

31. 32.

33.

34.

35.

36.

37.

38.

39.

40.

41.

42.

43.

44.

45.

46.

Ding S, Zhang J, Jia H, Qian J. An adaptive density data stream clustering algorithm. Cognit Comput 2015;30–8. Chen B, Shu H, Zhang H, Chen G, Luo L, Quaternions A. Color image analysis by quaternion zernike moments. 2010;7–10. Zhang N, Ding S, Shi Z. Denoising laplacian multi-layer extreme learning machine. Neurocomputing. 2016;171(C):1066–74. Jian Li, Xiaolong Li, Bin Yang, Xingming Sun. Segmentationbased Image copy-move forgery detection scheme, IEEE transactions on information forensics and security, vol. 10, no. 3, 2015. p. 507–518. Xia Z, Wang X, Sun X, Wang B. Steganalysis of least significant bit matching using multi-order differences. Security & Communication Networks, 2014;7(8):1283–91. Rao ACS, Somayajulu D, Banka H, Roy S. Feature binding technique for integration of biological databases with optimized search and retrieve. Procedia Technol. 2012;6:622–9. Wen X, Shao L, Xue Y, Fang W. A rapid learning algorithm for vehicle classification. Inf Sci (Ny) Elsevier Inc. 2015;295:395–406. Chen Y, Hao C, Wu W, Wu E. Robust dense reconstruction by range merging based on confidence estimation. Sci China Inf Sci 2016;59(9):1-11. Wang J, Li X-L, Xing G-G, Wan Y. The gamma frequency band neural oscillation: generation mechanisms and functions. Prog Biochem Biophys. 2011;38:688–93. Gray CM, König P, Engel AK, Singer W. Oscillatory responses in cat visual cortex exhibit inter-columnar synchronization which reflects global stimulus properties. Nature. 1989;338:334–7. Sheer DE. Focused arousal and the cognitive 40-Hz event-related potentials: differential diagnosis of Alzheimer’s disease. Prog Clin Biol Res. 1989;317:79–94. Murthy VN, Fetz EE. Coherent 25- to 35-Hz oscillations in the sensorimotor cortex of awake behaving monkeys. Proc Natl Acad Sci U S A. 1992;89:5670–4. Tallon-Baudry C, Bertrand O, Peronnet F, Pernier J. Induced γband activity during the delay of a visual short-term memory task in humans. J Neurosci. 1998;18:4244–54. Gruber T, Muller MM. Effects of repetition priming on induced gamma band responses in the human EEG. Psychophysiology. 2002;39:S39. Galambos R, Makeig S, Talmachoff PJ. A 40-Hz auditory potential recorded from the human scalp. Proc Natl Acad Sci U S A. 1981;78:2643–7. Babiloni C, Babiloni F, Carducci F, Cincotti F, Rosciarelli F, Arendt-Nielsen L, et al. Human brain oscillatory activity phaselocked to painful electrical stimulations: a multi-channel EEG study. Hum Brain Mapp. 2002;15:112–23. Kreiter AK, Singer W. Stimulus-dependent synchronization of neuronal responses in the visual cortex of the awake macaque monkey. J Neurosci. 1996;16:2381–96. Kaiser J, Buhler M, Lutzenberger W. Magnetoencephalographic gamma-band responses to illusory triangles in humans. NeuroImage. 2004;23:551–60. Gross J, Schnitzler A, Timmermann L, Ploner M. gamma Oscillations in human primary somatosensory cortex reflect pain perception. PLoS Biol 2007;5:e133. Yuval-Greenberg S, Deouell LY. What you see is not (always) what you hear: induced gamma band responses reflect cross-modal interactions in familiar object recognition. J Neurosci. 2007;27:1090–6. Fell J, Klaver P, Lehnertz K, Grunwald T, Schaller C, Elger CE, et al. Human memory formation is accompanied by rhinalhippocampal coupling and decoupling. Nat Neurosci. 2001;4: 1259–64. Sederberg PB, Schulze-Bonhage A, Madsen JR, Bromfield EB, McCarthy DC, Brandt A, et al. Hippocampal and neocortical gamma oscillations predict memory formation in humans. Cereb Cortex. 2007;17:1190–6.

47.

48.

49. 50. 51. 52. 53.

54.

55.

56. 57. 58.

59. 60.

61. 62. 63.

64.

65.

66. 67.

68.

69.

70.

Kaiser J, Ripper B, Birbaumer N, Lutzenberger W. Dynamics of gamma-band activity in human magnetoencephalogram during auditory pattern working memory. NeuroImage. 2003;20:816–27. Stam CJ, Van Cappellen van Walsum AM, Pijnenburg YAL, Berendse HW, de Munck JC, Scheltens P, et al. Generalized synchronization of MEG recordings in Alzheimer’s disease: evidence for involvement of the gamma band. J Clin Neurophysiol 2002;19: 562–574. Bouvier S, Treisman A. Feature binding signals in visual cortex. J Vis. 2010;10:96. Ashby FG, Prinzmetal W, Ivry R, Maddox WTA. Formal theory of feature binding in object perception. Psychol Rev. 1996;103:165–92. Wichert A. The role of attention in the context of associative memory. Cognit. Comput. 2011;3:311–20. Treisman A. Feature binding, attention, and object perception. Essent Sources Sci Study Conscious. 1998;8:226. Corbetta M, Shulman GL, Miezin FM, Petersen SE. Superior parietal cortex activation during spatial attention shifts and visual feature conjunction. Science. 1995;270:802–5. Ashbridge E, Walsh V, Cowey A. Temporal aspects of visual search studied by transcranial magnetic stimulation. Neuropsychologia. 1997;35:1121–31. DeSchepper B, Treisman A. Visual memory for novel shapes: implicit coding without attention. J Exp Psychol Learn Mem Cogn. 1996;22:27–47. Singer W, Gray CM. Visual feature integration and the temporal correlation hypothesis. Ann Rev Neurosci. 1995;18:555–86. Engel AK, Müller DJ. Observing single biomolecules at work with the atomic force microscope. Nat Struct Biol. 2000;7:715–8. Engel AK, Kreiter AK, Konig P, Singer W. Interhemispheric synchronization of oscillatory neuronal responses in cat visual cortex. Science. 1991;252:1177–9. Gold I. Does 40-Hz oscillation play a role in visual consciousness? Conscious Cogn. 1999;8:186–95. Hughes JR. Gamma, fast, and ultrafast waves of the brain: their relationships with epilepsy and behavior. Epilepsy Behav. 2008;13:25–31. Wersing H, Steil J, Ritter HA. Competitive-layer model for feature binding and sensory segmentation. Neural Comput. 2001;13:357–87. Ashbridge E, Cowey A, Wade D. Does parietal cortex contribute to feature binding? Neuropsychologia. 1999;37:999–1004. Mitchell KJ, Johnson MK, Raye CL, D’Esposito M. fMRI evidence of age-related hippocampal dysfunction in feature binding in working memory. Cogn Brain Res. 2000;10:197–206. Prabhakaran V, Narayanan K, Zhao Z, Gabrieli JD. Integration of diverse information in working memory within the frontal lobe. Nat Neurosci. 2000;3:85–90. Llinás RR. The intrinsic electrophysiological properties of mammalian neurons: insights into central nervous system function. Science. 1988;242:1654–64. Liu T, Tao D. Classification with noisy labels by importance reweighting. IEEE Trans Pattern Anal Mach Intell. 2016;38:447–61. Chen B, Shu H, Coatrieux G, Chen G, Sun X, Coatrieux JL. Color image analysis by quaternion-type moments. J Math Imaging Vision 2015;51(1):625–28. Colzato LS, van Wouwe NC, Hommel B. Feature binding and affect: emotional modulation of visuo-motor integration. Neuropsychologia. 2007;45:440–6. Castillo JC, Castro-González Á, Fernández-Caballero A, Latorre JM, Pastor JM, Fernández-Sotos A, et al. Software architecture for smart emotion recognition and regulation of the ageing adult. Cognit Comput. 2016;8:357–67. Sun R, Wilson N, Lynch M. Emotion: a unified mechanistic interpretation from a cognitive architecture. Cognit Comput Springer US. 2016;8:1–14.

Cogn Comput 71.

Castelo-Branco M, Goebel R, Neuenschwander S, Singer W. Neural synchrony correlates with surface segregation rules. Nature. 2000;405:685–9. 72. Tallon-Baudry C, Bertrand O, Delpuech C, Pernier J. Stimulus specificity of phase-locked and non-phase-locked 40 Hz visual responses in human. J Neurosci. 1996;16:4240–9. 73. Sato YD, Nagatomi T, Horio K, Miyamoto H. The cognitive mechanisms of multi-scale perception for the recognition of extremely similar faces. Cognit. Comput. Springer US. 2015;7:501–8. 74. Fries P, Roelfsema PR, Engel AK, König P, Singer W. Synchronization of oscillatory responses in visual cortex correlates with perception in interocular rivalry. Proc Natl Acad Sci U S A. 1997;94:12699–704. 75. Brecht M, Singer W, Engel AK. Patterns of synchronization in the superior colliculus of anesthetized cats. J Neurosci. 1999;19:3567–79. 76. Brown RJ, Norcia AMA. Method for investigating binocular rivalry in real-time with the steady-state VEP. Vis Res. 1997;37:2401–8. 77. Srinivasan R, Russell DP, Edelman GM, Tononi G. Increased synchronization of neuromagnetic responses during conscious perception. J Neurosci. 1999;19:5435–48. 78. Takac M, Knott A. A neural network model of episode representations in working memory. Cognit. Comput. Springer US. 2015;7: 509–25. 79. Keizer AW, Verment RS, Hommel B. Enhancing cognitive control through neurofeedback: a role of gamma-band activity in managing episodic retrieval. Neuroimage Elsevier Inc. 2010;49:3404–13.

80.

81.

82.

83. 84.

85.

86.

87. 88.

Sarnthein J, Petsche H, Rappelsberger P, Shaw GL, Von Stein A. Synchronization between prefrontal and posterior association cortex during human working memory. Proc Natl Acad Sci U S A. 1998;95:7092–6. Isella V, Molteni F, Mapelli C, Ferrarese C. Short term memory for single surface features and bindings in ageing: a replication study. Brain Cogn. 2015;96:38–42. Saiki J, Miyatsuji H. Feature binding in visual working memory evaluated by type identification paradigm. Cognition. 2007;102: 49–83. Bays PM, EY W, Husain M. Storage and binding of object features in visual working memory. Neuropsychologia. 2011;49:1622–31. Lekeu F, Marczewski P, Van Der Linden M, Collette F, Degueldre C, Del Fiore G, et al. Effects of incidental and intentional feature binding on recognition: a behavioural and PET activation study. Neuropsychologia. 2002;40:131–44. Chalfonte BL, Verfaellie M, Johnson MK, Reiss L. Spatial location memory in amnesia: binding item and location information under incidental and intentional encoding conditions. Memory. 1996;4: 591–614. Roy A. On findings of category and other concept cells in the brain: some theoretical perspectives on mental representation. Cognit. Comput. Springer US. 2014;7:279–84. Henderson J. Connectionist syntactic parsing using temporal variable binding. J Psycholinguist Res. 1994;23:353–79. Ajjanagadde V, Shastri L. Rules and variables in neural nets. Neural Comput. 1991;3:121–34.