Chapterr 1. Introduction

Chapterr 1. Introduction

ImagineImagine yourself walking on the street and someone stops you to ask for directions to the train station.station. While you explain how to get there, the conversation is briefly interrupted by construction workersworkers carrying a large solid door between you and the person. As the workers have passed, thethe person you were talking to has been replaced by another person, who carries on the conversationconversation as if nothing happened. Would you notice that this person is someone else? 'Of course'course' is the intuitive answer However, in a study in which this was actually done, the change waswas noticed in only 50% of the people (Simons & Levin, 1998). The ones who did not notice it areare not a special kind of people. A myriad of evidence shows that normal human beings are simplysimply very poor at detecting changes in the visual scene in a wide variety of situations. This hashas been termed 'change blindness'.

1.. Not all information enters awareness Intuitivee experience suggests that an extensive representation of the scene accumulates as wee scan the environment. Currently relevant information is assumed to be held 'online' in short termm memory. However, it has been revealed that sudden changes occurring during a visual disruptionn such as a saccade, flicker or a blank interval are often not noticed (change blindness) (Phillips,, 1974; Pashler, 1988; Grimes, 1996; Luck & Vogel, 1997; O'Regan, Rensink & Clark, 1997,, 1999; Rensink, 2000a; reviews: Simons & Levin, 1997; Simons, 2000a; Rensink, 2002). Inn typical change blindness paradigms (Fig 1), an image is successively presented two or more times,, separated by brief blank intervals. The observer's task is to judge whether something changedd or not. Without the blank interval, the change is usually very easy to detect. However, aa blank interval of 80ms is enough to make the task extremely difficult, so that it can take as manyy as 30 alternations before the change is detected. Surprisingly, once the change has been detected,, observers can hardly believe they ever missed it. Changee blindness belongs to a broader range of effects including inattentional blindness (Mackk & Rock, 1998) and the attentional blink (Raymond, Shapiro & Arnell, 1992), showing thatt under specific circumstances, observers are functionally blind to events that would normally bee quite easy to detect. Inattentional blindness refers to the failure to notice unexpected and irrelevantt events when people are engaged in an attention demanding task (Mack & Rock, 1998; Simons,, 2000b). In an extreme example, subjects watched a video of a ball game and were instructedd to keep track of which team had possession of the ball. Many subjects failed to notice thatt during the game, a person in a gorilla suit thumping his chest walked through the scene (Simonss & Chabris, 1999). The attentional blink is a similar phenomenon in the temporal domain: Whenn two targets are presented within half a second after one another and the first is detected, perceptionn of the second target is severely impaired (Raymond et al., 1992; Joseph, Chun & Nakayama,, 1997). Changee blindness, inattentional blindness and the inattentional blink, although different, all raisee the same sort of questions: What happens in the brain so that stimuli are not perceived? Whatt causes some stimuli to reach a conscious state, while others remain unnoticed? In what wayy are complex scenes represented in the brain anyhow? This thesis will provide answers to 77

ChapterChapter 1

Figi i

Reportt if change seen

Report when change seen

FigFig 1 (A) 'One shot' - paradigm. In this paradigm, the percentage correct responses typically decreasesdecreases with the number ofpotentially relevant figures (set size). A similar paradigm was used inin our experiments. Adapted from Rensink (2000a). (B) 'Flicker' - paradigm. In this paradigm, timetime to detection (search time) typically increases with the number ofpotentially relevant figures (set(set size). Adapted from Rensink (2000a). thesee questions. This chapter introduces the main issues. In section 2, we briefly address the rolee of attention (a further introduction to attention is given in BOX 1) and of working memory. Inn section 3, we discuss three hypotheses about how change blindness occurs, which differ in thee predicted fate of internal representations, thus introducing our psychophysical experiments describedd in chapter 2 and 3. A special section about 'iconic memory', a low level visual representation,, is included in BOX 2. In sections 4 and 5 we discuss neural correlates of lowandd mid-level visual representations with special emphasis on figure-ground segregation in area V I ,, followed by an introduction to our neurophysiological experiments, about the fate of these figure-groundd signals during change blindness (chapters 4 and 5).

2.. Attention and storage in working memory protects against change blindness AA crucial factor in change detection seems to be attention, a phenomenon discussed in BOX 1.. In the real world, changes almost always involve motion or luminance changes. This often evokess a visual transient that is unique, or very salient with respect to background noise, so that itt attracts attention (Phillips & Singer, 1974; Rensink, 2000b, 2002). Without a blank interval in thee change blindness paradigm, the unique transient makes the change easy to detect. With the 88

Introduction Introduction

Figg 2 AA

Natural stimuli

B

'Highh interest' regions

Artificial stimuli All items equally salient

'Loww interest' regions

FigFig 2 (A) In natural scenes, some regions are considered more important for the meaning of the scenescene than others. Based on subject's ratings, 'high'- and 'low '-interest regions can be distinguished. ChangesChanges to 'high'-interest regions have a higher chance of being noticed than changes to 'low interest'-regions,interest'-regions, presumably because people people pay more attention to what they consider importantimportant or meaningful. (B) Using artificial stimuli, all items can be made equally salient and equallyequally meaningful (or meaningless). Under these conditions, the chance that a change will be noticednoticed decreases with the number of items on display (set size).

blankk interval, the change no longer evokes a motion transient. Other ways to mask the motion orr luminance transient have been used as well. It is possible to mask the change signal by adding concurrentt transients, so that the transient associated with the change occurs no longer in isolation.. One example are 'mud-splashes' (high luminance, patterned patches), appearing on thee screen the same moment a change occurs. This makes the change more difficult to detect becausee the change is no longer the unique attention-grabbing event (Rensink, 2000b, 2002; O'Reganetal.,, 1999). Too detect a sudden change across a brief visual disruption, the relevant item has to be encodedd into memory prior to the change in such a way that it remains stable. In change detection experiments,, every trial contains a new image, only relevant for that trial, making 'working memory'' the suitable candidate for storage (Baddeley, 1986; Luck & Vogel, 1997; Cowan, 2001).. It is widely accepted that memory for attended stimuli is better than for stimuli we ignore. Thiss also applies to change detection tasks: A pre-cue, attracting attention to the location of a changee before it occurs, reduces change blindness (Scholl, 2000). In studies with natural images, 99

ChapterChapter 1 changee blindness is attenuated for 'high-interest' items (items which are thought to be important forr the meaning of the picture) (O'Regan et al., 1999). This is illustrated in Fig 2a. People's limited attentionall resources tend to be selectively focused on 'high interest' items, as judged by fixationn behavior in free viewing (Henderson & Hollingworth, 1999). In these conditions, change blindnesss for 'low interest' items may be very similar to inattentional blindness, since the event iss very likely to occur outside the observer's focus of attention (Shapiro, 2000). Usingg artificial stimuli (Fig 2b), as opposed to natural scenes, all items can be made equally salientt and equally important. In this condition, change detection accuracy depends on the number off potentially relevant items in the display (set size) (Luck & Vogel, 1997; Rensink, 2000a). It is estimatedd that people can monitor about 4 items for change. Set size effects are commonly foundd in tasks which require attentional processing (see BOX 1). In the biased competition modell of attention, a key neural mechanism of attention is that simultaneously presented objects competee with one another for cortical representation by mutual suppression in the extra-striate cortices.. Competition increases with the number of simultaneously presented items (Desimone && Duncan, 1995; Kastner & Ungerleider, 2000). Thus, specific predictions can be made about thee effect of increasing set size on behaviour and neuronal activity.

BOX1 1 Attention n Attentionn is a selection mechanism by which some stimuli get more processingg than others. It involves enhanced processing for attended, and inhibitionn for unattended stimuli. Attention is a capacity-limited selection thatt is partially automatic and partially effortf ul.thought of as the gateway to consciouss perception and memory (Broadbent, 1958; Neisser, 1967; Treisman && Gelade, 1980; Egeth & Yantis, 1997; Scholl, 2001; Pashler, Johnston & Ruthruff, 2001). . Visuall processing is commonly divided into two successive stages, 'preattentive'' and 'attentive' processing. Pre-attentive processing is viewed as a quick andd basic feature analysis of the whole visual field, on which attention can subsequentlyy operate. A central controversy in research on visual attention concernss the level of information processing at which visual stimuli are selected byy attention. An extention of this issue is the question whether the visual field is parsedd into objects preattentively, or whether attention is first directed to unstructuredd regions of the image. In the first view, attention can directly select objectss (object-based attention), and the limit of attention is in the number of objectss that can be processed at once (Neisser, 1967). In the second view, attention primarilyy selects regions in space like a 'spotlight' (space-based attention), and objectss are constructed by virtue of attention. Here, the limit is in the spatial area fromm which information can be picked up. The area can be made larger at the costt of a loss of detail (Eriksen & Hoffman, 1973; Posner, Snyder & Davidson, 1980).. Although most attention theories emphasize either object- or space-based selection,, it is generally acknowledged that they are not mutually exclusive. Se-

10 0

Introduction Introduction lectionn may occur at various levels of processing, depending on task demands (eg.. Pashler&Badgio, 1985;Treisman, 1993; Lavie&Tsal, 1994; Vecera & Farah, 1994). . Onee of the most influencial theories about the relation between attention and visionn is Feature Integration Theory (Treisman & Gelade, 1980). In its original form,, it stated that certain 'basic' features are represented in independent retinotopicc maps for orientation, color, motion etc., in accord with the existence of distinctt visual cortical areas. Features were thought to be processed pre-attentively inn parallel across the field. Spatial attention was proposed to be necessary for 'binding'' of features into objects, in a separate 'master map'. The central prediction off the theory was that visual search for a unique feature is parallel (i.e. the target iss found quickly regardless of the number of distractors: 'pop out'), whereas search involvingg feature conjunctions is serial (i.e. the search takes longer as more distractorss are present). Moree recent research shows that the strict separation between parallel and seriall search for unique features versus conjunctions is no longer tenable. Many instancess of parallel search for conjunctions have been reported (eg. Nakayama && Silverman, 1986; McLeod, Driver & Crisp, 1988). Also, 3D properties can support parallell search. Enns & Rensink (1990) used geometric stimuli suggesting cubes thatt differed in the apparent direction of illumination. Although such stimuli consist off complex conjunctions of lines and luminance, the odd one out could be detected inn parallel. Stimuli of matched complexity that could not be interpreted as 3D figuress did not support parallel search. Finally, grouping has been shown to significantlyy influence search behavior (Driver & Baylis, 1989; Duncan & Humphreys,, 1989; 1992). For example, similar distracters may be rejected groupwise,, speeding up search, whether the target is defined by a conjunction or aa single feature (Duncan & Humphreys, 1989; 1992). Effectss of grouping and findings of object-based attention (Duncan, 1984) have ledd to proposals that attention is constrained to select image elements that are pre-attentivelyy grouped according to Gestalt rules such as collinearity, good continuation,, proximity, common color, motion etc (Duncan, 1984; Driver & Baylis, 1989;; Duncan & Humphreys, 1992). Some have postulated an intermediate level off processing between basic feature extraction and attentive selection of objects: thee surface representation (He & Nakayama, 1992; Nakayama & Shimojo, 1992; Nakayama,, He & Shimojo, 1995), somewhat like the 2.5D sketch of Marr (1982). Thiss representation can be viewed as the result of pre-attentive grouping of image elements.. In this structure, complex objects consist of multiple surfaces, while multiplee objects can be aligned along a single surface. It has been argued that attentionn has little or no access to feature extraction but must have the surface representationn as its input (He & Nakayama, 1992). Iff image elements are pre-attentively grouped, how can we explain phenomena suchh as the 'attentional blink' (Raymond, Shapiro & Arnell, 1992) and 'inattentionall blindness' (Mack & Rock, 1998)? These suggest that under conditionss of 'true inattention', people can be functionally blind to otherwise salient events. . Onee explanation may be that grouping of image elements by Gestalt principles actuallyy requires some divided attention (Treisman, 1993). In this view, parallel searchh for 3D objects occurs because for the task at hand, the subjects' attentional

ll l

ChapterChapter 1 windoww has a global setting. When attention is narrowly focused elsewhere, the globall display properties do not reach awareness. However,, the fact that some properties do not reach awareness does not necessarilyy mean that they are not processed. Indirect methods have revealed thatt grouping does occur under conditions of inattention, without the observer becomingg aware of it (Moore& Egeth, 1997; Driver, Davis, Russell, Turatto & Freeman,, 2001). In the language domain, the meaning of words has been shown too be processed during the attentional blink, even though subjects cannot explicitly namee the word (Luck, Vogel & Shapiro, 1996). Ann interesting alternative to explain the attentional blink and inattentional blindnesss is that the target event transiently attracts attention, and may even be consciouslyy perceived, but rapidly forgotten (inattentional amnesia) (Wolfe, 1999). Inn line with this idea, it has been argued that as soon as attention is taken up by somethingg new, the previously attended items return to the fleeting and unbound pre-attentivee state (Wolfe, 1999; Rensink, 2000a; Rensink, 2002).

3.. Three explanations for change blindness Changee blindness and related phenomena have given impetus to new theories about the role of internall representations in visual perception, each with its own hypotheses to account for change blindnesss (eg. O'Regan & Noë, in press; Hollingworth & Henderson, 2002; for an overview of hypothesess for change blindness: Simons, 2000a). Three hypotheses will be discussed below. Att one extreme, change blindness is considered to result from the absence of a representation of thee relevant item before the change ('failed representations'-account) (Rensink, 2000a, O'Regan && Noë, in press). The other extreme is that there is an extensive representation of both the preandd post-change scene, but that change blindness occurs because of a failure to compare the twoo ('failed comparison-account) (Scott-Brown, Baker & Orbach, 2000; Shore & Klein, 2000; Hollingworth,, Williams & Henderson, 2001; Hollingworth & Henderson, 2002). Another view, whichh is sort of in between, holds that there may be a good representation of the pre-change scene,, but that this representation is overwritten by the post-change scene ('overwriting'account)) (Beck & Levin, 2000; Brawn, Snowden & Wolfe, 1999; Becker, Pashler & Anstis, 2000; Tatler,2001). . Proponentss of the 'failed representations'-account equate the internal representation to 'what iss inside the current focus of attention' (Rensink, 2000a, 2002; O'Regan & Noë, in press). In thiss account, working memory and focused attention are considered to be different aspects of thee same process: Items are held in a spatially and temporally coherent complex as long as they aree attended, but fall back to the fleeting and unbound pre-attentive state when attention is withdrawn.. Thus, visual representation is thought to be both local and transient. The observer doess not build up a detailed representation across fixations, but rather samples the environment byy moving the eyes and by moving attention. Inn contrast, the 'failed comparison'-account holds that a representation does accumulate overr the course of eyemovements and attention. This representation contains information about thee objects in the scene and their properties, but not about exact metric details. It has a large 12 2

Introduction Introduction capacityy and is retained over the long term. In this account, poor change detection is not due to limitss in visual memory, but due to limits in the ability to compare between two instances of a visuall stimulus. To detect a change, comparisons must be made between each corresponding pairr of elements. Instead of comparing the whole scene at once, we may be able to compare only one,, or a few elements at a time. Thee overwriting account also poses that we have an extensive representation of the scene. However,, in this view most of the representation is replaced with new input after vision has been brieflyy disrupted. There are two modes of representation, 'iconic memory' (see BOX 2) and 'workingg memory'. Almost the whole scene enters into 'iconic memory', a large capacity representationn which is easily overwritten. A small subset of items enters into the more stable 'workingg memory'. Whenever visual information enters the brain, a detailed representation is madee rapidly and in parallel across the visual field, independent of the current focus of attention. Whenn the image briefly disappears, this representation persists ('iconic memory'), but when neww inputs enter the system, it is replaced. Part of the information available in the iconic representationn can be protected against interference by transferring it to the more durable, but limitedd capacity working memory store. In contrast to the 'failed representations-account, the overwritingg account posits that the visual representation is global rather than local, and that this representationn is not dependent on focal attention. Compared to the 'failed-comparisons'-account, thee large representation is more detailed, but is not available after the appearance of new stimuli orr after eyemovements.

BOXX 2 Iconicc memory: a large capacity pre-attentive representation Inn traditional psychological models of memory, there are two stages, short term memoryy and long term memory. Short term memory however, can be further subdividedd into a large capacity, but fleeting, sensory memory store ('iconic' memory),, and a limited capacity, more durable, working memory store (Atkinson && Shiffrin, 1968). Evidencee for the large capacity store has been found using partial report methodss (see fig. 3) (Sperling, 1960; Averbach & Sperling, 1961). When an array off letters is briefly shown and subjects are asked to report as many letters as theyy can, they usually manage to recall only about 4 letters (whole report). However, whenn one row in the array is cued within about 500ms (partial report), subjects cann report the entire row, even when they cannot predict which row will be cued. Thiss indicates that information about the entire array remains available briefly afterr the presentation of an image, which has been called 'iconic memory' (Sperling,, 1960; Neisser, 1967). The iconic memory representation is established evenn when a display is shown for only 50ms, therefore the underlying process workss rapidly and in parallel across the field. Hence, it appears to be pre-attentive. Inn iconic memory experiments, whole report reflects the contents of working memory,, whereas partial report reflects iconic memory (Gegenfurtner & Sperling, 1993).. More specifically, the cue promotes the transfer of specific information fromm iconic memory to the more durable working memory store.

13 3

ChapterChapter 1

Figg 3 AA Variable e interval. .

Cue e

nn 'o > o> oo

00 H P AA Z K JJ D T

1m m1 50ms s Time e 00

Partiall report method

0.5 Cuee delay in seconds

1

FigFig 3 (A) The partial report method used in iconic memory studies. An array of letters is brieflybriefly presented and after an interval of 0-1 sec, a line appears, pointing at one row. SubjectsSubjects are asked to report the letter in the cued row. The percentage correctly reported lettersletters in the partial report conditions is contrasted with the percentage correct in the wholewhole report condition. In the whole report condition, no cue is given and subjects simply reportreport as many letters as they can. Based on the percentage correct, the number of letters availableavailable can be estimated. (B) Typical results in iconic memory studies. The graph showsshows the estimated number of letters available at varied cue delays in partial report conditionsconditions (solid line), compared to the number of letters available in whole report (dotted line). line).

Itt has been argued that the iconic representation does not play a significant rolee in visual perception (Haber, 1983), because the contents of iconic memory aree easily overwritten and because it rapidly decays: The partial report advantage doess not occur when eyemovements intervene between the display and the cue, orr when a masking stimulus (Breitmeyer, 1984) appears at the location of stimuli inn the initial display. The cue has to appear before iconic memory has decayed, withinn about 500ms after stimulus offset. However, iconic memory may reflect the outputt of 'early vision' (Rensink, 2000c), including feature extraction and perhaps thee establishment of global surface layout: important stages in visual processing (BOX1). . Thee role of iconic memory in change detection has been investigated by Becker, Pashlerr & Anstis (2000), who showed that a cue appearing during the interval betweenn successive images improved change detection. After the onset of the changedd image however, cueing was useless, suggesting that the iconic representationn of the pre-change image was overwritten by the post-change image. Thee cue during the interval may have allowed subjects to shift attention to the targett item and transfer it to working memory, thus protecting it from being overwritten. .

14 4

Introduction Introductio

Argumentss mustered in favor of the absence of a detailed representation include repeated searchh studies. In repeated search tasks, observers search the same display for hundreds of trials.. It has been shown that when search is inefficient in the first trial, it remains inefficient after hundredss of trials (Wolfe, Klempen & Dahlen, 2000). This suggests that a stable representation doess not accumulate even after many instances of focused attention to each individual item in thee array. However, there is evidence that visual information is retained over the long term. Recentt research has revealed that changes can be succesfully detected even across multiple interveningintervening eye fixations, especially when the relevant object has been fixated both before, and afterr the change (Hollingworth & Henderson, 2002). Also, cueing the location of the change afterr its occurrence improves change detection (Hollingworth, in press). This suggests that a largee capacity representation is available, but only used for change detection when the comparisoncomparison is restricted to the appropriate item. However, another study shows that a postchangee cue fails to improve the detection of a change in an array of letters across an interval (Beckerr et al., 2000). This change may require a more detailed representation to be detected. A detailedd representation is in fact available shortly after the presentation of an image, in the form off 'iconic memory'. Iconic memory is likely to be overwritten by new stimuli. The interval screen betweenn successive images is a new stimulus, and the changed image is another new stimulus. Thee existence of such widely different views indicates that there is not much certainty aboutt what really happens to the internal representation of a scene over the course of a change blindnesss trial. Attention may promote encoding of part of the image into working memory, but whatt happens to the rest of the image? Is the whole image represented and do we have a limited abilityy to compare? Is it represented but overwritten in some way, or does it rapidly decay? Or is theree no representation of unattended information? Inn this thesis we evaluate these hypotheses. In chapters 2 and 3, we monitor the number of itemss retained throughout a change detection trial, and ask: is there a representation when the stimulii have disappeared? Is this representation dependent on attention to be maintained? Does thee representation contain information in bound or in unbound form? What are the conditions thatt may result in an overwriting or disappearance of this representation? 4.. A neural correlate of mid level vision Psychophysicall experiments on change blindness probe into the nature of internal representations,, and how they relate to attention, memory, and awareness (chapters 2 & 3). An alternative,, and possibly more direct, method of looking at internal representations is to record neurall activity. Chapters 4 and 5 describe a set of experiments where we recorded from the primaryy visual cortex of monkeys involved in a change detection task. AA change detection trial involves encoding the pre-change image, retaining as much as possiblee across an interruption and comparing the new version with the encoded version. Encodingg of an item is facilitated by selective attention, but the preceding section and especially chapterss 2 and 3 of this thesis show that a great deal of visual representation may be independent off selective attention (pre-attentive). Pre-attentive processing involves basic feature analysis (loww level) and some coarse grouping of features according to Gestalt principles (mid-level). In thiss process, the features of simple figures are grouped so that the figure as a whole is segregated fromm the background. The resulting 'surface representation' (2.5D sketch) may be a critical intermediatee stage between basic feature extraction and 3D object perception (Marr, 1982; He &

15 5

ChapterChapter 1

Figg 4 AA Background d

Figure e

FigFig 4 (A) Contextual modulation: a surfacesurface representation in the brain (A)(A) When a patch of similarly orientedoriented lines is distinguished from aa differently oriented surround, it resultsresults in the subjective impression ofof a 'figure' on a 'background'. (B) FromFrom 80-]00ms after stimulus onset, thethe response to the interior of the figurefigure becomes enhanced compared toto the background. This is called contextualcontextual modulation. (C) As a resultresult of this process, a surface representationrepresentation of objects in the ima buildsbuilds up (Lamme, RodriguezRodriguezRodriguez & Spekreijse, 1999).

Background d

00

100

200

300

400

500

Modulation n

00

100

200 300 Timee (ms)

400

500

Modulation n RFF positions

16 6

Introduction Introduction Nakayama,, 1992; Nakayama & Shimojo, 1992; Nakayama, He & Shimojo, 1995). It has been proposedd that attention has little or no access to basic feature extraction but must have the surfacee representation as its input (He & Nakayama, 1992; 1994). Thus, the surface representation mayy be a stage where pre-attentive and attentive mechanisms interact. Inn our neural recordings we focussed in particular on a possible neural correlate of mid-level vision,, or the surface representation (fig 4). Visual responses in primary visual cortex are characterizedd by an early (~30-70ms) transient peak of activity, followed by a more sustained periodd of activity. During the early transient, classical receptive field properties, such as orientation orr direction of motion tuning are expressed. At longer delays, however, higher levels of processing aree expressed. For example, VI responses to the two stimuli shown in figure 4a are identical up to 1000 ms after stimulus onset, as the line segments within the receptive field are identical (fig 4b). Beyondd 100ms, however, the response to line segments belonging to a figure is stronger than the responsee to background. This phenomenon is called contextual modulation, as the perceptual contextt of the line segments that stimulate the RF influences the neural response. It can be evokedd by many other cues that segregate figure from ground, including motion, depth and color,, independent of the cell's preferred receptive field stimulus (Lamme, 1995; Zipser, Lamme & Schiller,, 1996). Contextual modulation reflects the grouping of image elements into coherent surfaces,, as illustrated by figure 4c. Here, modulation is recorded for many positions of the RF relativee to the figure-ground display. Modulation arises earliest (-80 ms) at the boundary between figuree and ground, which is followed by modulation of responses to surface elements. At a latencyy of about 120 ms all line segments belonging to the figure surface evoke a uniformly elevatedd response relative to the background (Lamme, 1995; Zipser et al., 1996; Lamme, RodriguezRodriguezz & Spekreijse, 1999). These, and many other observations (Kapadia, Ito, Gilbert & Westheimer,, 1995; Lamme, Zipser & Spekreijse, 2002) suggest that contextual modulation may be aa neural correlate of mid-level vision, representing the global surface layout (Marr, 1982; He & Nakayama,, 1992; Nakayama & Shimojo, 1992; Nakayama, He & Shimojo, 1995). Whilee the VI receptive field tuning properties are mediated by feedforward activation from thee thalamus, contextual modulation is a manifestation of recurrent interactions between VI and higherr visual areas (Hupe, James, Girard, Lomder, Payne & Bullier, 1998; Lamme, Super & Spekreijse,, 1998; Bullier, Hupe, James & Girard, 2001). In contrast to RF tuning, it is absent or stronglyy reduced in the anesthetized animal (Lamme, Zipser & Spekreijse, 1998; Nothdurft, Gallant && Van Essen, 1999), or in animals that report behaviorally not to have seen the figure in the displayy (Super, Spekreijse & Lamme, 2001). This suggests that contextual modulation is reflecting aa more or less active process that is not entirely stimulus driven, maybe related to visual awareness orr attention (Lamme, 2000; Lamme & Roelfsema, 2000).

5.. The fate of figure-ground signals during change blindness Inn chapters 4 and 5 we monitor the neural activity evoked by figure-ground displays during the coursee of a change blindness trial. First, we ask what the effect is of set-size on the figure-ground signals.. A longstanding debate in vision research is whether figure-ground segregation occurss pre-attentively, or whether attention is guided to unstructured regions of the image, to be activelyy involved in figure-ground segregation. Another possibility is that figure-ground segregationn marks an intermediate level of perceptual organization (the surface representation), betweenn low level feature extraction and the deployment of attention (He & Nakayama, 1994). If 17 7

ChapterChapter 1 figure-groundd segregation depends on attention, it will decrease in strength when attentional loadd increases. Thiss has implications for change blindness. Performance in an attention demanding task is affectedd by the number of potentially relevant items, or 'set size'. In visual search, limited attentionall resources affect performance already when set size is increased from 1 to 4 items (Palmer,, Verghese & Pavel, 2000). Change detection tasks also become more difficult when set sizee increases (Luck & Vogel, 1997; Rensink, 2000b). We have seen that attentional selection promotess pre-change encoding of objects into a stable working memory representation, which is necessaryy for succesful change detection (section 2). According to the 'failed representations'accountt of change blindness, the representation of stimuli is limited to what is in the current focuss of attention. When attentional load exceeds maximum capacity, excess information in the imagee may not even reach the stage of figure-ground segregation, but stay limited to basic featuree extraction. Wee describe experiments in which we recorded activity in the striate cortex during the presentationn of th& pre-change display in monkeys, while they were performing a change detection task.. Thus, we looked at activity during the encoding phase of the task. Like in humans, change detectionn performance also decreases with set size in monkeys. We hypothesized that if contextual modulationn is dependent on the availability of attentional resources, it should decrease with set sizee (chapter 4). Furthermore, we investigated whether contextual modulation during the prechangee display predicts whether a change will be detected or not. If change blindness occurs duee to the absence of a representation of the relevant item ('failed representations'-account) (Rensink,, 2002; O'Regan & Noë, in press), a mid-level vision surface representation should be absentt for those items whose change is not detected by the animal (chapter 5).

References s Allmann J, Miezin F & McGuinness E (1985) Stimulus specific responses from beyond the classical receptivee field: neurophysiological mechanisms for local-global comparisons in visual neurons. Annu Revv Neurosci 8: 407-430. Atkinsonn RC & Shiffrin RM (1968) Human memory: A proposed system and its control processes. In K.W.Spencee and J. T. Spence (Eds.), The Psychology of Learning and Motivation: Advances in Researchh and Theory (Vol. 2, pp. 89-195). New York: Academic Press. Averbachh E & Sperling G (1961) Short term storage of visual information. In C. Cherry, Information theoryy (pp. 196-211). London: Butterworth. Baddeleyy A (1986). Working memory. Oxford: Clarendon Press. Beckk MR & Levin DT (2000) Recognizing pre- and post-change object: The role of representational volatilityy in change detection. [ARVO abstract]. Invest Ophtalmol Vis Sci 41: S421, 2225. Beckerr MW, Pashler H & Anstis S M (2000) The role of iconic memory in change detection tasks. Perceptionn 29: 273-286. Brawnn PT, Snowden RJ & Wolfe JM (1999) The minimal conditions for 'change blindness': What hidess what was. [ARVO abstract]. Invest Ophtalmol Vis Sci 40: S49, 263.

18 8

Introduction Introduction Breitmeyerr BG (1984) Visual masking: An integrative approach. New York : Oxford University Press. Broadbentt DE (1958) Perception and communication. Oxford: Oxford University Press. Bullierr J, Hupe JM, James AC & Girard P (2001) The role of feedback connections in shaping the responsess of visual cortical neurons. Prog Brain Res 134: 193-204. Cowann N (2001) The magical number 4 in short term memory: A reconsideration of mental storage capacity.. Behav Brain Sci 24: 87-185. Desimonee R & Duncan J (1995). Neural mechanisms of selective visual attention. Annu Rev Neurosci 18:: 193-222. Driverr J & Baylis GC (1989) Movement and visual attention: the spotlight metaphor breaks down. J Expp Psychol Hum Percept Perform 15: 448-456. Driverr J, Davis G, Russell C, Turatto M & Freeman E (2001) Segmentation, attention and phenomenal visuall objects. Cognition 80: 61-95. Duncann J (1984) Selective attention and the organization of visual information. J Exp Psychol Gen 113:: 501-517. Duncann J & Humphreys GW (1989) Visual search and stimulus similarity. Psychol Rev 96: 433-458. Duncann J & Humphreys GW (1992) Beyond the search surface: visual search and attentional engagement.. J Exp Psychol Hum Percept Perform 18: 578-588. Egethh HE & Yantis S (1997) Visual attention: control, representation, and time course. Annu Rev Psycholl 48: 269-297. Ennss JT & Rensink RA (1990) Influence of scene-based properties on visual search. Science 247: 721723. . Eriksenn CW & Hoffman JE (1973) The extent of processing of noise elements during selective encodingg from visual displays. Percept Psychophys 14: 155-160. Grimess J (1996). On the failure to detect changes in scenes across saccades. In K. Atkins, Perception (ppp 89-110). Oxford: Oxford University Press. Haberr RN (1983) The impending demise of the icon: A critique of the concept of iconic storage in visuall information processing. Behav Brain Sci 6: 1-54. Hee ZJ & Nakayama K