Seeing, Hearing, and Touching: Putting It All Together

Seeing, Hearing, and Touching: Putting It All Together Seeing Module Rapid Vision Visual Encoding Procedural Vision Navigating Visual Space

Rensink Munzner Rensink Munzner

Overview Procedural Vision ( “things” and “space”) 1.

Scene Perception •

2.

Virtual Representation

Visual Attention •

3.

Limits on perception of dynamic stimuli

Space Perception •

4.

Invariance to geometric transformations

Nonattentional Perception •

Coercive graphics

Seeing Module - Procedural Vision (Rensink)

2

1

Procedural Vision Procedural vision is that part of visual processing carried out over relatively long stretches time (> several seconds) Characteristics of procedural processing: • extended/ongoing operation (more than 200 ms) • sequential operation from item to item • controlled (consciously or via learned routines) • effortful — especially if repeatedly done • selectively provides qualities not possible otherwise - good local spatial coherence (objects) - good temporal coherence (events) - global spatial organization (scenes) Seeing Module - Procedural Vision (Rensink)

3

1. Scene Perception How Do People See Scenes?


4

2

Change Blindness

Example: Ron Rensink Seeing Module - Procedural Vision (Rensink)

5

Example: Ron Rensink Seeing Module - Procedural Vision (Rensink)

6

3

Thus, we don’t always represent all aspects of the dynamic world around us

But what is going on? What do we represent? à try to better understand change blindness


7

Change blindness occurs for changes made during: • • • • • • •

image flicker (e.g., Rensink et al., 1997) saccades (e.g., Grimes, 1996; Henderson, 1997) eyeblinks (O'Regan, Deubel, et al., 2000) "splats" not on change (Rensink et al., 2000) movie cuts (Levin & Simons, 1997) gradual change (Simons et al, 2000) real-world interruptions (Simons & Levin, 1998)


8

4

Example Dan Simons


9

Proposal (Rensink et al, 1997): Attention is needed to see (=visually experience) a change in an object. Under normal circumstances, a change creates a motion transient, which draws attention. When change is made the same time as another event, transients interfere with drawing of attention, causing the change to effective become “invisible”.


10

5

An Apparent Problem… In this view, we only accumulate what we attend to However, we can’t attend to much (4-5 items) - e.g., Pylyshyn & Storm, 1988 - Observers have coherent representation of only a few objects at any moment (= objects that are attended) Problem: If we only attend to a few objects at a time, • how can we represent the dynamic world around us? • why do we feel we see all objects at once? (metacognitive gap)

Proposal: Virtual Representation (Rensink, 2000b) Observation: Although objects appear to be present simultaneously, they do not all need to be represented simultaneously All that is needed is that the properties of the objects can be accessed when requested. If co-ordination is successful, it will appear to higher levels as if representation is “real”, i.e., as if all items present simultaneously à virtual representation Seeing Module - Procedural Vision (Rensink)

12

6

For visual perception: Can always obtain information from the world - use the world itself as an external memory (e.g., Stroud, 1955; Brooks, 1991) To build a coherent representation of an object, focus eyes and attention on appropriate location whenever that object is needed - this representation “dissolves” once it is no longer needed - only a few objects represented at any one time


Visual system "sees": 1) speaker (R.R.) 2) left screen 3) right screen 4) …

13

World Millions of objects, each with lots of data

Virtual representation: Millions of objects speaker

(R.R.)

left screen right screen

Real representation: 1-2 objects at a time

podium stage ceiling

If object already attended, use it. Else locate appropriate protoobject, and make it coherent.


noisy person

14

7

How might this be done? à Rely on other, nonattentional systems to guide attention to the right items at the right time - gist: abstract meaning of a scene - layout: locations of items, with minimal description of each item


15

HIGHER LEVELS High level control Setting (nonattentional)

Object (attentional) Complex

Layout

Gist

Low level control

Elements Early (nonattentional)


16

8

Implications for Graphics and Visualization Implication 1:

Visual representation is dynamic, not static What is seen depends on the viewer & the task - there is no “universal” interpretation of an image - different people can literally see the same world very differently à can use flicker paradigm to measure which parts and properties of objects are attended (= most easily seen to change) under various conditions


17

Implication 2 Activities such as navigating through information space are fairly natural - can easily adapt to them if designed correctly - mechanisms already exist to couple our perceptual systems to environment. - “natural-born cyborgs” (Clark, 2003) à Can use these mechanisms as the basis of effective human-machine interactions


18

9

2. Visual Attention Can use change blindness to explore the nature of visual attention (the limits of conscious perception)

E.g., Attentional Capacity Approach: Visual Search for Change - use images that change back and forth in time, like the scene examples - but images that are much simpler in content - can control the number of items, the type of change, etc. Seeing Module - Procedural Vision (Rensink)

19

Search for Change (Rensink, 2000c) - on half the trials, one of the items changes (target) - observer must report if change present or absent Displays alternate until observer responds

blank display

2

blank display

1

Focused attention holds onto item, allowing change to be seen


20

10

Look at number of items held across each flicker (i.e., the number compared each time) - increases with increased display time - asymptote = capacity


21

Results (Change): Capacity = c. 5 items Asymptote: 5.5 items

hold (items)

6 5 4 3 2

display time (ms)

1 0

160

320

480

640


800

960

22

11

Search for Absence of Change -on half the trials, one of the items doesn’t change (target) -observer reports if a nonchange is present or absent Displays alternate until observer responds

blank display 2 blank

display 1

Focused attention holds onto item, allowing change to be seen


23

Results: absence of change (orientation) (task: look for items that remain horizontal or vertical)

1.6

hold (items)

1.4 1.2

Asymptote: 1.4 items

1.0 0.8

Constant rate: 320 ms / item

0.6 0.4 0.2

on-time (ms) 0

160

320

480

640


800 24

12

Presence vs. absence Asymptote: 5.5 items

hold (items)

6 5

Why?

4 3

Asymptote: 1.4 items

2

display time (ms)

1 0

160

320

480

640

800

960


25

Hypothesis: Attention pools information from 4-5 links (pointers) into a single nexus (a) Searching for presence of change

1 vs. 0

∆

If change present, nexus value = 1 If change absent, nexus value = 0 Thus, present vs absent is 1 vs 0 — strong signal

13

(b) Searching for absence of change

4 vs. 5

∆

∆

∆

∆

If nonchange present, nexus value = 4 If nonchange absent, nexus value = 5 Thus, present vs absent is 4 vs 5 — weak signal

1 0

(b) Searching for absence of change

∆

one item at a time

∆

∆ ∆

If nonchange present, nexus value = 0 If nonchange absent, nexus value = 1 Present vs absent is 0 vs 1 — strong signal Seeing Module - Procedural Vision (Rensink)

28

14

All attended items form a single integrated structure - 4-5 links pool into a single nexus (Rensink, 2002a) à If attended items are part of a single field, may not be possible to keep changing items separate. May only be able to see one change at a time


29

(Rensink, 2002b)

160 ms

160 ms

(a) Sequence doesn’t contain a simultaneous change

(b) Sequence does contains a simultaneous change

Observer asked to report if sequence contains a simultaneous change Seeing Module - Procedural Vision (Rensink)

30

15

Dual Orientation 100

Overlapping Separated

80

70

Accuracy (%)

90

60

50

On-time (ms)

400

800

1200

Alternation (ms)

560

960

1360


31

Dual Dual Orientation Color 100

Overlapping Overlapping Separated Separated

80

70

Accuracy (%)

90

60

50

On-time (ms)

400

800

1200

Alternation (ms)

560

960

1360

16

Cannot see more than one change at a time -> “change simultagnosia”

- this differs from change blindness in that: - change can be seen; it just can’t be “multiplied” - the effect never goes away - an inherent limitation in our ability to see “2nd order” quantities (i.e., transitions) - an inherent limitation in visual attention


33

Implications for Graphics and Visualization 1. Rapid Pickup of Information à maximum information that can be picked up “at a glance” is 4-5 items - 4-5 pointers, one for each item - only a small amount of info from each item à can use flicker paradigm to find the basic units of visual attention. - units = properties which have capacity of 4-5 - compound structures will need more pointers - take longer to see, - have lower capacities Seeing Module - Procedural Vision (Rensink)

34

17

This approach can be applied to other modalities à audition: “change deafness” - map out units of auditory attention à haptics: “change numbness” - map out units of haptic attention

Can also use this approach to explore interactions: • vision + audition • audition + haptics • vision + haptics Seeing Module - Procedural Vision (Rensink)

35

2. Decomposition of objects

Rendering is often computationally expensive Need to decide what to render and in what detail

Eye movements have been used to find important parts of objects, events (O’Sullivan et al., 2002) Seeing Module - Procedural Vision (Rensink)

36

18

Can use flicker paradigm to find which parts and properties of objects are most important. (= most easily seen to change)

àmay provide descriptions of objects using minimal number of vertices (“natural decomposition”) Seeing Module - Procedural Vision (Rensink)

37

3. Conveying information dynamically à No attentional distraction from other parts of display - would create change blindness (Nowell et al, 2001) à Only one dynamic information source at a time - can’t separate two simultaneous changes; will lead to “cross-talk” à Only a small amount of information can be conveyed - perception of dynamic patterns requires attention, and attention is severely limited in capacity. (e.g. blindness to more than one movement parameter) Seeing Module - Procedural Vision (Rensink)

38

19

4. Hiding LOD transitions àTransitions due to changing level of detail (LOD) become invisible if attention is not drawn to them. This could be done in several ways… à change LOD during an eye movement (O’Sullivan et al, 2002) à change LOD during the occlusion of item à change LOD during the onset of another event à change LOD via slow fade Seeing Module - Procedural Vision (Rensink)

39

5. Computer Animation à If attention is needed to see change (movement), and attention is severely limited, will only have an accurate perception of one moving thing at a time Thus, if a part of an item is not attended, can make changes to it that are difficult to notice - changes in link length of up to 20% can be obscured in conditions where attention is absent (Harrison, Rensink, and van de Panne, 2004)


40

20

3. Space Perception What is the nature of perceptual “space”? How abstract is it? -> What kind of transformations is it invariant to? -> What kind of transformations affect performance?


41

HIGHER LEVELS High level control Setting (nonattentional)

Object (attentional) Complex

Layout

Gist

Low level control

Elements Early (nonattentional) Seeing Module - Procedural Vision (Rensink)

42

21

Layout: abstract arrangement of items in the scene. - nonvolatile - held without attention (Simons, 1996) - can be picked up without attention (Chun & Jiang, 1998) Cloud

Cloud

Barn

Truck Gate

Child

Dog Toy


43

Exploration of spatial representation via Shaker Paradigm (Rensink, 2004) - compare performance on static image with performance on alternating pair of images (image + image transformed in some way, e.g., shifted, rotated, scaled, etc.)


44

22

E.g., visual search - look for “L” shape among “T” shapes Compare speed:

vs.

Static image

Alternating image pair (2-4 cyc/s)

If space is invariant -> no effect on performance Seeing Module - Procedural Vision (Rensink)

45

Results (alternations every 240 ms - 720 ms) Translation: no effect (at least up to 4°) -> large invariance Rotation:

no effect up to 15°‚ - from 30° on, slowdown in speed -> limited invariance (to about 15°)

Scaling:

no effect up to a factor of 2:1 - from 3:1, slowdown in speed -> large invariance (up to about 2:1)


46

23

Implications for Graphics and Visualization Space is largely invariant to translations and scaling -> can make a sudden shift of up to 4°, or up to a 2:1 change in size, and performance will be unaffected E.g. zooming - for some transformations, “jumps” are okay - it is not necessary to have animations make them continuous Can use this technique to find out which kinds of geometric transformations are “natural” ones Seeing Module - Procedural Vision (Rensink)

47

E.g., Nonlinear Focus+Context transformations (Lau, Rensink, & Munzner, 2004)

Courtesy of T. Alan Keahey Seeing Module - Procedural Vision (Rensink)

48

24

Although distortions cause deterioration in performance, making the sudden transition itself does not seem to… Seeing Module - Procedural Vision (Rensink)

49

4. Nonattentional Perception Virtual representation implies an important role for nonattentional systems in vision


50

25

Higher Levels

Attentional System (objects)

Nonattentional Systems

Etc.?

Gist

Layout

Early Visual System Incoming Light Seeing Module - Procedural Vision (Rensink)

51

4. Nonattentional Perception Virtual representation implies an important role for nonattentional systems in vision These systems operate independently of attention - not concerned with explicit (conscious) perception -> Mapped out via implicit (= unconscious) detection of change?


52

26

4.1 Implicit Detection of change - Visuomotor Bridgeman et al. (1975) — oculomotor response – target moves while observer saccades to it – eye makes corrective saccade, even though observers have no explicit perception of change

Goodale et al. (1986) — manual pointing – target moves while observer saccades to it – hand corrects its trajectory while reaching to target, even though observers have no explicit perception of change Seeing Module - Procedural Vision (Rensink)

53

Proposal: Two visual systems Milner & Goodale (1995) "What" system -

requires attention relatively slow (c. 300 ms) conscious "picture" of world basis for rational decisions

"How" system Eye

-

may not require attention quite fast visuomotor control emotions...

Two systems are largely separate - supported by two separate neural pathways The “how” system is essentially an “inner zombie” Seeing Module - Procedural Vision (Rensink)

54

27

4.2 Implicit Detection of Change - Perception Fernandez-Duque & Thornton (2000) – observers view 2-display sequence; each display is a simple array of rectangles – observers tested on two items: the item changed, and the item diagonally across from it

? time

– If observer did not notice change, asked to guess which item changed. Seeing Module - Procedural Vision (Rensink)

55

Results Observers could guess better than chance (55-63%) when change was not consciously noticed No attentional effects at location of unnoticed change involvement of purely nonattentional system


56

28

4.3 Visual Awareness Without Visual Experience Origin - reports by some observers that they “sensed” the change long before they saw (= visually experienced) it.


57

Rensink (2004b) – observers view flicker sequence (natural images) – asked to hit button (t1) when change was felt – then hit button (t2) when change was seen

inc

r

ea

g sin

tim

e

t2: time change first seen

t1: time change first felt


58

29

Results 1/2 of observers had no feeling of change without visual experience of it 1/3 of observers could accurately sense change Not a result of guessing: – accuracy on catch trials is good (82%)

Different than seeing with visual experience - different sensitivities to types of change


59

Mindsight: Conscious (mental) awareness without an accompanying visual experience

Mindsight may be due to a nonattentional system - provides an alert for the observer? - basis of the belief in a “sixth sense”???


60

30

Overview of Visual Systems Attention (Objects) Layout (Setting) "What"

Gist (Meaning) Statistics (?) ...

Vision Eye movements Head movements "How"

Hand movements Arm movements ...


61

Overview of Visual Systems Only one of these systems is involved with conscious visual perception

Attention (Objects) Layout (Setting) "What"

Gist (Meaning) Statistics (?) ...

Vision Eye movements Head movements "How"

Hand movements Arm movements ...


62

31

Implications for Graphics and Visualization Displays can influence other aspects of experience besides the “conscious image” in our heads 1. Visuomotor Actions - displays designed for the “inner zombie”: àcould guide user actions (e.g. control of mouse) - no conscious awareness of this à avoid problems with lag for visual feedback - open-loop control of pointing (Po, 2002)


63

2. Displays for “Sixth Sense” Experience - feeling that something is occurring, without an accompanying visual experience à use as a natural form of alert - increase user vigilance without disrupting normal attentional allocation during a task (e.g. when driving)


64

32

3. Coercive Graphics (Rensink, 2002c) - unconscious cues could send attention of user to appropriate location at appropriate time - viewer will see (or not see) selected items -> Soft warning: user automatically sees what they should see (e.g. incoming mail) - no need for hard warning (e.g. beep); attention is controlled in more natural way - such factors are what magicians use to control what an audience “sees” Seeing Module - Procedural Vision (Rensink)

65

Summing Up… Perception of scenes is not based on a detailed “picture” • visual perception is a dynamic, “just in time” process • vision is similar in operation to other senses (e.g., touch)

Nonattentional processes provide considerable guidance • graphics for “inner zombie” (reaching, grasping, walking…) • coercive graphics for “magical” guidance of perception

Vision is the result of not just one system, but several • attentional vision just the system concerned with object perception • these may interact much like the different senses do - similar ways of integrating information? • each of the other senses may comprise several systems


66

33