Proc. Inst. Radio Engr 47 - Center for Neural Science

V1

Hubel and Wiesel, 1968

MT

Maunsell and Van Essen, 1983

Relating MT responses to visual discrimination

0% coherence

50% coherence

100% coherence

Newsome and Paré, 1988

Downing & Movshon, 1989

Record LIP, 7a

VIP

DP FEF V1 V 2

M V4 T MST FST PITd

VP VOT

PITv

T P CITd CITv

Visual input

S

AITd AITv

Record LIP, 7a

VIP

DP FEF V1 V 2

M V4 T MST FST PITd

VP VOT

PITv

T P CITd CITv

Visual input

S

AITd AITv

Visual stimulus

Neuronal response

Behavioral judgement

Pref target

Receptive field

Dots Aperture

+ recording in MT

Null target 10 deg

Fix Pt Dots Targets 1 sec

Fixation Point

MT responses depend on motion coherence 200 100

Firing rate (impulses/trial)

150 100

50

Preferred direction Null direction

50 0

0

25

50

75

100

0

10

20

30

40

50

100

50 40

75

30

50

20

25

10 0

0

0

25

50

75

100

0

0

10

20

30

40

Coherence (%) Britten, Shadlen, Newsome & Movshon, 1993

Proportion correct

Behavioral performance from one session

Coherence (%)

Britten, Shadlen, Newsome & Movshon, 1992

Barlow, Levick and Yoon, 1971

Britten, Shadlen, Newsome & Movshon, 1992

MT cells are as sensitive as monkeys to visual motion

Number of cells

Proportion correct

60

40

20

0

0.1 1 10 Threshold ratio (neuron/behavior)

Coherence (%) Britten, Shadlen, Newsome & Movshon, 1992

Neuronal threshold, fixation (%)

MT cell firing does not require the observer to make a decision

100

10

1 1

10

100

Neuronal threshold, choice (%)

Britten, Shadlen, Newsome & Movshon, 1992

Visual stimulus

Neurometric function

Neuronal response

Psychometric function

Behavioral judgement

Visual stimulus

Neurometric function

Neuronal response

Psychometric function

?

Behavioral judgement

MT cell firing is correlated with behavioral choice

0% coherence Choose “preferred”

Choose “anti”

Britten, Newsome, Shadlen, Celebrini & Movshon, 1996

MT cell firing is correlated with behavioral choice

0% coherence Choose “preferred”

Choose “anti”

Britten, Newsome, Shadlen, Celebrini & Movshon, 1996

Response of "antipreferred" neuron

MT cell firing is correlated with behavioral choice

Response of "preferred" neuron

"p Ch re oo fe se rre d"

nt Cho ip o re se fe rre d" "a

Response of "antipreferred" neuron

MT cell firing is correlated with behavioral choice

Response of "preferred" neuron

"p Ch re oo fe se rre d"

nt Cho ip o re se fe rre d" "a

Response of "antipreferred" neuron

MT cell firing is correlated with behavioral choice

Response of "preferred" neuron Choice probability ~ 0.85

Response of "preferred" neuron

Correlation of activity to choice is not accidental

Britten, Newsome, Shadlen, Celebrini & Movshon, 1996

Difference in normalized response

Mean normalized response

Choice-related activity has a “forward” time course 0.4

0.2

0.0 0

500 1000 Time (msec)

1500

2000

0

500 1000 Time (msec)

1500

2000

0.1

0.0

-0.1

Britten, Newsome, Shadlen, Celebrini & Movshon, 1996

The most sensitive neurons are most correlated to choice

Britten, Newsome, Shadlen, Celebrini & Movshon, 1996

and simultaneously to measure neuronal subspace maps for disparity. First we examined how the monkeys weight the disparity signal in the stimulus to form their decision16. We calculated the difference between the average stimulus preceding the monkeys’ ‘near’ choices

neurons that had been recorded while the data for the psychophysical

Choice probability for depth discrimination in V2

a

n = 17,200

0.01

0.3

Disparity (º)

0.005

a Fixation marker Stimulus

0

0

Choice targets

–0.005

0 Trial start

–0.3

2 Time (s) Trial end

0

b Amplitude

0.3

c –0.3 0

10

20

30 Time (ms)

40

2,000

c Probability

Added signal:

–50%

0%

–25%

25%

0 –0.5

0 0.54 n = 57 cells

0.5 0

d

1,000 Time (ms)

2,000

n = 76 0.5

0

0.5

–0.5

0

0.5

–0.5

0

0.5

–0.5

0

0.5

–0.5

0

60

0.5

Percentage of near choices

Disparity (º) 1

R

d

0.5 0

40

0

20

–0.5

–50

–25

0 Added signal (%)

25

0.4

50

Figure 1 | Methods. a, Sketch of the sequence of events during one trial. b, Example time series of the stimulus. c, Probability mass distributions of the stimuli for one experiment (probability as a function of disparity), with signal disparities of 20.3u and 0.15u. Each plot depicts one signal condition (negative percentages indicate near signal disparities). d, The monkey’s performance for this experiment (in percentage ‘near’ choices as a function of percentage added signal).

utes of Health, 49 Convent Drive, Bethesda, Maryland 20892, USA.

blishers Limited. All rights reserved

–0.01

1

50%

0.5

2,000

2

Mean choice probability

Disparity (º)

b

1,000

89

0.6 0.8 Choice probability

Temporal integration time (ms)

ffects of ons, are we find ange in alysis of ally, we that are readily nal gain ntion19. -related es comneurons. neuronal r goal in t theory ce about se in the nse10–12, and the sensory not for ult from with one urons. A emselves ory neumarkedly forming timulus d by the en these tivity in ask, with allowed describe ubspace a kernel

Figure 2 | Psychophysical kernel and choice-related signal have different time courses. a, Psychophysical kernel (averaged over 76 experiments; n 5 17,200 trials; two monkeys) as a function of disparity and time. Colour represents amplitude (in occurrences per frame). b, Normalized amplitude of the psychophysical kernels decreases over time. c, Averaged choice-related signal over time. Shaded grey areas in b and c, 61 standard error. d, The correlation coefficient, R, over time between choice probability (for individual neurons) and the amplitude of the mean psychophysical kernel, plotted against a neuron’s mean choice probability. Colour represents temporal integration time (Supplementary Methods); bold symbols, significant R (P , 0.05, by resampling); circles, data from monkey 1; squares, data from monkey 2. 90

c p p

s m q r m s s n ( g

p c ( P ( c w p m F e n a n s c

f c a i p a r

l o i

Visual stimulus

Neurometric function

Neuronal response

Psychometric function

Choice probability

Behavioral judgement

Albright, 1984

Stimulate LIP, 7a

VIP

DP FEF V1 V 2

M V4 T MST FST PITd

VP VOT

PITv

T P CITd CITv

Visual input

S

AITd AITv

Pref target

Receptive field

Dots Aperture

+ microstimulation in MT

Null target 10 deg

Fix Pt Dots Elect Stim Targets 1 sec

Fixation Point

Proportion of choices of the preferred direction

Coherence (%) Salzman, Murasugi, Britten and Newsome, 1992

Number of cases

Equivalent visual coherence

Salzman, Murasugi, Britten and Newsome, 1992

Visual stimulus

Neurometric function

Neuronal response

Psychometric function

Choice probability Microstimulation

Behavioral judgement

Decoding MT neurons for visual motion discrimination

up

X1

up

X2

up

X3

Pooled MT Signal

● ●

〈X〉

up

● up

XN

Decision up

〈X〉

down

> 〈 X 〉down

X1

down

X2

down

X3

● ● ● down

XN

Pooled MT Signal down

〈X〉

Britten, Newsome, Shadlen, Celebrini & Movshon, 1996 Shadlen, Britten, Newsome & Movshon, 1996 Cohen & Newsome, 2009

Decoding MT neurons for visual motion discrimination

up

X1

up

X2

up

X3

Pooled MT Signal

● ●

〈X〉

up

● up

XN

Decision up

〈X〉

down

> 〈 X 〉down

X1

down

X2

down

X3

● ●

Pooled MT Signal down

〈X〉

● down

XN

Shadlen, Britten, Newsome & Movshon, 1996 Cohen & Newsome, 2009

Interneuronal correlation

Decoding MT neurons for visual motion discrimination MT neuron pairs (Zohary et al.)

up

0.5

0.0

X1

up

X2

up

X3

Pooled MT Signal

● ●

〈X〉

0

90 Difference in preferred direction (deg)

up

Zohary, Shadlen & Newsome, 1994

● up

XN

Decision up

〈X〉

down

> 〈 X 〉down

X1

down

X2

down

X3

● ●

Pooled MT Signal down

〈X〉

● down

XN

180

Shadlen, Britten, Newsome & Movshon, 1996 Cohen & Newsome, 2009

Where is sensory activity converted into decision and actions? Frontal eye field (FEF)

Lateral intra-parietal area (LIP)

V2d

7a 7b V4

V1 V2v MT/MST

LIP receives projections from MT and projects to areas that are known to contribute to the generation of saccadic eye movements

2ECORD ,)0 A

6)0

$0 &%& 6 6 

6 4 -34 &34 0)4D

60 6/4

0)4V

4 0 #)4D #)4V

6ISUAL INPUT

3

!)4D !)4V

A ●

●

Target 1

Target 2

✙

+ recording in LIP

FP

B Fixate 350 msec

✙ Targets appear 500 msec

●

●

Target 1

Target 2

✙ Random dot motion 2 sec

●

●

✙ Delay 500-1000 msec

●

●

✙ ●

●

Saccade

Shadlen and Newsome, 2001

task_panels_vert_noMF.isl

50

motion onset

Mean response (sp/s)

45

saccade

35

50 45

51.2% 25.6% 12.8% 6.4% 0%

40

N=106

40 35

30

30

25

25

20

20

15

15

10

10

-0.5

0

0.5

1

-0.5

0

0.5

Time (s) Mike Shadlen

Responses in a reaction-time version of the direction discrimination task

Mike Shadlen

Bounded accumulation of evidence

MT – sensory evidence Motion energy “step” Spikes/s

High motion strength

Low motion strength

LIP – decision formation Accumulation of evidence “ramp” Spikes/s

r

H

Time Stimulus on

~1 sec

h ig

m

n io t o

m Low

st

th g en

re n st o i t o

ngth

Threshold

Time Stimulus off

Stimulus on

~1 sec

Stimulus off

Mike Shadlen

Diffusion to bound model Criterion to answer “Right”

Accumulated evidence for Rightward and against Leftward

Momentary evidence e.g., ∆Spike rate: MTRight– MTLeft

µ = kC

Criterion to answer “Left”

C is motion strength (coherence)

Palmer et al (2005) Shadlen et al (2006)

Responses in a reaction-time version of the direction discrimination task are well described by the “race” model of integration to a decision boundary

1 P= 1 + e−2 k C B

t(C) =

B tanh(BkC) + t nd kC

saccade Grouped by RT RT (ms)

choose Tin

70

Firing rate (sp/s)

Firing rate (sp/s)

0 60 50 40

choose Tout

30 –1000

–500

0

Time from saccade (ms)

Roitman & Shadlen, 2002

Speed-Accuracy Tradeoff

e m i T n Saccade o ti

c

]

a e R

Motion Targets Fixation

e m i T

Hanks, Kiani & Shadlen, 2014

Speed-Accuracy Tradeoff

a

b 0.7

0.9

Reaction time (s)

Proportion correct

1

0.8 0.7 0.6

0.6 0.5 0.4

0.5 0 20 40 60 Motion strength (% coh)

c A

0

Tout accumulator

Tin accumulator

Bound for Tin

0 20 40 60 Motion strength (% coh)

A

0

Bound for Tout

Hanks, Kiani & Shadlen, 2014

Reacti

Proport

0.7

Speed-Accuracy Tradeoff 0.6

0.4

0.5 0 20 40 60 Motion strength (% coh)

c A

0

Tout accumulator

Tin accumulator

Bound for Tin

0 20 40 60 Motion strength (% coh)

A

Bound for Tout

0

Hanks, Kiani & Shadlen, 2014

Speed-Accuracy Tradeoff

Speed-Accuracy Tradeoff

b 1.2 1 0.8 0 20 40 60 Motion strength (% coh)

c Normalized firing rate

Normalized FR before saccade

a

Speed condition

Accuracy condition 1 0.5

1

Slower

Faster

−400 −200 0 Time from saccade (ms)

0.5

Slower

Faster

−400 −200 0 Time from saccade (ms)

Hanks, Kiani & Shadlen, 2014

Where is sensory activity converted into decision and actions? Frontal eye field (FEF)

Lateral intra-parietal area (LIP)

V2d

7a 7b V4

V1 V2v MT/MST

3TIMULATE ,)0 A

6)0

$0 &%& 6 6 

6 4 -34 &34 0)4D

60 6/4

0)4V

4 0 #)4D #)4V

6ISUAL INPUT

3

!)4D !)4V

%YE MOVEMENT

Fixation

Motion

Evoked saccade

Voluntary saccade

+ microstimulation in FEF Time

Fixation point Targets Motion Electrical stim Eye position 200 ms Gold and Shadlen, 2000

Y eye position (degrees)

5 Choice → alone

← Stimulation alone

0

-5

-5

0 5 10 X eye position (degrees)

Gold and Shadlen, 2000

Y eye position (degrees)

5 Choice → alone

Choice and ← stimulation ← Stimulation alone

0

-5

-5

0 5 10 X eye position (degrees)

Gold and Shadlen, 2000

Weak visual signal

4

Y eye position (degrees)

0 Choice and ← stimulation

5 Choice → alone

← Stimulation alone

-4 -4

0

4

-5

0

-5

0

5 10 X eye position (degrees)

0

4

Strong visual signal

-4 -4

0

4

Gold and Shadlen, 2000

Magnitude of deviation (degrees)

1.5

Strong visual signal

1.0 Weak visual signal 0.5

0

100

300 Viewing duration (ms)

500

Gold and Shadlen, 2000

Magnitude of deviation (degrees)

1.5

Strong visual signal

1.0 Weak visual signal 0.5

0

100

300 Viewing duration (ms)

500

Gold and Shadlen, 2000

Direction of motion map (MT)

DECISION

Leftward

Rightward Saccade vector map (FEF)

LIP, 7a

VIP

DP FEF V1 V 2

M V4 T MST FST PITd

VP VOT

PITv

T P CITd CITv

Visual input

S

AITd AITv

Eye movement

Motion Mmm .... left ... right?

Response How confident am I?

Certainty task

Topp

Tin o o o

Stimulus duration: 100-900 ms (truncated exponential) Sure target delay: 500-750 ms Sure reward / correct reward ≈ 0.8 Kiani & Shadlen, 2009

Post-decision wagering Motion Mmm .... left ... right?

Response How confident am I?

sure bet high stakes choice

Kiani & Shadlen, 2009

Kiani & Shadlen, 2009

accumulaBon-to-bound model

Choose right

Time

Momentary Momentary evidence e.g., evidence ∆Spike rate: MTRight– MTLeft

μ= kC – +

Decision variable (v)

Choose le7 Link 1992 Ratcliff & Smith, 2004 Palmer et al, 2005 Laming, 1968 Luce, 1986

ly e k i l t Mos ct e corr

tain r e c Un

Kiani & Shadlen, 2009

+B

μ= kC

Decline sure target -B

Choose sure target

Three free parameters: Decline sure target k, sensiBvity coefficient B, bound height θ, criterion on log-odds correct

Kiani & Shadlen, 2009

Model fits Kiani & Shadlen, 2009

PredicBons

Model fits Kiani & Shadlen, 2009

weak motion strength N = 70 neurons

Firing rate (normalized)

sure sure target target

dots motion

eye saccade movement

choose Tin

1.5 Waive good Decline good choose Tsure Reject bad

1

1

choose Tout

0.5

100 ms

t wo directions remember 2 dashed lines.

Kiani & Shadlen, 2009