Difference of gaussians type neural image filtering with ... - Google Sites

Difference of gaussians type neural image filtering with spiking neurons Bio-inspired preattentional vision system

Sylvain Chevallier LIMSI - CNRS Orsay, France [email protected]

October, 7th. 2009

Framework Neural image filtering

Visual attention Neural computation

Outline

1. Framework Visual attention Neural computation 2. Neural image filtering Methods Results Saliency extraction

S. Chevallier (LIMSI - CNRS)

Neural Image Filtering

ICNC’09

2 / 21

Framework Neural image filtering

Visual attention Neural computation

Change blindness

S. Chevallier (LIMSI - CNRS)

Neural Image Filtering

ICNC’09

3 / 21

Framework Neural image filtering

Visual attention Neural computation

Change blindness

S. Chevallier (LIMSI - CNRS)

Neural Image Filtering

ICNC’09

3 / 21

Framework Neural image filtering

Visual attention Neural computation

Change blindness

S. Chevallier (LIMSI - CNRS)

Neural Image Filtering

ICNC’09

3 / 21

Framework Neural image filtering

Visual attention Neural computation

Bio-inspired attentional vision systems Visual features I

Attentional spotlight metaphor

I

Reduce the search space

Definitions I

Preattention and attention

I

Covert attention et overt attention

Bio-inspired vision I

Propose efficient bio-inspired solutions

I

Between realistic models and artificial systems

S. Chevallier (LIMSI - CNRS)

Neural Image Filtering

ICNC’09

4 / 21

Framework Neural image filtering

Visual attention Neural computation

Existing implementations

Main characteristics 1. Feature decomposition 2. Combination on a saliency map 3. Focus of attention with WTA selection Neural models proposed in (2) and (3) [Itti & Koch, 98]

S. Chevallier (LIMSI - CNRS)

Neural Image Filtering

ICNC’09

5 / 21

Framework Neural image filtering

Visual attention Neural computation

Existing implementations

Some applications: I

Driver assistance [Michalke, 08]

I

Medical images [Fouquier, 08]

I

Robotics [Frintrop, 06]

S. Chevallier (LIMSI - CNRS)

Neural Image Filtering

ICNC’09

5 / 21

Framework Neural image filtering

Visual attention Neural computation

Spiking neurons

I

“Third generation” of neural models

I

Precise spike timing Selective information processing

I

I

I

Different behaviors I I

I

Implicite thresholding Temporal integrator Coincidence detector

Anytime computation

S. Chevallier (LIMSI - CNRS)

Neural Image Filtering

ICNC’09

6 / 21

Framework Neural image filtering

Visual attention Neural computation

Proposed architecture

Inputs I

Luminance

I

Color

Features I

Contrasts/Edges

I

Orientation Color opponency

I S. Chevallier (LIMSI - CNRS)

I

High and low frequencies

I

Saliency extraction

I

Focus of attention

Neural Image Filtering

ICNC’09

7 / 21

Framework Neural image filtering

Methods Results Saliency extraction

Outline

1. Framework Visual attention Neural computation 2. Neural image filtering Methods Results Saliency extraction

S. Chevallier (LIMSI - CNRS)

Neural Image Filtering

ICNC’09

8 / 21

Framework Neural image filtering

Methods Results Saliency extraction

Neural model

Leaky Integrate-and-Fire (LIF)  dV i dt = −λi Vi (t) + ui (t), if Vi < ϑ else trigger a spike and Vi ← Vreset I

Vi (t) : membrane potential

I

λi : relaxation constant

I

ui (t) : command

S. Chevallier (LIMSI - CNRS)

Neural Image Filtering

ICNC’09

9 / 21

Framework Neural image filtering

Methods Results Saliency extraction

From pixels to spikes dVi dt



with τ = 1/λ

= −λi Vi (t) + KLi , if Vi < ϑ else trigger a spike and Vi ← Vreset

details

S. Chevallier (LIMSI - CNRS)

Neural Image Filtering

ICNC’09

10 / 21

Framework Neural image filtering

Methods Results Saliency extraction

Neural image filtering

Neural maps: Input map Pixels to spikes Filter Filtering results Here : difference of gaussians (DOG) type filter

S. Chevallier (LIMSI - CNRS)

Neural Image Filtering

ICNC’09

11 / 21

Methods Results Saliency extraction

Framework Neural image filtering

Neural image filtering (

dVj dt

P

j = −λj Vj (t) + ∑i=1 wij Si (t), if Vj < ϑ else trigger a spike and Vj ← Vreset

DOG filter

Neural filter

details

S. Chevallier (LIMSI - CNRS)

Neural Image Filtering

ICNC’09

12 / 21

Framework Neural image filtering

Methods Results Saliency extraction

Edge preservation on artificial images

Validation process 1. Noise corruption 2. Neural and DOG filtering 3. Sobel and threshold 4. Comparison with original image

S. Chevallier (LIMSI - CNRS)

Neural Image Filtering

ICNC’09

13 / 21

Framework Neural image filtering

Methods Results Saliency extraction

Edge preservation on artificial images

"worst" neuronal

"best" neuronal

Gradual results

S. Chevallier (LIMSI - CNRS)

Neural Image Filtering

ICNC’09

14 / 21

Framework Neural image filtering

Methods Results Saliency extraction

Edge preservation on artificial images

Neural

DOG

Only a few neurons are activated

S. Chevallier (LIMSI - CNRS)

Neural Image Filtering

ICNC’09

14 / 21

Framework Neural image filtering

Methods Results Saliency extraction

Edge preservation on artificial images

Robust to noise Edge preservation

S. Chevallier (LIMSI - CNRS)

Neural Image Filtering

ICNC’09

14 / 21

Framework Neural image filtering

Methods Results Saliency extraction

Preattentional architecture

S. Chevallier (LIMSI - CNRS)

Neural Image Filtering

ICNC’09

15 / 21

Framework Neural image filtering

Methods Results Saliency extraction

Saliency extraction on natural images

Comparison with Itti algorithm

Itti

S. Chevallier (LIMSI - CNRS)

Original

Neural Image Filtering

Neuronal

ICNC’09

16 / 21

Framework Neural image filtering

Methods Results Saliency extraction

Saliency extraction on natural images

Quickly obtain a gradual result

S. Chevallier (LIMSI - CNRS)

Neural Image Filtering

ICNC’09

16 / 21

Framework Neural image filtering

Methods Results Saliency extraction

Conclusion

I

Bio-inspired neural image filtering

I

Temporal processing

I

Good edge preservation

I

Results obtained gradually

Perspectives I

Mathematical study of neural filtering

I

Extended experimental comparisons

S. Chevallier (LIMSI - CNRS)

Neural Image Filtering

ICNC’09

17 / 21

Framework Neural image filtering

Methods Results Saliency extraction

Annex

S. Chevallier (LIMSI - CNRS)

Neural Image Filtering

ICNC’09

18 / 21

Framework Neural image filtering

Methods Results Saliency extraction

Input maps



dVi dt

= −λi Vi (t) + KLi , if Vi < ϑ else trigger a spike and Vi ← Vreset

with Li the considered pixel value Φi

  ˆti = − 1 ln 1 − λi ϑ λi KLi

=

λi  = −  iϑ ln 1 − λKL i

back

≈

S. Chevallier (LIMSI - CNRS)

1 ˆti

Neural Image Filtering

K Li ϑ

ICNC’09

19 / 21

Framework Neural image filtering

Methods Results Saliency extraction

Input maps

back

S. Chevallier (LIMSI - CNRS)

Neural Image Filtering

ICNC’09

19 / 21

Methods Results Saliency extraction

Framework Neural image filtering

Integration maps

(

P

dVj dt

j = −λj Vj (t) + ∑i=1 wij Si (t), if Vj < ϑ else trigger a spike and Vj ← Vreset

Ni

Si (t) =

∑ δ (t − tif ) f =1

back

S. Chevallier (LIMSI - CNRS)

Neural Image Filtering

ICNC’09

20 / 21

Methods Results Saliency extraction

Framework Neural image filtering

Integration maps

(

P

dVj dt

j = −λj Vj (t) + ∑i=1 wij Si (t), if Vj < ϑ else trigger a spike and Vj ← Vreset

Pj

Vj (t) =

Ni

∑ wij

i=1

j

f =1 Pj

Vj (Tj ) ≈

ˆi

∑ e−λ (t−f t ) H(t, f ˆti )

∑ wij

i=1

1 − e−QNi /Li 1 − e−Q/Li

avec Q =

λj ϑ K

back

S. Chevallier (LIMSI - CNRS)

Neural Image Filtering

ICNC’09

20 / 21

Methods Results Saliency extraction

Framework Neural image filtering

Frequency coding

P1 P2 P3 P4

V ϑ S t ISI

6 ms

S. Chevallier (LIMSI - CNRS)

4 ms

5 ms

Neural Image Filtering

5 ms

4 ms

ICNC’09

21 / 21