Reduce the search space [Tsotsos, 90]. Attentional architecture. Feature extraction. Combination on saliency map. Focus
Efficient neural models for visual attention Sylvain Chevallier, Nicolas Cuperlier and Philippe Gaussier ETIS - Neurocybernetic team Univ. Cergy-Pontoise – ENSEA – CNRS Cergy, France
[email protected]
September, 22th. 2010
Framework
Outline
1
Framework Visual attention Neural models
2
Models and implementation Attentional architecture Implementations
3
Experimental results
S. Chevallier (ETIS)
Efficient neural models
September, 22th. 2010
2 / 20
Framework
Visual attention
Change blindness
S. Chevallier (ETIS)
Efficient neural models
September, 22th. 2010
3 / 20
Framework
Visual attention
Change blindness
S. Chevallier (ETIS)
Efficient neural models
September, 22th. 2010
3 / 20
Framework
Visual attention
Change blindness
S. Chevallier (ETIS)
Efficient neural models
September, 22th. 2010
3 / 20
Framework
Visual attention
Bio-inspired attentional vision systems Attentional spotlight metaphor Reduce the search space [Tsotsos, 90] Attentional architecture Feature extraction Combination on saliency map Focus selection through Winner-Take-All
[Itti & Koch, 98]
Applications Driver assistance [Michalke, 08] Retinal prostheses [Parikh, 10] Robotics [Frintrop, 06] S. Chevallier (ETIS)
Efficient neural models
September, 22th. 2010
4 / 20
Framework
Neural models
Bio-inspired information coding Neurons exchange information through spikes
Spikes have little variations in amplitude and duration Spikes are fully characterized by their emission dates Level of description for neural models: Neuron level Temporal coding, precise spike timing Population level Rate coding, mean firing rate S. Chevallier (ETIS)
Efficient neural models
September, 22th. 2010
5 / 20
Framework
Neural models
Neural models
Spiking Neuron Network Network of [1, . . . , i, . . . , N] spiking neurons: ( P P (s) dVi j∈Pre wij s∈Trainj δ(t − tj ) + I(t), if Vi < ϑ dt = −λi Vi (t) + else trigger a spike and Vi ← Vreset Frequency-based Neural Network Continuum neural field τ
∂u (x, t) = −u(x, t) + ∂t
S. Chevallier (ETIS)
Z
w(x − x0 )f [u(x0 , t)]dx0 + I(x, t) + h
Efficient neural models
September, 22th. 2010
6 / 20
Framework
Neural models
Goal of this paper
Question What is the most suited neural coding scheme for an efficient bio-inspired attentional architecture ?
Compare SNN and FNN Complexity analysis Quality of results Simple artificial images Natural images
S. Chevallier (ETIS)
Efficient neural models
September, 22th. 2010
7 / 20
Models and implementation
Outline
1
Framework Visual attention Neural models
2
Models and implementation Attentional architecture Implementations
3
Experimental results
S. Chevallier (ETIS)
Efficient neural models
September, 22th. 2010
8 / 20
Models and implementation
Attentional architecture
Preattentive visual architecture IOR
Input image
Low spatial frequencies
WTA
Saliency Input maps
High spatial frequencies
Multi-scale Features
FNN needs WTA to sort saliencies
Contrast of luminance (DOG) Orientations (Gabor) Color opponencies (DOG) S. Chevallier (ETIS)
Efficient neural models
SNN is an anytime process September, 22th. 2010
9 / 20
Models and implementation
Implementations
SNN implementation
DOG filter
details
Neural filter S. Chevallier (ETIS)
Efficient neural models
September, 22th. 2010
10 / 20
Models and implementation
Implementations
Complexity analysis FNN Filtering cost: for f features, s spatial scales, filters of size M and N input image pixels WTA cost: O(N) with ARGMAX O(f × s × M × N) SNN Hybrid synchronous simulator, with time step ∆t Total cost = Spike propagation cost + neuron update cost cp × F × M × N + cu ×
A ∆t
F is mean firing rate, A is number of active neurons. cu is 10 FLOP. S. Chevallier (ETIS)
Efficient neural models
September, 22th. 2010
11 / 20
Models and implementation
Implementations
Complexity analysis SNN computational cost depends on emitted spikes Is the number of spikes constant for processing different images ?
CPU cycles (106 )
2.5
1 patch 10 patchs 50 patchs 100 patchs
2 1.5 1 0.5 0
0
10
20 30 40 Simulated time (t)
50
60
For SNN, computational cost depends on the input image Rich images (w.r.t chosen filters) induce large number of spikes S. Chevallier (ETIS)
Efficient neural models
September, 22th. 2010
12 / 20
Experimental results
Outline
1
Framework Visual attention Neural models
2
Models and implementation Attentional architecture Implementations
3
Experimental results
S. Chevallier (ETIS)
Efficient neural models
September, 22th. 2010
13 / 20
Experimental results
Comparison on artificial images Pop-out artificial images
FNN Circle shows most salient region, winner of WTA (FNN) SNN Dots indicate the most salient pixels (SNN) Same salient items are found for FNN and SNN (20 images) S. Chevallier (ETIS)
Efficient neural models
September, 22th. 2010
14 / 20
Experimental results
Natural images 19 webcam images of 160x120 pixels
Salient regions might not be extracted in the same order Measured computational cost (as CPU cycle): Constant for FNN SNN can find salient regions before FNN (1/4 of the images) S. Chevallier (ETIS)
Efficient neural models
September, 22th. 2010
15 / 20
Experimental results
Conclusion and perspective Comparison of two neural models for an attentional system Frequency-based Neural Network: have a constant and lower computational cost, needs a WTA to sort saliencies Spiking Neuron Network: have a variable computational cost have anytime capabilities Perspective Formal analysis of spiking neuron processing Learning capability of neural network Attentional bias modulating salient regions Long term adaptation of input signal (slow variation of illumination) S. Chevallier (ETIS)
Efficient neural models
September, 22th. 2010
16 / 20
Experimental results
Annex
S. Chevallier (ETIS)
Efficient neural models
September, 22th. 2010
17 / 20
Experimental results
Input maps
dVi dt
= −λi Vi (t) + KLi , if Vi < ϑ else trigger a spike and Vi ← Vreset
with Li the considered pixel value Φi =
1 λϑ ˆti = − ln 1 − i λi KLi
λi
= −
ln 1 −
back
≈
S. Chevallier (ETIS)
1 ˆti
Efficient neural models
λi ϑ KLi
K Li ϑ
September, 22th. 2010
18 / 20
Experimental results
Input maps
S. Chevallier (ETIS)
Efficient neural models
September, 22th. 2010
18 / 20
Experimental results
Integration maps
PPj = −λj Vj (t) + i=1 wij Si (t), if Vj < ϑ else trigger a spike and Vj ← Vreset dVj dt
Si (t) =
Ni X
δ(t − tif )
f =1
back
S. Chevallier (ETIS)
Efficient neural models
September, 22th. 2010
19 / 20
Experimental results
Integration maps
PPj = −λj Vj (t) + i=1 wij Si (t), if Vj < ϑ else trigger a spike and Vj ← Vreset dVj dt
Vj (t) =
Pj X
wij
Ni X
e−λj (t−fˆti ) H(t, f ˆti )
f =1
i=1
Vj (Tj ) ≈
Pj X i=1
wij
1 − e−QNi /Li 1 − e−Q/Li
with Q =
λj ϑ K
back
S. Chevallier (ETIS)
Efficient neural models
September, 22th. 2010
19 / 20
Experimental results
Frequency coding
P1 P2 P3 P4
V
ϑ S t ISI
S. Chevallier (ETIS)
6 ms
4 ms
5 ms
Efficient neural models
5 ms
4 ms
September, 22th. 2010
20 / 20