http://ykonline.yksd.com/distanceedcourses/Courses09/PhysicalScience/Lessons/ThirdQuarter/ ... Optical flow in computer
Computational Vision Motion processing
Motion as the most primitive form of vision
image sources: http://www.onekind.org/be_inspired/animals_a_z/zebra/ http://ykonline.yksd.com/distanceedcourses/Courses09/PhysicalScience/Lessons/ThirdQuarter/ Chapter07/07-01/48RabbitFreez.jpg
Motion perception: Navigation
Motion perception: Navigation
source: http://opticflow.bu.edu http://psych.hanover.edu/krantz/motionparallax/motionparallax.html
Motion perception:
Figure ground
Motion perception:
Structure from motion
Motion perception: Kinetic depth effect
Motion perception: Action recognition
more demos: http://www.biomotionlab.ca/?page_id=11
Motion as correspondence problem Apparent motion
see Ternus demo at http://www.michaelbach.de/ot/mot_Ternus/index.html
Motion as a correspondence problem
Optical flow in computer vision
image source: Militello et al 2014
Optical flow in computer vision Int J Comput Vis (2011) 92: 1–31
source: Baker et al 2011
11
Optical flow in computer vision • Challenge: Non-rigid scenes!
• Uses high-frequency fluorescent texture (UV light)
source: Baker et al 2011
Int J Comput V
Optical flow inetcomputer vision 612 D.J. Butler al.
Fig.et 1.alMPI-Sintel Flow data set. Sample ground truth flow fields and correspondsource: Butler 2012
Ventral vs. dorsal streams
Seeing without motion • Patient LM: 43 yr old, stroke with bilateral posterior parietal and occipital regions
• Complete loss of motion perception
• Comment from scientists that have worked with her for years [Zihl et al ’83]:
- She had difficulty for example, in pouring tea or coffee because the fluid appeared to be frozen like a glacier [...]
- In a room where more than two people were walking [...] she usually left the room because “people were suddenly here or there but [she had] not seen them moving”
Motion in non-primates
source: Lettvin et al 1959
© 2000 Nature
Detecting motion: The Reichardt detector
experiments, using the optomotor response of the beetle Cholorphanus as a behavioral measure. This response is the animal’s tendency to follow the movement of the visual surround to compensate for its mistaken perception of selfmotion in the opposite direction. The beetle was glued to a rod so it could not move its body, head or eyes relative to the surround, but could express its behavior at decision points by rotating a ‘Y-maze globe’ Fig. 1. Tethered Chlorounder its feet (Fig. 1). Their results 3 led to the phanus walking on the Ydevelopment of a model for maze globe (from ref. 10). motion detection that became known as the ‘correlation-type motion sequence of d detector’, the ‘Hassenstein-Reichardt two neighbo model’ or briefly—omitting half the origpreferred or n inal team—the ‘Reichardt detector’ low and Levi (Fig. 2). The core computation in this the null dire model is a delay-and-compare mechacantly reduce nism: delaying the brightness signal as individual res measured by one photoreceptor by a lowto the prefer roughly equ responses. T The author is at the ESPM-Division of Insect mechanism o Biolog y, 201 Wellman Hall, Univ. of California as the basis f Berkeley, Berkeley, California 94720, USA. e-mail:
[email protected] this study, the 1168
© 2000 Nature
experiments, using the optomotor response of the beetle Cholorphanus as a behavioral measure. This response is the animal’s tendency to follow the movement of the visual surround to compensate for its mistaken perception of selfmotion in the opposite direction. The beetle was glued to a rod so it could not move its body, head or eyes relative to the surround, but could express its behavior at decision points by rotating a ‘Y-maze globe’ Fig. 1. Tethered Chlorounder its feet (Fig. 1). Their results 3 led to the phanus walking on the Ydevelopment of a model for maze globe (from ref. 10). motion detection that became known as the ‘correlation-type motion sequence of d detector’, the ‘Hassenstein-Reichardt two neighbo model’ or briefly—omitting half the origpreferred or n inal team—the ‘Reichardt detector’ low and Levi (Fig. 2). The core computation in this the null dire model is a delay-and-compare mechacantly reduce nism: delaying the brightness signal as individual res measured by one photoreceptor by a lowto the prefer roughly equ responses. T The author is at the ESPM-Division of Insect mechanism o Biolog y, 201 Wellman Hall, Univ. of California as the basis f Berkeley, Berkeley, California 94720, USA. e-mail:
[email protected] this study, the
Detecting motion: The Reichardt detector (1961) 2 mm
photoreceptors
1168
source: Stone & Frisby (2010)
excitatory
(e.g., ‘x’, ‘+’)
inhibitory
(e.g., ‘/‘, ‘-‘)
Borst & Helmstaedter 2015
Borst & Helmstaedter 2015
Visual motion detection in the fly
Visual motion detection in the fly
Takemura et al 2013
Direction selectivity in V1
Hubel & Wiesel
Motion selectivity in the primary visual cortex
Shmuel & Grinvald ’96
Motion as orientation in space-time
/February 19852/February 1985 A/Vol. 2, No.
E. H. Adelson and J. R. Bergenand J. R. Bergen E. H. Adelson
in rather the frequency domain. n, than in the frequency domain. nce the between the sampled cit difference between the sampled oving we simply ns of bar. the If moving bar. If we simply from(Fig. the 4b) sampled s. 4b) pattern from the sampled patiotemporal plot of the plot of the derive a new spatiotemporal Fig. 4c. Since the 4c. differillustrated in Fig. Since the differer have displayed it ondisplayed a negative, we have it on a dsraytocorresponds zero, white to to zero, posi-white to posithat the sampled-motion ative. Observe that the sampled-motion to be the sum to of be thethe realsum of the real be considered Fig.artifacts 4c. That the of is, Fig.we4c.canThat is, we can continuous motion with motion with motion as being continuous o it. equently in time, the apmpled more frequently in time, the aps improved, i n as shown i n uous motion as is shown improved, ig. 4f) have(Fig. rather4f)little he artifacts have rather little trequencies we can see. sampling 5. a-e (x, t)Fig plots of bars to bars the left or totothe at to the right at thatIf we can see. If Fig sampling 5. a-e (x, t)moving plots of moving the right left or f, Motion is like orientation in (x, t), and ain spaplainly a point come at various speeds. f, Motion is like orientation (x, t), and a spagh, there come will plainly avarious point speeds. at tiotemporally oriented receptive field can be used to detect g, The tiotemporally oriented receptive field can be it.used to detect it. g, The ve so little energy the energy omponents have soinlittle in the same oriented receptive field can respond to can sampled motion just as motion just as same oriented receptive field respond to sampled ncy range that they will mporal-frequency range that itthey will to continuous responds it respondsmotion. to continuous motion. atiotemporal structure of ce the fine spatiotemporal structure of bility the spatial by andthe spatial and Adelson & Bergen ’85 urred tobyinvisibility Fig. 5g, it responds to a sampled version of theversion same of the same is continuous Fig. 5g,well it responds well to a sampled he point, eye. Atthethis point, the continuous
Motion as orientation in space-time Reichardt detector in space-time Smoother Reichardt detector excitatory inhibitory
first RF time
second RF
space
excitatory inhibitory
time
space
2nd neuron has a spatially separated Receptive Field (RF), and a shorter temporal delay Like an oriented V1 receptive field, but oriented in space-time!
if the outputs of a large number of receptors are are temporall filtered in the same way, and the filtered outputs are combine with a spatial weighting function, then again the net respons will be separable. Separability is frequently observed in th early stages of cortical visual processing.25,26 Figure 7 illustrates how the spatiotemporal impulse re sponse may be used to analyze the way a unit will respond t a stimulus. The stimulus here is a light dark edge that i initially stationary, then moves to the right, and then become responsestationary again. The stimulus may be considered to lie o a continuous strip, which is drawn upward over time, as show in Fig. 7a. A picture of the unit's impulse response is overlai on the stimulus, to show how inputs at all points and times ar
Building spatio-temporal RFs (LGN and V1 L4) Spatiotemporally separable impulse
DeValois et al ’00
Fig. 6. A spatiotemporally separable impulse response. The spati and temporal impulse responses are shown along the margins. The product is shown schematically in the center. The spatiotempor impulse re-sponse is a weighting function that sums inputs at variou positions and times to determine the present output.
Adelson & Bergen ’85
Spatiotemporally separable impulse response Spatiotemporally separable vs. inseparable impulse response 290
J. Opt. Soc. Am. A/February 1985
E. H. Adelson and J. R. Bergen
t x
Fig. 8. a, An (x, t) plot of an edge that is stationary, then moves sinusoidally, and then is stationary again. b, A separable spatiotemporal impulse response magnified four times for clarity. c, The convolution of a and b, i,e., the output of a separable channel. There is no selectivity for direction of motion. d The stimulus again. e, A spatiotemporally oriented Gabor function, magnified four times. f, The convolution of d and e. The output is strongly selective for rightward motion.
edge detector of Fig. 5f, this is not an easy unit t o build in a physiological system, but there are ways of approximating it that are physiologically plausible.
cussed how the oscillations might be used to advantage i n computing velocity.) A phase-independent motion detector can be built as shown in Fig. 9. We begin with two units that act as linear spa-
Adelson & Bergen ’85
A variety of receptive field types in the primary visual cortex • Two types of cells in V1:
- tuned to static bars and gratings
- tuned to drifting bars and gratings
7736 J. Neurosci., December 1, 1996, 16(23):7733–7741
proportion of cells
separable
inseparable
Movshon and Newsome • V1 Neurons Projecting to MT
directionality index
Ventral vs. dorsal streams
Spatiotemporally separable impulse response Spatiotemporally inseparable impulse response 290
J. Opt. Soc. Am. A/February 1985
E. H. Adelson and J. R. Bergen
sensitivity to phase
t x
Fig. 8. a, An (x, t) plot of an edge that is stationary, then moves sinusoidally, and then is stationary again. b, A separable spatiotemporal impulse response magnified four times for clarity. c, The convolution of a and b, i,e., the output of a separable channel. There is no selectivity for direction of motion. d The stimulus again. e, A spatiotemporally oriented Gabor function, magnified four times. f, The convolution of d and e. The output is strongly selective for rightward motion.
edge detector of Fig. 5f, this is not an easy unit t o build in a physiological system, but there are ways of approximating it that are physiologically plausible.
cussed how the oscillations might be used to advantage i n computing velocity.) A phase-independent motion detector can be built as shown in Fig. 9. We begin with two units that act as linear spa-
Adelson & Bergen ’85
compressive nonlinearity following the sum-of-squares stage, in order to keep the outputs within a reasonable range (cf. the research of Pantle and Sekuler28 on the motion aftereffect). Such a monotone transformation would not affect the basic properties of the motion-extraction process but could have an effect on the performance observed in various tasks, such as the accuracy with which changes in speed and contrast could be judged. Energy was extracted in Fig. 9a by using the standard trick
Extracting spatio-temporal motion energy • Goal: To build a RF that takes on a constant value for a constant motion
back to H&W
pooling over position and phase but otherwise identical tuning properties phase-independent measure of motion information
pha fun ver ture wit 6.
Wa orie out por clud to c one the for spa por a s par asy ma deg A and sep the T to sho (Fi this sec imp
wh abo app N and let
Frequency specific motion aftereffect and opponent channels
https://www.youtube.com/watch?v=OAVXHzAWS60
