ICSP'O4 Proceedings
Lucas-Kanade Algorithm with GNC M.'Junghans, A.'Leich €3.-J. Jentschel Dresden University of Technology;, Germany
[email protected] Abstract: The siinilarity of hvo arbitraly real functions can be analysed calculating the squared Euclidean distance, the paraineter space. The application of this method to digital images was proposed by Lucos and Knnade. In this paper the concept of Grachated Non-Comexi~(GNC)is applied to the probIein of evaluating the p a m e t e r function. It is s l ~ that ~ ntlte application of GNC to the parameter function and the construction and evaluation of a pyramid of spatially smooth and sub-sampled iinages are equivalent operations in particdar.'but practically relevant cases. Key words: Lucas-Kanade, GNC, Coarse-to-fine. Euclidean distance
The calculation of the square of the Euclidean distance is a method to analyse the similarity of two arbitmy functions. This method is often applied to calculate optimal values for a set of parameters. wluch characterise the transformation of one function towards another. The application of tlus method to digital images and the determination of optimal parameters with the Gauss-Newton method is known as Lucos-Konode algorithm [4]. Another approach is to apply the concept o f Gru~iiatrd.Non-Co~exirv (GNC) [3] to evaluate the parameter space. Tliereby the parameter function is srnootled and global optimal values for lhe parameters are detenniiied iteratively. On the basis of regismtion problems in digital iinages it is shown that the GNC-concept and h e construction of a pyramid of spatially smooth and sampled iniages are equivalent operations. Tlie paper is structured as fotlows: In section 2 the calculation of the squared Euclidean distance is introduced. I1 will be shown that the cross Correlation function can be derived from the squared Euclidean distance for pafiicular cases. Subsequently. in section 3. the concept GNC is applied lo the pameter function and afterwards, to registration problems in image processing in section 4. The equivalent. connection between the GNC o in every step. Since convexity is necessary but not sufficient for the convergence of the Newton method. we controlled the stepsize of each Newton step.
Without loss of genenliiy other efficient minimisation methods can be applied. for instance the Lmenberg-Murquardf algorithm [l]. .
4
Registration Technique with GNC
Recall the two functions f and g! the similarity o f which can be analysed. We want to apply the squared Euclidean distance in connection with GNC to registration problems in image processing for images of the size M x N . The function f (2,y) is repmented by a discrete grayvalue function of an image of a Video sequence. And g (z, y, $) is the chronologically successive image containing displaced objects in comparison to f . The parameter vector (F' = (Ax, characterises the global optimal displacement of the pixels in image g towards image f. In Section 3 it was shown, that the cross correlation function can be derived from the Euclidean error norm if the vector $contains soleiy translational parameters and if the functions f and g are integrated within (-00, +CO). In contrast. real images are the result of windowing the "real w o r l d with a rectangle function o f the size M x N. The simplification shown in (3) can be applied to real inxiges too, if the following conditions are fdfilled:
E,
1. The image energy E, g2 ( r c , y, $1 must be approximately constant in dependence on the parameter vector ip". This god can be achieved, if the size of the image is large.
2. Furthennore it must be ensured, that no moving object in the image leaves the border of the image. i.e. the displacementvector must not point out of the image. Thus, the correlated images must be fitted according to the interval [Gmtn, &,ax]. wi-uch decreases the size of the image. Considering these conditions, the th~rdterm in (3) can 6e neglected in discrete real images too. Then we obtain M
N
3'= 1
y= I
as the negative cross correlation function of the two images with regard to the displacement vector ip"= (AX, TO apply GNC to the negative cross correlation function (61, f is decomposed into sub-functions, each of which containing exactly one of the pixels of f , beginning with the first. The rest of the sub-function is evaluated to zero. The pth
&
A
1380 1360
1340
1320
Figure 1 : Example of GNC: The two Gaussians in function g (dashed line) on the left side can be displaced differently towards function f (solid line). One is shifted by Azl = 1. the other one by A22 = 5. The parameter functions for different stages of GNC on the right picture show that there is one local ininiinm first. Lowering the smoothing parameter CJ results in two local minima in the paiameter Eunction. The minimum $51of stage 0 inoves towards the global minimum $3 in stage 3 sub-function is referred to as f , (2,9 ) . AIlke. the parameter function I I (d)can be decoinposed into sub-functions (1,' each of which is the result of the computation of the m o r metric for fF, (.I;. y ) and y ( x , y. 3):
AfxN
M
A'
(c).
(7)
AfxN
8
a.p ($3).
a ($3)= p= 1
As specified in (4}. the sub-function up (U') can be convolved with the smoothing kernel G (& a )to obtain smooth
sub-functions E!, (+! a).Since convolution is a linear operation. (9) can be derived froin (8). An interpretation of these equations is. that the smoothed pammeter function is a sum of smoothed sub-functions.
AlxN M
N
p=l z=ly=l Ill x N
(9) p=I
Function fi, is constant in respect of the parameter vector j?. so that (9) can be sinipItficd in the following manner:
p= 1
*=ly=l
As it can be seen in (10).smoothing in parameter space has solely to be applied to the function g ; there is no must to smooth the cqrrelation function in the parameter space. On the other hand, smoothing of solely function g in the pam e t e r space is not possible, because the parameter space is the result of the comparison of Uie two functions f and g. Though. it would be desirable to find a way to allow smoothing in the original region instead the parameter space. Fortunately. the ZD paraiiieter s ace is spanned by the displacement vector (p' = (Ax, Ay) . whcli characterises the spatial clunges of die locations of moving objects in the image. Consequently. these smooth spatial changes result in sinooth spatial images. Vice versa. smoothing the image spatially results in a smootlied p m n e t e r function implicitly. On this account. spatially sniooth images are sufficient for a smooth parameter space. In (10) only the function g Ius to be smoothed. Since spatial smoothing is not dependent on the displacement vector $. either image f or g need to be smoothed. Here, we sniootli fmction g. of which the smoothversion is tenned with g ~But. . if spatial smoothing is also sufficient for an implicit smootlung of the parameter space in case of arbit- parameters. has not been proved yet and hto be ipestigated. The sampling theorem allows to sub-sample sufficient smooth functions witliout loss of information. Furthermore. a coarse-to-fine method can be applied to construct a pyramid of spatially smooth and sampled images to cope with motions larger than half of the wavelength of a periodic pattern in the image, usually one pixel per frame [Z, 41. Beginning with the lowest spatial resolution the global optiinal estimation for (p' in the highest level of the pyramid can be obtained by d e t e n i h t i o n of the global minimum. Then. the result is projected (warped) towards the next lower level in the pyramid as starting value for the nest minimisation and projection. This process is repeated until the global o p timal solution has been computed at highest resolution of y
1090
.
Figure 2: Example for the calculation of the pymmid of spatially smooth and sub-sampled polygon lines: (a) A polygon line is placed on a lane of tlic road. Tlieir grayvalues represent the two functions f and y to be correlated. (b)The corresponding p r m i d cliaracterises the smootli and sampled versions of g. (c) The pyramid of associated correlation functions E (As:a).
in the lowest level of the pyramid. Due to tlie fact. that spatial smoothing of tlie images is equivalent to sinootlung the parameter function in parameter space in the case of translational parameters.*theconstruction and evaluation of a pyramid of spatially smooth and sub-sampled images is equivalent to the application of GNC to the parameter function Consequently. both possibilities are idcntical operations. In tlie following section the coarse-to-fine method is applied to the images of a tmffic video sequence.
5
'
Smootlmg the vector g results in the smootlivectorg~;.The smooth parameter function, of which the global mniinuin is to be determined. is obtained: N
& ( A ~ , o=) - C ~ ( S ) ~ ~ ( S , A S . ~ ) p= 1
6
In this paper a LmwKunnde algorithm with application of Graduated Non-Cnmexir?, (GNC) to the cross correlation function derived by the squared Euclidean distance *as introduced. It was shown, that for particular cases with practical relevance the coarse-to-fine method and the GNCconcept are equivalent operations. e.g. tlie determination of the mean velocity of vehicles on the road. The equivalence was due to the fact, that smoothing of the parameter space in the case of translational paratneters and spatial smootbing and sub-sampling of images are identically. Thicrc arc at lcast two opcn qucstions left. which characterise our future work. First. whether the application of a spatial coarse-to-fine method to images is equivalent to a parametric smoothing in parameter space for arbitrary parameters, e.g. affine motion. And second, whether the application of robust error functions instead.of the La-Norm 10 the GNC-concept is possible and leads to similar results.
Mean Motion Detection in Traffic Scenes
There are seven1 methods for motion detection of vehicles in video traffic scenes. wlucli can be applied using the options of inmge processing. One method is to place a detection area on a road in the image in wlucli tlie grayvalues are correlated. In tlus example the detection area is represented by a line with the length Ar {we Figure 2). Tlle two functions j' and 9 can now be understood as grayvalue vectors of the polygon lines of two chronological images. The N-dimensional grayvalue vectos f and g of the two chronological images represent the vehicles in the detection area in space and time. assuming the changes of the graydues depend on the displacement of the vehicles in the video sequence mainly. In th~scase (2) can be simplified in the same way as shown in (6) to (8) and the Esulting negative correlation function can be smoothed and analysed by the application of the GNC-concept as Shown in (9) and (10). Since the GNC-concept and the constmcl tion of a pyramid of smooth and sampled images are equivalent operations. the global niiniinum of the negative cross correlation function c m be detemuned level-wise frotn low lo high resolution of the grayvalue lines. Thereby tlie parameter vector is reduced to a parametric scalar s = s (x,g). since the velocity can only be determined along the line, so tlut +'=As. We derive the negative cross correlation function
Conclusion and Future Prospects
7 References
I-
N
AS) = - E f ( s ) g ( ~ . A s ) s= 1
1091
[ l ] S. Baker, I. Matthews, Lucas-Kailade 20 Years On: A Unifying Framework: Part 1. Technical report CMU-RI-TR02- 16, Robotics Institute, Camegie MeIlon'University. July x. 7002 [2'j M..I.Black, P.Anandan. The Robust Estimation of Mdtiple Motions: Paraintetric and Piecewise-Smooth Flow Fields. Campiifer Vision arid Inloge - Understaizdiiig - vol. 63. pp. . . 75104.-1996
I31 A. Blake. A . Zisserman. I6suol Recunstritcfioir. The MIT Press Series in Artificial Intelligence. Cambridge 1987 [41 B.D. Lucas, T. Kanade. An iterative Image Registration Technique with an Application to Stereo Vision. Proceedings 7fIrJoinr Corrference 011 Artificial Itjtelligence, pp. 674679. Vancouver 1981
