IMAGE REGISTRATION FOR PERSPECTIVE DEFORMATION RECOVERY George Wolberg Siavash Zokai Department of Computer Science City College of New York New York, NY 10031 fwolberg|
[email protected]
Keywords: affine transformation, nonlinear least-squares, mosaics, perspective transformation, registration
ABSTRACT This paper describes a hierarchical image registration algorithm to infer the perspective transformation that best matches a pair of images. This work estimates the perspective parameters by approximating the transformation to be piecewise affine. We demonstrate the process by subdividing a reference image into tiles and applying affine registration to match them in the target image. The affine parameters are computed iteratively in a coarse-to-fine hierarchical framework using a variation of the Levenberg-Marquadt nonlinear least squares optimization method. This approach yields a robust solution that precisely registers image tiles with subpixel accuracy. The corresponding image tiles are used to estimate a global perspective transformation. We demonstrate this approach on pairs of digital images subjected to large perspective deformation.
1. INTRODUCTION Image registration refers to the geometric alignment of a set of images. The set may consist of two or more digital images taken of a single scene at different times, from different sensors, or from different viewpoints. The goal of registration is to establish geometric correspondence between the images so that they may be transformed, compared, and analyzed in a common reference frame. This is of practical importance in many fields, including remote sensing, medical imaging, and computer vision. Registration is often necessary for (1) integrating information taken from different sensors (i.e., multisensor data fusion), (2) finding changes in images taken at different times or under different conditions, (3) inferring three-dimensional information from images in which either the camera or the objects in the scene have moved, and (4) for model-based object recognition.1 Common tasks associated with image registration include generating large panoramic images (image mosaics) from several overlapping images, producing super-resolution images from multiple images of the same scene, change detection, topographic mapping, and multisensor image fusion. This paper describes a hierarchical image registration system based on parameter estimation techniques. We consider pairs of digital images that were acquired under a weak perspective camera model or a fixed center of projection. Both constraints yield images that are free of parallax. This is necessary to facilitate the registration of an image pair by a global perspective transformation. There are three classes of images that we can register with this method: 1. images of any 3-D scene taken at large distances, e.g., high altitude surveillance imagery. 2. images of any 3-D scene taken at any distance but from a fixed center of projection, e.g., any set of images acquired from a (swiveling) camera on a tripod. 3. images of flat objects taken at any distance. Perspective transformations introduce space-variant scale changes to an image. This makes the solution to general perspective registration very difficult. We circumvent this problem by approximating perspective transformations to be piecewise affine. That is, a reference image is subdivided into tiles and affine registration is applied to find the best matching tiles in the target image. These tile pairs are used to estimate a global perspective transformation. This approach permits us to exploit recent results in affine registration to solve the more difficult perspective registration problem. Appears in Proc. SPIE, Automatic Target Recognition X, Orlando, FL, April 2000. This work was supported by a NASA FAR Award (NAG-57129).
1
There is a vast literature of work in the related fields of image registration, motion estimation, image mosaics, and video indexing. Most algorithms exploit a hierarchical approach due to computational efficiency in handling large displacements. Hierarchical motion estimation2–4 and image mosaic5–12 algorithms usually assume small deformations among image pairs. For instance, a dense image sequence is required to stitch the frames together.8,11 The problem of assembling a large set of images into a common reference frame is simplified when the inter-frame deformations are small. Our work addresses the problem of registering two input images in the presence of large deformations. We make use of hierarchical affine registration 13 to infer perspective. Although existing hierarchical affine registration work attempts to find local deformations,14 we infer perspective transformations since they are a predominant deformation model found in a large class of images. In Section 2, we review deformation models. Section 3 describes a nonlinear least-squares technique based on the Levenberg-Marquadt algorithm to solve the affine registration problem. Section 4 discusses how to assemble the results of the affine registration stage to infer the perspective parameters. Finally, Section 5 presents a summary and conclusions.
2. DEFORMATION MODELS A deformation model is necessary to define the class of spatial transformations that may apply between any two images, I1 and I2 , of the same scene. This defines a geometric relationship between each point in both images. The general form for the mapping function induced by the deformation is
x; y] = [X (u; v); Y (u; v)] (1) where [u; v] and [x; y] denote corresponding pixels in I1 and I2 , respectively, and X and Y , are arbitrary mapping functions that uniquely specify the spatial transformation. In registering I1 and I2 , we shall be interested in recovering the inverse mapping functions U and V that transform I2 back into I1 : [u; v ] = [U (x; y ); V (x; y )] (2) [
In this sense, the registration process is akin to geometric correction, whereby we attempt invert the distortion. The choice of deformation model must be closely tied to the imaging process by which the input data set was acquired. The complexity of the model dictates the choice of registration algorithm. Images which are known to differ by small translations, for example, require only a simple correlation technique to perform registration. Images which differ by elastic deformations, though, may require user-supplied correspondence points. These points, also known as control, fiducial, and tie points, are marked by the user at sparse and irregular landmark sites on the images. Since the mapping function is precisely known at only those points, a scattered data interpolation algorithm must be applied to smoothly propagate the mapping to all other points in the image. Note that in the former case the mapping function was derived directly from the image intensities, while in the latter case the mapping function was constructed from sparse correspondence points. Geometric correspondence is achieved by determining the mapping function that governs the relationship of all points among a pair of images. There are several common mapping function models in image registration. They include (1) 3-parameter rigid transformations (translation, rotation), (2) 6-parameter affine transformations (translation, rotation, scale, shear), (3) 8-parameter perspective transformations, and (4) local nonrigid transformations. In this paper, we shall address the problem of registering images misaligned due to a perspective transformation. The mapping function is given as
u
=
v
=
a1 x + a2 y + a3 a7 x + a 8 y + 1 a4 x + a5 y + a6 a7 x + a 8 y + 1
(3a) (3b)
The eight parameters that govern the transformation of observed image I2 into reference image I1 will be estimated by approximating perspective to be piecewise affine. This will permit us to use a robust parameter estimation algorithm to recover the six unknown parameters relating each pair of image tiles in I1 and I2 .
An affine transformation considers only the numerators in Eq. (3), i.e., a7 = a8 = 0. The affine parameters are estimated by minimizing the sum of squared differences between the registered tiles. A least squares method is applied to the collection of affine parameters recovered over all tile pairs in order to infer the eight parameters of a global perspective transformation.
2
3. AFFINE PARAMETER ESTIMATION In this section, we demonstrate how to determine the affine parameters using nonlinear least-squares optimization. We use the sum of squared differences (SSD) as the objective criterion that establishes a similarity measure between two images (or regions): Z Z
(a) 2
=
uZRZ
?
I1 (u) ? I20 (u) 2 du
?
I1 (u) ? TA fI2 (x)g 2 du
2
=
uR kI1 (u) ? TA fI2 (x)gk2 2
=
(4)
where T is a geometric transformation applied to observed image I2 to map it from its [x; y] coordinate system to the [u; v] coordinate system of reference image I2 . The subscript A denotes that this transformation is affine. For discrete data, we have
2(a)
=
=
=
N X i=1
N X i=1
N X i=1
I1 (ui ) ? I20 (ui ) 2
I1 (ui ) ? TA fI2 (xi )g 2
I1 (ui ; vi) ? I2 (a1 xi + a2yi + a3 ; a4xi + a5 yi + a6) 2
(5)
3.1. Newton-Raphson The minimum of a function occurs at points where the first derivatives of the function vanishes to zero. We now solve for:
Bk (a) = @ @a(a) = ?2 2
k
N X i=1
20 (ui) I1 (ui) ? I20 (ui) @I@a =0 k
(6)
where k = 1; 2; :::; 6. This gives us a system of six nonlinear equations:
Bk (a) = 0 k = 1; 2; :::; 6
(7)
The linear part of the Taylor series gives us:
Bk (a + ∆a) Bk (a) +
@Bk (a) ∆a @al l l=1
6 X
(8)
Six unknown variables and six linear equations gives us:
B(a + ∆a) = B(a) + H(a)∆a
(9)
a + ∆a) is the solution, then B(a + ∆a) = 0 and we have: H(a)∆a = ?B(a) H(a) is a 6 6 matrix and is known as the Hessian matrix: N 0 2 2 X @I2 (ui) @I20 (ui) ? I (u ) ? I 0 (u ) @ 2 I20 (ui) hkl = @Bk (a) = @ (a) = 2 i i 1 If (
@al Due to symmetry, hkl = hlk .
@ak @al
i=1
@ak
@al
a
2
@ak @al
(10)
(11)
We then solve for six values in ∆ by Gauss elimination. The advantage of this method is fast convergence near the solution. Its disadvantages include difficulty in finding a good initial estimate, and possibly slow convergence or oscillations if the function is not well-behaved. 3
3.2. Steepest (Gradient) Descent In any iterative minimization, the (i + 1)st iterate is related to the ith iterate by the equation:
ai
+1
a
a a C
=
ai + ∆ai
(12)
a
We call f i+1 g a descending sequence if 2 ( i+1 ) < 2 ( i ). One direction that surely produces a descent is the direction of the negative gradient. Therefore, i+1 = i ? r 2 ( ), where C controls the step size along each parameter in a.
a
a
a
∆ i
= = =
ai 1 ? ai ?Cr 2 (a) ?CB(a) +
(13)
An advantage of this method is that it always converges to a (local) minimum. A disadvantage, though, is its slow convergence.
3.3. Levenberg-Marquadt Algorithm This method is based on the combination of the Newton-Raphson method and the steepest descent method. From the NewtonRaphson method, we have ( )∆ = ? ( ). From the steepest descent method, we have ∆ = ? ( ). If we let 1 2 15 k = hkk , then hkk ∆ = ? ( ). Marquadt proved that the following hybrid method minimizes ( ) more robustly.
C
a
Ha a Ba
Ba
a
a
CB a
H(a) + I)∆a = ?B(a) (14) Parameter 0 controls the extent to which ∆a conforms to either the Newton-Raphson or steepest descent methods. If the previous update succeeds in reducing 2 (a), then is reduced to adopt the Newton-Raphson update. Otherwise, is increased (
to adopt a steepest descent update. The resulting method, known as the Levenberg-Marquardt algorithm (LMA), is given below. The algorithm requires threshold values T1 and T2 to specify stop conditions. The algorithm terminates when the change in ∆ 2 falls below T1 or rises above T2 . Levenberg-Marquadt Algorithm begin Initialize parameters to the identity matrix Initialize with a modest value, e.g., = 0:1 while (j∆ 2 ( )j > T1 or < T2 ) do Compute the 6 6 Hessian matrix H Apply affine transformation on I2 Compute vector Solve linear equations ∆ = ? for ∆ Evaluate 2 ( + ∆ ) if 2 ( + ∆ ) < 2 ( ) then do =10 +∆ else do 10 end if end while end
a
B
a a a a a a a a
Ha
B
a
3.4. Example Fig. 1 demonstrates affine registration. Image pair I1 and I2 are shown in Figs. 1(a) and 1(b), respectively. Affine registration estimates the six affine parameters necessary to transform I2 onto I1 . Fig. 1(c) shows the overlay of I1 and the transformed I2 . Minimal double-exposure (or ghosting) effects illustrate that the image pair is registered with high fidelity. 4
(a) image I1
(b) image I2
(c) overlay of registered images
Figure 1. Affine registration.
4. PERSPECTIVE PARAMETER ESTIMATION In this section, we extend the results of affine registration to handle perspective transformations. Fig. 2 demonstrates perspective registration. Image pair I1 and I2 are shown in Figs. 2(a) and 2(b), respectively. Perspective registration estimates the eight parameters necessary to transform I2 onto I1 . Fig. 2(c) shows the overlay of I1 and the transformed I2 . Minimal double-exposure (or ghosting) effects illustrate that the image pair is registered with high fidelity.
(a) image I1
(b) image I2
(c) overlay of registered images
Figure 2. Perspective registration. This method cannot, in general, register images subjected to large perspective distortion. The primary difficulty is that the initial guess may not lie close to the solution and the minimization procedure may get stuck in a local minima. This problem is more severe for perspective than affine because perspective transformations are nonlinear. We propose to circumvent this problem by approximating perspective to be piecewise affine. In this manner, we leverage the robust affine registration algorithm to handle the more difficult perspective registration problem. A local affine approximation is suggested by expanding the perspective transformation function (Eq. (3)) about a point using a first-order Taylor series. The approximation holds in a small neighborhood about point (x0 ; y0 ):
u
= = =
U (x; y) x0 ; y0) (x ? x ) + @U (x0 ; y0) (y ? y ) U (x0; y0) + @U (@x 0 0 @y A1 x + A 2 y + A 3 5
(15)
where A1
=
@U (x0;y0 ) , A 2 @x
=
@U (x0 ;y0 ) , and A 3 @y
U (x0; y0 ) ? A1x0 ? A2 y0 . The expression for v is
V (x; y) @V (x0; y0 ) (x ? x ) + @V (x0 ; y0) (y ? y ) = V (x0 ; y0 ) + 0 0 @x @y (16) = A4 x + A 5 y + A 6 x ;y ) , A = @V (x ;y ) , and A = V (x ; y ) ? A x ? A y . Notice that the expressions for u and v match the where A4 = @V (@x 5 6 0 0 4 0 5 0 @y 0
v
=
0
=
0
0
linear form required of affine transformations.
In the next two sections, we review the process of determining the affine transformation for a four-corner (tile-to-tile) mapping and its use in inferring perspective parameters. We have already reviewed how to perform affine registration on a rectangular image domain, e.g., tile. For the purpose of demonstrating the feasibility of piecewise affine approximation, we will show how to infer the affine parameters given the correspondence of the tile corners. This permits us to analyze the proposed technique without concern for errors that may be due to the affine registration process using the LMA algorithm described in Section 3.
4.1. Inferring Affine Parameters
Consider a tile in I1 and its corresponding quadrilateral in I2 . (Fig. 3). P’ 2 P1
P2
P’ 1
P’ 3
P4
P’ 4
P3
Figure 3. Four corner mapping. Although a perspective transformation can fully account for this four corner mapping, we are interested in finding the best affine transformation that approximates this mapping. Given the four corners of a tile in observed image I2 and their correspondences in reference image I1 , we may solve for the best affine fit by using the least squares approach. Let the affine mapping be given as:
u v
a1 x + a2 y + a3 = a4 x + a5 y + a6 We solve for the affine parameters by minimizing the expression for 2 below. 2 (a)
=
4 X
i=1
ui ? a1 xi ? a2yi ? a3 )2 + (vi ? a4 xi ? a5 yi ? a6 )2
(
We may relate these correspondences in the form U 2
3
=
2
=
WA: y1 1 y2 1 y3 1 y4 1
(17a) (17b)
(18)
3
u1 x1 0 0 0 2 3 6 u2 7 6 x2 0 0 0 7 6 7 6 7 a1 6 6 u3 7 6 x3 7 0 0 0 7 6 7 6 7 6 a2 7 6 6 u4 7 6 x4 7 0 0 0 7 6 a3 7 6 7 6 7 6 6 v1 7 = 6 0 7 0 0 x1 y1 1 7 6 7 6 7 6 a4 7 6 v2 7 6 0 7 4 0 0 x2 y2 1 7 a5 5 6 7 6 4 v3 5 4 0 0 0 x3 y3 1 5 a6 v4 0 0 0 x4 y4 1 The pseudoinverse solution A = (W T W )?1 W T U is computed to solve for the six affine coefficients. 6
(19)
4.2. Inferring Perspective Parameters The affine parameters computed in Eq. (19) apply to a single tile pair. We must infer the affine parameters for each of the N tile pairs in I1 and I2 , and apply them to all tile centers. This establishes N point correspondences between I1 and I2 , The eight unknown perspective parameters can be inferred by solving the resulting system of equations. Again, we may relate the correspondences in the form U = WA by rewriting Eq. (3) in the form
u v
a1 x + a2 y + a3 ? a7ux ? a8uy a4 x + a5 y + a6 ? a7vx ? a8 vy
= =
(20a) (20b)
This yields the following overdetermined system of equations. 2
ui0
3
2
6 7 6 7 6 .. 7 6 . 7 6 7 6 7 6 7 6 i 7 6 0 7 6 7 4 5
6 6 6 6 6 =6 6 6 0 6 6 4
v
.. .
x0i y0i
1
0
0
xi0 y0i
0
7 7 7 7 7 7 7 i i 7 0 0 7 7 5
6 6 6 6 6 6 6 6 6 6 4
1
?v0i x0i ?v y
.. .
2N 1
2
?ui0 x0i ?ui0y0i
.. . 0
3
0
2N 8
where (ui0 ; v0i ) and (xi0 ; y0i ) denote the centers of tile i in I1 and I2 for 1 W T W )?1 W T U is computed to solve for the eight perspective coefficients.
(
i N.
a1 a2 a3 a4 a5 a6 a7 a8
3 7 7 7 7 7 7 7 7 7 7 5
(21)
8 1
The pseudoinverse solution
A
=
We may augment these results by including additional constraints in the least squares estimation. In addition to considering the correspondence of tile centers, we may also consider the local affine parameter estimates. From Eqs. (15) and (16), we know that the affine approximation at a tile is given as
u v
= =
A1 x + A2 y + A3 A4 x + A5 y + A6
(22a) (22b)
where
A1
= =
A2
= =
A3
= = =
A4
= =
A5
= =
A6
= = =
@U (x0 ; y0 ) = a1 (a7x0 + a8 y0 + 1) ? a7 (a1x0 + a2 y0 + a3) = a1 ? a7 u0 @x (a7 x0 + a8 y0 + 1)2 a7x0 + a8 y0 + 1 a1 ? a7(A1 x0 + u0 ) ? a8A1 y0 @U (x0 ; y0 ) = a2 (a7x0 + a8 y0 + 1) ? a8 (a1x0 + a2 y0 + a3) = a2 ? a8 u0 @y (a7 x0 + a8 y0 + 1)2 a7x0 + a8 y0 + 1 a2 ? a7A2 x0 ? a8 (A2 y0 + u0 ) U (x0 ; y0) ? A1 x0 ? A2 y0 = aa1xx0 ++aa2yy0 ++a13 ? A1 x0 ? A2 y0 7 0 8 0 a1 x0 + a2 y0 + a3 ? a7(A1 x20 + A2 x0y0 + A3 x0) ? a8(A1 x0 y0 + A2 y02 + A3 y0 ) ? (A1 x0 + A2y0 ) a1 x0 + a2 y0 + a3 ? a7u0 x0 ? a8u0 y0 ? (u0 ? A3 ) @V (x0 ; y0) = a4(a7 x0 + a8y0 + 1) ? a7(a4 x0 + a5 y0 + a6 ) = a4 ? a7v0 @x (a7 x0 + a8 y0 + 1)2 a7 x0 + a8y0 + 1 a4 ? a7(A4 x0 + v0 ) ? a8A4 y0 @V (x0 ; y0) = a5(a7 x0 + a8y0 + 1) ? a8(a4 x0 + a5 y0 + a6 ) = a5 ? a8v0 @y (a7 x0 + a8 y0 + 1)2 a7 x0 + a8y0 + 1 a5 ? a7A5 x0 ? a8 (A5 y0 + v0 ) V (x0; y0 ) ? A4x0 ? A5 y0 = aa4xx0 ++aa5 yy0 ++a16 ? A4x0 ? A5 y0 7 0 8 0 a4 x0 + a5 y0 + a6 ? a7(A4 x20 + A5 x0y0 + A6 x0) ? a8(A4 x0 y0 + A5 y02 + A6 y0 ) ? (A4 x0 + A5y0 ) a4 x0 + a5 y0 + a6 ? a7v0 x0 ? a8 v0y0 ? (v0 ? A6) 7
(23a)
(23b)
(23c)
(23d)
(23e)
(23f)
A3 and A6 actually produce equations of the form u0 = a1 x0 + a2 y0 + a3 ? a7 u0x0 ? a8u0 y0 and v0 = a4 x0 + a5y0 + a6 ? a7v0 x0 ? a8v0 y0 , respectively. These are the same point correspondence relations defined in Eq. (21). Notice that the terms for
We may collect the expressions given above to yield the following system of equations that relates the affine parameters at all N tiles with the unknown parameters of the global perspective transformation. 2 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 4
A1i
2
3
.. 7 . 7 A2i 777 .. 7 . 7 7 u0i 77 .. 7 . 7 7 A4i 77 .. 7 . 7 7 A5i 77 .. 7 . 7 7 v0i 75 .. . 6 N 1
6 6 6 6 6 6 6 6 6 6 6 6 =6 6 6 6 6 6 6 6 6 6 6 6 4
1 0
0
0
0
0
0 .. .
1
0
0
0
x0i y0i
0 .. .
1
0
0
0 .. .
0
0
0
1
0
0 .. .
0
0
0
0
1
0 .. .
0
0
0
x0i y0i
1 .. .
?(Ai1 xi0 + u0i ) ?Ai2 x0i ?ui0 x0i ?(Ai4 x0i + v0i ) ?Ai5 x0i ?v0i xi0
3
?A1i y0i
7 7 7 ( 2i 0i + i0 ) 7 7 7 7 7 i i 7 0 0 7 7 7 7 i i 7 4 0 7 7 7 7 i i i ( 5 0 + 0) 7 7 7 7 7 i i 7 0 0 5
?Ay
u
2 6 6 6 6 6 6 6 6 6 6 4
?u y
?A y
?Ay
v
a1 a2 a3 a4 a5 a6 a7 a8
3 7 7 7 7 7 7 7 7 7 7 5
(24)
8 1
?v y
6N 8
where (u0i ; v0i ) and (xi0 ; y0i ) denote the centers of tile i in I1 and I2 for 1 i N . Note that Eq. (24) is a superset of Eq. (21). Again, the pseudoinverse solution is computed to solve for the eight perspective parameters. The solution to Eq. (24) is normally not stable if pixel coordinates are directly used. We therefore perform a simple normalization to bring the pixel coordinates into the [0,1] range.
4.3. Piecewise Affine Approximation Consider an observed image I2 subdivided into a regular grid of tiles. For each tile T2i , we perform affine registration using the Levenberg-Marquadt algorithm to search for the most similar tile T1i in reference image I1 . This yields a collection of affine parameters and tile centers. Note that all tiles must share the same dimensions in order to compute 2 . The algorithm is outlined below. Piecewise Affine Approximation Algorithm begin Partition I2 into a collection of N tiles: T2i , for 1 i 1 while (i < N ) do Select tile T i
iN
2
for all positions (x; y) in I1 do Crop tile T1i in I1 Register T2i with T1i using LMA if 2 ( ) is minimum then do xi x yi y i
a
a
a
end if end for i i+1 end while Solve Eq. (21) or Eq. (24) for perspective parameters using pseudoinverse solution (linear least squares) end
8
4.4. Examples
Fig. 4 demonstrates the proposed algorithm. The top row of the figure depicts an image subdivided into a uniform grid of 2 2, 4 4, and 8 8 tiles, respectively. We consider these three resolutions to demonstrate the relationship between tile size and approximation error. The second row shows the effects of a perspective warp applied to the input images. The overlaid grids and the marked tiles centers clearly illustrate the foreshortening effects of perspective. The piecewise affine approximation of a perspective transformation is shown in the third row. Notice that each input tile has undergone an affine warp that best approximates the four corner mapping to its corresponding tile in the warped image. The true perspective grids (second row) are superimposed to illustrate the error with respect to the ground truth. The positions of the tile centers from the input (first row) and piecewise affine images (third row) are supplied to Eq. (21) to infer a global perspective transformation. The results are shown in the fourth row, with the true perspective grids (and tile centers) overlaid on the figure. The mapping is due to the perspective transformation inferred directly from the correspondences of the tile centers. The last row of the figure shows the improved approximation due to Eq. (24), when the local affine parameter estimates are used alongside the point correspondences.
The Taylor series approximation is adequate in only a small neighborhood around a point, i.e., tile center. Clearly, a 2 2 tile subdivision is too coarse, as depicted in the first column of Fig. 4. Finer subdivision into 16 and 64 tiles, as shown in the second and third columns of the figure, produce greatly improved results. The use of small tiles thus enforces the assumption implicit in the Taylor series, and thereby helps reduce errors. Fig. 5 demonstrates the algorithm on a pair of images subjected to a large perspective deformation. Fig. 5(c) illustrates a set of manually chosen tiles drawn from Fig. 5(a). These tiles then undergo affine registration to establish correspondence with Fig. 5(b). The point correspondences of the tile centers were sufficient to achieve the perspective registration shown in Fig. 5(d). Notice that the minimal double-exposure effect in the figure is evidence of a good match.
The choice of tiles is important to the process. The perspective transformation computed above was applied to a regular grid. A piecewise affine approximation was applied to the warped grid to yield Fig. 6. Notice that many of the image tiles are lacking proper visual structure that would have made accurate matches possible. Therefore, only tiles replete with statistically significant data must be registered. The tiles shown in Fig. 5 were selected for their rich visual content.
5. SUMMARY AND CONCLUSIONS A method to register image pairs subjected to large perspective deformations has been presented. The images are assumed to be acquired under a weak perspective camera model or a fixed center of projection. These are necessary conditions to avoid parallax and facilitate registration in the presence of a global perspective transformation. We have demonstrated the feasibility of approximating a perspective transformation as piecewise affine. A reference image is subdivided into tiles and affine registration is applied to find the best matching tiles in the target image. These tile pairs are used to estimate a global perspective transformation. In this manner, we exploit recent results in affine registration to solve the more difficult perspective registration problem. In affine registration, we estimate the affine parameters necessary to register any two digital images misaligned due to rotation, scale, shear, and translation. The parameters are selected to minimize the sum of squared differences between the two images. They are computed iteratively in a coarse-to-fine hierarchical framework using a variation of the Levenberg-Marquadt nonlinear least squares optimization method. This approach yields a robust solution that precisely registers images with subpixel accuracy. After applying affine registration to all tiles, the correspondence of tile centers are used to solve for the eight perspective parameters. The accuracy of the technique is further refined by considering the local affine parameter estimates. A critical component to this work is the affine registration among tiles. As we pointed out in Section 4, the choice of tiles is important to the process. Future work will address the tile selection process. In particular, we will investigate the minimal set of tiles necessary to achieve accurate registration. The tile size is also critical to the accuracy and computational performance of the algorithm. There is a tradeoff between registration accuracy and the Taylor series approximation error. Although the use of small tiles conforms more closely to the Taylor series approximation, it lowers the registration accuracy because it offers less structure and visual detail. An automated method of determining this tradeoff will be investigated. We will also investigate the use of more robust optimization techniques for eliminating outliers. In particular, we will compare the least median squares with our existing linear least squares (pseudoinverse) approach. 9
REFERENCES 1. L. G. Brown, “A survey of image registration techniques,” ACM Computing Surveys 24, pp. 325–376, December 1992. 2. J. R. Bergen and E. Adelson, “Hierarchical, computationally efficient motion estimation algorithm,” J. Opt. Soc. Am. 4(35), 1987. 3. P. Anandan, “A computational framework and an algorithm for the measurement of visual motion,” Intl. J. Computer Vision 2, pp. 283–310, 1989. 4. J. R. Bergen, P. Anandan, K. J. Hanna, and R. Hingorani, “Hierarchical model-based motion estimation,” Proc. Euro. Conf. Computer Vision (ECCV) , pp. 237–252, 1992. 5. S. Chen, “Quicktime VR: An image-based approach to virtual environment navigation,” Proc. Siggraph ’95 , pp. 29–38, 1995. 6. M. Irani and P. Anandan, “Video indexing based on mosaic representations,” Proc. IEEE 86, pp. 237–252, May 1998. 7. R. Szeliski, “Image mosaicing for tele-reality applications,” Proc. IEEE Workshop on Applications of Computer Vision , pp. 230–236, 1994. 8. R. Szeliski and H.-Y. Shum, “Video mosaics for virtual environments,” IEEE Computer Graphics and Applications 16, pp. 22–30, 1996. 9. R. Szeliski and H.-Y. Shum, “Creating full view panoramic image mosaics and environment maps,” Proc. Siggraph ’97 , pp. 251–258, 1997. 10. H.-Y. Shum and R. Szeliski, “Construction and refinement of panoramic mosaics with global and local alignment,” Proc. Intl. Conf. Computer Vision , pp. 953–958, 1998. 11. S. Mann and R. W. Picard, “Video orbits of the projective group: A simple approach to featureless estimation of parameters,” IEEE Trans. Image Processing 6, pp. 1281–1295, September 1997. 12. J. Davis, “Mosaics of scenes with moving objects,” IEEE Conf. Computer Vision and Pattern Recognition (CVPR) , 1998. 13. P. Th´evenaz, U. E. Ruttimann, and M. Unser, “A pyramid approach to subpixel registration based on intensity,” IEEE Trans. Image Processing 7(1), pp. 27–41, 1998. 14. D. Aiger and D. Cohen-Or, “Mosaicing ultrasonic volumes for visual simulation,” IEEE Computer Graphics and Applications 20, pp. 53–61, March/April 2000. 15. D. W. Marquadt, “An algorithm for least-squares estimation of nonlinear parameters,” J. Society for Industrial and Applied Mathematics 11(2), pp. 431–441, 1963.
10
(a)
(b)
(c)
Figure 4. Piecewise affine approximation of perspective with (a) 2 2 tiles, (b) 4 4 tiles, and (c) 8 8 tiles.
11
(a)
(b)
(c)
(d)
Figure 5. Perspective registration. (a) Image I1 ; (b) Image I2 ; (c) Selected tiles; (d) Overlay of registered
Figure 6. Piecewise affine approximation.
12
I1 on I2 .
