Robust Parametric Image Registration

Robust Parametric Image Registration Felix Calderon1 , Juan J. Flores1 , and Leonardo Romero1 Division de Estudios de Postgrado. Facultad de Ingenier´ıa El´ectrica Universidad Michoacana de San Nicol´ as de Hidalgo. Santiago Tapia 403. Col Centro. Morelia, Michoacan, M´exico. CP 58000. {calderon, juanf, lromero}@umich.mx Summary. We present a hybrid method to perform parametric image registration. The objective is to find the best set of parameters to match a transformed image (possible with noise) to a target image. Hybridization occurs when Genetic Algorithms are able to determine rough areas of the parameter optimization space, but fail to produce fine tunings for those parameters. In that case, the Newton– Levenberg–Marquardt method is used to refine results. Another important combination of techniques results when we want to compare noisy images. In this case, the fitness function needs to be robust enough to discard misleading information contained in outliers. Statistical techniques are employed in this case to be able to compare images in the presence of noise. The resulting implementation, called GA–SSD–ARC–NLM was compared against robust registration methods used in the area of image processing. GA–SSD–ARC–NLM outperforms the RANSAC and the Lorentzian Estimator methods when images are noisy o they include occluded parts from objects, or even when new objects are on the target images.

1 Introduction The parametric image registration problem [19] consists of finding a parameter set for an Affine Transformation, which allows us to match an origin image with a target image as closely as possible. Occluded areas and noise (outliers) increase the complexity of the registration task, and some parametric registration techniques exhibit poor results under these circumstances. There are basically two sources of hibridization we are using in our proposed method: a gradient-based method is used to fine tune the results of Genetic Algorithms (GAs), and a stochastic method is used to produce a better evaluation function (i.e. a function that does not take into account outliers produced by noise). It is well known in the evolutionary computation community, that (GAs) are good tools to produce rough estimates of the region where optima appear, but they are not good at producing results with a high level of accuracy

2

Felix Calderon, Juan J. Flores, and Leonardo Romero

[8]. With that in mind, we made a combination of GAs and the Newton– Levenberg–Marquardt (NLM) method [4]. GAs find quickly a set of parameters around a global minimum, in the search region, and then NLM refine the results to reach the minimum. NLM, as a gradient-based method, suffers of getting stuck at local extrema. NLM fails if the initial parameters are not close enough to the right parameters (which minimize a given cost function). This problem is avoided by the use of GAs, which are not gradient-based, and they perform a better search on multimodal problems. On the other hand, GAs fail to give accurate results. So, the combination of GAs + NLM finds accurate results in a huge parameter space, avoiding the disadvantages of GAs and NLM. Now, talking about the fitness function, many algorithms try to minimize the Sum of Squared Differences (SSD) between the origin and target images. Successful SSD applications, including the classical Least Squared method (LS), are presented in [15, 14, 16]. Nevertheless, the SSD-based algorithms have poor performances in cases of noisy images. The problem with LS is that outliers have a heavy weight in the cost function and pull the solution towards them; robust methods try to exclude outliers in some way. Three well known robust methods, for parametric estimation, in the computer vision literature are the Least Median of Squares (LMedS) [12], the Lorentzian Estimator (LE) [6] and the Random Sample Consensus method (RANSAC) [7]. LMedS is a robust method which can deal with outliers (up to 50%); examples of its application to optical flow computation are presented in [1]. Some authors, instead of using LS, use the Lorenzian Estimator [2] in cases of noisy images, getting good results. The RANSAC method is a stochastic technique which presents good results with outliers [7]. An application of SSD for non-parametric image registration called SSDARC, is presented in [3]. The SSD–ARC cost function is minimized using a coarse to fine strategy and the Richardson’s Iteration. We take the core idea from SSD–ARC, and incorporate it in the fitness function, producing a function able to match noisy images, discarding the differences produced by noise. This chapter describes a new robust method named GA–SSD–ARC–NLM that combines the explicit outlier rejection idea of SSD–ARC [3] into a new cost function with a searching process in two steps. The first step is done with a Genetic Algorithm (GA), where the goal is to find an approximate solution inside a bounded parameter space. The second step refines the solution, reached by GA, using the NLM method. Both searching processes, consider an Affine Transformation to match the origin and target images, and try to minimize the same SSD–ARC cost function and compensate the disadvantages of one another. Experimental results show the robustness of GA–SSD–ARC–NLM for noise images and provides a comparison with the RANSAC and LE methods. The LE method was selected because it punishes outliers and its behavior is similar to the SSD–ARC cost function. The RANSAC method is a

Robust Parametric Image Registration

3

robust stochastic method, commonly used in the literature [7], which is not gradient-based. This chapter is organized as follows: Section 2 presents the image registration problem using an affine transformation; Section 3 describes the problem of estimating a good transformation in the presence of outliers (noisy data), and presents two alternative solutions to this problem: the Lorentzian Estimator (LE) and the RANSAC method, Subsection 3.3 presents the parametric cost function SSD–ARC; Section 4 presents the genetic coding scheme used to represent affine transformations, and describes the general framework for experiments, it also describes how the hybrid GA–NLM search process takes place; Section 5 presents experimental results, and a comparison against LE and RANSAC. Finally, Section 6 presents the conclusions.

2 Registration using an Affine Transformation The Affine Transformation (AT) [9, 18] allows us to compute, at the same time, translation, rotation, scaling, and shearing of images. An AT uses a sixparameter vector Θ, and maps a pixel at position ri (with integer coordinates [xi , yi ]) to new position rbi (with real coordinates [b xi , ybi ]) given by   θ0  θ1  · ¸ · ¸   xbi xi yi 1 0 0 0   θ2  rbi (Θ) = = (1)  ybi 0 0 0 xi yi 1  θ3    θ4  θ5 In matrix notation the AT transformation can be expressed as rbi (Θ) = M (ri )Θ

(2)

where M (ri ) and Θ are the matrix of coordinates and the parameter vector, respectively. Let I2 be the image computed by applying an affine transformation, given by Θ, to an input image I1 . The image registration problem would compute the vector parameter Θ given the images I1 and I2 . However, in practical cases, both images are corrupted by noise and the problem is to find the best AT to match the transformed origin image to the target image. To assign a value of matching quality, a common strategy, measures the distance between the source and target images, pixel by pixel, as in Equation (3). N X 2 ELS (Θ) = [I1 (rbi (Θ)) − I2 (ri )] (3) i=1

4


ELS =

N X

2

ei (Θ) =

i=1

N X

ρLS (ei (Θ))

(4)

i=1

where ei (Θ) = I1 (rbi (Θ)) − I2 (ri )

(5)

and N is the number of pixels, I(ri ) is the gray level value of pixel ri in image I and ρ is the error function given by ρLS (ei ) = e2i

(6)

Using this SSD error measurement, the registration image task consists of finding Θ∗ that makes E reaches a minimum value. This method is named Least Squared (LS) and some strategies to minimize Equation (3) are presented in [4].

3 Outliers and Parameter Estimation Assuming that ei (Θ) has a normal distribution with zero mean and constant variance, one method for computing Θ∗ is LS [10]. In this case, the likelihood p (Θ) is given by Equation (7) and Θ∗ is called the maximum likelihood estimator. p (Θ) =

N Y 2 1 exp−[ei (Θ)] 2π i=1

(7)

The goal of Equation (7) is to find a Θ set that maximizes the probability p(Θ). Taking the logarithm of Equation (7), one can see that this is equivalent to minimize the Equation (3) for the same data set. The influence function is defined as the derivative of the error function and helps seeing the contribution of the errors to the right solution (see [6]). In the LS case, the influence function is given by Equation (8) ψLS (ei ) =

∂ρLS (ei ) ∂e2 = i = 2ei ∂ei ∂ei

(8)

This means that big errors have a big contribution to the overall gradient vector. So when using a gradient-based search algorithm, this fact contributes to reach a solution biased by outliers. The problem with LS is its notorious sensibility to outliers, because they contribute too much to the general error term ELS , as Hampel described in [6]. To overcome the LS behavior, others interesting influence functions can be used. The following subsections present alternative solutions to the problem of producing a wrong estimation, biased by outliers.


5

3.1 Robust Statistical Estimation Robust Statistics [6] present the following main advantages: provide a better parameter set describing most of the input data and identify outliers. M. J. Black (in [2]) uses the T–student distribution with two degrees of freedom, instead of the normal distribution for maximum likelihood estimation, because the T–student distribution is used in populations with unknown variance. The T–student distribution is defined by Equation (9) (see [17]). h(t) =

Γ [(υ + 1) /2] √ Γ (υ/2) πυ

µ ¶−(υ+1)/2 t2 1+ υ

(9)

where Γ is the gamma function and υ is the degrees of freedom. The T– student distribution with two degrees of freedom and its logarithm is given by Equations (10) and (11), respectively. 1 h(t) = √ 2 2

µ ¶−1.5 t2 1+ 2

µ ¶ 1 t2 log(h(t)) = log( √ ) − 1.5 log 1 + 2 2 2

(10)

(11)

Black [2] substitutes t = ei /σ in the second term of Equation (11). Eliminating the associated factor, the resulting term, shown in Equation (12), is known as Lorentzian Estimator (LE), which has an influence function given by Equation (13). µ ¶ e2i ρLE (ei ) = log 1 + 2 (12) 2σ ψLE (ei ) =

2ei + e2i

2σ 2

(13)

The main advantage of using LE instead of the LS is its influence function, in which the factor 1/(2σ 2 + e2 ) reduces the outliers contribution (as you can see in Equations (8) and (13)), yielding, in cases of large error terms, derivatives near zero. Parameter σ controls the boundaries between inliers and outliers for LE. Nevertheless the outliers estimation is not clear, for LE, so Black and Rangarajan [2] proposed an EM algorithm extension. 3.2 RANSAC for Image Registration The RANSAC [5] is a method for robust fitting of models in the presence of multiple data outliers. Automatic Homography estimation using the RANSAC method is presented in [7] and RANSAC–based method for mosaic is presented in [11].

6


RANSAC–based algorithms for Image Registration compute a Homography given at least four point correspondences ri ↔ ri0 which are related by ri0 = Hri . Here ri and ri0 are given in homogeneous coordinates and Hi is a 3 × 3 matrix. This Equation may be expressed in terms of the vector cross product as ri0 × Hri = 0, and the solution must be found using the singular value decomposition [7]. The steps for Automatic Homography estimation for image registration (see [7]), given two images are: 1. Interest points: Compute interest points on each image using the Kanade– Lucas–Tomasi algorithm (KLT) [13]. 2. Putative Correspondences: Compute a set of interest points that match based on proximity and similarity of their intensity neighborhood (for instance using correlation). 3. RANSAC robust estimation: Repeat for M samples • Select a random sample of four correspondence points and compute the homography H (k) , solving ri0 × H (k) ri = 0 • Given H (k) , compute the distance for each putative correspondence • Compute the number of inliers consistent with H (k) • Choose the H (k) with the largest number of inliers 4. Optimal estimation: re-estimate H (∗) from all correspondence points classified as inliers (consensus set) using any error function (v.g. LS). 3.3 SSD-ARC The Sum of Squared Differences with Adaptive Rest Condition for non– parametric Image Registration was presented in [3] as the minimization of bSSD−ARC with a term hi li to reduce the huge a quadratic energy function E error contribution of outliers and it is given by Equation (14) bSSD−ARC (Θ, l) = E

N X

(ei (Θ) − hi li )2 + µ

i=1

N X

li2

(14)

i=1

where hi is desirable to be error dependent in order to weight the error, and li ∈ [0, 1] is an indicator function under the control of parameter µ. Assuming hi = ei , a particular SSD-ARC function can be obtained as bSSD−ARC (Θ, l) = E

N X i=1

e2i (Θ)(1 − li )2 + µ

N X

li2

(15)

i=1

bSSD−ARC function (Equation (15)), the term (1−li )2 allows For this new E us to discard outliers; one would expect values between zero and one for the variable li , zero for inliers, and one for outliers. The second term in Equation (15) restricts the number of outliers by means of µ. For instance, µ = 0 means


7

that all image pixels are outliers in contrast with µ = ∞, meaning that all the image pixels are inliers and the solution is close to LS. The minimizer li∗ , for Equation (15), can be computed by solving the independent system of equations given by (16) and (17) bSSD−ARC (Θ, l) ∂E =0 ∂li∗

(16)

−2e2i (Θ)(1 − li∗ ) + 2µli∗ = 0

(17)

for each image pixel, so the solution is given by Equation (18). li∗ =

e2i (Θ) µ + e2i (Θ)

(18)

The variable li∗ is bounded on the interval [0, 1], an indispensable condition to estimate and reject outliers. Replacing the value of li∗ in Equation (15), a new error function ESSD−ARC (Θ) results. ESSD−ARC (Θ) =

N N X X µe2i (Θ) = ρSSD−ARC (ei (Θ)) µ + e2i (Θ) i=1 i=1

(19)

µe2i µ + e2i

(20)

ρSSD−ARC (ei ) =

which is unimodal respect to e. The influence function (first derivative) and second derivatives are given by ψSSD−ARC (e) and ϕSSD−ARC (e) respectively ψSSD−ARC (ei ) =

ϕSSD−ARC (ei ) =

2µ2 ei 2

(21)

3

(22)

(µ + e2i ) ¢ ¡ 2µ2 µ − 3e2i (µ + e2i )

The error functions for ρLS , ρLE and ρSSD−ARC are unimodal with respect to the error term, as shown Figures 1(a), 1(c), and 1(e). LS does not have the possibility to reject outliers because its derivative function is proportional to the error term (see Figure 1(b)). Note that the SSD–ARC influence function, exhibits a behavior similar to the Lorentzian Estimator’s. In both functions, large errors yield derivative values close to zero, as they can be shown in Figures 1(d) and 1(f). This means that no matter how large the error provided by an outlier is, its contribution to the searching process is very low. p The SSD–ARC influence function has a maximum value located at eb = µ/3; that value can be used as threshold for data rejection; for instance, an error value greater than 2b e, will have a derivative value close to zero. This condition can be used to reject outliers further away than a given error threshold. µ can be taken as µ ≥ 3b e2 for a given error threshold nevertheless,

8


10000

200

9000

150

8000 100 7000 50

dE/de

E

6000

5000

4000

0

−50

3000 −100 2000 −150

1000

0 −100

−80

−60

−40

−20

0 e

20

40

60

80

−200 −100

100

(a) LS error function

−80

−60

−40

−20

0 e

20

40

60

80

100

(b) Influence function for LS

8 0.8

7

0.6

6

0.4

0.2

dE/de

dE/de

5

4

0

3

−0.2

2

−0.4

1

−0.6

0 −100

−80

−60

−40

−20

0 e

20

40

60

80

−0.8 −100

100

(c) LE error function with σ = 2

−80

−60

−40

−20

0 e

20

40

60

80

100

(d) Influence function for LE

7 1.5

6 1

5 0.5

E

dE/de

4 0

3

−0.5

2

−1

1

0 −100

−80

−60

−40

−20

0 e

20

40

60

80

100

(e) SSD–ARC error function with µ = 10

−1.5 −100

−80

−60

−40

−20

0 e

20

40

60

80

100

(f ) Influence function for SSDARC

Fig. 1. Error Function Examples and their influence functions


9

there is not gradient-based algorithm capable to reach the minimum if the initial value gives an error than 2b e. The algorithm converges only for ¯p greater ¯ ¯ ¯ error values less than ¯ µ/3¯ due the derivative goes zero as you can see in figure 1(f). In some experiments, parameter vector computed by LS was taken as initial value for the SSD–ARC; for small µ values, the solution given by LS could not be improved by SSD-ARC. Therefore a two-step hybrid minimization algorithm is proposed: 1. First a GA search is used and then 2. Refinement of the GA solution is done, using NLM as shown Algorithms 1 and 2 where the SSD-ARC error function, given by Equation (19), is used in both steps. The next section provides more details for this two-step minimization algorithm.

4 Hybrid Genetic/Gradient-based Optimization In this section we provide a description of the components of the two–step algorithm (GA–SSD–ARC–NLM) used to minimize the error function given by Equation (19). The problem to be solved, is the determination of the vector Θ that best approximates the transformation that maps the source image I1 to the target image I2 . We propose to start the search for an optimal vector Θ using GAs and then to refine the results using a gradient-based algorithm, namely, the Newton Levenberg–Marquardt (NLM) method. NLM is convenient in cases where a semipositive definite symmetric Hessian matrix exists [4]. 4.1 Genetic Algorithms Genetic Algorithms is an optimization technique inspired in the biological process of evolution and survival of the fittest individuals in a population. Given an initial population, GA provides the means for this population to reach a state of maximum fitness in successive generations. The general optimization procedure is: define a cost function, encode an individual in a chromosome (in our case, an individual represents an AT – encoded by a parameter vector), and create a random starting population. Evaluate the cost function for each individual, allow them to mate, reproduce, and mutate. Repeat these last four steps for as many generations as needed in order to reach a stopping criteria (v.g., a small enough error, or a fixed number of generations). Haupt and Haupt [8] describe the steps to minimize a continuous parameter function cost using GAs. In our case, the function cost given by Equation (19) is optimized and the six-vector parameter Θ will be the chromosome

10


for each individual. Thus, the chromosome is represented by a real-valued vector of six elements. GA needs a search interval for each one of the parameters. To establish the initial population, the jth parameter or gene is randomly computed by θj = (θjmax − θjmin ) ∗ α + θjmin (23) where θjmax and θjmin are the upper an lower bounds, and α is a number randomly selected in the interval [0, 1]. This strategy is applied for newly generated genes in an individual’s chromosome. The same strategy is used for determining new values for genes of individuals during the mutation process. In each generation, a fitness-based selection process indicates which individuals from the population will mate and reproduce, yielding new offsprings. Once we have selected two individuals Θp , and Θm for mating, cross-over is accomplished according to the following formulae θi,j = θm,j − β(θm,j − θp,j ) θi+1,j = θp,j + β(θm,j − θp,j )

(24) (25)

where β is a random number between zero and one, and θi,j denotes the j − th parameter of the vector parameter Θi . Newly born offsprings (Θi and Θi+1 ) are incorporated to the population, their fitness is computed, and the population is cut to maintain the same size across generations. To maintain the population size, the worst individuals are discarded. The complete GA–SSD–ARC algorithm, is presented as Algorithm 1. 4.2 SSD–ARC Minimization by NLM In order to find the minimizer Θ∗ for the general error function ESSD−ARC (Θ), given by Equation (19), a gradient-based algorithm, namely the Newton Levemberg Marquart method [4] is used. Equation (26) gives the steps to find the minimum using NLM. h ³ ´ i−1 ³ ´ Θ(k+1) = Θ(k) − HSSD−ARC Θ(k) + λ(k) I ∇ESSD−ARC Θ(k) (26) ¡ ¢ where ∇ESSD−ARC Θ(k) is the gradient at each iteration, computed by ∇ESSD−ARC (Θ) = 2

N −1 X

J(b ri (Θ))

i=0

µe2i µ + e2i

(27)

and the Hessian matrix is 2

∇ ESSD−ARC (Θ) = 2

N −1 X i=0

T

J (b ri (Θ))J(b ri (Θ))

¡ ¢ 2µ2 µ − 3e2i 3

(µ + e2i )

(28)


11

Algorithm 1 GA-SSD-ARC Given µ, I1 (ri ), I2 (ri ) and Npop then 1. Compute a initial population - For k = 1 to Npop Randomly compute Θ(k) using Equation (23), For each Θ(k) compute I1 (b ri (Θ(k) )) and ei (Θ(k) ), by Equation(5) Compute the error ESSD−ARC (Θ(k) ) using Equation (19) - Select the half population with least ESSD−ARC (Θ(k) ); Npop ← Npop /2.0 2. Matting. - Randomly select couples over the best population half given more matting probability those elements with least ESSD−ARC (Θ(k) ) 3. Reproduce - Replace the worst population half by the offsprings created by Equations (24) and (25) - and then, compute the Error ESSD−ARC (Θ(of f ) ) (Equation (19)) for each offspring. 4. Mutation - Randomly select the k − th population member and the j − th parameter, and replace it by a new parameter computed by Equation (23) - Never mutate the best population member. 5. Repeat steps 2, 3 and 4 until the population do not reach convergence. 6. For the best population member Θ(0) , compute ei (Θ(0) ) using Equation (5), and then, the outliers field li∗ by Equation (18)

Finally J(b ri (Θ)) is the Jacobian matrix given by £ ¤T J(b ri (Θ)) = A−T ∇In (ri ) M (ri ) M (ri ) is defined by Equation (1), In (ri ) = I1 (b ri (Θ)) and ¶ µ θ0 θ1 A= θ3 θ4

(29)

(30)

A new Θ(k+1) is accepted only when ESSD−ARC (Θ(k+1) ) < ESSD−ARC (Θ(k) ) which is evaluated using Equation (19). At the beginning λ takes a small values and the search is like the Newton method. During the iterative process λ is adjusted; Large λ values perform a Gradient based search, the final SSD– ARC–NLM is presented as Algorithm 2

5 Results In the rest of the chapter, the following name convention is used to distinguish between the different optimization methods used to minimize the SSD–ARC equation. If only the NLM algorithm is used, the minimization process is

12


Algorithm 2 SSD–ARC–NLM Given the µ, I1 (ri ), I2 (ri ), λ(0) and Θ(0) then: 1. Set k = 0 and compute E(Θ(0) ) by Equation (19) 2. Do Compute ∇E(Θ(k) ) and ∇2 E(Θ(k) ) by Equations (27) and (28) Do Update H(Θ(k) ) ← H(Θ(k) ) + λ(k) I Compute Θ(k+1) by Equation (26) and ESSD−ARC (Θ(k+1) ) by Equation (19), If (ESSD−ARC (Θ(k+1) ) > ESSD−ARC (Θ(k) ) λ(k+1) = 10λ(k) else λ(k+1) = λ(k) While ((ESSD−ARC (Θ(k+1) ) > ESSD−ARC (Θ(k) )) and (λ(k+1) < λmax ) If (λ(k) > λmin ) then λ(k) ← λ(k) /10 If (ESSD−ARC (Θ(k+1) ) > ESSD−ARC (Θ(k) )) then Θ(k+1) = Θ(k) Set k ← k + 1 While ((ESSD−ARC (Θ(k) ) < ESSD−ARC (Θ(k−1) )) 3. For the final Θ∗ , compute ei (Θ∗ ) using Equation (5), and then li∗ by Equation (18)

named SSD–ARC–NLM; if we use only GA the minimization procedure will be named GA–SSD–ARC; when we use a combination, performing first GA and then NLM, the process will be called GA–SSD–ARC–NLM. In this section, we compare the results of image registration using GA– SSD–ARC–NLM method against LE error function and RANSAC method; results are presented for both synthetic and real images. For LE, a minimization scheme similar to GA–SSD–ARC–NLM (using GA and NLM) is used in order to give similar minimization conditions, because LE and SSD–ARC present a similar behavior in their derivative functions. In experiments with synthetic images, we know the right parameter vector b for the transformed image take it as target. Under those circumstances, Θ we propose to use the Euclidean distance |∆Θ| between the known parameb and the estimated parameter vector Θ as a proximity measure ter vector Θ (for comparison purposes, only, not for optimization). Algorithms with better performance will have smaller proximity measures with the real parameter vector. 5.1 Synthetic Images In the synthetic images, the NLM uses the ¯vector Θ =¯ [1, 0, 0, 0, 1, 0]T as ¯ (k+1) −E (k) ¯ the initial value, and the stop criterion was ¯ E E (k+1) ¯ < 1e − 5 or 1000 iterations. For GA, a population of 3000 individuals and 100 generations was used. In order to accelerate the convergence procedure, in some cases, the error function was evaluated only on 50% of the image pixels. The GA search


13

boundaries, for each of the affine transformation parameters, are shown in Table 1. Table 1. GA Search Boundaries θ0 θ1 θ2 θ3 θ4 θ5

θjmin 0.5 -0.5 -10.0 -0.5 0.5 -10.0

θjmax 1.5 0.5 10.0 0.5 1.5 10.0

The parameters µ for SSD–ARC and σ for LE were hand-picked in order to yield the best performance for both algorithms, those values are 20 and 25, respectively, and they remain constants in all the experiments.

(a) Origin image

(b) Target image (c) Outliers image estimated by GA– SSD–ARC–NLM

(d) Image estimated by GA– SSD-ARC–NLM Fig. 2. Image registration performed by GA–SSD–ARC–NLM using the Peppers image

14


Peppers Image The main goal of this experiment is to give an example of how the two minimization steps for SSD–ARC error function, improved the solution obtained using only NLM. With this goal in mind, first the experiment was done using NLM for LS, LE, and SSD–ARC error functions. Next, a combination of GA and NLM is introduced for the LE and SSD–ARC error functions. In this b = [0.9511, -0.3090, 0.0000, experiment, a rotation of 18o degrees, given as Θ 0.3090, 0.9511, 0.0000] is applied to the Peppers image (the origin and target images are shown in Figures 2(a) and 2(b) respectively). Table 2. Results for the Peppers Image using LS, LE, SSD–ARC and RANSAC algorithm LS–NLM

parameters

SSD–ARC–NLM

µ = 20

SSD–ARC–NLM

µ = 5000

LE-NLM

σ = 25

RANSAC

0.9511 0.3090 1.0106 -0.0043 0.9511 0.3090 1.1127 -0.0443 0.9591 0.3140

Θ -0.3090 0.9511 -0.0002 1.0090 -0.3090 0.9511 -0.1655 0.7131 -0.3107 0.9595

0.0000 0.0000 -1.2631 0.4235 0.0000 0.0000 -5.8382 11.3104 -0.2383 -0.2065

|∆Θ| 0.0000 1.4054 0.0000 12.7373 0.3156

The results from LS, LE and SSD–ARC, when their respective error functions were minimized using only NLM, are presented in Table 2; the same Table presents the results for the RANSAC method using the same images. According with the results presented in Table 2, LS has better performance than RANSAC, due to the fact that the origin and target images do not have outliers, and the RANSAC method only used a subset of the whole image pixels. LE presents the worst results, perhaps it needs a minimization procedure in two steps as GA–SSD–ARC–NLM (see section 4). SSD–ARC–NLM presents different results using different µ values, with µ = 5000, SSD–ARC–NLM presents better result than SSD–ARC–NLM with µ = 20. This fact show us, for this experiment, that SSD–ARC–NLM with large µ value presents a behavior similar to LS–NLM. This condition can be used only in cases where images do not present outliers. We suggest to use low µ values in order to allow outlier rejections for the SSD–ARC function. In the second part of this experiment, the minimization of the SSD–ARC and LE error functions was done using first GA and then NLM (as it was proposed in section 4 for the GA–SSD–ARC–NLM algorithm). The same images (Figure 2) and the AT, were used for this experiment, and the results are shown in Table 3. Comparing these results against the results presented in Table 2 one can observe, for this images how the two steps (GA, NLM)


15

Table 3. Performance for SSD–ARC and LE error function, using the Peppers image, when GA-NLM are used algorithm GA-SSD-ARC

parameters µ = 20 0.9537 0.3090 GA–SSD-ARC–NLM µ = 20 0.9511 0.3090 GA-LE σ = 25 0.9508 0.3077 GA–LE–NLM σ = 25 0.9511 0.3090

Θ -0.3097 0.9509 -0.3089 0.9511 -0.3088 0.9505 -0.3089 0.9511

-0.0970 0.0254 0.0000 0.0000 0.0092 0.1313 0.0000 0.0000

|∆Θ| 0.1003 0.0001 0.1316 0.0001

improved the solution reach by SSD–ARC–NLM and µ = 20. For LE one can observe the same behavior. In conclusion the algorithm GA–SSD–ARC–NLM found the best parameter set for this images using µ = 20, outperforming RANSAC’s solution. Peppers images with outliers In this experiment, in contrast with the previous one, 10% of the image pixels were set to black color (zero value) in order to simulate an outlier field. Using the same AT as the previos experiment, the target image is shown in figure 3(b). Similar to the last experiment, the registration image task was performed using LS, LE, SSD–ARC and RANSAC in combination of GA and NLM minimization algorithms. The results are presented in Table 4. In this Table, the SSD–ARC error function presents a similar behavior to the the last experiment; the best result is reached again when GA-NLM are used. Note that the vector parameter estimated by GA–SSD–ARC–NLM is the nearest to the real vector value. The resulting image after applying the vector parameters estimated with GA–SSD–ARC–NLM is shown in Figure 3(d) and the outlier field (presented as image) is shown in Figure 3(c). The outlier field is computed as the product of Equation li∗ (see Equation (18)) and the resulting image (fig 3(d)). Gatinburg images with outliers In this experiment the complement of a circular-shaped outlier field and the b = [1.2, 0, 0, 0, 1.2, 0] are applied to the picture in Figure 4(a), AT, given by Θ yielding the picture in Figure 4(b). The solutions for the different minimization schemes are shown in Table 5. GA–SSD–ARC–NLM again presents the best result. Figure 4(c) shows the outlier field computed by GA–SSD–ARC– NLM. Figure 4(d) shows the origin image after applied the AT computed by GA–SSD–ARC–NLM. Note the accuracy between the outliers and the target image.

16


(a) Origin Image

(b) Target Image

(c) Outliers Image

(d) Resulting Image

Fig. 3. Image registration performed by GA–SSD–ARC–NLM using the Peppers image and a regular outliers field

Earth image with outliers In this experiment, we use an earth image (see Figure 5(a)), an AT given b =[0.65, 0.15, 0, 0.15, 0.65, 0] and a random outlier field are applied by Θ to the earth image. For the outlier field, one image pixel is replaced by zero if a random uniform variable is between 0 and 0.05; the final outlier image, computed in this way, is shown in Figure 5(e). The target image, Figure 5(b) is the resulting image after applying the affine transformation and the outlier field to the origin image figure 5(a). For this images, the affine transformations computed by the different algorithms are presented in Table 6; again, the best result is reached by GA– SSD–ARC–NLM. In contrast, the RANSAC method, presents one of the worst result. The resulting image, show in Figure 5(d), was computed using the ori-


17

Table 4. Comparative results for the Peppers image with a regular outlier field algorithm LS–NLM

parameters

SSD-ARC–NLM

µ = 20

GA-SSD-ARC

µ = 20

GA-SSD-ARC–NLM

µ = 20

LE–NLM

σ = 25

GA-LE

σ = 25

GA–LE–NLM

σ = 25

RANSAC

1.2861 0.0980 1.0116 -0.0057 0.9546 0.2990 0.9513 0.3091 1.2394 0.1004 0.9530 0.3091 0.9530 0.3091 0.9532 0.3127

Θ -0.3763 0.8820 -0.0008 1.0099 -0.3086 0.9502 -0.3089 0.9511 -0.1746 0.8587 -0.3084 0.9506 -0.3084 0.9506 -0.3053 0.9556

-5.3277 5.0703 -1.2478 0.4813 -0.2383 0.7354 -0.0097 0.0007 -9.3068 5.4461 -0.0963 0.0346 -0.0963 0.0346 -0.2267 -0.2035

|∆Θ| 7.3660 1.4106 0.7731 0.0097 10.7903 0.1023 0.1023 0.3047

Table 5. Results for Gatinburg images algorithm LS–NLM

parameters

SSD-ARC–NLM

µ = 20

GA-SSD-ARC

µ = 20

GA-SSD-ARC–NLM

µ = 20

LE–NLM

σ = 25

GA-LE

σ = 25

GA–LE–NLM

σ = 25

RANSAC

1.3115 -0.0061 1.0000 0.0000 1.2009 -0.0007 1.2000 0.0000 1.3487 0.0599 1.2000 0.0000 1.2000 0.0000 1.2003 0.0013

Θ -0.1439 0.8401 0.0000 1.0000 -0.0044 1.2037 0.0000 1.2000 -0.1384 0.2555 -0.0004 1.2011 -0.0004 1.2011 0.0009 1.2019

1.2116 8.1120 0.0000 0.0000 0.1778 -0.1046 0.0000 0.0000 -6.8888 12.7813 0.0287 -0.0637 0.0287 -0.0637 0.0740 -0.1179

|∆Θ| 8.2119 0.2828 0.2064 0.0000 14.5518 0.0699 0.0699 0.1392

gin image and the parameter vector computed by the GA–SSD–ARC–NLM. LE also gives good results. Finally, figure 5(f) shows the difference between the given outlier field (Figure 5(e)) and the estimated outliers field, computed by Equation (18), in Figure 5(c). Both images are very similar and the difference are close to zero. In the figure 5(f), black pixels mean zero differences, otherwise gray level pixels encode the absolute difference.

18


(a) Origin Image

(b) Target Image

(c) Outliers Image computed by (d) Estimated Image by GA– GA–SSD–ARC–NLM SSD–ARC–NLM Fig. 4. Image registration performance by GA–SSD–ARC–NLM using the Gatinburg image with a regular outlier field

The poor performance of RANSAC is due to the characteristics of the KLT algorithm, which is derivative-based; the outlier field creates false putative correspondences, for this couple of images. If the RANSAC algorithm has poor putative correspondences, the algorithm is not capable to find a good consensus set, resulting in a poor affine transformation model. This experiment shows the high sensitivity of the RANSAC method for this kind of noise, and the robustness of the GA–SSD–ARC–NLM method under the same circumstances. 5.2 Image Registration with Real Images The comparison between the different algorithms presented in this chapter using real images can only be done in a qualitative way. In all experiments done with synthetic images GA–SSD–ARC–NLM presents the best results. In general, second place was reached by the RANSAC method and it is not gradient based method. For this reason, we only present a comparison between GA–SSD–ARC–NLM and RANSAC.


(a) Origin image

(b) Target image

(c) Computed outlier image

(d) Computed image

(e) Original outlier image

(f ) Difference between target and computed outlier fields

19

Fig. 5. Image registration performed by GA–SSD–ARC–NLM using the Earth image and a random outlier field

20

Felix Calderon, Juan J. Flores, and Leonardo Romero Table 6. Results for the Earth experiment with random outlier field algorithm LS–NLM

parameters

SSD-ARC–NLM

µ = 20

GA-SSD-ARC

µ = 20

GA-SSD-ARC–NLM

µ = 20

LE–NLM

σ = 25

GA–LE

σ = 25

GA–LE–NLM

σ = 25

RANSAC

0.8598 0.0799 0.9931 0.0368 0.6505 0.1448 0.6500 0.1500 0.8593 0.0867 0.6472 0.1511 0.6501 0.1500 0.6368 0.1451

Θ -0.0103 0.9595 0.0118 1.0305 0.1497 0.6535 0.1500 0.6500 -0.0106 0.9536 0.1514 0.6529 0.1500 0.6500 0.1431 0.6296

6.4877 -17.0157 -1.3534 -6.8756 -0.0191 0.2642 0.0002 0.0004 6.4833 -17.6414 0.0612 -0.4626 -0.0067 0.0051 1.2729 1.9900

|∆Θ| 18.2152 7.0285 0.2650 0.0004 18.7994 0.4667 0.0084 2.3624

Car Park Images The first experiment uses the images shown in Figure 6. These images show some differences that can not be fully modelled using only an AT because, the target image (Figure 6(b)) has a boy in front of the car (which does not appear in the origin image as shown Figure 6(a)), in order to introduce more complexity in the outliers, and the camera was rotated. The goal is to obtain the boy image as part of the outlier field and the affine transformation introduced by the camera rotation. The transformation computed by GA–SSD–ARC–NLM and RANSAC are shown in Table 7, note the proximity between the parameter vectors computed by both algorithms but these values do not allow us to conclude which one is better. Using Equation (18) the outlier field is computed and its image is presented in Figure 6(c), note in this image the contour of the boy in front of the car. Figure 6(d) shows the resulting image computed by vector parameter computed by GA–SSD–ARC–NLM. Finally the figures 6(e) and 6(f) give a clear idea of the accuracy of both AT computed, in both figures the absolute value of the difference between the target image and the computed image by GA–SSD–ARC–NLM and RANSAC (dark areas correspond to low difference). Note the quality for the AT computed by GA–SSD–ARC–NLM, shown in figure 6(e), most static objects like the car or walls are almost perfectly matched, only the boy, the leaves of trees does not match. In the RANSAC difference image (figure 6(f)) even the car is not fully matched.


(a) Origin Image

21

(b) Target Image

(c) Outlier Image computed by (d) Final Image registration usGA–SSD–ARC–NLM ing GA–SSD–ARC–NLM

(e) Difference between com- (f ) Difference between computed image by GA–SSD– puted image by RANSAC and ARC–NLM and target image target image Fig. 6. Car park image registration

A Panoramic Mosaic The second experiment is composed of two shots taken in our campus from different angles. The goal is to produce an affine transformation that allows us to paste them together in a panoramic mosaic. Figures 7(a) and 7(b) show the origin and target images, respectively. The parameter vector computed by RANSAC and GA–SSD–ARC–NLM are shown in Table 8; comparing these

22


(a) Origin Image

(b) Target Image

(c) OutlierImage

(d) Mosaic Image Fig. 7. Example of a Panoramic Mosaic


23

Table 7. Resulting AT for RANSAC and GA–SSD–ARC–NLM method using car park images algorithm parameters GA–SSD–ARC–NLM µ = 20 0.9700 0.2413 RANSAC 0.9166 0.2151

Θ -0.2280 0.9768 -0.2473 0.9249

42.7323 -20.6006 47.1152 -15.3065

results, there is not a significant difference between them. The mosaic image (Figure 7(d)) is computed as the linear combination of the origin and target images, and the outlier field Equation (18) by Equation (31) I1 (ri (Θ))l∗ (ri ) + I2 (ri )(1 − l∗ (ri ))

(31)

Finally the image of the outlier field is shown in Figure 7(c). Table 8. Comparative Results between RANSAC and GA–SSD–ARC–NLM for images in figure 7 Algorithm Parameters GA–SSD–ARC–NLM µ = 20 0.9703 0.0139 RANSAC 0.9558 -0.0167

Θ 0.042 0.9758 0.0380 0.9986

56.0455 0.0051 57.1481 0.1160

6 Conclusions In this chapter, we presented an algorithm for parametric image registration, based on the non–parametric SSD–ARC error function. The error function is minimized in two steps, using GAs and NLM. The final algorithm improved the solution using only GAs and/or NLM and it is robust when the images are corrupted by noise. A comparison of GA–SSD–ARC–NLM with other registration algorithms, such RANSAC and LE, was presented in order to provide experimental proof of the robustness of GA–SSD–ARC–NLM. We tested GA–SSD–ARC–NLM using different kinds of images and outlier fields; in all the experimental tests, GA–SSD–ARC–NLM outperformed the results of the RANSAC and LE methods. Our method is similar to GA–LE–NLM but it is less sensitive to the particular parameter µ (or σ for LE), and it is easier to find better solutions. Additionally the GA–SSD–ARC–NLM provides an explicit way to compute the outliers and does not need an extra image processing. With synthetic images, we tested the robustness of GA–SSD–ARC–NLM and presented how, the two minimization steps improved the solution, using

24


only NLM. In case of real images, the comparison were done using only the final parameter vector and the difference between the computed image and the target image, computed by GA–SSD–ARC–NLM and RANSAC; both algorithms exhibited similar results. Furthermore, GA–SSD–ARC–NLM has the advantage of computing the outliers with accuracy as experiments proved it, even in cases of random outlier fields. In contrast with RANSAC, GA–SSD–ARC–NLM computes outliers for the whole image using a simple equation. Outlier rejection are explicitly modeled in in the parametric SSD–ARC function and the outlier field can be easily computed when the algorithm converge. A disadvantage GA–SSD–ARC algorithm is that GA needs the boundaries of the parameters to be specified. In future works, we plan to compute good values for µ parameter, given a pair of images. Also it is interesting to use RANSAC methods as a way to define the boundaries of the parameters.

References 1. Alireza Bab-Hadiashar and David Suter. Robust optic flow estimation using least median of squares. IEEE International Conference on Image Processing ICIP96, 1996. 2. Michael J. Black and Anand Rangarajan. On the unification of line processes, outlier rejection, and robust statistics with applications in early vision. International Journal of Computer Vision, 19:57–92, July 1996. 3. Felix Calderon and Jose L. Marroquin. Un nuevo algoritmo para el calculo de flujo optico y su aplicacion al registro de imagenes. Computacion y Sistemas, 6(3):213–226, January-March 2003. 4. J. E. Dennis and Robert B. Schnabel. Numerical Methods for Unconstrained Optimization and Nonlinear Equiations. Society for Industrial and Applied Mathematics. siam, Philadelphia, 1996. 5. M. A. Fischler and R. C. Bolles. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography, comm. ACM, 24(6):381–395, 1981. 6. F. R. Hampel, E. M. Ronchetti, P. J. Rousseeuw, and W. A. Stahel. Robust Statistics: The Approach Based on Influence Functions. John Wiley and Sons, New York, N.Y, 1986. 7. Richard Hartley and Andrew Zisserman. Multiple View Geometry in Computer Vision. Cambridge University Press, University of Oxford, UK, second edition, 2003. 8. Randy L. Haupt and Sue Ellen Haupt. Practical Genetic Algorithms. John Wiley and Sons, New York, N.Y, 1998. 9. Donald Hearn and M. Pauline Baker. Computer Graphics C Version. Prentice Hall, Inc., Simon and Schuster / A Viacom Upper Saddle River, New Jersey 07458, second edition, 1997. 10. Geoffrey J. McLachlan. Mixture Models, volume 84. Dekker, 270 Madison Avenue, New York, New York 10016, 1988. 11. Kenji Okuma. Mosaic based representation of motion. ACCV, 2004.


25

12. P. J. Rousseeuw and A. M. Leroy. Robust Regression and Outlier Detection. John Whiley and Sons, New York, 1987. 13. J Shi and C Tomasi. Good features to track. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR94), Seatle USA, June 94. IEEE. 14. R Szeliski. Video mosaics for virtual environments. In IEEE Computer Graphics and Applications, volume 16, pages 22–30. IEEE, March 1996. 15. Richard Szeliski and James Coughlan. Spline-based image registration. Technical Report 94/1, Harvard University, Department of Physics, Cambridge, Ma 02138, April 1994. 16. Baba C. Vemuri, Shuangying, Huang, Sartaj Sahni, Christiana M. Leonard, Cecile Mohr, Robin Gilmore, and Jeffrey Fitzsimmons. An efficient motion estimator with application to medical image registration. Medical Image Analysis, 2:79–98, 1998. 17. R. E. Walpole and R. H. Myers. Probabilidad y Estadistica para Ingenieros. Inteamericana, Mexico. D.F, 1987. 18. Allan Watt. 3D Computer Graphics. Pearson Education Limited, 3rd edition, 2000. 19. Barbara Zitova and Jan Flusser. Image registration methods: A survey. Image and vision computing, 21(11):977–1000, 2003.