2830
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 16, NO. 11, NOVEMBER 2007
A Nonlinear Least Square Technique for Simultaneous Image Registration and Super-Resolution Yu He, Kim-Hui Yap, Member, IEEE, Li Chen, Member, IEEE, and Lap-Pui Chau, Senior Member, IEEE
Abstract—This paper proposes a new algorithm to integrate image registration into image super-resolution (SR). Image SR is a process to reconstruct a high-resolution (HR) image by fusing multiple low-resolution (LR) images. A critical step in image SR is accurate registration of the LR images or, in other words, effective estimation of motion parameters. Conventional SR algorithms assume either the estimated motion parameters by existing registration methods to be error-free or the motion parameters are known a priori. This assumption, however, is impractical in many applications, as most existing registration algorithms still experience various degrees of errors, and the motion parameters among the LR images are generally unknown a priori. In view of this, this paper presents a new framework that performs simultaneous image registration and HR image reconstruction. As opposed to other current methods that treat image registration and HR reconstruction as disjoint processes, the new framework enables image registration and HR reconstruction to be estimated simultaneously and improved progressively. Further, unlike most algorithms that focus on the translational motion model, the proposed method adopts a more generic motion model that includes both translation as well as rotation. An iterative scheme is developed to solve the arising nonlinear least squares problem. Experimental results show that the proposed method is effective in performing image registration and SR for simulated as well as real-life images. Index Terms—Image super-resolution (SR), image registration, nonlinear least squares methods.
I. INTRODUCTION
I
MAGE super-resolution (SR) is a process to fuse multiple low-resolution (LR) images to produce a high-resolution (HR) image. The LR images are typically shifted up to subpixel level, and, hence, the information available in each LR image can be extracted and combined to obtain a HR image. Image SR has wide applications, including remote sensing, military surveillance, and medical imaging, among others. Registration is a critical step in image SR. Conventional SR algorithms [1] assume the motion parameters are known a priori (such as the image-formation system using multiple CCD sensor arrays in [2]). This assumption, however, is impractical in many
Manuscript received November 28, 2006; revised July 30, 2007. The associate editor coordinating the review of this manuscript and approving it for publication was Prof. Stanley J. Reeves. Y. He, K.-H. Yap, and L.-P. Chau are with the School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore 639798 (e-mail:
[email protected];
[email protected];
[email protected]). L. Chen is with the School of Computer Science and Technology, Wuhan University of Science and Technology, Wuhan, China 430081 (e-mail: chenli@ieee. org). Digital Object Identifier 10.1109/TIP.2007.908074
applications as perfect subpixel displacements are difficult to calibrate and implement. Current SR techniques [3]–[7] are commonly performed in two disjoint stages, namely 1) image registration from LR images, followed by 2) inverse estimation that integrates image fusion and deblurring into a single step. Generally, these SR algorithms ignore registration errors and assume the estimated motion parameters by existing registration methods to be error-free. Nevertheless, due to the presence of aliasing in the captured LR images, most existing registration algorithms for aliased images still experience subpixel errors. For instance, a frequency domain-based algorithm in [8] is developed for registering a set of aliased LR images with application to SR. Rotational and translational parameters are estimated based on the low-frequency part of the LR images. Nevertheless, accurate registration in the LR domain remains difficult to achieve, giving rise to suboptimal results in the reconstructed HR images [9]. This motivates the study of progressive image SR in this work, which takes into account the impact of unreliable initial registration. Another class of SR methods [10], [11] employs iterative schemes based on alternating minimization (AM) to estimate the HR image and the motion parameters alternatingly. In [11], a regularized constrained total least-squares method has been proposed to minimize a nonconvex cost function. Similar to [10], the cost function is projected onto the image and motion parameter domains one at a time, and minimized iteratively. However, it has been shown in [12] that the optimization using the AM approach tends to be trapped at local minima. Woods et al. [13] have proposed algorithms for joint estimation of HR image and motion parameters using Bayesian and Maximum a posteriori (MAP) formulation. Similar to the idea of the AM framework, expectation maximization (EM) is employed to solve the problem. It is observed that the performance of the method relies greatly on the initialization and it also experiences local convergence. It should be pointed out that the proposed algorithm in this paper may also experience local convergence if the initial estimate is poor. However, as we have demonstrated in various experiments in Section IV, the initial estimate of the motion parameters obtained using existing registration techniques is sufficient to produce satisfactory convergence and results in our algorithm. Recently, some image SR methods based on the principle of variable projection (VP) have been developed in [14] and [15]. In [14], Chung et al. propose a nonlinear cost function, and estimate the registration parameters and the HR image using the Gauss-Newton method. The method, up to a certain extent, shares many similarities with the AM method, as the cost function is projected onto the image and motion parameter domains
1057-7149/$25.00 © 2007 IEEE
HE et al.: NONLINEAR LEAST SQUARE TECHNIQUE FOR SIMULTANEOUS IMAGE REGISTRATION
iteratively. In [15], a VP-based SR method is also developed by Robinson et al. In their method, the registration parameters for general motion model are first estimated by minimizing a VP functional. Then, the estimates of the motion parameters are used to reconstruct the HR image using maximum likelihood (ML) or MAP estimation. The paper further describes an efficient implementation of the minimization process in the Fourier domain. In [9] and [16], two subspace methods are developed for registration of totally aliased signals. After the motion parameters are estimated, the HR image is reconstructed in the frequency domain by using the least squares method. These methods, nevertheless, do not consider the effect of uniform kernel during the downsampling process of LR image formation. Some other researchers [17], [18] attempt to model the registration errors using Gaussian noise and proceed to develop a regularized adaptive HR estimation method. However, these methods can only handle global translational shift. This precondition constrains the application of these methods. To the best of our knowledge, there have been limited studies on image SR that progressively improve the registration and the HR image simultaneously, while using a motion model that includes both translation and rotation. In view of this, this paper proposes a new framework for simultaneous image registration and HR image reconstruction. The main contribution of this paper is twofold. First, the proposed method integrates image registration and SR into a single estimation process. As opposed to the current two-stage SR methods that perform registration on the LR images, the image registration in the proposed method is performed iteratively using the progressively estimated HR image. This is more promising as more accurate motion parameters can be determined, thereby enhancing the performance of the HR reconstruction. Further, the new method can overcome the shortcoming of the iterative AM framework, as the cost function is not projected onto the image and motion parameter domains one at a time. Instead, an iterative scheme based on a nonlinear least squares (NLS) method is developed to estimate the motion parameters and the HR image simultaneously. Although there is no guarantee of global convergence through a complete mathematical study, various experiments have demonstrated that our algorithm is effective in achieving satisfactory convergence and results. The second contribution of this paper is that a more flexible motion model that consists of translational and rotational motion is developed. As opposed to the translational motion model used in other SR methods, the adopted motion model is more realistic. It is noted that with this new model, a more challenging problem arises, as the problem is no longer linear with respect to some motion parameters. This paper will propose an NLS method to address this issue by deriving the Jacobian matrix for our SR problem. Experimental results show that the proposed method is effective in performing image registration and SR for simulated, as well as real-life, images. The rest of this paper is organized as follows. The problem formulation of image SR taking account of registration errors is introduced in Section II. An iterative algorithm using the NLS method is developed in Section III. Experimental results on simulated and real-life images are presented and discussed in Section IV. A brief conclusion is given in Section V.
2831
II. PROBLEM FORMULATION Let us consider the LR image modeling. The th acquired LR image of sized , can be modeled by rotating the HR image of sized by , shifting it by a translational vector , blurring the rotated and shifted image by a point-spread function (PSF) , then down-sampling it to the resolution of the observed LR image by a decimation factor of . The process can be expressed as [7]
(1) where is the 2-D convolution operator, is the down-sampling operator, and denotes the additive white Gaussian noise represents the camera lens blur in each LR image. (AWGN). represents the effect of the spatial integration of light intensity over a square surface region to simulate image acquisition takes the form of a uniform PSF of the sensors. Therefore, , where is the decimation factor that relates with support the observed LR and the desired HR images. Here, we consider the decimation factors in the horizontal and vertical directions to be identical. Then the decimation factor can be represented . In this work, it is taken that the by is known. The SR formulation in (1) can lens condition then be expressed in a matrix-vector form as (2) where
denotes the lexicographically ordered original image and are the vectors representing the discrete, concatenated and lexicographically is formed by ordered and , respectively. The matrix nonlinear, differentiable functions of an unknown motion parametric vector . Without loss of generality, we have assumed the first LR image to be the referenced image. Hence, can be written as
.. .
.. .
.. .
.. .
where is the down-sampling operator which is identical for all denotes the corresponding matrix constructed LR images. . represents the geometric from motion operator for the th LR image. In this work, we consider the initial estimated motion parameters contain some errors. This is a more realistic assumption, as accurate registration for SR is difficult to achieve especially at the beginning of the algorithm. The objective of image SR, in this context, is to reconstruct the HR image from observed LR images with the unknown motion parametric vector . As the first LR image is
2832
used as the reference, we need to estimate motion parameters and the HR image.
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 16, NO. 11, NOVEMBER 2007
unknown
III. DEVELOPMENT OF AN ITERATIVE SR ALGORITHM USING A NONLINEAR LEAST SQUARES METHOD A. Iterative Simultaneous HR Estimation and Registration Current disjoint two-stage SR methods perform image registration on the LR images, followed by HR reconstruction. The disadvantage of these methods is that they rely on the rough initial registration heavily, hence reducing the performance of subsequent HR reconstruction. To address this difficulty, we propose an effective iterative algorithm that integrates image registration and SR into a single estimation process. The motion model that is considered in this work includes both translation as well as rotation. A challenge arising as a is no longer result of this is that the motion operator Toeplitz. Thus, traditional linear least squares methods cannot address image SR effectively [19]. In view of this, an NLS-based method is proposed. In this section, we will explain how the nonlinear parametric estimation problem is formulated and solved to simultaneously estimate the motion parameters and the HR image. Generally, the SR image reconstruction is an ill-posed problem. To address this issue, a regularization approach based on total variation (TV) technique in [20]–[23] is adopted in this paper. Given the imaging model (1), the estimate of the HR image and the unknown motion vector can be obtained by minimizing the following cost function:
(3.1) (3.2) where denotes the and vectors, where
norm. are defined as the residual and . In this equation, the first term in (3.1) represents the data fidelity of the estimate and with respect to their original values. The second term in (3.1) is a regularization functional that introduces stability into the solution. is the regularization parameter, which provides a compromise between the first and second term. To choose , we follow the algorithm in [22] to obtain an order-of-magnitude estimate. The basic idea behind this method is that the . Then original image should satisfy the equation the equation is solved using the least squares method. Due to its efficiency in edge preservation and noise suppression, TV technique [22] is used in the second term, which is given as
(4)
defined as and , respectively. Here, is the regularization constant [24], which is required in the smooth regions where the absolute values of the gradients in the and directions are zero or very close to zero. is constructed using the half-quadratic scheme in [7] and [22]. in Appendix A. We have provided a brief explanation of . Its pseudo-decomposition is given by In the proposed method, it is unnecessary to compute explicitly during the minimization process. This will become clear in Section III-B. As the proposed method adopts the motion model comprising the translation and rotation, the minimization problem in (3.2) is linear with respect to but nonlinear with respect to . In order to solve this optimization problem, we develop an NLS-based approach to estimate the motion parameters and the HR image simultaneously. We extend the principle of the nonlinear parametric estimation algorithm in [25] to derive a linear approxi. Let represent a small change in the HR mation for a small change in the motion vector . Ignoring image and since its value is generally the second order term small with small and , the residual vector can and as follows: be linearized with respect to
where
and
are
(5) is the Jacobian of with respect to . We where in Section III-B. Therewill discuss how to construct fore, given the current estimate of the HR image and the motion vector , the minimization problem in (3.2) can be modified as
(6) into Compared with (3.2), we have added a new term (6). This is the regularization functional that introduces sta. is an adaptive regularization mability into the solution , where is the regulartrix, which is given by ization parameter in the motion parameter domain and is a identity matrix. is chosen similar to the idea given in [19] to ensure the stability of the solutions. By using the proposed method, the original problem for the direct estimation of and by minimizing (3.2) has been transformed into the minimization problem for the increment and in (6). The proposed iterative algorithm to perform simultaneous image registration and HR reconstruction is summarized in Table I.
HE et al.: NONLINEAR LEAST SQUARE TECHNIQUE FOR SIMULTANEOUS IMAGE REGISTRATION
TABLE I SUMMARY OF THE PROPOSED ALGORITHM
2833
enhancing the performance of the HR reconstruction. Further, the new method can overcome the shortcoming of the iterative AM framework as the cost function is not projected onto the image and the motion parameter domains one at a time. Instead, an iterative NLS-based scheme is developed to estimate the motion parameters and the HR image simultaneously and progressively. The theoretical justification for our proposed algorithm is presented in Appendix B. B. Derivation of the Jacobian Matrix The main challenge in the development of the NLS method lies in the derivation and computation of the Jacobian matrix . is the Jacobian of with respect to . To minimize the cost function (6), has to be constructed explicitly. It should be noted that due to the inclusion is no longer of rotational motion, the motion operator is now nonlinear with respect to . Toeplitz. Thus, In other words, we cannot find an equivalent matrix such that . Therefore, we propose the following technique to solve the problem. can be As each LR observation is independent, written as .. .
..
.
.. .
(9)
where is the Jacobian of with respect . As the first LR image is used as the referto ence, we only need to estimate the unknown motion parameters images. Using the chain rule, can be exfor pressed as follows:
(10) During the initialization, the HR image is reconstructed by minimizing (3.1) based on the initialized . The minimization problem is equivalent to solving (7) in Table I. The closedin Table I requires inversion of the matrix form solution , which is computationally intensive. To solve this problem, a numerical approach using conjugate gradient (CG) optimization is adopted. Similarly, at step 3, it . Hence, the CG optimizais difficult to invert the matrix tion method is again adopted due to its fast convergence. In this work, the dimension of the unknown motion parametric vector is much smaller than the dimension of the unknown HR image . Hence, its computational cost is almost similar to the traditional SR algorithms where the estimated is considered to be accurate or known a priori. As opposed to the current two-stage SR methods that perform registration on the LR images in the first stage, the image registration in the proposed method is performed iteratively using the progressively estimated HR image. This is more promising as more accurate motion parameters can be determined, thereby
It is not easy to obtain directly since is nonlinear with respect to . To solve the problem, we propose to use bilinear interpolation to derive the relationship beand . tween We first introduce the formation of relative shifted HR images . The possible positions between their grids are shown in Fig. 1. We define as the coordinates of the reference as the coordinates of the th shifted HR HR grid and , the coordinate change, is defined as grid.
(11) From Fig. 1, it can be seen that there are four possible relative coordinate positions between and . The shifted pixels in the th HR grid with respect to the top-left, top-right, bottom-left, and bottom-right pixels in the reference may consist HR grid are shown. It is noted that
2834
Fig. 1. Four possible relative positions between (x HR grid.
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 16, NO. 11, NOVEMBER 2007
;y
) and (x
;y
); (a)–(d) denote the pixel (x
of both pixel-level and subpixel-level shifts. For simplicity of demonstration, the HR grids in Fig. 1 show the subpixel shift only. The pixel-level shift can be taken into consideration by operator in (14). Using the matrix-vector form, we the floor can express (11) as
) in the k th HR grid with respect to (x
;y
) in the reference
top-left, top-right, bottom-left, and bottom-right locations, reas the distance vector between spectively. We denote and the pixel at the top-left position in the refthe pixel erence HR grid. When examining the four possible coordinate and can be seen to satisfy the positions in Fig. 1(a)–(d), following relationships:
(12) where
;y
(14) where floor denotes the operator rounding the number to the nearest integer less than or equal to itself. Using bilinear intercan be obtained by polation, the shifted and rotated
where and are the vectors representing the discrete and lexicographically ordered displacement and , respectively. and vectors are vectors of all ones and zeros, respectively. From (12), is linear with respect to . it is clear that can be obtained as
(15) is an entry-by-entry multiplication operator. and are the vectors representing and , the lexicographically ordered to denote respectively. For simplicity, we will use for the rest of the derivation. Using (14) and (15), can then be written as where
(13) Next, we use bilinear interpolation to describe the depenand . The aim is to dedency between termine . The pixel in the th HR grid is determined by four neighboring pixel values and as shown in Fig. 1. These four points are the reference HR grid points surrounding the pixel at the
(16)
HE et al.: NONLINEAR LEAST SQUARE TECHNIQUE FOR SIMULTANEOUS IMAGE REGISTRATION
as equals to the identity matrix due to (14). It can be shown that the same (16) can be obtained for all the cases in Fig. 1(a)–(d). Combining (13) and (16), can then be obtained as
2835
vector and the peak signal-to-noise ratio (PSNR) for the reconstructed HR image. They are defined as follows: (19) (20)
(17) Substituting (17) into (10), we can obtain a simple expression which is . The Jacobian matrix in (9) for can finally be integrated into the scheme iteratively as highlighted in Table I. Finally, we can formulate (8) in Table I by using the ex. plicit equations in (18) to compute the increment in (18) is a It should be noted that the matrix sparse . matrix and the vector in (18) is To compute the closed-form solution for this equation, we need to invert the matrix , which is computationally intensive. To address this problem, we employ the conjugate gradient (CG) method as it can achieve fast convergence when compared with the other methods. After solving (8) in step 3 of Table I, the estimates and can then be updated using the new increment and . The algorithm will terminate until convergence or a maximum number of iterations is reached (18) where (see the equation shown at the bottom of the page). IV. EXPERIMENTAL RESULTS In this section, we will study the performance of the proposed method and compare it with other methods. We conducted various Monte-Carlo simulations to compare the results obtained using the proposed method with those of the two other SR algorithms, namely two-stage disjoint SR method similar to [6] and [7] and AM SR method similar to [10]. Finally, experiments using real-life images are presented to illustrate the effectiveness of the proposed method. To evaluate the performance, we used the following well-known metrics in this work: normalized mean square errors (NMSE) for the identified motion parametric
..
where “ ” denotes the currently computed estimate. Generally, good algorithm is reflected by low NMSE and high PSNR. Nevertheless, the best performance measure remains human inspection of the reconstructed HR images. A. Convergence Study In this section, we conducted various experiments to study the effect of the initial choices of motion parameters on the convergence of the proposed method. We conducted the experiments using the 50 50 “Building” and “Tree” images, shown LR images for each HR image, in Fig. 2. To generate the HR images were rotated by different randomly chosen degrees,1 shifted angles of by randomly chosen subpixel translations of (0,0), (0.79,0.96), (0.52,0.88), (0.17,0.98), (0.27,0.25) pixels, and blurred by 2 2 uniform kernel before subsampling by a decimation . The images were further degraded by AWGN factor of to produce a signal-to-noise ratio (SNR) at 35 dB. The convergence performance was evaluated using different initial motion and as the initial estimate parameters. We denote and true value for the rotation angle , respectively. and represent the initial estimate and true value for the shift , respectively. Then, we initialized the motion parameters by using the following:
(21) where denotes the estimated angle error, and repre. Using the existing sents the percentage of the error in registration algorithms such as [8], it is observed that is 1When rotations are performed in our simulations, the pixel values on the image corners are filled with values of the nearest pixels lying on the image border. After that, the rotated image is cropped in such a way that it contains the center of the image, and the cropped image has the same dimension as the original image.
.. .
.
.. .
2836
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 16, NO. 11, NOVEMBER 2007
Fig. 2. Test images (a) Original “Building” HR image. (b) Original “Tree” HR image.
Fig. 4. NMSE of the estimated motion parametric vectors with different initial conditions. (a) “Building” image. (b) “Tree” image.
Fig. 3. PSNR of the reconstructed HR images with different initial conditions. (a) “Building” image. (b) “Tree” image.
usually small and falls within the range of degrees. Therefore, we considered the worst-case scenario and fixed degrees in these experiments. Further, four were used as the errors in the values of initial estimated . It is noted that we randomly chose “ ” or “ ” in (21) for each LR image. In the experiment, we considered that the algorithm had converged if the following convergence criterion was satisfied: (22) We plotted the PSNR of the reconstructed HR images for our algorithm against the number of iterations in Fig. 3. It is observed that the experiments with these initial conditions all converge to the same solution. When the initial estimates are further
away from the true values, the algorithm requires more iterations to converge. This provides empirical evidence on the convergence of the proposed method and an indication on the tightness of the initial estimates that are required to achieve convergence. In order to further illustrate the registration performance, the NMSE of the estimated motion parametric vector is plotted in Fig. 4. From the figures, it is clear that the motion parameters with different initial conditions converge to the true value. These results further reconfirm the PSNR of the reconstructed HR images in Fig. 3. B. SR for LR Images Degraded at Various Noise Levels In this section, we conducted various Monte-Carlo simulations to perform image SR based on multiple noisy LR images at different noise levels. The number of Monte-Carlo simulations was set to 10 at different noise levels. The “Building” image in Fig. 2 was selected as the test image. To generate five different LR images in each simulation, the HR image was rotated by randomly selected angles from a uniform distribution over degrees, shifted by randomly selected subpixel translations from a uniform distribution over [0, 1], and blurred by 2 2 uniform kernel before subsampling by a decimation factor . The LR images were further degraded by AWGN to of produce different levels of SNR (45, 35, 25, 15 dB). Two other SR algorithms, namely two-stage disjoint SR method similar to [6] and [7] and AM SR method similar to [10], together with
HE et al.: NONLINEAR LEAST SQUARE TECHNIQUE FOR SIMULTANEOUS IMAGE REGISTRATION
2837
Fig. 5. PSNR of the reconstructed HR image against the SNR of the LR images. Fig. 7. SR for LR images degraded by AWGN. (a) Original HR image. (b) Sample of the scaled-up LR images. (c) Reconstructed image using the two-stage SR algorithm. (d) Reconstructed image using the AM method. (e) Reconstructed image using the proposed algorithm.
Fig. 6. NMSE of the estimated motion parametric vector against the SNR of the LR images.
the proposed method were applied to the observed LR images to perform image SR. We initialized the motion parameters by using the registration method in [8] for all three methods. The two-stage disjoint SR was implemented by minimizing the cost function in (3.1) with respect to the HR image. The AM method was implemented by projecting and minimizing the cost function in (3.1) with respect to the HR image and the motion parametric vector iteratively. The state-of-the-art AM method that is based on conjugate gradient optimization is used for comparison in this study. Our proposed method was run based on the procedure outlined in Table I. The PSNR of the reconstructed HR images for different SNR noise levels is given in Fig. 5. The PSNR obtained are based on 10 Monte-Carlo simulations. From the figure, it is observed that the proposed method produces the best results among the three methods. The two-stage disjoint SR method has the lowest PSNR as it does not take the impact of inaccurate initial registration into consideration. The proposed method, on the other hand, offers better results than the AM method as the HR image and the motion parameters are estimated simultaneously in the proposed method. This is as opposed to the AM method where
Fig. 8. PSNR of the reconstructed HR image.
the cost function is projected onto the HR image and motion parameter domains, and minimized iteratively. To further compare the registration performance of the three methods, the NMSE of the estimated motion parametric vector for different SNR levels is plotted in Fig. 6. From the figure, it is clear that the proposed method achieves the best estimated motion parameters. This observation further reconfirms the PSNR results given in Fig. 5. We also compared the reconstructed HR images of an experiment at 35-dB noise level using the proposed and AM methods. The results are given in Fig. 7. From the figures, it can be seen that the reconstructed HR images using the AM method contain more artifacts than those of the proposed method. Further, Figs. 8 and 9 show that the PSNR of the reconstructed HR image and the NMSE of the estimated motion parametric vector using the proposed method are superior to the AM method. Due to the space constraint, we are unable to show the results for other images but similar observations can be made when other test images are used.
2838
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 16, NO. 11, NOVEMBER 2007
Fig. 9. NMSE of the estimated motion parametric vector.
Finally, we will provide an analysis on the computational complexity of the proposed method. As the exact number of arithmetic operations required is difficult to determine for the proposed method, we will use computational time to show the complexity of the proposed method. Further, we have used the AM SR method as the benchmark for comparison. In order to provide a good indication of the computational time, the above 35 dB is used. Monte-Carlo simulation experiment at The simulation environments are given as follows: Windows XP, MATLAB 7.1, CPU P4 3.4 GHz, and 1-G RAM. The total computational time, averaged over ten runs, is 37.7 s for the proposed method, and 45.9 s for the AM method. This shows that the proposed method has better computational time. It is noted that although an average iteration in our algorithm requires more computation than that of the AM method, the total number of iterations required for the proposed method is much less than that of the AM method. This observation can be shown in Figs. 8 and 9, where the proposed method requires much smaller number of iterations to achieve convergence. It should also be noted that the proposed method is implemented in MATLAB for these Monte-Carlo simulations. MATLAB is an interpreter-programming platform with lower processing speed. The computational time of the experiments can be improved if the algorithm is implemented in a compiler language such as C. C. Experiments on Real-Life Images The real-life experiment was conducted by capturing five “bookshelf” images using a web camera with relative translations and rotations. The LR images are shown in Fig. 10(a) and a sample of the scaled-up LR images is shown in Fig. 10(b). We chose a decimation factor of 2 so that the estimated HR image will have twice the resolution of the LR images. The registration algorithm in [8] was again used to estimate the initial shifts and rotations among the LR images. Next the two-stage SR algorithm, the AM method and the proposed method were run to perform simultaneous image registration and SR image reconstruction. The reconstructed HR images using the three methods are given in Fig. 10(c)–(e), respectively. To provide a
Fig. 10. SR for real-life images. (a) Five LR images. (b) Sample of the scaled-up LR images. (c) Reconstructed image using the two-stage SR algorithm. (d) Reconstructed image using the AM algorithm. (e) Reconstructed image using the proposed algorithm. (f) Ground truth. (g)–(j) Selected enlarged region of (c)–(f), respectively.
fair comparison, an image with the resolution of the HR image was captured to be used as the ground truth in Fig. 10(f). From Fig. 10(c)–(e), it is observed that the considerable clarity of images has been recovered by both the AM and the proposed methods. Further, it can be seen that the result by our proposed method has less artifacts than that of the AM method, in particular near the words. A selected region of the reconstructed HR images by the three methods is enlarged in Fig. 10(g)–(i) for closer examination. Comparison reveals that our approach is superior in handling real-life image SR, as it is able to estimate motion parameters accurately, leading to superior HR image reconstruction. V. CONCLUSION This paper presents a new algorithm to integrate image registration into image SR. As opposed to the methods that treat image registration and HR reconstruction as disjoint processes, the new framework enables image registration and HR reconstruction to be estimated and improved progressively. Further, unlike most algorithms that focus on translational
HE et al.: NONLINEAR LEAST SQUARE TECHNIQUE FOR SIMULTANEOUS IMAGE REGISTRATION
motion model, the proposed framework adopts a more generic motion model that includes both translation as well as rotation. An iterative scheme is developed to solve the arising nonlinear least squares problem in order to perform simultaneous image registration and HR reconstruction. Experimental results show that the proposed method is effective in performing image registration and SR for simulated as well as real-life images.
APPENDIX A CONSTRUCTION OF In this Appendix, the construction of in (4) will be explained. In (4), can be divided into two parts as follows:
2839
The left-hand side of (24) can be computed as follows:
(26) where is given as . Then can be computed by . The extension to the direction follows the similar derivation. Then, can be ob, where is constructed tained by: by using . The advantage of using this half-quadratic scheme is that it can effectively avoid the difficulty of solving the nonlinear partial differential equations (PDEs) arising out of the TV norm. Further information on this scheme can be found in [22].
APPENDIX B JUSTIFICATION OF THE ALGORITHM (23) defined as and , respectively. Here, is the regularization constant, which is required in the smooth regions where the absolute values of the gradients in the and directions are zero or very close to zero. Based on the guideline for choosing in [24], we have adopted in our experiments. It is observed that in our experiments, the reconstructed HR images are relatively insensitive to variation in the parameters so long as they are within an order of . We will first explain how to construct magnitude of based on . should satisfy the following condition: where
and
are
(24) Adopting the fixed-point (FP) scheme in [26], the coefficient can be calculated by using the estimated HR image in the previous iteration of our algorithm. Suppose the values at each lexicographical location are given by of , then can be expressed in the matrix-vector form as follows:
In this Appendix, a brief theoretical justification for the algorithm is presented. As discussed in Section III, our joint framework for simultaneous image registration and SR can be stated in terms of minimizing the following differentiable function:
(27) The solution of the minimization problem can be determined . This condition can be by solving expressed as
(28) The aim is to show that our proposed method can satisfy this condition when the convergence is reached. In Section III, the original problem for direct estimation of and has been transformed into finding the increment and by minimizing (6). We first show that the minimization problem is equivalent to solving (8) in Table I. It should be noted that the sufficient condition for this equivalence is that the Hessian matrix of the minimized cost function is positive def, it is clear that inite [27]. As the Hessian matrix is is positive definite. Thus, we can obtain the minimum of (6) by solving (8) in our method. Further, when the convergence of will be zero, the algorithm is reached, the vector if and only if, in (18) vanishes. Combining the expression in (18) and (28), the following can be obtained at the convergence of the algorithm:
where (29) .. .
.. .
..
.
..
.
.. .
(25)
This means that the solution satisfying (28) is obtained when the iterative algorithm reaches convergence. Experimental results in Section IV support this theoretical analysis.
2840
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 16, NO. 11, NOVEMBER 2007
REFERENCES [1] G. Jacquemod, C. Odet, and R. Goutte, “Image resolution enhancement using subpixel camera displacement,” Signal Process., vol. 24, pp. 139–146, Jan. 1992. [2] N. K. Bose and K. J. Boo, “High-resolution image reconstruction with multisensors,” Int. J. Imag. Syst. Technol., vol. 9, pp. 294–304, Dec. 1998. [3] S. C. Park, M. K. Park, and M. G. Kang, “Super-resolution image reconstruction: A technical overview,” IEEE Signal Process. Mag., vol. 20, no. 5, pp. 21–36, May 2003. [4] M. Elad and A. Feuer, “Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images,” IEEE Trans. Image Process., vol. 6, no. 12, pp. 1646–1658, Dec. 1997. [5] R. R. Schultz and R. L. Stevenson, “Extraction of high-resolution frames from video sequences,” IEEE Trans. Image Process., vol. 5, no. 6, pp. 996–1011, Jun. 1996. [6] S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, “Advances and challenges in super-resolution,” Int. J. Imag. Syst. Technol., vol. 14, pp. 47–57, Aug. 2004. [7] Y. He, K.-H. Yap, L. Chen, and L.-P. Chau, “Blind super-resolution image reconstruction using a maximum a posteriori estimation,” in Proc. IEEE Int. Conf. Image Processing, Atlanta, GA, Oct. 2006, pp. 1729–1732. [8] P. Vandewalle, S. Susstrunk, and A. Vetterli, “A frequency domain approach to registration of aliased images with application to super-resolution,” EURASIP J. Appl. Signal Process. 2006, Article ID 71459. [9] P. Vandewalle, L. Sbaiz, J. Vandewalle, and M. Vetterli, “Super-Resolution from unregistered and totally aliased signals using subspace methods,” IEEE Trans. Signal Process., vol. 55, no. 7, pp. 3687–3703, Jul. 2007. [10] R. C. Hardie, K. J. Barnard, and E. E. Armstrong, “Joint MAP registration and high-resolution image estimation using a sequence of undersampled images,” IEEE Trans. Image Process., vol. 6, no. 12, pp. 1621–1633, Dec. 1997. [11] M. K. Ng, J. Koo, and N. K. Bose, “Constrained total least-squares computations for high-resolution image reconstruction with multisensors,” J. Imag. Sci. Technol., vol. 12, pp. 35–42, Jan. 2002. [12] G. Golub and V. Pereyra, “Separable nonlinear least squares: The variable projection method and its applications,” Inst. Phys. Inv. Probl., vol. 19, pp. R1–R26, 2003. [13] N. A. Woods, N. P. Galatsanos, and A. K. Katsaggelos, “Stochastic methods for joint registration, restoration, and interpolation of multiple undersampled images,” IEEE Trans. Image Process., vol. 15, no. 1, pp. 201–213, Jan. 2006. [14] J. Chung, E. Haber, and J. Nagy, “Numerical methods for coupled superresolution,” Inv. Probl., vol. 22, pp. 1261–1272, 2006. [15] D. Robinson, S. Farsiu, and P. Milanfar, “Optimal registration of aliased images using variable projection with applications to super-resolution,” The Computer J., Apr. 2007, Invited paper, DOI:10.1093/comjnl/bxm007. [16] P. Vandewalle, L. Sbaiz, M. Vetterli, and S. Susstrunk, “Super-Resolution from highly undersampled images,” in Proc. Int. Conf. Image Processing, Genova, Italy, Sept. 2005, pp. 889–892. [17] E. S. Lee and M. G. Kang, “Regularized adaptive high-resolution image reconstruction considering inaccurate subpixel registration,” IEEE Trans. Image Process., vol. 12, no. 7, pp. 826–837, Jul. 2003. [18] H. He and L. P. Kondi, “An image super-resolution algorithm for different error levels per frame,” IEEE Trans. Image Process., vol. 15, no. 3, pp. 592–603, Mar. 2006. [19] H. Fu and J. Barlow, “A regularized structured total least squares algorithm for high-resolution image reconstruction,” Linear Algebra Appl., vol. 391, pp. 75–98, Nov. 2004. [20] L. Rudin, S. Osher, and E. Fatemi, “Nonlinear total variation based noise removal algorithms,” Phys. D, vol. 10, pp. 259–268, Nov. 1992. [21] T. F. Chan, S. Osher, and J. Shen, “The digital TV filter and Nonlinear Denoising,” IEEE Trans. Image Process., vol. 10, no. 2, pp. 231–241, Feb. 2001. [22] F. Sroubek and J. Flusser, “Multichannel blind iterative image restoration,” IEEE Trans. Image Process., vol. 12, no. 9, pp. 1094–1106, Sep. 2003. [23] S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, “Fast and robust multiframe super resolution,” IEEE Trans. Image Process., vol. 13, no. 10, pp. 1327–1344, Oct. 2004.
[24] C. Vogel and M. Oman, “Iterative methods for total variation denoising,” SIAM J. Sci. Comput., vol. 17, pp. 227–238, Jan. 1996. [25] J. B. Rosen, H. Park, and J. Glick, “Structured total least norm for nonlinear problems,” SIAM J. Matrix Anal. Appl., vol. 20, pp. 14–30, Jan. 1998. [26] T. F. Chan and W. Chiu-Kwong, “Total variation blind deconvolution,” IEEE Trans. Image Process., vol. 7, no. 3, pp. 370–375, Mar. 1998. [27] M. T. Heath, Scientific Computing: An Introductory Survey. New York: McGraw-Hill, 2001.
Yu He received the B.Eng. degree in precision instrument and optoelectronics engineering from Tianjin University, Tianjin, China, in 2002. He is currently pursuing the Ph.D. degree at Nanyang Technological University, Singapore. He was a Research and Development Engineer with SAMSUNG Electronic Co., Ltd. (color TV) for two years. His research interests include image/video processing, computer vision, and image/video deconvolution, and super-resolution.
Kim-Hui Yap (S’99–M’03) received the B.Eng. and Ph.D. degrees in electrical engineering from the University of Sydney, Sydney, Australia, in 1998 and 2002, respectively. Currently, he is a faculty member at Nanyang Technological University, Singapore. He has worked on projects for image processing and computer vision, including image restoration, segmentation, super-resolution, and reconstruction. He is also the Group Leader of Content-Based Analysis for the Center for Signal Processing, Nanyang Technological University. His works cover image/video content analysis and understanding, media processing, image/video indexing, retrieval, and summarization. He has numerous publications in various international journals, book chapters, and conference proceedings. He is also the Editor of the book Intelligent Multimedia Processing with Soft Computing (Springer-Verlag, 2005). His main research interests include image/video processing, media content analysis, computer vision, and computational intelligence.
Li Chen (S’03–M’06) received the B.Eng. degree in industrial automation from the Wuhan University of Technology, Wuhan, China, in 1999, the M.Eng. degree in control theory from the Huazhong University of Science and Technology, Wuhan, China, in 2002, and the Ph.D. degree in information engineering from Nanyang Technological University, Singapore, in 2006. He was a Research Associate at Nanyang Technological University in 2006. Currently, he is a faculty member at the Wuhan University of Science and Technology. His research interests include image processing, computer vision, and statistical pattern recognition.
HE et al.: NONLINEAR LEAST SQUARE TECHNIQUE FOR SIMULTANEOUS IMAGE REGISTRATION
Lap-Pui Chau (SM’03) received the B. Eng. degree with first class honors in electronic engineering from Oxford Brookes University, Oxford, U.K., and the Ph.D. degree in electronic engineering from the Hong Kong Polytechnic University, Hong Kong, in 1992 and 1997, respectively. In June 1996, he joined Tritech Microelectronics as a Senior Engineer. In March 1997, he joined the School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore, as a Research Fellow, then he became an Assistant Professor, and, currently, he is an Associate Professor. His research interests include streaming media, computer animation processing, and VLSI for media signal processing. Dr. Chau is involved in the organization committees of international conferences, including the IEEE Asia-Pacific Conference on Circuits and Systems (APCCAS 2006), the IEEE International Conference on Image Processing (ICIP 2004), and International Conference on Information, Communication, and Signal Processing (ICICS 2001). He is also involved as a Track Chair/Co-Chair on the technical committees for some international conferences, including the International Symposium on Circuits and Systems (ISCAS 2008, ISCAS 2007, ISCAS 2006, ISCAS 2005), the Pacific-Rim Conference on Multimedia (PCM2007, PCM 2003), and the International Conference on Information, Communication and Signal Processing (ICICS 2007, ICICS 2005, ICICS 2001). In addition, he also served as a technical program committee member regularly for many international conferences. He is a Secretary of the Technical Committee on Circuits and Systems for Communications (TC-CASC), a member of Technical Committee on Multimedia Systems and Applications (TC-MSA), a member of Technical Committee on Visual Signal Processing and Communications (TC-VSPC), and a member of Technical Committee on Circuits and Systems for Communications (TC-CASC) of the IEEE Circuits and Systems Society. He also served as a member of Singapore Digital Television Technical Committee from 1998 to 1999. He has served as an Associate Editor for IEEE TRANSACTIONS ON MULTIMEDIA and is currently serving as an Associate Editor for IEEE TRANSACTIONS ON BROADCASTING.
2841
