Comparison of Subpixel Image Registration Algorithms

Comparison of Subpixel Image Registration Algorithms R.R. Boye and C.L. Nelson Sandia National Laboratories, Albuquerque, NM 87185 ABSTRACT Research into the use of multiframe superresolution has led to the development of algorithms for providing images with enhanced resolution using several lower resolution copies. An integral component of these algorithms is the determination of the registration of each of the low resolution images to a reference image. Without this information, no resolution enhancement can be attained. We have endeavored to find a suitable method for registering severely undersampled images by comparing several approaches. To test the algorithms, an ideal image is input to a simulated image formation program, creating several undersampled images with known geometric transformations. The registration algorithms are then applied to the set of low resolution images and the estimated registration parameters compared to the actual values. This investigation is limited to monochromatic images (extension to color images is not difficult) and only considers global geometric transformations. Each registration approach will be reviewed and evaluated with respect to the accuracy of the estimated registration parameters as well as the computational complexity required. In addition, the effects of image content, specifically spatial frequency content, as well as the immunity of the registration algorithms to noise will be discussed. Keywords: Super resolution, image registration, aliasing, subpixel

INTRODUCTION Multiframe resolution synthesizes a high resolution (HR) image from several undersampled low resolution (LR) images.1-4 Multiframe resolution has been comprehensively investigated for applications such as medical imaging, remote sensing, computer vision and video image enhancement. Unless the relative shifts of the low resolution images are known a priori, they must be determined as a first step in a superresolution algorithm. Errors in the estimation of these registration parameters can severely limit the ability of the system to properly determine the high resolution image. The work motivating our review of approaches for registering images is the development of a small form factor imaging system. Similar to the TOMBO 5,6 system, the proposed optical system uses an imaging optic with a drastically reduced focal length as compared to conventional imaging systems such as cell phone cameras. The resulting image size scales accordingly; unfortunately, the available detector pixel sizes can not provide the same scaling.7,8 In order to provide images with a useful amount of information, an array of imagers will provide several images which are then fused together to provide a single higher resolution image. For the imager sizes being considered, the required increase in resolution from the raw images to the final high resolution output needs to be 8x or better. The first implementation of the system only requires the determination of global lateral shift parameters. It is anticipated that subsequent implementations will require the determination of a global rotation as well. The next section presents an analysis of the effects of registration error on resulting high resolution images to help quantify the accuracy required of the registration algorithms. Section three presents the results of candidate algorithms against several test images. Section four discusses the effect of noise on the accuracy of these approaches. Finally, the conclusions of this study are presented as well as directions that will be further investigated.

Computational Imaging VII, edited by Charles A. Bouman, Eric L. Miller, Ilya Pollak, Proc. of SPIE-IS&T Electronic Imaging, SPIE Vol. 7246, 72460X · © 2009 SPIE-IS&T · CCC code: 0277-786X/09/$18 · doi: 10.1117/12.810369

SPIE-IS&T/ Vol. 7246 72460X-1

EFFECT OF REGISTRATION ERROR When comparing schemes for registering images, it is prudent to first understand what level of accuracy is required. This would avoid the pursuit of unnecessary levels of precision at the expense of increased computational complexity or speed. Simulations using noise free, down sampled images without blurring due to optics were used as inputs to a simple reconstruction. By ignoring optical blur and noise, the effect of registration error on the reconstructed image is effectively isolated. Errors with a Gaussian distribution were added to the known registration offsets of the individual images. The modified positions of the low resolution pixels were mapped to a high resolution grid and interpolated to provide a high resolution image. The error between the reconstructed image and the original were compared using the mean squared error (MSE) averaged over several trials for a given level of registration error. Three test images, see Figure 1, were used to test the effect of differing spatial frequency content and provide qualitative image results. The simulation used sixty four low resolution images that were sampled at one eighth the resolution of the final image with evenly spaced lateral shifts. Figure 2 shows an example LR image for each of the corresponding ideal images from Figure 1. The effects of aliasing are clearly displayed for “Concentric”, Figure 2b as well as the obvious degradation to “Face”, Figure 2a, and “Text”, Figure 2c.

OOMC PCROSYSTEMS TECNOLOG

TQC CO SflI&5 1EHNOOlU

AO!sC

.o*c

'flA

TIáE vt.c WE PP+OTONIC IcRSTEI TOUOGII5

OTOPdC IICROSYSTMS TECHNoLoG

(a) (b) (c) Figure 1. Test images used for determining level of acceptable registration error, (a) “Face”, (b) “Concentric” and (c) “Text”.

I

I

!Cl. riiiJ.cJ I -L?L?.I IFIL.V

,W.

I

49iG L' 4I

p

. 1.r. L.LL?.I I JI1W I

(a) (b) (c) Figure 2. Downsampled versions of the corresponding images from Figure 1 with a downsampling ratio of 8.

The change of MSE with registration error for the three cases is shown in Figure 3. The low spatial frequencies dominated in the “Face” image leading to a lower MSE than the other two cases. The “Concentric” image contains high spatial frequencies as well as gray levels and had the highest MSE. The “Text” image was essentially black and white, but the high spatial frequencies of the text resulted in an MSE almost as high as the “Concentric” image. In all cases, the change in MSE is effectively linear so as

SPIE-IS&T/ Vol. 7246 72460X-2

long as the computing complexity is not increasing faster than linearly, there is a benefit to increased accuracy. Of course, depending on the application, there may be more qualitative criteria that reduce the ultimate need for registration accuracy. 100

RMS Error

10

1

Face Concentric Text 0.1

0.01 0.0001

0.001

0.01

0.1

1

10

Absolute Error [LR pixels]

Figure 3. MSE for test images with respect to registration errors as measured in LR pixels.

Three different reconstructions of Figure 1a are shown in Figure 4. The first is easily identified while the third is not. While the image in Figure 4b is not ideal, it may very well be adequate for the application. For instance, the person can certainly be indentified as a young boy, but cannot be uniquely identified with absolute certainty. In contrast, Figure 5 shows a reconstruction of Figure 1c. Higher levels of accuracy must be achieved if the smallest font needs to be properly imaged. An accuracy of 0.0232 LR pixels, more than three times better than Figure 4a, was required to determine properly the inner most lines of text (which differ from the others). This makes intuitive sense since text is comprised of higher spatial frequencies requiring higher resolution enhancement than the image of a face. Alternatively, reconstructions of Figure 1b provide a quantitative method for determining the required accuracy. Figure 6 shows radial plots from Figure 1b and two reconstructions with different amounts of registration error. In this manner, a required accuracy for a particular application could be determined.

(a) (b) (c) Figure 4. Reconstructions of test image “Face” with average absolute registration errors of (a) 0.0724, (b) 0.148 and (b) 0.293 LR pixels.

SPIE-IS&T/ Vol. 7246 72460X-3

IOTONIC MICROSYSTEMS TECHNOLOGI PHQTONC MICROSYSTEMS TECH NOLQGES PROTcViIC MICROSYSTEMS TECH\OLOGES F oic*ic M IC RQSrEM$ TECHr4OLOO

4O'OC

ROWSTM TECI3NOLO

5

PWTQ C VKROSYSTEMS TFCI*OLOCE

44tt.

'-i.1 IL .

m.s.c L$ocIC,,. 1E 4Ib.QE *CW V

PNK tJ 5Y5T/S $LOF PHTOI IpKRO5V'FtM 1EGW4OLQCE PHOTONC ICRO5YSTEMS 14!KLOI E3 PI'OTOM MICROSYSEMS TEHNOLQGIES PHOTON IC CROSYTEWS TECHNOLOGIES

PHOTONIC MICROSYSTEMS TECHNOLOGIES

IOTONIC MICROSYSTEMS TECHNOLOGI Figure 5. Reconstruction of test image “Text” with registration errors of 0.0238 LR pixels. Note that the center lines (smallest text) are slightly different than the others.

(a)

(b) Figure 6. Radial plots from Figure 1b (solid) and two reconstructions with average registration error of (a) 0.1524 and (b) 0.0390 LR pixels.

SPIE-IS&T/ Vol. 7246 72460X-4

REGISTRATION TESTING Registration algorithms fall into two general classes, spatial 9-11 and frequency domain 12-15 approaches. The spatial domain has the advantage of being able to handle local shifts and image distortions; however, for this application we are specifically concerned with global shifts. Frequency domain approaches have been shown to work quite well for determining global registration parameters; however, as the amount of resolution enhancement increases, the computing complexity increases considerably. Three algorithms have been investigated for this work. One frequency domain algorithm 14 was implemented as well as two spatial domain approaches.10,11 The first algorithm 14 attempts to eliminate errors due to aliasing in the low resolution images by low-pass filtering these images. (In fact, the authors simply limit the application of their algorithm to the low frequencies within the frequency domain.) Rotations are determined by reducing the two-dimensional spectrum using an azimuthal integration and performing a correlation. After this, the lateral shift parameters are found from the phase shifts between the transforms of a reference and shifted image. The first spatial domain algorithm 10 models a shifted image as a perturbation of the reference image. Subtracting the reference image from a Taylor series expansion of the shifted image provides an error measure. Differentiating this error with respect to the shift parameters and setting these partial derivatives to zero provides a set of equations that can be solved for the shift parameters including rotation, if necessary. The final approach 10 uses a gradient based approach similar to the previous method where the Taylor series expansions are solved using a least squares approach and then further refined using a linear bias compensation step. These three approaches were each tested using several test images shown in Figure 7. The first test used images that were undersampled by a factor of four and the results are shown in Table 1.

--2

9a7-

6-

9876

1u1IIIIIIII1IIII 1

II

I

I

=

[=

Figure 7. Test images used for testing registration algorithms: “Horseback”, “Face”, “Concentric”, “Bullseye”, “Horn Resolution”, “Linear Resolution” and “Text Resolution”.

SPIE-IS&T/ Vol. 7246 72460X-5

Image Horseback Face Concentric Bullseye Horn Resolution Linear Resolution Text Resolution

Vandewalle, et.al. (Freq. Domain) 0.0258 0.0210 0.0039 0.0001 0.0002 0.0009 0.0005

Irani and Peleg (Spatial Domain) 0.0098 0.0086 0.0299 0.0071 0.0347 0.0286 0.0631

Davis and Freeman (Spatial Domain) 0.0065 0.0035 0.0095 0.0034 0.0176 0.0084 0.0484

Table 1. Average absolute error in estimated lateral shifts (LR pixels) - undersampling factor of 4.

The frequency domain approach performs the best overall for these cases. It was noted in 14 that images with a lot of directionality in the frequency domain tended to have better results. This can be seen in the results shown here by the excellent performance for the images “Horn Resolution”, “Linear Resolution” and “Text Resolution”. The authors in 14 also comment that their algorithm does not necessarily work well for resolution enhancements greater than approximately four times. Since our application will need a greater level of upsampling to provide useful images, a second test using LR images undersampled by a factor of eight was performed. Table 2 displays the resulting errors from this trial.

Image Horseback Face Concentric Bullseye Horn Resolution Linear Resolution Text Resolution

Vandewalle, et.al. (Freq. Domain) 0.0677 0.0524 0.0446 0.0149 0.0006 0.0102 0.0073

Irani and Peleg (Spatial Domain) 0.0058 0.0034 0.0273 0.0192 0.0096 0.0221 0.0157

Davis and Freeman (Spatial Domain) 0.0033 0.0020 0.0249 0.0073 0.0042 0.0084 0.0091

Table 2. Average absolute error in estimated lateral shifts (LR pixels) - undersampling factor of 8.

For this higher upsampling factor, the final algorithm performs the best. It should be noted that all of the algorithms provide the level of accuracy required for reproducing images such as “Face” with adequate resolution. Since our application will need high upsampling ratios, the spatial domain approaches provide the best overall approach for noise free images.

EFFECT OF IMAGE NOISE An advantage of frequency domain approaches is an inherent level of noise immunity. To insure that the spatial domain approaches still outperform the frequency domain approach, an analysis was done with images containing varying degrees of noise. Down sampled images were formed in the same way as the noise free analysis, then white Gaussian noise was added. These images were then registered using both spatial domain approaches. The results for the images “Face” and “Concentric” are shown in Figure 8a and 8b, respectively. For the “Face” image, both spatial domain approaches work better than the frequency domain approach for signal to noise ratios (SNR) larger than 4. In fact, the approach of 10 is comparable to the frequency domain method for all SNR tried. A similar result can be seen for the image “Concentric” despite the fact that the spatial domain methods did not outperform the frequency domain method as drastically as the previous case. For “Concentric”, the spatial domain methods work well for SNR greater than 6. It is interesting to note, that for very low SNR, the spatial domain method of 11 starts to perform quite poorly; however, for the applications being considered, this is not a realistic level of noise.

SPIE-IS&T/ Vol. 7246 72460X-6

-

':

- -

'

Face - Irani & Peleg Face

Davis & Freeman

Face - Vandewalle (No Noise)

0l a

. .. ., +

..

.. .....,

001

0001 100

10

SNR

(a)

,'

.

Concentric - Irani & Peleg - -

Concentric - Davis & Freeman Concentric - Vandewalle (No Noise)

0l

001

0oI 100

SNR

(b) Figure 7. Effect of noise on accuracy of spatial domain registration algorithms. Results are for (a) “Face” and (b) “Concentric”.

SPIE-IS&T/ Vol. 7246 72460X-7

SUMMARY AND CONCLUSIONS We have presented a comparison of three approaches for registering low resolution images for use within a multiframe superresolution algorithm. Two spatial domain methods and one frequency domain method were compared against several test images for low resolution images that were undersampled by 4x and 8x. For the lower undersampling ratio, the frequency domain method of 14 was shown to be the superior approach. As the undersampling ratio increased to 8x, the spatial domain methods provided better registration accuracy. Even the presence of substantial amounts of noise did not change the conclusion that for larger undersampling ratios, the spatial domain methods were superior with the approach of 11 being the best except for cases of extremely high noise. These conclusions are based on the determination of lateral shift parameters only. Future work will include rotations and that additional requirement may lead to differing results.

ACKNOWLEDGEMENTS Supported by the Laboratory Directed Research and Development program at Sandia National Laboratories. Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United States Department of Energy’s National Nuclear Security Administration under contract DE-AC04-94AL85000.

REFERENCES [1] R.Y. Tsai and T.S. Huang, "Multiframe image restoration and registration," in Advances in Computer Vision and Image Processing, R.Y. Tsai and T.S. Huang, eds., vol.1, 317-339, JAI Press Inc. (1984) [2] S.P. Kim, N.K. Bose and H.M. Valenzuela, "Recursive Reconstruction of High Resolution Image from Noisy Undersampled Multiframes," IEEE Trans. Acoustics Speech and Signal Processing 38(6), 1013-1027 (1990). [3] M. Elad and A. Feuer, "Restoration of a Single Superresolution Image from Several Blurred, Noisy, and Undersampled Measured Images," IEEE Trans. Image Processing 6(12), 1646-1658 (1997). [4] P.M. Shankar, W.C. Hasenplaugh, R.L. Morrison, R.A. Stack and M.A. Neifeld, "Multiaperture imaging," Applied Optics 45(13), 2871-2883 (2006). [5] J. Tanida, T. Kumagai, K. Yamada, S. Miyatake, K. Ishida, T. Morimoto, N. Kondou, D. Miyazaki and Y. Ichioka, "Thin observation module by bound optics (TOMBO): concept and experimental verification," Applied Optics 40(11), 1806-1813 (2001). [6] K. Nitta, R. Shogenji, S. Miyatake and J. Tanida, "Image reconstruction for thin observation module by bound optics by using the iterative backprojection method," Applied Optics 45(13), 2893-2900 (2006). [7] A.W. Lohmann, "Scaling laws for lens systems," Applied Optics 28(23), 4996-4998 (1989). [8] M.W. Haney, "Performance scaling in flat imagers," Applied Optics 45(13), 2901-2910 (2006). [9] D. Keren, S. Peleg and R. Brada, "Image Sequence Enhancement Using Sub-pixel Displacements," Proc. IEEE Comp. Soc. Conf. on Computer Vision and Pattern Recognition (CVPR ’88), June 1988, Ann Arbor, MI, 742-746 (1988). [10] M. Irani and S. Peleg, "Improving Resolution by Image Registration," CVGIP: Graphical Models and Image Processing 53(3), 231-239 (1991). [11] C.Q. Davis and D.M. Freeman, "Statistics of subpixel registration algorithms based on spatiotemporal gradients or block matching," Optical Engineering 37(4), 1290-1298 (1998). [12] L. Lucchese and G.M. Cortelazzo, "A Noise-Robust Frequency Domain Technique for Estimating Planar Roto-Translations," IEEE Trans. Signal Processing 48(6), 1769-1786 (2000). [13] H.S. Stone, M.T. Orchard, E.C. Chang and S.A. Martucci, "A Fast Direct Fourier-Based Algorithm for Subpixel Registration of Images," IEEE Trans. Geoscience and Remote Sensing 39(10), 2235-2243 (2001). [14] P. Vandewalle, S. Susstrunk and M. Vetterli, "A Frequency Domain Approach to Regristration of Aliased Images with Application to Super-resolution," EURASIP Jour. Applied Signal Processing (2006), Article ID 71459 (2006).

SPIE-IS&T/ Vol. 7246 72460X-8

[15] M. Guizar-Sicairos, S.T. Thurman, J.R. Fienup, "Efficient subpixel image registration algorithms," Optics Letters 33(2), 156-158 (2008).

SPIE-IS&T/ Vol. 7246 72460X-9