Class of algorithms for realtime subpixel registration - CiteSeerX

Class of algorithms for realtime subpixel registration Robert W. Frischholz and Klaus P. Spinnler Fraunhofer Institute for Integrated Circuits IIS Wetterkreuz 13, 91058 Erlangen, Germany

ABSTRACT In 1972, Barnea and Silverman presented a new approach to the wide eld of template matching, the SSD{algorithm. Further work has been done to adapt the method to gain subpixel accuracy. Intense investigation of the proposed algorithms led to our new approach: by interpolating the template instead of the reference image, and by applying sort of an error{correction to the resulting subpixel{value, both computation time and accuracy can be improved. Exhaustive experiments with a CCD{camera and various kinds of reference images showed that a maximum error of 10% of the pixel period can be expected. Depending on the kind of image, mean square errors range from 0.4% to 4%.

INTRODUCTION In 1972, Barnea and Silverman presented the SSD{algorithm, a fast way to solve the problem of image registration [1]. Extending it to subpixel accuracy ([2],[3]), nevertheless, increased the computational cost to an amount where real{time applications seemed almost impossible. In this paper we present an adapted algorithm with real{time acceptable speed and even better accuracy. To introduce the reader into the matter, the rst chapter reviews the basic concepts of subpixel{registration. Chapter 2 introduces our new algorithm, including the the-

oretical proof of its reliance. Chapter 3 oers equations for computational costs. A coarse prediction of the expectable accuracy in relation to image noise is given, too. The last chapter describes measurements of accuracy in CCD{ scanned noisy images.

1. SUBPIXEL REGISTRATION A pixel{level template matching algorithm searches for the greatest correlation, i. e. the smallest dierence, of a template (that is a small selected part of the whole image) within an image, and reports the position with an accuracy of  1 Pixel. In many applications, however, it is desirable to increase this accuracy to the fraction of a pixel period. There are principally two ways, each of them using interpolation:

1.1 Interpolating the distance{measure For clarity, we consider the case of 1{ dimensional template matching. Let the original image have N Pixels, the template we search within this image have M Pixels (M p2M (7)

r

Solving the inequation to r, we get an upper limit of the expectable resolution:

sM

r < 2 ATS

(8)

An example: Common templates are 100 Pixels wide, so M = 100. The mean error of the scanned image data is of course hardware{ dependent, but usually around 1, so  = 1:0. The normalized dierence of template amplitude steps is template speci c, a common value is around 5 , so ATS  5:0. Using the developed formula, we compute r = 35.36. That means a resolution of 0.028 Pixels is to be expected. Lots of experiments have proven the reliability of our prediction formula. Let's have a closer look at formula (8). The denominator consists of the midrange noise of the image concerned, and so large noise will decrease accuracy. That was to be expected, of course. The remaining terms represent the point of interest: subpixel p accuracy is direct proportional to factors M and ATS . At rst glance, it seems as if a larger template size might cause better subpixel accuracy. This is, of course, false in most cases. Increasing M will decrease ATS in general. For best results, it is necessary to nd the optimum template size M in conjunction with ATS . We implemented algorithms for that task, and the computed optimum templates have always given better results regarding accuracy than templates choosen by man. So the task of nding the best template and predicting

the expectable accuracy of that template can be almost completely automated (still man has to choose the coarse template location, depending on the part of the image to be registered). We want to stress another consequence arising out of formula (8). The term we denoted by ATS is the mean dierence of template amplitudes. As accuracy is direct proportional to ATS , we can say that the greater the amplitude dierences of the template, the better will be the results of registration. Easily said, the more consecutive amplitudes dier, the more we can achieve with interpolation. Finally, our prediction formula stresses contrast. Figures 3.1.a and 3.1.b illustrate that fact. Figure 3.1.a represents the grey{value{function of a white curtain on black background, scanned with high contrast. Figure 3.1.b shows the same scene, low pass ltered by defocussing the camera's lens. Even though the latter image might seem more pleasing to the reader's eye, its predicted and veri ed subpixel registration accuracy is about 4 times worse than the un ltered one. We consider this experience very important for all registration tasks, because many related articles suggest the defocussing used as a low{pass{ lter to increase subpixel accuracy (stated in [2],[8],[9]). None of them, however, have proven this assumption. With our results, based on theoretical analysis, we cannot agree with them.

6

Intensity 250 200 150 100 50

-

0 0

100

200

300

Pixel

400

500

Fig. 3.1.a Predicted and measured accuracy is 0.01

Page 6

' $ & %

6

Intensity 250

Start

?

200 150

Scan new CCD{Line

100 50 0 0

100

200

300

Pixel

400

-

500

Fig. 3.1.b Predicted and measured accuracy is 0.04

? Compute Pixel Hitpoint

?

Compute distance of templates to the left

3.2 Computational Costs This section will lead to the development of a formula describing the eciency of our class of algorithms. The time spent on computing the shifted template versions is not considered here, because the interpolation can be made prior to all measurements following. We use abbreviations as follows: N image size M template size R resolution factor (i.e. number of interpolated template versions) INP L1

time needed to scan 1 CCD{line time needed to calculate the sum of absolute dierences between template and image for a single position ERR time needed for parabolic error correction (that is solving the term 2(2fmf ??frf ?fl) , see also [7]) r

l

To discuss the overall costs of a single subpixel measure, it is useful to lay out the main steps of the algorithm:

?HHH  HH H HH n y Found better ?

l

?

Compute distances to the right

?

' $ & % ?

Apply error{correction

?

End

Fig. 3.2 Algorithm owchart

Considering the worst case, that is the search for a better subpixel hitpoint to the left failed, and so the search has to be continued to the right, the time needed to gain a subpixel measure is then INP + L1(N ? M ) + 2 L1(R ? 1) + ERR All time constants are hardware speci c, so just to express some realistic numbers, we use

Page 7

the values gained by the system in test by the Fraunhofer Institut. The system consists of a 486 PC/AT (50 MHz) equipped with a specially designed camera{card for scanning. In an image of size 512 pixels (N=512) and a template size of 100 pixels (M=100), we use 16 interpolated versions (R=16). The computational costs formula yields 33ms for a single measurement. In other words, our system is able to detect more than 30 subpixel positions per second. Although a rather good quota, certain industrial inspection applications require a better one. Of course some optimizations on computing the pixel hitpoint (as suggested in [4]) can increase the eciency by a certain amount. Increasing the eciency by a factor of at least 10 times, however, can be achieved by the use of tracking{ methods. Remember, the computational cost formula stands for the case, in which every new incoming image has to be processed by a complete template matching search. In tracking applications, it is no longer necessary to investigate the whole new image, instead only a small part, surrounding the last hitpoint, has to be investigated. Depending on a priori knowledge of the maximum expectable deviation of consecutive hitpoints (in consecutive image scans), a factor of about 300 measurements per second can easily be achieved.

4. EXPERIMENTAL RESULTS We used 4 dierent kinds of interpolation methods:

   

Linear interpolation Cubic B{Spline interpolation Hiresolution Cubic Spline interpolation FFT/IFFT interpolation

Except for the last one, all methods have been exhaustively discussed in [5]. FFT/IFFT interpolation is an interpolation{method in the frequency domain. The template is transferred from the spatial to the frequency domain by use of the Fast Fourier Transformation. Then the frequency spectrum is enlarged by inducing zeros (in the middle of the symmetric spectral values). The interpolated template is gained by the inverse transformation (IFFT). More details are explained in [6]. This approach is surely the most time consuming one, nevertheless interpolation time is neglegible with our class of algorithms.

4.1 Arti cial Images Although our intention was to design algorithms useable for the practical, noisy case, we have proven the introductory statement of improved accuracy with computer{generated images. To enable a direct comparision, we used the same sinc{function as Tian&Huhns did in [2]. With their algorithm, they reported a midrange error of 0.01 to 0.05 pixels. Our results are shown in Table 4.1: Interpolation method midrange error LINEAR 0.005 CUBIC B{SPLINE 0.008 HIRES. SPLINE 0.004 FFT/IFFT 0.004

max. error 0.018 0.027 0.016 0.016

Table 4.1 Subpixel{errors with sinc{function

Results were obtained by use of 50 interpolated template versions. The template size was 100 pixels, taken from the middle of the sinc{ function. Statistics were made out of all 100 possible shifts of the original function from 0.0 to 1.0 (stepwidth 0.01 pixel). Remember no comparision with our prediction formula is possible in this case, because  is zero in computer generated images.

Page 8

We will omit results for Hiresolution Cubic Spline interpolation furthermore, because | even though results above seem very promising | we encountered very poor results by using it on high contrast images. A possible explanation for its behaviour is also given in [5].

4.2 CCD{scanned images We investigated various kinds of reference images to reach an overall impression of possible accuracy. Our selected examples trange from simple edge{pictures to complex speckle images. We rst examine a knife{edge picture (i. e. a black edge on white background). Interpolation method midrange error LINEAR 0.022 CUBIC B{SPLINE 0.041 FFT/IFFT 0.013

max. error 0.048 0.074 0.032

Table 4.2.a Subpixel{errors with knife{edge image

Next image to be observed is a pretty rough emery{paper to give a coarse speckle image. Interpolation method midrange error LINEAR 0.010 CUBIC B{SPLINE 0.015 FFT/IFFT 0.012

max. error 0.022 0.032 0.024

Table 4.2.b Subpixel{errors with emery{paper image

Last example is a ne sand{paper: Interpolation method midrange error LINEAR 0.032 CUBIC B{SPLINE 0.031 FFT/IFFT 0.036

max. error 0.069 0.082 0.072

Table 4.2.c Subpixel{errors with sand{paper image

All images have been adjusted in contrast according to the guidelines in section 3. Templates have been trained in order to suppress noise (noise has zero mean, so by averaging several pictures of the same scene we eliminate noise). The image was then translated by 0.01 pixels stepwidth on a micrometer stage. The values shown arise out of statistics of 16 translations each.

CONCLUSIONS In this paper we presented various experimental data concerning the accuracy of subpixel registration algorithms. As we have shown above, even though a midrange accuracy of 0.004 pixel can be achieved in simulated pictures, the maximum error is still about 0.1 pixel in real scanned images. But at this point we want to stress the fact that little is gained by computing the mean error, in industrial applications it is vital to know the maximum error that is to be expected. By using a new approach in subpixel template matching | interpolating the template instead of the original image data | best results have been achieved by using linear interpolation and FFT/IFFT resampling. An implementation of the suggested algorithm with a real{time CCD camera/hardware{ controller resulted in an execution time of more than 30 measures per second, and should therefore be sucient for many industrial purposes.

References [1] D. I. Barnea and H. F. Silverman, "A Class Of Algorithms For Fast Digital Image Registration", IEEE Transactions On Computers, C-21, pp 179 { 186, 1972.

Page 9

[2] Q. Tian und M. N. Huhns, "Algorithms For Subpixel Registration", Computer Vision, Graphics and Image Processing, 35, pp 220 { 233, 1986. [3] Q. Tian und M. N. Huhns, "A Fast Iterative Hill { Climbing Algorithm For Subpixel Registration", Proceedings, 7th Int. Conf. on Pattern Recognition, 1, pp 13 { 16, 1984. [4] A. Rosenfeld and G. J. Vanderbrug, "Coarse { Fine Template Matching", IEEE Transactions On Systems, Man, And Cybernetics, 2, pp 104 { 107, 1977. [5] J. A. Parker, R. V. Kenyon and D. E. Troxel, "Comparision Of Interpolating Methods For Image Resampling", IEEE Transactions On Medical Imaging, 1, pp 31{39, 1983. [6] G. Oetken, "Ein Beitrag zur Interpolation mit digitalen Filtern", Doktorarbeit im Fachbereich Elektrotechnik (Universitat Erlangen{Nurnberg), Erlangen, 1978. [7] P. Seitz, "Optical Superresolution Using Solid{State Cameras And Digital Signal Processing", SPIE Optical Engineering, Vol. 27 No. 7, pp 535{540, 1988. [8] P. Seitz and J. M. Raynor, "Optische U berau osung mit CCD Kameras und digitaler Signalverarbeitung", Optical 3{D Measurement Techniques, Herbert Wichmann Verlag, Karlsruhe, 1989. [9] B. Kleinemeier, "Measurement Of CCD Interpolation Functions In The Subpixel Precision Range", SPIE Industrial Inspection, V0l. 1010, pp 158 { 165, 1988. [10] R. Y. Wong and E. L. Hall, "Performance Comparision Of Scene Matching Techniques", IEEE Transactions On Pattern Analysis And Machine Intelligence, 3, pp 325 { 330, 1979.

[11] R. Y. Wong, "Sensor Transformations", IEEE Transactions On Systems, Man, And Cybernetics, 12, pp 836 { 841, 1977. [12] E. V. Slud, "Subpixel Translation { Registration Of Random Fields", IEEE Transactions On Geoscience And Remote Sensing, 4, pp 487 { 490, 1988. [13] J. P. Secilla, N. Garcia and J. L. Carrascosa, "Template Location In Noisy Pictures", Signal Processing, 14, pp 347 { 361, 1988.

Page 10