Image Mosaicing using Corner Detection André Rocha1,2 , Ricardo Ferreira1,2, Aurélio Campilho1,2 1 Instituto de Engenharia Biomédica – INEB 2 Faculdade de Engenharia da Universidade do Porto - FEUP Praça Coronel Pacheco, 1 4050-453 Porto PORTUGAL
[email protected],
[email protected],
[email protected]
Abstract An image mosaicing algorithm using the Laganière method for corner detection and the so called Hat function for image blending is presented in this paper. Good results have been obtained within the mosaics created, clearing the way for a few considerations about the most important topics to keep in mind when implementing an image mosaicing algorithm. The work herein presented was developed in an undergraduate level study.
Keywords Image Mosaicing, Perspective Image Transformations, Laganière Corner Detection Method.
1. INTRODUCTION An image mosaicing operation consists of three main steps. The first one is Image Registration. In this phase the correspondence between features common to both images is defined in order to compute an image-to-image transformation that maps one image into the other. In this work we selected the features to be the corners in the images and we used the Laganière method [1] for their detection. In the next step, Image Warping, the two images that will form the mosaic are warped, by using the geometric transformation computed in the previous phase. The final stage, Image Compositing, consists of blending images together on the reference coordinate system. This paper is organised as follows. In section 2, the corner detection method is presented and discussed. In section 3, the proposed image mosaicing method is presented. Finally, section 4 presents some results and in section 5 the main conclusions are derived.
2. CORNER DETECTION – LAGANIÈRE PROPOSAL For image mosaicing we adopted a morphological approach for corner detection proposed in [1]. The corresponding corner detector is based in an operation namedasymmetrical close. A morphological close of an image first dilates it using a structuring element, followed by an erosion by the same element. In the asymmetrical close different structuring elements are used in the dilation and erosion. This operation is performed twice thereby using four different structuring elements (which are represented in Fig.1) and generating two intermediate images.
Fig.1 – Structuring Elements. This way, the image being processed is dilated by element in a) and then eroded by b), creating the first intermediate image which will be referred by Ic+,◊ . The second one, Icx, , is obtained this same way, making use of element c) for the dilation process and d) for the eroding part. By providing different shapes for the structuring elements, the Laganière detector guarantees that corners with almost all kinds of orientations are detected, reaching as well excellent insensitivity to small sized features like isolated dark or bright noise peaks. The two images generated are then combined by using c+, x = |Ic+, ◊ - Icx, | and the resulting image is thresholded by a value depending on the situation. Fig.2 shows intermediate and final results of the application of this method on a synthetized image (a)). Images b) and c) represent dilation of a) by elements a) and c) in Fig.1, respectively. The erosion of these results by elements b) and d) of Fig.1 are shown in Fig.2d) and e), whereas f) is the final result as described above.
a) Synthetized image
b) Dilation by Fig.1a)
c) Dilation by Fig.1c)
d) Erosion of b) by Fig.1b) e) Erosion of c) by Fig.1d)
f) Detected corners
Fig.2 – Intermediate and final results of the Laganière corner detector It is relevant to refer how important and delicate the thresholding step is. In fact, this is the most sensitive part of the whole process and it is the main reason for the corners missing in the right side of the inner rectangle shown in Fig.2f): intensity level differences between Fig.2d) and Fig.2e) in those two points are very small comparatively to the threshold level. A difference of only 1% in the threshold value can sometimes be sufficient to avoid the detection of multiple and false corners in the image. This effect is also illustrated in the following pictures (Fig.3), where the threshold values indicated are relative to an intensity scale going from 0 to 255. Corners detected are represented by small squares in the images.
a) Threshold = 13
b) Threshold = 20
c) Threshold = 26
Fig.3 – Variation of the result of the Laganière detector with threshold value Another important aspect which may be worthy of evidence is the referred noise robustness of this method, which can be verified in Fig.4. Fig.4a) represents a synthesized image to which was added Gaussian noise of null mean and standard deviation equal to 10. Fig.4b) shows the corners detected in a) by the Laganière corner detector.
a) Noisy image
b) Detected corners
Fig.4 – Performance of the Laganière detector in a noisy image Taking into account the considerable amount of noise in the image, we may say that only a small number of false corners have been detected. Also when compared with other corner detectors, this method provides very good results. For comparison purposes we used the following corner detectors: • the Plessey detector, based on the evaluation of a cornerness measure [4,7], produces the result in Fig.5b); • the result of the Kitchen-Rosenfeld method [5,7], is illustrated in Fig.5c); • The Susan method, proposed in [6,7], produces the result in Fig.5d).
a) Laganière Detector
b) Plessey Detector
c) Kitchen-Rosenfeld Detector
d) Susan Detector
Fig.5 – Comparison of the results obtained by different corner detectors For all these reasons we think that the Laganière method for corner detection is a good option for preparing the images for the subsequent mosaicing algorithm. 3. IMAGE MOSAICING The first part of an image mosaicing operation consists of defining correspondences between some features present in both images, in order to determine the geometric transformation necessary to provide their alignment – Image Registration. One of the images is then deformed according to the computed transformation, Image Warping, thereby changing the spatial relationship between pixels coordinates. That is, image coordinates [x,y]T are mapped into a new set of coordinates [x’,y’]T. The definition of the correspondences can be done manually or by an automatic process. In the latter, one can use a similarity measure like normalized crosscorrelation or correlation coefficient to match the two images. However, due to time restrictions and to the better precision naturally achieved with manual definition, this turned out to be our option for this operation. This is where we come to the importance of corner detection as a pre-processing stage for image mosaicing. When defining the referred correspondences between the features present in both images, we could of course just look at the pictures and trust in our quick glance to determine the positions of those features. What we do instead is to first compute the exact positions of a particular class of features – the corners – after what we are able to perform a safer definition of the correspondences. Manual definition of those correspondences between the two images was supported by a function which prompted the user for alternately clicking on corresponding items on each picture, thereby providing an user-friendly interface. However, in order to achieve good results on the mosaicing operation we thought it was of extreme relevance that the referred function sought the nearest corner (which was previously detected) to the pixel pointed by the user click. This way we
protected the computation of the necessary geometric transformation against the nice but not perfect user’s eyes, against his shaky hands and all other kinds of inaccuracies of this process. The reason why we emphasize so much this matter is thinking that, in an operation which requires such a high level accuracy like image mosaicing does, every single point of the process where we can improve that accuracy is precious. And this is certainly one of those. Fig.6a) and b) illustrate corners detected on two images, where the circles around some of them indicate which correspondences between the two pictures were selected. The minimum number of correspondences one must define is four.
a)
b)
Fig.6 – Circles indicate selected corner correspondences between two images Once corners in each picture have been associated we now have the necessary elements to compute the geometric transformation matrix, which is given by the following relation.
Pixel coordinates on the original image are given by [x/w, y/w]T and are mapped into [x’/w’, y’/w’]T after image warping. In order to enable reverse mapping one can, without loss of generality, force w’ to 1. The coefficients of the geometric transformation matrix mij are the solution of the following system of equations,
where point [xi,yi]T is mapped into the corresponding point in the other image [ui,vi]T. It was already said that the minimum number of correspondences to make the perspective transformation possible is four. Nevertheless, the more associations we have between corners on each image, the more accurate the perspective transformation will be. This way, when more than four correspondences are defined, solution in the least squares sense to the overdetermined system of equations is computed, thereby improving the quality and accuracy of the perspective transformation. Bilinear interpolation was used to determine pixel intensities after image warping. The last part of a mosaicing operation is image blending, which consists of bringing the two pictures together into one. This section should not present extra difficulties to the process, except for one question: what to do with the intensities of overlapping pixels of the two images? There are several ways to answer this question: choosing the pixel intensities of one image; taking the average or median of the intensities of overlapping pixels; considering a weighted average of the intensities according to the distance of the pixel to the center of each image (bilinear blending). The blending method we chose was the hat function, which can be considered as a particular case of bilinear blending since it calculates a weight for every pixel in each image according to the following relation
This weight distribution is illustrated in the chart of Fig.7, where we can see how near image center pixels are affected by a large factor and how this factor decreases as we move to the borders.
Fig.7 – Weight distribution according to hat function The resulting intensity of the pixels of the mosaic (Iblend ) is then given by the relation below, where x and y represent the initial coordinates, before the warping operation, and i is the image index.
∑ w (x, y )⋅I (x, y ) (x, y )= ∑ w (x, y ) i
I blend
i
i
i
i
Ii(x,y) is referred to the intensity value of the i th image pixel [x, y]T. 4. RESULTS Now that all the steps and relevant details of the mosaicing operation have been carefully described, some of the mosaics created will be presented.
a)
b)
c)
d) Resulting mosaic of a), b) and c) Fig.8 On the left corner of the central roof present in Fig.8d) we can see one of the blending problems that we are planning to soften in the future: brightness differences in the boundaries between overlapping and non-overlapping areas.
a)
b)
d) Resulting mosaic of a), b) and c) Fig.9
c)
In this mosaic it is possible to see an example of a non-resolvable problem in image mosaicing: object motion. There is a man walking at the farthest end of the image which results in a kind of shadow in the mosaic.
a)
b)
c)
d) Resulting mosaic of a), b) and c) Fig.10 In the mosaic in Fig.10d) one can see some distortion on both sides of the image. This was caused by the fact that the overlapping areas were not big enough to allow a good extrapolation factor, which makes this problem a non-resolvable one. 5. FUTURE WORK Due to the fact that this work was developed as an undergraduate level assignment, there were some time constraints that, in a certain way, restricted further developments. Nevertheless, there are a few ideas that we have already started exploring and which we are planning to research in the future. One of them has to do with the development of an automatic image registration method. We tried to use the Levenberg-Marquardt algorithm for non-linear brightness quadratic differences minimizing, which is known by fast converging to the final result. A resolution pyramid, to reduce the computational cost, and a
matching measurement are used to find the overlapping area, after what we are able to obtain the intial transformation matrix M. Then, the Levenberg-Marquardt algorithm is applied, using the original images, to iteratively refine the tranformation matrix. There are also some plans in order to develop an image warping method to an intermediate plane, which could be used to achieve better results in some kinds of images. Finally, an image blending algorithm is to be created in order to soften some brightness differences that often appear in the boundary between overlapping and non-overlapping areas. 6. CONCLUSIONS As we have seen, by developing some small but important details of the mosaicing process we have managed to significantly improve the resulting mosaics. The first of these details consists of using a corner detector to locate only a few well defined features on each picture. If an user-friendly interface (where correspondences are indicated by alternately clicking on the two images) is developed it is important not to consider the point indicated by the user’s imprecise click, but to instead seek for the previously detected corner nearest to the clicked pixel. Another important tip is to select corners that don’t define a straight line and that are as spread as possible in the overlaping area. This way we are reducing the distortion caused by the perspective transformation. In the image blending stage we think that the weight function we have chosen – hat function – provides very good results in what concerns to mixing overlapping pixels on the resulting mosaic. Finally, when working with RGB images the transformation matrix is computed based on the corresponding gray pictures. Image warping is then applied to each of the three RGB components of the original image. References [1]
R. Laganière, A Morphological operator for Corner Detection, in Pattern Recognition, vol. 32, N. 11, pp. 1643-1652, 1998.
[2] Paul Heckbert, Projective Mappings for Image Warping, in Fundamentals of Texture Mapping and Image Warping (Paul Heckbert, Master’s Thesis), U.C.Berkeley, 1989. [3] R. Szeliski, Image Mosaicing for Tele-Reality Applications, Cambridge Research laboratory, Tech. Report. CRL 94/2, 1994.
[4] C. G. Harris, Determination of ego-motion from matched points, in Proc. Alvey Vision Conf, Cambridge, UK, 1987. [5] L. Kitchen and A. Rosenfeld, Gray level corner detection, in Pattern Recognition Letters, pp. 95-102, 1982. [6] S. M. Smith and J. M. Brady, SUSAN - a new approach to low-level image processing, in International Journal of Computer Vision, 23(1):45-78, 1997. [7] F. Mokhtarian http://www.ee.surrey.ac.uk/Research/VSSP/demos/corners/results3.html
