Efficient Implementation of Gaussian and Laplacian Kernels for ... - MDPI

Journal of

Imaging Article

Efficient Implementation of Gaussian and Laplacian Kernels for Feature Extraction from IP Fisheye Cameras Konstantinos K. Delibasis Department of Computer Science and Biomedical Informatics, University of Thessaly, Lamia 35131, Greece; [email protected]; Tel.: +30-22310-66900 Received: 1 May 2018; Accepted: 17 May 2018; Published: 24 May 2018

Abstract: The Gaussian kernel, its partial derivatives and the Laplacian kernel, applied at different image scales, play a very important role in image processing and in feature extraction from images. Although they have been extensively studied in the case of images acquired by projective cameras, this is not the case for cameras with fisheye lenses. This type of cameras is becoming very popular, since it exhibits a Field of View of 180 degrees. The model of fisheye image formation differs substantially from the simple projective transformation, causing straight lines to be imaged as curves. Thus the traditional kernels used for processing images acquired by projective cameras, are not optimal for fisheye images. This work uses the calibration of the acquiring fisheye camera to define a geodesic metric for distance between pixels in fisheye images and subsequently redefines the Gaussian kernel, its partial derivatives, as well as the Laplacian kernel. Finally, algorithms for applying in the spatial domain these kernels, as well as the Harris corner detector, are proposed, using efficient computational implementations. Comparative results are shown, in terms of correctness of image processing, efficiency of application for multi scale processing, as well as salient point extraction. Thus we conclude that the proposed algorithms allow the efficient application of standard processing and analysis techniques of fisheye images, in the spatial domain, once the calibration of the specific camera is available. Keywords: Gaussian; Laplacian kernels; multi-resolution image processing; Harris corner detection; spatial domain; fisheye image; camera calibration

1. Introduction A number of types of very wide field cameras acquire images with 180◦ field of view (FoV). These cameras use dioptric systems, spherical lens, often called fisheye, or a combination of catadioptric (parabolic or spherical mirror) and dioptric (normal projection lens), often referred to as omnidirectional. The model of formation of all these types of images differs significantly from the traditional central projection model and introduces rapid deterioration of spatial resolution towards the periphery of the FoV, as well as strong deformations that map straight lines as curves. The model of image formation with 180◦ FoV can be treated in a unified mathematic way, utilizing a spherical optic lens and central projection [1,2]. Thus we will use the terms interchangeably for the rest of the paper. The use of very wide field cameras is becoming very wide in domains like security [3], robotics [4,5], public and private area monitoring, since they allow constant imaging of all directions with a single camera. Spherical fisheye panoramas are used to emulate or to register a PTZ camera and to integrating live webcam feeds within 2D and 3D GIS applications (such as Google Earth and Google Maps) [6,7]. Fisheye cameras with special processing operators are proposed for large space monitoring and crowd

J. Imaging 2018, 4, 73; doi:10.3390/jimaging4060073

www.mdpi.com/journal/jimaging

J. Imaging 2018, 4, 73

2 of 21

counting in [8]. In [9], a network of fisheye cameras is used to efficiently track persons in a space with obstacles that hinder visibility. Traffic management at road intersection has been proposed in [10] using network fisheye cameras. Pair of fisheye cameras has been used for pose estimation and motion capture in [11], employing a number of 3D Gaussians to generate a probabilistic body model. Gaussian kernels defined in 3D have also been used for scene modeling, in order to make discrete phenomena, like occlusion, continuous and thus, more easily optimizable [12]. The very different model of image formation for fisheye images has invoked a number of attempts to redefine image processing operators for this type of images. The basic morphology operators have been redefined in [8], although the computational cost has not been discussed. The well-known Scale-Invariant Feature Transform (SIFT) image descriptors that were introduced in [13], are redefined for omnidirectional images in [14] in the frequency domain. Fisheye images are defined over spherical coordinates (azimuth θ, elevation φ). Therefore, Spherical Fourier Transform (SFT) is used, that decomposes the image as a weighted sum of spherical harmonic functions Ylm (θ, ϕ) of degree l and order m, with |m| ≤ l. In [15], the Gaussian kernel has been defined in the (θ, φ) image domain using spherical harmonics of the 0th order. The work of [16] also employs the use of SFT to detect points of interest using the well-known SIFT. In [17] differential operators defined on Riemannian manifolds are used to compute scale space and perform smoothing in fisheye images. That work was furthr improved in [18], where the already developed differential operator was extended to perform SHIFT-like salient point detection in omnidirectional images. Others have reported exploiting the image space, rather than the frequency domain, for specific kernels. In [19], the authors also used a simplified version of SIFT feature extraction method (e.g., no multiresolution was used) for robot navigation by fisheye camera, obtaining good results. In [20], features are extracted from 360 FoV omnidirectional images in the spatial domain, but after the image has been mapped to a hexagonal grid. In [21], 4-neighbours and 8-neighbours Laplacian operators have been proposed for omnidirectional panoramic images. In [22], an approach for fisheye image processing in the spatial domain is proposed, using geodesic distance between pixels. The approach proposed in this work, is also based on the spatial image domain and utilizes the geodesic distance between pixels. The main difference from [22] is that the proposed method uses the geodesic distance metric between pixels to redefine the basic filter kernels, whereas in [22] the image neighborhood is being resampled, while keeping the filter kernels unchanged. In order to avoid computationally expensive image interpolation, [22] uses the nearest neighbor interpolation, which although not a good performing technique, still produced better results than the classic image processing operators. As it will be discussed later, there is no proposed efficient implementation for the method in [22], since it requires the geodesic neighborhood definition for every pixel in each new image, in contrast to the method proposed in this work that requires the construction of a small Gaussian kernel, only once for a calibrated camera. The rest of the paper is structured as following: In the methodology section, the geodesic distance metric is defined using the calibration of the fisheye camera. The Gaussian kernel, its partial derivatives and the Laplacian of Gaussian kernel is redefined, using the geodesic distance. All these kernels vary with their spatial location in the image, therefor, their direct convolution with the image is inefficient. Thus, an efficient computational implementation is proposed, which involves the initial pre-calculation of a single Gaussian kernel with small standard deviation σ0 and the approximation of the Image response of the Gaussian, its derivatives, as well as the Laplacian of Gaussian at larger values of σ. 2. Materials and Methods The methodology that is proposed in this work is tested with images acquired by a Mobotix Q24 fisheye camera, that is installed on the roof, imaging an indoor space. A typical frame of the camera is shown in Figure 1, with pixilation of 480 × 640. The camera has been calibrated using [23]. The result of the calibration is shown in Figure 2, where the azimuthal angle Θ and the elevation Φ is provided for each pixel of the fisheye image.

J. Imaging 2018, 4, 73

3 of 21

J. Imaging 2018, 4, 4, x FOR J. Imaging 2018, x FORPEER PEERREVIEW REVIEW

of 21 3 of321

Figure 1. A typical image from a roof-based fisheye camera. Figure Figure 1. 1. A A typical typical image image from from aa roof-based roof-based fisheye fisheye camera. camera.

(a)

(b)

(a)calibration of the fisheye camera, for each image pixel(b) Figure 2. The resulting (a) the azimuthal angle θ and (b) the elevation angle φ is provided. Figure 2. 2. The The resulting resulting calibration calibrationof ofthe thefisheye fisheyecamera, camera,for foreach eachimage imagepixel pixel(a)(a)the the azimuthal angle Figure azimuthal angle θ θ and (b) the elevation angle φ is provided. (b) theDistance elevationMetric angle between φ is provided. 2.1.and Geodesic Pixels of the Calibrated Fish-Eye Image The formation omni-directional image a spherical lens,Image as presented in detail in [23] can 2.1. Geodesic Distance of Metric between Pixels of theusing Calibrated Fish-Eye 2.1.beGeodesic Distance Metric between Pixels of the of Calibrated Image summarized as following: the intersection the line Fish-Eye connecting the real world point with the The formation of omni-directional image using a spherical lens, as presented in detail in [23] can center the optical is calculated withusing the optical element. This is detail then projected The of formation ofelement omni-directional image a spherical lens, as intersection presented in in [23] can becentrally summarized as following: the intersection of the line connecting the real world point with the on the sensor It has been shown [1] that by the choosing the center projection, be summarized as image following: theplane. intersection of the line connecting real world pointof with the center center of the optical element is calculated with the optical element. This intersection is then projected one optical can simulate use of any quadratic shape mirror (spherical, ellipsoid, paraboloid and of the elementthe is calculated with the optical element. This intersection is then projected centrally centrally on the This image sensor plane. It has been shown [1] thattransformation by choosing the center of between projection, hyperboloid). type of image formation induces non-linear of distances on the image sensor plane. It has been shown [1] that by choosing the center of projection, one can one can simulate the use of any quadratic shape mirror (spherical, ellipsoid, paraboloid and pixels. simulate the use of any quadratic shape mirror (spherical, ellipsoid, paraboloid and hyperboloid). hyperboloid). This type image formation induces non-linear transformation of distances between These concepts areof visualized in Figure 3, where the semi-spherical optical element of unit radius This type of image formation induces non-linear transformation of distances between pixels. and the image plane is displayed. The center of projection is placed at −f on the Y axis, with f set to pixels. These concepts are visualized in Figure 3, where the semi-spherical optical element of unit radius 0.2.These The image plane is visualized tessellated into 128 equidistant points to resemble optical the image pixels.of 21unit of these concepts are in Figure 3, where the semi-spherical element radius and the image plane is displayed. The center of projection is placed at −f on the Y axis, with f set “pixels” are backprojected on the spherical optical element (both shown in different color). It is selfand the image plane is displayed. The center of projection is placed at −f on the Y axis, with f set to to 0.2. The image plane is tessellated into 128 equidistant points to resemble the image pixels. 21 of evident that the back-projected are equidistant no longer equidistant. 0.2. The image plane is tessellatedpoints into 128 points to resemble the image pixels. 21 of these these “pixels” are backprojected on the spherical optical element (both shown in different color). It is “pixels” are backprojected on the spherical optical element (both shown in different color). It is selfself-evident that the back-projected points are no longer equidistant. evident that the back-projected points are no longer equidistant.

J. Imaging 2018, 4, 73

4 of 21

J. Imaging 2018, 4, x FOR PEER REVIEW

4 of 21

Figure 3. The basic concept of fisheye image formation, involving a (semi) spherical lens and central Figure 3. The basic concept of fisheye image formation, involving a (semi) spherical lens and central projection on the image plane. The difference between Euclidean distance between pixels on the image projection on the image plane. The difference between Euclidean distance between pixels on the plane and geodesic distance (on the spherical lens) becomes apparent. image plane and geodesic distance (on the spherical lens) becomes apparent.

If a window of size (2w + 1) is defined on the image plane (red dots in Figure 3, then the + 1, 1) this window, using the Euclidean distance between pixelspixels on theon image definition of ofaaGaussian Gaussianwithin within this window, using the Euclidean distance between the plane is visualized as the black curve in Figure 3. If this Gaussian is back-projected on the spherical image plane is visualized as the black curve in Figure 3. If this Gaussian is back-projected on the optical element, kernelthe depicted black (atinthe periphery) is produced. As expected, is different spherical opticalthe element, kernelindepicted black (at the periphery) is produced. Asitexpected, it from a Gaussian due kernel, to distance In order to In generate a Gaussian defined on is different from akernel, Gaussian due metric. to distance metric. order to generate akernel Gaussian kernel the image we have to modify metric between pixels on thepixels sensor, to defined onsensor, the image sensor, we havethe to distance modify the distance metric between onaccording the sensor, the geodesic distance of their back-projection on the circle.onFigure 3 is aFigure 2D abstraction of the real according to the geodesic distance of their back-projection the circle. 3 is a 2D abstraction 3Dthe camera setup. Thesetup. asymmetry that is evident the 21-point, kernel, defined ondefined the image of real 3D camera The asymmetry that in is evident in theblack 21-point, black kernel, on plane is exaggerated since the image plane (image line in this 2D abstraction) was tessellated to only the image plane is exaggerated since the image plane (image line in this 2D abstraction) was 128 pixels. Furthermore, in the case of a 2D kernel, asymmetry along oneisofpresent its two only axes, tessellated to only 128 pixels. Furthermore, in the case of a is 2Dpresent kernel,only asymmetry as it will along onebeoffurther its twodiscussed axes, as itbelow. will be further discussed below. In [24] the use of the geodesic distance has been proposed for re-defining the Zernike moment proposed approach requires the results of the calibration invariant for calibrated fish-eye images. The proposed of the fish-eye camera. Several approaches exist for fish-eye camera calibration, such as [1,25]. In this work, we utilize the calibration of the specific fish-eye camera, proposed in [23]. The results of the calibration nay be summarized in two look-up tables of size equal to an image frame, which hold the direction of (x,(x, y) in 3D world. The direction is defined by two by angles, azimuth ofeach eachimage imagepixel pixel y)real in real 3D world. The direction is defined two the angles, the Θ(x, y) and they)elevation y). During theDuring calibration these two angles pre-calculated for azimuth Θ(x, and the Φ(x, elevation Φ(x, y). the step, calibration step, theseare two angles are preall pixels offor theallvideo are frame storedand in two Θ and Φ, as shown 2. in calculated pixelsframe of theand video are look-up stored intables two look-up tables Θ andinΦ,Figure as shown It is Figure 2. clear that the distance of two pixels is different when measured on the image sensor and on the projection of the points on pixels the spherical optical element. Therefore, in order to produce It is clear that theimage distance of two is different when measured on the image sensor and on accurate results, image processing algorithms that use pixel distances have to be re-implemented for the projection of the image points on the spherical optical element. Therefore, in order to produce fish-eye (omni-directional) images.algorithms that use pixel distances have to be re-implemented for accurate results, image processing It is(omni-directional) well known that the geodesic curve of a sphere is a great circle (a circle that lies on the sphere fish-eye images. and has the same center with the sphere). Thus, the distance between any twothat points sphere is It is well known that the geodesic curve of a sphere is a great circle (a circle lieson on athe sphere the length the arc that with is defined by the two points and belongs to a any circle that passes and has theofsame center the sphere). Thus, the distance between two points onthrough a spherethe is the length of the arc that is defined by the two points and belongs to a circle that passes through the two points and has the same centre with the sphere. The geodesic distance between pixels p0 = ( x0 , y0 ) and p1 = ( x1 , y1 ) is defined in [22,24] and can be summarized as following:

J. Imaging 2018, 4, 73

5 of 21

two points and has the same centre with the sphere. The geodesic distance between pixels p0 = ( x0 , y0 ) 5 of 21 and can be summarized as following: 1 1 1 Let P0 and P1 be the projections of the two pixels on the spherical optical element and v0 , v1 be v 0 , vby Let P0 and P1 bepointing the projections of the on of thethe spherical optical element be the position vectors to P0 and P1 . two The pixels direction two pixels in real worldand is given 1 the pre-calculated look-up tables:to P0 and P1. The direction of the two pixels in real world is given by the the position vectors pointing pre-calculated look-up tables: (θ0 , ϕ0 ) = (Θ( x0 , y0 ), Φ( x0 , y0 )) (1) ϕ 0 )) ==( (ΘΘ( (xx0 , ,yy0 ) ,)Φ ( x0 ,1y, 0y)1))) (θ(θ1 0, ,ϕ 1 1 1 , Φ( x (1) (θ1 ,ϕ1 ) = ( Θ ( x1 , y1 ) , Φ ( x1 , y1 ) ) Thus, the position vectors are calculated: Thus, the position vectors are calculated: vi = (cos θi cos ϕi , sin θi cos ϕi , sin ϕi ), i = 0, 1 (2) vi = ( cosθi cos ϕi ,sin θi cos ϕi ,sin ϕi ) , i = 0, 1 (2) The distance r01 of points P0 and P1 on the unit sphere is easily calculated as the arc-length The distance r01 of points P0 and−1P1 on the unit sphere is easily calculated as the arc-length between P0 and P1 , assuming that cos returns the result in radians: between P0 and P1, assuming that cos-1 returns the result in radians:

J.and Imaging 2018, REVIEW p = defined in [22,24] ( x 4,, yx FOR ) is PEER

1 1 (v0 · v1 ) r01r == dd((vv0 , ,vv1 )) = cos−− = cos (v ⋅ v ) 01

0

1

0

1

(3)(3)

Figure 44shows showsthe thegeodesic geodesicdistance distancecomponent componentbetween between successive successive pixels pixels along along lines lines(left) (left)and and Figure columns (right), defined more formally as: columns (right), defined more formally as:

= dd(( , yy) ))) ( ( xx ++1,1,yy) ), (, (xx, DDxx((x,x,yy))= DDy ((x,x,yy))= = dd(( + 11)), ,( (xx, , yy) )) ( (x,x, yy + )

(4)(4)

y

Figure Figure 4. 4. Graphic Graphic representation representation of Dxx (left) (left)and and D Dyy(right). (right).

2.2. Based on on the the Geodesic Geodesic Distance Distance Metric Metricbetween betweenImage ImagePixels 2.2. Filter Filter Kernels Kernels for for Fish-Eye Fish-Eye Image Image Processing Processing Based Pixels The most versatile filter in image processing is the Gaussian. According to its definition for 2D Thethe most versatile filter in image images, Gaussian can be written asprocessing is the Gaussian. According to its definition for 2D images, the Gaussian can be written as 1 − r22 2 2σ r (5) g( x, y; σ) = 12 e − 2σ 2 2πσ (5)

g ( x, y; σ ) =

r2

x2

+ y2

2πσ 2

e

with = being the distance of pixel (x, y) from the centre of the kernel. The discrete Gaussian 2 2 least) 2 in the following is defined (at with r = x + y being the distance of pixel (x, y) from the centre of the kernel. The discrete

Gaussian is defined (at least) in the following ( x, y) ∈ [−3dσe, 3dσe] × [−3dσe, 3dσe]

( x, y ) ∈  −3 σ  ,3 σ   ×  −3 σ  ,3 σ  

where

 x 

(6)

(6)

is the operator returning the closest integer larger than x, for x > 0. The standard

deviation σ is defined by the user in pixels, which equivalently defines the size of the kernel, according to Equation (6).

J. Imaging 2018, 4, 73

6 of 21

where d x e is the operator returning the closest integer larger than x, for x > 0. The standard deviation σ is defined by the user in pixels, which equivalently defines the size of the kernel, according to Equation (6). The above “classical” definition of the Gaussian needs to be modified for calibrated fish-eye images. Let us suppose that we require the linear convolution of the Gaussian kernel with a fish-eye image, at pixel p0 = ( x0 , y0 ). Let Aσ denote the subimage centered at ( x0 , y0 ): Aσ = [ x0 − 3dσe, x0 + 3dσe] × [y0 − 3dσe, y0 + 3dσe]

(7)

In the case of a calibrated fish-eye image, the definition of Equation (4) is modified as following:

• •

It is straightforward to use the geodesic distance metric between p0 and any pixel pi = ( xi , yi ) in Aσ , thus ri = d(pi , p0 ) σ is initially defined by the user in pixels. However, since the geodesic distance between pixels is used in Equation (1), the value of σ (in pixels) cannot be used. This becomes clear if one considers that the Gaussian has to approach 0 within Aσ . Since Aσ was defined as a window centered on the current pixel, spanning ±3 σ, we set σ equal to (1/3) of the maximum geodesic distance between the central pixel p0 and any pixel in Aσ : σ=

•

1 max({ri = d(pi , p0 ), pi ∈ Aσ }). 3

(8)

Then σ is replaced in Equation (5). Thus, for a Gaussian centered at p0 = ( x0 , y0 ), the g( x, y; σ) = g(p; σ) =

1 − d(p,p20 ) e 2σ 2πσ2

2

(9)

It becomes obvious that the Gaussian kernel now depends on its location on the fish-eye image. In order to define the Gaussian kernel with a specific size for the whole fisheye image, the following Algorithm 1 is used: Algorithm 1. Initialization of the Gaussian kernel with small standard deviation. The size of the window kernel (2 w + 1) × (2 w + 1) is defined. The value of the standard deviation σ is calculated as following: The window A0 is placed near the edge of the FoV, where pixel distances are smaller than other locations with high elevation φ and σ0 is calculated according to Equation (8). This value of σ0 is used for each location in the fisheye image.

Figure 5a shows the Gaussian kernel as defined in Equation (9) at different position on the fisheye image. Radial lines emanating from the CoV (Center of View) have constant azimuthal θ and points on the concentric circles centered on the CoV have constant elevation φ. The same kernels are shown in (b), using color-scale. The CoV is marked on Figure 5b.

J. Imaging 2018, 4, 73 J. Imaging 2018, 4, x FOR PEER REVIEW

7 of 21 7 of 21

(a)

(b) Figure 5. The positions on fisheye image, image, using using the Figure 5. The Gaussian Gaussian defined defined in in Equation Equation (9) (9) at at different different positions on the the fisheye the geodetic distance is shown in (a) 3D and (b) as color-encoded image. geodetic distance is shown in (a) 3D and (b) as color-encoded image.

It is evident that the shape of the Gaussian kernel is a function of the kernel’s location on the It is evident that the shape of the Gaussian kernel is a function of the kernel’s location on the image. More specifically, the kernel at pixel (x, y) appears rotated by an angle equal to the azimuth image. More specifically, the kernel at pixel (x, y) appears rotated by an angle equal to the azimuth Θ(x, y) at this pixel. Furthermore, the Gaussian kernel exhibits asymmetry, as indicated in the 1D Θ(x, y) at this pixel. Furthermore, the Gaussian kernel exhibits asymmetry, as indicated in the 1D abstraction of Figure 3. It is easy to confirm that consecutive pixels along the direction with constant abstraction of Figure 3. It is easy to confirm that consecutive pixels along the direction with constant (iso-) elevation φ have constant geodesic distance. On the other hand, consecutive pixels along the (iso-) elevation φ have constant geodesic distance. On the other hand, consecutive pixels along the direction with constant (iso-) azimuth θ do not have constant geodesic distance. Thus, the geodetic direction with constant (iso-) azimuth θ do not have constant geodesic distance. Thus, the geodetic Gaussian kernel is expected to exhibit asymmetry along the axis with constant azimuth. An example Gaussian kernel is expected to exhibit asymmetry along the axis with constant azimuth. An example depicting this asymmetry is shown in Figure 6. The location that was selected is shown in (c) as a depicting this asymmetry is shown in Figure 6. The location that was selected is shown in (c) as a green circle. The iso-azimuth and iso-elevation contours are superimposed in purple and green color green circle. The iso-azimuth and iso-elevation contours are superimposed in purple and green color respectively. Figure 6a shows a 35 × 35 geodetic Gaussian kernel, generated by Equation (9) of the respectively. Figure 6a shows a 35 × 35 geodetic Gaussian kernel, generated by Equation (9) of the proposed method. The orientation of the kernel with respect to the directions of constant azimuth proposed method. The orientation of the kernel with respect to the directions of constant azimuth and and elevation, is as described above. Figure 6b displays the asymmetry of the kernel. More elevation, is as described above. Figure 6b displays the asymmetry of the kernel. More specifically, specifically, the profile of the kernel is plotted along the directions of constant azimuth θ and of the profile of the kernel is plotted along the directions of constant azimuth θ and of constant elevation constant elevation φ. Both profiles are also plotted after being mirrored, in order to make the φ. Both profiles are also plotted after being mirrored, in order to make the asymmetry apparent. It can asymmetry apparent. It can be observed that the kernel is symmetric along the direction of constant φ and slightly asymmetric along the direction of constant θ (the profile and its mirror do not match). Finally, the resampling of the image to construct the geodetic neighborhood round the selected point,

J. Imaging 2018, 4, 73

8 of 21

X

be observed that the kernel is symmetric along the direction of constant φ and slightly asymmetric along the direction of constant θ (the profile and its mirror do not match). Finally, the resampling 4, x FOR PEERthe REVIEW 8 of 21 of theJ. Imaging image2018, to construct geodetic neighborhood round the selected point, as described in [22], Equation (16), is displayed in Figure 6d. These neighborhood pixels should be equidistant using as described in [22], Equation (16), is displayed in Figure 6d. These neighborhood pixels should be geodesic distant metric and they were selected using the nearest neighbor interpolation, as explained equidistant using geodesic distant metric and they were selected using the nearest neighbor in Section 3.3 of [22]. interpolation, as explained in Section 3.3 of [22].

(a)

(b)

(c)

(d)

Figure 6. (a): The shape of the geodetic Gaussian kernel, generated at the location of the frame

Figure 6. (a): The shape of the geodetic Gaussian kernel, generated at the location of the frame indicated indicated by a green circle (c). Contours of iso-azimuth and iso-elevation are also superimposed. The by a shape green and circle (c). Contours of iso-azimuth and iso-elevation are also superimposed. The shape symmetry of the kernel is demonstrated in (b), along the direction of constant (iso-) and symmetry of kernel(iso-) is demonstrated in (b), along direction to of constant azimuth and azimuth and the constant elevation. Finally, image the resampling construct (iso-) the geodetic constant (iso-) elevation. Finally, image resampling neighborhood, according to [22] is shown in (d). to construct the geodetic neighborhood, according to [22] is shown in (d). The partial derivatives of the Gaussian kernel are easily calculated analytically:

The partial derivatives of the Gaussian kernel are easily calculated analytically: ∂ x

g ( x , y ; σ ) = g x = − 2 g ( x , y; σ ) (10) ∂x σ x ∂ g( x,as y; σthe gx = − distance g( x, y; σof) (x, y) from (x0, y), namely (10) ) =geodesic The x coordinate is calculated ∂x σ2 x = d ( ( xi , yi ) , ( x0 , yi ) ) . The same holds for the partial derivative with respect to y. The x coordinate is calculated as the geodesic distance of (x, y) from (x0 , y), namely d ( prespect ,p ) x = d(( xi , yi ), ( x0 , yi )). The same holds for thed partial to y. − ( x0 , y ) ) 1 with ( ( x, y ) ,derivative ∂ 2σ g ( x, y;σ ) = g x = − e ∂x 2πσ 2 d(p,p0 )2 σ2 (11) − d(( x,y),( x0 ,y)) 1 ∂ 2 ( x, y ) σ, (2x, y0 ) )2πσ12 e − d2σ( p2σ,p 2) ( ∂∂x g( x, y; σ) = gx =d− (11) g ( x, y;σ ) = g y = − ed(p,p ) ∂y∂ g( x, y; σ) = gy = − d(( x,y σ)2,( x,y0 )) 21πσe2− 2σ20 ∂y σ2 2πσ2 2

0 2

2

0 2


9 of 21 9 of 21

Figure 7 shows the two partial derivatives Gaussian kernel, as defined in Equation (11) at different J. Imaging 2018,74, shows x FOR PEER 9 ofat 21 Figure the REVIEW two partial derivatives Gaussian kernel, as defined in Equation (11) position on the fisheye image. Radial lines emanating from the CoV have constant azimuthal θ. Points different position on the fisheye image. Radial lines emanating from the CoV have constant azimuthal on the centered oncentered the derivatives CoVonhave constant elevation φ. Figure 7 circles shows the two partial Gaussian kernel, as definedφ.in Equation (11) at θ. concentric Points on the concentric circles the CoV have constant elevation different position on the fisheye image. Radial lines emanating from the CoV have constant azimuthal θ. Points on the concentric circles centered on the CoV have constant elevation φ.

Figure 7. The kernels of the first partial derivatives gx, gy of Equation (11) at different positions on the

Figure 7. The kernels of the first partial derivatives gx , gy of Equation (11) at different positions on the fisheye image. fisheye image. Figure 7. The kernels of the first partial derivatives gx, gy of Equation (11) at different positions on the fisheye image Similarly, the. Laplacian of Gaussian for the calibrated fish-eye image is defined: 2 Similarly, the Laplacian of Gaussian2 for the calibrated fish-eye image is defined: 2 d ( p , p0 ) − p, p0 ) − for 2σ 2the calibrated d (fish-eye p, p0 ) −image 2σ 2 is1 defined: Similarly, the Laplacian ofd Gaussian ( 2σ 2 (12) LoG ( x, y;σ ) = g ( p;σ ) = e 4 42 2 2 22πσ d(p,p20 )2 2 2σ 2 2 d(p, p0 )σ2 − d(p, pσ 1 − 2σ ) d p p , ( ) 2 0 0 2 1 2 e−− 2σ2σ 2 LoG ( x, y; σ) = d ( p, p40 ) − 2σ g(p; σ) = d ( p, p0 )4 − 2σ (12) (12) LoG ( is x, shown y;σ ) = at different g ( p;σ = fisheye e 8. It can be easily σ 4 locations σ 4 image in2πσ The LoG kernel on) the Figure 2 2πσ σ σ observed that the LoG defined using the geodetic distance between pixels depends on the location of The LoG kernel is shown at different locations on the fisheye image in Figure 8. It can be easily The that LoG is shown the pixel itkernel is applied to. at different locations on the fisheye image in Figure 8. It can be easily observed that the LoG defined using the betweenpixels pixels depends on the location observed that the LoG defined using thegeodetic geodetic distance distance between depends on the location of of the pixel that it is applied to. the pixel that it is applied to.

Figure 8. The LoG kernels of Equation (12) at different positions on the fisheye image. Figure 8. The LoG kernels Equation (12) at different ong, the 2.3. An Efficient Implementation of(12) the Geodetically Corrected gthe x,fisheye gyfisheye and image. LoGimage. Kernels for Figure 8.Computational The LoG kernels ofofEquation at differentpositions positions on Fisheye Images for Any Value of σ 2.3. An Efficient Computational Implementation of the Geodetically Corrected g, gx, gy and LoG Kernels for The spatial dependence of the proposed corrected g, Gaussian kernel and its for 2.3. An Efficient Computational of the geodetically Geodetically Corrected gx , gy and LoG Kernels Fisheye Images for Any Value Implementation of σ derivatives imposes a very high computational load, since the kernel itself has to be recomputed at Fisheye Images for Any Value of σ The spatial dependence of the proposed In geodetically Gaussian kernelthat anduses its each image pixel, prior to the actual convolution. this section,corrected an approach is presented The spatial dependence ofGaussian the proposed corrected Gaussian its derivatives imposes high computational the kernel itselfofhas to be and recomputed at aderivatives small look-up table aofvery the kernelgeodetically for aload, smallsince pre-selected value σ.kernel imposes very high load, since theInkernel itself has to be recomputed each image eachaimage pixel,computational prior to the actual convolution. this section, an approach is presentedatthat uses smalltolook-up tableconvolution. of the Gaussian for a small pre-selected value of σ.that uses a small look-up pixel,aprior the actual In kernel this section, an approach is presented

table of the Gaussian kernel for a small pre-selected value of σ.


10 of 21 10 of 21

First we kernel forfor an image, using Equation (8), defined over aover 5 × 5aimage wegenerate generatethe theGaussian Gaussian kernel an image, using Equation (8), defined 5×5 window. This is aThis demanding computation that only needs be performed after theonce calibration of image window. is a demanding computation that only needs beonce performed after the the camera. of The is storedkernel for each theeach fisheye image. a total memory calibration theGaussian camera. kernel The Gaussian is pixel storedoffor pixel of theThus fisheye image. Thusfor a atotal floatmemory array offor NLa×float Nc × 25 isofrequired. array NL × Nc × 25 is required. The method proposed in this work for the the implementation implementation of 2D convolution convolution using Gaussian kernel with higher σ, is successive successive convolutions convolutions with with aa smaller, smaller, initially initially constructed constructed Gaussian Gaussian kernel. kernel. It is well known known that that convolving convolving two two Gaussians Gaussians with with standard standard deviations deviations σσ11, σ22 results in another Gaussian This fact is tested ofof thethe definition in Equation (9). Gaussian with withσσ22 == σσ1122 + σ2222.. This testednumerically numericallyininthe thecase case definition in Equation More specifically, a point in the in fisheye image is selected near thenear edgethe of the FoV thewhere deviation (9). More specifically, a point the fisheye image is selected edge of where the FoV the from the simple projective image formation is more is significant. The initial Gaussian kernelkernel g0 is deviation from the simple projective image formation more significant. The initial Gaussian constructed according to Equation (9) with a stencil of 13 g0 is constructed according to Equation (9) with a stencil of × 13 13. × 13.The Thestandard standarddeviation deviationσσ0 0of of gg00 is calculated using Equation (8). A number of successive linear convolutions (*) is performed performed

g0 * g0 * g0 * …* g0 (13) g0 * g0 * g0 * . . . * g0 (13) 1/2 Thus, the result of the kth convolution should be a Gaussian with σ0·(k + 1) . Thus, result of kth convolution should be a Gaussian with σ0 ·(k(b), + 1)at1/2a. position near the Figurethe 9 shows thethe construction of a 33 × 33 stencil Gaussian kernel shows thesuccessive construction of a 33 × 33 Gaussian (b),kernel at a position near edgeFigure of FoV9 (a), using convolutions ofstencil an initial smallerkernel 11 × 11 (centered at the the edge of FoV (a), using successive convolutions of an initial smaller 11 × 11 kernel (centered at the same pixel). sameIn pixel). order to facilitate visual comparison, the midline of each kernel is plotted as a curve in Figure In order to facilitate visual comparison, the midline of each kernel is plotted a curve in FigureThe 9d, 9(d), up to the first 8 successive convolutions (the first kernel is the taller, as narrower curve). up to the first 8 successive convolutions (the first kernel is the taller, narrower curve). The geodesic geodesic Gaussian that is generated using Equation (9) directly with the appropriate σ, is Gaussian that isusing generated using directlythat withthe theresult appropriate is superimposed using superimposed symbol “□”.Equation It can be(9) observed of the 8σ,successive convolutions symbol “ ”. It can be observed that the result of the 8 successive convolutions with the small initial σ0 with the small initial σ0 is almost identical to the result of a single convolution with the much larger is almost identical to the result of a singlekernel. convolution with the much larger and computationally very and computationally very demanding The asymmetry of the Gaussian kernel is shown in demanding kernel. The asymmetry of the Gaussian kernel is shown in Figure 9c using the midline Figure 9c using the midline profile and the mid-column profile of the kernel, which in this specific profile profile of the which this specific of and the image, coincide locationand of the themid-column image, coincide with thekernel, direction of in constant (iso-)location azimuth constant (iso-) with the direction of two constant (iso-) and constant (iso-) elevation. Theto same two profiles elevation. The same profiles areazimuth also plotted after reflection with respect the middle pointare of also plotted after reflection with respect to the middle point of the profiles. It can be observed that the the profiles. It can be observed that the mid-column profile is symmetric (the reflected profile matches mid-column profile iswhile symmetric (the reflected the (the original profile), while the midline the original profile), the midline profile profile shows matches asymmetry profile and its reflection does profile showsThese asymmetry (theare profile and itsaccording reflectionto does match). These aresubsection expected, not match). findings expected, thenot discussion in thefindings previous according (Figure 6).to the discussion in the previous subsection (Figure 6).

(a)

(b) Figure 9. Cont.

J. Imaging 2018, 4, 73

11 of 21


11 of 21

10 -3

2.5

Midline profile Midline Profile reflected Mid-column profile Mid-column Profile reflected

2

1.5

1

0.5

0

0

5

10

15

20

25

30

35

Pixels

(c) 10

8

-3

7 6 5 4 3 2 1 0

0

5

10

15

20

25

30

35

pixels (d)

Figure 9. (a) (a) construction construction of of aa 33 ××33 33stencil stencilGaussian Gaussiankernel kernelatataaposition positionnear nearthe theedge edgeof ofFoV FoV(b), (b), Figure using successive of an smaller 11 × 11 the same pixel). (b): pixel). the constructed using successiveconvolutions convolutions of initial an initial smaller 11kernel × 11 (at kernel (at the same (b): the kernel and (c) its lineand profiles iso-azimuth (midline) and iso-elevation (mid-column) direction, constructed kernel (c) itsalong line the profiles along the iso-azimuth (midline) and iso-elevation (mid(d): The midline of the intermediately generated kernels up to the 8th iteration are shown with column) direction, (d): The midline of the intermediately generated kernels up to the 8th iterationdots. are The midline the directly constructed shown with “kernel ”. The successive convolutions shown with of dots. The midline of the kernel directlyis constructed is result shownofwith “□”. The result of of the initial Gaussian kernel. successive convolutions of the initial Gaussian kernel.

Thus, the the proposed, proposed, efficient efficient algorithm Thus, algorithm to to calculate calculatethe theconvolution convolutionof ofaacalibrated calibratedfisheye fisheyeimage imageI using geodetic geodetic pixel pixel distance, distance, I* *g(x, g(x,y;y;σ)σ)with withaaGaussian Gaussianwith withlarge largevalues values of of standard standard deviation deviation σ, σ, using is as following Algorithm 2. is as following Algorithm 2. Algorithm 2. 2. The Theproposed proposed algorithm algorithm for for calculating calculating the the response response of of aa geodetically geodetically defined defined Gaussian Gaussian kernel kernel Algorithm on a calibrated fisheye image, for large values of σ. on a calibrated fisheye image, for large values of σ.

kk == 00 Initialize gg00 == g(x, y; σ0 )0) Initialize √ g(x, y; σ while σ k + 1 < 0 while σ0 k + 1 < σσgg k=k+1

kI == kI *+g1 0 end I = I * g0 end

J. Imaging 2018, 4, 73

12 of 21

2.3.1. Approximating the Normalized Geodetic LoG Using DoG The use of Difference of Gaussians (DoG) to approximate a Laplacian of Gaussian is well known in the literature [13], although the relation between the standard deviation of the two Gaussians and the Laplacian is seldom discussed in detail. Many approaches use the concept of octaves, namely define a number of convolution operations (typically 5) in order to double the value of σ and then resample the image to reduce the number of lines and column by two. The difference of the results of successive convolutions (DoG) approximates the convolution of the original image with the LoG. This multi-resolution approach is not directly applicable in the case of fisheye image processing, where generation of Gaussian kernels with different σ is computationally expensive. Thus the following approach is proposed, which consists of constructing a single geodetically corrected Gaussian kernel with small standard deviation σ0 (whose values depend on its spatial location on the image) and perform successive convolutions with this kernel. At the kth iteration, the standard deviation of the equivalent Gaussian kernel will be equal to

√ σ = σ0 k + 1.

(14)

The iterations are stopped when the equivalent σ becomes greater than or equal to the required standard deviation. Differences of successive convolutions are assigned to IL variable, since they approximate the response of the original image with the LoG. The factor ak than normalizes the response of LoG with respect to σ is discussed. The steps of the Algorithm 3 for approximating the normalized LoG with DoG for a calibrated fisheye image are given below. A schematic diagram of the proposed algorithm is shown in Figure 10. Algorithm 3. Approximating normalized LoG with DoG in the case of fisheye image. k=0 Initialize g0 = g(x, y; σ0 ), as in Equation (9) √ while σ0 k + 1 < σg k=k+1 I 1 = I * g0 IL = (I1 − I)ak I = I1 end

Since LoG is used in the concept of different image scales, the response of the LoG needs to be normalized with respect to scale σ. It is well known that σ2 ·LoG is scale-normalized: σ22 ∑| LoG ( x; σ2 )| = x

∑ σ21 | LoG(x; σ1 )|, σ2 > σ1

(15)

x

It is easy to confirm that the partial derivative of a Gaussian with respect to σ is ∂g( x; σ) = σ · LoG ∂σ If the derivative is approximated using finite differences, then ∂g( x; σ) g( x; σ + δσ) − g( x; σ) = ≈ σ · LoG ⇒ DoG = δσ · σ · LoG ∂σ δσ

(16)

Thus the normalization factor in this case is ak =

σ δσ

(17)

J. Imaging 2018, 4, 73

13 of 21

√ √ During the kth iteration δσ is equal to σ0 k + 1 − σ0 k. Substituting δσ in (16) we obtain: J. Imaging 2018, 4, x FOR PEER REVIEW

13 of 21

√ ak = √

k+1 √

(18)

k +1 ak = k + 1 − k k +1 − k

σ0 4

(18)

I∗g∗g∗g∗g

∗

-

a3

I∗LoG

I∗g∗g∗g

σ0 3

1 0.9 0.8 0.7

∗

0.6 0.5 0.4 0.3

-

a2

I∗LoG

0.2 0.1 0 -4

-3

-2

-1

0

1

2

3

4

I∗g∗g

Gaussian g with σ0

σ0 2

I

I∗LoG

∗ I∗g

σ0

DoG approx. of LoG

∗ Figure 10. Schematic representation responseofofthethe Gaussian kernel the LoG, Figure 10. Schematic representation of of the the image image response Gaussian kernel and and the LoG, approximated as DoG, forfor increasingly approximated as DoG, increasinglyvalues valuesof ofσ. σ.

Approximation of First Order PartialDerivatives Derivatives of Gaussian 2.3.2.2.3.2. Approximation of First Order Partial ofthe theGeodesic Geodesic Gaussian First order partial derivatives of the Gaussian kernel are fundamental for image processing and

First order partial derivatives of the Gaussian kernel are fundamental for image processing and feature extraction. Although these kernels are separable and therefore efficiently computed under feature extraction. Although these kernels are separable and therefore efficiently computed under their their classic definition Equation (10), the geodesic definition of Equation (11) imposes high classic definition (10), the geodesic definition (11) method imposes computational computationalEquation complexity, especially for large values ofof σ.Equation The proposed forhigh approximating complexity, especially for large values of σ. The proposed method for approximating geodetically corrected gx, gy with large values of σ is the following. According to theirgeodetically classic corrected gx , gysince withthe large valuesisofa σslow is the following. According to their classic definition, since definition, Gaussian varying function, its derivative gx may be approximated as the finiteisdifferences. In thefunction, case of geodetically defined Gaussian, neighboring have variableIn the Gaussian a slow varying its derivative gx may be approximated as pixels finite differences. distance accordingdefined to its locations, given in Dx, Dy defined in Equation (4). More formally: case of geodetically Gaussian, neighboring pixels have variable distance according to its locations, given in Dx , Dy defined in Equation (4). 1 More formally: g x ( x, y ) =

Dx ( x , y )

[1, −1] ∗ g ( x, y )

gx ( x, y) = D (11x,y) [1, −1]T ∗ g( x, y) g y ( x, y ) = x1 [1, −1]T ∗ g ( x, y ) gy ( x, y) = DDy((x,y x,)y[)1, −1] ∗ g( x, y) y

(19)

(19)

2.3.3.2.3.3. Detecting Corner Pixels in Fisheye Image Detecting Corner Pixels in Fisheye Image Corner pixels are salient image pixels thatthat playplay important rolerole in many image analysis tasks [26,27]. Corner pixels are salient image pixels important in many image analysis tasks The Harris corner detector is a well-established detector that utilizes the local image structure [26,27]. The Harris corner detector is a well-established detector that utilizes the local image structure by by calculating 2nd order moment matrix at each image pixel: calculating the 2ndthe order moment matrix at each image pixel:   I x2 2 I x I y  !I    ∑ 2 Ixx , y∈w ∑ Ix Iy  I I  x , y∈w x x y 2 M = w ( x,Ixy )  Ix Iy 2  =  ∈w x,y ∈w   x,y I I   M = w( x, y) Iy = IxI y I y2  2     ∑ Ix Iy  x Iyy2 I I I ∑ x y w x , y∈w  x , y∈x,y  y ∈w x,y∈w

(20)

(20)


14 of 21

J. Imaging 2018, 4, 73

14 of 21

where I x , I y are the image partial derivatives and w(x, y) is a user defined window. The image derivatives I x , I y can be approximated using derivative of the Gaussian kernel with appropriate where Ix , Iy are the image partial derivatives and w(x, y) is a user defined window. The image The summations using over the window of w are as convolution with a standard deviation σD.be derivatives Ix , Iy can approximated derivative the implemented Gaussian kernel with appropriate Gaussian with standard deviation σ > σ / 0.7 [26]. I D standard deviation σD . The summations over the window w are implemented as convolution with a Gaussian with standard deviation σ I > σD /0.7 [26]. 2  I x I y# 2 " (Ix ) M = (σ D )2 g (σ I ) ∗ ( Ix )2 Ix Iy 2  M = (σ D ) g (σ I ) ∗  I x I y I2   Ix Iy Iy y 

( )

(21) (21)

Corner response is calculated using the standard metric: det(M)-λ·trace(M)2, where λ a constant Corner response is calculated using the standard metric: det(M)-λ·trace(M)2 , where λ a constant parameter with value in (0.04, 0.06). parameter with value in (0.04, 0.06). Direct application of the geodetically corrected derivative of the Gaussian, as defined in Direct application of the geodetically corrected derivative of the Gaussian, as defined in Equation (11) is not computationally efficient for arbitrary values of σ. The approximation of Equation (11) is not computationally efficient for arbitrary values of σ. The approximation of Equation (19) is used instead. Thus, the Algorithm 4 that we propose for estimation of matrix Μ in Equation (19) is used instead. Thus, the Algorithm 4 that we propose for estimation of matrix M in the the case of a calibrated fisheye image is described as following: case of a calibrated fisheye image is described as following: Algorithm 4. The approximation of the Harris matrix M of Equation (21). Algorithm 4. The approximation of the Harris matrix M of Equation (21).

Apply Algorithm 2 to calculate I * g Apply Algorithm 2 to calculate I * g Apply Equation (19) to calculate I x , I y Apply Equation (19) to calculateIx , Iy

Apply Algorithm 2 (I tox 2 ),( I(Ix v)2 ),, (IIxυIy) with with onetoiteration the with convolution with , I x I yonly Apply Algorithm 2 to oneonly iteration calculateto thecalculate convolution the geodetic 2

2

Gaussian g0 . Gaussian g0. the geodetic Construct matrix M for pixelpixel Constructthe thesymmetric symmetric matrix M each for each

3. Results 3. Results The fisheye camera areare shown in Figure 11 for The results results of ofAlgorithm Algorithm22for foran animage imageofofthe thecalibrated calibrated fisheye camera shown in Figure 11 iteration k = 1 (first row), k = 10, k = 20, k = 30 and k = 40 (last row). The left column shows the result of for iteration k = 1 (first row), k = 10, k = 20, k = 30 and k = 40 (last row). The left column shows the result Algorithm 2 (using the geodetically defined Gaussian kernel), the middle column the response of the of Algorithm 2 (using the geodetically defined Gaussian kernel), the middle column the response of the classic Gaussian kernel kernel and and the the right right column column the the absolute absolute difference, difference, scaled scaled for for visualization visualization purposes. purposes. classic Gaussian

Figure 11. The results of Algorithm 1 for the geodetic Gaussian (left column) for the classic Gaussian Figure 11. The results of Algorithm 1 for the geodetic Gaussian (left column) for the classic Gaussian (middle column and their absolute difference (appropriately scaled for visualization). (middle column and their absolute difference (appropriately scaled for visualization).


15 of 21 15 of 21

Resultsofofthe the response of geodesic the geodesic gx kernel, as approximated using Algorithm 2 and Results response of the gx kernel, as approximated using Algorithm 2 and Equation Equation (18), for kiteration k =shown 20 are shown in Figure (left column). the right column (18), for iteration = 1 andk k= =1 and 20 are in Figure 12 (left12column). In theInright column the the magnitude of the image gradient is shown, using responsesofofthe thetwo twopartial partialderivatives derivatives of of magnitude of the image gradient is shown, using thethe responses the geodetic geodetic Gaussian Gaussian kernel. kernel. It is is evident evident that that for for small small value value of of σσ == σσ00,, fine fine details details appear appear in in the the the √ magnitude 21 ),only onlythick thicklines lines appear appearin in magnitude of of the the gradient, gradient, whereas, whereas, in in the the case case of of kk == 20 (thus σ σ = σσ00 21), the magnitude of the gradient image. the magnitude of the gradient image.

(a)

(b)

using the the proposed proposed algorithm algorithm for for different different scales scales (k (k == 11 and and kk == 20 20 in in Figure 12. 12. (a) (a) The The resulting resulting IIx using Figure x Equation (19)); (b) the magnitude of the image gradient at the same scales. Equation (19)); (b) the magnitude of the image gradient at the same scales.

The results of the maximum response of the normalized geodetically corrected LoG, The results of the maximum response of the normalized geodetically corrected LoG, approximated approximated by DoG (Algorithm 3) are shown in Figure 13. The initial Gaussian kernel g0 was by DoG (Algorithm 3) are shown in Figure 13. The initial Gaussian kernel g0 was defined over a defined over a 7 × 7 window and 40 iterations were executed. Yellow indicates pixels that exhibited 7 × 7 window and 40 iterations were executed. Yellow indicates pixels that exhibited maximum DoG maximum DoG response within the first 4 iterations, whereas magenta indicates pixels with response within the first 4 iterations, whereas magenta indicates pixels with maximum DoG response maximum DoG response between 5 and 10 iterations. In (a) the whole fisheye frame is displayed, between 5 and 10 iterations. In (a) the whole fisheye frame is displayed, whereas in (b) details are whereas in (b) details are shown for a zoomed portion of the image. shown for a zoomed portion of the image. A comparison between the classic LoG response and the response of the geodetically defined A comparison between the classic LoG response and the response of the geodetically defined LoG, approximated by the DoG, (after normalization in both cases) is given in Figure 14(a). The LoG, approximated by the DoG, (after normalization in both cases) is given in Figure 14a. The curves curves have been produced for the pixel shown in the image portion of Figure 14(b). have been produced for the pixel shown in the image portion of Figure 14b.

J. Imaging 2018, 4, 73

16 of 21


16 of 21


16 of 21

(a)

(b)

Figure 13. (a) The results of the maximum response of the normalized geodetically corrected LoG,

Figure 13. (a) The results of(a)the maximum response of the normalized geodetically corrected LoG, approximated by DoG (Algorithm 3), exhibited within the first 4 iterations(b) (yellow pixels) and approximated by DoG (Algorithm 3), exhibited within the first 4 iterations (yellow pixels) and between between andThe 10 iterations (b) detailed image portion. Figure 13.5 (a) results of(magenta); the maximum response of the normalized geodetically corrected LoG, 5 and 10 iterations (magenta); (b) detailed image portion. approximated by DoG (Algorithm 3), exhibited within the first 4 iterations (yellow pixels) and between 5 and 10 iterations (magenta); (b) detailed image portion.

(a)

(b)

Figure 14. (a) The response of the normalized DoG for the pixel shown in (b). The small differences (b) between the two curves are(a) evident. Figure 14. (a) The response of the normalized DoG for the pixel shown in (b). The small differences

Figure 14. The response of the normalized forinthe pixel15shown (b). The small differences The(a) response Harris detection isDoG shown Figure for theinwhole frame. Yellow dots between the twoofcurves arecorner evident. indicate detected the proposed geodetic Harris detector, whereas magenta squares depict between thecorners two curves areby evident. corner detected by the classic Harris is detector blue 15 crosses are corners detected bydots the Thepixels response of Harris corner detection shown and in Figure for the whole frame. Yellow method corners described in [22].by Allthe detectors were appliedHarris for a number differentmagenta image scales. Common indicate detected proposed geodetic detector,ofwhereas squares depict The responseisof Harris corner detection is shown in Figurepresented 15 for the whole frame. Yellow dots image scale determined size of the kernels. the results in this section, theby initial corner pixels detected by by thethe classic Harris detectorInand blue crosses are corners detected the indicatemethod corners detected by the proposed geodetic Harris detector, whereas magenta squares small geodetic Gaussian is set to a size of 5applied × 5 pixels. sizeofofdifferent the equivalent kernelCommon after the depict described in [22]. All detectors were for a The number image scales. corner pixels detected by the classic detector and blue crosses are corners detected theismethod subsequent is shown 1 and The classic Harris-based detection image scale isconvolutions determined by theHarris sizeinofTables the kernels. In2.the results presented in thiscorner section, theby initial applied using this size is forset the kernel (which slightly largerCommon than the the image described in geodetic [22]. AllGaussian detectors were applied number ofsize different image scales. small to aintegration size of 5for ×Gaussian 5apixels. The of the is equivalent kernel after differentiating Gaussian). The geodetic neighborhood method [22] is parameterized by the kernel’s subsequent convolutions is shown in Tables 1 and 2. The classic Harris-based corner detection is small scale is determined by the size of the kernels. In the results presented in this section, the initial field of view, measured in radians or degrees Gaussian and the number of pixels is of the kernel’s stencil. After applied using this size for the integration kernel (which slightly larger than the geodetic Gaussian is set to a size of 5 × 5 pixels. The size of the equivalent kernel after the subsequent visually studying the image that is required by this (an example is shown in differentiating Gaussian). Theresampling geodetic neighborhood method [22]method is parameterized by the kernel’s convolutions is shown in Tables 1 and 2. The classic Harris-based corner detection iswas applied using Figure 6d) the selected parameterization was as following: a field of view (FoV) of 1 degree used field of view, measured in radians or degrees and the number of pixels of the kernel’s stencil. After this sizevisually for5 the integration Gaussian (which islarger slightly larger theexample differentiating Gaussian). for × 5 studying and 9 × 9 the andimage an FoV of 2kernel degrees for kernels. Thethan numerical assessment of the resampling that is the required by this method (an is shown in methods is two-fold. First the 250 strongest corner responses are considered and the number of The geodetic neighborhood method [22] is parameterized by the kernel’s field of view, measured Figure 6d) the selected parameterization was as following: a field of view (FoV) of 1 degree was used

in radians number of pixels of larger the kernel’s stencil. Afterassessment visually studying the for 5or × 5degrees and 9 × 9and and the an FoV of 2 degrees for the kernels. The numerical of the methods is two-fold. First the 250 strongest corner responses are considered and the number of image resampling that is required by this method (an example is shown in Figure 6d) the selected parameterization was as following: a field of view (FoV) of 1 degree was used for 5 × 5 and 9 × 9 and an FoV of 2 degrees for the larger kernels. The numerical assessment of the methods is two-fold. First the 250 strongest corner responses are considered and the number of correctly identified corners in the chessboard are enumerated. The chessboard contains 80 corners. The results are presented

J. Imaging 2018, 4, 73

17 of 21


17 of 21

in Tablecorrectly 1. Theidentified zoomedcorners chessboard areas are are shown in Figure for the contains first four in the chessboard enumerated. The 15 chessboard 80 kernel corners. sizes of Table 1.The Secondly, the number of wrong corner responses is measured by visual inspection results are presented in Table 1. The zoomed chessboard areas are shown in Figure 15 for the for all first four for kernel sizes Table 1. kernel Secondly, the (equivalently, number of wrong corner responses measured three methods, four (4) of different sizes image scales). Theisresults are by measured visual inspection for all three methods, for four (4) different kernel sizes (equivalently, image scales). considering the 250, as well as the 500 strongest corner responses from each method. The results are The results are measured considering the 250, as well as the 500 strongest corner responses from each presented in Table 2. method. The results are presented in Table 2.

(a)

(b)

(c)

(d)

Figure 15. The chessboard area containing 80 corners, with detected corners form the three methods,

Figure considering 15. The chessboard area containing 80 corners, with detected corners form the three methods, only the 250 strongest responses. (a) 5 × 5 kernels, (b) 9 × 9 kernels, (c) 13 × 13 kernels and considering only the 250 strongest responses. (a) 5 × 5 kernels, (b) 9 × 9 kernels, (c) 13 × 13 kernels (d) 17 × 17 kernels. and (d) 17 × 17 kernels. Table 1. Quantitative evaluation, in terms of corner detection, of the proposed geodetic kernel the geodetic neighborhood in detection, [22] and classic Harris-based detection. Quantitative evaluation, in termsdefinition of corner of the proposedcorner geodetic kernelThe definition, Table 1.definition, number of correctly detected corners in the chessboard (80 corners in total) by each method, the geodetic neighborhood definition in [22] and classic Harris-based corner detection. The number considering the 250 strongest responses are shown.

of correctly detected corners in the chessboard (80 corners in total) by each method, considering the Kernel responses Size Classic Harris Proposed Geodetic Kernels Geodetic Neighborhood [22] 250 strongest are shown. 5×5 ×9 Kernel 9Size 13 × 13 5 ×175 × 17 9 ×219 × 21

13 × 13 17 × 17 21 × 21

76 Classic80 Harris 75 7653 8048

75 53 48

80 80 Proposed Geodetic Kernels 80 8080 8080

80 80 80

80 Geodetic 80 Neighborhood [22] 80 80 80 80 80

80 80 80

J. Imaging 2018, 4, x FOR PEER REVIEW J. Imaging J. Imaging2018, 2018,4,4,73x FOR PEER REVIEW

18 of 21 18 of1821of 21

Table 2. False corner detection of the proposed geodetic kernel definition, the geodetic neighborhood Table 2. False corner of the proposed geodetic kernel definition, thewrongly geodeticdetected neighborhood definition in [22] anddetection classic Harris-based corner detection. The number of corners Table 2. False corner detection of the geodetic kernel definition, the geodetic neighborhood definition in [22] and classic Harris-based detection. The number of wrongly detected corners by each method, considering the 250proposed and corner 500 strongest responses are shown. definition in [22] and classic Harris-based corner detection. The are number of wrongly detected corners by each method, considering the 250 and 500 strongest responses shown. Kernel by each method, considering the 250 and 500 strongest responses are shown. 250 Strongest Corner Responses 500 Strongest Corner Responses

Kernel Size Size

Kernel Size 55×× 55 599×××599 13 × 13 9 ××913 13 17 × 17 13 17×××13 17 21 21

250 Strongest Corner Responses ProposedCorner Responses 250 Strongest Classic Geodetic Geodetic Proposed Classic Proposed Geodetic[22] Harris Neighborhood Classic Geodetic Kernels GeodeticKernels Neighborhood [22] Harris Geodetic Harris Neighborhood [22] Kernels 99 2 2 2121

500 Strongest Corner Responses ProposedCorner Responses 500 Strongest Classic Geodetic Geodetic Proposed Classic Proposed Geodetic[22] Harris Neighborhood Classic Geodetic Kernels Geodetic Harris Geodetic Kernels Neighborhood [22] Harris Neighborhood [22] Kernels 1111 4 4 68 68

91010

21 1

21 3131

11 3535

45 5

68 72 72

12 19 12 1928

2 4 24 8

17 28 17 2827

42 70 42 7068

7 8 78 12

69 72 69 72 65

17 21××17 21

19 28

48

28 27

70 68

812

72 65

21 × 21

28

8

27

68

12

65

10 12

12

31 17

35 42

57

72 69

Two different parts of the fisheye frame are zoomed in Figure 16. Visual inspection of the results Two different parts of the fisheye frame are zoomed in Figure 16. Visual inspection of the results suggests the proposed geodetic Harrisare detector has fewer16. false corners than of thetheclassic one. Twothat different of the fisheye frame zoomedhas in Figure Visual inspection results suggests that the parts proposed geodetic Harris detector fewer false corners than the classic one. Figure 17 shows a few sporadic false corners detected by the classic Harris method, probably due suggests the aproposed geodetic fewer falseHarris corners than the classicdue one.to to Figure 17that shows few sporadic false Harris cornersdetector detectedhas by the classic method, probably the texture ofof the floor. [22] false detections mainlymethod, onimage image edges.due to Figure 17 shows a floor. few The sporadic falseofof corners detected by the classic Harris probably the texture the Themethod method [22]exhibits exhibits false detections mainly on edges. the texture of the floor. The method of [22] exhibits false detections mainly on image edges.

(a) 250 strongest responses (b) 500 strongest responses (a) 250 strongest responses (b) 500 strongest responses Figure 16. Corner pixels detected by the proposed geodetically corrected Harris detector (shown in Figure 16. Corner pixels detected by the proposed geodetically corrected Harris detector (shown in Figure pixels detected by the proposed geodetically corrected Harris yellow)16.byCorner the classic detector (magenta) and by the geodetic Harris in [22] (bluedetector crosses),(shown with 5 in ×5 yellow) by the classic detector (magenta) and by the geodetic Harris in [22] (blue crosses), with 5 × 5 yellow) by the classic detector (magenta) and by the geodetic Harris in the [22]500 (blue crosses),responses. with 5 × 5 kernel size, considering: (a) the 250 strongest corner responses and (b) strongest kernel size, considering: (a) the 250 strongest corner responses and (b) the 500 strongest responses. kernel size, considering: (a) the 250 strongest corner responses and (b) the 500 strongest responses.

(a) (b) (a) (b) Figure 17. (a) and (b): Two different parts of the fisheye frame from the result with 500 strongest Figure (a) and using (b): Two different fisheye frame from the as result with 500 strongest corner 17. responses, kernels of sizeparts 17 × of 17 the (color encoding is the same in the previous Figure). Figure 17. (a) and (b): Two different parts of the fisheye frame from the result with 500 strongest corner corner responses, using kernels of size 17 × 17 (color encoding is the same as in the previous Figure). responses, using kernels of size 17 × 17 (color encoding is the same as in the previous Figure).

J. Imaging 2018, 4, 73

19 of 21

The presented results can be summarized as following. The proposed definition and implementation of image processing geodetic kernels exhibited very good corner detection over a number of different kernel sizes: it managed to detect all corners in the chessboard area for kernel size ranging between 5 × 5 to 21 × 21 pixels. The same holds for the method of geodetic image neighborhood resampling [22]. On the other hand, the classic Harris-based corner detection does not perform well on the fisheye images when the size of the kernels (image space) increases. Concerning false corner detection, the proposed algorithm achieved very low rate for the 5 different kernel sizes that it was tested. On the contrary, the classic Harris-based detector exhibited increased false detection rate as the kernel size increases, whereas the method of [22] exhibited high false detection rate for almost all tested kernel sizes. The main algorithms described in this work have been executed on a laptop with Intel core i7-7700 HQ, 16 GB, with Windows 10, using the Matlab programming environment. The algorithms were applied to 480 × 640 fisheye images. The recorded times of various steps of the methods are shown in Table 3. It has to be mentioned again that the slowest part of the proposed algorithm, the initial calculation of the geodetic Gaussian kernel g0 with small σ0 (first entry in Table 3) is executed only once, after the calibration of the camera and it can be subsequently used for any image acquired by the specific camera. As it can be observed in Table 3, Algorithm 1 is the computationally intensive part of the proposed method. Execution times using the classic kernels are also provided for comparison. The geodetic approach for fisheye image processing using geodetic neighborhood [22] has also been timed. The slowest part of the method is the image resampling, which was timed for the following parameterization: FoV of 1 degree, kernel size of 5 × 5. Increasing the FoV and consequently the kernel size imposes a significant overhead to the algorithm [22]. Furthermore, this operation has to be performed for every acquired image. Table 3. Execution times for the proposed method, corner detection in [22] and classic corner detection. Method

Calculation

Execution Time (Second)

Proposed Geodetic kernels Geodetic neighborhood [22] Proposed Geodetic kernels Geodetic neighborhood [22] Classic Harris Proposed Geodetic kernels Classic Harris Geodetic neighborhood [22]

geodetic Gaussian g0 (Algorithm 1) Image resampling, 5 × 5, FoV 1◦ Gaussian response (Algorithm 2) Gaussian response Gaussian response Harris Matrix M Harris Matrix M Harris Matrix M

5.5 150 0.15 0.017 0.017 0.5 0.5 0.5

4. Discussion In this work, the most important image processing kernels, namely, the Gaussian, its partial derivatives with respect to x and y and the Laplacian of Gaussian (LoG) have been redefined for a calibrated fisheye camera, using the geodesic distance between pixels, of the fisheye image. For each kernel, a computationally efficient algorithm that calculates/approximates the response with a fisheye image acquired by a calibrated camera, based on successive convolutions with a small pre-initialized geodetic Gaussian, has been presented. The Harris corner detection has also been implemented for a calibrated fisheye camera, using the image responses with the redefined geodetic kernels. The results show that the proposed definitions and the computational implementations are efficient and present subtle differences from the applications of the classic image processing kernels. Corner detection in calibrated fisheye images appears more robust when using the proposed operators. More specifically, both the proposed method of geodetic definition of image processing kernels and the method of geodetic image neighborhood resampling achieved very high corner detection for all tested kernel sizes. The performance of the classic Harris-based corner detector, deteriorates as the kernel size increases. The number of falsely indicated corners is much lower for the proposed method, consistently

J. Imaging 2018, 4, 73

20 of 21

for all tested kernel sizes. The geodetic neighborhood method [22] is producing a large number of false corner detections over all tested kernel sizes, whereas the classic corner detector exhibited false detection rate that deteriorates as the kernel size increases. The execution time of the proposed method was slower than the corresponding time using the classic kernels, but not prohibitive for practical use. The most demanding step was the initial calculation of the initial geodetic Gaussian g0 with small σ0 , which needs only be performed once, after camera calibration. The approximation of the Gaussian response which was averaged to 0.017 s per iteration for a 480 × 640 image. Possible parallel of this step may achieve important acceleration. Further work is needed to expand these results into more image analysis/feature extraction methods from fisheye images. Funding: This research received no external funding Conflicts of Interest: The author declares no conflict of interest.

References 1. 2. 3.

4.

5. 6.

7.

8. 9.

10.

11. 12.

13.

14.

Geyer, C.; Daniilidis, K. Catadioptric projective geometry. Int. J. Comput. Vis. 2001, 45, 223–243. [CrossRef] Bermudez-Cameo, J.; Lopez-Nicolas, G.; Guerrero, J.J. Automatic line extraction in uncalibrated omnidirectional cameras with revolution symmetry. Int. J. Comput. Vis. 2015, 114, 16–37. [CrossRef] Ahmed, M.S.; Gao, Z. Ultra-wide fast fisheye for security and monitoring applications. In Proceedings of the International Symposium on Optoelectronic Technology and Application 2014: Advanced Display Technology; and Nonimaging Optics: Efficient Design for Illumination and Solar Concentration, Beijing, China, 13–15 May 2014. Caruso, D.; Engel, J.; Cremers, D. Large-scale direct slam for omnidirectional cameras. In Proceedings of the 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Hamburg, Germany, 28 September–2 October 2015; pp. 141–148. Choi, Y.W.; Kwon, K.K.; Lee, S.I.; Choi, J.W.; Lee, S.G. Multi-robot Mapping Using Omnidirectional-Vision SLAM Based on Fisheye Images. ETRI J. 2014, 36, 913–923. [CrossRef] Abrams, A.D.; Pless, R.B. Webcams in context: Web interfaces to create live 3D environments. In Proceedings of the 18th ACM International Conference on Multimedia, Toronto, ON, Canada, 26–30 October 2010; pp. 331–340. Sankaranarayanan, K.; Davis, J.W. A fast linear registration framework for multi-camera GIS coordination. In Proceedings of the IEEE International Conference on Advanced Video and Signal Based Surveillance, Santa Fe, NM, USA, 1–3 September 2008. Hu, X.; Zheng, H.; Chen, Y.; Chen, L. Dense crowd counting based on perspective weight model using a fisheye camera. Optik-Int. J. Light Electron Opt. 2015, 126, 123–130. [CrossRef] Vandewiele, F.; Motamed, C.; Yahiaoui, T. Visibility management for object tracking in the context of a fisheye camera network. In Proceedings of the 2012 Sixth International Conference on Distributed Smart Cameras (ICDSC), Hong Kong, China, 30 October–2 November 2012; pp. 1–6. Wang, W.; Gee, T.; Price, J.; Qi, H. Real time multi-vehicle tracking and counting at intersections from a fisheye camera. In Proceedings of the 2015 IEEE Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA, 5–9 January 2015; pp. 17–24. Rhodin, H.; Richardt, C.; Casas, D.; Insafutdinov, E.; Shafiei, M.; Seidel, H.P.; Theobalt, C. Egocap: Egocentric marker-less motion capture with two fisheye cameras. ACM Trans. Graph. 2016, 35, 162. [CrossRef] Rhodin, H.; Robertini, N.; Richardt, C.; Seidel, H.P.; Theobalt, C. A versatile scene model with differentiable visibility applied to generative Pose Estimation. In Proceedings of the 2015 International Conference on Computer Vision (ICCV 2015), Tampa, FL, USA, 5–8 December 2015. Lowe, D.G. Object recognition from local scale-invariant features. In Proceedings of the Seventh IEEE International Conference on Computer Vision, Kerkyra, Greece, 20–27 September 1999; Volume 2, pp. 1150–1157. Hansen, P.; Corke, P.; Boles, W.; Daniilidis, K. Scale Invariant Feature Matching with Wide Angle Images. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, San Diego, CA, USA, 29 October–2 November 2007; pp. 1689–1694.

J. Imaging 2018, 4, 73

15. 16. 17.

18. 19.

20. 21. 22. 23. 24.

25. 26. 27.

21 of 21

Bulow, T. Spherical diffusion for 3D surface smoothing. IEEE Trans. Pattern Anal. Mach. Intell. 2004, 26, 1650–1654. [CrossRef] [PubMed] Cruz-Mota, J.; Bogdanova, I.; Paquier, B.; Bierlaire, M.; Thiran, J.P. Scale invariant feature transform on the sphere: Theory and applications. Int. J. Comput. Vis. 2012, 98, 217–241. [CrossRef] Puig, L.; Guerrero, J.J. Scale space for central catadioptric systems: Towards a generic camera feature extractor. In Proceedings of the 2011 IEEE International Conference on Computer Vision (ICCV), Barcelona, Spain, 6–13 November 2011; pp. 1599–1606. Puig, L.; Guerrero, J.J.; Daniilidis, K. Scale space for camera invariant features. IEEE Trans. Pattern Anal. Mach. Intell. 2014, 36, 1832–1846. [CrossRef] [PubMed] Andreasson, H.; Treptow, A.; Duckett, T. Localization for mobile robots using panoramic vision, local features and particle filter. In Proceedings of the 2005 IEEE International Conference on Robotics and Automation, Barcelona, Spain, 18–22 April 2005; pp. 3348–3353. Zhao, Q.; Feng, W.; Wan, L.; Zhang, J. SPHORB: A fast and robust binary feature on the sphere. Int. J. Comput. Vis. 2015, 113, 143–159. [CrossRef] Hara, K.; Inoue, K.; Urahama, K. Gradient operators for feature extraction from omnidirectional panoramic images. Pattern Recognit. Lett. 2015, 54, 89–96. [CrossRef] Demonceaux, C.; Vasseur, P.; Fougerolle, Y. Central catadioptric image processing with geodesic metric. Image Vis. Comput. 2011, 29, 840–849. [CrossRef] Delibasis, K.K.; Plagianakos, V.P.; Maglogiannis, I. Refinement of human silhouette segmentation in omni-directional indoor videos. Comput. Vis. Image Underst. 2014, 128, 65–83. [CrossRef] Delibasis, K.K.; Georgakopoulos, S.V.; Kottari, K.; Plagianakos, V.P.; Maglogiannis, I. Geodesically-corrected Zernike descriptors for pose recognition in omni-directional images. Integr. Comput.-Aided Eng. 2016, 23, 185–199. [CrossRef] Puig, L.; Bermúdez, J.; Sturm, P.; Guerrero, J.J. Calibration of omnidirectional cameras in practice: A comparison of methods. Comput. Vis. Image Underst. 2012, 116, 120–137. [CrossRef] Harris, C.; Stephens, M. A combined corner and edge detector. In Proceedings of the Alvey Vision Conference, Manchester, UK, 31 August–2 September 1988; Volume 15, pp. 10–5244. Mokhtarian, F.; Mohanna, F. Performance evaluation of corner detectors using consistency and accuracy measures. Comput. Vis. Image Underst. 2006, 102, 81–94. [CrossRef] © 2018 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).