A new sharpness measure based on Gaussian lines

A new sharpness measure based on Gaussian lines and edges Judith Dijk, Michael van Ginkel, Rutger J. van Asselt, Lucas J. van Vliet, Piet W. Verbeek Pattern Recognition Group, Delft University of Technology Lorentzweg 1 2628 CJ Delft The Netherlands {judith,michael,lucas,piet}@ph.tn.tudelft.nl

Abstract. We measure the sharpness of natural (complex) images using Gaussian models. We first locate lines and edges in the image. We apply Gaussian derivatives at different scales to the lines and edges. This yields a response function, to which we can fit the response function of model lines and edges. We can thus estimate the width and amplitude of the line or edge. As measure of the sharpness we propose the 5th percentile of the sigmas or the number of line/edge pixels with a sigma smaller than 1.

1

Introduction

The purpose of the research presented in this paper is to model the perceptual sharpness of natural (complex) images. Sharpness is an important perceptual attribute determining the perceptual quality of an image. In the past, we have already done some experiments regarding the relative sharpness of natural images [2]. This relative sharpness is defined with respect to an original image, whereas the sharpness measure proposed in this paper is a property of the image itself. In the previous study we have seen that images can be seen as a collection of areas that are more or less uniform, separated by lines and edges. We assume that perceptual sharpness is correlated to the sharpness of these lines and edges. To determine the sharpness we need to determine the location and orientation of the lines and edges so that we can perform measurements on their profile. To establish their sharpness we need a model for the profile. We use a Gaussian profile for the line and an integrated Gaussian (error function) for the edge. We compute Gaussian derivatives at several scales to obtain a response function or signature. At a singularity point, the response function can be predicted given the width and the amplitude of the line or edge. Conversely, we can estimate the width and amplitude of the line or edge from the measured response function. The sigmas of all points have to be combined to form one or a few measures of sharpness. An obvious measure is the smallest sigma in the image. Because measuring the smallest sigma is sensitive to noise and other artifacts, we use the

5th percentile instead of the minimum sigma. Another measure that can be used is the number of pixels for which the sigma is smaller than one. This gives some insight in the number of sharp lines and edges in the image. Kayargadde [5] proposed a similar measure for the perceptual sharpness of images. He used a polynomial transform (Hermite transform), to detect and estimate edge parameters, such as position, orientation, amplitude, mean value and sigma. Our methods differs from his on three points. The first is that we detect lines and edges and determine the width of both of them. The second difference is that we perform a numerical estimation of the amplitude, whereas Kayargadde derived an analytical relationship. And the last difference is that we estimate the orientation of the structures in the localization phase, where Kayargadde determines the orientation in the estimation phase.

2

Line and edge detection

Before we can determine the sharpness of individual lines and edges, transients or singularity points for short, we must extract them from the image. For each transient we must also establish whether it is a line or an edge and its orientation φ, as defined in figure 1. Since we must deal with both lines and edges, it is logical to use a filterbank based on quadrature filters 1 . A quadrature filter is a linear, complex-valued filter. The real and imaginary part act as a line and edge filter respectively. The magnitude of the response is phase-invariant, i.e. insensitive to whether the transient is a line or an edge. This is important when we discuss the suppression of spurious responses below.

φ

φ

Fig. 1. The definition of orientation φ for an edge (left) and a line (right)

The quadrature filter is sensitive to edges and lines under a limited range of orientations. To obtain the response under an arbitrary angle we use a steerable [3] quadrature filter: the response can be computed from the filter response under a finite set of angles.The details of the quadrature filter we use can be found in [4]. The filter’s characteristic frequency fc , the range of frequencies it is sensitive to bf and the orientation selectivity sa can be tuned independently. Our supposition is that the overall sharpness relates to the sharpest line or edge in the image. We have therefore tuned the filter in such a way that it will detect small-scale lines and edges (fc , bf ) = (0.16, 0.16). The orientation selectivity sa = 0.185 [4] (17 filters) was chosen as a trade-off between orientation selectivity, signal-to-noise ratio and localization of the filter response. 1

Knutsson and coworkers [6] were early advocates of the use of quadrature filters in image analysis.

1

1

0.5

0.9

0.9

0.45

0.8

0.8

0.4

0.4

0.7

0.7

0.35

0.35

0.6

0.6

0.3

0.3

0.5

0.5

0.25

0.25

0.4

0.4

0.2

0.2

0.3

0.3

0.15

0.15

0.2

0.2

0.1

0.1 0

0.1 100

110

120

130

(a)

140

150

160

0

line response edge response sqrt( lines 2 + edges2)

110

120

130

140

150

160

0

line response edge response sqrt( lines 2 + edges2)

0.1

0.05

100

0.5 0.45

0.05 100

110

(b)

120

130

(c)

140

150

160

0

100

110

120

130

140

150

160

(d)

Fig. 2. Line and edge response. a) A line. b) An edge. c) The filters responses to the line. d) The filter responses to the edge.

Our last task is to determine whether a detected transient is an edge or a line, and the suppression of spurious responses. To see why these occur, imagine a line. The line detector responds as it should, but the edge detector will also respond. It responds to the flanks of the line, although less strongly than the line detector. This is illustrated in figure 2. We resolve this problem by suppressing (inhibiting) the secondary responses. The final line response lline is given by  0 if ql (x, y) < maxN (x,y) qe (x, y) lline (x, y) = , (1) ql (x, y) elsewhere with ql and qe the quadrature line and edge component filter results respectively, and N (x, y) a neighbourhood around (x, y). The size of the neighbourhood must be roughly equal to the width of the response lobes. By swapping the roles of ql and qe the same technique can be used to obtain the final edge response. We select the most salient (strongest) lines and edges by selecting the 20% points with the strongest response.

3

Line and edge characterization

In this section we explain how we can determine the amplitude and sigma of Gaussian lines in images. The method we use is a variation of Mallat’s approach [8], using Gaussian derivatives rather than wavelets [11]. The idea is to compute the response of the Gaussian derivator operator applied across the transient, while varying the scale of the Gaussian. The response depends on both the scale of the Gaussian and that of the transient. Since we know the first, we may estimate the latter. It is convenient to adapt a local coordinate system (v, w) at each point (x0 , y0 ) that is aligned with the orientation φ at that point:      v cos φ sin φ x − x0 = . (2) w − sin φ cos φ y − y0 The singularity function of an infinitely long line and edge respectively are given by
hline = A δ(v) = A δ (x − x0 ) cos(φ) + (y − y0 ) sin(φ)
(3) hedge = A u(v) = A u (x − x0 ) cos(φ) + (y − y0 ) sin(φ)

where A is the amplitude of the transient, φ the angle of the transient with respect to the x-axis, δ(x) the Dirac delta function and u(x) the Heaviside step function. Real transients have a finite width. We model this by convolving these functions with a Gaussian with σl/e . The width σ1/e reflects the sharpness of the transient. To find an estimate for the σl/e and amplitude A of the lines and edges we construct a response function in the following way: we convolve the input image with directional Gaussian derivatives along φ increasing the scale exponentially: σ = bi (b > 1) with i the free scale parameter. The Gaussian regularisation has, in general, the effect that the response decreases as a function of scale, as noted by Lindeberg [7]. We follow [7] in using normalized, or scale-independent, Gaussian derivatives. This results in more pronounced response curves. The normalization consists of multiplying the response with σ. The response curve at (x0 , y0 ) is r(σ) = σ


∂gσ (v, w) ∗ h(v, w) ∗ gsl/e (v, w) ∂v

Using the commutativity of the convolution operator we obtain q
∂ 2 + σ2 . r(σ) = σ h(v, w) ∗ gs (v, w) with s = σl/e ∂v

(4)

(5)

In what follows we consider the modulus of the response M (σ) = |r(σ)|. The expression for M for a line and edge respectively is given by the following two equations: |A|σ ∂ v2 gs (v)| = √ |v| exp(− 2 ) ∂v 2s 2πs3 2 |A|σ v Medge (σ) = √ exp(− 2 ) 2s 2πs Mline (σ) = |Aσ

(6)

The modulus maxima per scale are given by |A|σ max Mline(σ) = √ at v = s and 2πes2

|A|σ at v = 0. (7) max Medge(σ) = √ 2πs

The modulus maxima for lines and edges with different sigmas are given in figure 3. These responses are the theoretical responses of lines and edges. We find the sigmas of the lines and edges in the image by fitting the measured responses to the theoretical responses. The selection of the modulus maximum for an edge is straightforward: the position of the maximum is the same as the position of the point itself. For the lines this is different, the maxima are shifted over s. To find these maxima we search for a maximum in a appropriately sized neighbourhood. The minimalization can be done by some numerical minimization method. We use a minimalization based on the Levenberg-Marquardt method. We start with the third maximum (σ = b3 ) and use 8 scales. b is chosen 21/3 , i.e. three samples per octave.

σl\e = 1 σ =2 l\e σ =4 l\e σ =8

0.14 modulus maxima

0.12 0.1

l\e

←Line upper bound

0.08 0.06 0.04

Edge upper bound

0.4 0.35 modulus maxima

0.16

0.3 0.25 0.2 σl\e = 1 σ =2 l\e σ =4 l\e σ =8

0.15 0.1

0.02

0.05

0 0

5

10

15

0 0

l\e

5

scale i

10

15

scale i

Fig. 3. The modulus maxima of the response at different scales. A = 1, b is 21/3

(a) trui

(b) erika

(c) cameron

(d) lenna

(e) bicycle

(f) portrait

Fig. 4. The test images.

4

Sharpness measures

The final step is to obtain one or a few sharpness measures using the sigmas. We define and evaluate three sharpness measures. The first measure is the value of the 80th percentile of the filter responses. This percentile was used before to select the lines and edges. Note that this measure can be determined before determining the sigma of the singularities. The second measure we looked into is the number of points with a sigma smaller than 1. This is a very intuitive measure, just the number of points with a small sigma. The third measure is the sharpest line or edge in the image. Ideally, we would want to use the smallest sigma. However, measuring the smallest sigma is sensitive to noise and other artifacts. Therefore, we use the 5th percentile of the sigma as an estimate for the smallest sigma.

5

Tests of the sharpness measures

We tested the three proposed sharpness measures using four test images. These images are given in figure 4 (a)-(d). The time needed to determine the sharpness of one image was approximately 80 minutes on a Pentium III (800 MHz). The images were manipulated in two ways, The first manipulation is a Gaussian blurring with σ 2 /2 = {0.5, 1.0, 1.5...3.0}. In the second manipulation we subtract from an image I, k times the Laplacian-of-Gaussian filtered version [12]

th

5 percentile σ

80th percentile # pixels with sigma < 1

trui erika cameron lenna

12 10

2500

trui erika cameron lenna

2000

trui erika cameron lenna → sharpening

2.4 2.2 2

1500

smoothing ←

8

1.8 1.6

1000

6 → sharpening

4

1.4

smoothing ←

2

−3

−2 −1 −(σ)2/2

0

1

2 k

3

0

smoothing ←

1.2

500

−3

−2 −1 −(σ)2/2

1 0

→ sharpening 1 2 k

3

0.8

−3

−2 −1 2 −(σ) /2

0

1

2 k

3

Fig. 5. Different sharpness measures. In (a) and (b) the values are inversely correlated with the sharpness of the image, in (c) the values are correlated with the sharpness of the image. The method of unsharp masking allows estimated sigmas smaller than 0.9 without problems of aliasing

(unsharp masking) with σ = 1.0. The k’s used are {0.5, 1.0, 1.5...3.0}. In the results we plot these two manipulations in one plot by putting the original in the middle (denoted by the dashed line), with the blurring to the left and the sharpening to the right. The spacing to the left is −σ 2 /2, the spacing to the right is k. The results for the three sharpness measures are given in figure 5. We expect the 80th percentile and the number of pixels with a sigma smaller than one to decrease for a larger blurring, and the smallest sigma to increase for a larger blurring. It can be seen that for all three measures this is indeed the case. This means that all three measures can be used as a measure for the difference in sharpness between images. We performed a small pairwise-comparison experiment to determine the sharpness of the original images with respect to each other. Six subjects compared all pairs, six subjects only the pairs without lenna. It was found that cameron is the sharpest image, closely followed by lenna. The most unsharp image is trui. If the measures are absolute sharpness measures, this order should also be found with the sharpness measures. This is not the case for the 80th percentile measure, because this measure depends upon the overall intensity of the image. For the other two measures, trui is indeed the most unsharp image. The ordering of the other three images is not the same as as was found with the pairwise comparison experiment, but these values are really close to each other, so it can be due to noise. We can conclude that the 5th percentile and the number of pixels with a sigma smaller than 1 are promising measures.

6

Results of perceptual experiments

In earlier experiments [2] we asked subjects to order images that were sharpened with unsharp masking and smoothed with Gaussians or with anisotropic diffusion [9][1]. The images that are used in the experiment are standard ISO images (CD-ROM 12640:1997). These images are given in figure 4 (e) and (f). The correlation between the perceived sharpness on the one hand and the proposed

sharpness measures on the other, is measured with the Spearman rank-order correlation coefficient rs [10]. The null hypothesis that is tested is that there is no association between the two rankings. Table 1. The Spearman rank-order coefficients per subject. For the values printed in italic, the null hypothesis cannot be rejected; that is, in these cases the perceived sharpness is independent of the measures. CV stands for critical value. Note that subject 4 did not participate in the bicycle experiment.

Range 1 2 3 4

1 0.98 1.00 1.00 1.00

Subject 2 3 0.98 0.88 0.98 0.69 0.98 0.76 0.96 0.93

4 0.90 0.17 0.98 1.00

N 8 8 8 7

CV 0.74 0.74 0.74 0.79

(a) portrait 5th percentile

Range 1 2 3 4

1 0.98 0.93 1.00 1.00

Subject 2 3 0.98 0.88 0.88 0.76 0.98 0.76 0.96 0.92

4 0.90 0.04 0.98 1.00

N 8 8 8 7

CV 0.74 0.74 0.74 0.79

(c) portrait number of points

Range 1 2 3 4

Subject 1 2 3 0.95 0.88 0.83 0.93 0.90 0.83 0.52 0.90 0.88 0.46 0.79 0.29

(b) bicycle centile

Range 1 2 3 4

5th

Subject 1 2 3 0.95 0.88 0.83 0.93 0.90 0.83 0.47 0.85 0.79 0.71 0.79 0.04

N 8 8 8 7

CV 0.74 0.74 0.74 0.79

per-

N 8 8 8 7

CV 0.74 0.74 0.74 0.79

(d) bicycle number of points

Each subject ordered 4 ranges per image, varying from low smoothing (range 1) to high smoothing (range 4). The rank-order coefficients for the different subjects and ranges are given in table 1. It can be seen that with both measures the null hypothesis can be rejected for most subjects and ranges. It can also be seen that this measure does not work well for the bicycle range with high smoothing. For the other images, 92% can be rejected. The conclusion is that both measures can be used as a measure for sharpness.

7

Conclusions and discussion

We found that we can measure the sharpness of simple line and edge images. We first located these lines and edges in the image. Then we determined the sharpness of these lines and edges by fitting a Gaussian line or edge profile to the Gaussian derivative signature. We defined three measures, from which the 5th percentile of the sigma and the number of pixels with a sigma smaller than one are the most promising. We found that these measures correlate to perceptual sharpness. The 80th percentile of the filter responses correlates to relative sharpness, but not to absolute sharpness.

This was expected, because the measure depends upon the overall intensity of the image. In the future, we will study the distribution of the sigmas and amplitudes to see if we can define other measures that correlate to the sharpness of images.

Acknowledgments This research is partly supported by Senter, Agency of the Ministry of Economic Affairs of the Netherlands.

References 1. F. Catt´e, P.-L. Lions, J.-M. Morel, and T. Coll. Image selective smoothing and edge detection by nonlinear diffusion. SIAM Journal on Numerical Analysis, 29(1):182– 193, 1992. 2. J. Dijk, D. de Ridder, P. W. Verbeek, J. Walraven, I. T. Young, and L. J. van Vliet. A new measure for the effect of sharpening and smoothing filters on images. In Proc. 11th Scandinavian Conference on Image Analysis, pages 213–220. SCIA ’99, 1999. 3. W. T. Freeman and E. H. Adelson. The design and use of steerable filters. IEEE transactions on Pattern Analysis and Machine Intelligence, 13(9):891–906, September 1991. 4. M. van Ginkel. Image Analysis using Orientation Space based on Steerable Filters. PhD thesis, Delft University of Technology, Delft, The Netherlands, 2002. 5. V. Kayargadde. Feature extraction for image quality prediction. PhD thesis, Technische universiteit Eindhoven, 1995. 6. H. Knutsson and G.H. Granlund. Texture analysis using two-dimensional quadrature filters. In IEEE Computer Society Workshop on Computer Architecture for Pattern Analysis and Image Database Management, Pasadena, October 1983. 7. T. Lindeberg. On scale selection for differential operators. In Proc. 8th Scandinavian Conference on Image Analysis, 1993. 8. S. Mallat and W. L. Hwang. Singularity detection and processing with wavelets. IEEE transactions on information theory, 38(2):617–643, March 1992. 9. P. Perona, T. Shiota, and J. Malik. Anisotropic diffusion. In B.M. ter Haar Romeny, editor, Geometry-driven diffusion in computer vision, pages 73–92, Dordrecht, 1994. Kluwer Academic Publishers. 10. S. Siegel and N. J. Castellan. Non parametric statistics for the behavioral sciences. McGraw-Hill, New York, NY, 1988. 11. R. J. van Asselt. Line and edge characterization using mallat wavelets. Master’s thesis, Delft University of Technology, December 1997. 12. I. T. Young, J. J. Gerbrands, and L. J. Van Vliet. Image processing fundamentals. In V.K. Madisetti and D.B. Williams, editors, The Digital Signal Processing handbook, chapter 51, pages 1 – 81. IEEE Press and CRC Press, 1998.