Encoding Visual Information Using Anisotropic Transformations ц

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,

Encoding Visual Information Using Anisotropic Transformations Giuseppe Boccignone, Member, IEEE, Mario Ferraro, and Terry Caelli AbstractÐThe evolution of information in images undergoing fine-to-coarse anisotropic transformations is analyzed by using an approach based on the theory of irreversible transformations. In particular, we show that, when an anisotropic diffusion model is used, local variation of entropy production over space and scale provides the basis for a general method to extract relevant image features.

VOL. 23, NO. 1,

H…f j f  † ˆ

1

INTRODUCTION AND PRELIMINARIES

IN a previous paper [3], we proposed a method based on the theory of thermodynamics of irreversible transformations to compute variations of local entropy in images undergoing fine-to-coarse transformations. We have applied this method using an isotropic diffusion model to produce image segmentation where region boundaries were depicted by different rates of local entropy variation [3]. Here, we combine the method with an anisotropic diffusion model. It will be shown that the measure of local entropy variation, which gauges image structure dissipation, contains a term accounting for the bias or a priori information introduced by anisotropy. As a result, more precise and more stable feature encoding is obtained than by using isotropic diffusion [3]. Let be a subset of R2 , …x; y† denote a point in , and the scalar field I : …x; y†  t ! I…x; y; t† represent a gray-level image. The nonnegative parameter t defines the scale of resolution at which the image is observed; small values of t correspond to fine scales, while large values correspond to coarse scales. A scale transformation is assumed here to be a noninvertible or irreversible transformation of I, which preserves the total intensity, i.e., Z Z I…x; y; t†dxdy ˆ const:

Let I  denote the fixed point of the transformation, that is, I  …
; 0† ˆ I  …
; t† for all t and let I  be stable, that is, limt!1 I…
; t† ˆ I  …
†. Let f be the normalized version of I, that R R  1 , then f…x; y† can be is, f…x; y; t† ˆ I…x; y; t†

I…x; y; t†dxdy interpreted as an estimate of the probability that a photon, from the 3D scene, impinge on the image domain at the point …x; y†. In [3], the conditional entropy H…f j f  † was introduced as the negative of the Kullback-Leibler distance. That is, . G. Boccignone is with the Dipartimento di Ingegneria dell' Informazione e Ingegneria Elettrica, UniversitaÁ di Salerno, via Ponte Don Melillo, 1 84084 Fisciano, Salerno Italy. E-mail [email protected]. . M. Ferraro is with the Dipartimento di Fisica Sperimentale, UniversitaÁ di Torino, via Pietro Giuria, 1 10125 Torino, Italy. E-mail: [email protected]. . T. Caelli is with the Department of Computer Science, General Services Building, The University of Alberta, Edmonton, Alberta, Canada T6G 2H1. E-mail: [email protected]. Manuscript received 12 July 1999; revised 25 July 2000; accepted 5 Sept. 2000. Recommended for acceptance by Y.-F. Wang. For information on obtaining reprints of this article, please send e-mail to: [email protected], and reference IEEECS Log Number 110216. 0162-8828/01/$10.00 ß 2001 IEEE

1

Z Z

f…x; y; t† ln

f…x; y; t† dxdy; f  …x; y†

and Z Z @H…f j f  † @H…f† @ ˆ ‡ f…x; y; t† ln f  …x; y†dxdy; …1† @t @t @t

RR where H…f† ˆ

f…x; y; t† ln f…x; y; t†dxdy is the BoltzmannGibbs entropy measure. In this case, the fine-to-coarse transformation can be modeled by a diffusion equation [4], namely @I…x; y; t†=@t ˆ r2 I…x; y; t†, and the evolution of H…f j f  † is RR determined solely by P ˆ @H…f†=@t ˆ f…x; y; t†…x; y; t†dxdy,

where

Index TermsÐScale space, anisotropic diffusion, entropy production, feature encoding.

æ

JANUARY 2001

…x; y; t† ˆ

rf…x; y; t†
rf…x; y; t† f…x; y; t†2

…2†

is the density of entropy production in the thermodynamical sense. Since P  0 is nonnegative, H…f† is an increasing function of t, and lim  ˆ lim P ˆ 0. t!1

2

t!1

INFORMATION FROM ANISOTROPIC TRANSFORMATIONS

Anisotropic fine-to-coarse transformations can be modeled by generalizing the isotropic diffusion equation as @ f…x; y; t† ˆ div……x; y†rf…x; y; t††; @t

…3†

where the function  is used to force convergence of the diffusion process toward some desired image representation. For instance, Perona and Malik [7] assume  to be a nonnegative, monotonically decreasing function of the magnitude of local image gradient. In this way, the diffusion mainly takes place in areas where intensity is constant or varies slowly, whereas it does not affect areas with large intensity transitions. As a result, small variations in f, such as noise, can be smoothed while edges are retained. Unfortunately, this diffusion process does not incorporate a convergence criterion. Nordstrom [6] conjectured that, at the limit t ! 1, the image would converge to a piecewise constant image and developed an algorithm, called ªbiased anisotropic diffusion,º closely related to Perona and Malik's, which converges to a steady state solution without prespecifying the number of iterations. In discrete simulations, the diffusion process may diverge depending on the different schemes and grid sizes [5], [8], [10]; a method was presented by Black et al. [1], which is well-posed and robust, in the sense that, given a piecewise constant image, diffusion will leave the image unchanged. In this note, we assume Nordstrom's conjecture in order to avoid the use of specific well-behaved diffusion, such as the one detailed in [1], which may not apply to the many different types of images of interest here. That is, we assume that, under a suitable choice of  and, for t ! 1, the stationary point f  of the transformation corresponds to a piecewise constant function with sharp boundaries between regions of constant intensity. In practice, such ideal fixed points can be obtained either by using a well-behaved diffusion such as the ones proposed in [1], [11], or by defining f  …
† ˆ f…
; t †, with t  0 (as it is actually done in the simulations).  Consider now the general R R form of @H…f j f †=@t (see (1)); by @ f…x; y;t†dxdy ˆ 0, after some algebra taking into account that @t

one obtains, by making use of (3):

2

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,

VOL. 23,

NO. 1,

JANUARY 2001

Fig. 1. Left image: The original ªAerial1º image. Right image: The fixed point of ªAerial1º obtained through 2,000 iterations of anisotropic diffusion. The graph below the images plots @H…f j f  †=@t, P, and S as a function of scale, represented by iterations of the anisotropic diffusion process. Units are nats=t.

@ H…f j f  † ˆ @t Z Z ‰ln f  …x; y†

Let ln f…x; y; t†Šdiv……x; y†rf…x; y;t††dxdy:

…4†

Applying the Gauss divergence theorem and making use of Neumann's boundary conditions, we eventually obtain the formula for the evolution of the conditional entropy H…f j f  † in the anisotropic case as: Z Z @ rf…x; y; t†
rf…x; y; t† H…f j f  † ˆ f…x; y; t†…x; y† dxdy @t f…x; y; t†2

Z Z rf…x; y; t†
rf  …x; y† dxdy; f…x; y; t†…x; y† f…x; y; t†f  …x; y; t†

…5† thus, during anisotropic diffusion, the evolution of the conditional entropy H…f j f  † is not determined solely by the entropy production P ˆ @H…f†=@t.

Z Z Pˆ



f…x; y; t†0an …x; y; t†dxdy

and Z Z Sˆ



f…x; y; t†00an …x; y; t†dxdy:

A local measure an of the variation rate of conditional entropy H…f j f  † can be defined by setting @ H…f j f  † ˆ @t Z Z f…x; y; t†an …x; y; t†dxdy; P Sˆ

…6†



and an can be written as the sum of two terms, denoted by 0an and 00 an , respectively,

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,

VOL. 23, NO. 1,

JANUARY 2001

3

Fig. 2. Results of region identification by activity for image ªAerial1º shown in Fig. 1. The left column displays results obtained experimenting with the isotropic process [3]: The top image presents h-type regions, whereas the bottom image presents m-type regions. The two images in the right column are the corresponding results obtained exploiting the anistropic process. In both experiments, the activity has been computed by integrating ober 100 iterations; the anisotrophic process used the fixed point achieved after 2,000 iterations. 00

an …x; y; t† ˆ 0an …x; y; t† an …x; y; t† rf…x; y; t†
rf…x; y; t† rf…x; y; t†
rf  …x; y† : ˆ …x; y† …x; y† 2 f…x; y; t†f  …x; y; t† f…x; y; t† …7† It is worth noting that an , as defined in (7), is not necessarily

evidence that, if  is a nonnegative decreasing function of j rf j , then 0 < S < P, so that P

S > 0.

We have used different kinds of images (animals, flowers, landscapes, objects, submarine, aerial) of various sizes and

always positive; however, numerical calculations of @H…f j f  †=@t

resolutions to evaluate the model. An example is provided by

and P, S, performed on a data set of 120 natural images provide

the image ªAerial1º presented in Fig. 1 (left image).

4

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,

VOL. 23,

NO. 1,

JANUARY 2001

The term 00

an ˆ …x; y†

rf…x; y; t†
rf  …x; y† f…x; y; t†f  …x; y; t†

is calculated by using an approximated fixed point image fe …x; y† ' f  …x; y† obtained by letting the anisotropic diffusion run for large t. As previously discussed, for simulation purposes, large values of t provide a suitable fixed point approximation. The right image in Fig. 1 shows the fixed point for the ªAerial1º image obtained for t ˆ 2000 iterations and using a ªbalancedº backward sharpening diffusion, as detailed in [11]. In the same figure, the graphs show the plots of @H…f j f  †=@t and P; S, as functions of t. By definition, it is clear that an is a function of …x; y† and depends on the local characteristics of the image. This property holds also for the entropy production  (see (2)), and it has been the basis for a method of region identification [3], which exploits the measurement of the activity of a across scales, defined as Z 1 rf…x; y; t†
rf…x; y; t† f…x; y; t† dt: a …x; y† ˆ f…x; y; t†2 0 Consider a generalization of the activity concept to the case of anisotropic diffusion. Since the anisotropic process performs information selection (see (7)), the region identification algorithm is, to some extent, simplified while increasing precision in identifying different information content structures. Define the R1 activity of an as aan …x; y† ˆ 0 f…x; y; t†an …x; y; t†dt; and likewise define Z 1 0 a0an ˆ f…x; y; t†an …x; y; t†dt; 0

and Z a00an ˆ

1 0

00

f…x; y; t†an …x; y; t†dt;

then, aan ˆ a0an a00an . First note that, ideally, by definition of 0an , the activity a0an is relatively large at points where different kinds of structures are present: textures, edges, etc. On the other hand, a00an is approximately different from zero only where j rf  j6ˆ 0 and this happens only along strong edges. Thus, the total activity aan is different from zero only along weak edges. In other terms, aan and a00an naturally encode medium information content (m-type) and high information content (h-type) regions, respectively. Clearly, this is an ideal model. When dealing with real images, various sources of noise and degradation must be taken into account; however, the information provided by aan and a0an can be recovered by means of a statistical technique, as follows. Consider the probability distributions Pb…aan †, Pb…a00an † estimated by means of the histograms of aan and a00an , respectively. Note that, in contrast with the isotropic case, anisotropic diffusion allows for the definition of two probability distributions which contain contributions from high and medium activity, respectivelyÐmaking the separation between the activity types clearer. Here, an iterative thresholding [3] is not needed (in the sense that preliminary h-type / m-type identification is incorporated in the process itself) but, rather, a one-pass thresholding, applied to Pb…aan † and to Pb…a00an †: Results using the anisotropic process are shown, for the ªAerial1º image, in the right column of Fig. 2. They demonstrate that the use of anisotropic activity makes localization of different features in the image more precise; in particular, localization of edges is improved since anisotropic diffusion avoids edge blurring and displacement. Moreover, the identification of h-type and m-type

Fig. 3. Edge detection results obtained on the fixed point of image ªAerial1º shown in Fig. 1 using the procedure reported in [7].

regions is much more biased to detect proper edge and texture regions, respectively. Clearly, the procedure discussed above encompasses more general aspects than just edge detection. However, since by using anisotropic activity differently from isotropic activity proposed in [3], h-type regions should represent reliably edges, it may be of some interest to compare such results with those achieved by other methods. Fig. 3 provides an example where edge detection has been performed on the fixed point image of Fig.1 by adopting the same procedure used in [7]. By comparing Fig. 3 with the top right image of Fig. 2, it can be noted a slight improvement, as regards the saliency of edge regions encoded. A further example of results obtained with the procedure proposed here is presented in Fig. 4.

3

DISCUSSION AND CONCLUSION

In this note, we have considered the case of anisotropic diffusion for the extraction of features over space and scale. It has been shown that, in this case, the rate of change of information across scales does not depend solely on entropy production, but that, due to the characteristics of the process, the loss of information is, at least, partially prevented by a term that depends on the degree of parallelism between the gradient of the image at scale t and that of the image representing the fixed point of the anisotropic diffusion process. It has also been shown that different types of regions and features, can be derived from local measures of variation of conditional entropy and that this new model is an improvement over the isotropic case. The reason for this is that the anisotropic diffusion imposes constraints that limit the destruction of structures in the image, which represent strongly oriented features. Then the term S, which tends to prevent entropy growth, can be interpreted as accounting for a priori knowledge or information introduced by the anisotropic process as opposed to the isotropic process.

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,

VOL. 23, NO. 1,

JANUARY 2001

5

i divJ~i , where J~i is a flow and the index i labels the namely @I @t ˆ bands of the image ~ I ˆ fI1 ; . . . ; In g, and by considering flows as functions of the thermodynamical forces expressed by the phenomenological laws of irreversible processes. Preliminary results for color images have been reported in [2].

REFERENCES [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11]

Fig. 4. Results of region identification by means of anisotropic activity on the natural image ªOld Ladyº characterized for varying details, low resolution, and limitied dynamic range. Diffusion iterations (100) have been used and a fixed point of 500 iterations. The top image is the original ªOld Lady,º images in the middle and on the bottom show, respectively, h-type regions and m-type regions.

The idea of applying information theory to scale-space representations has been discussed in a recent and interesting paper by Sporring and Weickert [9], who also present a generalization to previous methods by studying the behavior of Renyi's entropy throughout linear and nonlinear scale-spaces. However, a basic limitation of such approach is that only global measures can be defined for an image. Although this can be useful when general characteristics (e.g., finding the fingerprint of a texture [9]) are of interest, it cannot encode structures localized in space and spatial scale. An issue that remains to be investigated is the extension to vector-valued images of the framework developed here and in [3]. In this case, the transformation from fine-to-coarse representations can be modeled by a generalization of the diffusion equation,

M.J. Black, G. Sapiro, D.H. Marimont, and D. Heeger, ªRobust Anisotropic Diffusion,º IEEE Trans. Image Processing, vol. 7, no. 3, pp. 421±432, 1998. G. Boccignone, M. Ferraro, and T. Caelli, ªEntropy Production in Colour Images,º Proc. 15th Int'l Conf. Pattern Recognition, vol. 1, pp. 202±205, 2000. M. Ferraro, G. Boccignone, and T. Caelli, ªOn the Representation of Image Structures via Scale Space Entropy Conditionsº IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 21, no. 11, pp. 1,199±1,203, Nov. 1999. T. Lindeberg, Scale-Space Theory in Computer Vision. Dordrecht, The Netherlands: Kluwer Academic, 1994. M. Nitzberg and T. Shiota, ªNonlinear Image Filtering with Edge and Corner Enhancement,º IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 14, pp. 826±833, 1992. N. Nordstrom, ªBiased Anisotropic Diffusion: A Unified Regularization and Diffusion Approach to Edge Detection,º Image Vision Computing , vol. 8, pp. 318±327, 1990. P. Perona and J. Malik, ªScale-Space and Edge Detection Using Anisotropic Diffusion,º IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 12, pp. 629±639, 1990. L.I. Rudin, S. Osher, and E. Fatemi, ªNonlinear Total Variation Based Noise Removal Algorithms,º Physica D, vol. 60, pp. 259±268, 1992. J. Sporring and J. Weickert, ªInformation Measures in Scale-Spaces,º IEEE Trans. Information Theory, vol. 45, no. 3, pp. 1,051±1,058, 1999. R.T. Whitaker and S.M. Pizer, ªA Multiscale Approach to Nonuniform Diffusion,º CVIGP: Image Understanding, vol. 57, no 1, pp. 99±110, 1993. Y.L. You, X. Wenyuan, A. Tannenbaum, and M. Kaveh, ªBehavioral Analysis of Anisotropic Diffusion in Image Processing,º IEEE Trans. Image Processing, vol. 5, no. 11, pp. 1,539±1,552, 1996.