Geometry and Color in Natural Images - CiteSeerX

Download PDF
6 downloads 0 Views 404KB Size Report
Comment
Mar 8, 2000 - graphic map. This explains why Matheron and ...... 8] J. L. Doob, Stochastic Processes, Wiley Classics Library, 1953. 9] L.C. Evans and R.F. ...
Geometry and Color in Natural Images Vicent Caselles  Bartomeu Coll y Jean-Michel Morel z March 8, 2000

Abstract

Most image analysis algorithms are de ned for the grey level channel, particularly when geometric information is looked for in the digital image. We propose an experimental procedure in order to decide whether this attitude is sound or not. We adopt the hypothesis that the essential geometric contents of an image is contained in its level lines. The set of all level lines, or topographic map, is a complete contrast invariant image description : it yields a line structure by far more complete than any edge description, since we can fully reconstruct the image from it, up to a local contrast change. We then design an algorithm constraining the color channels of a given image to have the same geometry (i.e. the same level lines) as the grey level. If the assumption that the essential geometrical information is contained in the grey level is sound, then this algorithm should not alter the colors of the image or its visual aspect. We display several experiments con rming this hypothesis. Conversely, we also show the e ect of imposing the color of an image to the topographic map of another one : it results, in a striking way, in the dominance of grey level and the fading of a color deprived of its geometry. We nally give a mathematical proof that the algorithmic procedure is intrinsic, i.e. does not depend asymptotically upon the quantization mesh used for the topographic map. We also prove its contrast invariance. Dep. of Technology, University Pompeu Fabra, La Rambla, 30-32, 08002 Barcelona, Spain, [email protected] y Dep. of Mathematics and Informatics, University of Illes Balears, 07071 Palma de Mallorca, Spain, [email protected] z Ecole Normale Sup erieure de Cachan, 61 Avenue du Pdt Wilson, 94235 Cachan Cedex France, [email protected] 

1

1 Introduction : color from di erent angles In this paper, we shall rst review brie y some of the main and attitudes adopted towards color in art and science (Section 1) We then focus on image analysis algorithms and de ne some of the needed terminology (Section 2), in particular the topographic map : we support therein the view that the geometrical information of a grey level image is fully contained in the set of its level lines. Section 3 is devoted to the description of an experimental procedure to check whether the color information contents can be considered as a mere non geometrical complement to the geometry given by the topographic map or not. In continuation, a description of several experiments on color digital images is performed. In Section 4, which is mainly mathematical, we check from several point of views the soundness of the proposed algorithm, in particular its independence of the quantization procedure, its consistency with the assumption that the grey level image has bounded variation and the contrast invariance of the proposed algorithm.

1.1 Painting, linguistics

The color-geometry debate in the theory of painting has never been closed, each school of painters making a manifesto of its preference. Delacroix claimed "L'ennemi de toute peinture est le gris !" [23]. To impressionnists, "la couleur est tout"(Monet), while the role of contours and geometry is prominent in the Renaissance painting, but also for the surrealistic school or for cubists : this last school is mostly concerned with the deconstruction of perspective and shape [18]. To Kandinsky, founder and theoretician of abstract painting, color and drawing are treated in a totally separate way, color being associated with emotions and spirituality, but the building up of a painting being essentially a question of drawing and the (abstract) shape content relying on drawing, that is on black strokes, [12]. In this discussion, we do not forget that, while an accurate de nition of color is given in quantum mechanics by photon wavelength, the human or animal perception of it is extremely blurry and variable. Red, green and blue color captors on the retina have a strong and variable overlap and give a very poor wavelength resolution, not at all comparable to our auditive frequency receptivity to sounds. Di erent civilisations have even quite di erent color systems. For instance, the linguist Louis Hjelmslev : [10] : "Derriere les paradigmes qui, dans les di erentes langues sont formes par les designations de couleurs, nous pouvons, par soustraction des di erences, degager un tel continuum amorphe : le spectre des couleurs dans lequel chaque langue etablit directement ses frontieres. Alors que cette zone de sens se forme dans l'ensemble a peu pres de la m^eme facon dans les principales langues de l'Europe, il n'est pas dicile de trouver ailleurs des formations di erentes. En gallois "vert" est en partie gwyrdd et en partie glas, "bleu" correspond a glas, "gris" est soit glas soit llwyd, brun correspond a llwyd ; ce qui veut dire que le 2

domaine du spectre recouvert par le mot francais vert est, en gallois, traverse par une ligne qui en rapporte une partie au domaine recouvert par le francais bleu, et que la frontiere que trace la langue francaise entre vert et bleu n'existe pas en gallois ; la frontiere qui separe bleu et gris lui fait egalement defaut, de m^eme que celle qui oppose en francais gris et brun ; en revanche, le domaine represente en francais par gris est, en gallois, coupe en deux, de telle facon que la moitie se rapporte a la zone du francais bleu et l'autre moitie a brun." To summarize, the semantic division of colors is simply di erent in French (or English) and Welsh and there is no easy translation, four colors in French being covered by three di erent ones in Welsh.

1.2 Perception theory

Perception theory does not support anymore the absoluteness of color information. In his monumental work on visual perception, Wolfgang Metzger [15], dedicates only one tenth of his treatise (2 chapters over 19) to the perception of colors. Those chapters are mainly concerned with the impossibility to de ne absolute color systems, the variability of the de nition of color under di erent lighting conditions, and the consequent visual illusions. See in particular the subsection : "Gibt es eine physikalisch festliegende "normale" Beleuchtung und eine "eigentliche" Farbe ?" (Is there any physically well founded "normal" illumination and a proper color ?) Noticeably, in this treatise, 100 percent of the experiments not directly concerned with color are made with black and white drawings and pictures. In fact, the gestaltists not only question the existence of "color information", but go as far as to deny any physical reality to any grey level scale : the grey levels are not measurable physical quantities in the same level as, say, temperature, pression or velocity. A main reason invoked is that most images are generated under no control or even knowledge of the illumination conditions or the physical re ectance of objects. This may also explain the failure of several interesting attempts to use shape from shading information in shape analysis [11]. The contribution of black and white photographs and movies has been to demonstrate that essential shape content of images can be encoded in a gray scale and this attitude seems to be corroborated by the image processing research. Indeed, and although we are not able to deliver faithful statistics, we can state that an overwhelming majority of image processing algorithms are being designed, tested and validated on grey level images. Satellite multispectral images (SPOT images for instance) attribute a double resolution to panchromatic (i.e. grey level images) and a simple one to color channels : somehow, color is assumed to only give a semantic information, (like e.g. presence of vegetation) and not a geometric one : this engineering decision meets Kandinsky's claim !

1.3 Image processing algorithms

Let us now consider the practice of image processing. When an algorithm has to be applied to color images, it is generally rst designed and tested 3

on grey level images and then applied independently to each channel. It is experimentally dicult to demonstrate the improvements due to a joint use of the three color channels instead of this independent processing. Antonin Chambolle [6] tried several strategies to generalize mean curvature algorithms to color images. His overall conclusion (private communication) was that no perceptible improvement was made by de ning a color gradient : di using each channel independently led to essentially equal results, from a perception viewpoint. A more recent, and equivalent, attempt to de ne a "color gradient" in order to perform anisotropic di usion in a more sophisticated way than just di using each channel independently is given by [21]. The authors do not provide a comparison with an algorithm making an independent di usion on each channel, however, so that their study does not contradict the above mentioned conclusion. Pietro Perona [19] performed experiments on color images where he applied the Perona-Malik anisotropic di usion [20]. In this paper, we propose a numerical experimental procedure to check that color information does not contribute to our geometric understanding of natural images. This statement has to be made precise. We have rst considered it a common sense consequence of the arbitrariness of lighting conditions and total inaccuracy of our color wavelength perception, which makes the de nition of color channels very context-dependent. This explains why the literature on color is mainly devoted to a "restoration" of universal color characteristics such as saturation, hue and luminance. Now, of these three characteristics, only luminance, de ned as a sum of color channels has a (relative) physical meaning. Indeed, luminance, or "grey level" is de ned as a photon count over a period of time (exposure time). Thus, we can relate a linear combination of R, G, B, channels to this photon count. Now, we mentionned that from the perception theory viewpoint, (see e.g. Wertheimer [25]), even this grey level information is subject to so much variability due to unknown illumination and re ectance condition that we cannot consider it as a physical information.

2 Mathematical morphology and the topographic map This explains why Matheron and Serra [22] developed a theory of image analysis, focused on grey level and where, for most operators of the so called " at morphology", contrast invariance is the rule. Flat morphological operators (e.g. erosions, dilations, openings, closings, connected operators, etc.) commute with contrast changes and therefore process independently the level sets of a grey level image. In the following, let us denote by u(x) the grey level of an image u at point x. In digital images, the only accessible information is a quantized and sampled version of u, u(i; j ), where (i; j ) is a set of discrete points (in general on a grid) and u(i; j ) belongs in fact to a discrete set of values, 0; 1; :::; 255 in 4

many cases. Since, by Shannon theory, we can assume that u(x) is recoverable at any point from the samples u(i; j ), we can in a rst approximation assume that the image u(x) is known, up to the quantization noise. Now, since the illumination and re ectance conditions are arbitrary, we could as well have observed an image g(u(x)) where g is any increasing contrast change. Thus, what we really know are in fact the level sets

X u = fx; u(x)  g; where we somehow forget about the actual value of . According to the Mathematical Morphology doctrine, the reliable information in the image is contained in the level sets, independently of their actual levels. Thus, we are led to consider that the geometric information, the shape information, is contained in those level sets. This is how we de ne the geometry of the image. In this paragraph, we are simply summarizing some arguments contained explicitly or implicitly in the Mathematical Morphology theory, which were further developed in [5]. We can further describe the level sets by their boundaries, @X u, which are, under suitable very general assumptions, Jordan curves. Jordan curves are continuous maps from the circle into the plane IR2 without crossing points. To take an instance which we will invoke further on, if we assume that the image u has bounded variation, then for almost all levels of u, X u is a set with bounded perimeter and its boundary a countable family of Jordan curves with nite length [1]. In the mentioned work, it is demonstrated that the level line structure whose existence was assumed in [4] indeed is mathematically consistent if the image belongs to the space BV of functions with bounded variation. It is also proved that the connected components of level sets of the image give a locally contrast invariant description. In the digital framework, the assumption that the level lines are Jordan curves is straightforward if we adopt the nave but useful view that u is constant on each pixel. Then level lines are concatenations of vertical and horizontal segments and this is how we shall visualize them in practice. As explained in [5], level lines have a series of structure properties which make them most suitable as building blocks for image analysis algorithms. We call the set of all level lines of an image topographic map. The topographic map is invariant under a wide class of local contrast changes ([5]), so that it is a useful tool for comparing images of the same object with di erent illuminations. This application was developed in [2] who proposed an algorithm to create from two images a third whose topographic map is, roughly speaking, an intersection of the two di erent input images. Level lines never meet, so that they build an inclusion tree. A data structure giving fast access to each one of them is therefore possible and was developed in [17], who proved that the inclusion trees of upper and lower level sets merge. We can conceive the topographic map as a tool giving a complete description of the global geometry for grey level images. A further application to shape recognition and registration is developed in [16], who proposes a topographic map based contrast invariant registration algorithm and [13], who propose a registration algorithm based 5

on the recognition of pieces of level lines. Such an algorithm is not only contrast invariant, but occlusion stable. Several morphological lters are easy to formalize and implement in the topographic map formalism. For instance, the Vincent-Serra connected operators ([24], [14]). Such lters, as well as local quantization algorithms are easily de ned [5]. Our overall assumption is rst that all of the reliable geometric information is contained in the topographic map and second that this level line structure is under many aspects (completeness, inclusion structure) more stable than, say, the edges or regions obtained by edge detection or segmentation. In particular, the advantage of level lines over edges is striking from the reconstruction viewpoint : we can reconstruct exactly the original image from its topographic map, while any edge structure implies a loss of information and many artifacts. It is therefore appealing to extend the topographic map structure to color images and this is the aim we shall try to attain here. Now, we will attain it in the most trivial way. We intend to show by an experimental and mathematically founded procedure that the geometric structure of a color image is essentially contained in the topographic map of its grey level. In other terms, we propose the topographic map of color image to be simply the topographic map of a linear combination of its three channels. If that is true, then we can claim that tasks like shape recognition, and in general, anything related to the image geometry, should be performed on the subjacent grey level image. As we shall see in the next section where we review from several points of view the attitude of scientists and artists toward color, this claim is nothing new and implicit in the way we usually proceed in image processing. Our wish, is, however, to make it explicit and get rid of this bad consciousness we feel by ever working and thinking and teaching with grey level images. Somebody in the assistance always asks : "and what about color images". We propose to prove that we do not need them for geometric analysis.

2.1 De nition of the geometry

We are led to de ne the geometry of a digital image as de ned by its topographic map. What about color ? Our aim here is to prove that we can consider the color information as a subsidiary information, which may well be added to the geometric information but does not yield much more to it and never contradicts it. Here is how we proceed : In general words, we shall de ne an experimental procedure to prove that Replacing the colors in an image by their conditional expectation with respect to the grey level does not alter the color image.

Of course, this statement needs some mathematical formalization before being translated into an algorithm. We prefer, for a sake of clarity, to start with a description of the algorithm (in the next section). Then, we shall display some experiments since, as far as color is concerned, visual inspection 6

here is the ultimate decision tool used to check whether a colored image has been altered or not. Section 4 is devoted to the complete mathematical formalization, i.e. a proof that the de ned procedure converges and in no way depends upon the choice of special quantization parameters. Although this was not our aim here, let it be mentionned that extensions of this work for application to the compression of multi-channel images are in course [7]. The idea is to compress the grey level channel by encoding its topographic map. Then, instead of encoding the color channels separately, an encoding of the conditional expectation with respect to the topographic map is computed. This de nition will become clear in the next two sections.

3 Algorithm and experiments

3.1 The algorithm: Morphological Filtering of Color Images The algorithm we present is based on the idea that the color component of the image does not give contradictory geometric information to the grey level and, in any case, is complementary. Following the ideas of previous works (see [4], [5]), we describe the geometric contents of the image by its topographic map, a contrast invariant complete representation of the image. We discuss in Section 4 an extension to color channels of this contrast invariance property. The algorithm we shall de ne now imposes to the chromatic components, saturation and hue, to have the same geometry, or the same topographic map, as the luminance component. We shall experimentally check that by doing so, we do not create new colors and the overall color aspect of the image does not change with this operation. In some sense, and although this is not our aim, the algorithm is an alternative to color anisotropic di usions algorithms already mentioned ([3], [6], [21]). To de ne the algorithm, we take a partition P = fa0 = a < a1 < : : : < aN = bg of the grey level range of the luminance component. In practice a0 = 0 and aN = 255 and this partition is de ned by a grey level quantization step. The resulting set of level lines is a restriction to the chosen levels of the topographic map. We then consider the connected components of the set complementary to the level lines. In the discrete case, these connected components are sets of nite perimeter given by a nite number of pixels, and constitute a partition of the image. In the continuous case, we must impose some restriction about the space of functions we take (see Section 4). The algorithm we propose is the following. Let U : ! IR3 , U = (U1 ; U2; U3), where the three channels U1 ; U2; U3 are the intensity of red, green and blue. (i) From these channels, we compute the L, S and H values of the color signal, 7

i.e., the luminance, saturation and hue, de ned by p + blue L(U )(x) =< U (x);  >= red + green 3

011 1 with  = p @ 1 A ; 3 1 p

;2;3 Ui (x) ; S (U )(x) = 1 ? 3 mini=1 L(x)

(1)

or the perceptually less correct but simpler,

S~(U )(x) = kU (x)? < U (x);  > k = k  U (x)k

011  H (U )(x) = angle   U (x);   @ 0 A : 

0

(ii) Compute the topographic map associated to L with a xed quantization step. In other terms, we compute all level lines with levels multiple of a xed factor, for instance 10. This quantization process yields the topographic representation of a partial, coherent view of the image structure (see [5] for more details about visualization of the topographic map). This computation obvious : we rst compute the level sets, as union of pixels, for levels  which are multiples of the entire quantization step. We then simply compute their boundary as concatenations of vertical and horizontal lines. (iii) In each connected component A of the topographic map of L, we take the average value of S; H . More precisely, let fx1; : : : ; xng be the pixels of A. We compute the value Pn v(x ) vA = i=1n i ;

where v 2 fS; H g and vA will be the new constant value in the connected component A, for the components S and H . In other terms, we transform the  H , where S;  H have a constant value, in fact the average channels S; H into S; value, inside each connected component of the topographic map of L, the luminance component. As a consequence we obtain a new representation of the image, piecewise constant on the connected components of the topographic map and we therefore constrain the color channels S and H to have the same topographic map as the grey level. (iv) Finally, in order to visualize the color image, we compute (U1; U2; U3 ), de ned respectively as the new red, green and blue channels of the image by  H ): performing the inverse color coordinates change on (L; S; Remark 1 Note that in order to perform a visualization, each channel of the (U1 ; U2; U3)-space must have a range of values in a xed interval [a; b]. In 8

practice, in the discrete case, a = 0 and b = 255. After applying the preceding algorithm, this range can be altered and we can't recover the same range of values for the nal components. We therefore threshold these components so that their nal range be [0; 255]. Remark 2 A slight change of this algorithm can be obtained if we take, instead of the average on the chromatic components of the image in each connected components, the average of these components weighted by the modulus of the gradient of the luminance component (see the remark after Theorem 3). This means that we replace the step (iii) of the algorithm given above by (iii)'. That is, (iii)' In each connected component A of the topographicPmap of L let fx ; : : : ; xn g n v(x )jrL(x )j 1 A i =1 be the pixels of this region. Compute the value v = Pn jri L(xi)j i , where i=1 v 2 fS; H g and vA will be the new constant value in the connected component A, for the components S and H . We shall explain in Section 4 the measure geometric meaning of this variant.

3.2 Experiments and discussion

In Experiment 1, Image 1.1 is the original image, which we shall call the "tree" image. In the L-component of the (L; S; H ) color space, we take the topographic map of the level lines for all levels which are multiples of p = 5. This results in Image 1.2, where large enough regions are to be noticed, on which the grey level is therefore constant. We then apply the algorithm, by taking the average of S; H components on the connected components of the topographic map, i.e. the at regions of the grey level image quantized with a mesh equal to 5. Equivalently, this results in averaging the color of "tree" on each white region of Image I.2. We then obtain Image 1.3. In Image 1.4, we display, above, a detail of the original image "tree" and below the same detail after the color averaging process has been applied. This is the only part of the image where the algorithm brings some geometric alteration. Indeed, on this part, the grey level contrast between the sea's blue and the tree's light brown vanishes and a tiny part of the tree becomes blue. This detail is hardly perceptible in the full size image because the color coherence is preserved : no new color is created and, as a surprising matter of fact, we never observed the creation of new colors by the averaging process. In Experiment 2, Image 2.1 is the original image, which we call "peppers". Image 2.2 is the topographic map of "peppers" for levels which are multiples of 10. By a way of practical rule, quantizations of an image by a mesh of 5 are seldom visually noticeable ; in that case, we can go up to a mesh of 10 without visible alteration. Image 2.3 displays the outcome of averaging the colors on the at zones of Image 2, which is equivalent to averaging the colors on the white regions of Image 2.2. If color does not bring any relevant geometrical information complementary to the grey level, then it is to be expected that Image 2.3 will not have lost geometric details with respect to Image 2.1. This 9

is the case and, in all natural images where we applied the procedure, the outcome was the same : the conditional expectation of color with respect to grey level yields, perceptually, the same image as the original. We are aware that one can numerically create color images where the di erent colors have exactly the same grey level, so that the grey level image has no level lines at all ! If we applied the above procedure to such images, we would of course see a strong di erence between the processed one, which would become uniform, and the original. Now, the generic situation for natural images is that "color contrast always implies (some) grey level contrast". This empiric law is easily checked on natural images. We have seen in Experiment 1 the only case where we noticed that it was slightly violated, two di erent colors happening to have (locally) grey levels which di ered by less than 5 grey levels. In Image 2.4, we explore further the dominance of geometry on color by giving to "tree" the colors of "peppers". We obtained that image by averaging the colors of peppers on the white regions of "tree" and then reconstructing an image whose grey level was that of "tree" and the colors those of "peppers". In Experiment 3, we further explore the striking results of mixing the color of an image with the grey level of another one. Our conclusion will be in all cases that, like in Image 2.4, the dominance of grey level above color, as far as geometric information is concerned, is close to absolute. In Image 3.1, we do the converse procedure to the one in Image 2.4 : we average the colors of tree conditionally to the topographic map of peppers. The amazing result is a new pepper image, where no geometric content from "tree" is anymore visible. Of course, those two experiments are not totally fair, since we force the color of the second image to have the topographic map of the rst one. Thus, in Image 3.2, we simply constructed an image having the grey level of "peppers" and the colors of "tree". Notice again the dominance of "peppers" and the fading of the shapes of "tree". In Image 3.3 we display an original "baboon" image and in Image 3.4 the result of imposing the colors of "tree" to "baboon". Again, we mainly see "baboon". In all experiments, we used the rst version of the mean value algorithm and not the one described in Remark 2. Our experimental records show no visible di erence between both algorithms.

3.3 Conclusions

Our conclusions are contained in the text of the experiments. From the image processing viewpoint, we would like to mention some possible ways of applications. Although the presented algorithm has no more application than a visual inspection and a check of the independence of geometry and color, we can deduce from this inquiry some possible new ways to process color images. First of all, we can consider that the compression of multichannels images should be led back to the compression of the topographic map given by the panchromatic (grey level) image. This kind of attempt is in course [7]. Also, 10

one may ask whether color images, when given at a resolution smaller than the panchromatic image (this is e.g. the case for SPOT images) should not be deconvolved and led back to the grey level resolution. This seems possible, if this deconvolution is made under the (strong) constraint that the topographic map of each color coincides with the grey level topographic map.

Image 1.1

Image 1.2

Image 1.3 Image 1.4 Experiment 1. Image 1.1 is the original image. Image 1.2 is the topographic map of the L-component of the (L,S,H) color space for all level lines which are multiples of 5. The application of the algorithm by taking the average of S,H components on the connected components of Image 1.2 gives us Image 1.3. Finally, in Image 1.4 we display, above, a detail of the original image "tree" and below the same detail after the algorithm has been applied.

11

Image 2.1

Image 2.2

Image 2.3 Image 2.4 Experiment 2. Image 2.1 is the original image. Image 2.2 is the topographic map of of the L-component of the (L,S,H) color space for all level lines which are multiples of 10. The application of the algorithm to Image 2.1 over the white regions of Image 2.2 give us Image 2.3. Finally, we obtain Image 2.4 by averaging the colors of Image 2.1 on the white regions of Image 1.1.

12

Image 3.1

Image 3.2

Image 3.3 Image 3.4 Experiment 3. In Image 3.1, we average the colors of tree conditionally to the topographic map of Image 2.1. Image 3.2 shows the grey level of Image 2.1 and the colors of Image 1.1. Image 3.3 is the original image and Image 3.4 is the result of imposing the colors of Image 1.1 to Image 3.3.

4 Formalization of the Algorithm

Let (Y; B; ) be a measure space and F  B be a family of measurable subsets of Y . A connected component analysis of (Y; F ) is a map which assigns to each set X 2 F a family of subsets Cn(X ) 2 B of X such that i) (Cn(X )) > 0, (Cn(X ) \ Cm (X )) = 0 for all n; m 2 IN , n 6= m and

([1 n=1 Cn (X )) = meas(X ); ii) If X  X 0  E , then each Cn(X ) is contained in some Cm(X 0), n; m 2 IN . 13

Notice that this de nition asks more than the usual de nition [22], since we request that sets of F be essentially decomposable into connected components with positive measure. If, e.g., F is the set of open sets of IR2 , then the usual de nition of connectedness applies, i.e. satis es requirements i) and ii). Let (Y; B; ) and u : Y ! [a; b] be a measurable function. Let F (u) be a family of sets contained in the -algebra generated by the level sets fu?1 ([ ; )) : < g. Assume that a connected component analysis is given on ( ; F (u)). In other words, F (u) is a family of subsets where the connected components can be computed and satisfy i) and ii). Let P = fa0 = a < a1 < : : : < aN = bg be a partition of [a; b] such that [ai  u < ai+1 ] 2 F (u), i = 0; : : : ; N ? 1. Such partitions will be called admissible. Let us denote by CC (P ) the set of connected components of the sections [ai  u < ai+1 ], i = 0; 1; : : : ; N ? 1. Let v 2 L1(Y; B; ). For each connected component A 2 CC (P ) we de ne the average value of v on A by Z 1 vA = (A) v(x)d: A Then we de ne the function X vA A: E (vju; P ) = A2CC (P )

This function is nothing but the conditional expectation of v 2 L1(Y; B; ) with respect to the -algebra AP of subsets of Y generated by CC (P ). In the next proposition we summarize some basic properties of conditional expectations, [26], Ch. 9. Proposition 1 Let (X; B; ) be a measure space and let A be a sub--algebra of B. Let L1 (X; B; ) be the space of measurable (with respect to B) functions which are Lebesgue integrable with respect to . Let L1 (A) be the subspace of L1(X; B; ) of functions which are measurable with respect to A. Then E (:jA) is a bounded linear operator from L1 (X; B; ) onto L1 (A). i) If v 2 L1( ), 0  v  M , then 0  E (vjA)  M . We have E (1jA) = 1:

ii) If v 2 L1(X; B; ), z 2 L1(A) then E (zvjA) = zE (vjA): In particular, if v is measurable with respect to A then E (vjA) = v. iii) If A0 is a sub--algebra of A then E (E (vjA)jA0) = E (vjA0): In particular,

E (E (vjA)jA) = E (vjA): 14

(2)

(3)

Lemma 1 Let P = fa0 = a < a1 < ::: < aN = bg be an admissible partition of [a; b]. Let P 0 = fb0 = a < b1 < : : : < bM = bg be a re nement of P , i.e.,

a partition of [a; b] such that each ai coincides with one of the bj , and such that each section [bj  u < bj +1 ] 2 F (u), j = 0; : : : ; M ? 1. Then AP is a sub--algebra of AP . 0

Proof. Let A 2 CC (P ). Then for some i 2 f0; : : : ; N ? 1g, A is a connected component of [ai  u < ai+1 ]. Let j; k be such that ai = bj , ai+1 = bj+k . According to our Axiom ii), each connected component of [bl  u < bl+1 ] is contained in a connected component of [ai  u < ai+1 ], l 2 fj; : : : ; j + k ? 1g. Then, by Axiom i), each connected component of [bl  u < bl+1 ], l 2 fj; : : : ; j + k ? 1g, is either contained in A or disjoint to A (modulo a null set). Let (An)n be the set of connected components of [bl  u < bl+1 ], l 2 fj; : : : ; j + k ? 1g contained in A. We have that A = [1 (mod a null set): n=1 An Indeed, if this was not true, then we would nd by Axiom ii) another connected component of [bl  u < bl+1 ], l 2 fj; : : : ; j + k ? 1g, contained in A and we would obtain a contradiction. Let L1(Y; AP ; ) be the space of functions which are measurable with respect to the -algebra AP and -integrable. Proposition 1 can be translated as a statement about the operator E (:ju; P ) = E (:jAP ).

Proposition 2 Let P be an admissible partition of [a; b]. Then E (:ju; P ) is a bounded linear operator from L1 (Y; B; ) onto L1 (Y; AP ; ) satisfying properties i) ? ii) of Proposition 1. If P 0 is admissible and is a re nement of P then iii0) E (E (vju; P 0)ju; P ) = E (vju; P ): (4) Proof. We have shown that AP is a sub--algebra of AP . Let A 2 CC (P ) and An 2 CC (P 0) be such that A = [1 n=1 An (mod a null set). Then 0

Z Z 1 X vA = (1A) v(x)dx = (1A) v(x)dx A An n=1 1 X (An ) = vAn ; n=1 (A)

which is equivalent to (4). For any partition P = fa0 = a < a1 < : : : < aN = bg of [a; b] we de ne

kPk :=

sup

i2f0;1;:::;N ?1g

ai+1 ? ai :

Let Pn be a sequence of partitions of [a; b] such that 15

a) Pn are admissible, i.e., the sections of u associated to levels of Pn , are in F (u). b) Pn+1 is a re nement of Pn , n = 1; 2; : : :. c) kPn k ! 0 as n ! 1. Let v 2 L1(Y; B; ). We de ne vn = E (vju; Pn). Note that the -algebras APn form a ltration, i.e., an increasing sequence of -algebras contained in B and, by Proposition 2, we have vn 2 L1 (Y; APn ; ) and

vn = E (vn+1ju; Pn ): Thus, vn is a martingale relative to (fAPn gn; ) ([8], Ch. VII, Sect. 8). According to the martingale convergence theorem we have

Theorem 1 Let A1 be the -algebra generated by the sequence of -algebras APn , n  1, i.e., the smallest -algebra containing all of them. If v 2 Lp (Y; B; ), p  1, then vn converges in L1(Y; A1 ; ) and a.e. to a function v1 = E (vjA1 ), which may be considered as the projection of v into the space Lp (Y; A1 ; ). If v 2 L1 (A1 ), then v1 = v. In particular, u1 = u. Proof. Bounded martingales in Lp converge in Lp and a.e. if p > 1 and a.e. if p = 1 ([26], [8], Ch. VII, p. 319). Martingales generated as conditional expectations of a function v 2 L1( ) with respect to a ltration are equiintegrable and, thus, converge in L1 ([26], [8], Ch. VII, p. 319). Now, by Levy's Upward Theorem v1 = E (vjA1) ([26], Ch. 14, [8], Ch. VII, p. 331). The nal statement is a consequence of the properties of conditional expectations as described in Proposition 2.

4.1 The BV model Let be an open subset of IRN . A function u 2 L1( ) whose partial derivatives in the sense of distributions are measures with nite total variation in

is called a function of bounded variation. The class of such functions will be denoted by BV ( ). Thus u 2 BV ( ) if and only if there are Radon measures 1 ; : : : ; N de ned in with nite total mass in and

Z

Z



uDi 'dx = ? 'di

(5)

for all ' 2 C01( ), i = 1; : : : ; N . Thus the gradient of u is a vector valued measure whose nite total variation in an open set 0  is given by

k Du k ( 0) = supf

Z

u div ' dx : ' 2 C01( 0; IRN ); j'(x)j  1 for x 2 0g: (6) This de nes a positive measure called the variation measure of u. For further information concerning functions of bounded variation we refer to [9] and [27].

0

16

For a Lebesgue measurable subset E  IRN and a point x 2 RN , the following notation will be used: r)j D(x; E ) = lim sup jEjB\ (Bx;(x; (7) r ) j r!0 and jE \ B (x; r)j : D(x; E ) = lim inf (8) r!0 jB (x; r)j

D(x; E ) and D(x; E ) will be called the upper and lower densities of x in E . If the upper and lower densities are equal, then their common value will be called the density of x in E and will be denoted by D(x; E ). The measure theoretic boundary of E is de ned by @ M E = fx 2 IRN : D(x; E ) > 0; D(x; IRN n E ) > 0g:

(9)

Here and in what follows we shall denote by H the Hausdor measure of dimension 2 [0; N ]. In particular, H N ?1 denotes the (N ? 1)-dimensional Hausdor measure and H N , the N -dimensional Hausdor measure, coincides with the Lebesgue measure in IRN . Let E be a subset of IRN wih nite perimeter. This amounts to say that E 2 BV (IRN ), the space of functions of bounded variation. Then @ M E is recti able, i.e., @ M E  [1 i=0 Ei where each Ei is a (N ? 1)-dimensional 1 embedded C -submanifold of IRN and H N ?1 (E0) = 0 ([9], [27]). We also have that H N ?1(@  E ) =k DE k. We shall denote by P (E ) the perimeter of E , i. e., P (E ) = H N ?1(@  E ). As shown in [1], we can de ne the connected components of a set of nite perimeter E so that they are sets of nite perimeter and constitute a partition of E (mod H N ). Let us describe those results. De nition 1 Let E  IRN be a set with nite perimeter. We say that E is decomposable if there exists a partition (A; B ) of E such that P (E ) = P (A) + P (B ) and both, A and B , have strictly positive measure. We say that E is indecomposable if it is not decomposable. Theorem 2 ([1]) Let E be a set of nite perimeter in IRN . Then there exists a unique nite or countable family of pairwise disjoint indecomposable sets fYn gn2I such that i) meas(Yn ) > 0 for all n 2 I and meas([n2I Yn ) = meas(E ):

ii) Yn are sets of nite perimeter and X P (E ) = P (Yn ): n2I

17

(10)

iii) The sets Yn are maximal indecomposable, i.e. any indecomposable set F  E is contained, modulo a null set, in some set Yn . The sets Yn will be called the M -components of E . Moreover, we have If F is a set of nite perimeter contained in E , then each M -component of F is contained in a M -component of E . In other words, if Fper denotes the family of subsets of IRN with nite perimeter then the above statement gives a connected component analysis of (IRN ; Fper ). Let u 2 BV ( ) \ L1( ) and v 2 L1( ). Without loss of generality we may assume that u takes values in [a; b]. We know that for almost all levels  2 [a; b], the level set [u  ] is a set of nite perimeter. Let G (u) = fu?1 ([ ; )) : < g such that u?1 ([ ; )) is of nite perimeter. Then Theorem 2 describes a connected component analysis of G (u). Let Pn be a sequence of partitions of [a; b] such that a) Pn+1 is a re nement of Pn , n = 1; 2; : : :. b) For each n 2 IN , the sections of u associated to levels of Pn, are sets of nite perimeter. c) meas([u = ]) = 0 for each 2 Pn . d) kPn k ! 0 as n ! 1. Given a -algebra A and a measure  we denote by A the completion of A with respect to , i.e., A = fA [ B : A 2 A; B a -null setg. Lemma 2 Let Pn ; Qn be sequences of partitions of [a; b] satisfying a); b); c); d). Let A1 , resp. B1 , be the -algebra generated by the sequence of -algebras APn , resp. AQn . Then A1 = B1. Proof. It suces to prove that given n 2 IN and X an M -component of [c  u < d], [c; d) being an interval of Pn , then there is a set Z 2 B1 such that (X Z ) = 0. Let  > 0. Since ([u = c]) = ([u = d]) = 0, ?1 [b ; b ), [b ; b ) being let m 2 IN be large enough so that [c; d)  [pi=1 i i+1 i i+1 intervals of Qm , with c 2 [b1 ; b2), d 2 [bp?1 ; bp ), and ([b1  u < b2]) < , ([bp?1  u < bp ]) < . Now, since any M -component of a set [bi  u < bi+1 ], i = 2; :::; p ? 2, is either contained in X or disjoint to it, we have Z   X  Z  [ [b1  u < b2 ] [ [bp?1  u < bp ]; where Z  is the union of M -components of the sets [bi  u < bi+1 ], i = 2; :::; p ? 2, which are contained in X . Obviously, Z  2 B1 and (X n Z ) < 2. This implies our statement. The above Lemma proves that the -algebra generated by the sequence of -algebras APn , n  1, is independent of the sequence of partitions satisfying a); b); c); d). Let us denote by Au the -algebra given by the last Lemma. Observe that E (ujAu) = u. 18

Theorem 3 Let v 2 Lp( ), p  1. Let Pn be a sequence of partitions satisfying a); b); c); d). Let vn = E (vjPn ). Then vn is a martingale relative to (fAPn g)n : Moreover, vn converges in L1 (Au ) and a.e. to a function v1 = E (vjAu ), which may be considered as the projection of v into the space Lp (Au ). The limit is independent of the sequence of partitions satisfying a); b); c); d). If v 2 L1(Au ), then v1 = v. In particular, u1 = u. Remark. Let u : ! [a; b] is in BV ( ) and v 2 L1(jDujdx). Then we may also de ne X A v A E 0(vju; P ) = A2CC (P )

Z vjDujdx vA = ZA : jDujdx

where

A

Then results similar to Proposition 2 and Theorem 3 hold for E 0 as an operator from L1( ; jDujdx) into L1(Au ; jDujdx). Formally, we have

Z ZddZ Z vjDujdx v j Du j dx vjDujdx [cu t ] c A A A A v = Z = Z = ZddZ jDujdx jDujdx jDujdx A [cuA t] Z dZ vdH 1dt

c

@ M [uA t]

c

@ M [uA t]

= Z dZ

dH 1dt

for any connected component A of [c  u < d]. Now, if we let c; d !  then

Z

vA !

Zcc[u=]

vdH 1

cc[u=]

dH 1

where cc[u = ] denotes the connected component of [u = ] obtained from A when letting c; d ! . In the case under consideration, this amounts to interpret our algorithm as the computation of the average of v along the connected components of the level curves of u. Note that, if we take v = u the above formula gives vA ! ; when letting c; d ! .

19

4.2 Contrast invariance for color

We can state the contrast invariance axiom for color as Contrast Invariance of Operations on Color [6]: We say that an operator T is morphological if for any U and any ~ (11) h(U ) = h() U; where h~ is a continuous increasing real function we have

T (h(U )) = h(T (U )): We refer to such functions h as contrast changes for color vectors.

Proposition 3 Let U : ! IR3 be a color image such that L 2 BV ( ) \ ~ H 2 L1( ). Let AL be the -algebra associated to the luminance L1 ( ) and S; channel as described in the previous section. Let us de ne the lter 0 L(U ) 1 ~ H system): F (U ) = @ E (S~(U )jAL) A (in L; S; E (H (U )jAL) Then F is a morphological operator.

Proof. Let h be a contrast change for color vectors. Let V be any color

image. Then the Luminance, Saturation and Hue of the color vector h(V ) are given by L(h(V )) =< h(V );  >= h~ (L(V )) ~ S~(h(V )) = k  h(V )k = h(LL((VV))) S~(V );

011  H (h(V )) = angle   h(V );   @ 0 A = H (V ):

(12)



0 Then the Luminance, Saturation and Hue of the color vector F (h(U )) are 1 0 h~ (L(U )) (13) F (h(U )) = B A (in L; S;~ H system): @ E ( h~ (LL((UU))) S~(U )jAL) C E (H (U )jAL) Now, using Proposition 1, ii), we may write (13) as 1 0 h~ (L(U )) F (h(U )) = B A (in L; S;~ H system): @ h~ (LL((UU))) E (S~(U )jAL) C E (H (U )jAL) 20

(14)

By de nition of F (U ), it is immediate that 0 h~ (L(U )) 1 h(F (U )) = B A (in L; S;~ H system); @ h~ (LL((UU))) S~(F (U )) C H (F (U )) which is equal to 0 1 h~ (L(U )) B@ h~ (L(U )) E (S~(U )jAL) C A (in L; S;~ H system): L(U ) E (H (U )jAL)

(15)

(16)

Thus F (h(U )) = h(F (U )), i.e., the operator F is contrast invariant. Acknowledgement: We gratefully acknowledge partial support by CYCIT Project, reference TIC99-0266 and by the TMR European Project Viscosity Solutions and their Applications, FMRX-CT98-0234.

21

References [1] L. Ambrosio, V. Caselles, S. Masnou and J.M. Morel, Connected Components of Sets of Finite Perimeter and Applications to Image Processing, 1999 Preprint. [2] C. Ballester, E. Cubero-Castan, M. Gonzalez and J.M. Morel, Contrast Invariant Image Intersection, Cahiers de l'ENS Cachan, 9817, 1998. [3] P. Blomgren and T.F. Chan, Color TV: Total Variation Methods for Restoration of Vector-Valued Images, IEEE Transactions on Image Processing, vol. 7, pp. 304-309, 1998. [4] V. Caselles, B. Coll and J.M. Morel, A Kanizsa programme, Preprint CEREMADE, Univ. Paris-Dauphine, 1995. [5] V. Caselles, B. Coll and J.M. Morel, Topographic Maps and Local Contrast Changes in Natural Images, International Journal of Computer Vision 33(1), 5-27 1999. [6] A. Chambolle, Partial Di erential Equations and Image Processing, Proceedings of the Fourth IEEE Internatinal Conference on Image Processing, ICIP-94, Austin, Texas, pp. 16-20, Nov. 1994. [7] B. Coll and J. Froment, Topographics Maps of Color Images, In preparation. [8] J. L. Doob, Stochastic Processes, Wiley Classics Library, 1953. [9] L.C. Evans and R.F. Gariepy, Measure Theory and Fine Properties of Functions, Studies in Advanced Math., CRC Press, 1992. [10] L. Hjelmslev Prolegomenes a une theorie du langage, Minuit, (1971). [11] B.K. Horn, Robot Vision MIT Press, (1986) [12] W. Kandinsky, Point et ligne sur plan, Folio essais, Gallimard, (1991). [13] J.L. Lizani, P. Monasse, L. Moisan and J.M. Morel Reference [14] S. Masnou, Filtrage et desocclusion d'images par methodes d'ensembles de niveau PhD Thesis, Ceremade, Universite Paris-Dauphine, 1998. [15] W. Metzger, Gesetze des Sehens, Waldemar Kramer, 1975. [16] P. Monasse, Image Registration, Cahiers de l'ENS Cachan, 1999. [17] P. Monasse and F. Guichard, Fast Computation of a Contrast Invariant Image Representation, Cahiers de l'ENS Cachan, 9815, 1998. [18] J. Paulhan, La peinture cubiste, , Gallimard, 1990. [19] P. Perona, Private communication 1989. [20] P. Perona and J. Malik, Scale-space and edge detection using anisotropic di usion, IEEE Transactions on Pattern Anal. Machine Intell. 12, pp. 629-639, 1990. 22

[21] G. Sapiro and D.L. Ringach, Anisotropic Di usion of Multivalued Images with Applications to Color Filtering, IEEE Transactions on Image Processing, vol. 5, pp. 1582-1586, 1996. [22] J. Serra, Image Analysis and Mathematical Morphology, Academic Press, 1982. [23] L. Signac, D'Eugene Delacroix au neoimpressionnisme, Collection Savoir, Hermann 1987. [24] L. Vincent, Morphological area openings and closings for gray-scale images, Proceedings of the Worshop "Shape in Picture", 1992, Driebergen, The Netherlands, Springer, Berlin, 197-208, 1994. [25] M. Wertheimer Untersuchungen zur Lehre der Gestalt, II Psychologische Forschung, 4:301-350, 1923. [26] D. Williams, Probability with Martingales, Cambridge Mathematical Textbooks 1991. [27] W. P. Ziemer, Weakly Di erentiable Functions, GTM 120, Springer Verlag, 1989.

23