Morphological Algorithms for Color Images Based on a Generic-programming Approach M ARCOS C ORDEIRO D ’O RNELLAS1 , R EIN VAN
DEN
B OOMGAARD1 , J AN -M ARK G EUSEBROEK1
1
ISIS - Intelligent Sensory Information Systems University of Amsterdam, Faculty WINS Kruislaan, 403 - 1098 SJ Amsterdam, The Netherlands fornellas,rein,
[email protected]
Abstract. The main purpose of image processing and analysis is, often, to segment the image into objects in order to analyze the geometrical properties and the structure of the objects and recognize them. The analysis of the geometric objects must be quantitative, because only such an analysis and description of the geometric objects can provide a coherent mathematical framework for describing the spatial organization. The quantitative description of geometrical structures is the objective of mathematical morphology. So far, the use of such framework has allowed the development of a class of algorithms to deal with binary and grayscale images. Multi-channel processing has been of growing interest recently, especially in color image processing. In this paper, the extension of mathematical morphology to color images by treating multi-channel data as vectors are presented. An overview of several approaches to this extension using different ordering relations is also provided. Basic vector morphological algorithms are introduced based on reduced ordering. We also present experimental results and show that these algorithms can be easily mapped into the genericprogramming framework. Keywords: Multivalued morphology, Color image processing, Generic-programming, Ordering.
1
Introduction
Mathematical morphology is a powerful and unified approach for geometrical shape analysis and description based on set theoretic notions such as inclusion, union and intersection. Morphological operations require the selection of a probe called a structuring element. An image is analyzed by inspecting its interaction with the structuring element as this is moved around the image plane. Not only the choice of the structuring element but also the operation being performed with this structuring element determines the effect of a morphological operator. Image analysis using morphology usually consists of two steps: a transformation step, in which morphological operations are performed on an image, extracting some features from the image, and a measurement step, in which certain geometrical characteristics of the transformed image are measured as presented in Serra [19], and Haralick et all [11]. Initially, mathematical morphology was constructed for binary images, which can be represented mathematically as sets. When applied to a binary image, a morphological operation (dilation or erosion) produces an increase or decrease in spatial extent, respectively. These operations are based on the geometrical relationships or connectivities from pixels of the same type. Binary mor-
Formerly at Universidade Federal de Santa Maria, Departamento de Eletrˆonica e Computac¸a˜ o, Santa Maria-RS, Brazil. Supported by CAPES Foundation under grant BEX 2780/95-0.
phological operations are naturally expanded to grayscale morphology by using an infimum or supremum operation, as defined in Heijmans [13] [14]. Several techniques developed for use with grayscale images can be extended to color images by applying the algorithm to each one of the image components separately. The most straightforward way to process multichannel images is to treat each channel independently using single-channel techniques. However this approach does not exploit the correlation between channels. Multichannel techniques that take into account this correlation have been reported to be more effective. These algorithms are based on the concept of ranking multivariate data introduced by Barnett [2]. In this paper we describe some approaches for applying mathematical morphology to color images. We present the concept of vector morphological algorithms, which are based on vector ranking concepts proposed by Goutsias et all. [9], Serra [20], and Comer and Delp [5]. We compare the performance of some vector morphological algorithms using reduced ordering and show that these algorithms can be straightforwardly mapped into the generic-programming framework suggested by Ornellas and Boomgaard [7]. The organization of the rest of this paper is as follows. In section 2, we give a brief introduction to mathematical morphology and complete lattice theory. In sec-
Anais do XI SIBGRAPI (1998) 1–8
2
D ’O RNELLAS ,
tion 3 we describe the morphological development framework. We also introduce the generic-programming approach and how to represent color images as well as their color models. In section 4 we present the concept of multivalued ordering applied to color images. We discuss alternative approaches and discuss their advantages and drawbacks. Section 5 refers to color morphology and their operators. The evaluation performance and algorithm results are shown in section 6. Conclusion and further research are given in section 7. 2
Mathematical Morphology and Complete Lattice Theory
One of the basic ideas in mathematical morphology is that the set of all possible images constitutes a complete lattice. The following definition will be used in this paper: Definition 2.1 (Complete Lattice) A set L with a partial ordering is called complete lattice if every subset P of L has a least upper bound (supremum) _H and a greatest lower bound ^P . Let L,M be two lattices. The set of all operators mapping L into M forms a complete lattice under the partial ordering given by:
,
(X )
(X ); f or all X
2L
(1)
The set L is anti-symmetrical, reflexive, and transitive. The concept of an infimum and a supremum stems from the partial ordering relation. If the supremum and infimum exist for any collection of images taken from the lattice of all images, then that lattice is called a complete lattice as presented in Heijmans and Ronse [15]. An image filter is considered as a mapping performed within a complete lattice; the filter maps one lattice element (the input image) to another lattice element (the output image). The key insight provided by mathematical morphology is to construct these image filters from what are called erosions and dilations. Erosions and dilations have proved to be very powerful and can be combined to yield effective image filters with desirable filtering properties. From a theoretical point of view, dilation, erosion, adjunction, opening, and closing are the most important algebraic notions in mathematical morphology to characterize operators on lattices as defined as follows: Definition 2.2 (Dilation) An operator : L ! M is called a dilation if (_i2I Xi ) = _i2I (Xi ), for every collection fXi ji 2 I g. Definition 2.3 (Erosion) An operator : L ! M is called an erosion if (^i2I Xi ) = ^i2I (Xi ), for every collection fXi ji 2 I g. Definition 2.4 (Adjunction) Let : L ! M and : ! L be two operators. We say that the pair (; ) is an
M
Anais do XI SIBGRAPI, outubro de 1998
M.C. AND B OOMGAARD , R. V. D .
AND
G EUSEBROEK , J.
adjunction between L and M if (Y ) X $ Y (X ), for all X 2 L; Y 2 M . refers to erosion while refers to dilation. Definition 2.5 (Opening and Closing) The operators and form an opening on L and a closing on M , respectively. Grayscale morphological operations are an extension of binary morphological operations and serve as a basis to expand mathematical morphology to color images. For grayscale operations, the image will be represented by the function f (x; y ), where (x; y ) 2 R2 or Z 2 , or simply f , and the structuring element will be the function g (r; s) or g . Grayscale dilation, erosion, opening, and closing are defined as follows:
f g)(x; y) f g)(x; y) f g f g
(
=
(
= = =
sup( )2 ff (x ? r; y ? s) + g(r; s)g inf( )2 ff (x + r; y + s) ? g(r; s)g (f g ) g (f g ) g r;s
G
(2)
r;s
G
(3) (4) (5)
where sup and inf denote the supremum and infimum operators, respectively, and G R2 or Z 2 is the support of g (x; y ). A special class of grayscale morphological filters when g (x; y ) = 0 for every (x; y ) 2 G are referred to flat morphological operators as proposed by Crespo and Schafer [6]. 3 Morphological Development Framework 3.1 Generic-programming Approach The generic-programming paradigm within the C++ context was first proposed by Stepanov and Musser [21] and became popular with the inclusion of their Standard Template Library (STL) into the C++ standard. In C++, the template mechanism is used to build generic algorithms by means of compile-time polymorphism. Generic-programming represents efficient algorithms independently of any particular data structure and facilitates the development of more abstract algorithms. Moreover, all generic data structures bring the most advanced type of interface they can implement efficiently. In this way, an unique interface between morphological algorithms and the data structures can be defined so that the same algorithm implementation can be applied to any number of different image types (e.g. binary, grayscale, RGB color, HSI color, etc.). The intention is to assure that one can have algorithms defined as generically as possible without loosing efficiency as proposed by Ornellas and Boomgaard [7]. 3.2 Image Representation and Color Models Color is a visual object attribute that results from the combined output of three sets of retinal cones each sensitive to
M ORPHOLOGICAL A LGORITHMS
FOR
C OLOR I MAGES B ASED
different portions of the visible part of the electromagnetic spectrum. A lot of psychophysiology studies have shown that color information is generally tri-variant. The cones have peak sensitivities in the red, green and blue portions of the spectrum respectively. Any perceived color may usually be created by a variety of sets of colors when combined in the correct proportions as reported by Travis [22]. So, three variables are necessary to describe the color sensation. Serra [20] and Goutsias et all introduced the definition for a multivalued image in the scope of mathematical morphology. [9] as follows: Definition 3.1 (Multivalued Image) Given n totally ordered complete lattices T1 ; : : : ; Tn of product T; T = T1 T2 : : : Tn , a multivalued image is a mapping f : E ! T , where E is a multidimensional digital space Z m. A color model is a means of representing the three components of a color in terms of a position in a three dimensional space. RGB, HSI, and Lab are some of the color models which might be used, depending on the particular purpose of the analysis. 3.2.1 The RGB Color Model Most video monitors use red, green, and blue color sources as the primary colors for color image generation. RGB color model uses a rectangular coordinate system with one coordinate axis assigned to each of three color components, red, green, and blue. The RGB system is the classical physical color model used to digitize color images and many algorithms have been developed in the RGB space such as filtering, edge detection or region segmentation. Using the RGB color model insures that no distortion of the initial information is introduced as suggested by Golland and Bruckstein [8]. However, RGB representation is far from the human concept of color. In the RGB space, color features are highly correlated, and it is impossible to evaluate the similarity of two colors from their distance in this space. 3.2.2 The HSI Color Model The HSI color model is cylindrical with the intensity axis coinciding with the achromatic diagonal of the RGB system. Saturation is the radius from the intensity axis and hue is the angle with respect to the red direction. From a perceptual point of view, color can be described in HSI color model by three attributes. The Hue H is a value which represents the main color of the pixel in the RGB triplet; the saturation S describes the pureness of the color, and I represents the amount of light received by the sensor. It depends on the lighting conditions and on the light source emissivity.
ON A
G ENERIC - PROGRAMMING A PPROACH
3
According to Carron and Lambert [3], HSI is appealing color system for segmentation because it separates the intensity information from the chromatic information. 3.2.3 RGB versus HSI Color Model If we consider HSI color model, as long as the colors in the image are reasonably well saturated hue will tend to remain relatively constant in the presence of shadows and other lighting variations. In such cases an image based on hue alone may work better than traditional gray scale analysis. However, low saturation hue may be difficult to determine accurately and hence irrelevant. If saturation is high, Hue is then very relevant. When saturation is zero hue is undefined. Its sensitivity to the image noise is even lower than that of intensity. Serra [20] also commented that by changing the system of coordinates from a vector space (e.g. RGB) into a sort of a spherical one, we find that the HSI representation, which is also a complete lattice. Nevertheless, it is partly discrete and partially continuous. For applications that must be able to differentiate all colors, saturated and unsaturated, HSI representation can introduce significant problems. Also, McConnell [17] suggested that for segmentation and classification of objects, which may be multicolored, the disadvantages of HSI space will almost always outweigh any possible advantages. In this paper, without loss of generality, we will restrict ourselves to the RGB color model. 4 Multivalued Ordering Applied to Color Images The problem of ordering multivariate data is not straightforward to mathematical morphology. Although that there is no universally accepted multivariate ordering scheme for total ordering of multivariate samples, much work has been done to define concepts such as median, range, and extremes in multivariate analysis. Barnett [2] proposed the classification of the ordering principles in four groups: marginal ordering, reduced ordering, partial ordering, and conditional ordering. The key point to extend mathematical morphology to color images is to define a well-suited ordering relation. In mathematical morphology, we consider an image filter as a mapping performed within a complete lattice framework as proposed by Ronse [18]. Any morphological operators we apply to the color images can be applied to the components separately, due to the fact that these filters commute with infimum and supremum respectively. This kind of marginal processing is equivalent to the vectorial approach defined by the canonic lattice structure when only supremum and infimum operators and their compositions are involved and induces a totally ordered lattice as presented by Gu [10] and Goutsias et all. [9]:
Anais do XI SIBGRAPI, outubro de 1998
4
D ’O RNELLAS ,
X
Y ,
X (i)
Y (i); 8i 2 1; : : : ; M
(6)
With these relations, the supremum of a family fXj g is the vector _X where each component _X (i) is the supremum of the fXj (i)g. Respectively, the infimum of a family fXj g is the vector ^X where each component ^X (i) is the infimum of the fXj (i)g. However, this morphological procedure fails because every color can be seen as a vector in a spatial domain, which can not be ordered and so the supremum or infimum of the two is a mixture of both the colors. Besides, image components are highly correlated. Using this procedure, the lattice structure gives the same results as the simple marginal processing of the data and new colors not contained in the input image will appear even for flat structuring elements. It is indeed possible to process the image by using only the most significative component but this method gives too much emphasis to the selected component (e.g. intensity in the HSI model). Moreover, the minimum morphological theoretical background is not respected anymore since a fundamental constraint has been reduced. Other techniques based on reduced ordering consist in considering any distance metric between two vectors X and Y as explained by Comer and Delp [5]. The output of the vector filter will depend not only on the input image and the structuring element, but also on the scalar-valued function used to perform the reduced ordering. The selection of specific distance metric is very important when local statistics are taken into account as is the Mahalanobis distance [(xi ? a )T ??1 (xi ? a)]1=2 (? being the covariance matrix of the input data) suggested by Hardie and Arce [12]. The arithmetic mean, the marginal or vector as proposed median can be used as a reference vector a by Astola et all. [1]. Goutsias et all. [9] introduced another approach grounded on the use of a vector transformation from RM into RQ followed by a marginal ordering on RQ . If Q > 1, the marginal ordering induces a partial ordering on the vectors. Q = 1 is required to obtain a total ordering with an h-adjunction defined as follows: Definition 4.1 (h-adjunctions) Assume that R is a complete lattice and that T is a nonempty set. Furthermore, let h : T ! R be a surjective mapping. Define an equivalence relation =h and h on T as follows: t =h t0
h(t) = h(t0 ); t; t0
(7)
t
h(t)
(8)
, t0 , h
2T h(t0 ); t; t0 2 T
Let ; : T ! T be two mappings with the property that for St 2 T : (s)
t,s h
h
(t)
then the pair (; ) is called an h-adjunction.
Anais do XI SIBGRAPI, outubro de 1998
(9)
M.C. AND B OOMGAARD , R. V. D .
AND
G EUSEBROEK , J.
The key point is to transform the image data by means of a surjective transformation h. The underlying assumption is that the transformed image data is better suited for the morphological approach but typically, h is neither bijective nor injective: two vectors can have the same image h. This stands in need of a new corresponding equivalence relation =h . A major drawback in practice is that the extrema of a family fXj g are not necessarily unique. As a consequence, many different vectors can lead to the same result h(X ) = supi fXi g. The concept of anamorphoses and conditional lattices were introduced by Serra [20]. In this way, a color image should be morphologically processed, by introducing priorities in processing its individual components. This idea involves functional relationships whether we keep or not the symmetry among the components. Recently, Chanussot and Lambert [4] introduced the bit-mixing paradigm, following the idea of h-adjunctions. A mapping h : RM ! R is used to rank the vectors that means that each vector pixel is represented by a single scalar value. The main idea is that a bijective mapping is used, inducing a total ordering and determines clearly the infimum and supremum of each set of vectors. With the bit-mixing representation, it is possible to perform any classical morphological filter on the coded image, and to decode the result afterwards to get the color filtered image. The proposed order can be considered as an extension to a total order. Two vectors are always comparable. Moreover, the output vector of any morphological filter will necessarily be one of the input vectors. 5 Color Morphology 5.1 Ordering Color as Vectors To extend the vector approach to color images, it is necessary to define an order relation which orders the colors as vectors, rather than ordering the individual components as suggested by Jones and Talbot [16]. This can be done using reduced ordering. This kind of ordering imposes a total ordering relationship that can be accomplished by the lexicographical ordering 1 as reported by Woods [23]. The structuring element for the vector morphological operations defined here is the set G, and the scalar-valued function used for the reduced ordering is h : R3 ! R. The operation of vector dilation is represented by the symbol v . The value of the vector dilation of f by G at the point (x; y) is defined as: (f
h((f
v
G)(x; y )
2 ff (r; s) : (r; s) 2 G(
x;y )
g
(10)
and v
G)(x; y ))
h(f (r; s))8(r; s) 2 G(
x;y )
(11)
1 An ordered pair (i; j ) is lexicographically earlier than (i0 ; j 0 ) if eii0 or i = i0 and j j 0 . It is lexicographic because it correther i sponds to the dictionary ordering of two-letter words.
M ORPHOLOGICAL A LGORITHMS
FOR
C OLOR I MAGES B ASED
Similarly, vector erosion is represented by the symbol v , and the value of the vector erosion of f by G at the point (x; y ) is defined as: (f
v
G)(x; y )
2 ff (r; s) : (r; s) 2 G(
x;y )
g
(12)
ON A
v
G)(x; y ))
h(f (r; s))8(r; s) 2 G(
x;y )
5.2 Scalar-valued Functions and Algorithm Representation Since the output of the vector filter depends on the scalarvalued function used for reduced ordering, the selection of this function provides flexibility in incorporating spectral information into the multivalued image representation. For example, certain linear combinations of the tristimulus values can be used. This can be written as: t t0 $
(
h(t) = a1 t1 + a2 t2 + a3 t3 h(t0 ) = a1 t01 + a2 t02 + a3 t03 h(t) h(t0 )
t
(13)
Vector opening is defined as the sequence of vector erosion and vector dilation, and vector closing is defined as the sequence of vector dilation and vector erosion. With the above definitions for vector morphological operations we must impose the restriction that the set G is a finite set, because if G is not finite then it is possible that no value of satisfies the above equations. It is not difficult to see that using erosion and dilation in conjunction with a total ordering induced by a reduced ordering will generate no new colors. Under the total ordering relation, the infimum or supremum will be one of the actual colors (and not a mix of all the colors). Therefore, the only colors in the output image will be those obtained from translations of the input colors. A similar result holds for combinations of erosions and dilations such as openings and closings.
(14)
if the image is filtered in the RGB color model. For the case where a1 = 0:299; a2 = 0:587, and a3 = 0:114, h(t) and h(t0 ) become the luminance component. Multiscale opening in this case would suppress bright objects at each scale. The values of a1 ; a2 , and a3 can also be selected to enhance or suppress specific colors. For example, if a1 = 1; a2 = 0, and a3 = 0, then the effect of a multiscale opening would be to suppress objects with high red content. The same holds to the green and blue color channels. If we choose a running maximum approach, we start with an arbitrary pixel inside the mask and call that one the maximum. Then we compare the next pixels with this maximum. As soon as we found a pixel, which has a bigger rank, we assign that pixel as maximum and continue until all pixels are examined within the filter mask. Note that the difference between the red component and
5
the maximum of the other two channels is taken for ranking. This means that the purity of red determines the ranking order. For instance, if we have a yellow pixel [255; 255; 0], the ranking order for it is 0. This can be represented as:
and h((f
G ENERIC - PROGRAMMING A PPROACH
t0 $
(
h(t) = a1 t1 h(t0 ) = a1 t01 h(t) h(t0 )
? MAX(a2 t20 ; a3 t30 ) ? MAX(a2 t2 ; a3 t3 )
(15)
Serra [20] suggested that when linear relationships are employed the partial order relationship associated with a conditional lattice should be given for the case of three components by (symmetrical and not symmetrical respectively):
(
t010 t20 t3 + k1 (t01 ? t1 ) + k2 (t02 ? t2 ) ( t1 t01 0 t2 t02 + k0 (t01 ? t1 tt $ t3 t03 + k1 (t01 ? t1 ) + k2 (t02 ? t2 ) t
t0 $
t1 t2 t3
(16)
(17)
where k0 ; k1 , and k2 are constants. When the bit-mix paradigm proposed by Chanussot and Lambert [4] is used, the transform h is based on the representation of each component of T in the binary mode. Let T 2 RM with M components t(i), each one represented on p bits t(i)j 2 f0; 1g with j 2 f0; : : : ; pg. The considered mapping h can then be written as follows:
8
