Soft Data Fusion in Image Processing - Semantic Scholar

Soft Data Fusion in Image Processing Aureli Soria-Frisch Fraunhofer IPK, Dept. Pattern Recognition, Pascalstr. 8-9, 10587 Berlin, Germany; aureli.soria [email protected]

Abstract. Data fusion is a long term of research in image processing that is becom-

ing more and more relevant owing to the complementary developments of computer and sensory technologies. Although operator research related to soft-computing, specially in the eld of fuzzy systems, has evolved considerably during this last two decades, implemented frameworks of data fusion for image processing take seldom into consideration this kind of operators. Most of pattern recognition systems with image fusion are still based in basic operators, e.g. minimum or product. The purpose of the here presented tutorial is to analyze this fact, present some of the fuzzy aggregation operators in the context of data fusion for image processing and show some applications where the usage of the fuzzy integral, one of these operators, increased the performance of image processing systems considering data fusion.

1 Introduction The relevance of Data Fusion methodologies increases owing to the complementary development of computer and sensory technologies. The availability of novel and more reliable sensors is complemented by the existence of new computing facilities, whose memory capability and processor times allow the implementation of powerful signal processing methodologies at decreasing costs. Up-to-date hardware and software architectures encourage the implementation of parallel processing strategies and thus the integration of dierent sensors in the embedding system in order to achieve goals of increasing complexity. Mostly the attainment of such goals can only succeed on condition that a diverse information is collected and that this information is jointly exploited. Data Fusion methodologies play a capital role in this picture. Basically Data Fusion methodologies attain the transformation of the information delivered by multiple sources into one representational form [1]. In this sense multisensory fusion is considered to be a special case of multisensory integration, which is de ned as the mere inclusion of dierent sensor units in a system [18]. Furthermore the fused data re ects information that can be extracted from the individual sources but also information not derivable from any of them alone [1]. Such a gain on information characterizes the purpose of Data Fusion. Operator research in the context of fuzzy systems have generated a fruitful set of aggregation operators, e.g. OWAs, Fuzzy Integrals, Weighted Ranking Operators, etc. Such operators constitute a exible mean for the fusion of

2

Aureli Soria-Frisch

information represented by fuzzy membership functions. So-called fuzzy aggregation operators constitute a exible alternative to operators traditionally used in Data Fusion. In the following the term Fuzzy Fusion Operators will be preferred, in order to make clear the idea that such operators are over the concept of aggregation, which is more related to the mathematical addition. The here presented tutorial pursues to give the reader an overview of recent developments on fuzzy fusion operators and an introductory material to the sometimes complex theoretical background. The presentation will be centered in the image processing application eld. The paper is organized as following: in section 2 the eld of Data Fusion for Image Processing will be brie y introduced. An overview on fuzzy fusion operators can be found in section 3. In section 4 the Fuzzy Integral and the fusion paradigm Intelligent Localized Fusion, which was developed by the author based on the usage of the Fuzzy Integral in Image Processing tasks, will be described. Section 5 presents dierent applications of the described operators: color edge detection, image segmentation through bio-inspired multisensory fusion, preprocessing in the automated visual inspection of high re ective materials, image processing in document analysis, and a framework of mathematical morphology for color images. Finally the conclusions can be found in section 6. Due to the limited extension of the paper, the dierent applications will not be deeply analyzed. The reader is referred to the literature references for more information on the dierent frameworks.

2 Image Processing with Data Fusion Operators Specially in the elds of Image Processing and Computer Vision the development of sensory technologies is undertaking a very interesting evolution nowadays. Image devices at other spectral bands than the visible one are reaching the market at decreasing costs. The appearance of multisensory image acquisition devices, where dierent information sensor units are integrated, are accessible, e.g. multispectral cameras, range-grayvalue cameras. New visualization techniques are developed specially in medical imaging. Summarizing it can be stated that the imagery available in a computer vision system is broader than some years ago. This fact leads to a emergence of interest on Data Fusion methodologies for image processing. Data Fusion realizes in Image Processing the reduction of multiple image entities to one. Thus dierent types of fusion [18], namely signal, pixel, segment, and feature fusion can be found in Image Processing. This classi cation is based on the type of entities involved in the fusion procedure. At this point it is worth de ning the scope of the term Data Fusion. The representational form resulting form a data fusion operation must be in the same abstraction level as the elements being fused in order for the operation to be considered as Data Fusion. I.e. a procedure where a segment is obtained after operating on some pixels would not be considered as a fusion operation. Some image

Soft Data Fusion in Image Processing

3

processing frameworks presented as Data Fusion do not fall in this category. Moreover it is an important goal in Data Fusion for Image Processing, as in general Data Fusion methodologies, to preserve the information available from the individual sources, as well to complement it with a gain of information not that can be extracted from the information sources separately [1]. Data Fusion methodologies can be found in image processing systems treating : color [27], multispectral [17], multi-modal [17], range and grayvalue [40], infrared and visible [22], synthetic aperture radar (SAR) and grayvalue [9], temporal sequenced images [19], and nally images taken under dierent conditions of illumination [32], shuttle speed [7], focal distance [24] or camera position [25]. Medical imaging, microscopic imaging, remote sensing, automated visual inspection, robotics, automatic target recognition (ATR) and 3D model reconstruction are some of the application elds where Data Fusion is employed. A classi cation of operators used in Image Processing for Data Fusion can be made based on its theoretical background. Among all operators used in Data Fusion classical operators, i.e. product, sum, mean , median and their non-linear versions, are used in a great part of applications [2]. Other employed operators are related with theoretical frameworks used mostly for other purposes: Theory of Regularization [27], Bayesian Theory [8], Evidence Theory [12], Image Algebra [34], Neural Networks [20]. Moreover Kallman Filtering [3], Laplacian Pyramids [7] and Wavelet Transform [17] are some of the general purpose image processing methodologies used also for Data Fusion. Finally a set of operators is derived from the fruitful activity in operator research that is developed in the theoretical framework of Fuzzy Systems: fuzzy connectives, as T-, S- and Uni-norms [9], weighted ranking operators [36], Ordered Weighted Averaging (OWA) operators [15], and fuzzy integrals [35]. The operators belonging to this last group in front of classical ones emphasize the exibility and interpretability of the fusion operation. Furthermore these operators were developed specially for that purpose. Despite these positive features fuzzy fusion operators are not widely used, e.g. [3] [2] [17] [22]. The employment of fuzzy related fusion operators can be spread through the elucidation of their application possibilities and of their sometimes complex mathematical background.

3 From Hard to Soft Fusion Operators In an increasing number of applications operators of dierent theoretical backgrounds are mixed in order to take advantage of the positive features of each framework. As a matter of fact operators used in the context of fuzzy systems were established as generalizations of classical ones (see Fig. 1).

4

Aureli Soria-Frisch

STAT. MOMENTS (AVERAGE)

RANKING OP. (MIN, MEDIAN, MAX)

LOGICAL OP. (AND, OR)

WIGHTED SUM

ALGEBRAIC OP. (SUM, PROD)

classical operators

weighting operators

WEIGHTED RANKING (WMIN, WMED, WMAX) OWA

T-, S-NORMS

fuzzy logic related operators

UNI-NORMS CHOQUET FI

SUGENO FI fuzzy integrals

WEBER FI

Grade of Hard-/Softiness

Generalization relationship

Fig.1. Relational map of dierent fusion operators. The grade of softness increase

in the vertical axis from the top to the bottom and is a result of successive generalizations. In the horizontal axis the operators are grouped upon its avor, which de nes dierent families of operators. In the vertical axis, the operators are grouped upon dierent theoretical frameworks in operation research.

The evolution of fuzzy fusion operators from harder ones is based on the consideration of dierent factors in the fusion procedure. In classical operators the fusion result depends exclusively on the value being operated on. In weighting operators another factor is taken into consideration, namely the a priori importance of the information sources. These operators are used for instance in neural networks paradigms. In the theoretical framework of Fuzzy Logic the new degree of softness is achieved through the parameterization of the aggregation, e.g. T- and S-norms [36], or the consideration of the ranking as a factor upon which the already mentioned a priori importance can be modi ed. This strategy is used in OWAs and increases the adaptivity of the operators and its capability concerning compatibility, partial aggregation and reinforcement [41]. Finally fuzzy integrals re ect in the fusion result all this mentioned information: the value delivered by the dierent sources, their a priori importance, and their ranking. Therefore fuzzy integrals are the only fusion operators in this theoretical framework taking into consideration the importance of the individual information sources and that of their possible coalitions [10]. Actually the weighting scheme in this operators succeed in form of so-called fuzzy measures. Last but not least fuzzy integrals generalize most of fusion operators considered in the context of fuzzy sets (see Fig. 2).

Soft Data Fusion in Image Processing

5

Ft-cI CFI

OWA

min

weighted min

arithm.

SFI

med

mean

MAX

weighted sum

weigted MAX

Fig.2. Generalization relationship between fuzzy fusion operators. Ft-cI: Fuzzy

t-connorm Integral. CFI: Choquet Fuzzy Integral. SFI: Sugeno's Fuzzy Integral. OWA: Ordered Weighted Averaging. med: median. Modi ed from a gure in [11] (with permission).

4 Soft Fusion through the Fuzzy Integral The concept of fuzzy integral is due to Sugeno, who presented in his work [33] a mathematical approach for the simulation of multicriteria evaluation taking into consideration some cognitive aspects. Sugeno's hypothesis is that the process of multicriteria integration undertaken by human beings transcend the linear combination of the dierent criteria with numerically expressed priorities, i.e. weighting sum strategy. Thus fuzzy measures, a generalization of classical measures through the consideration of subjectiveness, were established as the mean of expressing the a priori importance of the integrands. A new type of measures led to a new integration: the fuzzy integral.

4.1 Theoretical Background The elements that can make a mathematical expression for the fusion of information follow the hypothesis posited by Sugeno are the utilization of fuzzy membership functions as integrands, their weighting through fuzzy measures, and the binding of these two elements through a combination of T- and Snorms [36]. The fuzzi cation of the data from the information sources xi is made through the application of fuzzy membership functions, which are named here as hi(x). Fuzzy measure coecients (Aj ) are de ned on all possible subsets Aj of the set of information sources X = fx1; : : :; xng. Sugeno proposed the following mathematical expression for the new integral, which is known as the Sugeno Fuzzy Integral (SFI):

S (x ; : : :; xn) = 1

_n [h i (xi) ^ (A i )];

i=1

( )

( )

(1)

6

Aureli Soria-Frisch

where the used T- and S-norms are in this case the minimum (^) and the maximum (_) operators. The enclosed subindex states for a sort operation previous to the integration itself, e.g.

h1  h3  h2 ! h(1) = h1; h(2) = h3; h(3) = h2 ; (2) operation that would also determine the coecients of the fuzzy measures employed in the integration as (A(1) ) = (fx1 g); (A(2)) = (fx1 ; x3g); (A(3) ) = (fx1; x2; x3g): (3) The consideration of the information to be integrated in form of fuzzy membership functions and the utilization of the fuzzy measures as weighting factors improve the interpretability of the operators. The dependence of the weighting scheme on the ranking and the usage of a combination of norms are responsible of the exibility of the operator. The Choquet Fuzzy Integral (CFI) was introduced as a fuzzy generalization of the Lesbesgue Integral [39]. The T- and S-norms are in this case the product and the addition. The resulting expression has the form:

C (x ; : : :; xn) = 1

Xn h i (xi)
[(A i ) , (A i, )]: i=1

( )

( )

(

1)

(4)

While the Choquet Fuzzy Integral can be considered as a mean-like fusion operation, the Sugeno Fuzzy Integral is near to the median operator (see Fig. 2). There are other types of fuzzy integral de ned by dierent authors in order to generalize the already mentioned types of fuzzy integral. For the sake of simplicity only the fuzzy t-conorm integral [21] is mentioned. This integral is a generalization of the Choquet and Sugeno Fuzzy Integrals. The existence of such a vast number of fuzzy integral operators [10] [38] improves the exibility of the fusion operators, but also makes the understanding of the theoretical background more dicult.

4.2 The Role of Fuzzy Measures The Theory of Fuzzy Measures [38] was built upon Sugeno's conclusions. Mathematically the fuzzy measures are functions on fuzzy sets,  : P (X ) ! [0; 1], satisfying the following conditions in the discrete case: I. f;g = 0; fXg = 1, II. A  B ! (A)  (B ) 8A; B 2 P (X ). The addition of subjectivity to the classical measure theory was done then through the relaxation of the additivity axiom in classical measures, i.e. probability measures. Thus fuzzy measures generalize dierent types of measures

Soft Data Fusion in Image Processing

7

used in dierent theoretical frameworks, i.e. probability, possibility, belief, and -fuzzy measures. In the fuzzy integral the fuzzy measures coecients are used for quantifying the a priori importance of the information sources being integrated. This quanti cation is realized on the sources individually, but also on their possible coalitions. An analogy to a decision making process simulated by the fuzzy integral can clarify this point. One can imagine a situation where an answer to a problem is sought and dierent experts are asked to solve it. The coecients of individual sources, e.g. (fx1g), will be used to quantify the a priori importance of one of the experts, e.g. the rst one. The other ones would quantify the agreement between experts, e.g. (fx1 ; x3g) would quantify the importance of the rst and third experts building a coalition. The eect of this parameterization on image processing can be pictorially shown (see Fig.3).

(a)

(b)

(d)

(c)

(e)

Fig.3. Exemplary eect of the fuzzy measure modi cation in the integration of

color channels through a fuzzy integral. (a-c) Input channels: (a) red, (b) green and (c) blue. (d-e) Results of the integration through the Sugeno Fuzzy Integral. (d) Result for a fuzzy measure: R = 75,G = 40,B = 150,RG = 100,RB = 170, GB = 200,RGB = 255. (e) Result for a fuzzy measure: R = 75,G = 40,B = 150,RG = 220,RB = 170, GB = 200,RGB = 255. The change of the fuzzy measure coecient RG , (d) vs. (e), modi es the result of the yellow areas, where red and green channels agree, i.e. present high values.

8

Aureli Soria-Frisch

4.3 Some Properties of Fuzzy Integrals Dierent properties of the fuzzy integral, which will be recalled in the application descriptions, can be found in this section. First the response of the operator in front of incoming data, what is called in lter theory the transfer function and in fuzzy sets related operator research known as level surfaces or curves, will be described (see gures 4 and 5). Both types of integral present monotone non decreasing functions in the feature space [0; 1]N . They divide the feature space in dominance areas, due to the sort operation previous to the aggregation itself (2). These dominance areas, which receive the name of canonical regions of the hypercube [10], are de ned by the hyperplane that joints the origin and the point where all the features present their maxima (see gure 4b). 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1

1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1

x2

x1

µ2 µ2

x2 x2

0

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 x1 0.8 0.9

1 0

0.2 0.1

0.6 0.5 0.4 0.3

0.9 0.8 0.7 x2

(a)

1

µ1

x1 µ1

x1 (b)

Fig.4. Level curve of the Sugeno Fuzzy integral. (a) Response for 1 = 0:2; 2 = 0:6. (b) Projection on the feature plane.

The Sugeno Fuzzy Integral (see gure 4) presents so-called constant regions in its level curve [10], whose hypervolume depends on the values of the fuzzy measure coecients. Here the output value of the fuzzy integral is constant and equal to the value of the corresponding fuzzy measure coecient (see gure 4b). In the Choquet Fuzzy Integral the action of the fuzzy measure coecients is to modify the gradient of the channel being evaluated by that coecient (see gure 5). The larger the coecient is, the larger is the gradient for that variable. Thus the variations of the features with larger fuzzy measure coecients will be re ected stronger in the fuzzy integral result than these variations of the features with smaller ones. The result of the fuzzy integral ranges from that of the minimum operator to that of the maximum [10]. Thus it covers all the range of reasonable

Soft Data Fusion in Image Processing

9

0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1

1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 x1 0.8 0.9

1 0

0.2 0.1

0.6 0.5 0.4 0.3

0.9 0.8 0.7

1

x2

Fig.5. Level curve of the Choquet Fuzzy integral for 1 = 0:2; 2 = 0:6. results for a fusion operator. As formerly mentioned this result depends on the value of the fuzzy measure coecients. Furthermore the fuzzy measure coecients are implicitly responsible for the generalization relationships between fuzzy fusion operators (see Fig.2). For instance a minimum operator can be expressed as a Sugeno Fuzzy Integral where all coecients except that for the set of all information sources are 0. Analogously an OWA, which has the expression (being W = fw1; w2; : : :; wng the set of weights)

OWA(x ; : : :; xn) = 1

Xn wix i ; i=1

(5)

( )

will deliver the same result as a Choquet Fuzzy Integral where the fuzzy measure coecients of the subsets with the same cardinality are equal to wi. The generalization relationship of fuzzy integrals to most of fuzzy fusion operators is treated from a mathematical point of view in [10]. Finally the utilization of fuzzy integrals to measure similarity will be analyzed. Fuzzy fusion operators have been presented in a common theoretical framework [5], which states that aggregation operators can be used to measure distances in metric spaces. That is due to the existence of an intrinsic relationship between the aggregation operator and the distance function being used. The theoretical framework is built upon the utilization of aggregation operators in Fuzzy Systems Theory to establish the membership function of the output variable. Here the result can be seen to measure the similarity to prototype elements of the output set, which receive the name of Ideal (I ) and Anti-Ideal (;) [5]. A pictorial representation of these sets can be found in Fig.6. In this context the membership functions of the Ideal (I ) and the AntiIdeal (AI ) are considered to be [5]

I (X ) = 1 , d(I; X )

AI (X ) = d(;; X );

(6)

10

Aureli Soria-Frisch µ2 Ideal µI(x) x(µ1,µ2) µAI(x) Anti-Ideal

µ1

Fig.6. Representation of a fuzzy set X in a two-dimensional space and the corresponding membership functions respect to the sets Ideal and Anti-Ideal.

where d are distance functions. Since these memberships can be calculated through aggregation operators, with the only condition of being monotonic, and having a result value between 0 and 1, following relationships can be set: d(I; X ) = 1 , M (X ) d(;; X ) = N (X ); (7) where M and N are arbitrary aggregation operators for fuzzy AND and OR respectively. For theorem, demonstration and deeper explanations the mentioned work is referred [5]. Being the fuzzy integrals monotonic operators limited in the range [0; 1] for regular fuzzy measures, it seems possible to use the fuzzy integral for the computation of similarity relations to the Ideal and Anti-Ideal points of the hypercube. One open question in this de nition is the dierentiation between M and N . The work on applications [31] has shown that a practicable strategy is to use the fuzzy integral respect to a fuzzy measure  for the computation of the distance to the Ideal, while the so-called dual fuzzy measure  is used for the distance to the Anti-Ideal. The duality relationship between fuzzy measures is usually de ned between possibility and necessity measures or between belief and plausibility measures [10]. A pair of dual fuzzy measures (; ) satis es:  (
) = 1 , (
c); (8) where (
c) states for the complement set.

4.4 Intelligent Localized Fusion Operators Intelligent Localized Fusion (ILF) is a new paradigm for fuzzy fusion in image processing presented by the author [30]. Fuzzy Integrals become so-called Intelligent Localized Fusion Operators (ILFO) through the local de nition of fuzzy measures.

Soft Data Fusion in Image Processing

11

Usually only one fuzzy measure is de ned when being used in a fusion operation through the fuzzy integral. The fuzzy measure, as already mentioned, are used to quantify the a priori importance of the information channels and of its coalitions. Traditionally such quanti cation has been made globally. On the other hand dierent fuzzy measures are de ned in the ILF paradigm. This fuzzy measures are locally distributed through a mask that points out the fuzzy measure to be used in each loci (see Fig.7). µ

1

µ2

(a)

(b)

Fig.7. Exemplary usage of an ILFO with the image in Fig.3. In the ILF paradigm

dierent fuzzy measures are used in a fuzzy integral. (a) Example of a mask whereby the dierent fuzzy measures, e.g. 1 and 2 , are localized. (b) Fusion result where the 1 adopted the values used for obtaining Fig.3d and 2 those used for obtaining Fig.3e

The purpose of such a localization is the exploitation of the generalization capability of the fuzzy integral respect to other fuzzy fusion operators. Through the utilization of more than a fuzzy measure a new dimension of the fusion is achieved, namely the possibility of using dierent fusion avors in the same image. For instance one could be interested of using an OWA strategy in a determined area, while preferring a minimum operator for another one. The application of the ILF paradigm facilitates the realization of such mixed strategies.

5 Image Processing Applications of the Fuzzy Integral Although Image Processing was one of the rst application elds of the Fuzzy Integral [26], data fusion operators used for image processing are still dominated by classical operators. Beyond the theoretical complex background, the lack of standardized procedures for the automated determination of the fuzzy measure coecients [13] seem to be one of the reasons of this underestimation. The here presented applications do not handle directly this question, although some important considerations for the attainment of this goal can be derived from its reading. The following sections demonstrate the exibility that the fuzzy integral can bring into fusion operations for image processing.

12

Aureli Soria-Frisch

Moreover the fuzzy measure coecients can be seen as a parameter, whose determination is application dependent.

5.1 Color Edge Detection Edge detection is a very important image processing task involved in numerous applications. Paradigms for image segmentation, pattern recognition, image understanding, edge preserving smoothing, and image compression include the edge detection as a pre-processing stage. Mathematical morphology [14], orthogonal polynomials [16], statistical moments [23], and statistical vector elds [37] are some of the theoretical frameworks employed till now in the resolution of this problem on color images. These works are based on the application of complex mathematical concepts in order to cope with the vectorial nature of color information. On the other hand the here presented framework uses the long-term experience on edge detection for grayvalue images. In order to deal with the special features of color vision [4] the employed strategy here is to use standard grayvalue edge detectors on the individual channels and to leave the consideration of those features to the fusion operator. The proposed scheme considers the modules Edge Detection (ED), Fuzzi er (FUZZ), Determination of Fuzzy Measures (DFM) and Edge Fusion (EF) as depicted in Fig.8. For more information about the framework and its application the reader is referred to [29]. Red Chanel Green Chanel

FUZZ

ED

EF

Color Edge Map

Blue Chanel

a priori Knowledge

DFM

Fig.8. Scheme for color edge detection with Intelligent Fusion Operators, ILFOs. ED: Edge Detection. FUZZ: Fuzzi er. EF: Edge Fusion. DFM: Determination of Fuzzy Measures

The proposed scheme for the color edge detection was exemplary implemented using a Haar lter based edge detector, sigmoid-type fuzzifying functions, and the Sugenos Fuzzy Integral respectively for the Edge Detection (ED), the Fuzzi er (FUZZ), and the Edge Fusion (EF) modules. The fuzzi cation of the input image channels is undertaken in order to suppress background pixels, acting as a noise lter. The obtained results are shown in Fig.9. A comparison of the dierent results shows the quality of using the fuzzy integral and thus of the parameterization of the fusion through the fuzzy

Soft Data Fusion in Image Processing

(a)

(b)

(c)

(d)

13

Fig.9. Results with the framework for Color Edge Detection employing dierent

values of the fuzzy measures coecients and Sugeno Fuzzy Integral (SFI). (a) Input image (original in color). (b) Color edge map for values R = 0:6, G = 0:25, B = 0:8 (fuzzy- measures). (c) Fusion with SFI as minimum. (d) Fusion with SFI as maximum.

measures. The operator used for Fig.9a shows no loss of edges with a more natural aspect than the result of a maximum in Fig.9b. The determination of the fuzzy measures was done empirically. The employment of an ILFO opens new perspectives (see Fig.10). Through the usage of a mask, where the shadow regions present a dierent fuzzy measure than other areas , the detection of shadow false edges can be avoided.

5.2 Bio-inspired Multisensory Fusion for Image Segmentation

In this work a bio-inspired system for the fusion of color and infrared images is implemented. The goal of the system is the segmentation of the images in the input channels taking into consideration texture and color information for visual inspection. The processing of the color image is inspired in the processing undertaken in the visual primary system [28]. Moreover a parallelism between multisensory fusion in human brain and the nal segment fusion is used (see Fig.11) [28]. The fuzzy integral is used for the fusion of color edge maps, for the fusion of color and texture information, and nally for the fusion of the segmented color and infra-red images. The fusion of color edge map, which result from the application of Gabor lters, follows the framework presented in the former section.

14

Aureli Soria-Frisch

(a)

(b)

Fig.10. Result of the fusion through a SFI-based ILFO, where the avoidance of

shadow false edges is achieved. (a) Mask for the fuzzy measures. (b) Color edge maps after fusion. Color Processing (V1, V2) R G B

RG RGB to YB OPP (Retina, LGN) BW

CPIV

Segmentation (V4) Form Processing (V1, V2)

PreProcessing

IF (Associative Areas)

IR-Label Image Segmentation

Form Processing

CPIR

V-Edge Image JIA (Hippocampus)

PreSegmentation IR

V-Label Image

FITS IR-Edge Image

Fig.11. Bio-inspired framework for the fusion of visual and infrared images. CPIV: Visual Image Processing Channel. CPIR: Infra-red Image Processing Channel. FTIS: Segmented Textural Information Fusion. RGBtoOPP: RGB to Color Opponencies. JIA: Joint Image Analysis. IF: Information Fusion. Inspiring biological modules from the primary visual system and high level areas are enclosed.

Since this work is currently in progress, only preliminary results of the fusion of segmented images is shown (see Fig.12). The color and infrared images were segmented through a Watersheed Transformation [6] after bio-inspired pre-processing. The segmented images are fused with the fuzzy integral, which achieves the successful fusion of reinforcing area labels (see Fig.12e).

5.3 Automated Visual Inspection of High Re ective Materials

The framework presented in this section is conceived to be used as preprocessing system in the automated visual inspection of materials with a high re ectivity. Dierent images of the object under inspection taken under dierent conditions (of illumination or shuttle time) are fused through a fuzzy fusion operator. The goal of the fusion in this case is the avoidance of the highlights produced by the re ective surface. The system was applied in

Soft Data Fusion in Image Processing

(a)

(c)

15

(b)

(d)

(e)

Fig.12. Preliminary results with bio-inspired framework for the fusion of visual and infrared images. The results are obtained from a system for the automated visual inspection of textiles. Here a failure in the printing process has to be detected. The result of such faulty process can be observed in the color image (a) as a white thin band on the ower leave. In the IR image (b), in black. (c) Segmented color image. (d) Segmented infrared image. (e) Fusion of both segmented images.

both identi cation and detection tasks (see Fig.13). A detailed explanation on the application of the here presented framework can be found in [32].

(a) (b)

Fig.13. Two of the input images in the pre-processing system of high re ective

material images. (a) Image used in an identi cation system of chocolate items in a pack. (b) Image used for the detection of failures in lamp body.

In the rst application the nal goal of the system was to detect the absence of some item in a chocolate pack (see Fig.13a). A pre-processing was necessary in order to avoid the re ectance of the plastic bundle. It can be observed in Fig.14 that the softer the used fuzzy fusion operator is, the better is the performance of the system in terms of image contrast. The utilization

16

Aureli Soria-Frisch

of genetic algorithms for the determination of the fuzzy measure improved the results (see Figs.14d and 14e).

(a)

(b)

(c)

(d)

(e)

Fig.14. Results of the pre-processing system for the avoidance of highlights in

the identi cation problem. Dierent fuzzy fusion operators were used. (a) Result with minimum operator. (b) With OWA. (c) With Choquet Fuzzy Integral (CFI). (d) With automated parameterized OWA. (e) With automated parameterized CFI. The automated parameterization was realized through genetic algorithms.

The framework was also used in a system, whose nal goal was the detection of structural faults in lamp bodies. The starting idea in this case was not to suppress all the re ections but to use the re ections on the faults as help for the detection. The usage of the ILFO paradigm allows this by preprocessing dierent image regions selectively. Thus a mask was used where the highlighted areas of smaller size are considered to be candidate faults. The results are encouraging, although further work have to be invested for the practical usage of the framework (see Fig.15).

(a)

(b)

(c)

Fig.15. Results of the pre-processing system for the avoidance of highlights in

the detection problem. The fault to be detected can be visualized as a triangular lighter area in the upper part of the lamp. Dierent fuzzy fusion operators were used. (a) Result with weighted minimum operator. (b) With OWA. (c) With ILFO (Choquet Fuzzy Integral with localized fuzzy measures). The usage of an ILFO does not suppress the highlight on the fault in order to facilitate its detection.

Soft Data Fusion in Image Processing

17

5.4 Document Analysis through Data Fusion As already mentioned the fuzzy integral was rst used in image processing as image segmentation tool []. This work is the starting point for the developments here presented. The existence of constant regions (see section 4.3) in the level curve of the SFI was exploited in a system for the classi cation of archive cards. The system can be used for the suppression of textured backgrounds in the archive cards, which facilitates the extraction of the alphanumeric information (see Fig.16).

(a)

(b)

Fig.16. Usage of the Sugeno Fuzzy Integral for background-foreground separation

on color images of archive cards. The textured background is substituted by a unique grayvalue. (a) Input image and detail. (b) Output image and detail.

The same principle can be used for the segmentation of cards with dierent colors. The obtained results are shown in Fig.17. After a morphological dilation of the input images the SFI operates over the input channels with coecients of the fuzzy measure adapted for each color. Thence the obtained label images are fused. This kind of segmentation presents some errors due to the incomplete suppression of text in the pre-processing stage (see Fig.17c). A bit more complex system was used for the segmentation of stamps on documents. The used strategy was to compute the fuzzy integral with two dierent fuzzy measures. These were selected in order for the stamp color color cluster to be maximally aected by the change. The dierence image of these results was fused with the input image through a logical AND. The CFI, which presents a smoother response than the SFI, was proven to present a better performance. The obtained results are depicted in Fig.18.

18

Aureli Soria-Frisch

(a)

(b)

(c)

Fig.17. Usage of the Sugeno Fuzzy Integral for segmentation of archive card color

images. (a) Input image with four cards. (b) Image after morphological dilation for the suppression of text. (c) Label image resulting from the segmentation.

(a)

(b)

(c)

(d)

Fig.18. Usage of the Sugeno Fuzzy Integral for segmentation of stamps. A fuzzy integral respect to two dierent fuzzy measures. The change of fuzzy measure aects the stamp color cluster, leaving the other components of the image unmodi ed. (a) Input image. (b) Fuzzy Integral result with the rst fuzzy measure. (c) Fuzzy Integral result with the second fuzzy measure. (d) Final result, where the dierence image between (b) and (c) was used as mask for the input one (a).

5.5 Color Morphology

Mathematical Morphology constitutes a well-known discipline for the analysis of spatial structures which has found an extended application in image processing. The basic element for the de nition of a morphological operation is the existence of a ranking scheme, which is applied over the pixels underlying an structuring element known as mask. In grayvalue morphology the concepts of maximum and minimum are clear (respectively white and black grayvalues). Due to the multi-dimensional nature of color images such concepts are here not well de ned. Moreover the color de nition presents some degree of uncertainty. In spite of this fact only one scheme for a color morphology [14] takes into consideration fuzzy concepts for the treatment of this uncertainty. The existence of Ideals and Anti-Ideals in the formerly described theoretical framework for fuzzy fusion operators (see section 4.3) was used as starting point for the de nition of a new color morphology [31]. The result of the fuzzy integral is used for the ranking of the pixel values under the mask. This result measures the distance of the point to the Ideal, which is

Soft Data Fusion in Image Processing

19

thence used in dilation operations. The corresponding fuzzy measure weights the color channels and breaks the indetermination due to the vectorial nature of color. For erosion operations the distance to the Anti-Ideal is computed. This operation is undertaken with the fuzzy integral and the fuzzy measure dual to the one formerly applied. Exemplary the pseudo-code for a dilation operation appears in the following: determine a fuzzy measure for each point in the mask compute distance to the Ideal if there exists only one point with minimum distance pixel in the mask center takes its color values else for each point with minimum distance compute distance to the Anti-Ideal if there exists only one point with minimum distance pixel in the mask center takes its color values else pixel in the mask center takes one of these c. values

One novelty of the here presented framework is the de nition of so-called directed morphological operations, which are based on the existence of the fuzzy measure coecients weighting the color channels. For instance in a reddilation the colors near to the red will be preferred over the others. Such a preference is established through the values of the fuzzy measure coecients. The example depicted in Fig.19 elucidates this concept.

(a)

(b)

(c)

(d)

Fig.19. Results of directed morphological operations with a cross mask on a ran-

dom color image. (a) Input image, where pixel color is randomly de ned. (b) Reddilation with R = 128, G = 1, B = 1, RG = 200, RB = 200, GB = 1, RGB = 255. (c) Green-dilation with R = 1 ,G = 128, B = 1, RG = 200, RB = 1, GB = 200, RGB = 255. (d) Blue-erosion with R = 1, G = 1, B = 128, RG = 1, RB = 200, GB = 200, RGB = 255.

The de ned morphology was employed in the pre-processing stage of a system for the automated visual inspection of textiles. The results can be observed in Fig.20. In this case the the application of morphological operations directed to the color of the faults succeed in enhancing its visualization.

20

Aureli Soria-Frisch

(a)

(b)

(c)

(d)

Fig.20. Results of the here presented color morphology for the automated visual inspection of textiles. (a) Input image. (b) Result on (a) of a non-directed dilation with a cross mask. (c) Input image. (d) Result on (c) of an erosion with a linear mask directed to the fault color.

6 Conclusions In spite of the great diversity of fusion operators, most of fusion methodologies in real applications employ data fusion hard operators, e.g. product, sum, ranking operators. Beside them there is a great number of softer operators mostly developed in the context of operators research related to fuzzy sets. The transition from hard- to soft-fusion is progressive. Fuzzy fusion operators generalize most known aggregation operators. The complexity and computational cost of fuzzy fusion operators is larger than that of traditionally used ones. Nevertheless Soft Data Fusion presents a exible theoretical framework, suitable for the implementation of robust image processing industrial applications. The here presented tutorial elucidated the mathematical background of fuzzy fusion operators in order for them to become broader applied. It was shown that the fuzzy integral can be employed for image processing with very dierent purposes. The new introduced ILFO's constitute a good example of this exibility. Some of the still open research questions on Soft Data Fusion are: the research on computationally ecient algorithms for the implementation of the dierent operators, the automation of their parameterization, and extending its employment in a larger number of real applications.

References 1. M.A. Abidi, R.C. Gonzalez, eds. (1992). Data Fusion in Robotics and Machine Intelligence. San Diego: Academic Press. 2. F. Alkoot and J.Kittler (2000). Improving the performance of the Product Fusion Strategy. Proc. 15th International Conference on Pattern Recognition, ICPR'2000, Barcelona, Catalonia. 3. N. Ayache, O. Faugueras (1989). Maintaining representations of the environment of a mobile robot. IEEE Trans. Robotics and Automatation, Vol. 5, No. 6:804819.

Soft Data Fusion in Image Processing

21

4. W.G.K. Backhaus, R. Kliegl and J.S. Werner eds. (1998). Color Vision: Perspectives from Dierent Disciplines. Berlin: Walter de Gruyter. 5. G. Beliakov (2000) Aggregation operators as similarity relations. In Information, Uncertainty and Fusion, B. Bouchon-Menier et al. eds. Boston: Kluwer Academic Publishers. 6. S. Beucher (1982). Watersheds of functions and picture segmentation. IEEE Int. Conf. on Acoustics, Speech and Signal Processing, Paris, 1928-1931. 7. L. Bogoni (2000). Extending Dynamic Range of Monochrome and Color Images through Fusion. Proc. Int. Conf. Pattern Recognition, ICPR'2000, Vol. 3: 7-12. 8. A. Elfes (1992). Multi-source Spatial Data Fusion Using Bayesian Reasoning. In [1]: 137-164. 9. A. Filippidis, L.C. Jain, N. Martin (2000). Fusion of Intelligent Agents for the Detection of Aircraft in SAR Images. IEEE Trans. Pattern Analysis and Machine Intelligence, Vol. 22, No. 4: 378-384. 10. M. Grabisch, H.T. Nguyen and E.A. Walker (1995). Fundamentals of Uncertainty Calculi with Applications to Fuzzy Inference, Kluwer Ac. Pub. 11. M. Grabisch (1997). Fuzzy Measures and Integrals for Decision Making and Pattern Recognition. Fuzzy Structures: Current Trends (R. Mesiar et al. eds.), TATRA MOUNTAINS Mathematical Publications. 12. S.A. Hutchinson, A.C. Kak (1992). Multisensor Strategies Using DempsterShafer Belief Accumulation. In [1]: 165-209. 13. G.J. Klir, Z. Wang and D. Harmanec (1997). Constructing Fuzzy Measures in Expert systems. Fuzzy sets and Systems, 92: 251-264. 14. M. Koppen, C. Nowack and G. Rsel (1999). Pareto-Morphology for Color Image Processing. Proc. of the 11th Scandinavian Conference in Image Analysis, Greenland, Denmark. 15. M. Koppen, K. Franke, O. Unold (2000). A survey on fuzzy morphology. Proc. PRIA-5: 424-427, Samara, Russia. 16. R. Krishnamoorti and P. Bhattacharya (1998). Color Edge Extraction Using Orthogonal Polynomials Based Zero Crossings Scheme. Information Sciences, 112, 51-65. 17. H. Li, B.S. Manjunath and S.K. Mitra (1995). Multisensor Image Fusion Using the Wavelet Transform. Graphical Models and Image Processing, 57 (3): 235-245. 18. R.C. Luo and M.G. Kay eds. (1995). Multisensor Integration and Fusion for Intelligent machines and systems. Norwood, NJ: Ablex Publishing Corporation. 19. G. Medioni et a. (2001). Event Detection and Analysis from Video Streams. IEEE Trans. Pattern Analysis and Machine Intelligence, Vol. 23, No. 8: 873-889. 20. Y. Miyamoto et al. (1996). Development of `AI-VISION' for uidized-bed incinerator. Proc. IEEE/SICE/RSJ Int. Conf. Multisensor Fusion and Integration for Intelligent Systems: 72-77. 21. T. Murofushi and M. Sugeno (1991). Fuzzy t-conorm integrals with respect to fuzzy measures: generalization of Sugeno integral and Choquet integral. Fuzzy Sets and Systems, 42: 57-71. 22. N. Nandhakumar (1994). Robust physics-based analysis of thermal and visual imagery. Journal of the Opt. Soc. Am. A, 1994: 2981-2989. 23. S.-C. Pei and C.-M. Cheng (1999). Color Image Processing by Using Binary Quaternion-Moment-Preserving Thresholding Technique. IEEE Trans. On Image Processing, 8 (5) 614-629.

22

Aureli Soria-Frisch

24. J.L. Pech-Pacheco et al. (2000). Diatom autofocusing in bright eld microscopy: a comparative study. Proc. Int. Conf. Pattern Recognition, ICPR'2000, Vol. 3: 318-325. 25. J. Porrill (1988). Optimal Combination and Constraints for Geometrical Sensor Data. Int. J. of Robotics Research, Vol. 7, No. 6: 66-77. 26. H. Quiu, J. Keller (1987). Multispectral image segmentation using fuzzy techniques. Proc. North American Fuzzy Information Processing Society, May 1987: 374-387. 27. R.A. Salinas, C. Richardson, M.A. Abidi and R.C. Gonzalez (1996). Data Fusion: Color Edge Detection and Surface Reconstruction Through Regularization. IEEE Trans. on Industrial Electronics, 43(3): 355-363. 28. J. Ruiz-del-Solar and A. Soria-Frisch (2000). Bio-inspired color vision for the fusion of chromatic, infrared and textural image information. Proc. 2nd International ICSC Symposium on Neural Computation NC2000: 786-792, Berlin, Germany. 29. A. Soria-Frisch (2000). Intelligent Localized Fusion Operators for Color Edge Detection. Proc. 12th Scandinavian Conference on Image Analysis, SCIA 2001: 177-184, Bergen, Norway. 30. A. Soria-Frisch (2001). A New Paradigm for Fuzzy Aggregation in Multisensory Image Processing. In Computational Intelligence: Theory and Applications. Proc. Int. Conf. 7th Fuzzy Days: 59-67, Dortmund Germany. 31. A. Soria-Frisch and M. Koppen (2001). Fuzzy Color Morphology based on the Fuzzy Integral. In Proc. International ICSC Congress on Computational Intelligence: Methods and Applications, CIMA'2001: 732-737, Bangor, Wales, United Kingdom. 32. A. Soria-Frisch (2002). Avoidance of Highlights through ILFOs in Automated Visual Inspection. To appear in edited volume Fuzzy Filters for Image Processing of the International Series Studies in Fuzziness and Soft Computing, Heidelberg: Springer Verlag. 33. M. Sugeno (1974). Theory of Fuzzy Integral and its applications. Ph.D. thesis. 34. P. Sussner (2000). Observations on morphological associative memories and the kernel method. Neurocomputing 31: 167-183. 35. H. Tahani and J. Keller (1990). Information Fusion in Computer Vision Using the Fuzzy Integral. IEEE Trans. Systems, Man and Cybernetics, 20 (3): 733-741. 36. H.R. Tizhoosh (1998). Fuzzy Bildverarbeitung. Heidelberg: Springer-Verlag (in german). 37. P.E. Trahanias, I. Pitas and A.N. Venetsanopoulus (1994). Color Image Processing. Control and Dynamic Systems, Nr. 67: Digital Image Processing. Academic Press. 38. Z. Wang and G.J. Klir (1992). Fuzzy Measure Theory, Plenum Press. 39. S. Weber (1984). ?-Decomposable measures and integrals for Archimidean tconorms ?. J. Mathematical Analysis and Applications, Vol. 101: 114-138. 40. P.Weckesser, R.Dillmann (1996). Sensor-Fusion of Intensity- and LaserrangeImages. Proc. IEEE/SICE/RSJ Int. Conf. Multisensor Fusion and Integration for Intelligent Systems: 501-508. 41. R.R. Yager and A. Kelman (1996). Fusion of Fuzzy Information With Considerations for Compatibility, Partial Aggregation, and Reinforcement. Int. J. of Approximate Reasoning, 15:93-122.