Soft Data Fusion for Computer Vision

Fraunhofer Institut f¨ ur Produktionsanlagen und Konstruktionstechnik Department Security Technology

Von der Fakultät für Verkehrs- und Maschinensysteme der Technischen Universität Berlin zur Erlangung des akademischen Grades Doktor der Ingenieurwissenschaften – Dr.-Ing. – genehmigte Dissertation

Soft Data Fusion for Computer Vision vorgelegt von Diplom-Ingenieur Aureli Soria-Frisch

Tag der wissenschaftlichen Aussprache: 1. September 2004 Promotionsausschuss Vorsitzender: Prof. Dr.-Ing. G. Seliger Gutachter: Prof. Dr.-Ing. J. Krüger Prof. Dr. F. Tarrés (UPC) Berlin 2004 D83

i

Preface The dissertation presented herein was realized at the Department Security Technologies of the Fraunhofer IPK, Berlin, where I have been working as research engineer for the last six years. I would like to thank first Prof. Dr.-Ing. Jörg Kr¨ uger, head of the Research Area Automation of the Fraunhofer IPK, for the interest in the dissertation, which made him take over its supervision in a very advanced state of the research works. His advice has helped me successfully completing this research enterprise. My acknowledge is specially devoted to Prof. Dr. Francesc Tarrés from the Polytechnic University of Catalonia (UPC), who reviewed the dissertation and took part in my final examination. His support was essential for me both from a professional and a personal perspective. I would like also to thank Dr.-Ing. Bertram Nickolay, head of the Department Security Technologies, for its support all over the promotion time. He contributed with several suggestions to the significance of the dissertation. The discussions and exchange with the Research Engineers and Students, who currently work or have worked at the Department, enormously enriched my perspective on the field of Computer Vision. Specially Dipl.-Phys. M. Köppen, Dipl.-Phys. D. Kottow, Dr.-Ing. Lutz Lohmann, Dipl.-Inform. Ch. Nowack, Dipl.-Inform. Ch. Veenhenius, Dipl.-Ing. R. Vicente-Garc´ıa left its mark on different aspects of the dissertation. In this context, I would like to specially thank Prof. Dr.-Ing. J. Ruiz-del-Solar for the passionate discussions, the review of the manuscript, and his faithful and friendly support. Finally, I would like to thank all these friends and family members who had made this path with me, supporting me in the bad moments and sharing with me the good ones. Both from a spiritual and practical perspective, this work would have not been successfully realized without their assistance. Berlin, September 1, 2004 Aureli Soria-Frisch

ii

iii

Contents

List of Acronyms

vii

Nomenclature

xi

List of Figures

xiii

List of Tables

xix

List of Algorithms

xxi

1 Introduction

1

2 Multi-sensory Computer Vision 2.1 Machine-based Pattern Recognition . . . . . . . . . . . . . . . . . 2.2 Computer Vision Systems . . . . . . . . . . . . . . . . . . . . . . 2.2.1 General structure of computer vision systems . . . . . . . 2.2.2 Imaging as measuring process on the electromagnetic spectrum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Integration and fusion of multi-sensory information . . . . . . . . 2.3.1 Integration vs. fusion . . . . . . . . . . . . . . . . . . . . . 2.3.2 Information fusion in Computer Vision . . . . . . . . . . . 2.3.3 Ugly Duckling Theorem and the usage of Soft Computing for information fusion . . . . . . . . . . . . . . . . . . . . . 2.4 Soft Computing for data processing . . . . . . . . . . . . . . . . . 2.5 Fuzzy Fusion Operators . . . . . . . . . . . . . . . . . . . . . . . 2.5.1 Fuzzy connectives: T- and S-Norms . . . . . . . . . . . . . 2.5.2 Fuzzy Integrals . . . . . . . . . . . . . . . . . . . . . . . . 2.6 Utilization of the Fuzzy Integral in Computer Vision . . . . . . . 2.7 Construction of the Fuzzy Measures . . . . . . . . . . . . . . . . . 2.7.1 Non-automated assessment of coefficients . . . . . . . . . . 2.7.2 Automated assessment of coefficients . . . . . . . . . . . . 2.8 Soft Computing methodologies for the construction of Fuzzy Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.8.1 Genetic Algorithms . . . . . . . . . . . . . . . . . . . . . .

3 3 4 6 7 12 13 15 18 20 23 23 24 28 29 29 30 32 32

iv

Contents 2.8.2

Neural based strategies . . . . . . . . . . . . . . . . . . . .

34

3 Dissertation’s motivation and framework definition 3.1 Problem statement . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Goals to be attained in the here presented work . . . . . . . . . . 3.3 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

37 37 38 39

4 Development of a Computer Vision System with Soft Data Fusion 4.1 New Model for Multi-sensory Fusion . . . . . . . . . . . . . . . . 4.2 Soft Data Fusion . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Engineering with the Fuzzy Integral . . . . . . . . . . . . . . . . . 4.3.1 The practical application of the Fuzzy Integral . . . . . . . 4.3.2 The Fuzzy Integral in Computer Vision . . . . . . . . . . . 4.4 Intelligent Localized Fusion Operators . . . . . . . . . . . . . . . 4.4.1 ILFOs’ mathematical foundations . . . . . . . . . . . . . . 4.4.2 ILFOs’ Framework . . . . . . . . . . . . . . . . . . . . . . 4.5 SC Methodologies for the Construction of Fuzzy Measures Revisited 4.5.1 Genetic Algorithms . . . . . . . . . . . . . . . . . . . . . . 4.5.2 Interactive Determination . . . . . . . . . . . . . . . . . . 4.5.3 SOM based Strategies . . . . . . . . . . . . . . . . . . . .

41 41 44 48 48 53 58 59 62 66 66 67 69

5 Evaluation of soft data fusion procedures in Image Processing 5.1 Edge fusion for color edge detection . . . . . . . . . . . . . . 5.1.1 Application on standard color images . . . . . . . . . 5.1.2 Application on satellite images . . . . . . . . . . . . . 5.2 Pre-processing for highlights filtering . . . . . . . . . . . . . 5.2.1 Pre-processing for visualization . . . . . . . . . . . . 5.2.2 Pre-processing for fault detection . . . . . . . . . . . 5.3 Color morphology for textile image processing . . . . . . . . 5.4 Document image processing through the fuzzy integral . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

75 75 78 81 84 87 97 99 103

6 Evaluation of soft data fusion procedures in Image Analysis 109 6.1 SOM - Fuzzy Integral hybrid system for image segmentation . . . 109 6.1.1 Framework for the segmentation of multi-dimensional images110 6.1.2 Segmentation of benchmark color images . . . . . . . . . . 113 6.1.3 Image segmentation in a market basket recognition problem 118 6.2 Automated visual inspection of collagen plates . . . . . . . . . . . 124 6.2.1 Framework for perceptual relevance evaluation . . . . . . . 125 6.2.2 Network of fuzzy aggregation operators for feature analysis 129 6.2.3 Perceptual relevance evaluation on collagen plates . . . . . 134

v

Contents 7 Conclusions 7.1 Multi-sensory computer vision systems and fuzzy fusion operators 7.2 Intelligent multi-sensory fusion, soft data fusion, and the fuzzy integral . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3 Soft data fusion for image processing: results evaluation . . . . . . 7.3.1 Color edge detection . . . . . . . . . . . . . . . . . . . . . 7.3.2 Highlights filtering . . . . . . . . . . . . . . . . . . . . . . 7.3.3 Color morphology . . . . . . . . . . . . . . . . . . . . . . . 7.3.4 Image segmentation for document analysis . . . . . . . . . 7.4 Soft data fusion for image analysis: results evaluation . . . . . . . 7.4.1 Multi-dimensional image segmentation . . . . . . . . . . . 7.4.2 Perceptual relevance evaluation . . . . . . . . . . . . . . .

139 139

8 Summary and projections

147

141 143 143 143 144 144 145 145 146

A Fuzzy Measures Theory 151 A.1 Fuzzy Logic vs. Fuzzy Measures . . . . . . . . . . . . . . . . . . . 151 A.2 Axiomatic Definition of Fuzzy Measures . . . . . . . . . . . . . . 153 A.3 Different Measures for the Treatment of Uncertainty . . . . . . . . 155 B Properties of the Choquet and Sugeno Fuzzy Integrals B.1 Generalization of other fusion operators . . . . . . . . B.2 Canonical regions of the hyper-cube . . . . . . . . . . B.3 Level curves . . . . . . . . . . . . . . . . . . . . . . . B.4 Distributivity with respect to the scalar product . . . B.5 Monotonicity with respect to the integrands . . . . . B.6 Monotonicity with respect to the fuzzy measures . . . B.7 Range of results . . . . . . . . . . . . . . . . . . . . . B.8 Fusion operators as measures of similarity . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

157 157 157 158 160 160 160 161 161

C Examplary classification of data with gaussian distributions

163

D Multi-sensory Fusion in Biological Systems D.1 Biological and Cognitive Perspective on Information Fusion D.2 Information processing in the human brain . . . . . . . . . D.2.1 Principles of Information Processing . . . . . . . . D.2.2 Neuronal coding . . . . . . . . . . . . . . . . . . . . D.2.3 Brain Maps . . . . . . . . . . . . . . . . . . . . . . D.2.4 Evolutionary Processes . . . . . . . . . . . . . . . . D.3 Multi-sensory fusion at different organizational levels . . . D.3.1 Cognitive level . . . . . . . . . . . . . . . . . . . . D.3.2 Systemic level . . . . . . . . . . . . . . . . . . . . . D.3.3 Neuronal level . . . . . . . . . . . . . . . . . . . . .

171 171 173 174 175 177 179 179 180 183 186

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

vi

Contents D.4 Fuzzy Measures Theory: Framework for cognitive information fusion188 D.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191

Bibliography

193

vii

List of Acronyms A

Auditory

A/D

Analogue/Digital

ATR

Automatic Target Recognition

Binar

Module for Binarization

BPPM

Binary Pattern Processing Modules

CCA

Connected Component Analysis

CCD

Charge-Coupled Device

CFI

Choquet Fuzzy Integral operator

Class

Module for Classification

CLOP

Classical Fusion Operators

CMOS

Complementary Metal Oxide Semiconductor

CNS

Central Nervous System

Codif

Codification

coeffs

Coefficients

COG

Center of Mass

Cooc

Coocurrence Matrix

CA

Cornu Amoni

DecMak

Module for Decision Making

DG

Dentate Gyrus

DetFM

Module for the Determination of Fuzzy Measures

viii

List of Acronyms

DigImAcq

Module for Digital Image Acquisition

EC

Evolutionary Computing

EDet

Module for Edge Detection

EFT

Error in Fault Type

EFus

Module for Edge Fusion

ENT

Entorhinal Cortex

ExpSys

Module implementing an Expert System

FA

False Acceptance

FC

Fuzzy Computing

FeatExt

Module for Feature Extraction

FuzFus

Module implementing a Fuzzy Fusion operator

FI

Fuzzy Integral

FuzInt

Module implementing a Fuzzy Integral

FM

Fuzzy Measure

FR

False Rejectance

Ft-cI

Fuzzy t-conorm Integral

FuzExpSys

Module implementing a Fuzzy Expert System

FuzMeCo

Module for the Construction of Fuzzy Measures

Fuzzif

Module for Fuzzification

GA

Genetic Algorithm

Gen

Number of Generations

HSI

Hue Saturation Intensity color model (or its components)

IC

Immunological Computing

ILF

Intelligent Localized Fusion paradigm

ILFO

Intelligent Localized Fusion Operator

ImAcq

Module for Image Acquisition

ix ImEnh

Module for Image Enhancement

ImTrans

Module for Image Transformation

LabImGen

Module for the Generation of a Label Image

LPS

Linear Pixel Shuffling

LVQ

Learning Vector Quantization

MAX

Maximum

MED

Median

MIN

Minimum

MSE

Mean Square Error

MType

Fuzzy Measure Type

NC

Neurocomputing

Op.

Operator

OWA

Ordered Weighted Averaging operator

PC

Probabilistic Computing

PCA

Principal Component Analysis

PCross

Probability of Crossover

PMut

Probability of Mutation

POLIMOD.

Polimodal

PPATH

Perforant Path

PRepl

Probability of Replacement

PROD

Product

RGB

Red Green Blue color model (or its components)

SAR

Synthetic Aperture Radar

SC

Soft Computing

Seg

Module for Image Segmentation

SFI

Sugeno Fuzzy Integral operator

x

List of Acronyms

SFX

Sensor Fusion Effects framework

S-norm

Triangular conorm

SOM

Self-Organizing Feature Map

T-norm

Triangular norm

UNIMOD.

Unimodal

V

Visual

WMAX

Weighted Maximum

WMED

Weighted Median

WMIN

Weighted Minimum

xi

Nomenclature A(i)

Set on the components of vector x formed after a sorting operation, A(i) = {x(1) , . . . , x(i) }

Cj

Class j

Cµ (x1 , . . . , xn ) Choquet Fuzzy Integral of xi ∀i ∈ [1, n] with respect to fuzzy measure µ Fµ (x1 , . . . , xn ) Fuzzy Integral of xi ∀i ∈ [1, n] with respect to fuzzy measure µ h(x)

Fuzzy membership function of x

λ

Additivity coefficient in a fuzzy-λ measure

µ

Fuzzy measure, i.e. set of coefficients of a fuzzy measure

µj

Fuzzy measure of class j

µi

Fuzzy density on xi , i.e. fuzzy measure coefficient on a set of cardinality one

µi...j

Fuzzy measure coefficient on set {xi , . . . , xj }

µji

Fuzzy density of class j on xi

µ(A)

Fuzzy measure coefficient on set A

OWA

Ordered Weighted Averaging operator

P(X)

Power set of vector x

P (A)

Probability measure on set A

T (x, y)

T-norm of x and y

S(x, y)

S-norm of x and y

Sµ (x1 , . . . , xn ) Sugeno Fuzzy Integral of xi ∀i ∈ [1, n] with respect to fuzzy measure µ

xii

Nomenclature

x(i)

Component of vector x with rank i after sorting operation

∨

Maximum operator

∧

Minimum operator

xiii

List of Figures 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 2.10 2.11 2.12 2.13 2.14

Sub-disciplines of Computer Vision . . . . . . . . . . . . . . Block diagram of a generic computer vision system. . . . . . Spectral sensitivity bands of different materials. . . . . . . . Grayvalue images acquired by a CCD and a CMOS camera. Color image acquisition architectures. . . . . . . . . . . . . . Images in the visible and the infrared spectral band. . . . . . Images in the visible and the ultrasound spectral band. . . . Exemplary hyper-spectral image. . . . . . . . . . . . . . . . Integrated, digital, and intelligent sensors. . . . . . . . . . . Block diagram of a generic multi-sensory integration system. Pictorial description of the Ugly Duckling Theorem. . . . . . The cognitive metaphor of soft computing systems. . . . . . Generalization relationship between fuzzy fusion operators. . Flow diagram of a generic genetic algorithm. . . . . . . . . .

. . . . . . . . . . . . . .

5 6 8 8 9 10 11 12 13 14 19 22 27 33

3.1

Goals attained in the here presented dissertation. . . . . . . . . .

39

4.1

Functionalities of multi-sensory integration undertaken by an intelligent multi-sensory fusion procedure. . . . . . . . . . . . . . . Relational map of different fusion operators. . . . . . . . . . . . . Exemplary fusion of a discrete signal with different operators. . . Lattice structure of a fuzzy measure. . . . . . . . . . . . . . . . . Exemplary operation of the iterative algorithm of the SFI (part I). Exemplary operation of the iterative algorithm of the SFI (part II). Discriminating capability of fuzzy measures. . . . . . . . . . . . . Exemplary result of the Choquet and Sugeno Fuzzy Integrals. . . Block diagram of a generic classification system based on the fuzzy integral in its possibilistic approach. . . . . . . . . . . . . . . . . . Block diagram of a generic classification system based on the fuzzy integral in its information fusion approach. . . . . . . . . . . . . . Exemplary usage of an Intelligent Localized Fusion Operator. . . . Relationship among feature sets and levels in the lattice structure of a fuzzy measure. . . . . . . . . . . . . . . . . . . . . . . . . . . Set diagram for the generation of a label image in ILFOs. . . . . .

4.2 4.3 4.4 4.5 4.6 4.7 4.8 4.9 4.10 4.11 4.12 4.13

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

43 45 46 49 51 52 53 54 56 56 59 60 61

xiv

List of Figures 4.14 4.15 4.16 4.17 4.18 4.19 4.20 4.21 4.22 5.1 5.2 5.3 5.4 5.5 5.6 5.7 5.8 5.9 5.10 5.11 5.12 5.13 5.14 5.15

5.16 5.17 5.18 5.19

Framework of a generic ILFO. . . . . . . . . . . . . . . . . . . . . Block diagram for the generation of a crisp label image in ILFOs. Block diagram for the generation of a fuzzy label image in ILFOs. Tolerance parameter ǫ in the fuzzification of the feature images in ILFOs with fuzzy label images. . . . . . . . . . . . . . . . . . . . Flow diagram of a generic interactive genetic algorithm. . . . . . . Inspiring employment of a SOM for the analysis of feature saliency. Exemplary U- and hit-matrices of a SOM. . . . . . . . . . . . . . Pictorial description of the operation for the extraction of fuzzy measures from a SOM. . . . . . . . . . . . . . . . . . . . . . . . . Pictorial description of the geodesic erosion. . . . . . . . . . . . . Block diagram of the presented framework for color edge detection. Haar Wavelet. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . S-shape membership function. . . . . . . . . . . . . . . . . . . . . Color edge detection with the SFI on the splash image. . . . . . . Color edge detection with SFI on the lenna image. . . . . . . . . . Color edge detection with SFI on the house image. . . . . . . . . Result of the fusion through a SFI-based ILFO, where the avoidance of shadow false edges is achieved. . . . . . . . . . . . . . . . Satellite image used in the detection of edge maps and result of the applied color space transformation. . . . . . . . . . . . . . . . Results of a Sugeno-based ILFO on satellite images. . . . . . . . . Results of a Choquet-based ILFO on satellite images. . . . . . . . Block diagram of the presented pre-processing system for highlights filtering through the application of ILFOs. . . . . . . . . . . Fuzzy membership functions for image quality assessment. . . . . Input images of the pre-processing stage of a system for the analysis of chocolate package completeness. . . . . . . . . . . . . . . . Results of the pre-processing stage with different fusion strategies in a system for the analysis of chocolate package completeness. . . Input images of the pre-processing stage of a system for the detection of structural faults on automobile headlamp reflectors (item I). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Results of the pre-processing stage of a system for the detection of structural faults on automobile headlamp reflectors (item II). . . . Results of the utilization of interactive genetic algorithms for the construction of fuzzy measures. . . . . . . . . . . . . . . . . . . . Exemplary result of the computation of the peak dynamics on histograms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Results of the binarization, the label image generation, and the fuzzy label image generation after automated determination of thresholds. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

63 63 64 65 68 69 70 72 72 76 77 77 78 79 80 81 82 83 83 85 86 87 88

88 89 90 92

93

List of Figures 5.20 Convergence of the GA over different generations in the computation of the fuzzy measure coefficients of an ILFO. . . . . . . . . . 5.21 Final results of an automated pre-processing system, where the fuzzy measures are constructed after a GA. . . . . . . . . . . . . . 5.22 Estimation of the dependence of the execution time of an ILFO based on the CFI. . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.23 Effect of the tolerance variation (ǫ) on the final results of the automated pre-processing system. . . . . . . . . . . . . . . . . . . . 5.24 Input images of the pre-processing stage in a system for the detection of structural faults on automobile headlamp reflectors (item II). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.25 Final results of the automated pre-processing system (item II). . . 5.26 Input images of the pre-processing stage in a system for the detection of structural faults on halogen bulbs. . . . . . . . . . . . . . . 5.27 Results of the pre-processing stage with different fusion strategies in a system for the detection of structural faults on halogen bulbs. 5.28 Results of targeted morphological operations with a cross mask on a random color image. . . . . . . . . . . . . . . . . . . . . . . . . 5.29 Results of basic color morphological operations for the automated visual inspection of textiles. . . . . . . . . . . . . . . . . . . . . . 5.30 Results of a color morphological gradient (item I). . . . . . . . . . 5.31 Results of a color morphological gradient (item II). . . . . . . . . 5.32 Result of the SFI in the segmentation of archive card color images. 5.33 Results of the CFI in the segmentation of seals. . . . . . . . . . . 5.34 Results of the SFI in the segmentation of seals on a post letter. . 5.35 Quantitative performance evaluation of the framework for seal segmentation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1

6.2 6.3 6.4 6.5 6.6 6.7

xv

94 95 96 96

97 97 98 98 100 101 101 102 103 104 105 106

Block diagram of the presented framework for the segmentation of multi-dimensional images within the information fusion theoretical framework. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 Unsupervised segmentation results on the peppers image in a 5dimensional color feature space. . . . . . . . . . . . . . . . . . . . 113 Unsupervised segmentation results on the peppers image in a 7dimensional color feature space. . . . . . . . . . . . . . . . . . . . 114 Unsupervised segmentation results on the peppers image in a 9dimensional color feature space. . . . . . . . . . . . . . . . . . . . 115 Supervised segmentation results on the peppers image in a 9-dimensional color feature space. . . . . . . . . . . . . . . . . . . . . . . . . . . 115 Unsupervised segmentation results on the baboon image in a 5dimensional feature space. . . . . . . . . . . . . . . . . . . . . . . 116 Unsupervised segmentation results on the baboon image in a 5dimensional feature space for different fuzzification functions. . . . 117

xvi

List of Figures 6.8 6.9 6.10 6.11 6.12 6.13 6.14 6.15 6.16 6.17 6.18 6.19 6.20 6.21 6.22

Unsupervised segmentation results on the baboon image in a 9dimensional feature space. . . . . . . . . . . . . . . . . . . . . . . 118 Supervised segmentation results on the baboon image in a 9-dimensional feature space. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 Unsupervised segmentation applied for market basket recognition (item I). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 Unsupervised segmentation applied for market basket recognition (item II). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 Unsupervised segmentation applied for market basket recognition (item III). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 Unsupervised segmentation applied for market basket recognition (item IV and V). . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 New paradigm for automated visual inspection: from detection to interpretation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 Framework for perceptual relevance evaluation. . . . . . . . . . . . 126 Block diagram of a Binary Pattern Processing Module. . . . . . . 127 Auto-Lookup based procedure. . . . . . . . . . . . . . . . . . . . . 128 Hierarchically organized network of fusion operators used for the fuzzy evaluation of binary images. . . . . . . . . . . . . . . . . . . 130 Measuring of binary patterns on connected components through classical fusion operators. . . . . . . . . . . . . . . . . . . . . . . . 131 Exemplary employment of OWAs in the computation of fuzzy metafeatures. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132 Results of collagen plate inspection for a training and a test data sets. Choquet Fuzzy Integral with respect to fuzzy-λ measures. . 136 Results of collagen plate inspection for a training data set. Comparison of Choquet Fuzzy Integral with respect to fuzzy-λ and general fuzzy measures. . . . . . . . . . . . . . . . . . . . . . . . . 137

A.1 Fuzzy membership functions. . . . . . . . . . . . . . . . . . . . . . 152 A.2 Types of fuzzy measures. . . . . . . . . . . . . . . . . . . . . . . . 156 B.1 B.2 B.3 B.4

Canonical regions of two- and three-dimensional hyper-cubes. . Level curve of the Sugeno Fuzzy Integral. . . . . . . . . . . . . Level curve of the Choquet Fuzzy Integral. . . . . . . . . . . . Ideal and Anti-Ideal in the hyper-cube. . . . . . . . . . . . . .

. . . .

. . . .

159 159 160 161

C.1 C.2 C.3 C.4 C.5 C.6

Classification Classification Classification Classification Classification Classification

. . . . . .

. . . . . .

165 166 167 168 169 170

of of of of of of

gaussian gaussian gaussian gaussian gaussian gaussian

clusters. clusters. clusters. clusters. clusters. clusters.

Case Case Case Case Case Case

I. . II. . III. IV. V. . VI.

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

xvii

List of Figures D.1 Neural representation through coarse vector coding. . . D.2 Neural representation through rank order coding. . . . D.3 Examplary brain maps. . . . . . . . . . . . . . . . . . . D.4 Evolutionary process in a cortical map. . . . . . . . . . D.5 Different sensory transducers in human beings. . . . . . D.6 Structural levels of organization in the nervous system. D.7 Different areas of the cerebral cortex. . . . . . . . . . . D.8 System diagram around the superior colliculus. . . . . . D.9 System diagram around the hippocampus. . . . . . . . D.10 Life-long learning schematic representation. . . . . . . . D.11 Schematic representation of an integrating neuron. . . . D.12 Multisensory enhancement at neuronal level. . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

175 177 178 180 181 182 184 185 185 186 187 189

xviii

xix

List of Tables 2.1

Basic T-norms and their dual S-norms. . . . . . . . . . . . . . . .

5.1

Type and parameterization of different interactive genetic algorithms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 Best parameterization of a steady-state genetic algorithm for the construction of a fuzzy measure for different fitness functions fi . . 92 Performance evaluation of framework for color seal extraction. . . 107

5.2 5.3 6.1 6.2

24

Statistical comparison of results for the inspection of collagen plates obtained through general and fuzzy-λ measures. . . . . . . . . . . 137 Statistical analysis of the results obtained by the automated industrial system for the inspection of collagen plates. . . . . . . . . 138

B.1 Generalization relationships between the fuzzy integral with respect to a particular fuzzy measure and other fuzzy aggregation operators. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158

xx

xxi

List of Algorithms 1 2 3

Iterative algorithm for the computation of the fuzzy integral on an image. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Morphological clustering of the U-matrix based on reconstruction by erosion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Targeted dilation operator of a color morphology. . . . . . . . . .

50 73 99

xxii

1

1 Introduction The relevance of information fusion methodologies increases due to the complementary development of computer and sensory technologies. Information fusion basically attains the transformation of the information delivered by multiple sources into one representational form [1]. The fused data does not only reflect information that can be extracted from the individual sources but also information not derivable from any of them on its own [1]. Such an information gain characterizes the purpose of information fusion. Operator research in the context of fuzzy systems has generated a fruitful set of aggregation operators [62], e.g. fuzzy connectives [82], weighted ranking operators [34][188], Ordered Weighted Averaging (OWA) operators [187], Fuzzy Integrals [163]. Such operators constitute a flexible means for the fusion of information represented by fuzzy membership functions. So-called fuzzy aggregation operators constitute a flexible alternative to operators traditionally used in information fusion. The concept of fuzzy integral is due to Sugeno, who presented in [163] a mathematical approach for the simulation of multi-criteria evaluation taking into consideration some cognitive aspects. Sugeno’s hypothesis is that the process of multi-criteria integration undertaken by human beings subsumes the linear combination of the different criteria with numerically expressed priorities, i.e. weighting sum strategy. Thus fuzzy measures, which generalize classical measures through the consideration of subjectiveness, were established in order to quantify the a priori importance of the integrands. A new type of measure led to a new integral operator: the fuzzy integral. The elements that make the mathematical expression of the fuzzy integral support the hypothesis posited by Sugeno are: the utilization of fuzzy membership functions as integrands, their weighting through fuzzy measures, and the binding of these two elements through a combination of T- and S-norms, which are the fuzzy connectives used in fuzzy logic [82]. Due to its relationship with cognitive processes and to its positive features as fusion operator, the fuzzy integral is employed in different application fields, where Decision Making [58] [81] and Subjective Evaluation [114] [168] [181] represent the most natural ones. Furthermore fuzzy integrals were used in Computer Vision problems, both on Image Processing and Image Analysis [80] [166], in a very early stage of research. In this case the fuzzy integral is mainly used because of its mathematical properties as fusion operator. The successful employment of the fuzzy integral in real applications depends

2

1 Introduction

on the existence of standard methodologies for the automated construction of fuzzy measures. The automated completion of this process is compared by some authors with the development of procedures for the automated determination of fuzzy membership functions in the fuzzy sets theoretical framework [83]. The appearance of a wide palette of methodologies for the construction of fuzzy membership functions encouraged the feasible application of Fuzzy Set Theory in a great number of application fields. On the contrary, there is a lack of efficient methods for constructing fuzzy measures [83], what hinders the extensive usage of the fuzzy integral in real applications. In this context, the application of the fuzzy integral in Computer Vision with information fusion does not constitute an exception. The here presented dissertation attains two main goals. First, it presents different computer vision applications, where the fuzzy integral is used as fusion operator, based on the further development of different aspects of the theoretical framework of information fusion through fuzzy aggregation operators. Second, it analyzes different Soft Computing methodologies for the automated construction of fuzzy measures. In spite of the flexibility, robustness, and interpretability that the fuzzy integral presents when being used as fusion operator, few information fusion applications, specially in Computer Vision, are based on it. This may lay on the complex theoretical background [62] [179] and on the lack of successful implementations of the methodology. Therefore the dissertation brings the fuzzy integral from a mathematical domain to the engineering domain. This goal is achieved in different steps. First, a common theoretical framework for all fuzzy fusion operators, which is denoted as Soft Data Fusion, is developed. The fuzzy integral, which is the main operator analyzed, is further develop from a theoretical point of view by taking into consideration some specific features of its application in Image Processing. Furthermore different processing frameworks, which go beyond the application of the fuzzy integral on its own, are developed. These frameworks are eventually applied on different computer vision applications and the obtained results evaluated. The second goal of the dissertation is achieved by developing different methodologies for the automated parameterization of the fuzzy integral. The development of feasible methodologies for the construction of fuzzy measures in real applications is based on the application of the research field of Soft Computing. In this context Soft Computing methodologies present the advantage of being data-driven, what facilitates the implementation of full automated systems for information fusion based on the fuzzy integral. Moreover the positive features of Soft Computing for the resolution of applications in Computer Vision encourage its application for the automated determination of the fuzzy measure coefficients as well. Neurocomputing and Evolutionary Computing are the paradigms selected for the resolution of this problem in the here presented dissertation.

3

2 Multi-sensory Computer Vision Computer Vision is a research field, where different technologies flow together in order to attain the final goal of making a computer “see”. Pattern recognition and signal processing technologies are the basic fields, from which computer vision evolve. These technologies together with the spread availability of imaging sensors led to the rapid development of this research field. Nevertheless computer vision presents no unified approach on aspects concerning nomenclature and formal definitions. This chapter presents the author’s perspective on the field. First different concepts on the field of Pattern Recognition are presented from a formal point of view. Computer vision systems are thence analyzed by taking into consideration their components and defining the associated research fields. In this context, the process of image acquisition is treated in detail because of its relationship with multi-sensory computer vision systems. The theoretical concepts related with the employment of different sensory units in a computer vision system are described. Next the different building methodologies of Soft Computing are described and the possibilities for the implementation of so-called hybrid systems presented. The second part of the chapter is devoted to the state of the art on fuzzy aggregation operators for information fusion. Thus fuzzy connectives and fuzzy integrals are described. Thence different computer vision systems, which are based on the application of the fuzzy integral, are presented. The methodologies for the construction of fuzzy measures, where existent Soft Computing methodologies are especially taken into consideration, are then analyzed.

2.1 Machine-based Pattern Recognition The practice of the scientific discipline known as Pattern Recognition attains a two-fold goal. On the one hand, it tries to have machines reproduce some cognitive capabilities similar to that of human beings. On the other hand, it contributes to the understanding of recognition capabilities that are found in the natural world [38]. The usefulness of pattern recognition have to be seen in the extension (and not substitution) of human capabilities.

4

2 Multi-sensory Computer Vision

Pattern recognition has been defined as the process that leads to discovering structure in data [14]. In order for this process to be fulfilled the structure should be of interest, i.e. this structure has a meaning in the outer world that makes it interesting to be revealed. Thus the word structure has to be understood in the sense of meaningful regularity [37]. In this sense pattern recognition can be seen as a process transforming perception into categorization [55] [182]. Since machines work within a numerical domain, the mechanical implementation of the pattern recognition process is based on the concept of features [38], which can be considered the computational counterpart of predicates in the process of logical inference [182] (see Sec. 2.3.3). In this context pattern recognition systems are defined as a mapping between a measurement space M, and the decision space D via the feature space F [30], which can be expressed as: M → F → D.

(2.1)

It has been stated that pattern recognition and inductive inference share an essential functionality, namely inferring a generality from concrete cases [182]. This statement elucidates the concept of generalization, which is one of the crucial aspects in the implementation of pattern recognition systems [38]. Furthermore pattern recognition systems must take into consideration the effect of working in spaces of increasing dimensionality, what has been denoted as the curse of dimensionality [15]. A pattern recognition system undertakes a process of dimensionality reduction on a multi-dimensional feature space yielding to a decision [182]. In this sense it is important not to forget that the mathematical behavior of high-dimensional feature spaces is not well-known and difficult to understand [86]. Pattern recognition systems have to take into consideration the question of complexity. Thus some authors consider that Occam’s razor, i.e. a simple solution is preferred to a more complex one, applies in the context of pattern recognition as well [15] [38].

2.2 Computer Vision Systems The application of pattern recognition methodologies to the two-dimensional representation of data known as image constitutes one of the elements of Computer Vision. The objects of study in this discipline range from the acquisition of the data through an imaging device to the decision making process based on the information extracted from it (see Fig. 2.1). Therefore computer vision systems can be defined as a mapping by particularizing the measurement space M of the definition given above (2.1) through the image space I: I → F → D.

(2.2)

The study of computer vision methodologies are grouped in different subdisciplines, whose operational domain is depicted in Fig. 2.1. As it can be ob-

5

2.2 Computer Vision Systems Image Processing Image Acquisition IMAGE

Computer Graphics

Computer Vision

Image Analysis

IMAGE INFORMATION

Decision Making Image Information Processing

Figure 2.1: Application domain of the different sub-disciplines of Computer Vision (modified from [134]). The Image Acquisition sub-discipline groups methodologies for the implementation of the measuring process on the environment through imaging sensors. The methodologies grouped under the Image Processing sub-discipline operate on the image domain (I). Methodologies mapping this measurement space into the image information domain are studied in Image Analysis. Methodologies employed in the transformation of the acquired information are grouped into Image Information Processing. Finally Decision Making manages the information in order for the computer vision system to have an effect on the environment. The possible recursion of a computer vision system is completed by the methodologies grouped into Computer Graphics, which do not belong to Computer Vision.

served, computer vision methodologies can be classified within those operating in the image domain, which are grouped in the sub-discipline called Image Processing1 , those mainly operating in the feature domain, which are studied in Image Analysis2 , and those processing the image information extracted from it, which are included in Image Information Processing. The techniques grouped in the sub-disciplines Image Acquisition and Decision Making allow respectively entering and leaving the operational domain of Computer Vision. Finally it can be stated that computer vision systems combine image processing methodologies 1

Image Processing is described within the context of Computer Vision. Therefore the term excludes these parts of the discipline related to the transmission and codification of images. Computer Vision is related to Computer Science, while Image Processing including Image Coding and Transmission is related to an Electrical Engineering perspective. 2 Image Analysis is denoted as Image Understanding as well.

6


with pattern recognition ones (see Fig. 2.2), i.e. pattern recognition methodologies are used with other types of data as well.

2.2.1 General structure of computer vision systems The stages of a Computer Vision system have been traditionally structured by a processing chain that follows the sequential input-output structure of more general discrete-time systems [118]. This linear structure, which computer vision systems present from an scholastic point of view, e.g. [76], have been modified through the introduction of a feedback (see Fig. 2.2). This feedback indicates that the involved processes are sometimes repeated at decreasing levels of abstraction, which correspond to increasing levels of resolution3 . Pattern Recognition Image Processing DigImAcq

ImEnh

ImTrans

Image Analysis Seg

FeatExt

Image Information Processing Class

DecMak

Figure 2.2: Block diagram of a generic computer vision system. DigImAcq: Digital image acquisition. ImEnh: Image enhancement. ImTrans: Image transformation. Seg: Segmentation. FeatExt: Feature extraction. Class: Classification. DecMak: Decision making. Methodologies for the accomplishment of these different stages are studied in the computer vision sub-disciplines Image Processing, Image Analysis, and Image Information Processing as indicated in the figure (see Fig. 2.1 as well). Furthermore Image Analysis and Image Information Processing are part of the more general discipline of Pattern Recognition.

Any Computer Vision system presents as a first stage the acquisition of the image to be analyzed (DigImAcq), which constitutes the attained measure of the observed scene through an imaging sensor. Furthermore while image enhancement sub-systems (ImEnh) attain a general suppression of image features or elements that hinder the right visualization of the acquired image, image transformation ones (ImTrans) attain the goal-driven modification of the image, e.g. edge detection, color space transformation. After an image segmentation (Seg) task, the image remains divided in different image sub-domains that satisfy a particular condition of perceptual homogeneity. Furthermore the numerical characterization of different image elements through some vectorial features is achieved by 3

The concepts and diagram described in this section resulted from fruitful personal discussions with Prof. Dr.-Ing. Javier Ruiz-del-Solar (Universidad de Chile).

2.2 Computer Vision Systems

7

the feature extraction stage (FeatExt). Classification sub-systems (Class) attain the categorization [182] of the features, i.e. they divide the feature domain in different sub-domains, whose components satisfy an ideal homogeneity condition. Finally the decision making process (DecMak ) transforms the achieved classification into a selection among different alternatives in order to fulfill the demands requested on the system.

2.2.2 Imaging as measuring process on the electromagnetic spectrum The Image Acquisition stage (see Fig. 2.2) plays a crucial role in the development of computer vision systems in practical applications. This stage attains the extraction of the measures upon which the rest of the system is based. The obtained measures are represented in the discrete domain through a regular twodimensional grid of squared pixels, which is denoted as image. The selection of the sensor that undertakes the measure and the conditions of the measurement process shape the performance of the system. This fact is mostly disregarded from a theoretical point of view. Different phenomena are detected by different kinds of devices. Furthermore, the nature of the acquired measures depends on the characteristics of the detector embedded in the imaging device. The material of the detector fixes up its spectral response and thus its ability to gather the measurement of the energy in a particular spectral band (see Fig. 2.3)4 . A detailed description of the physical principles of these measurement processes is out of the scope of this work. The interested reader is referred to specialized literature on this topic [2] [184]. The purpose of this section is a qualitative description of the possibilities offered by up-to-date imaging technologies. The first images taken into consideration in computer vision systems were grayvalue ones. The most used detector technologies for delivering grayvalue images are known as Charge-Coupled Device (CCD) and Complementary Metal Oxide Semiconductor (CMOS). Both detectors are based on sillicium (Si) but differ in the used semiconductor technology and therefore in its microelectronic architecture. While CCD sensors are superior in terms of sensitivity, dynamic range, uniformity (specially in dark regions), and easiness of shuttering, CMOS sensors are superior in terms of speed, windowing capabilities, and avoidance of blooming [95] (see Fig. 2.4). Moreover the architecture of CMOS detectors allows the access to the information of each individual pixel and facilitates the integration of these detectors with other microelectronic elements. Summarizing, it can be 4

Further information on this aspect can be found in the following URLs: Optical detectors and human vision (http://cord.org/step_online/st1-6/st16eii1.htm), Temperature measurements (http://www.me.uvic.ca/~mech455/Lecture14-IRFPA.pdf), and PhotriX Spectral Wavelength and Bandwidth Advantages (http://www.luxtron.com/ pdf/photrix/TN04_OFT02.pdf).

8

1/2

detector sensitivity (D) [cm Hz / W]


1015 1014

1: CdS@300K 1

1013

2: Si@300K

3: GaAs@300K

2 5

4

1012

4: InGaAs@300K 5: InAs@77K

3

1011

6: PbS@300K

6 8

1010

7: Ge:Au@77K

9 10

7

10 9

8: InSb@77K 9: HgCdTe@77K 10: Ge:[email protected]

10 8 0,1

0,3

1,0 3,0 wavelength [µm]

10

30

Figure 2.3: Sensitivity (D) bands of different materials employed in different image detectors with respect to the wavelength. The bands are depicted by characterizing them through the maximum sensitivity values at a particular operating temperature, which is specified in the legend.

(a)

(b)

Figure 2.4: Grayvalue images acquired by two cameras with respectively a CCD (a) and a CMOS (b) detector architecture. The CMOS camera, which was an experimental camera of first generation, clearly shows its worst performance in terms of sensitivity.

9


stated by comparing the two technologies with analogous spatial resolution that CMOS detectors offer better facilities for low-consuming high-integrated acquisition devices, but not for applications with a demand on image quality5 . Grayvalue imaging devices are taken into consideration in order to analyze geometrical features, both static and dynamic, of the acquired scene. CCD (green) trim filter

CCD (red) bayer filter prisma

CCD (blue)

CCD (a)

(b)

Figure 2.5: The measurement of color information is based on gathering the spectral bands centered in the red, green, and blue colors. The most common architectures used for implementing this measurement process are based on one or three CCD chips. (a) Bayer filter, which assigns the sensitivity of each pixel in a CCD chip to a particular spectral band. (b) Three CCDs architecture.

Color imaging is based on the formerly presented detectors. In this case, the scene is simultaneously measured on different bands of the visible spectrum through the employment of filters. Two kinds of filters are used for color measurement in two different types of devices. On the one hand, one chip systems use so-called Bayer filters, which assigns a particular spectral responsivity for each pixel in the image (see Fig. 2.5a). On the other hand, acquisition devices with three detectors split the incoming light in three different beams, which fall upon them (see Fig. 2.5b). Thus these last mentioned acquisition devices are multi-sensory in nature. Furthermore, they present a more exact color fidelity. Nevertheless, the most modern trend concerning color fidelity is achieved by multi-spectral cameras [67], where multiple narrow-band optical filters (typically around 20) are set up in front of a CCD detector. The employment of color imaging devices principally allows the reproduction of this important perceptual cue, but it can be used in order to collect a multi-sensory set of images as well. Sensing the spectrum through imaging is not restricted to the visible spectral band. Image acquisition devices exclusively sensing the infrared spectral band 5

Professional digital photography cameras of the last generation are based on CMOS sensors. The high quality of the acquired images is due to the improvement of the S/N ratio of the CMOS detectors, which is newly achieved by powerful in-chip signal conditioning.

10


have been used in computer vision as well. The first infrared cameras available in the market were based on a unique photodetector made from HgCdTe [172], which needed to be cooled through the application of liquid nitrogen (see Fig. 2.3). On the contrary the most modern devices embed a so-called Focal Plane Array (FPA) of InSb- or PtSi-detectors, which are sensitive in the spectral band from 1000 to 7000nm [17] (see Fig. 2.3). These devices are electronically cooled. This fact, together with the increment in the resolution of the detectors, is thought to widen the employment of infrared cameras in computer vision applications [45]. The images acquired with an infrared camera can deliver information about the temperature distribution and the material composition of the inspected scene. Furthermore, structure and sub-surface information, till approximately 20µm depth, can be extracted from the analysis of infrared images (see Fig. 2.6). The exact penetration depth within this range depends on the spectral band of the detector’s sensitivity curve (see Fig. 2.3), the employment of an energy source for the thermal activation of the inspected material, the wavelength of this source, and the reflection features of the material itself.

(a)

(b)

Figure 2.6: Comparison of images acquired in the visible and infrared spectral bands on hand of a cloth piece. (a) Color image acquired with a 3-CCD camera. (b) Infrared image acquired through active thermography with an infrared camera embedding a InSb detector, i.e. the image is acquired after applying a light pulse on the inspected object. Image acquired in a research stay at the Fraunhofer Institute for Wood Research, FhG-WKI (Braunschweig).

Sub-surface structural information can also be attained through the employment of ultrasonic imaging devices. The different image acquisition devices functioning in the ultrasonic spectral band are known as A-, B-, and C-scans. Furthermore some research efforts are being devoted to new and more efficient ultrasonic

11


sensors (see Fig. 2.7). The penetration depth achieved in the ultrasonic inspection reaches the 10mm depending on analogous parameters as in the infrared measuring process.

(a)

(b)

Figure 2.7: Comparison of images acquired in the visible and ultrasound spectral bands on hand of a PVC object. (a) Color image acquired with a 1-CCD camera. (b) Ultrasound image acquired with a camera prototype. The difference of height among the holes in the PVC object are displayed with different grayvalues in the ultrasound image. It is worth mentioning that the image on the PVC object is acquired from the opposite site as the one displayed in (a). The employed ultrasonic camera was developed in a research project realized together with the Fraunhofer Institute for Non-Destructive Testing, FhG-IZfP (Saarbr¨ ucken), and the Fraunhofer Institute for Silicate Research, FhG-ISC (W¨ urzburg).

Finally there are some imaging devices, e.g. the ones used in remote sensing applications, that simultaneously measure the spectrum in visible and non-visible spectral bands. The detectors used for remote sensing are made of different materials and their functionality is based on different physical principles. Multispectral sensors collecting information in few discrete non-continuous bands are being substituted by hyper-spectral devices [145]. Hyper-spectral sensors measure the spectrum from the visible region to the near-infrared in hundreds of narrow contiguous bands about 10nm wide (see Fig. 2.8). The generated images can be used in other application fields of computer vision as stated by [50]. Hyperspectral imaging devices are usually employed in order to obtain the so-called spectral signature of the image elements, which characterizes their molecular composition [145]. These are some of the most used sensors in practical applications. Nevertheless, several more imaging devices are taken into consideration in computer vision

12


Figure 2.8: Exemplary hyper-spectral image of a brain slice. The spectrum is sampled at the visible and near-infrared spectral bands and the result is delivered as an hyper-spectral cube [49].

systems. All these devices are jointly employed in multi-sensory computer vision systems for the generation of what can be denoted as multi-sensory image sets. The following are some examples of the employment of different imaging devices, in multi-sensory computer vision systems: color [137], multi-spectral [94], bimodal [138], multi-modal [139], range and grayvalue [185], infrared and visible [115], synthetic aperture radar (SAR) and grayvalue [43], temporal sequenced images [106], and, finally, images taken under different conditions of illumination [101], shutter time [16], focal distance [123] or camera position [128]. Medical imaging, microscopic imaging, remote sensing, automated visual inspection, robotics, automatic target recognition (ATR) and 3D model reconstruction are some of the application fields where multi-sensory image sets are employed. There are numerous devices that can be employed in the development of multisensory computer vision systems. The proliferation of imaging devices increases the technological demand on paradigms for jointly using the extracted data. This is the final goal aimed by integration and fusion methodologies.

2.3 Integration and fusion of multi-sensory information The relevance of information fusion methodologies increases due to the complementary development of computer and sensory technologies. The availability of novel and more reliable sensors (see Sec. 2.2.2) is complemented by the existence of more advanced computing technologies, whose memory capability and processor times allow the implementation of powerful signal processing methodologies

13

2.3 Integration and fusion of multi-sensory information

at decreasing costs. Furthermore up-to-date hardware and software architectures, e.g. multiprocessor systems, parallel programming languages, encourage the implementation of parallel processing strategies. Therefore the integration of different sensors in a processing system is newly assured. This fact can lead computer vision systems to achieve goals of increasing complexity, what can only succeed by both collecting and jointly exploiting a diversity of information on the environment. Sensors, which measure the physical variables involved in the implementation of a particular system, constitute the first stage of the information collection process. The difference between sensors and sensor devices is fuzzy in nature, where sensor definitions go from the detectors itself to the results of a complex processing system (see [2] for a good discussion on this topic). Specially with the development of so-called smart and intelligent sensors [19], which integrate the processing technologies into the sensor unit or device, the sensor definition becomes even fuzzier (see Fig. 2.9). In the here presented work a sensor is defined as the element of an instrumentation process that converts a physical input variable into a signal output variable [65]. intelligent sensor digital sensor integrated sensor

physical variable

DETECTION

CONDITIONING

A/D CONVERSION

PROCESSING

signal variable

Figure 2.9: The different types of sensors, i.e. integrated, digital, and intelligent, are defined with respect to the stages used in the sensing process, which constitutes a mapping between a physical input variable and a signal output variable. Thus the signal variable results after different degrees of processing.

2.3.1 Integration vs. fusion It is first worth taking into consideration the difference between multi-sensory integration and multi-sensory fusion. These are two concepts that are usually confused even in the specialized literature.

14


The inclusion of different sensors in a technical system is denoted as multisensory integration [99]. The goal of this inclusion is the reduction of the uncertainty resulting from the limitations of a particular sensor, e.g. its sensitivity to a particular phenomena (see Fig. 2.3). This uncertainty can hinder the system to attain its final goal. Thus, the multi-sensory integration is undertaken in order for the system to gather a diversity of information from the environment and, therefore, to reduce this uncertainty. As with most engineering systems, the more accurate multi-sensory information should be obtained in the shortest possible time at the lowest possible cost [99]. Nevertheless, a generalized bad practice is to include as many sensors as possible or as available. This may result from the lack of a mature theoretical background on multi-sensory systems research. MULTISENSORY INTEGRATION SEPARATE OPERATION SENSOR 1

SENSOR MODEL

SENSOR i

SENSOR MODEL

SENSOR n

SENSOR MODEL

SENSOR REGISTRATION

MULTISENSORY FUSION

WORLD MODEL

GUIDING/CUEING

SENSOR SELECTION SENSORY PROCESSING

SENSOR CONTROLLER

SYSTEM CONTROLLER

Figure 2.10: Block diagram of a generic system with multi-sensory integration [99].

A generic system with multi-sensory integration presents the block diagram depicted in Fig. 2.10. The first stage in a process of multi-sensory integration manages to model the answer of the sensor (Sensor Model in Fig. 2.10). In this context, the quality of the sensor in terms of uncertainty of the generated data and the error expectance is mostly established a priori. One usual approximation is the characterization of the data uncertainty through a Gaussian distribution [99]. The registration of the data (Sensor Registration in Fig. 2.10) transforms the coordinate system of each sensor in terms of one or more particular physical variables into a common one, e.g. time synchronization, planar position alignment. This leads the transformed data to present the same reference system.


15

In the multi-sensory application field, the data delivered by two different sensors is denoted as redundant if the same physical variable is gathered by both of them, whereas it will be considered complementary on the contrary case [99]. This characterization is based on the subjective knowledge of the system developer and applied in the selection of the sensors to be embedded in the system (Sensor Selection in Fig. 2.10). This process normally represents the first stage in the system implementation. As it can be observed (Sensory Processing in Fig. 2.10) the multi-sensory data can be processed following different strategies. The information of one sensor can be used in order to guide the sensing process of another one (Guiding/Cueing) or they can be used separately (Separate Operation). The multi-sensory fusion (Multisensory Fusion) is the third of these options. The fusion of information is undertaken when different information sources act as different pieces of evidence in the resolution of the problem that constitutes the goal of the processing system. Thus information fusion methodologies attain the transformation of the information delivered by multiple sources into one representational form [1]. Furthermore the fused data reflects information that can be extracted from the individual sources but also information not derivable by any of them on its own [1]. As already mentioned, such a gain on information characterizes the purpose of information fusion.

2.3.2 Information fusion in Computer Vision The idea of combining different pieces of evidence for the resolution of a processing goal is specially important in computer vision systems. In the context of imaging devices the uncertainty derived form the inclusion of a sole sensor present different faces. The occlusion of one part of the scene, the “invisibility” of particular features in some spectral bands (see Sec. 2.2.2) or the deliverance of ambiguous information, e.g. edges caused by a shadow, can be overcome by taking into consideration more than one imaging sensor [113]. On the other hand this uncertainty can be considered from an algorithmical point of view. Since the successful implementation of real applications in the field of computer vision results rather complex, it has been posited that the utilization of different image transformations is often required for the successful implementation of a computer vision system [150]. These two concepts can be generalized in order to justify the consideration of different information sources and thus the employment of information fusion methodologies in Computer Vision. Therefore it can be stated that the complexity of computer vision applications makes necessary the joint utilization of different information sources, i.e. image sensors or methodologies, in a particular processing stage. The data delivered by these multiple sources are used as different pieces of information. This measure leads to the successful attainment of the system’s goal, if the fusion methodology manages to preserve the information

16


available from the individual sources, as well as to complement it with a gain of information that can not be extracted from the information sources separately [1]. In this sense the goal of the fusion stage in computer vision systems does not differ from that of other application fields. The here presented work is focused in the concept of intelligent sensor fusion, which was first introduced in the context of computer vision for robot navigation applications [113]. This concept is characterized by the following features: • robustness, • consideration of contextual knowledge, • responsiveness in front of sensor failures, • and adaptability with respect to changes in the environment. It is worth pointing out the parallelism between these demands and the features of soft computing methodologies for data processing (see Sec. 2.4). Thus, the application of Soft Computing allows extending this model in order to establish a new theoretical model for multi-sensory fusion (see Sec. 4.1). Taxonomy on information fusion The fusion of information can succeed at different stages of a processing system. A taxonomy on the different types of multi-sensory fusion can be established by taking into consideration all stages of the block diagram of a generic computer vision system (see Fig. 2.2). This classification is originally proposed in [99] and extended in [136] as follows: • Signal fusion considers the fusion of the signals obtained through the image acquisition stage in order to improve the reliability of the images. • Pixel fusion occurs when different pixel information is fused in order to increase the useful information available on each pixel. • Segment fusion considers the fusion of information contained within the different regions resulting from a segmentation process. • ”Dif-fusion” is a type of fusion procedure derived from the analysis of biological systems [136]. The diffusion considers the integration of information in such a way that, one information source expands its validity in a domain whose limits are established by the other information source. This kind of fusion is used in morphological segmentation procedures such as the Watershed Transformation [149]. • Feature fusion considers the fusion of features extracted by different procedures in order to increase the available information.


17

• Classification fusion is defined as the fusion of the results of a classification process. • Symbol fusion allows the generalization of a group of symbolic representations by a hierarchically highest concept. This kind of fusion has been also called intelligence fusion [103]. As it can be observed the pixel fusion, the segment fusion, and the so-called “dif-fusion” are the only fusion types that exclusively belong to the application field of Computer Vision. Theoretical frameworks for the fusion of multi-sensory information Different fusion operators are used in different fusion methodologies for Computer Vision. Among all operators used in information fusion, classical operators, i.e. product, sum, mean, median and their non-linear versions, are used in a great part of applications [3]. Other employed operators are related to the theoretical frameworks used mostly for other purposes: Theory of Regularization [137], Bayesian Theory [39], Evidence Theory [74], Image Algebra [164], and Neural Networks [110]. Kalman Filtering [7], Laplacian Pyramids [16] and Wavelet Transform [94] are some of the general purpose signal processing methodologies also used for information fusion. A set of operators have been employed for information fusion within the theoretical framework of Fuzzy Computing: fuzzy connectives, as T-, S- and Uni-norms [43], weighted ranking operators [170], Ordered Weighted Averaging (OWA) operators [87], and Fuzzy Integrals [166]. The operators related to Fuzzy Computing constitute a flexible means for the fusion of information represented by fuzzy membership functions. These operators, which are generally denoted as fuzzy aggregation operators, constitute an alternative to operators traditionally used in information fusion. In the following, the term fuzzy fusion operators will be preferred in order to make clear the idea that such operators are beyond the concept of aggregation, which is more related to the mathematical addition. Fuzzy Computing is the only herein mentioned theoretical framework, where operators specifically devoted to aggregation are developed. Fuzzy fusion operators, specially the fuzzy integral, emphasize the flexibility and interpretability of the fusion operation in contrast to classical ones. Furthermore these operators were developed specifically for the attainment of dimensionality reduction with information gain. Despite these positive features, fuzzy fusion operators are not widely used in computer vision applications, e.g. [3] [7] [94] [115]. The employment of fuzzy fusion operators can be spread through the elucidation of their application possibilities and their sometimes complex mathematical background. In this sense it is worth first focusing some attention to the Ugly Duckling Theorem in order to give a theoretical motivation for the employment of fuzzy fusion operators.

18


2.3.3 Ugly Duckling Theorem and the usage of Soft Computing for information fusion Logical inference is the basic process used in formal logics for the implementation of reasoning, what is undertaken by combining different predicates in order to reach a conclusion. Artificial intelligence has extended this concept into the combination of information for the implementation of artificial systems, making it the fundamental mechanism of reasoning in expert systems [48]. The Ugly Duckling Theorem, which was introduced in [182], constitutes one of the fundamental theorems of pattern recognition [38] and, as it will be presented in this section, is deeply related to the field of Soft Computing (see Sec. 2.4 for introduction on the field). The theorem relates the categorization of objects with the process of association of logical predicates. In this context, enumerating the logical predicates that apply to different objects is a common fashion of establishing a similarity relationship among them, i.e. the similarity among objects is quantized by the number of predicates shared by them. The computation of the similarity in this manner, which is undertaken by fusing some information, leads to the categorization of the objects as described in the following example. Imagine a logical system that tries to establish categories on three different liquids: lemonade, water, and oil (see Fig. 2.11). Suppose the predicates colored and potable are taken into consideration for the categorization of the three liquids. The different logical combinations of the predicates, i.e. colorful AND NOT potable, colorful OR potable, are valid predicates as well and, therefore, must be taken into consideration in order to find out the number of predicates shared by the three liquids. The following nomenclature is established. The number of atoms taken into consideration is denoted as d. An atom is defined as an element in the Venn ¯ and thus d = 4 diagram that can not be further divided [182], e.g. A∩B or A∩ B, (see Fig. 2.11). Moreover the rank r states for the cardinality of the predicates that can be established on the problem at hand. In the example colored is a predicate of rank r = 1, while colored OR NOT potable presents r = 2. The Ugly Duckling Theorem establishes that the number of predicates shared by any of two objects is d−2 X d−2 = 2d−2 . (2.3) r − 2 r=2

Thus, the number of predicates shared by any two objects is constant and independent of the two objects considered (see Fig. 2.11). Therefore 24−2 = 4 predicates are shared by any two objects in the example. A detailed mathematical demonstration of the theorem can be found in [182]. Pattern recognition systems [38], where features are employed instead of logical predicates as shown by (2.2), are faced with similar problems. The consequences of the Ugly Duckling Theorem for the field of Pattern Recognition are diverse.

19


A

B

x1

x3 x2

C1

{B ,

A B

C1

,

A¯ B , U}

Figure 2.11: Exemplary pictorial description of the Ugly Duckling Theorem in a Venn diagram for n = 2 elemental predicates, A =colored and B =potable, thus the number of atoms (indivisible areas) is d = 4: ¯ A¯ ∩ B, and A¯ ∩ B. ¯ The predicates are used to form A ∩ B, A ∩ B, categories on three liquids: x1 oil, x2 lemonade, and x3 water. All pairs to be formed on the three objects share the same number of predicates 2d−2 = 4 up to the Ugly Duckling Theorem. The class formed by lemonade and water, which define the category C1 , share the predicates potable, potable AND colored, potable AND NOT colored, and the true predicate U, which takes into consideration all the universe of discourse. This category presents a cardinality |C1 | = 4. The category lemonade-oil share colored, potable OR colored, colored OR NOT potable, and the true predicate, hence presenting the same cardinality as C1 . As a corollary it can be stated that both categories can not be establish just by using as similarity measure the number of predicates shared by the elements of the category unless a ponderation on the importance of the predicates is taken into account.

Among them it is worth mentioning herein the fact that there is not a set of features that can be considered as “best” without taking into consideration the problem at hand [38], i.e. for trying to answer the question what-to-drink? the predicate potable would be the most appropriate. Nevertheless this a priori knowledge is not always available, specially in the field of Pattern Recognition. In order to break the ambiguity exposed by the Ugly Duckling theorem, [182] brings up the concept of pattern recognition as value-oriented ponderation. Thus the categorization among objects can be established upon the number of predicates shared by them, but only after introducing some weighting that makes some predicates more important than others. The role of weighting and the preference on flexible weighting schemes has been taken into consideration for developing the paradigm of soft data fusion (see Sec. 4.2), where fuzzy fusion operators find a common theoretical framework.

20


Being fuzzy fusion operators a part of Fuzzy Computing (FC), and FC one of the main building blocks of Soft Computing, this last field is analyzed in the following section.

2.4 Soft Computing for data processing Soft Computing is an alliance of different processing methodologies that seek the resolution of real applications. In order to fulfill this objective, the methodologies included in the theoretical framework of Soft Computing share the following features [12][195]: • adaptability, • tolerance in front of errors, • biological and cognitive inspiration, • parallelism, • and treatment of numeric data in a bottom-up approach. The adaptability of Soft Computing methodologies relies on the inclusion in the procedures of algorithms for their automated parameterization. This fact allows them to apply on different data sets and applications without substantially modifying the implementation of the processing systems. Hence Soft Computing methodologies outperform other processing techniques in terms of flexibility. The tolerance in front of errors is achieved by relaxing the optimality constraint in the solution space. Soft Computing methodologies do not principally target finding out the best solution of the problem at hand, but they do approximate it with sub-optimality. This relaxation of the goal allows the system to robustly react in front of unexpected inputs. The biological and cognitive inspiration of the methodologies attains emulating the performance of biological systems in real applications. Since information processing in biological and cognitive systems is parallel in nature [12], the parallelism of the different methodologies can be considered a consequence of this inspiration. Moreover the exploitation of the parallelism, which is currently achieved by means of hardware implementations of the methodologies, can lead the processing systems to operate in real time and therefore eases the implementation of real applications. The treatment of numeric data is a consequence of the poor performance of traditional artificial intelligence approaches, which operate on symbolic data, in real applications. In this context, it is worth mentioning that the term Computational Intelligence is used to denote the joint implementation of Soft Computing

2.4 Soft Computing for data processing

21

methodologies in a processing system for the resolution of a complex problem [12]. All these facts allow us to summarize that Soft Computing achieves the robust and flexible resolution of complex processing problems present in real applications. For that purpose Soft Computing presents methodologies for the machinebased representation, acquisition, and processing of knowledge [33]. Since Pattern Recognition takes into consideration the processing of data and knowledge, the implementation of real applications of pattern recognition finds in soft computing methodologies an interesting theoretical playground. In order to cope with real life problems, Soft Computing does not rely on any particular methodology, but on the combination of the so-called characteristic function [46] of different ones. The resulting hybrid processing system seeks fitting the characterizing functionality of each methodology in order to succeed in the resolution of the problem at hand. Neurocomputing (NC), Fuzzy Computing (FC), and Evolutionary Computing (EC) are the main theoretical building blocks of Soft Computing [195], although some others have been appended later, e.g. Immunological Computing (IC), Machine Learning, Chaotic Computing, and Probabilistic Computing (PC). Neurocomputing groups different procedures based on so-called neural networks. Neural networks are distributed processing systems, whose structure and operation simulates these of connectionist models established on neuronal circuits present in natural systems. The characteristic function of neural networks is the non-linear approximation of functions [33]. The combination of the neural network structure with learning procedures, which are mainly studied in Machine Learning, confers to the system the capability of acquiring knowledge up to a set of examples [46]. Fuzzy Computing was built on the Theory of Fuzzy Sets [192]. Fuzzy Sets are a generalization of classical sets in terms of the membership definition, which becomes a real-valued function in contrast to the binary definition within the classical Theory of Sets [21]. Other facets that conform Fuzzy Computing were developed up to this set-theoretic one [195]: the logical facet, which presents a multivalued logical system (Fuzzy Logic); the relational facet, which is concerned with the representation and binding of imprecisely defined functions (Fuzzy Rules); and the epistemic facet, which is focused on meaning, knowledge and decision (Fuzzy Decision Making). The Theory of Fuzzy Measures, which constitutes the theoretical background of the here presented work (see Sec. A.1), originally belongs to this last facet. The characteristic function of Fuzzy Computing is the approximation of a mapping function that relates a symbolic domain with a pattern domain [46]. Since the symbolic domain is usually defined in linguistic terms, it can be stated that the achieved approximation goes beyond the black box approach of neural networks by focusing on general interpretability [33]. Evolutionary Computing results from the application of the principles of the biological Theory of Natural Selection [28] into computational search procedures.

22


The characteristic function of Evolutionary Computing is the obtainment of a solution in non-linear optimization problems [46]. For that purpose the evolutionary methodologies, which include Genetic Algorithms, Evolutionary Programming, Evolutionary Strategies, and Genetic Programming, take into consideration a multi-point search in the solution space. The iterative search is driven by a so-called fitness function, which characterizes the optimality of the individual solutions in each step. NC

inspiration EC 001010 101101 001101 001001 111000 010111 010000 001010

FC

µ

LINE

CIRCLE

SC

IC

PC

BIOLOGICAL INTELLIGENCE

capability extension

COMPUTATIONAL INTELLIGENCE

Figure 2.12: The cognitive metaphor of soft computing systems: Soft computing (SC) systems take into consideration different methodologies inspired by biological and cognitive models for the implementation of intelligent systems. The result of this implementation is known as Computational Intelligence [12]. These methodologies are mainly grouped into the disciplines: Neurocomputing (NC), Fuzzy Computing (FC), Evolutionary Computation (EC), Immunological Computing (IC), and Probabilistic Computing (PC). The combination of the methodologies of these disciplines in hybrid processing systems, which is an open research topic, can be biologically inspired by the interaction in biological autonomous systems among the corresponding inspiring principles.

One of the most interesting aspects of Soft Computing is the formerly mentioned implementation of hybrid systems for the resolution of complex real problems. How to combine the different methodologies in a hybrid system is an open research topic [46]. These hybrid systems process information/knowledge by taking into consideration the principles of emergence, self-organization, reflectivity, and interaction [33]. These principles have been used in the Cognitive Sciences to define the new paradigm of the so-called enactive models of cognition, which

23

2.5 Fuzzy Fusion Operators

overcome the cognitivist and the connectionist approaches [174]. The enactive cognition is based on the concept of embodiment. Thus Soft Computing can be taken as a theoretical body of knowledge, whose realization targets in the long term the autonomous relationship of a system with its environment (see Fig. 2.12). Each of the elements of this body has a characteristic function as already mentioned. Its assembly in hybrid systems can be itself inspired by the relationship among the inspiring biological and cognitive elements in order to implement more intelligent machines.

2.5 Fuzzy Fusion Operators Once the theoretical framework of Soft Computing has been described, the attention can be focused on the field of Fuzzy Computing, and particularly on fuzzy fusion operators, which tackle the question of how to combine information extracted from different sources. In this context it is worth reminding the conclusions set upon the Ugly Duckling Theorem (see Sec. 2.3.3). Thus Pattern Recognition have to be understood in this context as a process of value oriented ponderation, where weighting features makes possible to establish similarities among data points in a feature space. This concept is implemented at its best by the Fuzzy Integral, since it presents the most discriminative weighting schema of all fuzzy fusion operators. From a mathematical point of view the Fuzzy Integral results from the combination of two fuzzy connectives, which are more simple operators.

2.5.1 Fuzzy connectives: T- and S-Norms Fuzzy Computing has been one of the faster application fields of the concept of triangular norms and conorms [82], which are respectively called T- and S-norms for short, since the seminal suggestion for using them in the aggregation of fuzzy sets [192]. T-norms were introduced within the framework of statistical metric spaces [107], which take into consideration probability distributions in the computation of the elements of the space. The basic idea is that the points in these spaces are a result of a measurement process, where the uncertainty due to measurement errors can be modeled through probability. In this context T-norms were defined in order to compute distances in these spaces [107] by generalizing the triangle inequality from classical metric spaces. Therefore T-norms are defined as binary operations T on the unit interval [0, 1]: T : [0, 1]2 → [0, 1], that satisfy ∀x, y, z ∈ [0, 1] the following axioms [141]:

24

2 Multi-sensory Computer Vision I. Commutativity: T (x, y) = T (y, x).

II. Associativity: T (x, T (y, z)) = T (T (x, y), z). III. Monotonicity: T (x, y) ≤ T (x, z) ↔ y ≤ z. IV. Boundary condition: T (x, 0) = 0, T (x, 1) = x. S-norms were introduced as the dual operations of T-norms [141]. Thus they satisfy the same axioms up to the boundary condition, which turns into: S(x, 0) = x, S(x, 1) = 1. The four basic T-norms [82] are the minimum, the product, the Lukasiewicz t-norm, and the drastic product. On the other hand the basic S-norms [82] are the maximum, the probabilistic sum, the Lukasiewicz t-conorm, and the drastic sum (see Tab. 2.1). Furthermore other T- and S-norms, which can be found in [82], have been developed. Table 2.1: Expressions of the basic T-norms and their dual S-norms after [82]. ∧: Minimum. ∨: Maximum. Luk. T-: Lukasiewicz T-norm. Luk. S-: Lukasiewicz S-norm. prod.: Product. prob. sum: Probabilistic sum. dras. prod.: Drastic product. dras. sum: Drastic sum. T-norm ∧(x, y) ∨(x + y − 1, 0) x·y ( ∧(x, y) if x ∨ y = 1 dras. prod. 0 otherwise minimum Luk. Tprod.

maximum Luk. Sprob. sum dras. sum

S-norm ∨(x, y) ∧(x + y, 1) x + y − xy ( ∨(x, y) if x ∧ y = 0 1 otherwise

The fuzzy integral constitutes an aggregation operator built upon these norms [62]. Beyond its utilization for aggregating fuzzy membership functions, T- and S-norms are related to fuzzy integrals. Thus all types of fuzzy integrals (see Sec. 2.5.2) employ a combination of a T-norm and an S-norm.

2.5.2 Fuzzy Integrals The concept of fuzzy integral is due to Sugeno, who presented in [163] a mathematical approach for the simulation of multicriteria evaluation taking into consideration some cognitive aspects. Sugeno’s hypothesis is that the process of multicriteria integration undertaken by human beings transcends the linear combination of the different criteria with numerically expressed priorities, i.e. weighting sum strategy. The elements that make the mathematical expression of the fuzzy integral follow the hypothesis posited by Sugeno are: the utilization of fuzzy membership


25

functions as integrands, their weighting through fuzzy measures, and the binding of these two elements through a combination of T- and S-norms (see Sec. 2.5.1). The fuzzification of the data from the information sources xi is made through the application of the fuzzy membership functions hi (x) on the data delivered by the information sources. Fuzzy Measures Fuzzy measures, which are denoted as µ, were established as a means of expressing the a priori importance of the integrands. They characterize the a priori importance of the information sources xi , where i = 1, ..., n, through the coefficients: µi = µ({xi }) , (2.4) which are denoted as fuzzy densities [166] as well. Moreover they characterize the importance of their coalition6 , which is represented here as the union of the corresponding subsets, µij = µ({xi , xj }) ∀i, j = 1, . . . , n i 6= j .. (2.5) . µ1...n = µ(X) = µ({x1 , . . . , xn }). Thus fuzzy measure coefficients µ(Aj ) are defined on all possible subsets Aj of the set of information sources X = {x1 , . . . , xn }. Fuzzy measures are, from a mathematical point of view, a generalization of the weights in a weighting sum operator, i.e. in this fusion operator the weights are just given on the subsets of cardinality one w1 , ..., wn . In this sense the concept of Pattern Recognition as a value oriented ponderation (see Sec. 2.3.3) can be improved. The coefficients of the fuzzy measures satisfy two conditions (see Sec. A.2). First, they are generally defined in the interval [0, 1]. Second, they satisfy the so-called monotonicity condition, which states: Aj ⊂ Ak → µ(Aj ) ≤ µ(Ak ) ∀Aj , Ak ∈ X.

(2.6)

General fuzzy measures strictly follow these two conditions. Other types of fuzzy measures specify the monotonicity condition among the fuzzy densities and the coefficients of the coalitions through a mathematical expression (see Sec. A.3). For instance, the coefficients of fuzzy-λ measures, which are the most used in engineering systems, fulfill the expression: µ(Ai ∪ Aj ) = µ(Ai) + µ(Aj ) + λµ(Ai )µ(Aj ) ∀i, j ∈ I, 6

(2.7)

Coalition states for the fact that the information delivered by different information sources agree (large value of the coefficient) or not (small value of the coefficient)

26


where the interaction among the information sources is characterized by λ. The value of this parameter is found after assigning the fuzzy densities and thence solving the equation [166]: " n # 1 Y µ(X) = [1 + λ · µ({xi })] − 1 . (2.8) λ i=1 The interested reader can find a deeper explanation on the theoretical background of the Theory of Fuzzy Measures in Appendix B. The definition of this new type of measures led to the definition of a new integral operator: the fuzzy integral. Fuzzy Integral Operators Sugeno proposed the following mathematical expression for the new integral, which is known as the Sugeno Fuzzy Integral (SFI): Sµ [h1 (x1 ), . . . , hn (xn )] =

n _

[h(i) (xi ) ∧ µ(A(i) )],

(2.9)

i=1

where the used T- and S-norms are in this case the minimum (∧) and the maximum (∨) operators. The enclosed sub-index states for a sort operation previous to the integration itself, e.g. for n = 3 if h1 (x1 ) ≥ h3 (x3 ) ≥ h2 (x2 ), then h(1) (x1 ) = h1 (x1 ); h(2) (x3 ) = h3 (x3 ); h(3) (x2 ) = h2 (x2 ), operation that would also determine the coefficients of the fuzzy measures employed in the integration as µ(A(1) ) = µ({x1 }); µ(A(2) ) = µ({x1 , x3 }); µ(A(3) ) = µ({x1 , x2 , x3 }). Thus the index of the different subsets in the expression of the fuzzy integral denotes their cardinality. The Choquet Fuzzy Integral (CFI) was introduced as a fuzzy generalization of the Lesbesgue Integral [111]. Therefore the employed T- and S-norms are in this case the product and the addition. The resulting expression has the form: Cµ [h1 (x1 ), . . . , hn (xn )] =

n X

h(i) (xi ) · [µ(A(i) ) − µ(A(i−1) )].

(2.10)

i=1

While the Choquet Fuzzy Integral can be considered as a mean-like fusion operation, the Sugeno Fuzzy Integral is near to some ranking operators as the

27


median (see Fig. 2.13 and Sec. B.1). Intuitively the Choquet Fuzzy Integral weighs the data delivered by the information sources through the coefficients of the fuzzy measure selected as a result of the previous sorting operation. On the other hand the Sugeno Fuzzy Integral meets a tradeoff between the subjective a priori importance embedded in the coefficients and the data actually delivered by the information sources with the same structure as the well known max-min optimization criteria. Both types of integrals are characterized by the definition of a set of weighting coefficients in each canonical region of the hyper-cube (see Sec. B.2)

Ft-cI CFI

OWA

min

weighted min

arithm. mean

SFI

med

MAX

weigted MAX

weighted sum

Figure 2.13: Generalization relationship between fuzzy fusion operators (modified from a figure in [58]). Ft-cI: Fuzzy t-conorm Integral. CFI: Choquet Fuzzy Integral. SFI: Sugeno’s Fuzzy Integral. OWA: Ordered Weighted Averaging. med: median.

There are other types of fuzzy integral [112] [179] [183] that generalize the already mentioned fuzzy integrals. For the sake of simplicity only the fuzzy t-conorm integral [112] is mentioned. This integral is a generalization of the Choquet and Sugeno Fuzzy Integrals in the sense that the used T- and S-norms are not set a priori. The existence of such a vast number of fuzzy integral operators improves the flexibility of the fusion operation, but makes more difficult the understanding of the theoretical background. Although promising alternatives [78], these fuzzy integral operators have not been used in image processing nor in pattern recognition so extensively to dare further attention herein. Thus the dissertation is exclusively focused in the utilization of the Sugeno and the Choquet Fuzzy Integrals in computer vision systems. Appendix B further discussed on these two fusion operators from a theoretical point of view by giving an overview on its mathematical properties.

28


2.6 Utilization of the Fuzzy Integral in Computer Vision The present section describes different applications of the fuzzy integral for computer vision in chronological order of appearance. The different applications are analyzed by taking into consideration the attained goal, the processing framework implemented for this purpose, and the existence, or not, of procedures for the automated construction of the fuzzy measures. The first application of the fuzzy integral for computer vision [80] attains the segmentation of color images. The segmentation is based on the computation of different features on the grayvalue images of the channels of the input color images. The coefficients of the fuzzy measures are heuristically found. However [129] presents a very similar work on image segmentation, where these coefficients are computed by taking into consideration the overlapping areas of the histogram peeks, which represent a class each. Therefore the histograms are used as fuzzifying functions for each of these classes as well. The aforementioned frameworks for image processing are generalized by the work presented in [166], which constitutes the seminal application of the fuzzy integral for information fusion in Computer Vision. The general methodology for feature and classification fusion (see Sec. 2.3.2) based on the Sugeno Fuzzy Integral is established and tested in an application for ATR. The fuzzy membership functions are computed through a fuzzy two-mean procedure [13]. The fuzzy measure coefficients are heuristically found by taking into consideration the confusion matrix of the classification results. The quality evaluation on printed color images is attained in [168] based on the Choquet Fuzzy Integral. In this case a decision network with a two consecutive levels undertaking a feature fusion is implemented. The fuzzy measures are constructed by applying a numerical optimization procedure based on quadratic programming. The framework presented in [191] seeks the segmentation of color images. The approach is similar to this presented in [166] based on the computation of the Sugeno Fuzzy Integral, but in this case with respect to possibility fuzzy measures (see Sec. A.3). Two mathematical morphology frameworks for grayvalue images have been presented in [61] [64]. In this case the fuzzy integral of the pixels under an structuring element is computed by manually assigning the corresponding fuzzy densities. The same strategy has been used in different non-linear filters based on the application of the fuzzy integral [56] [70] [71] [146] These frameworks are applied for image enhancement and transformation (see Sec. 2.2.1). The recognition of handwritten words is achieved through the application of the fuzzy integral in [25]. This work presents a complex pattern recognition system based on the combination of a Self-Organizing Feature Map (SOM) and a

2.7 Construction of the Fuzzy Measures

29

Choquet Fuzzy Integral. Since the SOM is used for the automated construction of fuzzy measures, the framework is described with more detail in Sec. 2.8.2. A complex processing system for ATR on LADAR images takes the fuzzy integral into account [77]. The Choquet Fuzzy Integral is used a first stage in the detection, which is based in the fusion of the result of three simple classifiers. The fuzzy densities are determined as the relative number of detections of each classifier on the training set. The fuzzy integral has been used in another framework for color image segmentation [126]. However, in this case the segmentation do not succeed by applying the fuzzy integral on the color channels, but by employing a clustering procedure denoted as mountain clustering. The fuzzy integral is used in order to compute the similarity among color pixels within this clustering algorithm. No details on the construction of the fuzzy densities are given. At this point it is worth mentioning that few methodologies [25] [77] [129] [168] attain the automated determination of the fuzzy measure coefficients. As it can be appreciated in the following section this has been heretofore one of the shortcomings of the application of the fuzzy integral in Computer Vision.

2.7 Construction of the Fuzzy Measures As already mentioned the fuzzy measures constitute the weighting function used by the fuzzy integral. Therefore the result of the application of the fuzzy integral remains parameterized by the values of the fuzzy measures coefficients. These values can be established upon different procedures, which are described in this section. Fuzzy measures are used in the fuzzy integral in order to characterize the a priori importance of the information sources, whose data is being fused. The knowledge needed to set the value of the fuzzy measure coefficients is not well known or difficult to collect in a large number of applications. Even if this is not the case, the translation of a statement like “information source xi is very important” into the numerical value of the corresponding coefficient is complex. The characterization of the coefficients of the coalitions results even more difficult. Thus different methodologies for the construction of fuzzy measures are presented in the literature. A first classification can be made between methodologies just assisting the developer or user in the assessment (Sec. 2.7.1) and those proposing computing procedures for the automated completion of this task (Sec. 2.7.2).

2.7.1 Non-automated assessment of coefficients In the first group three different methodologies are found. The extensive search of the numerical values of the fuzzy measure coefficients, what is known in the context of optimization as enumerative search [54], constitutes the first alternative.

30


Since the non linearity of the fuzzy integral leads to non intuitive results, which difficult the comprehension of the parameter modification process, the heuristic extensive search is not recommended. On the other hand the automated extensive search of the fuzzy measures coefficients is very expensive in terms of execution time, even for applications considering a relatively small number of information sources. The lack of efficiency of the enumerative search is well known [54]. The second procedure takes into account the utilization of questionnaires in the determination of the fuzzy measure coefficients [114]. A pool of experts are requested to go through the different characteristics of the problem. Based upon the result of the questionnaires, an evaluation model of the problem at hand is established. This methodology increases the time for the development of the system and is not feasible in most industrial applications. Finally a procedure is presented in the literature [58], wherein the numerical features of the problem are represented in form of histograms and scatter plots. In this way the characterization of the information sources is based on the characterization of the overlapping among the different features and of its capability to represent a particular group of analyzed items. The lack of a numerical relationship between these qualitative factors and the resulting values of the coefficients, let the procedure just become an improved version of the heuristic extensive search.

2.7.2 Automated assessment of coefficients The automated characterization of the a priori importance of the information sources speeds up the implementation stage of a system where the fuzzy integral is employed. Furthermore it helps in redefining the system in case of a change in the importance relationship among the information sources, the inclusion of new ones, or a modification of the problem specifications. Taking all these facts into consideration, it can be stated that the successful employment of the fuzzy integral in real applications depends on the existence of methodologies for the automated construction of fuzzy measures. The automated completion of this process can be compared to the development of procedures for the automated determination of fuzzy membership functions for fuzzy control, where the appearance of a wide palette of methodologies attaining this goal, encouraged the feasible application of Fuzzy Set Theory in a great number of application fields [83]. On the contrary, there is a lack of efficient methods for constructing fuzzy measures [83]. A taxonomy of the methodologies for the automated construction of fuzzy measures is presented in [83]: 1. Usage of Sugeno and Choquet fuzzy integrals. Given a known fuzzy measure µ0 and a set of characteristics the new fuzzy measure µ should satisfy, a Choquet or a Sugeno fuzzy integral is applied on a function f in order to

2.7 Construction of the Fuzzy Measures

31

find out µ, thus applying: µ = Fµ0 (f ). In this procedure f should be also determined, for instance from given data. Since f presents just n unknown coefficients this results in reducing the number of coefficients to be assessed. The procedure is extensively described in [179]. 2. Constructing fuzzy measures by transformations. Given a fuzzy measure µ0 , the new fuzzy measure µ is found by applying a mathematical transformation θ on the given measure as µ(A) = θ[µ0 (A)]. Here the transformation pursues modifying the properties of µ0 in order to obtain a µ with some other particular properties needed for the problem at hand. A tabular description of the transformation functions θ to be used can be found in [83] together with their characteristics. 3. Data-driven construction methods. In this case the methodology proposes determining µ by solving the inverse problem on given data. Hence the different values to be integrated xi , ∀i = 1, . . . , n are supplied together with the expected fuzzy integral result Fbµ (x) in a training data set. Different data-driven procedures are applied on this data set in order to find out the coefficients of µ that minimize the difference between the expected and the current result of the fuzzy integral as expressed by: ǫ = |Fbµ (x) − Fµ (x)|.

(2.11)

Heretofore the following methodologies has been used for this purpose: numerical optimization [57][168][178], random search with simulated annealing [191], probability analysis [77], neural networks [25][79], and genetic algorithms [23][83][180]. 4. Utilization of modal logic. This methodology is based on the interpretation of fuzzy measures through the usual semantics of modal logic. The intuitive example given in Sec. 4.2 considers the determination of the coefficients based on common sense reasoning, what constitutes a similar approach to the modal logic. 5. Utilization of uncertainty principles. In this case fuzzy measures are specially considered as means for representing uncertainty. Fuzzy measures are computed by following different principles for the management of uncertainty (minimum uncertainty, maximum uncertainty and uncertainty invariance) and by taking into consideration a meaningful measure of uncertainty.

32


As stated in [83] all the categorized strategies need further research in order to become standard procedures in the construction of fuzzy measures. Among them Soft Computing methodologies, which belong to the data-driven strategies, are successfully being used in practical applications [23][25][79][177]. This type of strategies present the advantage with respect to e.g. transformations, extensions, etc., that are not based on a previously defined fuzzy measure. Moreover the positive features of Soft Computing for the resolution of applications in Computer Vision encourage its application to this problem as well.

2.8 Soft Computing methodologies for the construction of Fuzzy Measures In the following section, different data-driven procedures for the construction of fuzzy measures that belong to the theoretical framework of Soft Computing (see Sec. 2.4) are detailed. Within this theoretical framework the fields of Neurocomputing [25] [79] and Evolutionary Computation [23][83][180] are taken into account. Furthermore the methodologies denoted as Self-Organizing Features Maps (SOM) and Genetic Algorithms (GA) are selected for further development.

2.8.1 Genetic Algorithms Genetic Algorithms were already presented within Evolutionary Computation (see Sec. 2.4). In this context, the term Genetic Algorithms groups some methodologies that can be characterized as follows. The general goal of Genetic Algorithms is the determination of a sub-optimal, but sufficient solution to a particular problem [54]. For that purpose genetic laws are taken as the inspiring biological framework. Solution candidates are codified and grouped in a set. The solution candidates, the codification, and the set respectively receive the name of individuals, gene, and population by following the biological terminology. Furthermore, the driving force of the searching procedure, as recognized by Holland [75] in natural systems, is the tension between exploitation of the already found solutions and the exploration of new ones [109]. In Genetic Algorithms this tension is represented by the employment of socalled genetic operators (see Fig. 2.14), which iteratively operate on the population in the way leading to the solution. The exploration in the search space is implemented by so-called mating and crossover operators, whereas mutation and selection operators are related to exploitation [54]. In the general procedure, the discrimination among the solutions being found relies on the so-called fitness function. This plays the role of indirectly building up the conditions to be fulfilled by the final solution. The basic procedure of a generic genetic algorithm is depicted in Fig. 2.14. In the first stage the members of the population are gathered in groups of two

2.8 Soft Computing methodologies for the construction of Fuzzy Measures

33

elements through the mating operator. In the next stage, implemented by means of the crossover operator, the genotype of the mated individuals are split apart and recombined. Thence some of the genes of the new individuals are randomly modified through the mutation operator. Finally the fitness of these individuals is computed, whereby the selection operator discriminates among the new individuals. Those with a fitness large enough are appended into the old population by substituting the individuals that present a lower fitness, whereas the rest are lost. This basic procedure is iteratively repeated till a particular stopping condition is fulfilled. 001010 101101 001101 001001 111000 010111 010000 001010 Generation n

001010

111000

000001

101101

010111

011110

001101

010000

011101

...

101001

Crossover

001001 1

010100 0

011101

...

101101 0

Mutation

...

110001 Mating 101011

Fitness 010100 001010 101101 001101 001001 111000 010111 010000 001010

...

101101 Selection

Generation (n+1)

Figure 2.14: General flow of a genetic algorithm: a set (population) of coded (genes) possible solutions (individuals) sequentially undergoes: grouping in two element sets (mating), exchange of a part of the coding in these elements (crossover), modification of some of the coding bits of the resulting elements (mutation), computation of the degree of fitting to the optimal solution (fitness assessment), and selection of those elements nearer to the optimal solution (selection). Taken from [87] with permission.

Application in the construction of fuzzy measures Genetic algorithms have been successfully used for constructing fuzzy measures in different application fields [23] [83] [180]. The fuzzy measures coefficients are coded by arrays of real numbers. The general methodology of Genetic Algorithms with standard mating, crossover, mutation, and selection operators [54] applies for this problem. Choosing one or another particular type of genetic operator is

34


heuristically done, and no general assessments can be made on this point based on the available literature [23] [180]. The fulfillment of the monotonicity axiom of fuzzy measures (2.6) constitutes an important question to be solved in the application of genetic algorithms when constructing fuzzy measures. The problem can be generally described as a constraint in the search space, since some of the possible solutions to be found are not allowed in the context of the fuzzy integral. The utilization of any particular type of measures (see Sec. A.3) constitutes the first alternative used in order to overcome this problem. In this case only the fuzzy densities µi are to be found, what simultaneously reduces the dimensionality of the search space. Due to its greater flexibility in the interaction characterization, fuzzy-λ measures represent the best alternative in this case (as used in [125] [177]). If this flexibility is not enough, the utilization of general fuzzy measures is advised. In case general measures are being constructed, a conformance function downplays the fitness of those individuals, whose fuzzy measures do not fulfill the monotonicity relationship among the coefficients [23]. In this work, a monotonicity conformance in form of a trapezoidal fuzzy membership function δ() is defined over the coefficient values µ(Ai ) as:  µ(A )−µ− (A )+Cp (i) (i)  : µ− (A(i) ) − Cp ≤ µ(A(i) ) < µ− (A(i) )  Cp   1 : µ− (A(i) ) ≤ µ(A(i) ) < µ+ (A(i) ) δ(µ[A(i) ]) = , µ+ (A(i) )−µ(A(i) )+Cp + +  : µ (A ) ≤ µ(A ) < µ (A ) + C  p (i) (i) (i) Cp   0 : otherwise (2.12) where the parameter Cp defines the tolerance interval of the conformance function. This parameter gradually decreases as the search process completes. Moreover µ− (A(i) ) and µ+ (A(i) ) are respectively the maximum of the linked coefficients in the previous layer and the minimum of the linked coefficients in the posterior one (see Fig. 4.4): µ− (A(i) ) = ∨[µ(A(i−1) )] µ+ (A(i) ) = ∧[µ(A(i+1) )].

(2.13) (2.14)

The overall conformance of each individual is computed as the minimum of the conformance degrees of all its genes and used as a weighting factor in the computation of its fitness. Therefore the selection procedure filters out those individuals presenting a non-monotonicity out of the tolerance interval.

2.8.2 Neural based strategies The SOM [85] paradigm presents an unsupervised neural network based on the biological model of Hebbian learning [68] and on some structural properties of

2.8 Soft Computing methodologies for the construction of Fuzzy Measures

35

so-called brain maps [85] (see Sec. D.2.3). Self Organizing Feature Maps (SOMs) are mostly exploited as approximation and representation methodologies [85]. A SOM seeks the representation of a particular data set through a grid of nodes. The nodes of the grid contain vector data, which are denoted in this context as the weights of the neural network. These weights present the same dimension as the data being analyzed. In order to obtain a representation of the incoming data, the neural network first goes through the so-called training phase, which can be described as follows. Once the weights are randomly initialized, a distance function between them and one data sample is computed. Thence the weights of the node with a minimal distance to the data point mc () are adapted to this particular data point by applying [85]: mi (t + 1) = mi (t) + α · hci (t)[x(t) − mi (t)],

(2.15)

where hci (t) is a neighborhood function changing over time and α the learning rate. Furthermore hci(t) determines which neighbors of the winning node are modified as well. This process is repeated iteratively on the training data set till convergence is reached. As a result, an approximation of the probability density function of the incoming data in the form of an ordered two-dimensional map is obtained [85]. Therefore, the topological properties of the output map resulting from the application of a Self Organizing Feature Map (SOM) are a consequence of its statistical properties as representational tool [85]. Application in the construction of fuzzy measures As formerly mentioned (see Sec. 2.6), there is an example of employment of a SOM neural network for the construction of fuzzy measures [25]. The referred application tackles a handwritten recognition problem through a hybrid fuzzyneural system. The system can be subdivided into two subsystems respectively attaining the recognition of characters and the recognition of handwritten words. The former subsystem, which is used in order to compute the membership degree of the input pattern in 28 character classes, is the one presenting the hybrid combination of a SOM and a Choquet Fuzzy Integral. The SOM is first used in order to project the 100-dimensional data. This projection allows obtaining the fuzzifying functions of the input variables by applying the expression: h(Sk , i) =

1 , 1 + dηiki

(2.16)

where ηi is the average distance of the prototypes forming a cluster when the input pattern Sk is input and the center of this cluster υi , and dik stands for the distance between Sk and υi . The SOM is employed in the construction of a fuzzy-λ measure. This measure is used in a Choquet Fuzzy Integral, which is applied over the prototypes forming

36


a connected region R of prototypes belonging to a class C, in order to find the membership function of Sk to the character of this class. The coefficients of the prototypes i are computed through: µR,C = i

NCi , N i · BCi

(2.17)

where NCi is the number of times that class C wins at prototype i, N i is the total number of times prototype i wins, and BCi a factor expressing the number of neighbors-4 of prototype i at which class C wins at least once. Thus the coefficients are computed upon the concept of relative winning frequency. Finally, it is worth pointing out that all the hybrid subsystem operates in a supervised fashion.

37

3 Dissertation’s motivation and framework definition After summarizing the state of the art in multi-sensory Computer Vision and in fuzzy fusion operators, the following section gives an overview of the work developed in the here presented dissertation. First, the motivation of the different developments is given. Second, the goals to be attained are described.

3.1 Problem statement The complexity involved in the implementation of computer vision applications often requires the consideration of more than one information source in the attainment of a particular stage in the processing chain (see Fig. 2.2). This multisensory approach in computer vision concerns any of the processing stages from the image acquisition to the decision making one. Furthermore, the state of the art in computer, sensor and processing technologies encourages the implementation of multi-sensory systems in Computer Vision. In some multi-sensory systems, the diversity of gathered information needs from a dimensionality reduction in order to achieve the final goal. The dimensionality reduction should manage to avoid any loss of information and further to achieve the gain of new one. These are the basic goals of information fusion methodologies. Nevertheless, there is a need for further developing the field of information fusion from a theoretical point of view before attaining its practical aspects. This will be accomplished within the application field of Computer Vision in the here presented dissertation. Fuzzy fusion operators, which were developed in the theoretical framework of Soft Computing particularly in the field of Fuzzy Computing, constitute good candidates for the implementation of the fusion stage. On the one hand, these operators were specially thought for the resolution of fusion problems. On the other hand, they present some positive features in terms of flexibility, robustness, and interpretability. Among the fuzzy fusion operators, these positive features can specially be found in the fuzzy integral, which mathematically generalizes most of them. In spite of these positive features, fuzzy fusion methodologies in general, and particularly the fuzzy integral, are not widely employed in computer vision sys-

38

3 Dissertation’s motivation and framework definition

tems. This fact may rely on two main points. On the one hand, the mathematical background of the fuzzy integral, which is based on the Fuzzy Measure Theory [62] [179], can act as a discouraging factor in its employment. On the other hand, there is a lack of practical work on the particular utilization of the fuzzy integral for Computer Vision. The implementation of the fuzzy integral for information fusion in computer vision requires the further theoretical and practical development of this fusion operator.

3.2 Goals to be attained in the here presented work The successful implementation of computer vision systems with information fusion based on the fuzzy fusion operators depends on the attainment of the following goals: • The field of multi-sensory computer vision and, especially, the theoretical background on the process of multi-sensory fusion has to be further developed. Multi-sensory fusion is related to the fusion operation in the existent model of multi-sensory integration [99] (see Sec. 2.3.1). This functionality has to be extended in order for other procedures of this model, which are intimately related to the multi-sensory fusion, to be globally taken into consideration. Furthermore the information gain of the multi-sensory fusion operation [1] have to be further specified and, if possible, formally defined. • The relative large number of fuzzy fusion operators (see Sec. 2.5) have to be embedded in a common theoretical framework in order to ease its employment from an engineering point of view, and the process of selection of one or another operator. Specially the utilization of the fuzzy integral have to be analyzed from this perspective in order to ease the exploitation of its positive features as fusion operator. • The fuzzy integral have been developed as fusion operator in a mathematical context (see Sec. 2.5.2). The employment of this operator in Computer Vision can be tailored in order to adapt these properties to the processing of images. The fuzzy integral have to be further developed taking this fact into consideration. • The employment of the fuzzy integral has mostly succeeded in computer vision systems by the mere application of the operator, i.e. few processing frameworks have been developed based on this fuzzy fusion operator (see Sec. 2.6). Such frameworks can help exploiting the capabilities of the operator for different applications of computer vision systems. Therefore more complex processing frameworks based on the application of the fuzzy integral have to be developed.

39

3.3 Methodology

• Among these processing frameworks there are few methodologies for constructing fuzzy measures. All the existant methodologies in this context are supervised (see Sec. 2.8). It is especially important to count with methodologies that function in an unsupervised fashion, which are particularly needed for the implementation of industrial applications of computer vision systems. • The performance of the different frameworks based on the fuzzy integral have to be eventually evaluated on hand of industrial applications. This evaluation can give an idea of the feasibility of its employment. The aforementioned goals have been summarized in Fig. 3.1, where the difference between theoretical and application related goals can be observed.

New model multisensory fusion Theoretical Engineering perspective on fuzzy fusion

Simplified theoretical framework for fuzzy fusion Developing operators new operators for applications in computer vision

Processing frameworks based on FI

GOALS Application

Automated determination of FM Performance evaluation

Figure 3.1: Goals attained in the here presented dissertation. The goals succeed both in a theoretical and application related context.

3.3 Methodology In order to achieved the formerly described goals different developments, implementations and evaluation tests have been realized. These are herein described as follows:

40

3 Dissertation’s motivation and framework definition • A new model of multi-sensory fusion, which further develops the theoretical background on this process for systems with multi-sensory integration is presented in Sec. 4.1. • The theoretical framework denoted as soft data fusion, which is developed in order to serve as a common platform for the employment of fuzzy fusion operators in engineering systems, is elucidated in Sec. 4.2. Moreover Sec. 4.3 describes the employment of the fuzzy integral from an engineering point of view. • The paradigm Intelligent Localized Fusion (ILF), whereby the fuzzy integral is further developed for its particular utilization in image processing applications, can be found in Sec. 4.4. • Sec. 4.5 presents three different soft computing methodologies for the assessment of the fuzzy measure coefficients. These are based on a further development of existant genetic algorithms (Sec. 4.5.1), the utilization of interactive genetic algorithms (Sec. 4.5.2), and an unsupervised methodology based on the utilization of SOMs (Sec. 4.5.3). In this context it is worth pointing out that interactive genetic algorithms have not been previously used for the construction of fuzzy measures. Moreover the unsupervised construction of fuzzy measures based on the application of a SOM constitutes a novelty as well. • Finally, different computing framework, which realize diverse image processing (Chap. 5) and analysis applications (Chap. 6), are develop and evaluated. Color edge detection (Sec. 5.1), pre-processing for highlights filtering (Sec. 5.2), mathematical morphology for color images (Sec. 5.3), and color image segmentation (Sec. 5.4) are the image processing tasks achieved by the developed frameworks. These frameworks are respectively applied for remote sensing, the visual inspection of high-reflective materials, the visual inspection of textiles, and the processing of document images. On the other hand, two different frameworks for image analysis, which attain the segmentation of multi-dimensional images (Sec. 6.1) and the detection of texture faults (Sec. 6.2), are developed. Their performance is evaluated in two industrial applications for market basket recognition and the visual inspection of collagen plates.

41

4 Development of a Computer Vision System with Soft Data Fusion Although the employment of the fuzzy integral in computer vision applications succeeded at a relative early stage of research, a large number of processing frameworks for information fusion still use to take into account more traditional fusion operators. This fact lays on different reasons. First the field of information fusion is an emerging research field, whose theoretical foundations are still being developed. Therefore the properties of the fuzzy integral for the implementation of fusion operations are not appreciated. Furthermore, the existence of few methodologies for the automated assessment of fuzzy measures hinders the development of full automated tasks using this fusion operator. Finally these shortcomings do not allow the number of frameworks for image processing and understanding based on the fuzzy integral increase. The situation becomes thus a vicious circle. This chapter tackles the two first mentioned problems and leaves the third one to be treated in the next two chapters. Thus it develops two different theoretical concepts concerning general aspects of information fusion, which are related to the employment of the fuzzy integral. Moreover, a new theoretical framework specially devoted to the application of the fuzzy integral in image processing is developed and described. Different novel methodologies for the construction of the fuzzy measures are presented in the last sections of the chapter.

4.1 New Model for Multi-sensory Fusion Taking into consideration the characteristics of the different fuzzy fusion operators, and especially of the fuzzy integral, (see Sec. 2.5) a new theoretical model for the process of multi-sensory fusion is developed. The model of intelligent multi-sensory fusion considers the fusion of information in a broader sense than the model traditionally used in systems with multi-sensory integration [99] (see Sec. 2.3.1). These systems consider the gain on information achieved in multi-sensory systems through the fusion operation

42

4 Development of a Computer Vision System with Soft Data Fusion

from a quantitative point of view. Thus it is an important goal in classical Data Fusion methodologies to preserve the information available from the individual sources, as well to complement it with a gain of information that can not be extracted from the information sources separately [1]. Summarizing it is stated that the sum of the parts should be quantitatively something more than just the summed parts1 . It is worth mentioning that this theoretical principle lays behind the Theory of Fuzzy Measures [179], since the generalization of classical measures succeeds by taking into account this point. Thus additive measures, e.g. probability measures P (), sustain that the union of two sets is so important from the informational point of view as the sum of the degree of importance of the separate sets, as stated by the expression: P (A ∪ B) = P (A) + P (B).

(4.1)

On the other hand, super-additive measures fulfill: µ(A ∪ B) > µ(A) + µ(B).

(4.2)

Super-additive measures are supposed to model the complementary interaction of two information sources [62] (see Sec. A.2 for a further theoretical discussion). By comparing the entropy of additive and super-additive measures the gain on information, which is posited in [1] and [99] just from a conceptual point of view, can be quantitatively assessed. Operators as the fuzzy integral achieve a gain of qualitatively different information beside the informational quantitative gain. In this case the goal is the attainment of a qualitatively more complete representation of the knowledge embedded in the multi-sensory data and in the system processing it. The point leading to the fulfillment of such a model is the usage of fuzzy fusion operators with both the automated determination of the fuzzifying functions and the automated construction of fuzzy measures. On the one hand, the utilization of fuzzy fusion operators increases the interpretability of the fusion operator, i.e. the behavior of the fuzzy fusion operator can be better understood by an external observer than the behavior of other fusion operators. This fact is based on the “real” meaning of the parameters in the fusion operation, i.e. fuzzy measures and fuzzy membership functions. Thus fuzzy membership functions allow assigning a linguistic descriptor to the analyzed features, and fuzzy measures state for the importance of these features. On the other hand, the implementation of the complementary procedures for the determination of the fuzzy functions and the automated construction of fuzzy measures leads the fuzzy fusion operator to attain the fusion of information, but the modeling and selection of sensors as well. This functionality is achieved in the classical model of multi-sensory fusion by 1

The quantitative gain of information on these systems is implicitly supposed by the existence of the system itself. If no information gain would be achieved, the fusion would not be considered. The statement is not supported by quantitative computations.

43

4.1 New Model for Multi-sensory Fusion

other modules of the multi-sensory integration (see Sec. 2.3.1), i.e. Sensor model and Sensor Selection (see Fig. 4.1). This statement is elucidated in the following paragraphs. MULTISENSORY INTEGRATION SEPARATE OPERATION SENSOR 1

SENSOR MODEL

SENSOR i

SENSOR MODEL

SENSOR n

SENSOR MODEL

SENSOR REGISTRATION

MULTISENSORY FUSION

WORLD MODEL

GUIDING/CUEING

SENSOR SELECTION SENSORY PROCESSING

SENSOR CONTROLLER

SYSTEM CONTROLLER

Figure 4.1: Functionalities of multi-sensory integration undertaken by an intelligent multi-sensory fusion procedure. A fusion operator following this model realizes the modeling (Sensor Model ), the selection (Sensor Selection), and the fusion of the sensor information (Multisensory Fusion) itself. The computation of fuzzy functions on the incoming data previous to the fusion, which can be considered as a part of the methodology for the utilization of the fuzzy integral (see Sec. 2.5.2), extend the treatment of the uncertainty realized in processing systems with multi-sensory integration (see Sec. 2.3.1) by the module denoted as Sensor model (see Fig. 2.10). Thus the computed fuzzy functions overtake the role of the Gaussian probability distribution that customary models the error in the sensor data [99], whereby the flexibility and adaptability of the fusion operator increases. Furthermore the automated construction of fuzzy measures assumes the functionality of the Sensor Selection module of the classical multi-sensory integration. First the type of additivity fulfilled by the fuzzy measure after its automated determination enlighten the type of interaction among the information sources, i.e. information sources can be redundant, complementary or independent (see Sec. A.2). Qualitatively, the characterization of this interaction is an important point to be attained by an intelligent fusion procedure in order to obtain a more complex and complete information. This increases the interpretability of the processing system behavior. Furthermore the consideration of the sorting

44


operation in the fusion operator allows it to respond with increased adaptability to a change in the environment or to a sensor failure. Finally the coefficients of the fuzzy measures express the importance of the information sources. The interpretation of the results obtained through the automated construction of the fuzzy measure can speed up the redesign of the processing system with respect to the sensors to be integrated in it, i.e. a sensor with a low value in all the fuzzy measure coefficients can be eliminated. Taking all the formerly mentioned facts into account, it can be posited that the fusion of multi-sensory information is expected to involve both a qualitative and a quantitative gain of information. The use of fuzzy fusion methodologies encourages the fusion of multi-sensory information in the broader sense represented by the new model of intelligent multi-sensory fusion. Two factors influence the development of such a model: the type of fusion undertaken (see Sec. 2.3.2) and the characteristics of the used fusion operator. While the first point is application dependent, the selection of the fuzzy integral as fusion operator facilitates the fulfillment of the model, specially when used with automatically constructed fuzzy measures. This fact is related to the concept of soft data fusion, which is elucidated in the following section.

4.2 Soft Data Fusion Fusion operators traditionally used can be considered as hard. Fuzzy fusion operators were established as generalizations of classical ones (see Fig. 4.2). This mathematical generalization can be considered as a softening process of the operator [153]. The evolution of fuzzy fusion operators from harder to softer ones is based on the inclusion of an increasing number of factors in the fusion procedure. In classical operators the fusion result exclusively depends on the value being operated on. E.g. the result of the sum operator just depends on the summands and thus 1.9 + 3.1 is always 5. In the weighted operators another factor is taken into consideration, namely the a priori importance of the information sources. These operators are used for instance in neural networks, which are based on the application of the weighted sum. In the theoretical framework of Fuzzy Logic the new degree of softness is achieved through the parameterization of the aggregation, e.g. T- and S-norms (see Sec. 2.5.1), or the consideration of the ranking as a factor upon which the already mentioned a priori importance can be modified. This strategy is employed in Ordered Weighting Averaging (OWA) operators [187], whose mathematical expression is given in eq. (4.3), and increases the adaptability of the operators and its capability concerning compatibility, partial aggregation, and reinforcement

45

4.2 Soft Data Fusion

STAT. MOMENTS (AVERAGE)

RANKING OP. (MIN, MEDIAN, MAX)

LOGICAL OP. (AND, OR)

ALGEBRAIC OP. (SUM, PROD)

WEIGHTED SUM

classical operators

weighting operators

WEIGHTED RANKING (WMIN, WMED, WMAX) OWA T-, S-NORMS

fuzzy logic related operators

UNI-NORMS CHOQUET FI

SUGENO FI fuzzy integrals

T-CONORM FI

Grade of Hard-/Softiness

Generalization relationship

Figure 4.2: Relational map of different fusion operators. The grade of softness increase in the vertical axis from the top to the bottom and is a result of successive generalizations. In the horizontal axis the operators are grouped upon its flavor, which defines different families of operators. In the vertical axis, the operators are grouped upon different theoretical frameworks in operation research.

[190]. OWA =

Pn

i=1 wi x(i)

∀wi ∈ [0, 1]

Pn

i=1

wi = 1.

(4.3)

Fuzzy integrals reflect in the fusion result all this mentioned information: the value delivered by the different sources, their a priori importance, and their ranking. This can be exemplary observed in the mathematical expression of the Sugeno Fuzzy Integral: Sµ [h1 (x1 ), . . . , hn (xn )] =

n _

[h(i) (xi ) ∧ µ(A(i) )],

(4.4)

i=1

where µ(A(i) ) = µ({x(1) , ..., x(i) }). Thus a set of weighting coefficients is defined for each canonical region of the feature hyper-cube (see Sec. B.2). Moreover, fuzzy integrals are the only fusion operators in this theoretical framework taking into consideration the importance of the individual information sources and that of their possible coalitions [62]. Fuzzy integrals generalize most of fusion operators considered in the context of fuzzy sets (see Fig. 2.13). This generalization is further analyzed from a mathematical point of view in Sec. B.1. The following example is expected to elucidate the implications of using fusion operators with different degrees of softness in a simple operation. A woman

46


travels in a week, from home to the work place and back home, the following temporal intervals (in minutes): travel time = {mo, tu, we, th, f r} = {80, 40, 50, 60, 120}.

data sample value

The example can be described in a signal processing context as well, where a discrete signal (see Fig. 4.3) is fused with different operators in order to obtain different representative values. As it can be observed in the following paragraphs, each of these representative values leads to different decisions.

CFI OWA WSUM AVE

data sample index

Figure 4.3: Exemplary fusion of a discrete signal with different operators in order to obtain a representative value. The samples of the signal are successively fused with an average (AVE), a weighted sum (WSUM), an Ordered Weighted Averaging (OWA), and a Choquet Fuzzy Integral (CFI) operators. The weighting importance of the samples with a larger value is increased on each operator and thus the result follows this evolution.

The fabric, where she is working, have just moved to the present location. She decides to speak with the boss about an economical compensation. The boss applies an average on the travel times and rejects the proposition as the result (70.0) looks reasonable. Not being happy with the decision, the worker appeals to the trade union. Based on scientific studies, which state that the most annoying days for traveling are on the extremes and on the center of the week, the trade

47

4.2 Soft Data Fusion union expertise apply a weighted sum with weights: W = {0.3, 0.1, 0.2, 0.15, 0.25},

obtaining 77.0 of travel minutes per day, not such a great quantity to go on courts. The work colleagues decide to apply an operator which reflects better the fact that our worker needs to expend more time at home on the weekends because of some familiar problems they know about. They have heard about the works of Yager, and thus use an OWA with a distribution of weights, which ponders harder the days with a great quantity of travel minutes, and so they: 1. Sort the travel times in ascending order, travel time = {40, 50, 60, 80, 120}. 2. Apply the weights W = {0.1, 0.15, 0.2, 0.25, 0.3}. 3. Obtain result of 79.5 travel minutes per day Our worker knows about the complexity of such an evaluation and tries to use something called Choquet Fuzzy Integral she discovered in a science television program a time ago. She establishes following fuzzy densities for the individual days taking into consideration the annoyance of arriving so late on Monday and Friday at home: µmo = µf r = 0.2; µtu = µwe = µth = 0.1, and following for the coalitions (she knows also about monotonicity of fuzzy measures): µmo,f r = 0.4; µother two coalitions = 0.3; µmo,we,f r = µmo,tu,f r = µmo,th,f r = 0.5; µother three coalitions = 0.45; µmo,tu,th,f r = µmo,tu,we,f r = µmo,we,th,f r = 0.7 ; µother f our coalitions = 0.6. Since her husband work in another city and flies back on Friday to spend with her the weekend and the morning of Monday, she finds both Monday and Friday, or three consecutive days, really unbearable. She applies thus the following expression: CF I1 = 40 ∗ µtu = 4; CF I2 = 4 + 50 ∗ (µtu,we − µtu ) = 14; CF I3 = 14 + 60 ∗ (µtu,we,th − µtu,we ) = 23; CF I4 = 23 + 80 ∗ (µmo,tu,we,th − µtu,we,th ) = 35; CF I5 = 35 + 120 ∗ (1 − µmo,tu,we,th ) = 83.0

48


She experiences this way, that the CFI allows her to consider better each situation and combination of days. She decides to talk again with the boss. Summarizing it can be stated, that the utilization of fuzzy fusion operators allows the flexible representation of subjective reasoning present in evidence combination processes. Moreover, the admission of several operators and the possibility of considering different types of weighting overcomes the classical linear weighting approach. The flexibility in the weighting allows softer operators to cope better with the ambiguity stated by the Ugly Duckling Theorem (see Sec. 2.3.3). Among all fuzzy fusion operators fuzzy integrals play a preponderant role due to its generalization property with respect to other fuzzy fusion operators. Soft data fusion is the theoretical paradigm defined by taking all these facts into consideration.

4.3 Engineering with the Fuzzy Integral In the following section different practical issues related with the application of the fuzzy integral are described. The elucidation is first given in a more general sense, and then focused in the research field of Computer Vision.

4.3.1 The practical application of the Fuzzy Integral From a practical point of view the fuzzy integral is a weighted operator that uses so-called fuzzy measures as weighting functions. As in other engineering problems, the question is how to parameterize the operator taken into consideration. This process has been described as the process of construction of the fuzzy measures (see Sec. 2.7). In this context the different particular types of fuzzy measures, e.g. fuzzy-λ (see Sec. A.3 for a description of further types of fuzzy measures), are used from a practical point of view in order to decrease the dimensionality of the fuzzy measure search space [83]. Taking n information sources into account, the number of coefficients N to be determined in a regular fuzzy measure is 2n − 2, which results from the cardinality of P(X): n n n = 2n − 2. + ...+ + N = |P(X) − {∅, X}| = n−1 2 1

(4.5)

This implies an important computational cost when trying to automate the construction of the fuzzy measure. Since the different types of regular fuzzy measures fix the relationship among the fuzzy densities and the remaining coefficients through the type of additivity fulfilled by the fuzzy measure, the number of unknown coefficients reduces to n (see Sec. A.3). Fuzzy-λ measures constitute, due to the lack of a prefixed additivity condition as expressed by eq. (2.7), the most flexible alternative among the mentioned fuzzy measures types. Nevertheless fuzzy-λ measures present in front of general

49

4.3 Engineering with the Fuzzy Integral

fuzzy measures the disadvantage that the kind of interaction among the information sources, which is determined by the parameter λ, can not differ among the different subsets. Summarizing, it can be stated that the selection of one or another type of fuzzy measure presents a tradeoff between the flexibility in the characterization of the interaction among information sources and the number of coefficients to be determined. µ 0/

µ1

µ12

µ13

µ123

µ2

µ14

µ124

µ X

layer 0

0

µ3

µ4

µ24

µ23

µ134

µ1234

layer 1

µ234

1

µ34

layer 2

coe f f s

layer 3

coe f f s

layer n

n 1

coe f f s

4

coe f f s

n

4 1

4

4 2

4

4 3

2

n 3

n 4

4 4

Figure 4.4: Lattice structure of a fuzzy measure up to [60]. The data structure constitutes a graph with unidirectional links from µ{∅} to µ{X}. The nodes are occupied by the corresponding fuzzy measure coefficient µi . The sorting operation of the fuzzy integral, eqs. (4.4) and (2.10), fixes up the path for the selection of the fuzzy measure coefficients in a particular integration with respect to the fuzzy measure, e.g. a dotted line joins the coefficients selected when x2 > x1 > x3 > x4 . The graph presents n+ 1 layers, where n is the number of information sources, with ni coefficients in each level ∀i = 0, . . . , n. Each layer is occupied by the coefficients of the subsets with equal cardinality. From a computational point of view fuzzy measures present a complex structure that demands an explicit treatment of the data structures embedding a fuzzy measure. Hence, a lattice structure was proposed for this purpose in [60]. Such a graph structure (see Fig. 4.4) presents n + 1 layers, where n is the number of information sources, connected by a determined number of unidirectional links. The nodes of the graph are occupied by the coefficients corresponding to each of the subsets of P(X). Each layer is occupied by the coefficients of the subsets that present the same cardinality. The links connect the subsets among the different layers, which satisfy the inclusion relationship. By restricting the number of

50


paths among the different layers, the data structure assures the satisfaction of the monotonicity condition eq. (2.6) of the fuzzy measure. The usage of the lattice structure optimizes the computation of the fuzzy integral. Since in the computation of the fuzzy integral different sets of coefficients have to be used depending on the current ranking of the information sources, the employment of the lattice structure allows systematizing the computation and the extraction of the coefficients corresponding to this ranking. Thus the fuzzy integral can be computed in an iterative way by following the links in the graph structure. The employment of look-up tables constitutes an alternative to this data structure. Nevertheless this option demands the computation of all the fuzzy measure coefficients prior to the computation of the fuzzy integral, what slows down the processing time in case of a large number of information sources (n). Algorithm 1 : Iterative algorithm for the computation of the fuzzy integral on an image. The algorithm makes use of the lattice structure of fuzzy measures (see Fig. 4.4). construct a fuzzy measure F M for all pixels P in image do FI ← 0 current node ← µ{∅} 5: µcurrent ← 0 sort pixel channels for all element CH in sorted sequence do µprior ← µcurrent follow link in lattice corresponding to the current element 10: µcurrent ← coefficient in current node if Choquet Integral then µcurrent ← µcurrent − µprior end if T R ← T norm(CH, µcurrent) 15: F I ← Snorm(F I, T R) end for P ← FI end for An algorithm of the fuzzy integral, which employs the mentioned lattice structure for the fuzzy measure, is presented (see Alg. 1). The presented algorithm is iterative. As already mentioned, the used T- and S-norms are defined by the type of fuzzy integral employed (see Sec. 2.10). The content of “image element” and “data in the information channels” depend on the type of information fusion undertaken (see Sec. 2.3.2), e.g. for a pixel fusion these will be respectively pixel and the pixel values in each channel, for a feature fusion, the data point and the

51

4.3 Engineering with the Fuzzy Integral µ 0/

0

0

µR

µG

µB

µ 30R

µ 80G

µ 100

µ RG

µ RB

µ GB

µ117 RG

139

205

µ X

µ RGB

1

255

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

Figure 4.5: The algorithm presented in Alg. 1 is exemplary undertaken as a SFI on the color channels of the image depicted in (a) with respect to the fuzzy measure (b). The coefficients of this fuzzy measure present the values (c). The squares’ colors in (c) are used for labeling the areas of the images, where the coefficients result from the corresponding operation. The result of the different steps are depicted after operating with the algorithm for all pixels in the image. (d) Red channel of the input image. (e) Green channel of the input image. (f) Blue channel of the input image. (g) Maximum of the color values after sorting (after Alg. 1, line 6). (h) Median of the color values after sorting. (i) Minimum of the color values after sorting. (Remaining steps in Fig. 4.6)

52


(a)

(b)

(c)

(d)

(e)

(f)

Figure 4.6: The algorithm presented in Alg. 1 is exemplary undertaken as a SFI on the color channels of the image depicted in Fig. 4.5a with respect to the fuzzy measure, whose lattice is depicted in Figs. 4.5b and c. This last figure presents the pseudo-colors used in the following figures. (a) Coefficients of the first level in the lattice (see Fig. 4.5c) selected up to the result of the sorting operation (after Alg. 1, line 10, for the first CH in the iteration). (b) Coefficients of the second level in the lattice (see Fig. 4.5c) selected up to the result of the sorting operation (after Alg. 1, line 10, for the second CH in the iteration). (c) Result of the S-norm in the first iteration (after Alg. 1, line 15, for the first CH in the iteration). The circles indicate the areas where the original grayvalue of the largest color pixel (see Fig. 4.5g) win in front of the value of the coefficient (see Fig. 4.6a). (d) Result of the S-norm in the second iteration (after Alg. 1, line 15, for the second CH in the iteration). The circles indicate the areas where the value of the former result (see Fig. 4.6c) win in front of the value of the coefficient (see Fig. 4.6b). (e) Result of the S-norm in the third iteration (after Alg. 1, line 15, for the third CH in the iteration) and thus final result of the SFI. The circles indicate the areas where the value of the former result (see Fig. 4.6d) are modified by this last step. (f) Final result of the SFI without pseudo-color.

53


components of the feature vector. Exemplary the transformations suffered by an image undergoing the presented algorithm are depicted in Figs. 4.5 and 4.6 for the case of a Sugeno Fuzzy Integral. The result of the different steps can be observed in order to gain a more intuitive understanding on its effect in the image domain.

4.3.2 The Fuzzy Integral in Computer Vision Computer Vision is one of the engineering fields, where the positive features of the fuzzy integral can be exploited. In this following section, the special features of the employment of the fuzzy integral in computer vision applications and its possibilities will be analyzed. The analysis will be separately undertaken for Image Processing and Image Analysis due to the different quality of the data processed by systems in both application fields (see Sec. 2.2).

(b)

(c)

(d)

(e)

(f)

(g)

(a)

Figure 4.7: Example of the finer discriminating capability of fuzzy measures with respect to other weighting schemes. The R, G and B channels of the input image (a) are fused through a weighted sum (b-d) and a CFI (e-g). (b) Result for weights wR = 0.12, wG = 0.24, wB = 0.36. (c) Result for weights wR = 0.12, wG = 0.24, wB = 0.47. (d) Difference image of (b) and (c). (e) Results for fuzzy measure coefficients µR = 0.12, µG = 0.24, µB = 0.36, µRG = 0.59, µRB = 0.71, µGB = 0.82. (f) Results for fuzzy measure coefficients µR = 0.12, µG = 0.24, µB = 0.47, µRG = 0.59, µRB = 0.71, µGB = 0.82. (g) Difference image of (e) and (f).

54


Image Processing Being the fuzzy integral a fusion operator, its basic function in Image Processing is the reduction of the dimensionality of a pictorial representation. Thus n image channels are reduced to a single channel image through the fuzzy integral. This operation is used in different application fields with different types of images, where the previous fuzzification stage is not compulsory. The features of the resulting images depend on the parameterization through the fuzzy measures and on the used type of integral. The usage of fuzzy measures as the weighting scheme in the fuzzy integral improves the discrimination of the operator with respect to the multidimensional data. The consideration of the canonical regions of the hyper-cube in the weighting scheme (see Secs. 4.2 and B.2), leads its weights to present a finer discriminating capability than those of other fusion operators. The pictorial example in Fig. 4.7 can help in the comprehension of that effect. The behavior for image fusion of the Sugeno Fuzzy Integral eq. (4.4) can be intuitively understood as a multi-threshold in an image processing context. The constant regions in the hyper-cube generated by the values of the different coefficients (see Sec. B.3) makes the pixel data falling in these regions to present a crisp value at the output of the operator (see Fig. 4.8b). This effect lets the operator be successfully employed in segmentation problems [129][170].

A

C D B

(a)

(b)

(c)

Figure 4.8: Exemplary result of the Choquet and Sugeno Fuzzy Integrals on an image with a red-blue transition. (b) Result of the Sugeno Fuzzy Integral with respect to a regular possibility measure with µR = 0.47, µG = 0.001, µB = 0.785. The areas A and B present crisp values because they are caught in the constants regions of the hyper-cube (see Fig. B.2). (c) Result of the Choquet Fuzzy Integral with respect to the same measure. The difference in the number of levels between areas C and D is due to the difference in the fuzzy measure coefficients. The larger the coefficient of the corresponding channel, the larger the gradient of the level curve for that channel (see Fig. B.3). In case of the Choquet Fuzzy Integral eq. (2.10), the coefficients control the

55


gradient of the level curve (see Sec. B.3). Therefore the discrimination increases in those channels with larger coefficients. In other words a variation in the features or in the features subset with larger coefficient will influence more the result than a change in a feature with a smaller coefficient (see Fig. 4.8c). Image Analysis Although the field of Image Analysis is a bit wider (see Sec. 2.2), the present section will be centered on the usage of the fuzzy integral as classificator. This is justified since in most of the frameworks used for Image Analysis, the classification stage plays a principal role (see Sec. 2.2). The fuzzy integral for the resolution of classification problems has been employed in two different theoretical frameworks: possibility theory and information fusion [60]. A classification task can be defined as follows. Suppose n different information sources, delivering a feature vector x with components xi ∀i = 1, . . . , n. The classification problem is solved through a mapping function c on x, which delivers for each data point to be classified a label vector C formed by m components C j , ∀j = 1, . . . , m. Different types of classifiers, i.e. possibilistic (Cp ), probabilistic or fuzzy2 (Cf ), and crisp (Cc ), can be defined with respect to the interval where the label vectors are defined [13]: Cp ∈ ℜ m C j ∈ (0, 1] ∀j = 1, . . . , m (4.6) m X j Cf = Cp C =1 (4.7) Cc ∈ ℜ

m

j=1

C j ∈ {0, 1}

∀j = 1, . . . , m..

(4.8)

The fuzzy integral realizes a possibilistic classification by computing the integral with respect to m different fuzzy measures µj on the feature vector x. Therefore this operation delivers the components C j of the label vector, which fulfill eq. (4.6). The two theoretical approaches, wherein the fuzzy integral is used as a classifier, namely the possibilistic and the information fusion, differ in the type of information being fused by the fuzzy integral3 . This difference results in two different frameworks for classification through the fuzzy integral (see Figs. 4.9 and 4.10), where the fusion operator is seen from two different perspectives. In the possibilistic approach (see Fig. 4.9), which receives also the name of fuzzy pattern matching approach [60], the fuzzifying functions, denoted as hi (x) in eqs. (4.4) and (2.10), are possibility distributions computed with standard 2

The distinction between probabilistic and fuzzy classifiers is of a qualitative nature and not of a numerical one. See [13][14] for a detailed discussion on this topic. 3 The terms possibilistic and information fusion do not denote the type of classification accomplished by the fuzzy integral, since both approaches attain a possibilistic classification.

56


Defuzz

Figure 4.9: Block diagram of a generic classification system based on the fuzzy integral in the framework of the possibility theory. hji : possibility distributions. Fµ : Fuzzy integral. Defuzz: Module for Defuzzification.

methodologies, e.g Parzen windows [38], possibility histograms [35]4 . The sense of such functions is to express the degree to which individual attributes make the data points belong to each class defined in the label vector and expressed through C j . Thus the class membership degrees based on each individual attribute are fused through the fuzzy integral in order to obtain the final class membership degrees. In other words, the fuzzifying functions act as a pre-classifier, whose result is refined by the fuzzy integral.

Defuzz

Figure 4.10: Block diagram of a general classification system based on the fuzzy integral in the framework of information fusion. hi : fuzzy membership functions. Fµ : fuzzy integral. Defuzz: Module for Defuzzification. In the information fusion approach (see Fig. 4.10) the fuzzification through 4

Although this operation is undertaken from a practical point of view, the theoretical foundations are not well established. Methodologies for computing possibility distributions from data are an open research question (D. Dubois in a personal communication)

57


hi (x) assigns a linguistic attribute to the data present in the information channels. In some cases hi () are identity functions. Such an approach is more simple, since just one fuzzifying function for each attribute has to be generated. In this case the fuzzy integral combines the different linguistic expressed attributes for the final classification. Due to the nature of the fuzzy integral, a defuzzification stage (Defuzz ), which maps the resulting possibilistic label vectors of eq. (4.6) into crisp ones eq. (4.8), is needed in some applications. In this case the final classification is obtained in both approaches by turning the larger component of the label vector into 1 and the rest into 0. Although the employment of the maximum operator as just described is usual, other procedures could be used for the defuzzification stage. In both theoretical frameworks the fuzzy measures coefficients represent the degree of importance of the integrands for a classification, considered individually and in coalitions. Thus from a practical point of view, a fuzzy measure for each class has to be generated in the classification procedure, what will be denoted as: µj = {µj1 , . . . , µjn , µj12 , µj13 , . . . , µj1...n }

∀j = 1, . . . , m.

(4.9)

The exponentially increasing number of coefficients of the fuzzy measures makes the methodologies for its automated construction play a crucial role in the practical employment of the fuzzy integral in classification problems. In the Appendix C the capability of the fuzzy integral for classifying twodimensional Gaussian distributions is evaluated. The following conclusions can be derived from the exemplary results: • If the clusters are in different canonical regions, the clustering is possible and relative simple. • The results for different types of fuzzy measures are similar. No general assessment can be made on the type of fuzzy measure to be used within classification problems for the usage of the fuzzy integral. • The nearer one of the clusters is to the Ideal point (see Sec. B.8), the more difficult the clustering is. In this case the Choquet integral performs better than the Sugeno integral. • In the possibilistic theoretical framework such a situation is very common. Here the clustering capability mostly relays on the performance of the preclassification achieved by the possibility distributions hji (x). In this theoretical framework the data points belonging to the class, whose fuzzifying functions deliver all feature values near the Ideal, have a fuzzy integral result larger than the other classes. This fact is theoretically supported by the monotonicity of the operator with respect to the integrands (see Sec. B.5).

58

4 Development of a Computer Vision System with Soft Data Fusion • In the information fusion framework the feature extraction stage is more relevant than in the possibilistic approach. The different features are expected to be chosen previous to the clustering because they enhance some feature characteristic for some of the classes.

Summarizing it can be stated that fuzzy fusion operators and especially the fuzzy integral present very positive features for the implementation of computer vision systems, both for Image Processing and Image Analysis. Among them the flexibility and the discrimination capability of the operators, which are framed in the theoretical paradigm soft data fusion, play a principal role. These features are further developed in the paradigm presented in the following section.

4.4 Intelligent Localized Fusion Operators Operators for soft data fusion can achieve a higher degree of flexibility in Image Processing by defining local application domains of the weighting schemes. This is attained within the paradigm denoted as Intelligent Localized Fusion (ILF), which was presented in [152] as a new paradigm for fuzzy fusion in image processing. In this paradigm fuzzy integrals become so-called Intelligent Localized Fusion Operators (ILFO) by locally defining the employed fuzzy measure. Usually the a priori importance of the information channels, which is realized through the fuzzy measure coefficients, is evaluated in image processing applications from a global perspective. In other words, the fuzzy measure coefficients characterize the importance of each image channel and that of their possible coalitions taking the image as a unit (see Sec. 4.3.2). A novelty concerning this last aspect is presented in the ILF paradigm. Here the used fuzzy measure is not unique, but depends on the region where the fusion is undertaken, i.e. the fuzzy measure changes over the image. Therefore one can talk of “localized” fuzzy measures. The localization of the different fuzzy measures conforming a “localized” fuzzy measure is given through a label image, where the application area of each fuzzy measure is codified through a label (see Fig. 4.11). A fuzzy integral with respect to such locally defined fuzzy measures becomes a so-called Intelligent Localized Fusion Operator, ILFO. This approach is analogous to the application of the fuzzy integral in classification problems, where a fuzzy measure (µj ) is defined for each class C j ∀j = 1, . . . , m (see Sec. 4.3.2). What is the purpose of localizing the definition of the fuzzy measures? The utilization of “localized” fuzzy measures allows operating in the same image with different fusion strategies, thus exploiting the generalization property of the fuzzy integral with respect to other fuzzy fusion operators, which is modeled as soft data fusion (see Sec. 4.2). For instance, it could be interesting to use an OWA strategy in a particular area, while preferring a minimum operator for another one. Moreover the existence of such locally defined fuzzy measures reflects the

59

4.4 Intelligent Localized Fusion Operators

fact that the information sources do not necessarily have to interact in the same way all over the image, i.e. in some areas of the image the different channels can be redundant while in other ones independent or complementary (see Sec. A.3). Therefore the additivity property is locally defined. Both features are particular cases of a more general one related to the basic role of the fuzzy measures in the fuzzy integral, namely the characterization of the image channels importance (see Sec. 4.3.2). Since each pixel represents a different measure point in the spatial domain, this importance have not to be uniform in the image space. µ1 µ2

(a)

µ3

(b)

(c)

Figure 4.11: Exemplary usage of an ILFO. In the ILF paradigm a fuzzy integral with respect to different fuzzy measures is computed (see Fig. 4.7 for comparison with the normal application of the fuzzy integral). (a) Input image. (b) Example of a label image whereby the different fuzzy measures, e.g. µ1 , µ2 , and µ3 , are localized. (c) Fusion result with µ1 = {µR = 0.24, µG = 0.36, µB = 0.36, µRG = 0.59, µRB = 0.71, µGB = 0.90}, µ2 = {µR = 0.12, µG = 0.24, µB = 0.47, µRG = 0.59, µRB = 0.71, µGB = 0.82}, and µ3 = {µR = 0.12, µG = 0.24, µB = 0.36, µRG = 0.59, µRB = 0.71, µGB = 0.82}. Summarizing it can be stated that the label image splits the image space into different subspaces, where the fuzzy integral is operated with respect to different fuzzy measures. In this context it is noteworthy to take into consideration how ILFOs are mathematically defined and how they can be implemented in engineering systems.

4.4.1 ILFOs’ mathematical foundations The image space (x, y) is partitioned by the label image in different subspaces (x, y)j where different fuzzy measures µj are defined. The definition of these fuzzy measures attains the characterization of different importance relationships among the image channels in the corresponding subspaces. The image space is partitioned based on a process of feature analysis. The goal of this feature analysis procedure is the determination and extraction of a set of features upon which the importance of the information channels can be established. Finally the label image codifies the spatial distribution of the analyzed features over the different channels of the input image.

60


The label image should allow the definition of subspaces where the coalition of two channels could be less important than the channels individually taken in order to increase the flexibility in the application of the fuzzy integral. Since the result of Choquet and the Sugeno Fuzzy Integrals ranges from the result of the minimum of the image channels to the maximum of them (see Sec. B.7), the maximal flexibility would be achieved when the fuzzy integral operates as minimum in a subspace while operating as maximum in the other one. The generation of the label image can be formalized as follows. For the sake of simplicity the description is made for a two channel image (xi , i = 1, 2). Being the image space X, where X = {X1 , X2 }, a feature extraction procedure is applied on each channel set Ai . Thus a feature (or a group of them) a priori characterizing the importance for the fusion operation is firstly extracted. Thence thresholding the resulting feature maps leads to the definition of the following sets: F1 ∩ F2 = {x x presents the computed f eature} Fi = (4.10) {x xi presents the computed f eature} ¯ ¯ F1 ∪ F2 = {x x does not present the computed f eature}. If we suppose for instance that the feature analyzed influences the fusion result in a positive sense, this set definitions are equivalent to the following ones: F1 ∩ F2 = {x both channels are important f or the f usion} Fi = {x channel i is important f or the f usion } (4.11) ¯ ¯ F1 ∪ F2 = {x no channel is important f or the f usion }. Hence these sets can be related to the different levels of the fuzzy measure defined in the image space (see Fig. 4.12). µ 0/

¯2 F¯1 F

µ1

µ2

µ X

µ12

Fi i

1 2

F1 F2

Figure 4.12: Relationship among feature sets and levels in the lattice structure of a fuzzy measure which is used for deriving the expression of an ILFO. The intersection F1 ∩ F2 is related to the coefficient in the level 2, namely µ(A1 ∪ A2 ), the sets Fi are related to µ(Ai), and the union of the complements

61


F¯1 ∪ F¯2 to µ(∅). If the goal to be achieved by partitioning the image space is the increment of the flexibility of the fuzzy integral as described in the former paragraph, the monotonicity of the fuzzy measure have to be broken. This can not be achieved with just one measurable space as stated by (2.6). Therefore the image space is divided in three different subspaces (X, C j ) ∀j = 1, 2, 3, where three different fuzzy measures µj can be defined. The number of subspaces is determined by the number of levels to be singularly considered in order to overcome the monotonicity of the fuzzy measure (see Fig. 4.12). Thus these fuzzy measures can break the monotonicity condition by presenting: µ1 ({x1 , x2 }) ≤ µ2 ({x1 }) µ1 ({x1 , x2 }) ≤ µ2 ({x2 }).

(4.12) (4.13)

F3

F1 F2

(a)

(b)

(c)

(d)

(e)

C1 C2 C3 C4

Figure 4.13: Set diagram for the generation of a label image to be used in an ILFO, exemplary shown for three input channels. The process is based on the following set and class definitions: Fi = {xi presents computed f eature} ∀i = 1, 2, 3; C 1 = {F1 ∩ F2 ∩ F3 }; C 2 = {(F1 ∩ F2 ) − C 1 , (F1 ∩ F3 ) − C 1 , (F2 ∩ F3 ) − C 1 }; C 3 = {F1 − C 2 , F2 − C 2 , F3 − C 2 }; C 4 = {F¯1 ∪ F¯2 ∪ F¯3 }. (a-c) Binary maps resulting from the binarization of the corresponding feature map. (d) Set representation. (e) Resulting label image. The subspaces, where the different fuzzy measures are localized, can be defined upon the following classes: C1 = {F1 ∩ F2 } 2 C = {F1 − (F1 ∩ F2 ), F2 − (F1 ∩ F2 )} C3 = {F¯1 ∪ F¯2 }.

(4.14)

The set difference of the second class assures the classes to be disjoint, i.e. each point can just belong to one class. Finally it is worth mentioning that if the

62


analyzed feature influences negatively the fusion result the class definitions are made over F¯i instead of Fi . The generalization of these definitions for three channels is depicted in Fig. 4.13. The class definitions for a larger number of classes can be analogously found. The former definitions can be also generalized by applying the Fuzzy Set Theory [192] (see Sec. A.1). In this case, the classes become also fuzzy classes. Moreover the points of the label image present so many membership degrees as classes are defined. Therefore each point will present a membership degree for each of the classes, what will be denoted as ζj , ∀j = 1, . . . , m. The result of the ILFO is then a linear combination of the fuzzy integrals for each class: ILF O[h1 (x1 ), . . . , hn (xn )] = ζ1 Fµ1 + . . . + ζm Fµm ,

(4.15)

where Fµm = Fµm [h1 (x1 ), . . . , hn (xn )]. Since the Choquet fuzzy integral presents distributivity with respect to the scalar product (see Sec. B.4), the usage of this integral type allows simplifying the computation of the ILFO. In this case the ILFO will be computed as the Choquet fuzzy integral with respect to the fuzzy measure resulting from the linear combination with coefficients ζj of the fuzzy measures for each class, as expressed by: ILF O[h1 (x1 ), . . . , hn (xn )] = Cζ1 µ1 +...+ζm µm [h1 (x1 ), . . . , hn (xn )].

(4.16)

4.4.2 ILFOs’ Framework The following section presents the framework of a generic ILFO (see Fig. 4.14). The implementation of an ILFO is composed by different modules for the generation of the label image for the localization of the fuzzy measures (LabImGen), the construction of these measures (FuzMeCo), and the fusion of the image data through a localized fusion operator (FuzFus). Although the theoretical development takes into consideration the usage of a fuzzy integral for the implementation of this last module, the framework of an ILFO can employ other weighted fuzzy fusion operators, e.g. an OWA. While the theoretical background of the modules FuzFus and FuzMeCo are respectively analyzed in Secs. 4.2 and 4.5, the generation of the label image is elucidated in the following paragraphs. The block diagram of the module used for the generation of the label image (LabImGen), which was described in in Sec. 4.4.1, is shown in Fig. 4.15. In the first stage the feature of interest is analyzed and extracted (FeatExt). Thence the resulting feature map, which represents the spatial distribution of the feature, is binarized in each channel (Binar ). The obtained binary images enter afterwards an expert system, which implements the characterization of the spatial distribution of the different permutations of the binary maps (ExpSys). The resulting images of this stage are binary ones as well. Finally the binary maps of the different permutations are codified through different labels in order to be represented

63


LabImGen

Input images

FuzMeCo

Fused image

FuzFus ILFO

Figure 4.14: Framework of a generic ILFO. An Intelligent Localized Fusion Operator (ILFO) is implemented through the following modules. LabImGen: Module for the generation of the label image. FuzMeCo: Module for the construction of the localized fuzzy measures. FuzFus: Module implementing the corresponding fuzzy fusion operator, generally the Fuzzy Integral.

br 1 i1

fm 1

bm

1

l FeatExt

in

Binar

fm n

ExpSys

bm

Codif

n

br m

Figure 4.15: Block diagram for the generation of a crisp label image in an ILFO. Signals. ii : Input channels. f mi : Features maps. bmi : Binary feature maps. bri : Binary class maps for different relationships. l: Crisp label image. Modules. FeatExt: Feature extraction. Binar : Binarization. ExpSys: Expert system. Codif : Codification.

64


in a unique image (Codif ). The modules of the block diagram are described in the following. Through the FeatExt module a particular feature of the image is analyzed and characterized. A numerical expressed feature is extracted from each of the input channels in this feature stage. Examples of such a characterization are the extraction of a blurring coefficient or of the areas with low luminance. This feature extraction takes place in the image space, i.e. each pixel represents then its corresponding feature value. The feature extraction stage can be left in case the feature is already represented in the input channels (e.g. low luminance on grayvalue images), which are then directly binarized. In the binarization stage, which is implemented through the module denoted as Binar, the feature distribution maps are thresholded with a grayvalue θ. The parameter θ represents the point up to which the feature is considered to be important enough in order to influence the fusion result. Thus the resulting binary maps represent this importance through a binary variable (see Figs. 4.13ac). Furthermore this module implements the set definition exemplary given in eq. (4.10). The purpose of the expert system (in the module ExpSys) is the computation of the classes for the definition of the different fuzzy measures as stated for instance in eq. (4.14). The expert system generates the binary images whose true values indicate whether (and where) the feature is important in the first channel, in the second channel, in the first and second one, and so on. . . Afterwards these sets are grouped in the different classes. Therefore the intersection set operator of the expressions is implemented through a logical AND operator and the sets of each class are joint through a logical OR operator. Finally the module delivers a binary map for each class. fr 1 i1 FeatExt

in

fr 1 Fuzzif

fm n

l

fz1

fm 1

FuzExpSys

Codif

fz n frm

frm

Figure 4.16: Block diagram for the generation of a fuzzy label image in an ILFO. Signals. ii : Input channels. f mi : Features maps. f zi : Fuzzy feature maps. f rj : Fuzzy class maps for different relationships. l: Crisp label image. Modules. FeatExt: Feature extraction. Fuzzif :Fuzzification. FuzExpSys: Fuzzy expert system. Codif : Codification. The last module Codif takes the binary maps of the former module as input. Codif generates the label image (see Fig. 4.13e) by codifying the image points

65


of each class with a grayvalue. The outgoing label image will be used for the computation of the ILFO as already described. The block diagram of the procedure for the generation of a fuzzy label image is shown in Fig. 4.16. This procedure constitutes a generalization of the procedure for the generation of a crisp label image by taking into account Fuzzy Set Theory (see Sec. A.1). Thus the module Binar is substituted by Fuzzif. This module implements the fuzzification of the feature images with a monotonic increasing fuzzy membership function [24]. It delivers a set of fuzzy images to the module FuzExpSys. The fuzzification of the feature images is done taking into consideration a parameter ε that denotes the tolerance of the feature importance. The corresponding fuzzy function is centered in the grayvalue θ, which has been used in the generation of the binary feature maps through Binar (see Fig. 4.17). h(x) 1.0

0.5

0.0

θ

ε

θ

θ

ε

x

Figure 4.17: Tolerance parameter ǫ for the fuzzification of the feature images in an ILFO with a fuzzy label image. The parameter turns the step function centered on θ (dotted thick line), which would have been used as threshold for the generation of a crisp label image, into a ramp-shaped fuzzy membership function (continuous thick line). The rule system (ExpSys) becomes a fuzzy rule system, which is implemented in FuzExpSys. This module attains the treatment of the fuzzy images resulting from the former module (Fuzzif ). Therefore the implementation of FuzExpSys is realized by applying T-norm, S-norm, and fuzzy complement operators. The output of the procedure changes as well. A set of fuzzy images, which constitute the fuzzy label image, is delivered together with the crisp label image (see Fig. 5.19 for an exemplary comparison between the two approaches). The membership degrees contained in these fuzzy images will be used as the real coefficients in the linear combination expressed in eq. (4.15). It is worth pointing out that the employment of an ILFO increases the number of fuzzy measures to be constructed from one, i.e. µ, to j, i.e. µj . Therefore the existence of methodologies for the automated assessment of the fuzzy measure coefficients (see Sec. 2.7) becomes even more important within the ILF paradigm.

66


4.5 Soft Computing Methodologies for the Construction of Fuzzy Measures Revisited Different Soft Computing methodologies for the construction of fuzzy measures are formerly presented (see Sec. 2.8). The following section extends the number of methodologies that can be used for this purpose. These extension takes place within the theoretical framework of Soft Computing, what is formerly justified (see Sec. 2.7.2).

4.5.1 Genetic Algorithms Genetic Algorithms are already presented as a methodology for the construction of the fuzzy measures (see Sec. 4.5.1). A modification of an existent GA methodology is presented in this section. The novelty in the application of GAs for the construction of fuzzy measures is based on the simplification of the procedure presented in Sec. 2.8.1 for assuring the fulfillment of the monotonicity axiom given in eq. (2.6). Thus when constructing general fuzzy measures a simplified version of that procedure can be used. This simplification takes into consideration the computation of a non-monotonicity degree, which quantizes how apart is the set of coefficients of a particular fuzzy measures of being monotone. The non-monotonicity degree for each individual δi is computed as δi =

n X

∨[0, µ− (Ai ) − µ(Ai )],

(4.17)

i=1

where µ− (Ai ) is again the maximum coefficient of the previous level as stated by (2.13). This quantity is used in order to downplay the fitness degree of each individual without monotonicity considerations, which is denoted as φi. Thus the actual fitness of each individual is computed as φ′i = φi ± δi ,

(4.18)

where the sign of δ depends on the type of optimization (minimization / maximization) undertaken. Especial attention have to be devoted to the fitness function as well. However the fitness function to be used in each application is problem dependent. Thus only one general conclusion can be made related to this question. Genetic algorithms present a structure that encourage driving the search procedure in form of a trend instead of looking for an exact value of the fitness function. It is worth exploiting this property in practical applications for the constructions of fuzzy measures.

4.5 SC Methodologies for the Construction of Fuzzy Measures Revisited

67

4.5.2 Interactive Determination Recent developments in the context of evolutionary computation methodologies consider the substitution of the fitness function by the direct evaluation of a user [167]. Thus the searching process is not driven anymore by the analysis of a mathematical function, but by a human evaluator, who interactively influences the search through the expression of preferences on the population. This fact leads to the appearance of different methodologies generally denoted as Interactive Evolutionary Computation [167]. These offer an alternative to classical evolutionary computation procedures when the optimized item is to be exploited by a user or the assessment of a fitness function is too complex. This is the case for instance when the evaluated items are images, since the determination of a fitness function based on qualitative aspects of the image is not trivial. Thus Interactive Evolutionary Computation takes advantage of the expression of the fitness by a user and simultaneously exploits the positive strategy of an evolutionary search in the parameter space. This cooperative process is expressed as an implicit mapping between the parameter space and the psychological space of the user undertaking the evaluation [167]. Interactive Evolutionary Computation considers the interactive version of the computing paradigms included in Evolutionary Computation, namely interactive genetic algorithms, interactive evolutionary strategy, interactive evolutionary programming, and interactive genetic programming [167]. In the general methodology of interactive evolutionary computation the successive populations are presented to the user. This presentation can be made with the phenotypes of the individuals or alternatively with the system outputs generated by the parameters embedded in these phenotypes. Thence the user interacts with the evolutionary search procedure by rating the final results that can be obtained up to the different individuals in the population [167]. This interaction substitutes the fitness computation and/or the selection operator stages of a classical evolutionary computation strategy (compare Figs. 2.14 and 4.18). Up to this point though the normal sequence of genetic operators is applied as usual. This basic procedure is sequentially repeated till a solution in some generation is accepted by the user. The application of interactive evolutionary computation presents some conditioning limitations. First the number and resolution of the items to be evaluated is limited by different characteristics of the viewing device and the capabilities of the human evaluator. Thus small population sizes are normally considered. Furthermore a large number of generations can excessively tire or bore the human operator, decreasing its concentration. Since these limitations can affect the quality of the final result, the possibility to use interactive evolutionary computation depends on the application. No methodologies for the construction of fuzzy measures through interactive evolutionary methodologies have been presented heretofore. The construction of

68


001010 101101 001101 001001 111000 010111 010000 001010 Generation n

001010

111000

000001

101101

010111

011110

001101

010000

011101

...

101001

Crossover

001001 1

010100 0

011101

...

101101 0

Mutation

...

101101

010100 001010 101101 001101 001001 111000 010111 010000 001010

...

110001 Mating 101011

User

Generation (n+1)

Figure 4.18: Flow diagram of a generic interactive genetic algorithm. The user substitutes the fitness and/or the selection stages of a genetic algorithm (see Fig. 2.14). Modified from [87] with permission.

the fuzzy measures through an interactive evolutionary methodology is appropriate for those image processing applications using the fuzzy integral as fusion operator, where subjective features on the fusion result, e.g. image quality, have to be assessed. Since genetic algorithms are already used for constructing fuzzy measures, the utilization of interactive genetic algorithms is the more natural extension of this work. If the fitness function is based on subjective features of the image fusion result, substituting its computation by the direct evaluation of a user simplifies the assessment of the coefficients. This results from the fact that the fitness assessment rests in the psychological domain. Furthermore interactive genetic algorithms have been already used in image processing applications [119]. The most convenient alternative in this case is the presentation of the final image result obtained through the fuzzy measure coefficients codified in the individuals. Thus the surface of the displaying monitor limits the number of individuals of the population. Thus populations between 10 and 20 individuals are used, where the exact number of individuals depends on the size of the images and on the goal to be attained by the fusion. The number of generations is strong limited as well. Not only by the already mentioned factors affecting the concentration of the evaluating user, but also by the execution time of the fuzzy integral that can made a user wait too much time between assessment of the different generations. The possibility of directly selecting one of the resulting images and comparing it to the results of each new generation helps improving the performance of the methodology.


69

4.5.3 SOM based Strategies This section presents a methodological approach for the construction of fuzzy measures through Self-Organizing Maps (SOM) [85]. The difference with the system presented in [25], where a SOM is used for constructing the fuzzy measures of a fuzzy integral based classificator, is due to the unsupervised nature of the methodology presented in this section. This approach takes as starting point the apparent property of the SOM of capturing feature saliency. The goal to be attained is the construction of fuzzy measures in high-dimensional spaces, where the positive performance of the SOM is well known. Such an approach is specially relevant in the segmentation of textures, where the employment of large feature spaces5 is very usual, e.g. Gabor Filters. Furthermore the combination of a SOM and an LVQ for texture segmentation have been successfully used [85][135]. By substituting the LVQ with a fuzzy integral a fuzzy texture segmentation can be achieved. RD

V

BG

H

LD

Figure 4.19: The utilization of a SOM for the analysis of feature saliency inspired its employment for the construction of fuzzy measures in the fuzzy integral. The application of a SOM on Gabor features [29] generate output maps as the one depicted in the figure. The output map can be divided in areas of different orientation saliency: right diagonal (RD), vertical (V), horizontal (H), left-diagonal (LD), and background (BG), where no orientation is salient.

The usage of a SOM in [22] for the analysis of texture saliency on natural images inspired the employment of a SOM for the construction the fuzzy measures in a hybrid framework [156] similar to this presented in [25]. In [22] the following utilization of the SOM is described. After the textural feature extraction, which is undertaken through a set of Gabor Filters [29], the resulting features are input in two cascade-connected SOMs for the saliency analysis. The first SOM analyzes 5

The term large is relative to the number of coefficients of the fuzzy measure that have to be found, e.g. ten-dimensional feature spaces afford the assessment of 210 = 1024 coefficients. However a 10-dimensional feature space is not considered to be a large one in pattern recognition systems

70


the eight orientations of the three considered frequency bands separately and the second one joints the results of each band. The final output of the system is a map with four zones of orientation saliency: RD (right diagonal), LD (left diagonal), H (horizontal), and V (vertical), and a zone BG (background) where no orientation in no frequency is salient (see Fig. 4.19). Such a map is used to codify the pixels on the input images with the corresponding saliency, obtaining as a result a socalled perceptual feature image [22]. The map represents four areas where some Gabor features are salient. The saliency can be interpreted as the subjective knowledge about importance of each feature needed for the construction of the fuzzy measure. This assessment can be made assuming that saliency and feature importance are in some sense equivalent concepts. The assumption will be made, trying to evaluate if some contradiction appears in the final results.

(a)

(b)

Figure 4.20: (a) Exemplary U-matrix of a SOM. Points represent prototypes. Crosses, interpolating prototypes (i.e. prototypes that were not affected by the data set). Grayvalues represent distances between prototypes (black for larger ones). (b) Exemplary hit-matrix of a SOM. Two-dimensional histogram, where the grayvalue is proportional to the number of winning times of SOM neurons.

Beyond its basic properties the SOM paradigm presents two interesting associated methodologies that will be used in the construction of the fuzzy measures. The first one is related to the possibility of clustering the results of the output map [176]. The so-called U-matrix (see Fig. 4.20a) represents the distances between the prototypes of the output map in a two-dimensional symbolic space. Such a representation can help in getting a first idea of the cluster distribution. Clusters are represented in the grayvalue image as homogeneous regions of large grayvalues separated by edge-wise areas of low ones. The second methodology is denoted as hit-matrix. After training the output map, the data set can be input once again in order to obtain the winning neurons of each data point. This information is accumulated in a two-dimensional histogram and represented in an image space, where the grayvalue will be proportional to the number of times a neuron has won (see Fig. 4.20b) [175]. Such prototypes with a larger number of hits are the most-frequent winning values, and thus the most representative ones. The so called fuzzy hit-matrix [175] presents a value F Hj for each neuron mj of the output map, which is computed through

4.5 SC Methodologies for the Construction of Fuzzy Measures Revisited the expression: F Hj =

p X i=1

1 ¯ 2, 1 + (||xi − mj ||/Q)

71

(4.19)

¯ is the average where p is the number of data samples xi in the training set, and Q quantization error in the codification achieved through the output map, i.e. the number of neurons of the output map q fulfills q < p (see Fig. 4.20b). The U- and hit-matrices are presented in the literature as tools for knowledge discovery [69]. They overcome the usage of histograms and two-dimensional scatter plots for the determination of the fuzzy measure coefficients [59], since the utilization of scatter plots limits the subjective importance assessment to the coalition sets of cardinality two. The usage of the U- and hit-matrices allow assessing the coefficients of sets with larger cardinality. On the other hand it is worth observing the relationship between the here presented matrices and the concepts used in a formerly presented hybrid system [25] for handwritten word recognition (see Sec. 2.8.2). Taking all these facts into consideration a procedure for the construction of fuzzy measures is developed. The procedure is based on the application of a SOM on the data set to be classified. The obtained output map is thence used for the extraction of the fuzzy measure coefficients. First, the computation of the output map is described. It has been proven the convenience of realizing a previous statistical normalization of the SOM input data for that purpose [85]. Such a normalization makes the data sets present zero mean and unity standard deviation. Moreover, the implemented methodology adapts the dimensions of the SOM output map through the standard application of a previous PCA analysis [85]. Once the training data have been normalized, and the dimensions of the output map established, the neural network is trained with these data. Thence the U- and hit-matrices are computed. Since the SOM is used for the codification of the input data with few prototypes representing it, ¯ could have been taken as stopping criteria the quantification error of the SOM Q ¯ is not a of the training phase. However some preliminary analysis showed that Q useful stopping criteria. Furthermore the analysis demonstrated the importance of obtaining good U- and hit-matrices, what can be achieved with a sufficient number of iterations (200000). Other SOM parameters do not deeply influence the results. Second, the extraction of the fuzzy measure coefficients is detailed. The methodology allows the determination of the fuzzy measure coefficients of the individual information sources by taking into account the U- and hit-matrices. The procedure is described in the following. The U-matrix is first used in order to obtain different clusters on the data. The delimiters of these clusters are the interpolating neurons (neurons which are not hit by the data set and that are marked by a cross in the U-matrix ) together with cells of high distance between neurons. Thence the components of the neuron in

72


the hit matrix with a maximum number of hits inside each cluster are used as the individual coefficients of the different fuzzy measures. A pictorial description of this process is shown in Fig. 4.21.

(a)

(b)

Figure 4.21: Pictorial description of the operation for the extraction of fuzzy measures on the U- and hit-matrices. (a) U-matrix : Different clusters are found, which are delimited by interpolating neurons (represented by a cross) and cells of large distance between neurons (cells with lower grayvalue). (b) Hit matrix: The neuron with a maximum number of hits in each cluster is extracted (shown by a green arrow) and its components used as the fuzzy measure coefficients of the individual channels. A methodology based on mathematical morphology that takes into consideration the U- and hit-matrices as grayvalue images has been implemented in order to automate the formerly described process. The methodology, which can be denoted as morphological clustering of SOM [158], is based on so-called geodesic transformations [150]. A pictorial description of a geodesic dilation can be observed in Fig. 4.22.

(1)

Figure 4.22: Pictorial description of the geodesic erosion ǫg (f ) with marker set f and geodesic mask g. S: Structuring element. ǫ(1) (f ): Erosion. ∨: Maximum. While the basic morphological operations dilation δ (1) (f ) and erosion ǫ(1) (f )


73

are based on the grayvalue image f to be operated on and a particular structur(1) (1) ing element, the geodesic dilation δg (f ) and geodesic erosion ǫg (f ) [150] take into consideration a third element, which receives the name of geodesic mask g. The geodesic mask limits the result of the basic morphological operation to a particular image domain. Thus these operations are defined as: δg(1) (f ) = δ (1) (f ) ∧ g

(4.20)

(1) ǫ(1) g (f ) = ǫ (f ) ∨ g.

(4.21)

The so-called morphological reconstruction results from the iterative repetition of a geodesic transformation until stability is achieved. Thus the reconstruction by dilation Rg (f ) and the reconstruction by erosion Rg⋆ (f ), which is used in the morphological clustering of the SOM, can be expressed as [150]: Rg (f ) = δg(i) (f ) ∀i δg(i) (f ) = δg(i+1) (f ) (i) (i+1) Rg⋆ (f ) = ǫ(i) (f ). g (f ) ∀i ǫg (f ) = ǫ

(4.22) (4.23)

The morphological clustering of the SOM, which is applied for the automation of the fuzzy measures assessment, is presented in the following (see Alg. 2). It is worth reminding the goal to be attained by the procedure, namely the selection of a particular number (M) of prototypes. Moreover just one prototype in each cluster of the U-matrix can be selected. First the U-matrix is Algorithm 2 : Morphological clustering of the U-matrix (U) based on reconstruction by erosion (Rg⋆ (f )). P A: Set of possible areas. γ: Threshold of the U-matrix, which determines the cluster structure. SP : Set of selected prototypes. M: Number of prototypes to be selected. F H: Fuzzy hit-matrix. CP : Set of candidate prototypes. sp: element of selected prototypes. P A ← {U > γ} θ ← 256 SP ← {∅} while kSP k < M do 5: θ ← θ−1 CP ← {F H > θ} ∩ P A while kCP k = 6 0 do sp ← CPi SA ← RP⋆ A (sp) 10: SP ← SP ∪ {sp} P A ← P A − SA CP ← CP − SA end while end while

74


initialized as geodesic mask g after undergoing a binarization through a threshold γ. The binarization of the U-matrix defines a set of the possible areas, from which a prototype can be selected. The fuzzy hit-matrix is thence binarized through a threshold corresponding to its maximal value. In this way a set of candidate prototypes is defined. Thence a reconstruction by dilation is computed using the candidate set as the marker set and the set of possible areas as geodesic mask. The resulting image is used two-fold. On the one hand its set intersection with the candidate set returns the prototypes to be added to the set of selected prototypes. On the other hand the set difference between the resulting image and the set of possible areas actualize this set. Thus the selection of just one prototype per cluster is achieved. The detailed operation is sequentially repeated by decreasing the threshold of the fuzzy hit-matrix until M prototypes have been selected. Summarizing the procedure determines the clusters on the U-matrix through the parameter γ and then select the M larger local maxima of the fuzzy hitmatrix that fall within these clusters. The prototypes related to these maxima are used as the fuzzy densities µi (see Sec. 2.5.2). Thence fuzzy-λ measures are constructed by applying eq. (2.7): µij = µi + µj + λµi µj .

75

5 Evaluation of soft data fusion procedures in Image Processing This chapter attains the description of different frameworks for image processing based on soft data fusion. Different methodologies for the detection of color edges, for pre-processing in automated inspection systems, for processing color images through color morphological operators, and for the segmentation of clusters with a particular color were implemented and are presented in the following sections. Furthermore their results are evaluated in different industrial applications.

5.1 Edge fusion for color edge detection Edge detection is an important image transformation involved in numerous applications. Paradigms for image segmentation, pattern recognition, image understanding, edge preserving smoothing, and image compression include the edge detection as a processing stage. The two most usual approaches for the detection of edges on color images are based on the application of an edge detector operating in the grayvalue domain. The first approach takes into account the detection of edges in the three color channels plus the application of a simple fusion operator, usually the maximum operator. In the other approach, the intensity image of the three color channels is first computed and afterwards the selected operator is applied on this image. Furthermore, approaches based on regularization theory [137], mathematical morphology [88], orthogonal polynomials [91], statistical moments [124], and statistical vector fields [171] have been developed for color edge detection. These works are based on the application of complex mathematical concepts in order to cope with the vectorial nature of color information and simultaneously to reflect the special features of human color vision. On the other hand the here presented framework follows the first mentioned strategy for color edge detection. Nevertheless the maximum operator is substituted by a softer one [151] (see Sec. 4.2). Thus standard grayvalue edge detectors are first used on the individual channels. Thence the special features of color vision are implicitly taken into consideration by fusing the resulting edge maps through a fuzzy fusion operator. The proposed framework [151] includes different modules for edge detection (EDet), fuzzification (Fuzzif ), determination of fuzzy measures (DetFM ), and edge fusion (EFus) as depicted in Fig. 5.1.

76

5 Evaluation of soft data fusion procedures in Image Processing Red Channel Green Channel Blue Channel

EDet

Fuzzif

EFus

Color Edge Map

DetFM

Figure 5.1: Block diagram of the presented framework for color edge detection. EDet: Edge Detection. Fuzzif : Fuzzification. EFus: Edge Fusion. DetFM : Determination of Fuzzy Measures

The framework for color edge detection is exemplary realized with a Haar wavelet based edge detector for the implementation of EDet, and a S-shape fuzzifying function [24] for the module Fuzzif. The module EFus is implemented through the Choquet or the Sugeno fuzzy integrals (in its traditional usage or as an ILFO) with heuristically determined fuzzy measure coefficients. This configuration of the framework is used in all the results presented in this section. The module EDet is implemented following the procedure presented in [97], where an algorithm up to the Canny’s paradigm for edge detection and the equivalence between wavelets and filter banks is developed. Taking as background the Haar wavelet (see Fig. 5.2), which presents the following mathematical expression [97]:   1 : ∀x ∈ [0, 21 ) −1 : ∀x ∈ [ 21 , 1) , Ψ(x) = (5.1)  0 : ∀x ∈ / [0, 1)

a so-called Haar filter is designed. The obtained filter is approximated in order to optimize its performance in terms of computation time. As a result, a five-pixel mask with linear form, and values given by {1, 1, 0, −1, −1}, is vertically and horizontally run over the image. Finally the results obtained in both components are joint together through a maximum operator [97]. The fuzzification of the edge images for each individual channel in the module Fuzzif is carried out through S-shape membership functions (see Fig. 5.3) [24]. The mathematical expression of such functions is:  0 : ∀x ∈ [0, a]     x−a 2 2 c−a : ∀x ∈ (a, b] 2 S(x, a, c) = , (5.2) x−c  : ∀x ∈ (b, c] 1 − 2  c−a   1 : ∀x ∈ (c, 1] where b = a+c (see Fig. 5.3). 2 Employing the fuzzy membership function on the edge channel map allows removing some spurious background of no interest for its detection. The semantic

77

5.1 Edge fusion for color edge detection

1

0.5

Ψ(x)

0

-0.5

-1 -0.5

0

0.5

1

1.5

x

Figure 5.2: Mathematical function of a Haar Wavelet (Ψ(x)). A discrete Haar Wavelet is used as edge detector in the here presented framework.

1

0.8

0.6

S(x) 0.4

0.2

0

0

0.2

0.4

x

0.6

0.8

1

Figure 5.3: Mathematical function of a S-shape membership function (S(x, a, c)) for a = 0.2, b = 0.45, and c = 0.7. S-shape fuzzy membership functions are used in the here presented framework in order to filter background edges.

78

5 Evaluation of soft data fusion procedures in Image Processing

meaning conferred to the image through fuzzification refers in this case to the membership of the detected edges to the foreground of the image. Although the automatic determination of the parameters involved in the fuzzification is not undertaken in the here presented implementation, well-known methodologies for the resolution of this problem exist [170].

5.1.1 Application on standard color images The presented framework is first used on standard color images. This work is showed in order to assist on the comparison of the results obtained with the here presented scheme and those of other theoretical frameworks for color edge detection [88][91][124][137][171]. The standard input images were taken from the -SIPI Image Database1 .

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

Figure 5.4: Color edge detection with the SFI on the splash image with respect to a fuzzy-λ measure with densities µR = 0.25, µG = 0.5, and µB = 0.75. (a-c) Red, green, and blue channels. (d) Input image. (e-g) Red, green, and blue edge maps after fuzzification. (h) Color edge map after fusion. The fuzzy measure coefficients are heuristically found in order to achieve no loss of color edges.

First the color edges on the splash image (see Fig. 5.4) are detected. The simplicity of this image can help to visually understand the performance of the presented framework in color edge detection. The fuzzification of the edge images for each channel shows the capability of the framework to cope with the noisy background information. Nevertheless its elimination is not optimal and some information both of the reflection (see Fig. 5.4c) and the shadow (see Fig. 5.4a) 1

USC-SIPI Image Database (http://sipi.usc.edu/services/database/Database.html)

79


of the splash disappears as well. Furthermore some point artifacts provoked by the usage of a linear mask in the edge detection can also be observed. The utilization of a more complex edge detector could correct these shortcomings. The result shows how the presence of a combination of T- and S-norms in the fuzzy integral positively influences the obtained result. Therefore no color edges are lost.

(a)

(b)

(c)

(d)

Figure 5.5: Color edge detection with Sugeno fuzzy integral on the lenna image with respect to a fuzzy-λ measure with densities µR = 0.6, µG = 0.3, and µB = 0.8. (a) Input image. (b) Color edge map after fusion. The fuzzy measure coefficients are heuristically found in order to achieve no loss of color edges. (c) Results attained by the framework presented in [124]. (d) Results attained by the framework presented in [137].

The lenna image is used (see Fig. 5.5) in order to support the comparison of the results obtained with the color edge detectors presented in [124], [137], and [171], and the one of the here presented scheme. The visual inspection of the here presented results shows a better performance in terms of loss of color edges (see Figs. 5.5c and d). Among these methodologies, the here presented is the only one, where no loss of color edges on lenna is achieved.

80


It could be claimed that the utilization of a maximum operator as fusion operator of the individual channels’ edge maps would lead to similar results. The great differences of both approaches can be confirmed on the next results (see Fig. 5.6). A comparison of the different results shows the quality of using the fuzzy integral and thus of the parameterization of the fusion through the fuzzy measures. This parameterization improves the flexibility of the fusion operator (see Sec. 4.2). Thus the result of the fuzzy integral (see Fig. 5.6a) shows no loss of edges with a more natural aspect than the result of a maximum operator (see Fig. 5.6b). Furthermore the most important point is that the employment of the fuzzy integral generalizes the result to be obtained with the most common used fusion operators, i.e. maximum or average. Since the edge detection stage is usually not an end goal in a computer vision system, i.e. its performance have to be analyzed in the more general context of the application, the greater flexibility of the fuzzy integral makes it outperform over other frameworks.

(a)

(b)

(c)

(d)

Figure 5.6: Results on the house image of the framework for color edge detection employing different values of the fuzzy measures coefficients and Sugeno Fuzzy Integral (SFI). (a) Input image. (b) Color edge map for values µR = 0.6, µG = 0.25, µB = 0.8 (fuzzy-λ measure). The fuzzy measure coefficients are selected in order for the color edge map to present a ”natural” aspect. (c) Fusion with SFI as minimum. (d) Fusion with SFI as maximum.

The employment of an ILFO opens new perspectives for the further employment of the here presented framework (see Fig. 5.7). Through the usage of a label image, where the shadow regions present a different fuzzy measure than other areas, the detection of shadow false edges can be avoided. Therefore the analyzed feature for the generation of the label image (see Sec. 4.4.1) is in this case an intensity low level. A fuzzy measure leading to a minimum-like fuzzy integral is used in the shadow areas in order to avoid the detection of shadow edges, while a maximum-like integral is applied on the rest of the image.

81


(a)

(b)

Figure 5.7: Result of the fusion through a SFI-based ILFO, where the avoidance of shadow false edges is achieved. (a) Label image for the fuzzy measures. (b) Color edge maps after fusion. The fuzzy measure coefficients are heuristically found in order for the fuzzy integral to behave as a minimum operator in the white labeled areas and as maximum operator in the black labeled ones (see Sec. B.1).

5.1.2 Application on satellite images The ILFO based methodology is applied in the detection of edge maps on satellite images. The segmentation of these images is eventually attained. Since edge maps are used in different segmentation procedures [11][44][100], the edges of the shadow regions can mislead the segmentation procedure. Therefore, it is interesting to apply the presented methodology (see Fig. 5.1), in order to avoid the detection of shadow edges. Some of the results obtained on a database2 of aerial images are presented in the following (see Figs. 5.8-5.10). First a transformation from the input color space RGB into the Gaussian one is implemented (see Fig. 5.8). This color space presents some channels, where the shadow areas present a low contrast (see Fig. 5.8e), what can be exploited for the avoidance of its detection with the here presented framework. The discrete Haar wavelet is used for the edge detection on the Gaussian color channels. The results are shown in Fig. 5.8(g-i). Thence an ILFO is applied over these images in order to obtain the final edge map. The performance of two ILFOs, respectively based on the Choquet and the Sugeno Fuzzy Integrals, can be observed on hand of Fig. 5.9. The employed label image (see Fig. 5.9a) is computed in two stages. Firstly the binarization of the luminance channel (see Fig. 5.8d) is undertaken. The threshold is analytically selected as the first minimum of the image histogram. Finally a morphological binary erosion with a 3 × 3 mask is applied on the result. In the ILFO the fuzzy measure coefficients are heuristically determined and no fuzzification of the input images is undertaken for the completion of these results. 2

The images analyzed are delivered by the enterprise Definiens, AG, which distributes the IKONOS satellite images for Europe (http://www.definiens-imaging.com/).

82


(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

Figure 5.8: Input image for the detection of edge maps showing a scene from Paris and result of the applied color space transformation from a RGB into a Gaussian color model [52]. (a) R input channel. (b) G input channel. (c) B input channel. Channels of the Gaussian color model: (d) Eˆ ˆλ channel (analogous to an channel (equivalent to luminance); (e) E opponent band R-G); (f) Eˆλλ channel (analogous to an opponent band B-Y=B-R-G). Edge maps detected through a simplified Haar wavelet [97]: (g) On the Eˆ channel; (h) on the Eλ channel; (i) on the Eˆλλ channel.

83


(a)

(b)

(c)

Figure 5.9: Results obtained from the images in 5.8(g-i) through the application of an ILFO based on the Sugeno Fuzzy Integral (b) and on the Choquet Fuzzy Integral (c). (a) Employed label image.

(b)

(a)

(d)

(c)

(e)

Figure 5.10: Results obtained from the images in 5.8g-i through the application of an ILFO based on the Choquet Fuzzy Integral (d) with fuzzified images, respectively red, green, and blue fuzzy edge maps (a-c). (e) Employed label image.

84


The results of the CFI-based ILFO, which outperforms the SFI-based one, can be improved through the fuzzification of the input edge maps (in Fig. 5.8(gi)). Therefore, heuristically parameterized S-shape fuzzy membership functions [24] (see Fig. 5.3) are applied on the edge maps (see Fig. 5.10(a-c)). The fuzzy membership functions leave the edges of the foreground, and suppress these of the background. Moreover, the label image is generated with the procedure described in the former paragraph, but eroding the binary image twice (see Fig. 5.10e). The final result is shown in Fig. 5.10d.

5.2 Pre-processing for highlights filtering Objects of highly reflective materials, e.g. plastic envelopes, glass plates, mirrorlike surfaces, are very difficult to analyze with automated inspection systems. Traditionally such objects are inspected after a tedious process for the selection of a suitable lighting system. This selection process often remains unsolved because no lighting system can be successfully applied. Therefore, the automated inspection can only succeed after enhancing the input images. The ILF paradigm is applied in the implementation of the pre-processing framework for the automated visual inspection of objects made from highly reflective materials [154]. The presence of so-called highlights, areas where the camera detector is saturated, constitute the principal problem in the inspection of the mentioned objects. Highlight areas are characterized by the absence of visual information. Therefore the failing information has to be generated. The methodology takes into consideration the generation of a set of images where complementary visual information about the inspected object is contained. After the set of images has been generated, an ILFO is applied on them in order to suppress the highlight areas. The block diagram of the pre-processing system for highlight filtering is depicted in Fig. 5.11. The system is built by an Image Acquisition module (ImAcq) and a module implementing an automated ILFO, which consists of the submodules: Label Image Generator (LabImGen), Fuzzy Measure Constructor (FuzMeCo), and Fuzzy Integral (FuzInt). First an input image set is generated (ImAcq). For that purpose different images of the underlying object are taken with a camera from the same position but with different focal distances, shutter times and/or illumination conditions, i.e. lamp type, lighting direction. Therefore the highlights appear in the image set with different structures and/or positions, while the object information remains the same. This operation is not trivial since the high reflective nature of the objects being inspected makes the reflection of the environment on the surface almost unavoidable. Once the image set has been generated the label image for the fusion is computed (LabImGen). This is attained through the application on the input images

85

5.2 Pre-processing for highlights filtering

LabImGen

ImAcq

FuzMeCo

FuzInt ILFO ImPrePro

Figure 5.11: Block diagram of a pre-processing system for the highlight filtering through the application of ILFOs. ImPrePro: Image Pre-Processing. ImAcq: Image Acquisition. ILFO: Intelligent Localized Fusion Operator. LabImGen: Label Image Generator. FuzMeCo: Fuzzy Measure Constructor. FuzInt: Fuzzy Integral.

of the frameworks depicted in Figs. 4.15 and 4.16. In this case the extracted feature characterizes areas with high saturation, the highlights. Thus the module remains characterized by the highlight importance, which are parameterized by the threshold θ and its tolerance ǫ. The label image codifies all the different highlight interaction areas: no highlights, highlight in the first image, highlight in the first and second image, etc. A process for the determination of the fuzzy measures is thence undertaken (FuzMeCo). The number of different labels in the label image fixes up how many fuzzy measures have to be constructed. Genetic algorithms (see Sec. 4.5.1) and interactive genetic algorithms are used (see Sec. 4.5.2) for that purpose. In the construction through genetic algorithms the selection of an adequate fitness function, which characterizes the quality of the output image, is not trivial. Hence three different fitness functions fi (x) are evaluated. The first function (f1 ) presents a weighted sum of three features, namely the contrast of the fused image (characterized by the variance of its grayvalues σg2 ), and the fuzzy variables “bright” hb () and “dark” hd () (applied on the grayvalues gi of the pixels in the fused image): N −1 X 2 f1 (x) = 0.8σg + [0.75hb (gi ) + 0.2hd (gi )], (5.3) i=0

where N states for the number of pixels of the final image. The weights of this fitness function are heuristically found. The used trapezoidal fuzzy membership functions, which are depicted in Fig. 5.12, present the following mathematical

86


expressions: hbright (gi ) = hdark (gi) =

gi −200 55

: 0 ≤ gi < 200 : 200 ≤ gi ≤ 255

(5.4)

gi 1 − 15 0

: 0 ≤ gi < 15 : 15 ≤ gi ≤ 255

(5.5)

0

h(g) 1.0

hdark

hbright

15

200

255

gi

Figure 5.12: Fuzzy membership functions used for image quality assessment used in the fitness function of eq. (5.3). The fuzzy membership function hdark (gi ) accounts for the pixels with a low grayvalue (gi ), and hbright (gi ), for those with a large grayvalue.

The grayvalue distribution of the fused image is driven to present g¯ = 128 and σg = 64 by applying the second fitness function (f2 ) , which is mathematically expressed as: f2 (x) = k¯ g − 128k + 2kσg − 64k. (5.6) The third fitness function (f3 ) pursues a minimization of the fused image contrast, which is again characterized by σg2 : f3 (x) = σg2 .

(5.7)

Interactive genetic algorithms are employed as an alternative to GAs for the implementation of FuzMeCo. The application of interactive genetic algorithms allows evaluating the final result directly by a user, avoiding the complex determination of a fitness function. Simultaneously the search procedure is improved by the search mechanisms of standard genetic algorithms (see Sec. 4.5.2). After determining the values of the fuzzy measure coefficients, the fuzzy integral is applied on each pixel of the image set with respect to the fuzzy measure codified in the label image (FuzInt). The result is an image, where the highlight areas have been filtered. The following sections show some of the results obtained in different automated inspection systems through the application of the described framework. Two different pre-processing goals are attained in the presented results. On the one hand

87


(a)

(b)

(c)

Figure 5.13: Input images of the pre-processing stage of a system for the analysis of chocolate package completeness.

some of the applications request just a right visualization of the underlying object. In this case the ILFO is the only part of an image enhancement system, whose objective is the highlights suppression. A second type of pre-processing system goes further by overtaking some of the functions of a subsequent fault detection subsystem. Since the reflection on the possible faults can ease its detection, the highlights in these positions are left untouched, while being suppressed in the rest of the image.

5.2.1 Pre-processing for visualization First the results from an automated visual inspection system, whose final goal is the analysis of the chocolate package completeness, are presented. For this purpose a contrast enhanced image with no highlights is required. Three different images of the chocolate package are generated in the module ImAcq, where the presence of a bundle produces highlights (see Fig. 5.13). Different results, which are obtained with different fusion strategies, are shown in Fig. 5.14. The utilization of more complex strategies than the minimum increases the brightness of the final result (compare Fig. 5.14a with Figs. 5.14bd). Furthermore the utilization of an ILFO improves the contrast of the fused image (see Fig. 5.14f). The label image, Fig. 5.14e, was built up to an addition of the three input images followed by a multi-threshold procedure. Once the label image was computed a genetic algorithm (GA) with the fitness function given in eq. (5.3) was applied for the construction of the three fuzzy measures. The automated visual inspection of automotive headlamp reflectors becomes more complex due to the presence of mirror-like surfaces. The images generated by ImAcq are shown in Fig. 5.15. The obtained preliminary results are depicted in Fig. 5.16. The label image is obtained after the application of a heuristically found highlight threshold (θ) in each of the individual images (see Fig. 5.16a). All results

88


(a)

(b)

(c)

(d)

(e)

(f)

Figure 5.14: Results of the pre-processing stage with different fusion strategies in a system for the analysis of chocolate package completeness. (a) Minimum. (b) OWA. (c) OWA with GA optimized weights. (d) Choquet Fuzzy Integral with respect to a GA optimized fuzzy-λ measure. (e) Label image used with the ILFO. (f) ILFO with label image in previous figure and fuzzy-λ measures, whose coefficients were optimized through a genetic algorithm (GA).

are obtained after the construction of the corresponding fuzzy measures through a GA with 40 individuals of population size, 60 generations, 0.8 of crossover probability, and 0.001 of mutation probability.

(a)

(b)

(c)

Figure 5.15: Input images of the pre-processing stage of a system for the detection of structural faults on automobile headlamp reflectors (item I). The results of the pre-processing subsystem ( in Figs. 5.16b-f) are obtained for different types of fuzzy integral, of fuzzy measures, and fitness functions fi (x). The result of the utilization of an ILFO based on the Sugeno Fuzzy Integral, where the fuzzy measures are determined with a GA, is shown in Fig. 5.16b. The GA is run over the fitness function in eq. (5.6). Some artifacts, areas with a grayvalue equal to one of the coefficients of the fuzzy measure, appear very easily

89


(a)

(b)

(c)

(d)

(e)

(f)

Figure 5.16: Preliminary results of the pre-processing stage of a system for the detection of structural faults on automobile headlamp reflectors, where the highlights avoidance is undertaken through the application of ILFOs. (a) Label image for results. (b) Sugeno-based ILFO with GA optimized fuzzy-λ measures with respect to the fitness function (5.6). Areas with grayvalue artifacts are marked with an orange circle. (c) Choquet Fuzzy Integral based ILFO with GA optimized fuzzy-λ measures (fitness (5.3)). Pseudo-edges are marked with circles. (d) Choquet Fuzzy Integral based ILFO with GA optimized fuzzy-λ measures (fitness (5.7)). Pseudo-edges are marked with circles. (e) Choquet-based ILFO with general fuzzy measures, which were heuristically found up to fuzzy measures used in the computation of result of the previous figure. (f) Choquet-based ILFO with fuzzy-λ measures found for computing (d), now applied with a fuzzy label image.

when applying this operator. On the other hand some pseudo-edges appear when using the Choquet-based ILFO (see Fig. 5.16c), where the fitness function of the GA for the fuzzy-λ measures is given by eq. (5.3). The utilization of another fitness function, eq. (5.7), does not produce any improvement (see Fig. 5.16d). However an heuristic and slight modification of the results obtained by the GA improves the ILFO performance as can be observed on Fig. 5.16e. It is worth mentioning that the modification changes the type of the used fuzzy measures from λ measures to general fuzzy measures (see Sec. A.3). The image depicted in Fig. 5.16f was obtained with the mentioned fuzzy-λ measures applied with the

90


fuzzy label images generated following the block diagram in Fig. 4.16. As already mentioned, interactive genetic algorithms can be used for the construction of fuzzy measures in the here presented application as well. Therefore, an interactive genetic algorithm is used for the construction of the fuzzy measures leading to the avoidance of highlights in the fused image. The experiment was undertaken on the images shown in Fig. 5.15. The obtained results are shown in Fig. 5.17.

(a)

(b)

(c)

(d)

Figure 5.17: Preliminary results of the utilization of interactive genetic algorithms for the construction of fuzzy measures. Results obtained on images depicted in Fig. 5.15 for different types of interactive genetic algorithms and different parameterizations (see Tab. 5.1).

Table 5.1: Type and parameterization of different interactive genetic algorithms used in the computation of the preliminary results depicted in Fig. 5.17. Gen: Number of generations. PCross: Crossover probability. PMut: Mutation probability. El: Elitism. PRepl: Replacement probability. MType: Fuzzy Measure type. StSt: Steady State Fig. (a) (b) (c) (d)

GA Type Simple StSt StSt StSt

Gen 20 8 9 10

PCross PMut 0.8 0.1 0.95 0.1 0.7 0.048 0.81 0.024

El/PRepl MType elitism λ 0.9 λ 0.9 λ 0.9 general

The interactive genetic algorithm strategy is tested with different parameter-


91

izations and different types of fuzzy measures (see Tab. 5.1). The number of individuals is limited to 10 pro generation in order for all of them to fit in the displaying surface of a 17′ computer monitor. The variance of the individuals quickly decreases in each generation, i.e. the individuals tend to be equal up to the fifth generation. Although this shortcoming can be controlled (though can not be avoided) through the crossover and mutation probabilities, this relationship does not seem to be deterministic. The interactive determination becomes tedious up to the seventh generation. The utilization of an estimation of the fitness of each individual, which can then be modified by the user, improves the results (see Fig. 5.17(c-d)). This succeeds for a smaller number of generations than those obtained when the user “blindly” gives the fitness of each individual (see Fig. 5.17(a-b)). After having shown some preliminary results of the presented framework, which were undertaken in order to fix up the type of fuzzy integral and of the fuzzy measures, and the strategy for the determination of the fuzzy measures coefficients, the performance of an automated pre-processing system for highlight filtering is analyzed. This automated framework presents the same block diagram already presented (see Fig. 5.11). Furthermore the module LabImGen is composed by a procedure for the determination of the parameter θ on hand of the image histograms and the implementation of the block diagrams shown in Figs. 4.15 and 4.16. The tolerance of the highlights (ǫ) remains a parameter. Since the results of the interactive genetic algorithm are not satisfactory, FuzMeCo is implemented through a GA. The fuzzy measures to be constructed are fuzzy-λ measures. Although general fuzzy measures seems to deliver better results (see Fig. 5.16e), the computational cost of its consideration increases with the number of fuzzy measures. Therefore, its utilization is not feasible. Moreover, it can not be fixed up, which fitness function to use. A Choquet fuzzy integral is selected for the implementation of the module FuzInt. First a procedure used in the generation of the label image is implemented for thresholding the feature maps, whereby the determination of θ is automated. The selected algorithm is part of the watershed transformation for the segmentation of multidimensional histograms [149]. It is based on the concept of the dynamic of a histogram peak [63], which can be applied for finding out the local extrema of an histogram. Therefore, the local minimum of the histogram larger than a low limit, whose grayvalue 200 is set a priori, and presenting the largest dynamic value is selected as the highlight threshold θ (see Fig. 4.17). One threshold θi for each input channel results from this procedure. Figure 5.18 exemplary depicts the result of the computation of the dynamic peaks for the histogram of the image shown in Fig. 5.15c. The binary images obtained through the automated parameterization, the resulting label image, and the fuzzy label images for ǫ = 50 are depicted in Fig. 5.19. It is worth mentioning that the values of θi obtained through the computation of the peak dynamics were almost the same as manually set in the generation of

92

5 Evaluation of soft data fusion procedures in Image Processing 1200

distribution function

1000

800

600

θ

400

227

200

0 0

50

100

150 grayvalue

200

250

300

Figure 5.18: Exemplary result of the computation of the peak dynamic on the inverted histogram of Fig. 5.15c. Value manually set for the generation of Fig. 5.16a was θ = 225. The dotted line sets the low limit (200) of the area where the local minima is expected to be found.

the label image based on the observation of the histogram (compare Fig. 5.16a and Fig. 5.19d). Furthermore the generation of the label image follows in this case the mathematical framework described in Sec. 4.4.1. Table 5.2: Best parameterization of a steady-state genetic algorithm for the construction of a fuzzy measure for different fitness functions fi . Parameter Crossover probability Mutation probability Replacement probability Generation number Population size

f1 (5.3) f2 (5.6) f3 (5.7) 0.7 0.9 0.8 0.05 0.005 0.05 0.9 0.9 0.9 80 63 65 40 40 40

A second experiment pursue the determination of the best parameterization of the genetic algorithm in the construction of the fuzzy measure. Thus a genetic algorithm with different crossover and mutation probabilities are taken into consideration. The analysis is conducted for the fitness functions f1 , eq. (5.3), f2 , eq. (5.6), and f3 , eq. (5.7). The evolution of these fitness functions for the best individual of the population in each generation is analyzed for three values of crossover probability and four of mutation probability. The evolution of each fitness function reaching the absolute minimum of all these combinations is depicted in Fig. 5.20 .

93


(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

Figure 5.19: Results of the binarization, the label image generation, and the fuzzy label image generation after automated determination of thresholds θi . (a) Binarization of image in Fig. 5.15a with θ1 = 243. (b) Binarization of image in Fig. 5.15b with θ2 = 225. (c) Binarization of image in Fig. 5.15c with θ3 = 227. (d) Resulting crisp label image. Fuzzy label image with fuzzy membership functions characterizing areas with: (e) No Highlight, (g) Highlight in one channel, (h) Highlight in two channels, and (i) Highlight in all channels. (f) Crisp label image for another object computed with the same procedure.

94


best fitness value f1(x)

P(crossover) = 0.7

generation index (a)


P(crossover) = 0.9

generation index (b)


P(crossover) = 0.8

generation index (c)

Figure 5.20: Convergence of the GA over different generations in the computation of the fuzzy measure coefficients of the ILFO applied for obtaining the images depicted in Fig. 5.21(a-c). The convergence is shown for the best crossover-mutation combination of each considered fitness function fi (x) computed for the best individual in each generation. (a) Fitness function f1 (5.3). (b) Fitness function f2 (5.6). (c) Fitness function f3 (5.7).

95


Table 5.2 summarizes the best parameters resulting from the application of each fitness function. The evolution of the fitness function f1 eq. (5.6) does not show any difference among the different values of mutation probability up to a particular number of generations (see Fig. 5.20b). In this case the fuzzy measures were selected after a subjective evaluation of the resulting images for the last generation. The fuzzy measure coefficients obtained through the GAs parameterized as stated in Tab. 5.2 are selected. Thence the final results are computed (see Fig. 5.21) based on this parameterization.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 5.21: Final results of an automated system for the visual inspection of headlamp reflectors, where the fuzzy measures are constructed after a GA with different fitness functions and parameterized upon Tab. 5.2. The results are obtained for two highlight tolerance (ǫ) values (see Sec. 4.4.2). (a) Fitness function f1 (5.3) and ǫ = 0. (b) Fitness function f2 (5.6) and ǫ = 0. (c) Fitness function f3 (5.7) and ǫ = 0. (d) Fitness function f1 (5.3) and ǫ = 50. (e) Fitness function f2 (5.6) and ǫ = 50. (f) Fitness function f3 (5.7) and ǫ = 50.

Since the execution time of the framework linearly varies with the increment of the tolerance (see Fig. 5.22), the training through GAs is first done by taking into consideration the “crisp” label image. A trade-off among the value of this parameter and the quality of the final system output have to be undertaken. As it can be observed in Fig. 5.23, a larger tolerance does not necessarily implies a better performance of the system.


execution time [ms/pixel]

96

tolerance

Figure 5.22: Estimation of the dependence of the execution time [ms/pixel] of an ILFO based on the Choquet fuzzy integral with respect to the highlight tolerance (ǫ) parameter (see Sec. 4.4.2).

(a)

(b)

(c)

Figure 5.23: Effect of the tolerance variation (ǫ) on the final results of the automated pre-processing system (see Sec. 4.4.2), where the fuzzy measures are constructed after a GA with the fitness function f3 (5.7) and parameterized upon Tab. 5.2. Highlight tolerance (ǫ) values: (a) epsilon = 0. (b) ǫ = 10. (c) ǫ = 50.

Finally Fig. 5.24 shows the input images generated by ImAcq over another object as the one considered up to now. An ILFO with the same fuzzy measures found for the object considered up to now is applied on these input images (see Fig. 5.25). The experiment gives an idea of the generalization capability of the here presented framework for different fitness functions and different tolerance values. The label image (see Fig. 5.19f) is automatically generated. The results (see Fig. 5.25) are obtained by applying the same conditions in the acquisition of the input images.

97


(a)

(b)

(c)

Figure 5.24: Input images (of a second object) of the pre-processing stage in a system for the detection of structural faults on automobile headlamp reflectors.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 5.25: Final results of the automated pre-processing system, where the fuzzy measures are obtained with a training on the object depicted in Fig. 5.15. The results are obtained for two highlight tolerance (ǫ) values (see Sec. 4.4.2). (a) Fitness function f1 (5.3) and epsilon = 0. (b) Fitness function f2 (5.6) and ǫ = 0. (c) Fitness function f3 (5.7) and ǫ = 0. (d) Fitness function f1 (5.3) and ǫ = 50. (e) Fitness function f2 (5.6) and ǫ = 50. (f) Fitness function f3 (5.7) and ǫ = 50.

5.2.2 Pre-processing for fault detection The results obtained in the pre-processing of a structural fault detection system, which was applied on halogen bulbs, are described in the following. The input images of the system are shown in Fig. 5.26. The fault to be detected can be

98


distinguished as a triangular area in the right part of the bulb neck.

(a)

(b)

(c)

Figure 5.26: Input images of the pre-processing stage in a system for the detection of structural faults on halogen bulbs.

(a)

(b)

(c)

(d)

Figure 5.27: Results of the pre-processing stage with different fusion strategies in a system for the detection of structural faults on halogen bulbs. (a) Weighted minimum. (b) OWA. (c) Label image used with the ILFO. (d) Choquet Fuzzy Integral based ILFO with label image in (c) and heuristically determined fuzzy-λ measures . A comparison of the results obtained through a weighted minimum (Fig. 5.27a) and an OWA (Fig. 5.27b) shows again that the utilization of a softer operator

5.3 Color morphology for textile image processing

99

(OWA in front of weighted minimum) improves the result of the fusion in terms of presence of structural information. The usage of an ILFO (Fig. 5.27d) can ease the posterior detection of the fault. This is achieved by taking into consideration geometric information of the highlight areas on the defects. The saturation area in the center of the bulb body is due to an environment reflection and can not be eliminated because of its presence in all three input images (see Fig. 5.26).

5.3 Color morphology based on the fuzzy integral for textile image processing Algorithm 3 : Exemplary algorithm for the computation of a dilation operation based on the here presented color morphology. construct a fuzzy measure F M for all pixels Pi under the morphological mask do Distance to the ideal DIi ← F I F M (Pi ) if only one minimum DIi then 5: Pcenter ← Parg mini (DIi ) else for all pixels Pi with minimum DIi do Distance to the anti-ideal DAIi ← F I F¯M (Pi ) if only one minimum DAIi then 10: Pcenter ← Parg mini (DAIi ) else Pcenter ← Parg mini (DAIi ) for any i end if end for 15: end if end for Mathematical Morphology constitutes a well-known discipline for the analysis of spatial structures that has found an extended application in image processing. The basic element for the definition of a morphological operation is the existence of a ranking scheme [142], which is applied over the pixels underlying an structuring element known as mask (see Sec. 4.5.3). In grayvalue morphology the concepts of maximum and minimum are clear (respectively white and black grayvalues). On the other hand, such concepts are not well defined on color images due to its multi-dimensional nature. Moreover the color definition presents some degree of uncertainty. In spite of this fact only one scheme for a color morphology [88] takes into consideration fuzzy concepts for the treatment of this uncertainty. Fuzzy fusion operators can be used upon [10] in order to measure the distance of a point x in the feature hyper-cube [0, 1]n to the characteristic points denoted

100


as Ideal 1 n and Anti-Ideal 0 n (see Sec. B.8). This property is used as starting point for the definition of a new color morphology [157]. Thus, the result of the fuzzy integral is used for ranking the pixel values under the mask. If for instance a dilation operation is considered, the fuzzy integral result measures the distance of the point to the Ideal, which is used for ranking the values under the mask. The corresponding fuzzy measure weights the color channels and breaks the indetermination due to the vectorial nature of color. Furthermore if some pixels under the mask present the same distance to the Ideal, an additional criteria is used, namely the distance to the Anti-Ideal. This last distance is computed through the fuzzy integral with respect to the dual fuzzy measure [62] of the fuzzy measure previously used. On the contrary the distance to the Anti-Ideal is first computed for the implementation of the morphological erosion. In case of some pixels tying, the distance to the Ideal (the fuzzy integral with respect to the dual fuzzy measure) is used. Exemplary the pseudo-code for a dilation operation appears in Alg. 3. One novelty of the here presented color morphology is the definition of so-called targeted morphological operations, which are based on the existence of the fuzzy measure coefficients weighting the color channels. For instance in a red-dilation the colors near to the red will be preferred over the others. Such a preference is established through the values of the fuzzy measure coefficients. The example depicted in Fig. 5.28 elucidates this concept.

(a)

(b)

(c)

(d)

Figure 5.28: Results of targeted morphological operations with a cross mask on a random color image. (a) Input image. (b) Red-dilation with µR = 200, µG = 1, µB = 1, µRG = 200, µRB = 200, µGB = 1, µRGB = 255. (c) Green-dilation with µR = 1 ,µG = 128, µB = 1, µRG = 200, µRB = 1, µGB = 200, µRGB = 255. (d) Blue-erosion with µR = 1, µG = 1, µB = 128, µRG = 1, µRB = 128, µGB = 128, µRGB = 255.

The targeted morphology is employed in the pre-processing stage of a system for the automated visual inspection of textiles. The results of the application of basic morphological operations, i.e. dilation/erosion, can be observed in Fig. 5.29. In this case the usage of the morphological operations targeted to the color of the faults succeed in enhancing its compactness. Thus this operation facilitates the posterior detection of the faults. The construction of the fuzzy measures is realized by asking the user to select an area of the image, whose color is used in order to target the morphological operation. The average of the extracted color values for each channel are used as the fuzzy densities of a fuzzy-λ measure.

101

5.3 Color morphology for textile image processing

(a)

(c)

(b)

(d)

Figure 5.29: Results of basic morphological operations of the here presented color morphology for the automated visual inspection of textiles. (a) Input image where the fault to be detected was caused by the existence of dust particles on the textile during the print process. (b) Result on (a) of a white-targeted dilation with a cross mask. (c) Input image of a textile where the color could not be right fixed on the textile. (d) Result on (c) of an erosion with a linear mask targeted to the fault color.

(a)

(c)

(b)

(d)

(e)

Figure 5.30: Morphological gradient of the here presented color morphology for the automated visual inspection of textiles. (a) Input image where the fault to be detected was caused by the existence of strange particles on the textile while printing it. (b) Morphological gradient after a white-targeted dilation and erosion with a cross mask. (c-e) Individual channels of (b).

102


The basic morphological operations can be combined in order to derive more complex ones. The morphological gradient [142], which is used in image processing for the detection of edges, constitutes one of these composed operations. Different results of the morphological gradient on images of the application at hand are depicted in Figs. 5.30 and 5.31. The computation of the morphological gradient delivers a color image in contrast to the algorithm presented in [88]. This fact allows using some of the channels of the resulting image as the detector of the fault in the textile (see Fig. 5.30e).

(a)

(c)

(b)

(d)

(e)

Figure 5.31: Morphological gradient of the here presented color morphology for the automated visual inspection of textiles. (a) Input image with no visible fault. (b) Morphological gradient after an orange-targeted dilation and erosion with a cross mask. (c-e) Individual channels of (b).

The morphological gradient computed through the targeted morphological operations is expected to result in a selective detection of edges of the targeted color. Nevertheless this goal is not successfully attained by the here presented morphology in its current configuration (see Fig. 5.31b). Although the dual fuzzy measure for the computation of the distance to the Anti-Ideal was selected because of its mathematical plausibility, it does not seem the most appropriate for the attainment of this last goal.

103

5.4 Document image processing through the fuzzy integral

5.4 Document image processing through the fuzzy integral As already mentioned, the fuzzy integral was first used in image processing as image segmentation tool [129]. This work is the starting point for the development of the following application. The existence of constant regions in the level curve of the SFI (see Sec. B.3) is exploited in a system for the classification of archive cards after its background color. The obtained results are shown in Fig. 5.32. After a morphological dilation of the input images, the SFI operates over the input channels with coefficients of the fuzzy measure adapted for each color. Thence the obtained label images are fused. This kind of segmentation presents some errors due to the incomplete suppression of text in the pre-processing stage (see Fig. 5.32c).

(a)

(b)

(c)

Figure 5.32: Usage of the Sugeno Fuzzy Integral for segmentation of archive card color images. (a) Input image with four cards. (b) Image after morphological dilation for the suppression of text. (c) Label image resulting from the segmentation. A different system [155] is used for the segmentation of ink seals on documents (see Fig. 5.33). The used strategy takes into consideration the computation of the fuzzy integral with respect to two different fuzzy measures. These are selected in order for the seal color color cluster to be maximally affected by the change. The difference image of these results is thresholded, dilated, and is thence fused with the input image through a logical AND. The CFI, which presents a smoother response than the SFI, was proven to present a better performance. The obtained results on two test images with the Choquet and Sugeno the Fuzzy Integrals are respectively depicted in Figs. 5.33 and 5.34. The methodology can be formally described as: Id (x) = Fµ1 (x) − Fµ2 (x),

(5.8)

104


(a)

(b)

(c)

(d)

Figure 5.33: Usage of the Choquet Fuzzy Integral for the segmentation of seals. A fuzzy integral is firstly computed with respect to two different fuzzy measures. The change of fuzzy measure affects the seal color cluster, leaving the other components of the image unmodified. A binary label image is obtained by subtracting those images and thresholding the result. (a) Input image. (b) Fuzzy Integral result with the first fuzzy measure. (c) Fuzzy Integral result with the second fuzzy measure. (d) Final result, where the difference image between (b) and (c) plus a morphological dilation is used as mask for the input one (a).

where Id states for the the difference image, x for the color vector in each pixel of the input image, and Id (x) for the grayvalue image resulting from the difference. Thence the resulting image is binarized and dilated through mathematical morphology. The resulting binary image is then used as binary mask for extracting the ink seal from the original image (see Figs. 5.33 and 5.34). This methodology is applied on 20 documents from an industrial application, which attains the segmentation of customs seals for falsification detection. The documents are sampled from real documents, i.e. documents worked out in customs. Thus, they present the seal to be segmented together with different other elements of similar color hue, e.g. pen notations, other seals. Due to a nondisclosure agreement with the enterprise delivering the images, these and the results obtained on them can not be depicted. Therefore, the performance of the segmentation framework is quantitatively analyzed. Among all the criteria presented in [196], the so-called goodness from region shape is selected because of the circular form of the analyzed seal. An eccentricity coefficient [76] is computed on Id (x) of each document image for evaluating the performance of the framework. The coefficient presents the following expression: (m2,0 − m0,2 )2 + 4m21,1 ε= , (5.9) (m2,0 + m0,2 )2 where mp,q are the central moments of the grayvalue image [76]. Its real value, which is used in order to characterize form information, ranges from 0.0 for circular forms to 1.0 for linear ones. The eccentricity coefficient is rotation, scale, and


(a)

(c)

(b)

(d)

105

Figure 5.34: Exemplary usage of the Sugeno Fuzzy Integral for the segmentation of seals on a post letter. (a) Input image. (b) Sugeno Fuzzy Integral result with the first fuzzy measure. (c) Sugeno Fuzzy Integral result with the second fuzzy measure. (d) Final result.

translation invariant, what compensates for the different position and orientation of the seals in the document images used for the here presented evaluation. In a previous analysis the employed fuzzy measures µ1 and µ2 are determined. The color cluster of the seal is maximally affected by a change in the coefficient µGB . This is due to the location of the color cluster in the canonical region dominated by this coefficient. Thus, ε is computed for different values of µ2GB coefficient, while maintaining µ1GB = 5 and the remaining coefficients of both fuzzy measures constant. The result is depicted in Fig. 5.35. Two synthetically generated images are added to the data set in order to have a reference for the evaluation of the results. The first reference image is generated by adding a disturbing region with the same color as the seal. This region is placed at 180 pixel distance of the seal center and presents an area of 56 pixels. Its detection is equivalent to erroneously segmenting a 0.065% of the image (equivalent error rate). The second reference image is generated by placing a color area of 168 pixels (0.196% of the image) at 210 pixels of the seal center. As it can be observed (see Fig. 5.35) the eccentricity coefficient ε of all resulting segmented images are clearly below the error rate of the second reference image (0.196%). Moreover only four images presented an equivalent error between this and 0.065% of false segmented pixels. The graphical results are summarized in Tab. 5.3. Finally it is worth mentioning that the eccentricity coefficient is relatively constant for the different values of µ2GB . Thus the automatic determination of this coefficient can be coarsely approached. Taking into consideration this fact, the

106

5 Evaluation of soft data fusion procedures in Image Processing 0.3

0.25

eccentricity coefficient

0.196% 0.2

0.15

0.1

0.065%

0.05

0 50

100

fuzzy measure coefficient

150

200

250

[grayvalue]

Figure 5.35: Quantitative performance evaluation of the framework for seal segmentation. Eccentricity coefficient ε [76] for 20 tax forms of a data set from an industrial application. The coefficient is computed on image resulting from (5.8) with the Choquet fuzzy integral. The two used fuzzy measures differ in µiGB : µ1GB = 0.0 and µ2GB = i/255 ∀i = 5, 10, . . . , 250 (x-axis). The eccentricity coefficient of two reference images are provided for comparison (the mean value of these references is depicted through a dotted line together labeled with the percentage of false segmented pixels). A third reference is given by the mean value of the eccentricity coefficient for the image depicted in Fig. 5.33 (bottom of the figure, mean value in a dotted line as well).

only parameter to be determined in the system remains the threshold for the binarization of the difference image.


107

Table 5.3: Performance evaluation of framework for color seal extraction. The following table summarizes the graphical results depicted in Fig. 5.35. i: Image index. ε¯i: Mean value of the eccentricity coefficient over the values of µ2GB . σε2i : Variance for the eccentricity coefficient. eep: Equivalent number of erroneously segmented pixels over the total number of pixels of the document image. This parameter can just be computed for the reference images. The value is given in order for the reader to establish an implicit equivalence between the value of the eccentricity coefficient and the expected percentage of false segmented pixels in the remaining evaluation images. i 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

ε¯i σε2i [·10−4 ] 0.009 0.508 0.013 0.533 0.014 0.554 0.018 1.192 0.023 1.062 0.025 2.112 0.029 1.911 0.036 1.883 0.038 2.794 0.042 3.015 0.043 3.23 0.045 4.345 0.069 8.242 0.073 10.24 0.074 8.553 0.075 8.571 0.075 10.96 0.086 3.401 0.097 12.9 0.098 16.04 0.114 16.56 0.146 21.16 0.243 13.73

eep 0% 0.065% 0.196%

108

109

6 Evaluation of soft data fusion procedures in Image Analysis This chapter presents two different applications for image analysis, which are developed within the theoretical framework of soft data fusion. These procedures are based on the utilization of the fuzzy integral for the implementation of the classification stage. The first one takes into consideration the images of a market basket recognition system, where different images taken under bad illumination conditions are segmented based on color information. On the other hand, an industrial application of the fuzzy integral for the automated inspection of collagen plates was developed and implemented. In this case the general framework, which is based in the binarization of textured images, is presented drawing special attention to the parts where fusion operators are involved.

6.1 SOM - Fuzzy Integral hybrid system for multi-dimensional image segmentation A hybrid system (see Sec. 2.4) for the segmentation of multi-dimensional images is implemented [158]. The system considers the application of a SOM for the automated determination of the fuzzy measures (see Sec. 4.5.3) and of the fuzzy integral for the classification of the pixels in the input image. In this framework, various color spaces are simultaneously used as a multi-dimensional feature pace. Because of its novelty for the realization of image segmentation on color images, the approach is elucidated in the following. Traditionally, the first step in the realization of a color processing system takes into account the selection of the color space to operate on. The selection of the color space is a previous operation, some kind of a priori feature selection. A recent work [52] presents the characteristics of different color spaces with respect to different image features, e.g. shadows, highlights, color illumination. The here presented framework for segmentation simultaneously uses different components of the color spaces in order to take advantage of the positive characteristics of each color space in front of the corresponding image features. Thus, the framework is applied on multi-dimensional images in the sense that these images are composed

110

6 Evaluation of soft data fusion procedures in Image Analysis

by the components that traditionally define the color spaces and here become image features. Since the SOM paradigm is well-known to present positive characteristics in high-dimensional feature spaces, it is applied in the construction of fuzzy measures for the segmentation of those multi-dimensional images. Thus the fuzzy measure coefficients of the individual channels are assessed through the morphological clustering of the SOM output map presented in Sec. 4.5.3. The utilization of the morphological clustering allows partitioning the SOM prototypes somehow independently from the dimensionality of the feature space, i.e. just the response of the U-matrix to the high-dimensional space influences the result. Furthermore, the computational time of the procedure depends exclusively on the number of prototypes of the output map and not on the dimensionality of the feature space. The fuzzy integral operates herein as classifier (see Sec. 4.3.2), where one fuzzy measure for each class have been determined through the morphological clustering. Thence the fuzzy integral with respect to these fuzzy measures is applied on each pixel of the input image in order to assign a membership degree on each of the classes represented by the selected prototypes. The index of the class with maximal membership degree is then given as the label of the pixel.

6.1.1 Framework for the segmentation of multi-dimensional images The fuzzy integral, which is used as classifier of the image pixels, constitutes the nuke of the framework for the segmentation of multi-dimensional images. The fuzzy integral can be used in the framework both within the information fusion and the possibilistic theoretical frameworks (see Sec. 4.3.2). As already mentioned, the information fusion framework presents an unsupervised classification, while the possibilistic uses a supervised one. The description of the framework is first undertaken on the unsupervised approach and thence the modifications to implement a supervised segmentation methodology will be described. Finally some light modifications on the fuzzy measures, which can be applied in order to improve the results in segmentation procedures using the fuzzy integral, are elucidated. Unsupervised approach The block diagram of the framework for the segmentation of multi-dimensional images within the information fusion approach is depicted in Fig. 6.1. Some details on it are given in the following paragraphs. A SOM is first trained with the features present in the image to be segmented. The training is undertaken with a subset of image pixels. The pixels of this subset are randomly selected following a quasi-homogeneous sampling algorithm called

111

6.1 SOM - Fuzzy Integral hybrid system for image segmentation

FuzInt fuzMember hi

Multin -dimensional Image

n

SOM

m

Defuzz

Segmented Image

m*n

Umat Omap

MorphClust

FHmat

Figure 6.1: Block diagram of the here presented framework for the segmentation of multi-dimensional images within the information fusion theoretical framework. Umat: U-matrix. Omap: SOM output map. FHmat: Fuzzy hit-matrix. fuzMember: Fuzzy membership functions.

Linear Pixel Shuffling (LPS) [5]. Once selected, these pixel features are statistical normalized and the SOM output map is trained (see Sec. 4.5.3). After the training phase is completed, different operations are undertaken on the resulting output map. First the fuzzy hit- and U-matrices, which constitute the basis for the morphological clustering, are computed. It is worth reminding that the fuzzy hit-matrix is a fuzzified two-dimensional histogram of the winning neurons, while the U-matrix is a grayvalue image, where the grayvalues correspond to the distance among the prototypes of the output map (see Sec. 4.5.3). The output map and both matrices are delivered to the module implementing the morphological clustering (MorphClust). On the other hand, the fuzzy hit-matrix is transformed into a one-dimensional histogram for each feature. These histograms are used as fuzzy membership functions in the fuzzification of the input images previous to the fuzzy integral computation. Alternatively the fuzzy membership functions can be computed as sum histograms or smoothed by applying Parzen windows [38]. The morphological clustering of the SOM is applied on the U-matrix in order to obtain m different vectors of n components, which are the fuzzy densities (see Sec. 4.5.3). These coefficients are delivered to the module implementing the fuzzy integral (FuzInt). In the fuzzy integral module (FuzInt), the complete fuzzy measure is constructed as a fuzzy-λ measure, thus, following eq. (2.7). After applying the fuzzy integral with respect to the m constructed fuzzy measures, the fuzzy integral results have to be defuzzified. This is attained in Defuzz by finding out the argument which delivers a maximum fuzzy integral result. The class of each pixel is finally assigned to this argument.

112


Supervised approach Three are the main differences between the supervised framework and the unsupervised one. In the supervised one the user interactively selects the training set. The training set is formed by the pixel features plus a label indicating the class to which the pixels belong. Moreover a set of fuzzy membership functions for each final class is computed j h (see Fig. 6.1, where hi is to be substituted by hji for the supervised framework). The fuzzy membership functions are computed through the fuzzy hit-matrix as in the unsupervised approach. Nevertheless the computation of the set of fuzzy membership functions for a particular class is undertaken upon the fuzzy hitmatrix computed with the training data points exclusively belonging to this class and not with the whole training set. Finally another difference between both approaches is related to the necessity of using the morphological clustering. The morphological clustering is really useful in the information fusion framework. Since one fuzzy hit-matrix pro class is computed within the possibilistic theoretical framework, the prototype with the global maximum of these fuzzy hit-matrices is selected. The application of the fuzzy clustering could have been applied in this case as well, i.e. the results would have been the same, but it is not really necessary because of the existence of the labeled data. Modifications on fuzzy measures Two light modifications are undertaken in the here presented framework in order to improve the discrimination capability of the fuzzy integral for the segmentation. The first one is related to the employment of the SOM for the construction of fuzzy measures. Since the extraction of a prototype from each cluster delivers a first estimation of the fuzzy measure to be constructed related to the saliency of the feature vector, the canonical region of the clusters to be classified by the fuzzy integral can be determined. Thus a fuzzy measure can be constructed such that the coefficients affecting this canonical region satisfy the condition of a fuzzy-λ measure given in eq. (2.7): µij = µi + µj + λµi µj , while the remaining coefficients satisfy the condition of a possibility one, eq. (A.10): µij = ∨[µi , µj ]. Therefore the pixels that occupy the same canonical region as the selected prototype present larger coefficients than the rest. This measure will be denoted in the following as a mix fuzzy-λ measure. A normalization on the coefficients of the individual channels in a fuzzy measure is sometimes undertaken in the employment of the fuzzy integral as a classifier


113

[59]. Although not explicitly explained there, this operation improves the fuzzy integration. This fact can be better appreciated on the expression of the Choquet Fuzzy Integral in eq. (2.10), where the coefficients are subtracted from each other. If the values of the individual channels are very large, their difference tends to 0 and thus the result of the integral is almost equal to the larger channel. This can be avoided by applying the mentioned normalization.

6.1.2 Segmentation of benchmark color images The presented framework is first applied for the segmentation of benchmark color images from the USC-SIPI Image Database1 . Although a post-processing of the resulting images could have been applied, the results are shown “out-of-the-box” for the sake of analysis. The application of the unsupervised approach takes into account a training data set that is collected by sub-sampling the input image through LPS [5]. Moreover a 2% of the input image pixels are selected for this purpose.

(e)

(a)

(b)

(c)

(d)

(f)

(g)

Figure 6.2: Unsupervised segmentation results on the peppers image (e) in a 5dimensional color feature space RGBHS. (a) Hit-matrix. (b) Fuzzy hit-matrix. (c) Projection of RGB features on the output grid achieved by the SOM. (d) U-matrix of the output map. (f) Final label image. (g) Final label image attained by adding the pixel position (x, y) to the pixel features. The results of the framework using the fuzzy integral within the information fusion theoretical framework are given first. The peppers image is first segmented 1

USC-SIPI Image Database (http://sipi.usc.edu/services/database/Database.html)

114


with the here presented framework on a 5-dimensional color space composed by the three color channels of a camera’s RGB color space plus the hue and saturation of the HSI color space. The obtained results can be observed in Fig. 6.2. The inclusion of the pixel position in the data samples of the training set improved the results with respect to class separation (compare Figs. 6.2f and g), i.e. the “yellow” class2 is only detected after this consideration. The employment of the fuzzy hit-matrix instead of the hit-matrix improved the classification results (compare Figs. 6.2a and b). The color of the clusters in the U-matrix (see Fig. 6.2d) can be identified in the grid projection achieved through the SOM, which is depicted in Fig. 6.2c. The segmentation is achieved by binarizing the U-matrix with γ = 40 and searching for the characteristic prototype of M = 6 clusters (see Sec. 4.5.3) in the morphological clustering (see Fig. 6.2f).

Figure 6.3: Unsupervised segmentation results of the peppers image (Fig. 6.2d) on a 7-dimensional feature space RGBHS + (x, y). The results differ on the methodology used for finding out the fuzzification functions, which are applied on these features previous to the fuzzy integral. (top-left) No fuzzification function is applied. Fuzzification up to (bottom-left) probability histograms, (top-right) sum histograms, and (bottom-right) Parzen windows.

The application of fuzzy membership functions is analyzed as well. The following results (see Fig. 6.3) are obtained through the application of different 2

“Yellow” class is used in order to denote the colors similar to that of the pepper in the left-bottom corner of the image depicted in Fig. 6.2e


115

methodologies for the computation of the fuzzy membership functions, namely probability histograms, sum histograms, and Parzen windows [38].

Figure 6.4: Unsupervised segmentation results on the peppers image (Fig. 6.2d) in a 9-dimensional feature space RGBHSI + gaussian by using the fuzzy integral with respect to: (left) a fuzzy-λ measures, and (right) a mix fuzzy-λ measure.

Figure 6.5: Supervised segmentation results on the peppers image (Fig. 6.2d) in a 9-dimensional feature space RGBHSI + gaussian by using the fuzzy integral with respect to: (top-left) a fuzzy λ measure, (bottom-left) mix fuzzy-λ measure, (top-right) normalized fuzzy λ measure, and (bottom-right) normalized mix fuzzy-λ measure. The dimensionality of the output space was increased up to nine by taking into consideration the RGB, the HSI, and the Gaussian [52] color spaces. The obtained

116


results are shown in Fig. 6.4. The employment of the mix fuzzy-λ measure (see Sec. 6.1.1) lightly improve the results (see Fig. 6.4), i.e. the “yellow” class appear without taking into consideration the pixel position as feature. The performance of the framework with the fuzzy integral functioning as classifier in the possibilistic framework (supervised) is tested on that 9-dimensional color feature space (see Fig. 6.5). In this case the performance is tested for the novel type of measure, i.e. mix fuzzy-λ measure (see Fig. 6.5b), a normalized fuzzy-λ measure (see Fig. 6.5c), and both modifications simultaneously applied (see Fig. 6.5d). The most significant improvement with respect to the utilization of the fuzzy integral within the information fusion framework is the successful detection of all segments present in the training set, i.e. the “white” class belonging to the highlights of the image is not confused with the “green” one (compare Figs. 6.4 and 6.5).

(a)

(b)

(c)

Figure 6.6: Unsupervised segmentation results on the baboon image (b) in a 5dimensional feature space RGBHS. (a) U-matrix with selected prototypes in color. The color of the prototype corresponds to the color in the label image (c). The prototype in magenta is filtered out by the defuzzification stage. The framework is tested on the baboon image as well. The obtained results are depicted in Fig. 6.6. The segmentation is first undertaken on a five-dimensional color feature space RGBHS (the consideration of pixel position demonstrated to be unnecessary on this image) with parameters γ = 20 and M = 6. The position in the U-matrix of the prototypes selected through morphological clustering can be observed in Fig. 6.6a. It is worth mentioning that there are some prototypes whose results are filtered out by the defuzzification stage, i.e. the corresponding fuzzy integral result is lower than the result corresponding to all other classes for


117

Figure 6.7: Unsupervised segmentation results of the baboon image (Fig. 6.6b) in a 5-dimensional feature space RGBHS. The results differ on the methodology used for finding out the fuzzification functions, which are applied on these features previous to the fuzzy integral. (top-left) No fuzzification function is applied. Fuzzification up to (bottom-left) probability histograms, (top-right) sum histograms, and (bottomright) Parzen windows.

all pixels of the image. The results obtained on this five-dimensional space by applying the fuzzy membership functions obtained up to different methodologies are depicted in Fig. 6.7. The following results are obtained taking into consideration the RGB, the HSI, and the gaussian [52] color spaces, i.e. a 9-dimensional color feature space. These results serve the comparison between normalizing and not normalizing the fuzzy measure coefficients obtained (depicted respectively on the left and right sides of Fig. 6.8), and using a fuzzy-λ measure and the new mix fuzzy-λ measure (depicted respectively on the top and bottom parts of Fig. 6.8). The larger the dimensionality of the feature space considered, the larger are these differences, i.e. the normalization of the coefficients and the usage of the mix fuzzy-λ measure affect the result much more when taking into consideration a large dimensional space. The supervised framework is evaluated on this image as well. The results of the segmentation of the nine-dimensional color feature space are shown in Fig. 6.9.

118


Figure 6.8: Unsupervised segmentation results on the baboon image (Fig. 6.6b) in a 9-dimensional feature space RGBHSI+gaussian by using the fuzzy integral with respect to: (top-left) a fuzzy λ measure, (bottom left) new type of fuzzy measure, (top-right) normalized fuzzy λ measure, and (bottom right) normalized mix fuzzy-λ measure.

6.1.3 Image segmentation in a market basket recognition problem The developed framework is applied on the images taken for an industrial application, namely an automated cashier system for a supermarket. For this purpose the recognition of the market basket items have to be resolved. The here presented image segmentation results are an intermediate stage of this hypothetical cashier system. In this context the segmentation of the color images serves the color codification of the different items in order to characterize them. The color characterization of objects have been successfully used in the retrieval of object images in large databases [165]. The acquired images present different problems with the illumination, i.e. highlights, shadowed areas (see for instance Fig. 6.10a). The employment of the multisensory approach, which is described in the introduction of this section, seems to be justified. Thus different color spaces, which present a relative independence in front of different illumination problems [52], are computed. The features of these color spaces are expected to act as complementary pieces of information in the segmentation of the color images. The system is demonstrated on five images from the system displaying different beer cans (see Figs. 6.10a, 6.11a, 6.12a,


119

Figure 6.9: Supervised segmentation results on the baboon image (Fig. 6.6b) in a 9-dimensional feature space RGBHSI + gaussian by using the fuzzy integral with respect to: (top-left) a fuzzy λ measure, (bottom-left) mix fuzzy-λ measure, (top-right) normalized fuzzy λ measure, and (bottom-right) normalized mix fuzzy-λ measure.

6.13a, and 6.13b). The selected color spaces for the application at hand are the Gaussian, the c1c2c3 and the l1l2l3 color models [53], which are heuristically selected. However up to the preliminary results just the feature l2 of the third color space is taken into consideration. Thus the considered feature space presents 7-dimensionality. Since the performance of the unsupervised framework in the simulations described in the former section is better, this framework is the one used in the here presented application (see Fig. 6.1). The training set is made up to 5% of the image pixels, which are selected by applying the LPS algorithm [5]. No fuzzification of the features previous to the fuzzy integral is used. Therefore, the only parameters of the system are the threshold of the U-matrix (γ), the number of prototypes to be extracted from the SOM output map (M), and the normalization factor to be applied on the components of these vector prototypes before constructing the fuzzy measures (e.g. see Fig. 6.6). Although the range of the threshold to initially segment the U-matrix in the morphological clustering of the SOM is constant for the different images, i.e. γ ∈ [20, 50], its exact value depends on the properties of the matrix (see Figs. 6.10, 6.11, and 6.12). Therefore, both γ and M are determined ad hoc. The deter-

120


(c)

(d) (a)

(b)

(e)

Figure 6.10: Unsupervised segmentation of beer can of Type I with the here presented framework binarizing the U-matrix with grayvalue γ = 25, and extracting M = 6 clusters. (a) Input image. (b) Input image in Gaussian color space. (c) Projection of the Gaussian features achieved by the output map that can be used in order to identify the different color clusters in the U-matrix (d). (e) Results of the defuzzification of the fuzzy integral with respect to (from left to right): fuzzy-λ measures, mix fuzzy-λ measure, fuzzy-λ measures with previous normalization (factor 6) of the fuzzy densities, and mix fuzzy-λ measure with previous normalization (factor 6) of the fuzzy densities


121

(c)

(d) (a)

(b)

(e)

Figure 6.11: Unsupervised segmentation of beer can of Type II with the here presented framework thresholding the U-matrix with grayvalue γ = 20, and extracting M = 6 clusters. (a) Input image. (b) Input image in Gaussian color space. (c) Projection of the Gaussian features achieved by the output map that can be used in order to identify the different color clusters in the U-matrix (d). (e) Results of the defuzzification of the fuzzy integral with respect to (from left to right): fuzzy-λ measures, mix fuzzy-λ measure, fuzzy-λ measures with previous normalization (factor 4) of the fuzzy densities, and mix fuzzy-λ measure with previous normalization (factor 4) of the fuzzy densities

122


(c)

(d) (a)

(b)

(e)

Figure 6.12: Unsupervised segmentation of beer can of Type III with the here presented framework thresholding the U-matrix with γ = 33, and extracting M = 6 clusters. (a) Input image. (b) Input image in Gaussian color space. (c) Projection of the Gaussian features achieved by the output map that can be used in order to identify the different color clusters in the U-matrix (d). (e) Results of the defuzzification of the fuzzy integral with respect to (from left to right): fuzzy-λ measures, mix fuzzy-λ measure, fuzzy-λ measures with previous normalization (factor 2) of the fuzzy densities, and mix fuzzy-λ measure with previous normalization (factor 2) of the fuzzy densities


123

mination is based on the observation of the U-matrix and of the projection of the color features achieved by the SOM. The parameters are modified till one prototype pro color cluster is selected. One of the detected problems in this process is the existence of smooth transitions among some clusters in the U-matrix, i.e. the clusters are not separated by an “edge” of high valued distances (see for instance the prototypes marked in green and blue on Fig. 6.10d). Moreover small color clusters, which are very well detected by the human eye, do not have a high number of hits in the matrix (e.g. see the cluster corresponding to the golden stripes on Fig. 6.12d). Thus, it results quite difficult to refine the value of γ in order for the two clusters to be separated.

(a)

(b)

(c)

(d)

Figure 6.13: Unsupervised segmentation of beer cans of Type IV and Type V with the here presented framework. (a) Type IV input image. (b) Type V input image. Results of the defuzzification of the fuzzy integral with respect to (from left to right): fuzzy-λ measures, mix fuzzy-λ measure, fuzzy-λ measures with previous normalization of the fuzzy densities, and mix fuzzy-λ measure with previous normalization of the fuzzy densities for (c) Type IV, and (d) Type V. The framework is tested by computing the fuzzy integral with respect to fuzzyλ measures and to the novel type of measure, i.e. mix fuzzy-λ measure. In

124


both cases the normalization of the fuzzy densities is taken into consideration. The different results are depicted in Figs. 6.10e, 6.11e, 6.12e, and 6.13cd. No systematic approach for the determination of the normalization factor of the fuzzy densities could be found.

6.2 Automated visual inspection of collagen plates The utilization of automated visual inspection systems for quality control on textured surfaces is an outstanding field of research in computer vision. In this context industrial visual inspection is leaving behind the traditional application goal of finding functional faults in textured materials trying to conquer new and more complex goals. The hitherto most used approach of detecting a fault, assigning it to a fault class, and delivering the crisp results to the superposed production system has lost importance due to economic demands. The possibility to reuse some parts of the produced goods, to redirect its selling to alternative channels of distribution or to take advantage of the long experience of human operators in inspection tasks encourage the establishment of new paradigms for automated visual inspection systems. These new paradigms should consider the detection of perceptually relevant anomalies of any kind, which might disturb the total visual appearance of the surface texture. The resolution of such problems are characterized by the following main features: • The items subject to inspection are in its majority end consumer goods. • Only part of such faults are of functional nature, or at least malfunction plays a secondary role. • It is desired that the inspection system delivers a bit more complex description of the faultiness degree than the two-valued decision fault or not. Being the inspected objects end consumer goods, the goal to be attained by the inspecting system is the prediction of the response of the end user in front of this item. This response is not based in a clear decision on some quantitative or clearly defined aspect, like the decision weather a bulb is broken or not, but on its subjective impression. Therefore no clear parameters, which otherwise would mean the malfunctioning of the item, can be defined on the pieces to be rejected. Furthermore the subjective impression of the end user is affected by different factors, which yield her or him to make a decision. Thus the quality control is preventing the product end user of eventually rejecting some items due to their aesthetic value. As all aesthetic evaluation this can not be a crisp one, but have to take into consideration the way humans usually express values, namely in a fuzzy way [192]. Taking all these facts into consideration it can be stated that traditional visual inspection systems detect, whereas modern ones have to interpret (see fig. 6.14).

6.2 Automated visual inspection of collagen plates

(a)

125

(b)

Figure 6.14: Challenge for automated visual inspection systems of textured surfaces: from the detection of faults eventual producing malfunctioning (a) to the the interpretation of “ugliness” (b). (a) Automotive mechanical piece. (b) Organic material plate used in the cosmetic industry.

Wooden and textile surfaces have been a long-termed research target in this direction. However, a general attempt to treat this class of inspection problems has not been considered so far. In this context this novel approach for automated visual inspection will be denoted as perceptual relevance evaluation [89][159]. Perceptual relevance evaluation is mainly characterized by its subjective nature. So far, the consideration of Fuzzy Computing (see Secs. 2.4 and A.1) in the quantification process seems unavoidable [159]. Furthermore perceptual faults are stuck on both, local and global image properties. A floor tile might successfully pass all fault tests by itself, whereas it may fail in a pavement due to its perceptual relevance (e.g. a large-scale regularity). Thus the goal of the inspecting system is to detect these different properties as a human being would do, to quantify the importance of these features in the assessment of the final aesthetic value of the item, and finally the reproduction of the end user reaction. A framework for fuzzy evaluation of perceptually relevant faults was developed [89][159] in order to fulfill the formerly described challenges of this new paradigm in automated visual inspection. In the following the different fuzzy strategies employed in the framework will be analyzed and the obtained results presented.

6.2.1 Framework for perceptual relevance evaluation A system for the automated visual inspection of texture surfaces was implemented [89]. The purpose of the visual inspection is the evaluation of the perceptual relevance of different fault types on plates of an organic material. The framework is basically composed of a processing chain of alternating Binary Pattern Processing Modules (BPPM), each of them having the same internal structure. Some

126

6 Evaluation of soft data fusion procedures in Image Analysis Acquired Image

BPPM 1 N F

Y

Test 12

Y

BPPM 2 N

F

Y

Test 23 N

Y

BPPM 3

...

BPPM n Y ok

...

N F

Figure 6.15: Proposed framework for perceptual relevance evaluation. Different types of fault are analyzed by the Binary Pattern Processing Modules (BPPM ), which take as input the acquired image. These modules are organized from less to more complex (from left to right). The modules denoted as Test are faster routines that check if the fault to be analyzed by the next BPPM may be found in the inspected item. These two features attain the optimization of the system processing time.

additional testing modules can be found between such modules (see Fig. 6.15). The testing modules, where a fast testing routine based on reduced information is realized, are optional design components. Their main purpose is to bypass a BPPM, if there is no evidence for the faults, which are processed by that BPPM, to exist. Each BPPM has access to the acquired image and to the evaluation of the foregoing testing module. Hence the processing of each BPPM is independent of the processing of the others, but may refer to the results of the foregoing modules. Modules for the detection of the more frequent faults, or of the more simple to detect faults should come first in the chain. This way the system is computationally optimized, since the appearance of a sufficient relevant fault in foregoing modules can interrupt the more time consuming computation in later ones. Each single BPPM is designed for a special fault class related to the fault appearance. However it is always composed by a preprocessing, a binarization, and a fuzzy evaluation stages (see Fig. 6.16). These stages are described in the following paragraphs. First, the image delivered by the camera is preprocessed in the module Preprocessing in order to avoid the inhomogeneities introduced by the lighting system. This stage is specially important in case the system is applied for the inspection of organic textures. In the binarization stage (modules Binarization1 - Binarizationk ) a set of k algorithms is performed in parallel, what results in k binary images. The purpose of the binarization multiplicity within a BPPM is the acquisition of complemen-

127


... ... ... ...

Binarizationk

Binarization 2

Binarization 1

Preprocessing

next test module

Binary Fusion CCA Fuzzy - Features Fuzzy Fusion

Decision

Y

N fault

Figure 6.16: Block diagram of a Binary Pattern Processing Module (BPPM). After pre-processing the input image several binarization procedures are applied in parallel. The resulting binary images undergo a fusion operation. Thence isolated pixels are filtered out and regions are made more compact by a process of Connected Component Analysis (CCA). Afterwards different fuzzy features are extracted from these compact blobs. Finally a fuzzy fusion operator is applied in order to evaluate the perceptual relevance of the analyzed fault.

tary pieces of evidence, which have the form of binary images, in the analysis of the object under inspection. Four basic designs of binarization have been implemented so far. Each of these different strategies is expected to cope with the following types of faults (for a deeper explanation the reader is referred to [89]): Strong-contrast localized faults appear as a strong contrast of the faults with respect to the background resulting from the material’s normal aspect. Therefore an interval thresholding normally suffices in the successful detection of this type of faults, i.e. only one binarization (k = 1) operation is applied in this case. Long-range faults are not related to a strong local contrast, but to a distortion within the global distribution of grayvalues within the image. Such faults may be detected by employing the auto-lookup procedure [87]. This procedure, which acts as a novelty filter, delivers a look-up table by computing the cooccurrence matrix [66] on a subset of pixels from the image. The look-up table is thence employed in order to transform the input image with a reduced number

128

6 Evaluation of soft data fusion procedures in Image Analysis Auto-Lookup Images Cooc1

Cooc2

Binary Fusion

Original Image Cooc3

Cooc4

Cooc5

Figure 6.17: Auto-Lookup based procedure. From the acquired image, five cooccurrence matrices [66] are derived at five randomly located windows. Then, for each co-occurrence matrix, the auto-lookup procedure [87] is performed on the whole image. As a result dark regions stand for “atypical” regions.

of grayvalues (see Fig. 6.17). Low-contrast faults are characterized by its low contrast in the image grayvalues. For their treatment, the framework called Lucifer2 [87] may be used. The purpose of the framework is the automated generation of texture filters given the original and the expected goal images through the application of a genetic programming procedure, where different basic operations, e.g. erosion, addition of pixel values, are combined. Frequency related faults appear in form of a disruption of the uniformity resulting from the repetition of a basic element. For its detection a Gabor image decomposition [29] [47] for different spectral bands are calculated following the scheme presented in [116]. Finally, the total image energy in the analyzed spectral bands is computed and thresholded. A fusion procedure derives a binary image by applying logical operators on the k images delivered by the binarization stage. The foreground (or black pixels) of that binary image are candidates for perceptual faults which will be fuzzy processed in the following stages. The module CCA (connection component analysis) is performed in order to select connected components from the binary fused image, which are perceptual fault regions, and to remove isolated pixels that


129

remained from the binary fusion, which are expected to be noisy pixels. The fuzzy evaluation of the binary image is undertaken by a network of fusion operators [159]. Thus, the network achieves the path between binary pixel information and the relevance of the defect under inspection. This stage is implemented in the modules Fuzzy-Features, and Fuzzy Fusion, which will be deeper analyzed in Sec. 6.2.2 as a whole. As a result each BPPM delivers a duple of class membership degrees (“no relevant faultiness”, “relevant faultiness”). For instance, the vector (0.3, 0.7) would indicate an inspected object where the analyzed texture fault are perceptually relevant with degree 0.7. The final decision for the rejection of a piece upon each defect type is undertaken by the production system based on this information. In case of a very complex interaction between the different defect types another fuzzy integral could be used in the highest level of abstraction for the integration of their “faultiness”/”non-faultiness” membership degrees.

6.2.2 Network of fuzzy aggregation operators for feature analysis The binary images extracted by the parallel binarization submodules and already filtered by the module CCA (see Fig. 6.16) have to be fuzzy processed in order to better approximate the desired subjective evaluation of the textured plates [159]. Therefore, the analysis of the resulting binary images is undertaken at this point by a network of fuzzy aggregation operators (see Fig. 6.18), where the complexity of the problem attached in each following stage needs the usage of operators of also increasing complexity. This scheme follows the theoretical considerations refer to the theoretical framework of soft data fusion (see Sec. 4.2). With each fusion operator in the network a new abstraction level in the way from pixel information to the quantification of the defects relevance is achieved. The different stages will be analyzed in the following. The network of fuzzy aggregation operators is implemented through the stages of the BPPMs Fuzzy-Features, and Fuzzy Fusion (see Fig. 6.16). First different functionalities implemented though the module Fuzzy-Features are described. The connected components, which result from the CCA stage, are first measured (see Fig. 6.19) through classical operators. These operators are for instance the statistical first moment of the black pixel positions, which is used in the determination of the center of mass (COG) of the faults, or the sum of pixel values (1 or 0), which is used for the determination of the area or the perimeter of the faults. Thus some local features for each detected component of the inspected object are obtained. These features geometrically describe each component e.g. height, width, position, area, perimeter, roundness. Once the local features are obtained, a holistic description of them is needed in order to characterize the plates under inspection as a whole. This global

130


binary image fault type i pixels

CLOP

local features

CLOP

OWA

global features

FUZZ fuzzy features

big cracks 0.79

near border 0.35

defect i relevance

faultiness 0.647

non-faultiness 0.181

CFI

Figure 6.18: Hierarchically organized network of fusion operators used for the fuzzy evaluation of binary images. The operators of increasing softness are in charge of tasks of increasing complexity. These correspond to increasing levels of abstraction in the evaluation from grayvalue pixel information, g(x, y), to the result in form of a double fuzzy membership degree of perceptual relevance (“relevant faultiness”, “non-relevant faultiness”). CLOP: Classical Operators; OWA: Ordered Weighted Averaging operators; FUZZ: fuzzification stage; CFI: Choquet Fuzzy Integral.

131


Width: 23Pixel COG: (893,1712) Height: 23Pixel

Area: 173Pixel Perimeter: 43Pixel

Figure 6.19: Measuring of binary patterns on connected components through classical fusion operators.

descriptor can be considered as a meta-feature, which results from the fuzzy aggregation of the local fault features. Two different strategies are used for that end. On the one hand there are some global features, whose trivial computation can be directly attained based on the information already obtained. For instance the computation of the number of candidate faults in a plate can be achieved by a sum operator on the connected components already detected. On the other hand Ordered Weighted Averaging (OWA) operators can be applied on the local features in order to obtain some global features, whose computation results a bit more complex. The weighting configuration of the OWAs, which are softer aggregation operators than classical ones, increase the flexibility in obtaining the global descriptors of the inspected objects (see Sec. 4.2). Furthermore, such a weighting process is made in these fusion operators taking into consideration the numerical ranking of the features. Therefore the result can be biased for giving preference to a determined range of values or for reinforcing the presence of coincident ones [190]. An example of the usage of the OWA weights and its effect on the result is shown in Fig. 6.20. The result is a generalization of the mean value of the local features embedded in the plate (see Sec. B.1). The definition of the weights with respect to the ranking of the incoming values, which results from softening the operator (see Sec. 4.2), allows overcoming the low-pass effect of the average. Thus extreme values will not necessarily be filtered out (this will depend on the selected weight distribution). The consideration of the OWA instead of a weighted sum operator is compulsory, since the local features are not given in any prefixed channel distribution. The application of the OWA operators practically succeeds as described in the following. First the weight distribution is built upon the kind of meta-feature that is going to be computed. Once the distribution is established the continuous

132

6 Evaluation of soft data fusion procedures in Image Analysis WOWA

OWA = 0.164

P/A = 0.08

0.65

P/A = 0.62

0.1

0.2 0.05 (1)

(2)

(3)

(4)

ranking

1.0

LINE

CIRCLE

WOWA OWA = 0.4895 P/A = 0.14

P/A = 0.53

0.45 0.35 0.15 0.05 (1)

(a)

meta-roundness

(2) (b)

(3)

(4)

ranking (c)

Figure 6.20: Exemplary employment of OWAs in the computation of fuzzy metafeatures. The image shows the exemplary computation of the metaroundness of a plate based on the fuzzy aggregation of the local features Perimeter/Area (P/A) (a). (b) A weight distribution very sensitive to the presence of just one fault with circular form is displayed at the top. This distribution will operate on the depicted features as: OW A = 0.62∗0.65+0.53∗0.2+0.14∗0.1+0.08∗0.05 = 0.164. On the contrary if a detection of more than one elongated fault is desired, a configuration like the one at the bottom would be used. This distribution results as OW A = 0.62 ∗ 0.05 + 0.53 ∗ 0.15 + 0.14 ∗ 0.35 + 0.08 ∗ 0.45 = 0.4895 (c) Example of fuzzification functions for the computation of fuzzy meta-features (in this case, meta-roundness). The result of the OWAs is fuzzified with trapezoidal fuzzifying functions.

curve is sampled in order to obtain so many weights as needed for the computation of the OWA, i.e. the number of features that the object presents. This procedure allows to obtain similar values of the meta-feature for a different number of features. This property, namely allowing the system to compare features of different dimensionality, is an significant contribution to the field of Pattern Recognition. The obtained meta-features are fuzzified by defining linguistic terms with trapezoidal fuzzy membership functions (see Fig. 6.20c). By means of the fuzzification the global descriptors are obtained in a linguistic manner, what eases the conceptual development and parameterization of the following stages. Two different sets of fuzzy membership functions are used in the fuzzification of the extracted meta-features (see Fig. 6.18). The first set refers to positive features, e.g. “small fault”, while the other one refers to its negative counterpart, e.g. “large fault”.


133

Moreover the usage of two opposed fuzzy features increase the robustness of the evaluation system. Once the global fuzzy meta-features are computed, they are fused in the module Fuzzy Fusion. For each perceptual fault class, these features are fused into a value within the interval [0, 1] by using the Choquet Fuzzy Integral. This operation aims finding the perceptual relevance of possible faults in each inspected object in form of a membership degree of the fault being analyzed. The application of a fuzzy fusion operator can produce such a membership function (see Sec. B.8). The most important reason for the application of the fuzzy integral is the capability of this fuzzy operator for fusing information taking into consideration the a priori importance of both individual and groups of attributes. The fusion of the different fuzzy meta-features is needed in order to find the joint perceptual relevance of the faults in the object under inspection. In such a process the interaction between the different fuzzy meta-features has to be considered. The fuzzy integral is the only fuzzy fusion operator to allow such a characterization [58]. The kind of analysis undertaken reflects in the result the different possibilities of interaction, e.g.: if the presence of defects on the plate border is very important, the result of the relevance quantification should increase; if such a presence is not so important but coincides with a very big defect, the relevance should also increase; if the defects are small and there are not so much of them, the relevance decreases. Such a characterization could have been undertaken with a system of fuzzy rules as well. However, the fuzzy integral approach is more synthetic and understandable from the developer point of view. When the kind of interactions to be characterized are very complex or numerous, the number of rules increases so much that the problem is not more tractable. Furthermore, in many cases the descriptions delivered by the inspection experts can be ambiguous. The data-driven construction of the fuzzy measures helps overcoming this problem and allow an easier redesign of the feature extraction stage. The Choquet Fuzzy Integral is selected for the classification stage. This type of integral has been already used as classifier in pattern recognition problems, where its performance was superior to that of more traditional classifiers, e.g. multilayer perceptron and Bayesian independent classification, in three standard benchmark classification problems [60]. Furthermore the parameterization of the fuzzy integral is taken into consideration for the selection of this type of fuzzy integral. Diverse algorithms have been presented for the parameterization of the Choquet integral, while the Sugeno integral lacks of such a diversity [58][60][178]. It is worth making a deeper analysis of the automated parameterization of the Choquet Fuzzy Integral. In any industrial application, as the one presented here the expert knowledge is difficult to collect from the system end-users. The characterization of the different fault types is not exact, if not presenting some contradictions. Moreover the importance of the different fault types and of its interaction is neither easy to characterize. The automated parameterization of the fuzzy integral based on a training set of plates can facilitate the characterization

134


of the faults’ importance and that of their interaction. The training sets are collected by the human visual inspectors, which deliver together with the items to be evaluated by the automated system their own evaluation on the acceptance degree. Finally the automated parameterization increases the flexibility of the system in front of changes in the quality standard specification. So far, some experiments were done for the automated assessment of the fuzzy measures for the CFI by using an optimization technique based on quadratic programming [57][60]. Those experiments did not present any successful result. This may lie on the necessity for the optimization technique to have a clear distinction between the membership of the classes. In case of an ambiguous definition of the classes, the quadratic programming delivers trivial solutions. Therefore, Genetic Algorithms are selected for the construction of fuzzy measures (see Sec. 4.5.1). One important point of discussion in the parameterization of the Choquet fuzzy integral through the application of genetic algorithms is which strategy to use when computing the fitness value. Two conditions were used for that purpose. The first condition was the minimization of the error between the fault perceptual relevance established by the experts (Fbµ (x)) on the plates belonging to the training set and the result delivered by the fuzzy integral (Fµ (x)), as stated by eq. (2.11): ǫ = |Fbµ (x) − Fµ (x)|.

The second one was the consideration of a trend complementing such error minimization. Such a trend succeed in form of a condition that biases the result of the fuzzy integral in order to present a maximal difference between the classes (“no relevant faultiness”, “relevant faultiness”). The computation of the fuzzy integral is undertaken with respect to a general fuzzy and to a fuzzy-λ measure (see Sec. A.3). The usage of fuzzy-λ measures avoids the consideration of the monotonicity condition of general fuzzy measures. Moreover, it reduces the search space of the GA, what leads to shorter training times. On the other hand the consideration of the monotonicity condition of the general fuzzy measures requires a modification of the results rawly delivered by the genetic algorithm (see Sec. 2.8.1). The here proposed strategy, where δi as expressed by eq. (4.17) is used to decrease the fitness value of the individuals (see Sec. 4.5.1), is used in the final system because of its simplicity.

6.2.3 Perceptual relevance evaluation on collagen plates The presented framework for the evaluation of fault perceptual relevance is applied in an industrial application for the evaluation of collagen plates. These collagen plates are used in skin care applications both as post-surgical treatment in the health sector and as beauty treatment sold by the cosmetic industry. The system analyzes different types of faults, whose existence may cause rejection on


135

the end users although the functionality of the product may not be necessarily affected by them. Different faults are automatically inspected by the system, e.g. presence of holes, thinner areas, dirty areas, so-called knife faults, harder areas, texturedareas, three-dimensional structures on the plate. These faults were binarized through different BPPMs for the detection of strong-localized, long-range, and low-contrast faults (see Sec. 6.2.1). The current configuration of the system can not be disclosed due to an agreement with the contracting enterprise. The implemented system does not make a decision on the general perceptual relevance of all the faults, but on the perceptual relevance of each type of fault individually taken, i.e. the system ends up with the fuzzy fusion stage (see Fig. 6.16). Thus, the result of the system is a two-component vector for each plate and fault, which expresses the membership degree of the plate to the “accepted” and the “rejected” classes. This vector can be understood as a two-component feature for each fault type, upon which an end decision of the commercialization channel of the corresponding plate is to be made, e.g. some plates are sold as first class, other ones as second class, and some of them are not sold at all. The results of the system are analyzed by taking into consideration just the image analysis part of the system. Thus the binary images are supposed to be already computed. Up to this point the features are extracted and fuzzified by undergoing the described network of fuzzy fusion operators (see Fig. 6.18). It is worth mentioning that the fuzzy membership functions are heuristically defined on hand of the classification results. This process is not trivial and the achieved classification results demonstrated to depend on this stage. Therefore its automation can systemize the future implementations of the framework. The results of the stage, where the resulting two sets of fuzzy meta-features are fused with a Choquet Fuzzy Integral with respect to two different fuzzy measures, are analyzed in the following. The fuzzy measures are constructed through Genetic Algorithms. The genetic algorithm implemented is a steady-state genetic algorithm, which guarantees not losing the better result in each generation. The individuals of the population code the fuzzy measure coefficients in the real domain. The genetic algorithms employs: random initialization, a rank selection operator, a two-point crossover operator with probability of 0.7, mutation probability of 0.01, and replacement probability of 0.95 [54]. The probabilities of the genetic algorithm are heuristically found. Finally the perceptual relevance of the fault is delivered in form of a twocomponent vector, where each component is defined in [0, 1]. The generalization capability of the system is first analyzed (see Fig. 6.21). Fuzzy-λ measures are used for that purpose. The BPPM taken into consideration analyzes the relevance of textured areas on the plates. The training of the fuzzy integral was undertaken for this BPPM based on a set of 39 plates. The obtained results on the training set after determining the fuzzy-λ measure are depicted in Fig. 6.21a. The consideration of the trend condition in the fitness

136

6 Evaluation of soft data fusion procedures in Image Analysis (a)

(b)

fuzzy integral result

1

1

0.9

0.9

0.8

0.8

0.7

0.7

0.6

0.6

0.5

0.5

0.4

0.4

0.3

0.3

0.2

0.2

0.1

0.1

0

0 0

5

10

15

20

25

data sample index

30

35

40

0

5

10

15

20

25

30

35

40

45

50

55

60

65

70

75

80

85

data sample index

Non-relevance degree as set by experts "Non-relevant faultiness" membership degree computed by the Fuzzy Integral "Relevant faultiness" membership degree computed by the Fuzzy Integral

Figure 6.21: Results of collagen plate inspection for a training and a test data sets. (a) Results obtained on a 39 plates training set. (b) Results obtained on a 84 plates test set with the automatically determined fuzzy measure coefficients. The results are computed through a Choquet Fuzzy Integral with respect to fuzzy-λ measures.

function as described in the former section improved the results in terms of false accepted/rejected rates and of approximation to the relevance established by the experts. Taking into consideration the larger membership degree as a crisp result the system achieved a recognition rate of 84.61% on the training set. The results obtained on a test set of 84 plates are depicted in Fig. 6.21b. The recognition rate with the test set was of 70.23%. On hand of these results the generalization capability of the evaluation framework can be analyzed. In another simulation the influence of type of fuzzy measures to be used in the classification result is analyzed. Thus the training phase is completed using a data set of 129 items. The simulation compares the classification results obtained by using λ− and general fuzzy measures (see Fig. 6.22). Although the recognition rate is the same (86.82%) for both types of fuzzy measure in the here presented simulation, the greater flexibility of the general fuzzy measures in front of the λ ones improve the performance of the system (see Tab. 6.1). This lightly greater flexibility can be better observed on hand of the mean square error (MSE) between the relevance curve determined by the experts Fbµ (x) (see gray surface in Fig. 6.22) and the results of the fuzzy integral Fµ (x). The MSEs and their variances are summarized in Table 6.1. Although the general fuzzy measures present in all data sets a better performance, the difference between the results obtained with both types of measures strongly depends on the considered data set. Since the fuzzy-λ measures are

137



(a)

(b)

1

1

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2

0

0 0

20

40

60

80

100

120

0

20

data sample index

40

60

80

100

120

data sample index

Non-relevance degree as set by experts "Non-relevant faultiness" membership degree computed by the Fuzzy Integral "Relevant faultiness" membership degree computed by the Fuzzy Integral

Figure 6.22: Comparison of Choquet Fuzzy Integral with respect to fuzzy-λ and general fuzzy measures in a system for collagen plates inspection. Results obtained on a 129 plates training set (a) using general fuzzy measures and (b) fuzzy-λ measures.

Table 6.1: Statistical comparison of results obtained through the application of the fuzzy integral with respect to general (µ) and fuzzy-λ measures (µλ). ρ: Crisp recognition rate of the system by taking into consideration the larger membership degree as a crisp result. MSE: Mean square error between the “relevance” membership established by the experts and the one obtained. σM SE : Variance of the mean square error. Values are taken as average over different simulations and computed for three different data sets.

µ µλ

Data Set I ρ MSE σM SE 92.2 0.15 0.01 83.0 0.15 0.01

Data Set ρ MSE 86.7 0.167 83.0 0.16

II σM SE 0.041 0.04

Data Set I+II ρ MSE σM SE 86.82 0.184 0.028 86.82 0.184 0.034

a particular case of general fuzzy measures, this last type is preferred for the implementation of the final system. The implemented industrial system was tested with 11 different evaluation data sets, which are composed by 100 plates taken directly from the production line. The results obtained on these evaluation sets can be observed in Tab. 6.2. The system had been previously trained and tested with other data sets. Thus the results can be used for the evaluation of the system’s generalization as well. Three parameters characterize the goodness of the system. First the percentage

138


Table 6.2: Statistical analysis of the results obtained by the automated industrial system on 11 evaluation data sets each formed by 100 plates taken from production line. EFT: Error in fault type. FR: False rejection. FA: False acceptance. ⋆m : Minimum. ¯⋆: Average. σ⋆ : Variance. ⋆M : Maximum. These operators are applied to the formerly mentioned error rates, i.e. ⋆ ∈ {EF T, F R, F A}. Set EFT FR FA

1 2 1 1 1 3 24 9

3 7 1 21

4 5 6 13 9 2 0 3 9 17 11 3

7 5 2 5

8 9 10 11 13 4 0 4 4 5 6 5.3

11 4 4 11

⋆m 1 0 3

¯⋆ σ⋆ ⋆M 6.4 4.5 13 2.8 2.6 9 10.7 7.1 24

of plates that present a false fault type is given (EFT). This error occurs when a particular fault type, e.g. holes, is detected on a plate instead of the existent one, e.g. dirtiness. The false rejection (FR) and false acceptance (FA) rates are given as well. These error rates characterize the detection of a fault in a right plate (FR) or the acceptance of a faulty plate (FA). The results substantially differ from the performance on the training and test data sets, because these evaluation data sets, as already mentioned, are taken from the production line, and, therefore, present a majority of non-faulty collagen plates.

139

7 Conclusions This chapter is devoted to the analysis of different contributions attained in the here presented dissertation. For the sake of compactness the conclusions are summarized in this unique chapter instead of the customary concluding remarks at the end of each section. The contributions related to computer vision systems and specially those related to the inclusion of diverse imaging information sources are first described. Second the main theoretical contributions of the here presented dissertation in the revisited field of intelligent multi-sensory fusion, which are achieved through the employment of the fuzzy integral as fusion operator, are analyzed. Finally the results of the developed frameworks in different applications for Image Processing and Image Analysis are commented.

7.1 Multi-sensory computer vision systems and fuzzy fusion operators The realization of computer vision systems for solving applications in the real world has demonstrated much more complex than the visionary analysis in the field had predicted. In spite of the relative maturity of computer vision as research field, few systems have reached the “real world”, where the system implementation is challenged by the absence of constraints. Thus, two different conceptual aspects are introduced in Chap. 2, which are expected to offer new perspectives in the field and mention in the sequel. The first concept is related to the structure of computer vision systems. Their generic linear structure, i.e. a chain of image pre-processing, segmentation, feature extraction, and classification functions, has been substituted by a recursive one, in which these stages are repeated at decreasing scale levels until the goal object has been selected and analyzed (see Sec. 2.2.1). This conceptual approach copes better with the complexity of visual scenes than the traditional one. Second the goal complexity in computer vision systems can be tackled by taking into account different information sources, which are then used as different pieces of evidence in the resolution of a particular problem (see Sec. 2.3.2). Thus any stage in a computer vision system is better attained by complementary systems analyzing the imagery under different perspectives. Furthermore this fact specially relates to the image acquisition stage, which traditionally attains the

140

7 Conclusions

measurement of the environment with a particular imaging device (see Sec. 2.2.2). In the multi-sensory approach the scene being analyzed is on the contrary gathered in different spectral bands, which allows different features of the objects become visible to the system. The diversity of information at any processing stage implies the necessity of summarizing it at any posterior stage. This is achieved by means of multisensory fusion, whose difference with the multi-sensory integration is elucidated (see Sec. 2.3.1). The multi-sensory fusion attains the transformation of the data set delivered by diverse sensors into one representational form. Under this definition, different theoretical frameworks have been used in computer vision for the fusion of multi-sensory information (see Sec. 2.3.2). Nevertheless, this definition can be further particularized through the concept of fusion operator, which is a mathematical function transforming the incoming multi-dimensional data into a one-dimensional representational form. Fusion operators, which have been especially developed in the theoretical framework of Soft Computing, particularly in Fuzzy Computing (see Sec. 2.5), are herein denoted as fuzzy fusion operators. The necessity for the existence of such operators founds its theoretical background in a the Ugly Duckling Theorem (see Sec. 2.3.3), whose relationship with fuzzy fusion operators constitutes one of the novelties brought up in the here presented dissertation. Among all fuzzy fusion operators the fuzzy integral plays a principal role. The fuzzy integral is briefly analyzed in Sec. 2.5.2 from a mathematical point of view. In this context, the fuzzy integral presents a great flexibility and robustness for the resolution of image processing and pattern recognition problems, constituting the main reason for its selection as fusion operator in real applications. The mentioned section presents an overview on the fuzzy integral (see Appendices A and B as well) and summarizes the most important points of its mathematical background, whose lack of broad dissemination can be considered as one of the most important obstacles for its widening application in computer vision systems. The fuzzy integral results from the generalization of most known fusion operators (see Sec. B for a formal discussion). In this sense, they extend the concept of norm in statistical metric spaces (see Sec. 2.5.1), whose implementation is used for the computation of similarities in the feature hyper-cube. The definition of one set of weighting functions for each canonical region of feature hyper-cube improves the robustness of the operator. There are still some open questions on the research around the fuzzy integral. From a theoretical point of view, the definition of the concept of interactivity among the information sources included in an artificial system and its relationship with the additivity of the fuzzy measures deserves further research attention. Furthermore it is not well defined which type of integral to choose for a particular application. Finally the most important problem for the wide utilization of the fuzzy integral in industrial applications is related to the assessment of the fuzzy measure coefficients (see Sec. 2.7). In this context, data-driven procedures, which

7.2 Intelligent multi-sensory fusion, soft data fusion, and the fuzzy integral 141 are embodied in the theoretical framework of Soft Computing, offer the most promising options (see Sec. 2.8). The most used methodology for this purpose is the application of Genetic Algorithms, which have to take into account the monotonicity of the fuzzy measures in order to succeed. Moreover, the SOM has been used for the supervised construction of fuzzy measures.

7.2 Intelligent multi-sensory fusion, soft data fusion, and the fuzzy integral A framework for the implementation of computer vision systems with information fusion based on fuzzy fusion operators, devoting especial attention to the fuzzy integral, (see Chap. 4) is realized by further developing different conceptual aspects around this fusion operator. The utilization of such flexible and robust fusion operators supports the conceptualization of a theoretical model of information fusion, which is denoted as intelligent multi-sensory fusion (see Sec. 4.1). In this conceptual context, the fusion operator subsumes the mere fusion of the data by modeling the extracted information and selecting among the information sources. This functionality is reserved to independent modules in the classical approach of multi-sensory integration (see Sec. 2.3.1). Therefore its integration in one module characterizes the concept of intelligent multi-sensory fusion, which can be just realized through the automated assessment of the different parameters involved in the operation of the fuzzy integral. The flexibility of the fuzzy integral as fusion operator is conceptually denoted by the term soft data fusion, which is elucidated in Sec. 4.2. Soft data fusion is a theoretical framework, whereby fuzzy fusion operators are characterized by the number of freedom degrees enabled for the computation of the fusion operation result. The existence of soft data fusion as theoretical framework is expected to help an engineer to select among the different fuzzy fusion operators. The characterization through the softness degree of the operators facilitates meeting the trade-off between the necessary flexibility and the computational cost of using a particular fuzzy fusion operator for a particular application. The aforementioned robustness of the fuzzy integral is based on the consideration of the ranking among the data delivered by the information sources, which results from the extension of the freedom degrees. Different questions related with the practical implementation of the fuzzy integral are herein taken into account (see Sec. 4.3.1). These include a suitable data structure for the implementation of fuzzy measures -which is given in form of a lattice-, the customary employment of a normalization factor on the coefficients of the individual channels, and a mixed type of fuzzy measure -which is denoted as mix fuzzy-λ measure and expected to outperform other types of measures in critical cases.

142

7 Conclusions

The complexity of the fuzzy integral makes necessary the analysis of the particularities of its usage in computer vision applications (see Sec. 4.3.2). In its application for Image Processing, the Sugeno Fuzzy Integral behaves like a multithreshold procedure, whereas the Choquet one operates average-wise on the input image channels. Both integral types take advantage of the definition of the fuzzy measure coefficients, i.e. the weighting scheme used by the operator, with respect to the canonical regions of the image channels hyper-cube. When using the fuzzy integral as classifier, what is applied for Image Analysis, the Choquet integral generally outperforms the Sugeno integral. Other types of fuzzy integral, e.g. the Fuzzy T-conorm Integral, have been seldom used for Image Processing and for Image Analysis. The flexibility of the operator is improved by the local definition of the fuzzy measure coefficients that is attained through the novel image processing paradigm denoted as intelligent localized fusion (ILF) (see Sec. 4.4). The operators developed within it are tailored for the application of the fuzzy integral in image processing applications. In this sense, the fuzzy integral achieves a new degree of softness through the development of ILFOs, because of the increment of the number of weighting parameters used in the fusion operator, New methodologies for the automated construction of the fuzzy measures have been presented in this chapter (see Sec. 4.5). The usage of Genetic Algorithms, which is the most used methodology employed with this purpose, is improved by simplifying the procedure that takes into consideration the monotonicity of the fuzzy measures in the genetic search (see Sec. 4.5.1). The simplification facilitates the employment of the methodology in industrial applications. It is worth mentioning that the application of Genetic Algorithms is based on the mathematical characterization of the expected result, e.g. the expected value of the integral in a classification procedure or the image resulting from the application of this operator. The difference between this expected value and the actual result of the fuzzy integral is used as fitness function. This methodology is then not enough for the practical application of the fuzzy integral in computer vision systems because the mathematical characterization of global image features does not result trivial. Two new methodologies based on Interactive Genetic Algorithms and Neurocomputing are developed. The first one allows the evaluation of the output image by a human operator simultaneously improving the search procedure through the application of genetic operators (see Sec. 4.5.2). The second approach is implemented through a SOM. It is characterized through the unsupervised clustering of the input data, which contrast with the already existent methodology based on this neural model (see Sec. 4.5.3). Both methodologies allow constructing fuzzy measures without making any mathematical computation on the result of the image fusion operation. This fact supposes a clear progress in the application of the fuzzy integral in computer vision systems.

7.3 Soft data fusion for image processing: results evaluation

143

7.3 Soft data fusion for image processing: results evaluation Different methodological approaches for image processing are developed and implemented (see Chap. 5) up to the theoretical framework of soft data fusion. It can be generally stated that the fuzzy integral can be successfully applied for the resolution of different tasks, where different image sources are taken into consideration. The results obtained in each application are commented in the following sections.

7.3.1 Color edge detection The results obtained for the edge detection on color images (see Sec. 5.1) present no loss of edges, what is considered as an important feature in such applications. The application of the ILF paradigm (see Sec. 4.4) for avoiding the detection of shadow false edges is promising. Since the quality of the result in an edge detection stage can not be evaluated on its own but relative to the requests of the posterior processing stages, the improvement of the flexibility on the operator developed within ILF can be very helpful. Nevertheless, the detection of color edges should be further automated for the employment of the presented approach in industrial applications. The results obtained on satellite images do not demonstrate the color edge detection to be fully satisfactory because of the high resolution of this type of images. In this context, improved edge detection operators that avoid the detection of spurious edges have to be implemented.

7.3.2 Highlights filtering The application of ILF operators (ILFOs) for filtering the highlights produced by materials with high reflective surfaces presents a successful performance in terms of image enhancement (see Sec. 5.2). Two different methodologies are presented for the automated assessment of the fuzzy measure coefficients in this application. The application of Interactive Genetic Algorithms do not succeed in delivering a result without false edges among the different areas defined on the label image. Nevertheless, it is worth mentioning the advancements attained in the utilization of this methodology. In this sense, the consideration of some helping measures for the guidance of the user in charge of the evaluation improves the interactive search. The development of genetic operators adapted to the construction of fuzzy measures can further contribute to the feasible implementation of this methodology in industrial applications. The shortcomings of the methodology based on the interactive determination are solved by the application of customary Genetic Algorithms. Thus, a full

144

7 Conclusions

automated procedure, which manages to filter the highlights without generating any artifacts, is achieved by taking into consideration the Choquet Fuzzy Integral with respect to diverse fuzzy measures localized in a fuzzy label image. The only parameters of the framework remain the expected limit of the highlight area and its tolerance. In this context, it is worth mentioning that the employment of the peak dynamic of the histograms allows the successful automation of the presented approach. Since the algorithm results exclusively depend on the employed imaging and lighting devices, it can be posited that its generalization capability has practically been demonstrated. The results can be even further improved by modifying the mechanical device where the images are acquired in order to generate an input multi-sensory set with no redundant reflections and to increase the degrees of freedom of the lighting system. This modification can lead to the successful detection of structural faults based on the spatial analysis of particular reflections, which is undertaken in the second approach based on the developed methodology.

7.3.3 Color morphology The definition of so-called targeted morphological operations constitutes the main feature of the color morphology presented in Sec. 5.3, which is built on the property of fusion operators to measure similarities with respect to the Ideal and Anti-Ideal points of the color hyper-cube. The developed morphological operators are used in order to selectively modify a particular color defined by the user. This allows the definition of color centered morphological dilation, erosion, and the derived complex operators of any mathematical morphology. The intuitive idea behind this feature can be described as follows. In the same manner as morphological operators for grayvalue images are selected by the user taking into account the goal to be attained, i.e. the suppression of annoying small dark regions request the application of a dilation operation, the user of a targeted color morphology can selectively operate on some color areas of a particular image. In this context, the improved selectivity of the fuzzy integral with respect to other fusion operators constitutes an important advantage. The implemented operators are successfully applied on different textile images. Their employment facilitates the detection of different faults in an industrial application for the automated visual inspection of textiles.

7.3.4 Image segmentation for document analysis Two different approaches for image segmentation, which are applied on document color images, are implemented (see Sec. 5.4). Only the second approach, which results from the application of a difference operator after the integration with two different fuzzy measures, constitutes a novelty.

7.4 Soft data fusion for image analysis: results evaluation

145

The methodology succeeds in segmenting particular color clusters in an industrial application for the detection of falsified customs seals. In this context it is successfully employed with a data set taken from the industrial application, where it presents an error rate around the 0.065% of bad segmented pixels. This extraordinary performance is based on the robustness of the fuzzy integral achieved by taking into consideration the canonical regions of the hyper-cube in the weighting scheme. This property allows the developed methodology to robustly segment the ink seals in spite of the changes in its saturation.

7.4 Soft data fusion for image analysis: results evaluation The usage of the fuzzy integral for image analysis is focused on the application of this fusion operator as classifier. Two frameworks for the segmentation of multi-dimensional images and for perceptual relevance evaluation are herein developed by taking into consideration this functionality of the fuzzy integral. The first approach is evaluated against some benchmark images and some images extracted from an automated cashier system. The second one is implemented in an industrial system for the inspection of collagen plates. The obtained results are commented in the following sections.

7.4.1 Multi-dimensional image segmentation A framework for multi-dimensional image segmentation, which is based on the consideration of the components of color spaces as color features, is implemented (see Sec. 6.1.1). The framework is first applied on different benchmark images (see Sec. 6.1.2). These images are used both with the unsupervised and supervised variants of the framework, where the unsupervised one copes successfully with the increment of the dimensionality in the input feature space. Although the transition form a 5- to a 9-dimensional color feature space results in different output images, the obtained changes seem to lay on the selected color spaces, and not in the increment of dimensionality. In any case, successful results in terms of a weak segmentation, which is defined on the uniformity of the segments and not on the object-centered segmentation, are obtained. The employment of the newly presented mix fuzzy-λ measures improves the obtained results in the larger feature spaces. The fuzzification of the input images prior to the fusion results in a non-significant improvement. The supervised version of the framework is computationally more expensive than the unsupervised one. The obtained results present no advantages in order for it to be considered in applications. Its usage is therefore only justified if similar color areas have to be segmented. Nevertheless it is worth pointing out that the supervised approach succeeds in the segmentation of small color areas.

146

7 Conclusions

Taking the aforementioned facts into consideration, the unsupervised framework for multi-dimensional image segmentation is used in the automated cashier application for market basket recognition (see Sec. 6.1.3). The obtained weak segmentation of the inspected objects allows its differentiation. Therefore, the here presented framework can be successfully used in an automated cashier system. The normalization of the coefficients obtained from the morphological clustering of the U-matrix improves the performance of the system.

7.4.2 Perceptual relevance evaluation The paradigm of perceptual relevance evaluation for automated visual inspection systems is introduced in Sec. 6.2. Due to economic factors, modern inspection systems attain the prediction of the end user subjective evaluation of produced objects. This fact leads to the conceptual development of the new paradigm for automated visual inspection denoted as perceptual relevance evaluation, which is based on the employment of fuzzy classifier methodologies. A framework for the perceptual relevance evaluation of collagen plates is realized by applying the fuzzy integral (see Sec. 6.2.1). In the developed framework, the fuzzy membership functions needed for the successful performance of the system are heuristically defined. The procedure for their assessment is not trivial and should be automated in the further development of the framework in order to make it more comfortable from the developer’s perspective. On the contrary the construction of fuzzy measures is fully automated. GAs are used for this purpose. The simplification of the procedure for taking into account the monotonicity in the construction of general fuzzy measures succeeds in the here evaluated industrial application. The employment of general fuzzy measures improves the results obtained with fuzzy-λ measures. This relays on the greater flexibility of the general fuzzy measures. The implemented framework is first evaluated with a data set, where the great majority of the collagen plates presents some type of fault. The obtained results reach a 16% of errors on the training set and a 30% on the test set. The result shows the generalization capability of the implemented approach. Although the attained error rates can be considered to be too large from an absolute point of view, the performance in this sense is not significant. There are no data sets with such a percentage of faulty plates in a real inspection scenario. In this context, it is worth mentioning the framework results achieved on 11 different data sets of 100 items each, which were directly taken from the production line. The system performance, which reaches the 3% of false rejections and the 11% of false acceptances, demonstrated to be good enough for the quality standards fixed up by the contracting company.

147

8 Summary and projections The here presented dissertation constitutes an advancement in the employment of fuzzy fusion operators, especially of the fuzzy integral, in Computer Vision, both for Image Processing and Image Analysis. The achieved development fulfills the research goals set at the beginning of the work (see Chap. 3.2), as described in the following paragraphs. The model of intelligent multi-sensory fusion is established herein. In contrast with the existent model, i.e. multi-sensory fusion, this one takes into consideration the functionality of the fusion operation itself but also the modeling of the sensors embedded in a processing system and their selection. The model is based on the employment of fuzzy fusion operators. Therefore, the sensor model can be established upon fuzzy membership functions, which generalize the commonly used gaussian functions. The sensor selection is achieved by the automation of the sensor weighting functions. The sensor selection can be adapted to the actual performance of the sensors by regularly running the procedures that assess their weighting. The large number of fuzzy fusion operators related to Fuzzy Computing find its theoretical framework in the here presented soft data fusion. In this framework the guidelines for the selection of a particular operator are given. The selection is driven by the number of freedom degrees that characterized the fuzzy fusion operators. Therefore, the value delivered by the information sources, the a priori importance of the sensory units, and their ranking relationship are taken into consideration in the computation of the fusion result. Thus, the fusion operation improves its flexibility and robustness. All these features are analyzed from an engineering point of view, particularly tailored to the field of Computer Vision. Especially an analysis of the Choquet and the Sugeno fuzzy integrals from this perspective is undertaken. In this context a data structure for the implementation of fuzzy measures, an algorithmical description of the fuzzy integral operators, and some intuitive explanations on them are given. The mathematical properties of the fuzzy integral make it a good candidate for the implementation of flexible and robust methodologies of information fusion for Computer Vision. The further development of the operator, which is especially achieved through the establishment of localized fuzzy measures, open new perspectives in the field of data fusion for Image Processing. The employment of localized fuzzy measures lead to the development of novel soft data fusion operators denoted as Intelligent Localized Fusion Operators. These are formally described in the here presented dissertation for the first time. The new operators

148

8 Summary and projections

are tailored to image processing applications by redefining the operation scope of the fuzzy measures from image channels to image sub-domains. This feature improves the flexibility of the Intelligent Localized Fusion Operators with respect to existent fuzzy fusion operators. By increasing the number of fuzzy measures included in a fuzzy integral operation, the methodologies for constructing this weighting elements gain on relevance. Thus three different methodologies for the realization of this assessment are presented. The construction of fuzzy measures is successfully developed as it can be appreciated in the following overview. Genetic Algorithms, which are hitherto the most used methodology in this context, request the mathematical characterization of the search goal through the so-called fitness function. Because of the difficulty of numerically expressing qualitative features, e.g. image features, the application of this methodology results especially challenging in computer vision applications. It is thus necessary to count on methodologies that do not depend on a numerical expression or, at least, that are not supervised. The procedures based on Interactive Genetic Algorithms and Self-Organizing Feature Maps are relevant for the successful implementation of computer vision systems. The methodologies based on the SOM neural paradigm -which is unsupervised-, and on the application of Interactive Genetic Algorithms -where the evaluation of the resulting image can be directly undertaken by a user- are introduced in the here presented research work. Furthermore, the employment of Genetic Algorithms is lightly modified in order to simplify the construction of general fuzzy measures, where the monotonicity among the coefficients constrains the search space. In summary, the application of Genetic Algorithms and of the SOM is successfully attained, whereby the the parameterization of the fuzzy integral can be respectively undertaken in a supervised and unsupervised fashion. The application of Interactive Genetic Algorithms should be further developed. It can finally be posited that the development of new methodologies is still an open research question, whose realization will allow the extended utilization of the fuzzy integral. Soft data fusion operators are embedded in different processing frameworks, which overcome the mere utilization of the fuzzy integral out of the box. The different presented methodologies succeed fulfilling the processing goals of the different engineering systems taken into account. The methodologies for highlights filtering, color morphology and color cluster segmentation are mature enough to be further applied in industrial applications. The methodology for color edge detection will be improved in order to attain the same degree of feasibility. Moreover the analysis of new types of fuzzy integral can expand the possibilities of these methodologies and make possible the attainment of even more complex goals. The two different methodologies devoted to Image Analysis succeed in fulfilling the system specifications. The application of the presented methodology for

149 multi-dimensional image segmentation can be applied in future research works for texture segmentation. The here described new paradigm of perceptual relevance evaluation can profit from the consideration of new types of fuzzy integrals in order to consolidate as a paradigm for automated visual inspection in industrial applications.

150

151

Appendix A Fuzzy Measures Theory The Theory of Fuzzy Measures is based on the work of Sugeno [163]. The introduction of fuzzy sets [192], which result from a generalization of classical sets, encouraged the redefinition of set measures. Sugeno achieved this definition by introducing so-called fuzzy measures, with respect to which fuzzy integrals can be defined. Thus fuzzy measures generalize classical measures, i.e. probability measures. Although the Fuzzy Set Theory and the Fuzzy Measure Theory are considered to be different facets of Fuzzy Computing (see Sec. 2.4), they constitute two different theoretical frameworks. The following sections present the theoretical background on Fuzzy Measure Theory from different perspectives.

A.1 Fuzzy Logic vs. Fuzzy Measures Fuzzy measures are measures on fuzzy sets. The theoretical frameworks of both fuzzy sets and fuzzy measures pursue the treatment of uncertainty in computational systems. Nevertheless, the kind of uncertainty these theoretical frameworks deal with are different. On the one hand fuzzy sets are a generalization of classical sets derived from the consideration of the inaccuracy in natural language. The fundamental point of the theory is the consideration of linguistic labels as the key elements of human thinking [193]. The necessity of including common sense knowledge, which is generally expressed in linguistic terms [62], and the desire of reaching some of the positive features of human reasoning for problem solving in computational systems motivated the development of this theoretical framework. Thus, sets are not defined through precise labels but with linguistic ones, which reflect the inaccuracy of natural language. Moreover, the membership of fuzzy set elements is not defined as true or false, but becomes a matter of degree represented by a rational number defined in the interval [0, 1]. In this way a variable, upon which a fuzzy set is defined, presents a membership degree h(xi ) depending on its current value, as depicted in Fig. A.1. This definition makes possible the application of quantitative techniques on

152

Appendix A Fuzzy Measures Theory h(x) 1.0

SMALL hSM x0

0 77

hBIG x0

0 46

BIG

0.0

x0

x

Figure A.1: Example of fuzzification of a variable x. The variable x is fuzzified with the linguistic terms “SMALL” and “BIG”. The fuzzy membership functions hSM () and hBIG () are applied on the variable x in order to obtain the membership degrees hSM (x0 ) and hBIG (x0 ) of its current value x0 .

complex systems, whose feasible treatment through classical numerical techniques is denied by the principle of incompatibility [193]1 . Thus Fuzzy Logic, the logical inference system built upon fuzzy sets, presents tolerance in front of imprecision, ambiguity, and partial truths or occurrence of events. Moreover the consideration of such a multi-value logical system makes possible the treatment of apparently contradictory statements as classical paradoxes [90]. Fuzzy logic, among other multi-value logical systems, overcome the constraints imposed by the law of excluded middle [90]. All these facts show the straight relationship between Fuzzy Sets Theory and reasoning processes. On the other hand Fuzzy Measure Theory tackles the combination of evidence in expert systems [62]. This theoretical framework allows the treatment of the uncertainty derived from the partiality of the information sources, i.e.: when taking into consideration different criteria in the validation of a hypothesis, each of the information sources, which deliver some evidence on these criteria, support the hypothesis to a certain degree; moreover, no information source holds enough evidence to support the hypothesis on its own. Fuzzy measures allow to quantitatively express the importance of the information sources taken into consideration, while the degree of the hypothesis fulfillment is delivered by the result of the fuzzy integral. The importance quantification succeeds with respect to the information sources individually taken, but also with respect to their possible coalitions. 1

The principle of incompatibility establishes that in a computing system the achieved accuracy is proportional to the cost of the system and inversely proportional to its robustness.

A.2 Axiomatic Definition of Fuzzy Measures

153

Intuitively the quantification of the a priori importance through the fuzzy measures in the fuzzy integral and the general functionality of this operator can be better understood through the following example. Someone tries to solve a complex question and for that purpose calls different experts on the field in order to obtain a solution. A hypothesis to be evaluated by the experts is set up. The person waiting for an answer will weigh the opinion of the different experts based on their a priori importance. He will take into consideration different subjective factors, e.g. belief, intuition, confidence, past experiences, in order to quantify this a priori importance. This quantification will be applied not only to the opinion of the individual experts, but also to the fact that an opinion could simultaneously be held by more than one of the experts. Thus the different elements of this operation are the opinion of the experts -which state for the integrands in the fuzzy integral-, the a priori importance established -which is represented by the fuzzy measures coefficients-, and finally the degree of support of the hypothesis solving the question -which is quantified by the result of the fuzzy integral. A fuzzy measure µ is composed by its coefficients µ(), which are defined on the different subsets Aj of the n information sources xi . One coefficient is established for each information source individually considered, µi = µ({xi })

∀i = 1, . . . , n,

(A.1)

but also on their coalition, which is represented here as the union of the corresponding subsets, µij = µ({xi , xj }) ∀i, j = 1, . . . , n i 6= j .. (A.2) . µ1...n = µ(X) = µ({x1 , . . . , xn }). In the given example µ({x2 }) would characterize the a priori importance of the second expert’s opinion, while µ({x2 , x3 }), the importance of the second and third experts agreeing. The point of connection between the fuzzy logic and the fuzzy measure theoretical frameworks is the definition of fuzzy measures as functions on fuzzy sets, which implies that the integrand data generally suffers a fuzzification prior to the integration. This operation will be represented here through h() instead of the normally used µ() in order to avoid any confusion with the nomenclature customary used on fuzzy measures (see Fig. A.1).

A.2 Axiomatic Definition of Fuzzy Measures The Fuzzy Measures Theory [179] was built upon Sugeno’s conclusions as a generalization of the classical measure theory following the path open by Zadeh when

154

Appendix A Fuzzy Measures Theory

introducing fuzzy sets [192], namely the convenience of considering subjectiveness in mathematical analysis tools. This generalization of the classical measure theory is done through the relaxation of the additivity axiom of classical measures, i.e. probability measures. From a mathematical point of view fuzzy measures are defined in so-called fuzzy measure spaces, a tuple represented by (X, C, µ), where X represents a fuzzy set, and C a class with X ∈ C [179]. Both together form the measurable space (X, C). Thus fuzzy measures can be defined as functions on fuzzy sets, µ : P(X) → [0, ∞], where P(X) is the power set of information sources, satisfying the following axioms in the discrete case: I. Limits µ{∅} = 0,

(A.3)

Aj ⊂ Ak → µ(Aj ) ≤ µ(Ak ) ∀Aj , Ak ∈ X.

(A.4)

II. Monotonicity

The most usual fuzzy measures fulfill an additional condition concerning the limits of the measure, namely µ(X) = 1.0. In this case the measure is called a regular fuzzy measure [179]. Thus regular fuzzy measures are defined in a reduced domain, namely µ : P(X) → [0, 1]. Regular fuzzy measures present so-called dual fuzzy measures [179],which are defined as ¯ ν(A) = 1 − µ(A). (A.5) The additivity axiom of probability measures P (), which established for mutually exclusive sets [121] that P (A ∪ B) = P (A) + P (B),

(A.6)

is relaxed in fuzzy measures by the monotonicity condition (A.4). This relaxation can be understood as a means of making more flexible the measuring process with respect to the characterization of the interaction among the criteria being considered in the fusion operation. This interaction is defined by the complementarity, redundancy or independence among the individual information sources. In the Fuzzy Measure Theory the additivity property intuitively characterizes this interaction [62]. Thus sub-additive measures, which satisfy µ(A ∪ B) < µ(A) + µ(B),

(A.7)

characterize redundancy among the information sources, while super-additivity, expressed by µ(A ∪ B) > µ(A) + µ(B), (A.8)

A.3 Different Measures for the Treatment of Uncertainty

155

characterize complementarity among them. Thus additive measures, i.e. probability measures (A.6), are only capable of characterizing independence among information sources, at least from the perspective of Fuzzy Measure Theory. It is worth mentioning that this concept is not deeply discussed in the literature on Fuzzy Measure Theory. Moreover, it is qualified as an intuition demanding further research [62]. Neither the relationship between the independence in the interaction among information sources, nor the concepts of mutual exclusivity and statistical independence [121], which are derived from the probabilistic theoretical framework, are analyzed [13] [62]. Such a discussion overextends the scope of the present work. Thus, it can just be summarized that the relaxation of the additivity axiom allows fuzzy measures to describe different types of interaction among information sources.

A.3 Different Measures for the Treatment of Uncertainty From the more flexible definition of fuzzy measures, which is based on (A.3) and (A.4), a huge amount of functions on fuzzy sets can be defined. Different types of measures are used in different theoretical frameworks for information fusion in order to treat the uncertainty related to the partiality of the information sources. As formerly mentioned, probability measures are a particular case of regular fuzzy measures fulfilling additivity as expressed by (A.6). These measures are used, for instance, in Bayesian Decision Theory [38], where probability is used in order to quantify the trade-off between different possible decisions [38]. Another particular case of regular fuzzy measures is the so-called belief measures and its dual plausibility measures [179]. This type of measures is extensively used in the Dempster-Schafer Theory of Evidence [143]. Belief measures are denoted as Bel() and satisfy [179]: Bel

n [

{xi }

i=1

!

≥

X

! \ (−1)|I|+1Bel {xi } .

I⊂{1,...,n},I6=∅

(A.9)

i∈I

This condition makes them be denoted as super-additive measures (see [179] p.58 for proof). Plausibility measures, P l(), are defined as the dual measures (see (A.5)) of belief measures. Possibility and their dual necessity measures were introduced in the context of fuzzy sets in [194]. The Possibility Theory [36] is built on this type of fuzzy measures. Possibility measures, denoted by π(), are sub-additive regular fuzzy measures since π(Ai ∪ Aj ) = π(Ai ) ∨ π(Aj )

∀i, j ∈ I.

(A.10)

156

Appendix A Fuzzy Measures Theory

Fuzzy-λ measures, which were introduced by Sugeno [163], makes the additivity property depend on a parameter λ > −1 by applying: µ(Ai ∪ Aj ) = µ(Ai ) + µ(Aj ) + λµ(Ai )µ(Aj ) ∀i, j ∈ I.

(A.11)

Thus they can act as additive (λ = 0), sub-additive (−1 < λ < 0) or superadditive (λ > 0) measures. The parameter λ is found after assigning the coefficients of the individual information sources by solving the equation [166]: " n # 1 Y µ(X) = [1 + λ · µ({xi })] − 1 . (A.12) λ i=1 Figure A.2 shows a pictorial overview of all these types of fuzzy measures with respect to the additivity property. All these fuzzy measures offer a vast landscape of possibilities in the treatment of the uncertainty present in the combination of evidence, which constitutes the application scope of the fuzzy integral. Each of the theoretical frameworks formerly mentioned deal with the uncertainty in different ways and thus its application lead to different results. One of the most interesting points of the Theory of Fuzzy Measures and Fuzzy Integrals is the consideration of the type of interaction between the evidence sources, although, as already mentioned, such a consideration demands further theoretical research efforts. fuzzy measures

subadditive measures plausability measures probability measures

possibility measures

fuzzy- λ

λ

0

λ

0

1

λ

0

necessity measures belief measures superadditive measures

Figure A.2: Relationship among different types of fuzzy measures. Modified from a figure in [58] (with permission).

157

Appendix B Properties of the Choquet and Sugeno Fuzzy Integrals as Fusion Operators The transformation of the data delivered by multiple information sources into one representational form [1] constitutes the basic goal of fusion operator (see Sec. 2.3.1). Some of the mathematical properties that improve the performance of the fuzzy integral in front of other fusion operators are described in the this section.

B.1 Generalization of other fusion operators The analysis of the generalization property of the fuzzy integrals does not pretend to be herein exhaustive, but related to the operators employed in the presented research works. The generalization of the fuzzy integral with respect to other fuzzy fusion operators succeeds through the values of the fuzzy measure coefficients. Due to the mathematical expressions of the fuzzy integrals, the computation of their result with respect to a particular fuzzy measure is equivalent to the application of other fuzzy aggregation operators (see Fig. 2.13). Table B.1 gives the particular fuzzy measures for some operators. It is even known that the Choquet and the Sugeno fuzzy integrals succeed approximating, though not with arbitrary precision, any connective H from the unit hyper-cube [0, 1]n to [0, 1], if H is increasing and fulfills both H(0, . . . , 0) = 0 and H(1, . . . , 1) = 1 [62].

B.2 Canonical regions of the hyper-cube The sorting operation previous to the integration itself (see Sec. 2.5.2) divides the hyper-cube into n! different regions, one for each permutation of the information sources. These regions receive the name of canonical regions of the hypercube

158

Appendix B Properties of the Choquet and Sugeno Fuzzy Integrals

Table B.1: Generalization relationships between the fuzzy integral with respect to a particular fuzzy measure and other fuzzy aggregation operators (Op.). min: minimum. max : maximum. wmin: weighted minimum [62]. med : median. wsum: weighted sum. OWA: Ordered Weighted Averaging [189]. Sµ : Sugeno Fuzzy Integral. Cµ : Choquet Fuzzy Integral. Nπ states for a necessity measure induced (→) by a possibility distribution π, induced itself by the set of weights W [62], p.148. Op.

Integral

min max wmin med wsum OWA

Sµ , Cµ Sµ , Cµ Sµ Sµ Cµ Cµ

Fuzzy Measure µ(Aj ) = 0, ∀j |Aj | < |X | µ(Aj ) = 1, ∀j |Aj | = 1 µ = Nπ , W = {w1 , . . . , wn } → π → Nπ V V µ(Aj ) < µ(Ak ) µ(Ak ) = 1, ∀j, k |Aj | < |Ak | |Ak | = n/2 µ(Aj ∪ Ak ) = µ(Aj ) + µ(Ak ) = wj + wk , ∀j, k |Aj | = |Ak | = 1 µ(Aj ) = µ(Ak ) = wi , ∀j, k |Aj | = |Ak |

[62]. Figure B.1 depicts the canonical regions in both a two-dimensional, and a three-dimensional feature space. Furthermore, the variables satisfy a different ranking order in each of the canonical regions. In case of the Choquet integral its mathematical function is linear in each of the canonical regions.

B.3 Level curves The level curve of a fuzzy integral depicts its behavior in the hypercube for a particular fuzzy measure. The level curve of a fusion operator can be defined as the representation in a three-dimensional space of the result values (z) computed by applying the fusion operator on two variables (x and y). This concept is taken from Decision Making, where the level curves are used in order to place the alternatives left undecided by different operators and thence establishing equivalence classes among them [62]. Furthermore the level curves can be used to gain a qualitative description of the operator performance. The Sugeno Fuzzy Integral (see Fig. B.2) presents so-called constant regions in its level curve [62], whose hyper-volumes depend on the particular values of the fuzzy measure coefficients. Here the output value of the fuzzy integral is constant and equal to the value of the corresponding fuzzy measure coefficient. In the Choquet Fuzzy Integral the fuzzy measure coefficients modify the gradient of the information source being affected by that coefficient (see Fig. B.3). The larger the coefficient, the larger the gradient for that variable. Thus the variations of the features with larger fuzzy measure coefficients will be reflected

159

B.3 Level curves xi

x2

x1

x1

xi x1

x1

x1

x1

x3

x3

x2

x2

x3

x2

x1

xi

x2

x2

x3

x1

(a)

x3

x2

(b)

(c)

x2

Figure B.1: (a) Canonical regions of a two-dimensional hypercube. The canonical regions are divided by the hyper-plane where all the feature variables are equal. (b) The larger feature variable divide the threedimensional feature space in three regions, wherein two canonical regions for each one can be defined. (c) The two canonical regions for one of these regions are depicted. The other ones present an analogous distribution.

Sµ

0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1

1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0

1

0

0.1 0.2 0.3 0.4 0.5 0.6 0.7 x1 0.8

0.9

1

0

0.2 0.1

0.6 0.5 0.4 0.3

0.9 0.8 0.7 x2

Figure B.2: Level curve of the Sugeno Fuzzy Integral on two variables. Exemplary response for µ1 = 0.2, µ2 = 0.6, thus expressed by Sµ (x1 , x2 ) = [(x1 ∧ 0.2) ∨ x2 ] for x1 ≥ x2 and Sµ (x1 , x2 ) = [x1 ∨ (x2 ∧ 0.6)] for x1 < x2 . The constants regions of the hypercube, whose volume depends on the values of the fuzzy measure coefficients, can be observed in the projection on the feature plane x1 x2 .

stronger in the fuzzy integral result than these variations of the features with lower ones.

160


Cµ

0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1

1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0

0

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 x1 0.9

1

0

0.2 0.1

0.6 0.5 0.4 0.3

1 0.9 0.8 0.7 x2

Figure B.3: Level curve of the Choquet Fuzzy Integral of two variables. Exemplary response for µ1 = 0.2, µ2 = 0.6 thus expressed by Cµ (x1 , x2 ) = [0.2·x1 +x2 ·(1−0.2)] for x1 ≥ x2 and Cµ (x1 , x2 ) = [x1 ·(1−0.6)+0.6·x2] for x1 < x2 . The larger gradient of x2 can be observed on the level lines at the plane x1 x2 .

B.4 Distributivity with respect to the scalar product The Choquet fuzzy integral is distributive with respect to the scalar product, thus satisfying for a scalar α ≥ 0: α·Cµ [h1 (x1 ), . . . , hn (xn )] = Cµ [α·h1 (x1 ), . . . , α·hn (xn )] = Cα·µ [h1 (x1 ), . . . , hn (xn )]. (B.1)

B.5 Monotonicity with respect to the integrands Being h1i and h2i two sets of functions on X, µa fuzzy measure defined on (X, P(X)), and Fµ the Choquet or the Sugeno fuzzy integral with respect to this measure, then ∀x ∈ X: h1i ≤ h2i

∀i ∈ I

Fµ [h1i (xi )] ≤ Fµ [h2i (xi )].

→

(B.2)

B.6 Monotonicity with respect to the fuzzy measures Being µ1 and µ2 two fuzzy measures on (X, P(X)), then: µ1 (A) ≤ µ2 (A)

∀A ∈ P(X)

→

Fµ1 [hi (xi )] ≤ Fµ2 [hi (xi )].

(B.3)

161

B.7 Range of results

B.7 Range of results The result of the Choquet and the Sugeno fuzzy integrals range from the minimum to the maximum of the integrands, ^

hi (xi ) ≤ Fµ [h1 (x1 ), ..., hn (xn )] ≤

i

_

hi (xi ).

(B.4)

i

B.8 Fusion operators as measures of similarity A common theoretical framework for fuzzy fusion operators has been presented in [10], which states that aggregation operators can be used to measure distances in metric spaces. This fact is based on the existence of an intrinsic relationship between the aggregation operator and the distance function being used. The theoretical framework is built upon the utilization of aggregation operators in Fuzzy Systems Theory to establish the membership function of the output variable. Here the result can be seen to measure the similarity to prototype elements of the output set, which receive the name of Ideal (I) and Anti-Ideal (∅) [10]. A pictorial representation of these sets can be found in Fig.B.4. h2 Ideal hI(x) x(h1,h2)

hAI(x) Anti-Ideal

h1

Figure B.4: Representation of a fuzzy set X in a two-dimensional space and the corresponding membership functions of the sets Ideal and Anti- Ideal.

In this context the membership functions of the Ideal (hI ) and the Anti- Ideal (hAI ) are considered to be [10]: hI (X) = 1 − d(I, X) hAI (X) = d(∅, X),

(B.5) (B.6)

where d are distance functions. Since these membership degrees can be calculated through aggregation operators, with the only condition of being monotonic, and

162


having a result value between 0 and 1, following relationships can be set: d(I, X) = 1 − F1 (X) d(∅, X) = F2 (X),

(B.7) (B.8)

where F1 and F2 are arbitrary aggregation operators for fuzzy AND and fuzzy OR respectively. For theorem, demonstration and deeper explanations the mentioned work is referred [10]. Being the fuzzy integrals with respect to regular fuzzy measures monotonic operators limited in the range [0, 1], it seems possible to use the fuzzy integral for the computation of similarity relations to the Ideal and Anti-Ideal points of the hypercube.

163

Appendix C Examplary classification of data with gaussian distributions In the following the capability of the fuzzy integral for classifying two-dimensional Gaussian distributions is evaluated. The example is thought to familiarize the reader with the general methodology and to derive some general considerations from the results. It is worth first analytically expressing the undertaken procedure. When trying to classify two different clusters of data,a fuzzy integral with respect to two different measures will be undertaken µj j = 1, 2. Being µj (X) = 1 ∀j the expression of the Choquet fuzzy integral (2.10) reduces in a two-dimensional space to:

Cµj (x1 , x2 ) =

2 X

[h(i) (x(i) ) − h(i+1) (x(i+1) )] · µj (A(i) ) =

i=1

= [h(1) (x(1) ) − h(2) (x(2) )] · µj (A(1) ) + h(2) (x(2) ),

(C.1)

since µj (X) = µj (A(2) ) = 1. The sorting operation makes the integral behave different in each canonical region, which are determined as formerly mentioned by the sorting on the integrands: Cµj (x1 , x2 ) =

(

h1 (x1 ) − h2 (x2 ) · µj ({x1 }) + h2 (x2 ) : h1 ≥ h2 h2 (x2 ) − h1 (x1 ) · µj ({x2 }) + h1 (x1 ) : h1 < h2

(C.2)

Thus the operation at hand assures a minimal value for the integral, this of the smaller of the integrands (see Sec. B.7), and adds to this term a quantity proportional to the difference between the integrands. Here the multiplying factor is the value of the a priori importance of the integrand with a larger value, i.e. fuzzy measure coefficient for the larger integrand. On the other hand the Sugeno fuzzy integral (4.4) reduces in a two-dimensional

164

Appendix C Examplary classification of data with gaussian distributions

feature space to: Sµj (x1 , x2 ) =

2 _

[h(i) (xi ) ∧ µj (A(i) )] =

i=1

= [h(1) (x(1) ) ∧ µj (A(1) )] ∨ h(2) (x(2) ).

(C.3)

Also here the sorting operation establishes two different operations for the two canonical regions: ( h1 (x1 ) ∧ µj ({x1 }) ∨ h2 (x2 ) : h1 ≥ h2 (C.4) Sµj (x1 , x2 ) = h2 (x2 ) ∧ µj ({x2 }) ∨ h1 (x1 ) : h1 < h2

In this case the result of the integral is the median of the three elements present in the integration, namely the two integrands and the a priori importance of the information source presenting a larger value. In the following figures, different results of the fuzzy integral in the classification of two clusters of data with a normal distribution are depicted. In the different figures the cluster distribution is shown first, followed by the result of the Sugeno Fuzzy Integral, on the left, and this of the Choquet one, on the right. The result was obtained for diverse types of fuzzy measures (see Sec. A.3), but just the best one is depicted. The fuzzy measure coefficients for the individual information sources are taken as the mean values of the distributions. Clusters with mean (0.5,0.1) and (0.1,0.5), Fig. C.1. Sugeno presents better results due to the presence of the constant regions of the hypercube (see Sec. B.3). Only Choquet present differences between the different types of fuzzy measures. The presented result was obtained with probability fuzzy measures. Clusters with mean (0.1,0.3) and (0.9,0.1), Fig. C.2. Sugeno again presents better results due to the constant regions. No differences for different types of fuzzy measures. On the contrary Choquet presents greater differences between fuzzy measure types. The represented result was obtained with fuzzy-λ measures. It can be observed that the cluster with lower mean values is difficult to classify. Clusters mean (0.5,0.8) and (0.9,0.1), Fig. C.3. Both types of integral present good classification performance. Clusters mean (0.9,0.9) and (0.9,0.1), Fig. C.4. Both results are bad, but Choquet presents a slightly better one with probability fuzzy measures. For clusters whose mean values are not in different canonical regions, the clustering does not succeed employing the current strategy. No systematic approach for the assessment of the fuzzy measure coefficients could be recognized (Figs. C.5 and C.6). Good classification results are heuristically obtained.

165

1

feature x2

0.8

0.6

0.4

0.2

0 0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

feature x1

(a) 1

1

0.8



0.8

0.6

0.4

0.2

0.6

0.4

0.2

0

0 0

20

40

60

80

100

120

140

160

180

200

0

20

40

60

80

100

120

140

data sample index

data sample index

(b)

(c)

160

180

200

Figure C.1: Classification of 200 data samples of two classes : xi ∈ C 1 ∀i ∈ [0, 99] and xi ∈ C 2 ∀i ∈ [100, 199]. The data samples are generated through two gaussian distributions with mean values: (0.5, 0.1) and (0.1, 0.5). Values of fuzzy densities: µ11 = 0.5, µ12 = 0.1, and µ21 = 0.1, µ22 = 0.5.

166


1

feature x2

0.8

0.6

0.4

0.2

0 0

0.2

0.4

0.6

0.8

1

feature x1

(a) 1

1

0.8



0.8

0.6

0.4

0.2

0.6

0.4

0.2

0

0 0

20

40

60

80

100

120

140

data sample index

(b)

160

180

200

0

20

40

60

80

100

120

140

160

180

200

data sample index

(c)


167

1

feature x2

0.8

0.6

0.4

0.2

0 0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

feature x1

(a) 1

1

0.8



0.8

0.6

0.4

0.2

0.6

0.4

0.2

0

0 0

20

40

60

80

100

120

140

160

180

200

0

20

40

60

80

100

120

140

data sample index

data sample index

(b)

(c)

160

180

200


168


1

feature x2

0.8

0.6

0.4

0.2

0 0.6

0.65

0.7

0.75

0.8

0.85

0.9

0.95

1

feature x1

(a) 1

1

0.8



0.8

0.6

0.4

0.2

0.6

0.4

0.2

0

0 0

20

40

60

80

100

120

140

160

180

200

0

20

40

60

80

100

120

140

data sample index

data sample index

(b)

(c)

160

180

200

Figure C.4: Classification of 200 data samples of two classes : xi ∈ C 1 ∀i ∈ [0, 99] and xi ∈ C 2 ∀i ∈ [100, 199]. The data samples are generated through two gaussian distributions with mean values: (0.9, 0.9) and (0.9, 0.1). Values of fuzzy densities µ11 = 0.9, µ12 = 0.9, and µ21 = 0.9, µ22 = 0.1.

169

1

feature x2

0.8

0.6

0.4

0.2

0 0.6

0.65

0.7

0.75

0.8

0.85

0.9

0.95

1

feature x1

(a) 1

1

0.8



0.8

0.6

0.4

0.2

0.6

0.4

0.2

0

0 0

20

40

60

80

100

120

140

160

180

200

0

40

60

80

100

120

140

data sample index

(b)

(c)

1


20

data sample index

160

180

200

160

180

200

1

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2

0

0 0

20

40

60

80

100

120

140

160

180

200

0

20

40

60

80

100

120

140

data sample index

data sample index

(d)

(e)

Figure C.5: Classification of 200 data samples of two classes : xi ∈ C 1 ∀i ∈ [0, 99] and xi ∈ C 2 ∀i ∈ [100, 199]. The data samples are generated through two gaussian distributions with mean values: (0.9, 0.5) and (0.9, 0.1). Values of fuzzy densities (b-c) µ11 = 0.9, µ12 = 0.5 and µ21 = 0.9, µ22 = 0.1; (d-e) µ11 = 0.9, µ12 = 0.1, and µ21 = 0.3, µ22 = 0.1.

170


1

feature x2

0.8

0.6

0.4

0.2

0 0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

feature x1

(a) 1

1

0.8



0.8

0.6

0.4

0.2

0.6

0.4

0.2

0

0 20

40

60

80

100

120

140

160

180

200

0

20

40

60

80

100

120

140

data sample index

data sample index

(b)

(c)

0.9

0.9

0.8

0.8

160

180

200

160

180

200

0.7

0.7



0

0.6

0.5

0.4

0.3

0.6

0.5

0.4

0.3

0.2

0.2

0.1

0.1

0 0

20

40

60

80

100

120

140

160

180

200

0

20

40

60

80

100

120

140

data sample index

data sample index

(d)

(e)

Figure C.6: Classification of 200 data samples of two classes : xi ∈ C 1 ∀i ∈ [0, 99] and xi ∈ C 2 ∀i ∈ [100, 199]. The data samples are generated through two gaussian distributions with mean values: (0.5, 0.5) and (0.9, 0.1). Values of fuzzy densities: (b-c) µ11 = 0.5, µ12 = 0.5 and µ21 = 0.9, µ22 = 0.1; (d-e) µ11 = 0.1, µ12 = 0.9, and µ21 = 0.9, µ22 = 0.1.

171

Appendix D Multi-sensory Fusion in Biological Systems Although the capability for using different sensor modalities and for joint exploiting the information derived from them have been denoted as one of the principle foundations of the intelligence shown by biological systems [99], the neuronal mechanisms, principles and structures supporting this capability have not been extensively used in the implementation of technological systems with information fusion. The most important handicap for the existence of such bio-inspired and cognitive based information fusion frameworks is doubtless the lack of detailed knowledge on the cerebral areas participating of this kind of processes, mainly on the so-called cortical association areas [113]. Furthermore the existing knowledge is disseminated among different information sources, what hinders researchers not directly working in neurosciences becoming a general perspective. This chapter presents a unified perspective on different aspects of cognitive and biological principles that can be used as foundation of the emergent field of bio-inspired and cognitive based information fusion in computer vision. Therefore an overview of different systems, which could be embedded in this incipient field, is first given. Thence some principles of cerebral information processing, which are used in the here presented dissertation, are grouped together. Finally the process of information fusion in the nervous system is analyzed at different levels of abstraction.

D.1 Biological and Cognitive Perspective on Information Fusion As in other technological fields the development of computer vision systems is inspired in some cases by the biological implementation of visual cognitive capabilities. [113] constitutes a seminal paper for the elaboration of biological and

172

Appendix D Multi-sensory Fusion in Biological Systems

cognitive based fusion approaches in Computer Vision. In the following different computer vision systems inspired by biological or cognitive processes of multisensory fusion strategies are described. First three different systems based on concepts taken mainly from the analysis of human cerebral structures are presented. The Sensor Fusion Effects (SFX) architecture [113] attains the implementation of navigational and perceptual capabilities in robotic agents. With this purpose in mind different biological and cognitive principles are analyzed and translated into technological concepts. While the first ones inspired “how” the multi-sensory fusion is undertaken, the cognitive ones are taken into consideration in order to answer “why” it is interesting for a system to present multi-sensory fusion capabilities [113]. The developed architecture uses different organizational concepts based on the Schema Theory [6] that allows the system to operate with different fusion strategies. These follow the operational relationship among the information sources that is formerly presented for sensory processing in the general block diagram of multi-sensory integration (see Sec. 2.3.1). The implementation of this so-called fusion modes, which are based on a taxonomy presented in [18], can be understood as a perception management system [132]. Another interesting concept is the division of the fusion operation into an investigatory and a performatory phases based on a cognitive concept from [92]. The fusion theoretical framework is based on the DempsterSchaffer Evidence Theory [143][74]. This framework presents a fusion operator that overcomes the mere averaging of the input sources and allows contextual information to influence the result of the operation. Both features characterize the neurons found in the superior colliculus of cat’s brain that realize multi-sensory fusion [162] (see Sec. D.3.2). Furthermore the Evidence Theory allows to establish a common representation prior to the fusion through the application of belief functions. This feature is thought to be characteristic of sensor fusion in humans [102]. A system for the visualization of a multi-sensory image set formed by images taken in the visible and infrared spectral bands is based on the opponent band representation present in the primary visual system [42]. The framework uses a neural network [41], whose response simulates the ON-OFF receptive fields of the bipolar and ganglion cells found in the retina, and the cells forming the layer 4C of the primary visual system [140]. The system attains the decorrelation of the input channels in order to augment the color contrast of the output three-dimensional color image, what is achieved through the application of the mentioned fusion methodology. The last processing system based on human models is used for the fusion of different images by taking into consideration the wavelet transform [186]. This system undertakes a wavelet decomposition of the hyperspectral input image set and thence fuses the images through the application of a weighted sum. Thus the fusion operation is not perceptually-based, but the selection of the weights in this operation. This assessment of the weights is made in order for the final

D.2 Information processing in the human brain

173

image to increase the contrast among the different spectral bands by taking into consideration the dependence between contrast and spatial frequency in human perception [161]. On the other hand two different frameworks for multi-sensory fusion have been based on biological structures found on different animals. The first one [73] is based on the bimodal neurons that fuse visible and thermal information in the neuronal system of rattlesnakes [117]. It presents three processing stages implemented by two different neural network paradigms. A modified SOM [85] implements both the first stage, where fuzzy membership functions are generated and applied on the input images, and the third one, which realizes the attained pattern recognition task. On the other hand a single layer perceptron [133] realizes the six different types of fusion operators, which are used in the rattlesnake’s tectum for binding the infrared and visual information. The first group of these operators presents a response analogous to some logical functions (AND, OR, and XOR). The second group answers moderately when one sensor is activated and stronger when both are active. The implemented system is applied for the fusion of visible and range images. A system for the fusion of acoustic and visual information is implemented [122]. The resulting framework models a mechanism with the same functionality that can be found in the neuronal system of barn owls. Moreover the map models the senso-motoric map of the barn owl situated in the optic tectum, where the retinal information and the information of the so-called acoustic retina are fused in order to control the movement of the head towards a possible target [84]. Thus a neuronal map, which is structurally functionally similar to a SOM [85], is developed. This includes the modification of the equations for the actualization of the map weights. In the following sections the different principles and mechanisms underlying the incipient field of bio-inspired and cognitive based multi-sensory fusion will be reviewed.

D.2 Information processing in the human brain In this section, some of the building blocks of information processing in the human brain are presented. Further than a general interest the presentation of these different themes serves as an introduction for some questions that found its counterparts in different frameworks presented in the dissertation. The further development of the fuzzy integral and its application in computer vision applications can be be inspired by the following points.

174


D.2.1 Principles of Information Processing The information processing undertaken in the human brain are thought to follow some general principles, which have been described since the beginning of the 60’s as follows: 1. Cortical cells present selective response [26], e.g. basic units of cortical processing in the primary visual system are cells with selective response to a particular visual feature. Thus these cells plays an analogous role as feature detectors in pattern recognition approaches (see Sec. 2.1). This role of neurons as feature detectors was introduced in [93], which analyzes this aspect in the visual primary system. This principle have been extrapolated by some authors as a general principle beyond the primary visual system [26]. 2. The responses of the cortical cells are hierarchically organized along the nervous system from less to more specific [26]. Thence the larger the distance of the processing unit from the sensory one, the more specific and complex the information caught by a particular cell [9]. This principle have been exploited in Computer Vision, e.g. VisNet [131], which uses a hierarchical network of self-organizing maps for object recognition. 3. The segregation of different visual information in parallel processing channels present in the visual primary system [96] can be extrapolated to other sensor modalities [98]. Nevertheless some recent findings contradict this fact and advocate for an interaction of the different sensory modalities all over the nervous system [147]. Beside these general principles some other ones rule the process of multi-sensory fusion: 1. There is a decreasing number of neurons answering to an stimulus [26]. Thus there exist a process of information fusion along the nervous system. This process of information fusion is undertaken among the different sensor modalities as well [98]. Although still not known in detail, some works posit the existence of a special mechanism for the fusion of information [162]. As it will be explained in the next section such a representation is complemented by a distributed one. 2. Such a mechanism demands the transformation into a common representation of the information prior to the fusion operation of the different sensory modalities [102].


175

D.2.2 Neuronal coding Since multi-sensory fusion is basically a transformation of different information sources into one representational form [1], the question of how the world is represented in the brain is a central question to information fusion. The representation of information is undertaken at the neuronal level by a codification schema, which receives the name of neuronal code [51]. In the literature different codification schemes can be found.

Figure D.1: Some concepts are not represented by the frequency of the spike train generated by just one neuron, but by these of several ones (vector coding). Furthermore the afferent neuron is not exactly tuned to detect a particular feature, e.g. 90◦ , but presents an smooth response (coarse coding). This type of codification, which can be called coarse vector coding resembles the output representation of fuzzy systems [170].

The principles of general information processing described in the former section led to the conclusion that the processing in the cortex culminate in a local coding of the pattern being sensed [26]. Thus, each of the different elements perceived in a scene would be represented by a unique neuron. Furthermore this representation is achieved through the frequency of the spike train generated by the stimulus. In this context local coding, which is known as the theory of the grandmother cell as well, supposes the existence of a fusion operator that would generate this representation up to the representations incoming the fusing neuron. The response of this operator seems to present reinforcement capabilities, i.e. the answer of the fusion operation is stronger in front of several weak inputs as it would be in front of a unique strong one [162]. Such a synthetic representation of information speeds up the recognition process for known objects and allows a clean sensori-motor mapping [26]. The critique to this theory of neural coding [26] is centered on the fact that the existence of such a coding mechanism would hinder

176


the generation of right answers in front of unknown patterns. Furthermore it is difficult to imagine how this coding scheme could cope with the fuzzy definition of certain concepts (red as color is not the same as the color meant by red hair) if the existence of a limited number of neurons is taken into consideration. In [26] an alternative to local coding which receives the name of vector coding is proposed. This type of coding is known as distributed representation, state space representation or multi-dimensional representation as well. Such an scheme proposes that one neuron can participate in the distributed representation of different concepts and does not represent an item by itself. In this case the representation is undertaken by the frequency of the spike trains of the different coding neurons. [20] presents a general theoretical framework of representation in the brain through the consideration of a spatial distributed representation analogous to the vector coding. Similar models have been used to explain object recognition based on visual features [131]. Furthermore, the representation of objects through different modalities in the association cortex is related to the existence of such distributed representations as well [20]. Furthermore a processing strategy called vector averaging [26] is used in order to translate a vector coarse coded information into a synthetic one. The vector averaging operates through a component-wise weighted sum of the vectors, where the weights are established upon the activity level of the neurons forming the vector coding. It is worthwhile considering the codification capacity of the vectorial scheme in front of that of the local one. Suppose for instance five neurons with two possible states each one. While the local scheme offers 10(= 5 ∗ 2) codification possibilities, these turn into 25(= 52 ) in the vectorial scheme. However, vector coding is not considered to be the only neuronal codification scheme. Actually both types of codification are thought to coexist in the cortex for representation: local coding for the representation of scalar magnitudes and vector coding for the representation of vectorial ones [6] [26]. Both types of neural coding can be found in two different forms of representation called fine and coarse coding [26]. In this case the word coding is not referred to the type of achieved representation, but to the type of receptive field that interprets the incoming information in order to generate this code. Thus fine coding neurons present a response fine tuned to a very specific interval of the input variable, while the coarse coding presents a smooth response, e.g. some cells response to exactly vertical bars, while others show as response a bell-shape function centered at 90◦ (see Fig. D.1). This kind of representation shows some similarity to the concepts used in Fuzzy Computing (see Sec. A.1). Recently another type of coding has been presented that receives the name of rank order coding [51]. In this case the information is not carried by the frequency of the spike trains, but by the order in which they arrive at the afferent neuron. The mechanism is suppose to be behind the analysis of visual properties in short time observation windows [173]. In this context contrast in the input image is


177

Figure D.2: In the visual system, rank order coding [51] is interpreted as the analysis of rank order among the input signals of a neuron in an observation window of short duration. This fact lays on the translation of stimulus contrast into neuronal latency achieved through this type of coding. From [173].

coded through the latency1 of the spike trains generated in each sensing unit. In this context a neuronal model have been proposed whereby visual saliency, e.g. contrast in the retina, translates to latency [173] (see Fig. D.2). Therefore, rank order coding allows the comparison among the input signals in order to assess their ranking order. Furthermore the neurons using this codification scheme manage to answer to a particular ranking (see Fig. D.2). This type of coding have been used in a diverse systems for: analysis of natural images [31], simulation of attentional mechanisms [173], face detection [173], and face recognition [32]. One of the questions that support the existence of the rank order coding is the better performance of this scheme in terms of coding capability. For the example formerly given this coding scheme offers 120(= 5!) coding possibilities. Furthermore rank order coding explains the short response times of the visual system, since this type of coding suppress the necessity of having a time window in order to analyze the frequency rate of the incoming signal [173].

D.2.3 Brain Maps So-called brain maps exist in different cortical areas, which represent the most relevant information of the outer world into reduced two-dimensional spaces. These maps present some topological order of the stimuli while reducing the dimensionality of the representation space [85]. Because of this characteristic 1

Latency is a concept expressing the period of time between the emission and the reception of a spike pulse or train between two neurons

178


(a)

(b)

Figure D.3: Two different examples of brain maps, where the excitation of some outer stimuli are projected on. (a) Portion of the primary visual system of a cat, where information about orientation (A, color coded) and frequency (B, black low frequency) are projected. From [72]. (b) Portion of the somatosensory cortex of three monkeys (M3, M4, and M5), where somatotopic information of the fingers is projected. The overlapping among regions can be observed in the left column (areas delimited with lines). From [148].

they receive the name of topographic maps as well [26]. All sensory modalities project onto such two-dimensional spaces (see Fig. D.3). Beyond these uni-modal maps similar structures exist in other cortical areas related with the fusion of multi-sensory information, e.g. associative areas, and the hippocampus [85]. While some maps maintain the spatial proximity of the stimuli in the outer world, e.g. retinotopical maps in the visual primary system [134], there are others that group similar features, e.g. frequency and interaural phase from the acoustic signals in the nucleus laminaris of the barn owl [84]. As the result of an evolutionary process (see Sec. D.2.4), which is denoted with the general name of topological self-organization [131], clusters of similar features on the surface of the neuronal cortex are formed. Furthermore there exist maps known as category maps because of its capacity of categorizing similar objects [169] or concepts [85]. In these maps the limits among clusters are not just an abstract concept, but a physical one formed by tiny bands of neurons which are not excitable by the outer stimuli [20] (see Fig. D.4).

D.3 Multi-sensory fusion at different organizational levels

179

D.2.4 Evolutionary Processes Although the scientific value of Evolutionary Theory [28] is popularly thought to be restricted to the “successes” of species evolution, modern research have shown the real worth of the Darwinist theory. The classical interpretation of evolution is centered in the concept of optimization [28] [174]. Nevertheless evolutionary processes are more complex than that. They are based on some basic principles, which iteratively repeat on time: reproduction, production of variants, competition driven by a multifaceted environment, and increment of the reproduction chances of those groups of more successful individuals [20]. This success has been also described as the adaptation capability or in another terms as the congruency with the reality [40]. In this context it is worth mentioning that this success do not imply optimization but satisfaction in a sub-optimal local sense [174]. These principles allow evolutionary processes to be behind a great number of search procedures in nature at very different hierarchical levels. For instance the organization of the initial chemical bindings on the earth, which afterwards would origin the life, have been considered an evolutionary process [130]. In another completely different context these processes yield to the appearance of languages and thence of the reasoning capability as well [40]. In the brain, different neurons compete for the representation of concepts by undergoing an evolutionary process [20]. In this case different features embedded in a cortical map undergo a competitive process in another cortical map of a higher hierarchical level in order to codify the object being observed . Evolutionary principles, which rule this competitive process, are engaged at all these different levels (see Fig. D.4). First, different hypothesis are established. Thence these hypothesis spread over the cortical surfaces by cloning, where this cloning is driven by the action of the outer stimuli and of other cerebral areas. In this cloning process some variations may occur, that generate lightly different representations. It is worth mentioning that the territorial competition among the different candidates is spatially limited by areas that do not manage to be excited (see Fig. D.4, gray hexagons). Eventually the actual identity of the object is established. In summary, it can be posited that the force guiding the multi-point search procedure on a population of different possibilities is the adaptation to the outer conditions, or expressed in another way, the capability of the population to fit in the Umwelt, where it is embedded.

D.3 Multi-sensory fusion at different organizational levels Different sensory transducers (see Fig. D.5) transform physical variables into neuronal currents achieving the same functionality of sensors in technological systems

180


Figure D.4: Different candidates undergo an evolutionary process in order to occupy the maximum extension of a cortical map. Initial hypothesis are cloned into initially not occupied areas (1). The cloning procedure, whereby light variations of the initial hypothesis are generated, is limited by areas that are not excitable (2). The evolutionary competition is driven by the outer stimuli (3). The process finishes when a critical mass is reached that is sufficient congruent with the outer reality (4). Modified from [20].

(see Sec. 2.3.1). Thus light and sound wavelengths, chemical composition, motion, pressure, temperature, and other energy forms in the external world make these transducers depolarize or hyperpolarized in a graded fashion [26]. These different signals are processed within the brain and therefore build the complex cognitive processes that the embedding organisms present. The modalities of sensory processing undertaken in the organism do not differ from these presented in the context of technological systems with multi-sensory integration, i.e. independent, cueing, and fusion [104] (see Sec. 2.3.1). In this section the process of multi-sensory fusion present in biological systems will be described. This description is done on different levels of organization [26], which consider multi-sensory fusion at different structural levels (see Fig. D.6). As you will observe there are some organizational levels missing. Although there are examples of multi-sensory fusion at each level, this section concentrates in those aspects that are relevant to the here presented dissertation. Moreover as it has been reported in different works [162] [147] the analysis of multi-sensory fusion are just at the starting line and further research has to be conducted in order to understand the mechanisms underlying multi-sensory fusion in neuronal systems .

D.3.1 Cognitive level The importance of multi-sensory fusion in cognitive processes is out of doubt, since multi-sensory fusion underlies a great diversity of high level perceptual, cognitive, attentive, and learning processes [162]. Furthermore emotional responses are thought to be related with multi-sensory fusion as well [98] [162]. In spite of this ubiquity in neuronal processing, few cognitive processes related with multi-sensory fusion are described in the cognitive sciences’ literature. A list of

181


(a)

(b)

(c)

(d)

(e)

Figure D.5: Sensory transducers transform different types of energy in the external world into neural currents by means of polarization. The sensory transducers of the different senses in human beings are depicted. (a) Vision: Rod (left) and cones (right). (b) Hearing. (c) Smell. (d) Taste. (e) Touch: Free nerve ending (left) and Meissner corpuscle (right). From [127].

these processes include: • Ventriloquist effect. Humans perceived an anthropomorphic inanimate object as being talking when its mouth synchronously moves with the ventriloquist speech [162]. This perception is thus achieved under particular geometrical and temporal constrains. • Speech recognition is achieved not only by the hearing sense but with help of the vision one. In this case the acoustic signals are fused with the visual features derived from the analysis of the lip movements [104] [162]. • McGurk effect. In this cognitive event, related to speech perception, a particular phoneme combination is misperceived when the acoustic information gathered by the hearing system and the visual information generated by reading it differs. For instance a phonem will be perceived as /da/, if the acoustic signal embeds the sound /ba/ and is coupled with a written phoneme /ga/ [147]. Even two non-sense sentences, one read and the other heared, can become understandable upon this effect [104]. • Synesthetic effects are suffered by particular persons upon which the perception of some object through particular senses generates sensations on

182


Figure D.6: Structural levels of organization for the study of the nervous system. Each level has its corresponding scale as its denoted in the picture. These different levels fix up the different perspectives for the description of cognitive functions, which in the here presented work relate to multi-sensory fusion. From [27].

other ones, e.g. perception of color when hearing to music [162]. • Color perception is one of the most obvious processes of multi-sensory fusion since the signals of three receptors to three different spectral bands is transformed into a linguistic label after recognition [174]. • The modification of the intensity, duration and frequency of visual impulses by coupling an acoustic one has been reported as well [147]. E.g. a light flash is perceived as two ones if being coupled with two beeps [144]. • Subjective impression from a particular person based on the integration of some characteristics, e.g. physical features [163], qualifying adjectives [4]. In such tasks a descriptive general impression is achived by fusing the evaluation of individual features. Few computational models at the cognitive level that explain the formerly mentioned processes have been established. Furthermore the formerly mentioned literature usually treats these processes from a descriptive perspective. In the


183

following the exceptions analyzing the mechanisms that underly the process of multi-sensory fusion within these cognitive experiments from a computational point of view are described. Speech perception constitutes one of these exceptions. Thus a cognitive model have been tested in cognitive tasks of simultaneous perception of acoustical and visual stimuli [104]. This model is based on four stages: evaluation, integration, assessment, and response selection. In the evaluation stage the parallel processing of the data delivered by the information sources and its transformation into a psychological feature space by means of fuzzy logic is undertaken. The integration stage proposes the multiplication of the resulting fuzzy features, generating different degrees of support for the different speech alternatives. One of these alternatives is chosen through assessment and converted into behavior via the response selection stage. This model has been presented as a hypothetical general model of processes with multi-sensory fusion [105]. Although there is no complete cognitive model for color perception, Fuzzy Computing related concepts have been denoted from different perspectives to play a crucial role in the explanation of color perceptual processes as well. Thus [174] states that the cognitive process behind color perception and specially color categorization can be modeled at its best using elements of fuzzy set theory. Furthermore the mapping between the cone responses and the perception of over 10, 000 hues have been presented as a consequence of vector coarse coding [26], which is intimately related to Fuzzy Computing concepts as formerly described (see Sec. D.2.2). The seminal research on the fuzzy integral [163] takes into consideration a cognitive experiment for the simulation of the subjective impression derived from the analysis of different parts of a female face by a male subject. The obtained experimental data can be approximated through the application of the fuzzy integral as fusion operator. This question is further analyzed below (see Sec. D.3.3).

D.3.2 Systemic level Most of the areas of the cerebral cortex receive the name of association cortex [26]. These association areas are divided in those receiving signals from just one sensory modality, which are denoted as uni-modal association or secondary sensory areas, and those binding information from several modalities, which are denoted as poli-modal association areas or association cortex (see Fig. D.7). Thus the rest of the cerebral cortex, which occupy a minimal part of the cortical surface, is formed by the primary sensory and motor areas, whose functionality is better known. From a systemic perspective multi-sensory fusion have been reported mainly on two areas centered on the superior colliculus [162] and the hippocampus [6], which

184

Appendix D Multi-sensory Fusion in Biological Systems primary motor cortex premotor cortex primary auditory cortex

primary somatosensory cortex

prefrontal association cortex secondary visual cortex

orbital frontal cortex primary visual cortex limbic association cortex secondary auditory cortex

parietal-temporal-occipital association cortex

Figure D.7: Different areas of the cerebral cortex. Modified from [140].

are embedded in the mid-brain2 and related to both unimodal and polimodal association areas. While the first system is involved in the generation of motoric responses as an answer to attentive processes, the second one tackles tasks related to memory formation with different purposes, e.g. long-term declarative memory, navigational memory. The multi-sensory fusion around the superior colliculus serves the coordinated movement of different areas via a multi-motor maps found in this structure [162]. In this case the multi-sensory fusion assures that different areas move together as an answer to related stimuli. This multi-motor maps receive information from multisensory and unisensory maps, which are embedded in the superior colliculus as well. While the input signals to both types of maps come indistinguishably from several unimodal and polimodal association areas (see Fig. D.8), the nature of their output signals clearly differ. Thus multisensory maps generate signals that result from the fusion of its multisensory inputs [162] (see Sec. D.3.3). The fusion process is controlled by unimodal sensory signals from different parts of the cortex [162]. Therefore it exists a basic structure, where the fusion is realized in a particular subsystem being controlled by the information generated in another one. A similar structural organization have been reported around the hippocampus [136]. Thus in this case the hippocampus acts as an additional source of excitation for binding multisensory signal in polimodal association areas of the cortex [120]. This function is related with the capability of the hippocampus to generate 2

This is the central area of the brain under the neuronal cortex.

185


information about context [108]. Although the big picture is not quite clear, a block diagram of the hippocampal complex can be made based on information about the afferent and the efferents of the different areas [6] (see Fig. D.9).

UNIMOD. AREAS SUPERIOR COLLICULUS

MOTOR AREAS

POLIMOD. AREAS

Figure D.8: Block diagram of the system around the superior colliculus, where multi-sensory fusion is undertaken. Unimodal sensory signals from unimodal (UNIMOD. AREAS ) and polimodal (POLIMOD. AREAS ) cortical areas are fused in order to generate commands of coordinated movements for the motor cortical areas (MOTOR AREAS ).

HIP

UNIMOD. AREAS

DG

PPATH POLIMOD. AREAS

ENT

CA3

CA1

Figure D.9: Block diagram of the system around the hippocampus related with multi-sensory fusion process. The hippocampus is formed by the dentate gyrus (DG) and the cornu amoni (CA), which is divided in four different areas [6]. The hippocampus itself is connected to the other part of the so-called hippocampal complex, the entorhinal cortex (ENT ) via the perforant path (PPATH ). The hippocampal complex is connected to uni-modal (UNIMOD. AREAS ) and polymodal association (POLIMOD. AREAS ) areas of the cortex. The depicted structure takes part in navigational and memory formation processes.

186


The depicted structure plays a role in very different types of processes like shortand long-term memory formation [6] [26], memory consolidation [8], navigation [6], and in conjunction with the amygdala in emotional processing [98] [160]. Thus it is clear that the hippocampus and the entorhinal cortex are involve in binding information coming from different sensory modalities accomplishing some kind of “recognition memory” functionality [26]. Where the multi-sensory fusion actually takes place and even weather it does take place are still open research questions. In a computational model of memory consolidation processes [8] the interplay between association areas and the hippocampal complex has been modeled. Thus the functionality of the association areas for categorization is simulated through a self-organizing neural network, whose purpose is the reduction of the dimensionality by preserving the topological structure. The clusters in this network, which represent different concepts, recall the responses of an auto-associative neural network, which models the functionality of the area CA3 in the hippocampus (see Fig. D.10). This recalling process is mediated by the entorhinal cortex.

Figure D.10: Life-long learning processes, which are denoted as memory consolidation processes as well, are schematically represented by the interplay between a self-organizing neural map, representing the action of association areas in the cortex in categorization, and an associative network, representing the hippocampus. Modified from [8].

D.3.3 Neuronal level The following section treats the neuronal mechanisms that implements the multisensory fusion itself (see Fig. D.11). These neurons bind together two or three


187

sensory modalities and transform this multisensory input into one representational form. There are neurons integrating3 auditive and visual, somatosensory and visual, and these three modalities [162]. In mammals the following areas of the brain present such neurons: superior colliculus, reticular activating system4 , external nucleus of the inferior colliculus, thalamus, and the cortical association areas (see Fig. D.7) superior temporal, parietal, prefrontal, and orbito-frontal [162].

Figure D.11: Neurons that fuse the acoustic, visual and somatosensory modalities in different combinations have been in the brain. These neurons take into account the following rules in order to realize the process of multi-sensory fusion [162]: • Stimuli that are spatially coincident suffer an enhancement. This effect is accomplished by the registration of the receptive fields of the unimodal signals in the multisensory maps, e.g. the multisensory maps of the superior colliculus. • Stimuli that are temporarily coincident suffer an enhancement. This is accomplished by the existence of long observation windows that compensate for the different transmission time of the sensed energy and the processing times of the different modalities. • The aforementioned enhancement is lower when the inputs to a multisensory neuron are of the same sensory modality. 3

Although in the technological context integration and fusion are two different concepts (see Sec. 2.3.1) the literature on cognitive sciences both terms are indistinguishably used. 4 The reticular activating system is situated in the brain stem and in charge of arousal processes

188


• The relative enhancement achieved by multi-sensory fusion increases as the efficiency of the individual stimuli decreases (see Fig. D.12). By means of these rules the cognitive subject manages to increase its awareness in front of stimuli that are temporality synchronized and spatially coincident, and thus that are probably generated by the same event. In this context the mechanism maintains the response level when the efficiency of the stimuli decreases. This is an interesting property since it allows the subject to offer the same behavior while being exposed to ambiguous stimuli. Indeed these rules have been demonstrated by taking behavioral experiments into account [162]. The mechanism underlying the reinforced responses has been qualified as multiplicative, not in the sense of the mathematical product but of the non-linearity [162]. The question about the fusion operator of different frameworks tested on cognitive experiments is a central question in the discussion presented in [105]. Therefore different frameworks based on basically three different fusion operators, namely sum, weighted sum, and product, are compared. The result of the comparison is the better performance of the product when tested against behavioral data. Nevertheless a sentence of this text referring to other models of the multi-sensory fusion procedure seems to be premonitory: “Perhaps people use exemplar-based categorization5 early in learning before reliable summary description is developed”. In this context it is worth mentioning the description of the non-linear behavior of the fuzzy integral, which therefore models the process of multi-sensory fusion in subjective evaluation tasks [163].

D.4 Fuzzy Measures Theory: Framework for bio-inspired and cognitive based information fusion After realizing the importance of multi-sensory fusion for the implementation of industrial applications of Computer Vision, the question arises of how this process is undertaken in natural systems because of its inspiring value for this research field. Although the field of bio-inspired and cognitive based multi-sensory fusion in artificial systems is emerging and still not well consolidated (see Sec. D.1), some general principles can be established from the existing frameworks developed under the formerly mentioned approach. These principles have to be taken into account when developing a methodology for cognitive based multi-sensory fusion. In the following, the principles extracted from different computer vision systems with multi-sensory fusion developed by taking into account the biological and cognitive perspectives (see Sec. D.1) are summarized. multi-sensory fusion from a cognitive perspective is characterized by a common representation of the incoming 5

Another model for multi-sensory fusion commented in [105]

D.4 Fuzzy Measures Theory: Framework for cognitive information fusion

189

Figure D.12: Multisensory enhancement at the neuronal level relatively increases as the efficiency of the stimuli decreases. This feature have been denoted in the literature on fuzzy fusion operators as reinforcement capability [190].

190


signals previous to the fusion operation. Two principles characterize the operator in charge of the fusion itself. First, the information about the context, where the fusion operation occurs, influence the result of the operator. Second the operator overcomes the averaging nature of traditionally used operators. Furthermore it is worth mentioning the local quality of biological based information fusion for image processing and the relevance of SOM related methodologies to this field of research. The aforementioned foundations of bio-inspired and cognitive based multisensory fusion can be completed and extended taking into consideration the general principles of information processing in the brain and the most recent research results of neuroscience on multi-sensory fusion, which are respectively described in Secs. D.2 and D.3. Concerning these principles the following conclusions can be made. Technologies related with Fuzzy Computing play a principal role in the processes of neuronal information processing. Two different schemes for neuronal coding support this assertion (see Sec. D.2.2). Vector coarse coding reproduce the general representation attained by fuzzy systems, namely the distributed representation that is based on the real codification of information in contrast to the binary codification. In this sense, diverse neurons present a jointly and gradually changing response to a single particular outer stimulus. The second scheme, denoted as rank order coding, introduces the ranking relationship among the afferent signals to a neuron in the codification of the information. This is a very novel aspect which has driven the evolution of operators for soft data fusion (see Sec. 4.2). Other technologies modeled by methodologies embedded in Soft Computing underlay the processing of information in the human brain. In this context, the outer information is projected onto so-called brain maps, where a competitive evolutionary process takes place among different neurons in order to represent it. Brain maps have been modeled by different neural networks paradigms, i.e. SOM and analogous approaches, and evolutionary processes are the inspiring force of Evolutionary Computing. Taking into consideration the cognitive process of multi-sensory fusion, some assertion can be derived from the different levels of analysis. The different cognitive models of multi-sensory fusion confirm the necessity of a common representation prior to the fusion. Moreover this representation is implemented in the presented models (see Sec. D.3.1), which take into consideration processes of speech perception, color perception, and subjective evaluation, through different Fuzzy Computing approaches. At the systemic level (see Sec. D.3.2), the process of multi-sensory fusion is split into two different structures, one realizing the fusion itself and the othe inducing it. The analysis at the neuronal level (see Sec. D.3.3) confirms the non-linear behavior of the neuronal fusion operator. In this context it is worth mentioning its reinforcement capability. Although this property has been modeled through the application of a product operator in some cognitive models, this may lie on the knowledge gap between the neuroscientific

D.5 Conclusions

191

and the signal processing research communities. Taking all these facts into consideration, it can be concluded that the fuzzy integral and the theoretical framework of the fuzzy measures can emerge as a paradigm for the further development of the bio-inspired and cognitive based multi-sensory fusion. The fuzzy integral uses fuzzy membership functions as a common representational form prior to the fusion. Furthermore the fuzzy integral itself realizes the fusion operation, whereas the construction of the weighting fuzzy measures can be automated by a separate module. Thus, the hybridization of the fuzzy integral with other Soft Computing methodologies is an advancement in this direction. The reinforcement capability of the fuzzy integral is a known property of this operator. Further research on computational neuroscience nevertheless has to be undertaken in order to fully support the establishment of the fuzzy measures theoretical framework for the implementation of bio-inspired and cognitive based multi-sensory fusion systems.

D.5 Conclusions Different principles for the development of bio-inspired and cognitive perspective on information fusion methodologies are elucidated in the here presented Appendix. The connection between this research field and the theoretical framework of fuzzy measures and fuzzy integrals is established. Nevertheless the implementation of genuine bio-inspired and cognitive based computational systems that implement information fusion can only succeed after the further research on processes with information fusion in biological systems. This should be the first research goal in order to manage consolidating the field of cognitive based information fusion in engineering. The fuzzy integral can be used as fusion operator in the development of cognitive models of information fusion, since its mathematical properties fulfill the cognitive principles of the existent models. In this context special attention should be devoted to the implementation of methodologies for the automated construction of fuzzy measures. The fields of automated robot navigation, multi-modal speech recognition, and color perception offer the best playgrounds for the further development of such systems.

192

193

Bibliography [1] M. A. Abidi and R. G. Gonzalez, eds., Data Fusion In Robotics and Machine Intelligence, Academic Press, Inc., San Diego, 1992. [2] W. Adam, M. Busch, and B. Nickolay, Sensoren f¨ ur die Produktionstechnik, Springer-Verlag, Berlin, 1997. [3] F. Alkoot and J. Kittler, Improving the performance of the product fusion strategy, in Proc. 15th International Conference on Pattern Recognition, ICPR, Vol. 2, Barcelona, Catalonia, 2000, pp. 2164–2167. [4] N. Anderson, Foundations of information integration theory, Academic Press, New York, 1981. [5] P. G. Anderson, Linear pixel shuffling for image processing, an introduction, Journal of Electronic Imaging, April (1993), pp. 147–154. ´ [6] M. A. Arbib, P. Erdi, and J. Szentgothai, Neural Organization: Structure, Function, and Dynamics, MIT Press, Cambridge, MA, 1998. [7] N. Ayache and O. Faugueras, Maintaining representations of the environment of a mobile robot, IEEE Trans. Robotics and Automation, 5 (1989), pp. 804–819. [8] E. Barakova, T. Lourens, and Y. Yamaguchi, Life-long learning: Consolidation of novel events into dynamic memory representations, in Computational Methods in Neural Modeling, Proc. 7th Int. WorkConference on Artificial and Natural Neural Networks, IWANN, LNCS 2686, Part I, Maó, Balearic Islands, 2003, pp. 110–117. [9] H. Barlow, Single unit and sensation: a neurone doctrine for perceptual psychology?, Perception, 1 (1972), pp. 371–394. [10] G. Beliakov, Aggregation operators as similarity relations, in Information, Uncertainty and Fusion, B. Bouchon-Menier, R. R. Yager, and L. A. Zadeh, eds., Kluwer Academic Pub., Boston, 2000, pp. 331–341.

194

Bibliography

[11] S. Beucher and F. Meyer, The morphological approach of segmentation: the watershed transformation, in Mathematical Morphology in Image Processing, E. Dogherty, ed., Marcel Dekker, New York, 1992. [12] J. C. Bezdek, What is computational intelligence?, in Computational Intelligence: Imitating Life, J. M. Zurada, ed., IEEE Press, 1994. [13] J. C. Bezdek, J. M. Keller, R. Krisnapuram, and N. R. Pal, Fuzzy Models and Algorithms for Pattern Recognition and Image Processing, Kluwer Academic Pub., Boston, MA, 1999. [14] J. C. Bezdek and S. K. Pal, eds., Fuzzy Models for Pattern Recognition, IEEE Press, New York, 1991. [15] C. M. Bishop, Neural Networks for Pattern Recognition, Oxford University Press Inc., New York, 1995. [16] L. Bogoni, Extending dynamic range of monochrome and color images through fusion, in Proc. Int. Conf. Pattern Recognition, ICPR, Vol. 3, Barcelona, Catalonia, 2000, pp. 7–12. [17] K. W. Bonfig, ed., Temperatursensoren: Prinzipien und Applikationen., expert-Verlag, Renningen-Malmsheim, 1995. [18] T. Bower, The evolution of sensory systems, in Perception: Essays in honor of James J. Gibson, R. MacLeod and H. E. E. Pick, Jr., eds., Cornell Univ. Press, 1974, pp. 141–153. [19] J. E. Brignell, The future of intelligent sensors: A problem of technology or ethics?, Sensors and Actuators A, 56 (1996), pp. 11–15. [20] W. H. Calvin, The cerebral code: Thinking a thought in the mosaics of mind, MIT Press, Cambridge, MA, 1996. [21] G. Cantor, Gesammelte Abhandlungen, Berlin: Springer-Verlag, 1932. [22] M. J. Carreira, M. Mirmehdi, B. Thomas, and J. Haddon, Grouping of directional features using an extended Hough Transform, in Proc. International Conference Pattern Recognition, ICPR, Vol. 3, Barcelona, Catalonia, 2000, pp. 1002–1005. [23] T.-Y. Chen, J.-C. Wang, and G.-H. Tzeng, Identification of General Fuzzy Measures by Genetic Algorithms Based on Partial Information, IEEE Trans. on Systems, Man, and Cybernetics - Part B, 30 (2000), pp. 517–528. [24] Z. Chi, H. Yan, and T. Pham, Fuzzy Algorithms: With Applications to Image Processing and Pattern Recognition, World Scientific Publishing Co., 1996, ch. 2 Membership Functions, pp. 17–44.

Bibliography

195

[25] J.-H. Chiang and P. Gader, Hybrid fuzzy-neural systems in handwritten word recognition, IEEE Transactions on Fuzzy Systems, 4 (1997), pp. 497– 510. [26] P. S. Churchland and T. J. Sejnowski, The Computational Brain, MIT Press, Cambridge, MA, 1992. [27]

, Perspectives on cognitive neuroscience, in Neuro-Vision Systems: Principles and Applications, M. Gupta and G. Knopf, eds., IEEE Inc., New York, 1994, pp. 39–43.

[28] C. Darwin, On the origin of species by means of natural selection or the preservation of favored races in the struggle for life, Murray, London, 1859. [29] J. G. Daugman, Complete discrete 2-D gabor transforms by neural networks for image analysis and compression, in Neuro-Vision Systems: Principles and Applications, M. Gupta and G. Knopf, eds., IEEE, Inc., New York, 1993, pp. 233–243. [30] R. K. De, N. R. Pal, and S. K. Pal, Feature analysis: Neural network and fuzzy set theoretic approaches, Pattern Recognition, 30 (1997), pp. 1579–1590. [31] A. Delorme, L. Perinnet, and S. J. Thorpe, Networks of integrateand-fire neuron using rank order coding B: Spike timing dependent plasticity and emergence of orientation selectivity, Neurocomputing, 38-40 (2001), pp. 539–545. [32] A. Delorme and S. J. Thorpe, Face identification using one spike per neuron: resistance to image degradation, Neural Networks, 14 (2001), pp. 795–803. [33] Y. Dote and S. J. Ovaska, Industrial applications of soft computing: A review, Proceedings of the IEEE, 89 (2001). [34] D. Dubois, H. Fargier, and H. Prade, Beyond min aggregation in multicriteria decision: (ordered) weighted-min; discri-min; leximin, in The ordered weighted averaging operators: theory and applications, R. Yager and J. Kaprzyck, eds., Kluwer Academic Publishers, Boston, MA, 1997, pp. 181–192. [35] D. Dubois and H. Prade, Unfair coins and necessity measures: Towards a possibilistic interppretation of histograms, Fuzzy Sets and Systems, 10 (1983), pp. 15–20. [36]

, Possibility Theory: An Approach to Computerized Processing of Uncertainty, Plenum Press, New York, 1988.

196

Bibliography

[37] R. O. Duda and P. E. Hart, Pattern Classification and Scene Analysis, John Wiley & Sons, New York, 1973. [38] R. O. Duda, P. E. Hart, and D. G. Stork, Pattern Classification, John Willey & Sons, Inc., New York, 2001. [39] A. Elfes, Multi-source spatial data fusion using Bayesian reasoning, in Data Fusion in Robotics and Machine Intelligence, M. A. Abidi and R. C. Gonzalez, eds., Academic Press, San Diego, 1992, pp. 137–164. [40] N. Elias, Teor´ıa del S´ımbolo: Un ensayo de antropolog´ıa cultural, Editorial Pen´ınsula, Barcelona, 1994. [41] S. Ellias and S. Grossberg, Pattern formation, contrast control, and oscillations in the short memory of shunting on-center off-surround networks, Biological Cybernetics, 20 (1975), pp. 69–98. [42] D. Fay, A. Waxman, M. Aguilar, D. Ireland, J. Racamato, W. Ross, W. Streilein, and M. Braun, Fusion of multi-sensor imagery for night vision: Color visualization, target learning and search, in Proc. 3rd International Conference on Information Fusion, Vol. 1, 2000, pp. 3–10. [43] A. Filippidis, L. Jain, and N. Martin, Fusion of intelligent agents for the detection of aircraft in SAR images, IEEE Transactions on Pattern Analysis and Machine Inteligence, 22 (2000), pp. 378–384. [44] G. D. Finalyson, S. D. Hordley, and M. S. Drew, Removing shadows from images, in Proc. European Conference on Computer Vision, ECCV, vol. LNCS 2353, Coppenhagen, 2002, pp. 823–836. [45] G. F. Fulop, The U.S. market for infrared thermography equipment, in Proceedings of the SPIE, Vol. 2473, 1995, pp. 119–125. [46] T. Furuhashi, Fusion of fuzzy/neuro/evolutionary computing for knowledge acquisition, Proceedings of the IEEE, 89 (2001), pp. 1266–1274. [47] D. Gabor, Theory of communication, J. Institution of Electrical Engineers, 93 (1946), pp. 429–457. [48] T. D. Garvey, A survey of AI approaches to the integration of information, in Multisensor Integration and Fusion for Intelligent Machines and Systems, R. C. Luo and M. G. E. Kay, eds., Ablex Publishing Corporation, Norwood, NJ, 1995, pp. 49–79. [49] N. Gat and S. Subramanian, Spectral imaging: Technology & applications, Hyperspectrum News Letter, 3 (1997).

Bibliography

197

[50] N. Gat, S. Subramanian, J. Barhen, and N. Toomarian, Spectral imaging applications: Remote sensing, environmental monitoring, medicine, military operations, factory automation and manufacturing., in Proc. 25th AIPR Workshop on Emerging Applications of Computer Vision, SPIE Vol. 2962, 1997, pp. 63–77. [51] J. Gautrais and S. J. Thorpe, Rate coding versus temporal order coding: a theoretical approach, Neurocomputing, 48 (1998), pp. 57–65. [52] J.-M. Geusebroek, R. van den Boomgaard, A. W. Smeulders, and H. Geerts, Color invariance, IEEE Trans. Pattern Analysis and Machine Intelligence, 23 (2001), pp. 1338–1350. [53] T. Gevers, Color based image retrieval, in Principles of Visual Information Retrieval, M. Lew, ed., Springer-Verlag, 2001, pp. 11–50. [54] D. E. Goldberg, Genetic Algorithms in Search, Optimization and Machine Learning, Addison-Wesley Pub. Co., Reading, MA, 1989. [55] R. C. Gonzalez and M. G. Thomason, Syntactic Pattern Recognition: An Introduction, Addison-Wesley, Reading, MA, 1978. [56] M. Grabisch, Fuzzy integrals as a generalized class of order filters, in Proc. of the SPIE, 2315, 1994, pp. 128–136. [57]

, A new algorithm for identifying fuzzy measures and its application to pattern recognition, in Proc. Int. Joint Conf of the 4th IEEE Int.Conf. on Fuzzy Systems and the 2nd Int. Fuzzy Engineering, 1995, pp. 145–150.

[58]

, Fuzzy measures and integrals for decision making and pattern recognition, in Fuzzy Structures: Current Trends, R. Mesiar and B. Rieca, eds., Math. Inst., Slovak Academy of Sc, Bratislava, 1997, pp. 7–35.

[59]

, Fuzzy integral for classification and feature extraction, in Fuzzy Measures and Integrals: Theory and Applications, M. Grabisch, T. Murofushi, and M. Sugeno, eds., Studies in Fuzziness and Soft Computing, PhysicaVerlag, Heidelberg, 2000, pp. 415–434.

[60] M. Grabisch and J.-M. Nicolas, Classification by fuzzy integral: Performance and tests, Fuzzy Sets and Systems, 65 (1994), pp. 255–271. [61] M. Grabisch and M. Schmitt, Mathematical morphology, order filters, and fuzzy logic, in Proc. IEEE/IFES Int. Joint Conf. on Fuzzy Systems, Yokohama, Japan, 1995, pp. 2103–2108.

198

Bibliography

[62] M. Grabisch, H. T.Nguyen, and A. A. Walker, Fundamentals of Uncertainty Calculi with Applications to Fuzzy Inference, Kluwer Academic Pub., Dordrecht, 1995. [63] M. Grimaud, New measure of contrast: dynamics, Image algebra and morphological image processing III, 1769 (1992), pp. 292–305. ¨ ppen, and B. Nickolay, A new approach to fuzzy [64] S. Großert, M. Ko morphology based on fuzzy integral and its application in image processing, in Proc. of Int. Conf. on Pattern Recognition, ICPR, 1996, pp. 625–630. [65] R. J. J. Hansman, Characteristics of instrumentation, in The Measurement, Instrumentation, and Sensors Handbook., J. G. Webster, ed., CRC Press and Springer-Verlag, 1999, pp. I.1–I.8. [66] R. Haralick, K. Shanmugam, and I. Dinstein, Textural features for image calssification, IEEE Trans. on System, Man, and Cybernetics, 3 (1973), pp. 610–621. [67] J. Y. Hardeberg, Acquisition and Reproduction of Color Images: Colorimetric and Multispectral Approaches, Universal Publishers / Dissertation.com, Parkland, FL, 2001. [68] D. Hebb, Organization of Behaviour, Wiley, New York, 1949. [69] J. Himberg, J. Ahola, E. Alhoniemi, J. Vesanto, and O. Simula, The self-organizing map as a tool in knowledge engineering, in Pattern Recognition in Soft Computing Paradigm, N. R. Pal, ed., World Scientific Pub., Singapore, 2001, pp. 38–65. [70] A. Hocaoglu and P. Gader, Choquet integral representations of nonlinear filters with applications to LADAR image processing, in Proc. of the SPIE Conf. on Nonlinear Image Processing IX, San Jose, CA, 1998, pp. 66– 72. [71] A. Hocaoglu, P. Gader, and J. M. Keller, A fuzzy integral filter for object detection in LADAR images, in Proc. North American Fuzzy Information Processing Society, Syracuse, NY, 1997, pp. 177–182. ¨bener, D. Shoham, A. Grinvald, and T. Bonhoeffer, Spa[72] M. Hu tial relationships among three columnar systems in cat area 17, Journal of Neuroscience, 17 (1997), pp. 9270–9284. [73] T. Huntsberger, Data fusion: A neural networks implementation, in Data Fusion in Robotics and Machine Intelligence, M. A. Abidi and R. C. E. Gonzalez, eds., Academic Press, 1992, pp. 507–535.

Bibliography

199

[74] S. A. Hutchinson and A. C. Kak, Multisensor strategies using Dempster-Schafer belief, in Data Fusion in Robotics and Machine Intelligence, M. A. Abidi and R. C. Gonzalez, eds., Academic Press, San Diego, 1992, pp. 251–264. [75] J. H. Holland, Adaptation in natural and artificial systems, The University of Michigan Press, Ann Arbor, 1975. ¨hne, Digital Image Processing, Springer-Verlag, Berlin, 2002. [76] B. Ja [77] J. M. Keller, P. Gader, R. Krishnapuram, X. Wang, A. Hocaoglu, H. Frigui, and J. Moore, Fuzzy logic automatic target recognition systm for LADAR range images, in Proc. IEEE Int. Conf. on Fuzzy Systemes, FUZZ’IEEE, 1998, pp. 71–76. [78] J. M. Keller, P. Gader, H. Tahani, J.-H. Chiang, and M. Mohamed, Advances in fuzzy integration for pattern recognition, Fuzzy sets and systems, 65 (1994), pp. 255–271. [79] J. M. Keller and J. Osborn, Training the fuzzy integral, Int. Journal of Approximate Reasoning, 15 (1996), pp. 1–24. [80] J. M. Keller, H. Qiu, and H. Tahani, The fuzzy integral and image segmentation, in Proc. North American Fuzzy Information Processing Society, June 1986, pp. 334–338. [81] J. M. Keller and B. Yan, Possibility expectation and its decision making algorithm, in Proc. IEEE International Conference on Fuzzy Systems, FUZZ’IEEE, 1992, pp. 661–668. [82] E. P. Klement, R. Mesiar, and E. Pap, Triangular Norms, Kluwer Academic Pub., Dordrecht, 2000. [83] G. J. Klir, Z. Wang, and D. Harmanec, Constructing fuzzy measures in expert systems, Fuzzy Sets and Systems, 92 (1997), pp. 251–264. [84] E. Knudsen, S. du Lac, and S. Esterly, Computational maps in the brain, Annual Review of Neuroscience, 10 (1987), pp. 41–65. [85] T. Kohonen, Self-Organizing Maps, Springer-Verlag, Heidelberg, 1995. ¨ ppen, The curse of dimensionality, in Proc. 5th Online Conference [86] M. Ko on Soft Computing in Industrial Applications, WSC5, 2000, 22 pp. ¨ ppen and B. Nickolay, Genetic programming based texture filter[87] M. Ko ing framework, in Pattern Recognition in Soft Computing Paradigm, N. R. Pal, ed., World Scientific Pub., Singapore, 2001, pp. 275–329.

200

Bibliography

¨ ppen, C. Nowack, and G. Ro ¨ sel, Pareto-Morphology for color [88] M. Ko image processing, in Proc. of the 11th Scandinavian Conference in Image Analysis, SCIA, Greenland, Denmark, 1999, pp. 195–202. ¨ ppen, A. Soria-Frisch, and T. Sy, Binary pattern processing [89] M. Ko framework for perceptual fault detection, in Proc. 12th Scandinavian Conf. Image Analysis, SCIA, Bergen, Norway, 2001, pp. 363–370. [90] B. Kosko, Neural Networks and Fuzzy Systems: A Dynamical Systems Approach to Machine Intelligence, Prentice-Hall, Inc., Englewood Cliffs, N.J., 1992. [91] R. Krishnamoorthi and P. Bhattacharya, Color edge extraction using orthogonal polynomials based zero crossings scheme, Information Sciences, 112 (1998), pp. 51–65. [92] D. Lee, The functions of vision, in Modes of perceiving and processing information, H. Pick Jr. and E. E. Saltzman, eds., Willey, 1978, pp. 159– 170. [93] J. Lettvin, H. Maturana, W. McCulloch, and W. Pitts, What the frog’s eye tells the frog’s brain, Proceedings of the Institute of Radio Engineers, 47 (1959), pp. 1940–1951. [94] H. Li, B. Manjunath, and S. Mitra, Multisensor image fusion using the wavelet transform, Graphical Models and Image Processing, 57 (1995), pp. 235–245. [95] D. Littwiller, CCD vs. CMOS: Facts and fiction, Photonics Spectra, January (2001), pp. 154–158. [96] M. Livingstone and D. Hubel, Segregation of form, color, movement, and depth: Anatomy, physiology, and perception, in Neuro-Vision Systems: Principles and Applications, M. Gupta and G. Knopf, eds., IEEE, Inc., New York, 1993, pp. 177–186. [97] L. Lohmann, J. Ruiz-del-Solar, C. Nowack, and B. Nickolay, A robust architecture for the automatic edge detection in real-world scenes, in Proc. Int. Conf. Soft Computing, SOCO, Genua, Italy, 1999, pp. 152–159. [98] T. Lourens, E. Barakova, and H. Tsujino, Interacting modalities through functional brain modeling, in Computational Methods in Neural Modeling, Proc. 7th Int. Work-Conference on Artificial and Natural Neural Networks, IWANN, vol. LNCS 2686, Part I, Maó, Balearic Islands, 2003, pp. 102–109.

Bibliography

201

[99] R. C. Luo and M. G. Kay, eds., Multisensor Integration and Fusion for Intelligent Machines and Systems, Ablex Publishing Corporation, Norwood, NJ, 1995. [100] W.-Y. Ma and B. Manjunath, Edgeflow: A technique for boundary detection and image segmentation, IEEE Trans. on Image Processing, 9 (2000), pp. 1375–1387. [101] S. Mann, Intelligent Image Processing, Willey Interscience, New York, 2002. [102] L. Marks, The Unity of the Senses: Interrelations among the Modalities, Academic Pub., New York, 1978. [103] S. Martin dit Neuville, Intelligence fusion, in Proc. of the 1996 IEE/SICE/RSJ Int. Conf. on Multisensor Fusion and Integration, 1996, pp. 781–787. [104] D. Massaro, Speech reading: illusion or window into pattern recognition, Trends in Cognitive Sciences, 3 (1999), pp. 310–317. [105] D. Massaro and D. Friedman, Models of integration given multiple sources of information, Psychological Review, 97 (1990), pp. 225–252. [106] G. Medioni, I. Cohen, F. Brmond, S. Hongeng, and R. Nevatia, Event detection and analysis from video streams, IEEE Trans. on Pattern Analysis and Machine Intelligence, 23 (2001), pp. 873–889. [107] K. Menger, Statistical metric spaces, Proc. Natural Academy of Sciences U.S.A., 8 (1942), pp. 535–537. [108] R. Miller, Cortico-Hippocampal Interplay and the Representation of Contexts in the Brain, Springer-Verlag, Berlin, 1991. [109] M. Mitchell, An Introduction to Genetic Algorithms, MIT Press, Cambridge, MA, 1996. [110] Y. Miyamoto, K. Nishino, T. Sawai, and E.Nambu, Development of ’AI-VISION’ for fluidized-bed incinerator, in Proc. of the 1996 IEE/SICE/RSJ Int. Conf. on Multisensor Fusion and Integration, 1996, pp. 72–77. [111] T. Murofushi and M. Sugeno, An interpretation of fuzzy measure and the Choquet integral as an integral with respect to a fuzzy measure, Fuzzy Sets and Systems, 29 (1989), pp. 201–227.

202 [112]

Bibliography , Fuzzy t-conorm integral with respect to fuzzy measures: Generalization of Sugeno integral and Choquet integral, Fuzzy Sets and Systems, 42 (1991), pp. 57–71.

[113] R. R. Murphy, Biological and cognitive foundations of intelligent sensor fusion, IEEE Trans. on Systems, Man, and Cybernetics-A: Systems and Humans, 26 (1996), pp. 42–51. [114] Y. Nakamori and T. Terada, Analysis of subjective evaluation structure by the choquet integral model, in Methodologies for the Conception, Design and Application of Soft Computing. Proc. of IIZUKA, Iizuka, Japan, 1998, pp. 390–393. [115] N. Nandhakumar, Robust physics-based analysis of thermal and visual imagery, Journals of the Opt. Soc. Am. A, 11 (1994), pp. 2981–2989. [116] O. Nestares, R. Navarro, J. Portilla, and A. Tabernero, Efficient spatial-domain implementation of a multiscale image representation based on Gabor functions, J. Electronic Imaging, 7 (1998), pp. 166–173. [117] E. Newman and P. Hartline, Integration of visual and infrared information in bimodal neurons of the rattlesnake tectum, Science, 213 (1981), pp. 789–791. [118] A. V. Oppenheim and R. W. Schafer, Discrete-Time Signal Processing, Prentice-Hall International Inc., Englewood Cliffs, NJ, 1989. [119] K. Otoba, K. Tanaka, and M. Hitafuji, Image processing and interactive selection with Java based on genetic algorithms, in Proc. 3rd IFAC/CIGR Workshop on Artificial Intelligence in Agriculture, Makuhari, Japan, Apr. 1998, pp. 83–88. [120] G. Palm, On the internal structure of cell assemblies, in Brain Theory, A. Aertsen, ed., Elsevier Science Publishers B.V. Amsterdam, 1993, pp. 261–270. [121] A. Papoulis, Probability, Random Variables, and Stochastic Processes, McGraw-Hill Inc., Singapore, 1984. [122] J. C. Pearson, J. J. Gelfand, W. Sullivan, R. M. Peterson, and C. D. Spence, Neural network approach to sensory fusion, in Multisensor Integration and Fusion for Intelligent Machines and Systems, R. C. Luo and M. G. E. Kay, eds., Ablex Publishing Corporation, Norwood, NJ, 1995, pp. 111–120.

Bibliography

203

´ bal, and J. Chamorro-Mart´ınez, [123] J. L. Pech-Pacheco, G. Cristo ńdez-Valdivia, Diatom autofocusing in brightfield microscopy: J. Ferna a comparative study, in Proc. Int. Conf. Pattern Recognition, ICPR, Vol. 3, Barcelona, Catalonia, 2000, pp. 318–325. [124] S.-C. Pei and C.-M. Cheng, Color image processing by using binary quaternion-moment-preserving thresholding technique, IEEE Trans. on Image Processing, 8 (1999), pp. 614–628. [125] T. Pham, Genetic learning of multi-attribute interactions in speaker verification, in Proc. IEEE Congress on Evolutionary Computation, CEC, 2000, pp. 379–383. [126] T. D. Pham and H. Yan, Color image segmentation using fuzzy integral and mountain cluctering, Fuzzy Sets and Systems, 107 (1999), pp. 121–130. [127] M. Pines ed., Seeing, smelling, and hearing the world, tech. rep., Howard Hughes Medical Institute, 1995. [128] J. Porrill, Optimal combination and constraints for geometrical sensor data, J. Robotics Research, 7 (1988), pp. 66–77. [129] H. Qiu and J. M. Kelller, Multiple spectral image segmentation using fuzzy techniques, in Proc. North American Fuzzy Information Processing Society, 1987, pp. 374–387. [130] H. Reeves, J. de Rosnay, Y. Coppens, and D. Simonnet, La història més bella del món, Edicions 62, Barcelona, 1997. [131] E. T. Rolls and G. Decco, Computational Neuroscience of Vision, Oxford University Press, Oxford, 2002. [132] L. Ronnie, M. Johansson, and N. Xiong, Perception management: an emerging concept for information fusion, Information Fusion, 4 (2003), pp. 231–234. [133] R. Rosenblatt, Principles of Neurodynamics, Saprtan Books, New York, 1959. [134] J. Ruiz-del Solar, Biologisch basierte Verfahren zur Objekterkennung und Texturanalyse, PhD thesis, Technical University of Berlin, Germany, 1997. [135] J. Ruiz-del-Solar, TEXSOM: Texture segmentation using SelfOrganizing Maps, Neurocomputing, 21 (1998), pp. 7–18.

204

Bibliography

[136] J. Ruiz-del-Solar and A. Soria-Frisch, Toward a bioinspired fusion of color and infrared textural image information, Journal of Advanced Computational Intelligence, 5 (2001), pp. 326–332. [137] R. A. Salinas, C. Richardson, M. A. Abidi, and R. C. Gonzalez, Data fusion: Color edge detection and surface reconstruction through regularization, IEEE Trans. on Industrial Electronics, 43 (1996), pp. 355–363. [138] C. Sanderson and K. K. Paliwal, Noise compensation in a person verifictaion system using face and multiple speech features, Pattern Recognition, 36 (2003), pp. 293–302. [139] M. Santarelli, V. Positano, P. Marcheschi, L. Landini, P. Marzullo, and A. Benassi, Multimodal cardiac image fusion by geometrical features registration and warping, Computers in Cardiology, (2001), pp. 277–280. [140] R. F. Schmidt, ed., Neuro- und Sinnes-Phisiologie, Springer-Verlag, Heidelberg, 1993. [141] B. Schweizer and A. Sklar, Associative funtions and statistical triangle inequalities, Publ. Math. Debrecen, 8 (1961), pp. 169–186. [142] J. Serra, Image Analysis and Mathematical Morphology, Academic Press, New York, 1982. [143] G. Shafer, A Mathematical Theory of Evidence, Princeton University Press, Princeton, N.J., 1976. [144] L. Shams, Y. Kamitani, and S. Shimojo, Visual illusion induced by sound, Cognitive Brain Research, 14 (2002), pp. 147–152. [145] G. Shaw and D. Manolakis, Signal processing for hyperspectral image exploitation, IEEE Signal Processing Magazine, 19 (2002), pp. 29–43. [146] H. Shi, P. Gader, and W. Chen, Fuzzy integral filters: properties and parallel implementations, Journal of Real-Time Imaging, 2 (1997), pp. 233– 241. [147] S. Shimojo and L. Shams, Sensory modalities are not separate modalities: plasticity and interactions, Current Opinion In Neurobiology, 11 (2001), pp. 505–509. [148] D. Shoham and A. Grinvald, The cortical representation of the hand in macaque and human area S-I: High resolution optical imaging, The Journal of Neuroscience, 21 (2001), pp. 6820–6835.

Bibliography

205

[149] P. Soille, Partitioning of multispectral images using mathematical morphology, Vision Jahrbuch, II (1993), pp. 33–43. [150]

, Morphological Image Analysis: Principles and Applications, SpringerVerlag, Berlin, 1999.

[151] A. Soria-Frisch, Intelligent localized fusion operators for color edge detection, in Proc. Scandinavian Conference on Image Analysis, SCIA, Bergen, Norway, 2001, pp. 177–184. [152]

, A new paradigm for fuzzy aggregation in multisensory image processing, in Computational Intelligence: Theory and Applications. Proc. 7th Int. Conf. Fuzzy Days, Dortmund, Germany, 2001, pp. 59–67.

[153]

, Soft data fusion in image processing, in Soft-Computing and Industry: Recent Advances, R. Roy, M. Kppen, S. Ovaska, T. Furuhashi, and F. E. Hoffmann, eds., Springer-Verlag, 2002, pp. 423–444.

[154]

, Avoidance of highlights through ILFOs in automated visual inspection, in Fuzzy Filters for Image Processing, M. Nachtgael, D. Van der Weken, D. Van De Ville, and E. E. E. Kerre, eds., Springer-Verlag, Berlin, 2003, pp. 356–371.

[155]

, The fuzzy integral for color seal segmentation on document images, in Proceedings International Conference on Image Processing, ICIP 2003, Part 1, Barcelona, 2003, pp. 157–160.

[156]

, Hybrid SOM and fuzzy integral frameworks for fuzzy classification, in Proc. IEEE International Conference on Fuzzy Systems, FUZZ’IEEE, St. Louis, MO, 2003, pp. 840–845.

¨ ppen, The fuzzy integral as similarity mea[157] A. Soria-Frisch and M. Ko sure for a new color morphology, in Proc. CGIV’2002: First European Conference in Graphics, Imaging, and Vision, Poitiers, France, 2002, pp. 523– 526. [158]

, Morphological clustering of the SOM for multi-dimensional image segmentation, in Computational Methods in Neural Modeling, Proc. 7th Int. Work-Conference on Artificial and Natural Neural Networks, IWANN, LNCS 2686, Part I, Maó, Balearic Islands, 2003, pp. 582–589.

¨ ppen, and T. Sy, Is she gonna like it? auto[159] A. Soria-Frisch, M. Ko mated inspection system using fuzzy aggregation, in Intelligent Systems for Information Processing: From Representation to Applications, B. BouchonMeunier, L. Foulloy, and R. R. E. Yager, eds., Elsevier Science B.V., 2003, pp. 465–476.

206

Bibliography

[160] L. Squire, F. Bloom, S. McConnel, J. Roberts, N. Spitzer, and M. E. Zigmond, eds., Fundamental Neuroscience, Academic Press, 2002. [161] P. Stager and D. Hameluck, Contrast sensitivity and visual detection in search and rescue, tech. rep., Defence and Civil Institute of Environmental Medecine, Toronto, Canada, 1986. [162] B. E. Stein and M. A. Meredith, The merging of senses, MIT Press, Cambridge, MA, 1993. [163] M. Sugeno, The Theory of Fuzzy Integrals and Its Applications, PhD thesis, Tokyo Institute of Technology, Japan, 1974. [164] P. Sussner, Observations on morphological associative memories and kernel methods, Neurocomputing, 31 (2000), pp. 167–183. [165] M. J. Swain and D. H. Ballard, Color indexing, Int. Journal of Computer Vision, 7 (1991), pp. 11–32. [166] H. Tahani and J. M. Keller, Information fusion in computer vision using the fuzzy integral, IEEE Trans. on Systems, Man, and Cybernetics, 20 (1990), pp. 733–741. [167] H. Takagi, Interactive evolutionary computation: Fusion of the capabilities of EC optimization and human evaluation, Proceedings of the IEEE, 89 (2001), pp. 1275–1296. [168] K. Tanaka and M. Sugeno, A study on subjective evaluations of printed color images, in Fuzzy Models for Pattern Recognition, J. Bezdek and S. Pal, eds., IEEE Press, New York, 1991, pp. 362–368. [169] S. J. Thorpe and M. Fabre-Thorpe, Seeking categories in the brain, Science, 291 (2001), pp. 260–263. [170] H. R. Tizhoosh, Fuzzy-Bildverarbeitung: Einf¨ uhrung in Theorie und Praxis, Springer-Verlag, Berlin, 1998. [171] P. Trahanias, L. Pitas, and A. Venetsanopoulus, Color image processing, in Control and Dynamic Systems, vol. 67: Digital Image Processing: Techniques and Applications, Academic Press, 1994, pp. 45–90. [172] F. Ullmann, Temperaturbestimmung beim Drehen faserverstärkter Kunststoffe, Carl Hanser Verlag, 1992. [173] R. Van Rullen and S. J. Thorpe, Surfing a spike wave down the ventral system, Vision Research, 42 (2002), pp. 2593–2615.

Bibliography

207

[174] F. J. Varela, E. Thompson, and E. Rosch, The embodied mind, MIT Press, Cambridge, MA, 1991. [175] J. Vesanto, SOM-Based data visualization methods, Intelligent Data Analysis, 3 (1999), pp. 111–126. [176] J. Vesanto and E. Alhoniemi, Clustering of the Self-Organising Map, IEEE Trans. on Neural Networks, 11 (2000), pp. 586–600. [177] D. Wang, X. Wang, and J. M. Keller, Determining fuzzy integral densities using a genetic algorithm for pattern recognition, in Proc. North American Fuzzy Information Processing Society, 1997, pp. 263–267. [178] J. Wang and Z. Wang, Using neural networks to determine Sugeno measures by statistics, Neural Networks, 10 (1997), pp. 183–195. [179] Z. Wang and G. Klir, Fuzzy Measure Theory, Plenum Press, New York, 1992. [180] Z. Wang, K.-S. Leung, and J. Wang, A genetic algorithm for determining nonadditive set functions in information fusion, Fuzzy Sets and Systems, 102 (1999), pp. 463–469. [181] L. Warren, An industrial application of the Choquet integral for global performance evaluation operations, in Methodologies for the Conception, Design and Application of Soft Computing. Proc. of IIZUKA’98, Iizuka, Japan, 1998, pp. 381–385. [182] S. Watanabe, Pattern Recognition: Human and Mechanical, John Willey & Sons, New York, 1985. [183] S. Weber, ⊥-descomposable measures and integrals for Archimidean tconorms, Journal of Mathematical Analysis and Applications, 101 (1984), pp. 114–138. [184] J. Webster, ed., The measurement, instrumentation, and sensors handbook, CRC Press and Springer-Verlag, 1999. [185] P. Weckesser and R. Dillmann, Sensor-fusion of intensity- and laserrange-images, in Proc. of the 1996 IEE/SICE/RSJ Int. Conf. on Multisensor Fusion and Integration, 1996, pp. 501–508. [186] T. A. Wilson, S. K. Rogers, and L. R. Myers, Perceptual-based hyperspectral image fusion using multiresolution analysis, Optical Engineering, 34 (1995), pp. 3154–3164.

208

Bibliography

[187] R. R. Yager, On ordered weighted averaging aggregation operators in multicriteria decision making, IEEE Trans. on Systems, Man and Cybernetics, 18 (1988), pp. 183–190. [188]

, Fusion of ordinal information usig weighted median aggregation., International Journal of Approximate Reasoning, 18 (1998), pp. 35–52.

[189] R. R. Yager and J. Kaprzyck, eds., The ordered weighted averaging operators: theory and applications, Kluwer Academic Pub., Boston, 1997. [190] R. R. Yager and A. Kelman, Fusion of fuzzy information with considerations for compatibility, partial aggregation and reinforcement, International Journal of Approximate Reasoning, 15 (1996), pp. 93–122. [191] B. Yan, Optimization of the fuzzy integral in computer vision and pattern recognition, PhD thesis, University of Missouri, Columbia, MO, 1993. [192] L. A. Zadeh, Fuzzy sets, Information Control, 8 (1965), pp. 338–353. [193]

, Outline of a new approach to the analysis of complex systems and decision processes, IEEE Trans. on Systems, Man, and Cybernetics, 3 (1973), pp. 28–44.

[194]

, Fuzzy sets as a basis for a theory of possibility, Fuzzy sets and systems, 1 (1978), pp. 3–28.

[195]

, Some reflections on soft computing, granular computing and their roles in the conception, design and utilization of information/intelligent systems, Soft Copmputing, 2 (1998), pp. 23–25.

[196] Y. Zhang, A survey on evaluation methods for image segmentaion, Pattern Recognition, 29 (1996), pp. 1335–1346.

Soft Data Fusion for Computer Vision

Soft Data Fusion for Computer Vision

Suggest Documents

SENSOR AND DATA FUSION CONTEST - Computer Vision & Remote

Sensor Fusion in Computer Vision - Semantic Scholar

Using Propositional Graphs for Soft Information Fusion - Computer ...

Computer Vision, Sonar and Sensor fusion for ... - CiteSeerX

Computer Vision, Sonar and Sensor fusion for ... - CiteSeerX

complementary vision based data fusion for robust positioning and ...

Fusion Between Laser and Stereo Vision Data For Moving Objects

for computer vision

Mathematics for Computer Vision

From Data Fusion to Knowledge Fusion - UBC Computer Science

From Data Fusion to Knowledge Fusion - UBC Computer Science

Multimodal Stereo Vision System: 3D Data ... - Computer Vision Center

Combining Computer Vision & Data Stream Processing - eSprockets

Symmetry of Fuzzy Data - Computer Vision Lab

Computer vision for computer games - Merl

Computer Vision for Interactive Computer Graphics - Merl

Computer Vision

Machine Learning and Computer Vision System for Phenotype Data ...

CV-HAZOP: Introducing Test Data Validation for Computer Vision

Harmonic Filters for 3D Multi-Channel Data - Computer Vision Group ...

Web-Scale Computer Vision using MapReduce for Multimedia Data ...

Text Data-Hiding for Digital and Printed Documents - Computer Vision ...

Immersive Simulation for Computer Vision

Soft Data Fusion in Image Processing - Semantic Scholar