AUTOMATIC RECOGNITION OF OCEAN STRUCTURES FROM SATELLITE IMAGES BY MEANS OF NEURAL NETS AND EXPERT SYSTEMS Guindos-Rojas, Francisco(1); Cantón-Garbín, Manuel(2), Torres-Arriaza, José Antonio(3); Peralta-López, Mercedes(4); Piedra-Fernández, José Antonio(5) and Molina-Martínez, Alberto(6) Universidad de Almería. Depto. de Lenguajes y Computación. 04120 Almería – SPAIN. Email: (1)
[email protected] (2)
[email protected] (3)
[email protected] (4)
[email protected] (5)
[email protected] (6)
[email protected]
ABSTRACT Images received from satellites have became a great source of information about our environment. This is raw information that needs experts to make the most of it, but there are not many experts and the work is too much. The solution to this problem is the compilation of human experience into automatic systems that could do the same work. We depict here the structure for a knowledge based system capable of taking the place of human experts when it is properly trained. This structure has been used to build an automatic recognition system that process AVHRR images from NOAA satellites to detect and locate ocean phenomena of interest like upwellings, eddies and island wakes. The model covers every phase of the process from the source image, once it is corrected and geocoded, to the final features map. In the most delicate phase of the process pipeline, artificial neural nets and rule-based expert systems are used in a parallel redundant way so results can be validated by comparing the outcome of both subsystems. The automatic knowledge driven image processing system has been trained with ubiquitous and localized information and has proved his qualities with images of Canary Island, Mediterranean Sea and Cantabric and Portuguese coasts.
approximation that show the results obtained with an automatic interpretation system for AVHRR images, developed to detect and to label mesoscale oceanic structures. Such a system is developed with the basic goal to avoid human operation intervention. It includes an automatic cloud mask system developed using a neural approach, an iterative segmentation region builder based in intelligent threshold detection and a high-level information classifier which uses functional and operational principles derived from the ANNs (Artificial Neural Nets) and Rule-Based Knowledge Systems. 2. OVERALL STRUCTURE OF THE SYSTEM Fig. 1 depicts the overall structure of the system that has been developed.
RAW IMAGE
PREPROCESSING
CLOUD MASKING
1. INTRODUCTION During the last three decades the big volume of data about the Earth received from the observation and resource management satellites, mainly as images from the surface of our planet, has been enormous. At present, the major part of this information is stored at the archiving and processing satellite centers without interpretation, because this can be done only by human experts. This problem is of particular importance in the ocean to manage meso and macroscale phenomena. The difficulty of the image analysis and understanding problem for satellite data is due, in large part, to the lack of a precise mathematical description of the observed structures and to their variability. This work is a first
SEGMENTATION
A.N.N. CLASSIFIER
GRAPHIC E.S.
LABEL IMAGE
Fig. 1 Structure of the ocean feature recognition system
In a first step, the raw image is processed by means of algorithms such as radiometric correction, map projection and land masking. These are well known techniques also used when the analysis is made by human experts. However, we don’t make any image enhancement like histogram equalization or contrast stretch that are appropriate to make some features visible to human eye but have no positive effects when the images are going to be processed by artificial means. The second step aims to get ride of clouds that are opaque to radiance data measured in the AVHRR scenes. Cloudy images are distorted in such a way that the zone affected isn’t of any value for our later processing, so we apply a mask of 0s that will exclude these pixels. The following task is the segmentation that will divide the whole image in regions. The idea is that each phenomenon of interest should coincide with one or a small set of the cut off regions. The nature of ocean dynamics makes very difficult this process that is nevertheless fundamental, so we’ve designed an iterative knowledge-driven method to perform this part of the process pipeline. The last step is the recognition. Each region produced in the segmentation is analyzed and, if the recognition is positive, it is labeled with the identifier of the matching structure. The structures of interest in the Canary Islands zone as defined in [1] are: 1 Coastal upwelling. 2 Warm eddies. 3 Cold eddies. 4 Island wake. The vast majority of the regions that appear in the segmentation are of no special interest and they are labeled with a 0. We have implemented a redundant recognition subsystem. It has an ANN-based Symbolic Processing Element (SPE) module and a rule-based Graphic E.S. (GES), both performing the same task. The purpose is to test both technologies and provide a way to validate the results. 3. DETAILED PROCESS 3.1 Cloud Masking The technique used is described in [8]. The basic idea is that all cloud pixels are considered to have one or both of the following features: a) Very low value in channel 4 compared to is neighbors.
b) High variability. The system exploits this idea, locating in a first estimation those pixels affected by both characteristics or very strongly by one of them. In a second step, the system incorporates a refinement of the mask that considers reflectivity (a high value in channel 2). The implementation is made by means of two ANNs with a back propagation structure. The first one is used to derive a candidate pixel map and the second one will make the final selection from this map. Results of this phase are shown in Fig. 2, where we can see a cloudy AVHRR scene of the Canary Islands region taken by NOAA-14 satellite and the resulting mask that, when imposed onto the original image, prevents further processing of pixels labeled as “cloud”.
Fig. 2. AVHRR scene and cloud mask 3.2 Segmentation Segmentation is a key task for any automatic image processing system. The final results will be good only if high quality segmentation is achieved. But, as explained in [5] and [7], AVHRR images of the ocean are very difficult to segment due to a high variability of the pixel values, yielding poor results with conventional techniques relying merely on gradients or textures. The method proposed in [5] uses isothermal lines that are proven to produce better segmentations in this kind or images. This, like any other segmentation based on thresholding, has the inherent drawback of threshold selection, but we have solved it applying the knowledge obtained in the next phase creating an iterative procedure. The initial threshold can be set to a fixed empiric value or to the mean of the water pixels. With it, the segmentation is performed and results are given to the next phase that will analyze every region. Then, based on the knowledge about the regions we are looking for, the threshold level can be incremented or decremented and a segmentation with a new threshold level asked for. Results are then compared to previous ones and the system determines if the change went in the right direction or it must be done in the opposite one or
settled in the preceding value and end the task. A segmentation is considered to be better than the previous one if the regions that match with a class of interest have a larger area. This method yields good segmentations in AVHRR images, producing compact regions and reducing the oversegmentation obtained with other method tested (watersheds [2] and Canny [3]). An example is shown in Fig. 3.
In order to train this component, every ocean feature of interest has to be defined by a human expert using some numerical or symbolic descriptors. Each piece of this information may refer to a region by itself (like “size” or “temperature”) or in relation to some others regions (like “colder than…”). Usually, the knowledge is based on common well known characteristics but they may appear more elaborated ones like moments or variance. Also, human experts use to refer to symbolic values like “big”, “small” that will have to be compared to values calculated numerically from pixel values, so the relationship between them has to be defined using intervals. This knowledge is translated to production rules (Fig. 4 and Fig. 5) that conform the reasoning nucleus.
Fig. 3. AVHRR scene segmentation 3.3 Symbolic Processing Element The basic element of the high-level information classifier is a Symbolic Processing Element (SPE) that takes the basis operational principles from the formal model of the artificial neuron. In the SPE the activation and output functions are implemented by mean of decision rules, which generate outputs with dependence of the feature vector in the inputs [9].
Fig. 4. Example of production rule (graphic view)
The high-level information classifier is composed of a competitive structure of SPE which are trained using symbolic information taken of well known examples of several mesoscalar oceanic structures, upwellings at the Sahara coast, several kind of warm and cold eddies located near Canary Islands and warm wakes at the S of them. With this knowledge, the interpretation system is capable to identify similar structures but in different areas like on the Mediterranean Sea or Cantabric and Portuguese coasts with quite acceptable accuracy. 3.4 Graphic Expert System Also, a Graphic Expert System (GES) has been designed and tested with the same image data set as the SPEs. This system works like an oceanographer, collecting the ocean knowledge and representing it as production rules. The system also filter the ocean information that arrives to the GES in order to obtain only the relevant information for the ocean structures recognition present in the image.
Fig. 5. Example of production rule (text view) The hard work here is to figure the imprecise, sometimes intuitive, deduction scheme that human experts apply to perform the task. Also, the inexistence of an exact model for each feature of interest in our problem can lead to gaps or inconsistencies in the knowledge that the expert provides to the system. This is a general issue for expert systems so some techniques have arisen to manage this kind of imprecise or incomplete knowledge (bayesian networks [4] or fuzzy methods [6]). In our system the answer adopted is to consider that hypothesis are not absolutely proved when conditions of any of its rules are true. Instead, we add some verisimilitude to it and accumulate it through
every step of the iterative segmentation-recognition cycle. In production time, the system begins calculating the numeric characteristics in which expert knowledge relies on. It is done for every region of the current segmentation. Then, production rules are evaluated to identify if there is any region belonging to a class of interest, in which case it is labeled with the class identifier. As seen before, at this time the knowledge driven subsystem may ask for a new segmentation, modifying the threshold level, or decide to consider the process finished. The results of this component is just a list or database that connects region identifiers with the identifier of the class they belong to. The most frequent class is the “not interesting” one, which denotes than this region is part of the sea zone but doesn’t belong to any ocean feature of interest. This class assignment may be omitted to keep the results small and human readable. 4. RESULTANT PRODUCT The information provided by knowledge driven processing units (SPE and GES) is refurbished with the original segmentation to create images (Fig. 6) that show in a visual way each feature of interest recognized. Each feature is represented by different colored region in a map where each color represents one of the ocean phenomena we are looking for.
Fig. 7. Original AVHRR scene with detected features 5. RESULTS Results from systems, SPEs and GES, depend mainly on the quality of the images. In the case of SPE subsystem, also a good number of training images is needed. For GES, the best results are achieved when the human expert provides the system with specific knowledge about the target area. When this requisites are met, both systems produce positive ocean structures recognition between the 80% and 95%. The developed system is still a prototype that mixes several programming environments. It’s built using C++ programs, perl ant Matlab scripts and Nexpert Objects knowledge bases. This isn’t the ideal scenario for speed purposes and though the timing measures aren’t as fast as they could be. The whole process of an image, included the iterative segmentation-recognition cycle using a 1GHz PC takes about one hour. 6. REFERENCES 1. Arístegui J., Sangrá P., Hernández-León S., Cantón M., Hernández-Guerra A. and Kerling J.L., Islandinduced eddies in the Canary Islands. Deep-Sea Research, 41 (10): 1509-1525, 1994.
Fig. 6. AVHRR scene (equalized) and feature map As a last step, the feature map can be applied to original image (Fig. 7). This is done using a color code to identify the class of each element of the mask but here, in order to get a better printable image, we have used a black mask for all the classes.
2. Beucher S., Segmentation Tools in Mathematical Morphology, SPIE-1350 Image Algebra and Morphological Image Processing, 70-84, 1990. 3. Canny J., Finding Edges and Lines in Images, Massachusetts Institute of Technology, 1983 4. Dubois, D.; Prade, H.; Smets, P., Representing partial ignorance, Systems, Man and Cybernetics, Part A, IEEE Transactions on, Vol.26, Iss.3, May 1996, Pages:361-377 5. Guindos Rojas F., Torres Arriaza J.A., Peralta López M and Cantón Garbín M., Segmentación iterativa basada en conocimiento del afloramiento de aguas frías
en la costa sahariana, Teledetección, Medio Ambiente y Cambio Global, 591-594, 2001. 6. Intan, R.; Mukaidono, M., Approximate reasoning in knowledge-based fuzzy sets, Fuzzy Information Processing Society, Proceedings. NAFIPS. 2002 Annual Meeting of the North American 439-444, 2002 7. Thonet H., Lemonnier B. and Delmas R. Automatic segmentation of oceanic eddies on AVHRR thermal infrared sea surface images, Challenges of Our Changing Global Env. Conference Proceedings, OCEANS ’95, vol. 2, 1122-1127, 1995. 8. Torres Arriaza J.A., Guindos Rojas F., Peralta López M. and Cantón M., An Automatic Cloud-Masking System Using Backpro Neural Nets for AVHRR Scenes, IEEE Transactions On Geoscience and Remote Sensing, vol. 41, no. 4, 826-831, 2003. 9. Torres Arriaza, J.A.; Guindos Rojas, F.; López Peralta, M.; Canton Garbín, M. Competitive neural-netbased system for the automatic detection of oceanic mesoscalar structures on AVHRR scenes. Geoscience and Remote Sensing, IEEE Transactions on, Vol.41, Iss.4, April 2003Pages: 845- 852
