Comparing several AFE tools in the context of ships and vehicles detection based on RGB and EO data François Leduc Optronic Surveillance Section DRDC Valcartier Quebec, Canada
[email protected] Abstract – In this paper we present a comparison of three AFE tools used in the context of ship and vehicle detection based on high resolution data. The three tools (Genie Pro – Los Alamos National Laboratory, Feature Analyst – Visual Learning Systems and eCognition – Definiens AG) were chosen because they were defined as promising and were to be analyzed by NGA in the framework of the STAR program. The comparison is presented here in terms of detection and false alarm rates and also in terms of pros and cons of each tool. Keywords: ATD/ATR, AFE, target detection, Feature Analyst, Genie Pro, Definiens, eCognition.
1
Introduction
Assisted Feature Extraction (AFE) is typically a three-step process involving samples selection, training, and extraction in order to label image objects with real-world attributes. AFE allows an image analyst to identify a few objects of interest (samples) on one image and let the computer analyze the entire image. Such a process is only possible after the AFE tool performed some learning (training) from the samples. The ultimate goal of such an approach is to obtain a learned process that could be applied on several images so that a new image could be analyzed without being displayed. The selection of the three tools Genie Pro, Feature Analyst and Definiens (formerly named eCognition) was justified by the fact that they are part of the four tools being retained for detailed analysis by NGA in the framework of the STAR program [1]. Genie Pro and Feature Analyst follow the three-step process (sample, training, extraction), while Definiens differs from them by being an object-oriented rule-based tool. The interest in the study consists of identifying which tool performs better in terms of accuracy assessment and in terms of processing time regarding ships and vehicles detection based on high resolution imagery. This paper is structured as follows. Section 2 gives a brief description of the three tools while section 3 describes the
Daniel A. Lavigne Optronic Surveillance Section DRDC Valcartier Quebec, Canada
[email protected] data set and the preprocessing steps. Section 4 concerns the experiments and the results and section 5 describes the comparison of the three tools. Finally, section 6 concludes this paper.
2 2.1
Tools description Genie Pro
Genie Pro is a pixel-based evolutionary computation tool developed by the ISIS group at Los Alamos National Laboratory [2]. It employs user-defined training inputs to derive automatic pixel classification algorithms for remotely sensed imagery by assigning labels to pixels. From a collection of image operators, a chain of image processing tools is assembled using genetic algorithms. Genie Pro integrates spectral information and spatial clues such as local morphology, texture, and large-scale shape information in a sophisticated way. Designed at first for detecting complex spatio-spectral terrain features in multispectral imagery, it is used primarily for deriving vector overlays and semantically meaningful maps. Genie Pro version 1.2.2 has been used in this research.
2.2
Feature Analyst
Feature Analyst is a machine learning tool developed by Visual Learning Systems Inc. [3] to extract geospatial features in remotely sensed imagery. Its classification scheme incorporates a contextual classifier set by the user, accordingly to the spatial distribution of the image features to be extracted. Feature Analyst uses temporal, spatial, spectral, and ancillary information (such as shape, size, pattern, color, shadow, texture, and spatial association) to model the feature extraction process. As a pixel-based AFE tool, it employs small and simple sets of user-defined training examples within a hierarchical learning procedure, embedded to iteratively improve the classification results. Feature Analyst version 4.1 has been used during the scope of this research.
2.3
Definiens (eCognition)
Definiens (formerly named eCognition) is an objectoriented rule-based AFE tool developed by Definiens AG [4]. Definiens includes segmentation algorithms that allow extracting homogeneous regions thus permitting to process polygons instead of individual pixels. Polygons or regions can be described by several features such as radiometry, texture, shape, size, difference to neighbors, etc. Regions can be split into sub-regions or grouped into larger ones thus allowing multiscale analysis. Working with Definiens implies writing or developing rule sets which consists of selecting processes in the Definiens library. Once a process is selected it has to be tuned properly. Generally, a rule set begins with a segmentation process followed by some feature extraction and refinement. Processes can be iterative, new segmented levels can be created anywhere in the rule set and levels can be deleted. Parallel to the rule set development there is the class hierarchy definition. Finally, with Definiens there are several paths that can be used to arrive at a same point. A good introduction to Definiens can be found in [5] although it is concerned with eCognition 4 and that the actual work is done with Definiens Developer 5.
3
Figure 1 – RGB imagery.
Data set
In order to test the different tools, one colour image of Quebec City was selected (Figure 1). This digital airborne data imagery was obtained by scanning a photo negative with a resolution of 1815 dots per inch. As the paper photo was characterized with a 1:3000 scale, this operation produced a spatial resolution of 4.2 cm. Finally, the image was downsampled to a spatial resolution of 32 cm in order to decrease the required processing time. Part of the image seen at full resolution is presented in Figure 2. The panchromatic band was derived from the three RGB bands by computing a principal component analysis and by preserving the first component.
Figure 2 – RGB data enlarged at full resolution.
Ten Regions of interest (ROIs) were digitized with ArcGIS (Figure 3). On these ten ROIs, the largest one (cyan) corresponds to the water body and the nine others correspond to parking lots. The left-most parking (red) has the particularity of containing ships while other parkings (yellow) contain only ground vehicles. These ROIs contain 909 targets distributed as follows: 124 ships in water, 70 ships on the parking lot and 715 ground vehicles. All these targets were digitized with ArcGIS and used in the validation process. Vector ground truth data was converted into raster format using the Spatial Analyst ArcGIS module. The raster ground truth data was filtered with a 3x3 morphological erosion filter in order to eliminate ambiguities due to digitizing errors
Figure 3 – Regions of interests (ROIs).
4
Experiments
For each result produced by the three tools, an accuracy assessment is computed based on the use of confusion matrices [6] from which three metrics are extracted:
producer’s and user’s accuracies (PA, UA) and false alarm (FA) rates. The producer’s accuracy, for a real class, is the percentage of correctly classified pixels while the user’s accuracy, for a labelled class, is the ratio of correctly classified pixels to all pixels assigned to that class. Finally, the false alarm rate, for a real class, is the ratio of all pixels assigned to a class on the pixels really belonging to that class. Note that all statistics computed in this study are pixel-based.
4.1
Genie Pro
Genie Pro provides a clever process that is relatively straightforward. By using an evolutionary algorithm to explore potential attribute extractors in the neighborhood of local pixels, Genie Pro uses manually user-defined training samples on specific regions of interest, and derives pixel classification algorithms by assigning labels to pixels [2]. A four-class classification scheme has been used to segment the ships, vehicles, land, and water pixels. Figure 4 illustrates an example of some initialization of three-class training samples using Genie Pro on three ROIs.
Figure 5 – Classification results using Genie Pro on RGB data. Table 1. Producer’s accuracy, user’s accuracy and false alarm rates obtained with Genie Pro using the RGB data set. (%) ship vehicle land water 92,6 65,8 81,5 96,7 P.A. 64,4 53,3 87,0 98,5 U.A. 51,5 57,6 12,2 1,5 F.A. Table 2. Producer’s accuracy, user’s accuracy and false alarm rates obtained with Genie Pro using the panchromatic band. (%) ship vehicle land water 82,8 55,5 77,4 96,3 P.A. 49,2 42,0 86,8 98,6 U.A. 85,5 76,8 11,8 1,4 F.A.
Figure 4 – Training samples using three ROIs. Ship pixels are manually edited in red, vehicle pixels in yellow, while water and land pixels were identified in blue and green colors respectively. Figure 5 shows the resulting four-class image classification results for RGB data by letting Genie Pro run for about 20 minutes. Table 1 and Table 2 give some accuracy assessment measures obtained with the RGB data and the panchromatic band. Note that although Genie Pro classified all the image data, only the pixels contained in the ten ROIs of Figure 3 are used for accuracy assessment. Classification errors outside these ROIs are not taken into acount.
The percentage of correctly detected ships (92,6%) and vehicles (65,8%) in RGB data is greater than the one obtained with the panchromatic band (ships - 82,8% and vehicles - 55,5%). The false alarm rates were also better in the cases of ships (51,5%) and vehicles (57,6%) with the RGB data than with the panchromatic band (85,5% and 76,8% respectively). As with other AFE tools with machine learning algorithms, it is not easy to find an “optimal” solution for the extraction of features for a given imagery. Indeed, trying to decrease the false alarm rate is usually achieved at the price of missing some desired features to extract. Thus, there is a tradeoff between the accuracy of a given solution and the detection rate that can be achieved for some specific image features. Furthermore, the extraction of ship and vehicle pixels from land and water areas (i.e.
via a four-class classification scheme) generated numerous misclassifications (as illustrated in Figure 5, where many vehicles were incorrectly classified as ships in the parking areas). Thus, the discrimination between some classes can be somehow problematic, increasing the false alarm rates and decreasing the detection results. Finally, with Genie Pro, it is not trivial to know when the training process has reached an optimum in order to produce the best classification results. In this research, several experiments were computed with different running times and the presented results are the ones that generated the best classification results.
4.2
Feature Analyst
Feature Analyst is also a pixel-based AFE tool like Genie Pro but it uses a hierarchical learning scheme to iteratively improve the extraction results. Using four classes representing the main events of interest to be detected (i.e. ship, vehicle, land and water), pixels are classified using a hierarchical learning process. Figure 6 illustrates the classification results for RGB data. A post-processing was performed which consists in the aggregation of areas smaller than 5 pixels. As with Genie Pro, only the pixels contained in the ROIs were used for accuracy assessment. The total time required to process the image was about 5 minutes.
89.2% of the ships were extracted with the RGB data while only 79.4% of the ships were detected with the panchromatic band. The main difference between the RGB data and the panchromatic band resides in the high false alarm rate obtained with the panchromatic band for both ships and vehicles. Table 3. Producer’s accuracy, user’s accuracy and false alarm rates obtained with Feature Analyst using RGB data set. (%) ship vehicle land water 89,2 52,4 81,0 94,8 P.A. 53,8 30,4 95,7 97,9 U.A. 76,7 120,2 3,7 2,0 F.A. Table 4. Producer’s accuracy, user ’s accuracy and false alarm rates obtained with Feature Analyst using the panchromatic band. (%) ship vehicle land water 79,4 40,9 64,9 94,1 P.A. 35,0 21,5 88,8 98,1 U.A. 147,7 149,9 8,2 1,8 F.A. The most difficult task in this classification problem is the discrimination between ship and vehicle image features. As this can be observed with Genie Pro (Figure 5) and Feature Analyst (Figure 6), it might be an indication of the limitations of such pixel-based AFE tools to achieve interesting target detection capabilities using such high resolution image data.
4.3
Figure 6 – Classification results using Feature Analyst on RGB data. Even though Feature Analyst is a pixel-based AFE tool, it also takes into account the spatial and the multispectral information content of the image features to be extracted. As such, the ships and vehicles are better extracted with the RGB data set than with the single panchromatic band. Table 3 and Table 4 give some accuracy assessment measures obtained with the RGB and panchromatic data.
Definiens Developer
The Definiens rule set development for vehicles and ships detection required about 8 hours of work. This includes the identification of the appropriate segmentation method, the selection of appropriate descriptors and processes and the computation of other features used as input data. One of the main differences between Definiens and the two other tools is that Genie Pro and Feature Analyst use internally computed features, while with Definiens the user has to select manually the adequate features. Although it is possible to select features among the internal library, it is also possible to compute features and to use them in the input data set. Hence, the input data set was composed of eight features as follows: 1-2-3-4: RGB bands plus the panchromatic one 5-6: texture (occurrence) entropy and variance 7: texture (co-occurrence) homogeneity 8: Sobel gradient Features 5-8 are pixel-based computed using the panchromatic band and more specifically features 5-7 are computed using a 7x7 kernel. The rule set for vehicle and ship detection was developed with Definiens Developer 5 and contains six main parts. The first part is concerned with the image segmentation
allowing recreating the 10 regions of interest and the second part is concerned with water detection. For this, the water body polygon is segmented with the chessboard algorithm using squares of 3x3 pixels. Water is then defined as being dark in the panchromatic band and characterized with low variance (Definiens internal standard deviation). The third part is the detection of ships being characterized by high values in the three RGB bands. The fourth part is concerned with ship detection being parked on asphalt in the left parking lot. For this, fuzzy classification is performed by considering ships and land. Ships are characterized with high entropy and variance values (occurrence texture) while land is characterized with low variance and medium entropy. Finally, the fifth part concerns ground vehicle detection that is characterized with high entropy value and medium Sobel gradient value. The homogeneity feature is used to detect an intermediate class. The sixth part is the classification refinement, which is mainly the elimination of intermediate classes leaving the result presented in Figure 7. Some accuracy assessment measures are given in Table 5.
Table 5. Producer’s accuracy, user’s accuracy and false alarm rates obtained with Definiens using the RGB data set. (%) ship vehicle land water 96,6 76,1 81,3 99,3 P.A. 73,4 62,9 96,8 99,9 U.A. 35,0 44,9 2,7 0,1 F.A.
Table 6. Producer’s accuracy, user’s accuracy and false alarm rates obtained with Definiens Developer using the panchromatic band data set. (%) ship vehicle land water 98,1 46,7 86,7 99,3 P.A. 70,5 65,4 91,1 99,9 U.A. 41,1 24,7 8,5 0,1 F.A.
Figure 8 - Target detection results obtained with Definiens Developer using the panchromatic data set.
Figure 7 - Target detection results obtained with Definiens Developer using RGB data. In the context of surveillance, it may happen that only the panchromatic band is available, so the rule set has been modified and the panchromatic band replaces the three RGB bands in the descriptors. Hence, the panchromatic band data set contains five features (features 4-8 described above). The classification result obtained with the panchromatic data set is presented in Figure 8 and the accuracy assessment measures are given in Table 6.
The differences in the results obtained with the RGB and the panchromatic data sets are various. With the RGB data set, 76% of the vehicles are detected with 45% of false alarms while with the panchromatic band, only 47% of the vehicles are detected with a false alarm rate decreased to 25%. Concerning the ships, both data sets produced relatively similar results with high detection rates and relatively low false alarm rates. This similarity of the results between both data sets can be explained by the fact that ships are generally bright and contrasting objects. As it is quite easy to discriminate water from ships, 88% of the false alarms for ships come from the land class in the case of
RGB data set. In the case of the panchromatic data set, land contributes to 73% of the ship false alarms while water contributes to 23%. If we compare ships and vehicles, the lower rate of vehicle detection can be explained by a lack of discrimination between this class and land. This lack of discrimination is caused by the great diversity of vehicles and by the land not being homogeneous.
5
Discussion
In terms of accuracy assessment, we concentrate the results’ analysis on ship and vehicle detections as they compose the classes of interest. In terms of producer’s accuracy (Figure 9, Figure 11), Definiens performed better than the two other tools, except for vehicle detection based on the panchromatic band where Genie Pro performed better (Figure 11). Feature Analyst arrives behind the two other AFE tools for ship and vehicle detection, in both RGB and panchromatic data. However, it is worthwhile to note that Feature Analyst’s producer’s accuracy values were slightly lower than the ones obtained from Genie Pro and Definiens, except in the case of vehicle detection based on RGB data. Regarding the false alarm rates (Figure 10, Figure 12), the best performance is given by Definiens which does not produce FA rates greater than 45%. The worst performance is given by Feature Analyst with a FA rate reaching 120% and 150% for vehicles detection based on the RGB data and on the panchromatic band, respectively. Between both tools comes Genie Pro.
Figure 9– Comparing producer’s accuracy for ships and vehicles obtained with the RGB data set.
Figure 10 – Comparing false alarm rate for ships and vehicles obtained with the RGB data set.
Figure 11 – Comparing producer’s accuracy for ships and vehicles obtained with the panchromatic band.
Figure 12 – Comparing false alarm rate for ships and vehicles obtained with the panchromatic band. By combining the false alarm rates and the producer accuracy, we can rank the three tools in order of decreasing performance as follow: 1 – Definiens, 2 – Genie Pro and 3 – Feature Analyst. This ranking is also confirmed with the user’s accuracy measures.
Besides this comparison, some other criteria need to be taken into account. Genie Pro and Feature Analyst are much easier to use by loading imagery, selecting samples, training the tool, and classifying the data. On the other hand, these tools use internal features and as such are black boxes to the user so that results can hardly be interpreted. It is also difficult with these tools to export knowledge hence every time a new image is loaded, new samples are drawn and a new training process is computed. Definiens differs from these two tools as the user controls all the processes to be performed. One inconvenience with this fact is that in the present comparison of the three tools, the evaluation of Definiens is also the evaluation of user’s ability to use Definiens while the evaluation of Genie Pro and Feature Analyst is really the evaluation of the tools. In other words, the performance of Definiens is greatly correlated to the user comprehension of the tool and to user expertise in selecting and tuning appropriate features. Using Definiens required approximately 8 hours of work compared to the 5 – 20 minutes for the two other tools. Although it may appear much too long, the advantage of this tool is that the rule set is exportable and applicable to new imagery provided similar radiometric and geometric conditions. This comparison of required time from data ingestion to results production is independent of the learning curve although we can estimate the learning time for Genie Pro to 4 hours and the learning time for Feature Analyst to 8 hours. Learning Definiens requires about 40 hours. These learning times are given as crude indications for being able to properly use the tools, not necessarily to become an expert with them. Another major advantage of Definiens over the two other tools is that it is possible to control which areas are searched for a given object. In this study only the eight parking lots in yellow (Figure 3) are searched for vehicles, while ships on the ground are only searched in the parking lot in red. This kind of control avoids detecting water on the parking lots and detecting vehicles in the water. An interesting aspect to note is that the parking lot in red contains a few vehicles that have all been identified as ships. This shows that spectral and textural features alone are not sufficient to discriminate ships from vehicles.
6
Conclusion
In this study we compared three AFE tools, Genie Pro, Feature Analyst and Definiens, applied to vehicle and ship detection based on RGB and panchromatic data. In terms of producer’s accuracy (good detection rate) and false alarm rate, Definiens was identified as the best tool followed by Genie Pro. Feature Analyst arrives third in
this competition by producing too many false alarms. Although this study might seem incomplete by analyzing a single scene, it corroborates some results obtained internally within our research group. Although Definiens performed better, it required much more time to process the scene. It also requires a longer learning curve, but on the long-term, it becomes more powerful as knowledge is exportable. It was also mentioned that the evaluation of Definiens’ performance was also the evaluation of the user performance. In this case, a more skilled user could also mean even better results...
References [1] M.A. O’Brien and J.M. Irvine, Information Fusion for Feature Extraction and the Development of Geospatial Information, Fusion 2004, June 28 – July 1, Stockholm, Sweden, 2004. [2] S. Perkins et al., Genie Pro: Robust image classification using shape, texture and spectral information. Proceedings SPIE Algorithms and Technologies for Multispectral, Hyperspectral, and Ultraspectral Imagery XI, Vol. 5806, pp. 139-148, 2005. [3] M.A. O’Brien, Feature Extraction with the VLS Feature Analyst system, ASPRS 2003 Annual Conference, Anchorage, Alaska, May 5-9, 2003. [4] U.C. Benz, et al., Multi-resolution, object-oriented fuzzy analysis of remote sensing data for GIS-ready information, ISPRS Journal of Photogrammetry & Remote Sensing, Vol. 58, pp. 239-258, 2004. [5] K. Navulur, Multispectral Image Analysis Using the Object-Oriented Paradigm, CRC Press, New York, 165 p., 2006. [6] T.M. Lillesand and R.W. Kiefer, Remote Sensing and Image Interpretation, John Wiley and Sons, 396 p., 1979.
