evaluated to filter/enhance SAR Digital Elevation. Models (DEMs). ... by using a segmentation algorithm on the image data. ..... the next paragraph. Edition.
SYNERGY OF IMAGE AND DIGITAL ELEVATION MODELS (DEMS) INFORMATION FOR VIRTUAL REALITY C. Maire and M. Datcu DLR German Aerospace Center, D-82230 Wessling , Germany Tel / Fax number: +49 81 53 28 - 1388 / 1444 e-mail: {cyrille.maire, mihai.datcu }@dlr.de
ABSTRACT In the framework of 3D visualization and real-time rendering of large remote sensing image databases, several signal processing techniques are presented and evaluated to filter/enhance SAR Digital Elevation Models (DEMs). Through the SRTM DEM, the interest of InSAR data for such applications is illustrated. A non stationary bayesian filter is presented to remove noise and small artefacts which pervade the SAR DEM while preserving structures and information content. Results obtained are very good, nevertheless large artefacts cannot be filtered and some artefacts remain. Therefore, image information have to be inserted to produce more realistic views. This second step is done by using a segmentation algorithm on the image data. By a topology analysis, the extracted objects are classified/stored in a tree structure to describe the topologic relations between the objects and reflect their interdependencies. An interactive learning procedure is done through a Graphical User Interface to link the signal classes to the semantic ones, i.e. to include human knowledge in the system. The selected information in form of objets are merged/fused in the DEM by assigning regularisation constraints.
1. INTRODUCTION With the development of remote sensing (RS) technology, (e.g. high-resolution images, higher acquisition frequencies and new sensor types), the scope of RS Images allows users to perform a large variety of specific applications. For instance, land cover mapping is realized thanks to increased levels of information and synergy between data sets.
Digital Elevation Model (DEM) provides a basic spatial reference system, and images/vector data can be draped over the DEM for more advanced analyses. The “naturalness” of 3D representation enhances our ability to interpret/exploit two-dimensional imagery. For example, the representation of the third dimension provides important information about the relations between land coverages: shapes, structures and slopes appear to describe scene objects such as waterways, surface material and vegetative growth. Fig 1 illustrates the synergy achieved by 3D representations: Whereas the analysis of the 3 left images requires knowledge in RS data, the 3D view is intuitively/directly understandable. Therefore, information in form of geo-referenced DEMs & Images should be used for 3D real-time visualisation over large areas (country scale) while preserving high resolution data/content and realistic rendering. Up to now, the exploitation/integration of large and complex databases integration in the overall interactive 3D visualisation process is extremely difficult. However, the demand for such technologies has exploded in the last years and leads to a reinforcement of world acquisition programs (SRTM (Shuttle Radar Topography Mission), largest homogenous coverage). Therefore, RS and Virtual Reality communities are faced to several challenges to deal in real-time with such an amount of data: e.g. Landsat images mosaic/ERS Tandem DEM over the whole Germany (gridding: 25m) ⇔ (~40000² pixels) × (R-G-B channels + elevation information). In [1] an efficient rendering algorithm is proposed.
Fig 1. Data content: left to right: DEM, indexed DEM, image, synergy DEM-image
In addition to the rendering time constraints, realistic visualisations require the enhancement/regularisation of the database. In this purpose, an important level of understanding and content extraction has to be reached to perform significant improvements in the data and particularly in the DEMs which bring the geometry information. Nevertheless, despite high accuracies, EO DEM are still pervaded with errors and artefacts mainly due to the acquisition/generation techniques. As a consequence, the elevation data have to be analysed, filtered and enhanced. These pre-processing steps are determinant in order to cope the artefacts, to generate a higher level of realism and to simplify the data for 3D rendering processes (meshes simplification, hierarchical decomposition...). Since world DEM coverages are available and generated by InSAR (Interferometric Synthetic Aperture Radar) processes, InSAR DEM characterisctics are presented in section 2. In section3, a database preparation line for Virtual Reality scenarios is dressed, the algorithms which compose the system are presented. In section 4, a DEM regularization process is proposed. Perspectives and conclusion are sketched in section 5.
2. INSAR DEM CHARACTERISTICS DEM information is now recognised as one of the most important data structures used for geo-spatial analysis and 3D rendering. DTMs (Digital Terrain Model) and DEMs can be considered as “2.5” dimensional representation, i.e. in the nature of 2-D but contain 3D information. The formerly most used method to produce a DTM was the interpolation of topographic digitised level curves. Interpolation artefacts and smooth behaviour are introduced
during the data sampling. Moreover, the resulting information is different from RS DEM. It contains only the terrain elevation information, while forest, urban elevations are not included. Indeed, for 3D visualization in urban areas, such data are not efficient enough to reach the high level of realism required by users. Fig. 2. illustrates SRTM InSAR DEM data, generated by using an X-band single pass sensor on board of the space shuttle. It represents the largest homogenous DEM database. 1.
InSAR elevation content
InSAR processing enables to compute DEMs in a different way than the traditional one and to include at least partially some objects (forest, buildings…see Fig 2 and Fig 4) according to the acquisition process (resolution, sensor frequencies). Therefore, we can classify the DEMs according to their resolution/scale: Low resolution 30-100m: Elevation and Terrain models are not too different. The height of scene objects is not observable from the terrain surface. Medium resolution 5-30m: DEM contains partial height information of large objects (buildings, forest...). However, additive processes are needed for realistic visualization of the 3D geometry of objects. High resolution 1-5m: Scene objects are in 3D representation. Nevertheless, enhancements are required for visualization of details. Occlusions and shadows limit the data utilization. Very high resolution centi/deci-metric: Satellite or airborne data have to be complemented with ground images, synthetic 3D objects. The complexity can be very high, depending on the scene, mainly for human environments.
SRTM DEM: Information content: Forest Bridges Large buildings Valleys Amiens area (Fr) DEM, scale 1/200000
Amiens area; SRTM DEM, scale 1/50000
Artefacts: Noise Unwrapping Phase Geometry acquisition Specular reflexion, i.e. Lake, charc. Low SNR DEM
Nice area (Fr) SRTM DEM / SPOT5 image
Fig 2. InSAR SRTM DEM content
2.
InSAR DEM artefacts
The combination of acquisition and relief infers in SAR information occurrences, geometric displacements and artefacts characterised by an excessive texture roughness (too high fractal dimension and local variance). An essential issue is to remove the noises (thermal noise and low coherence noise) which pervade InSAR DEM. Moreover, land-covers have a direct influence; for instance specular reflections on flat surfaces such as rivers or lakes result in artefacts in the InSAR unwrapping phase (PU) (Fig 2). These artefacts have to be excluded for a better 3D rendering.
method and an object extraction algorithm. The extracted objects are classified/stored in a tree structure to preserve topology relations between the objects and therefore reflect their dependencies. An interactive learning procedure is done through a Graphical User Interface in order to link the signal classes to the semantic ones, e.g. to include human knowledge in the system. The selected information in form of objects are merged with the filtered DEM by assigning regularisation elevation constraints. A least Square Problem is formulated to optimize the adjustment. The following section describes more precisely the different algorithms of the system. 2.
3. SYSTEM ARCHITECTURE 1.
System flowchart
A modular approach has been choosen to build independent algorithms based on several signal processing techniques and information extraction algorithms: •
Non-stationary approaches (Bayesian approach, Gauss Markov Random Fields(GMRF))
•
Segmentation algorithms
•
Object Extraction /Topology analysis
• Merging/Data fusion This architecture (Fig 3), will allow us to improve the processing chain just by updating a part of the system. 3D real time rendering
3D rend. engine
Database preparation
Images
Enhanced DEM Texture Image
DEM/DTM
Feature extraction
Merging
DEM Filter
Fig 3. Flowchart of the system
The principle of the DEM regularization (Merging) is the following: Based on the DEM data: A DEM filter is used to reduce noise and artefacts while preserving contours/objects included in the data. Based on the image data: To extract relevant information, two algorithms are presented: a segmentation algorithm based on region growing
DEM filter
A bayesian filter has been developed to deal with nonstationary data such as InSAR DEM [2, 3, 4, 5]. It attempts to remove the noise inherent to the InSAR DEM data while preserving structural information and a high level of details included. Assuming the noise and SAR model formation, the process of filtering is considered as an ill-posed inverse problem, and formulated in the general frame of Bayesian inference in two levels: •
•
model fitting: To filter the data: Maximum A Posteriori (MAP) method with a Gaussian distribution for the likelihood and Gauss Markov Random Field (GMRF) models employed as prior.
model selection: Evidence maximization computed for a library of small models. For analytical tractability, we use a library of small dimensional models and perform a model selection (second level of inference) in order to get the model that best describes the data through the calculation of the evidence. Since texture representation by GMRF models leads to poor results when sharp edges are occurred, a region growing algorithm is computed to separate homogenous regions. The strong scatterers are detected by the prior model during the filtering and excluded. After the filtering has been made, the characteristic features extracted in the previous steps are reinserted in the DEM. Fig 4 illustrates the non-stationary filtering performed on the raw SRTM DEM. The noise restoration is efficient due to the complete and local noise modelling. The global aspect of the DEM is more properly realistic. A comparison with other filters (Gamma Map, Frost, Lee, Wavelet based..) is presented in [5] and illustrates the quality of filtering. In spite of the good results obtained in terms of statistical analyses and rendering aspects, the filtering is still not sufficient to cope with important artefacts (specular reflexion, PU). Indeed, complementary information have to be added to reach very realistic flight simulations.
Fig 4 DEM filtering
3.
Segmentation algorithm
Among the various segmentation methods, ranging from bayesian to morphological methods, a recursive region growing algorithm is used, based on the Mumford and Shah criterion [6] (minimisation of the simplest case of the Mumford-Shah functional). Starting from an initial grid where each pixel is considered as a region, a merging cost is computed for each boundary between two adjacent regions. All these results are stored in a priority queue. The regions which
correspond to the boundary with the smallest merging cost are merged. The priority queue is updated for the new merged region. Iteratively, regions are merged. In our implementation, only one parameter is needed: that one to stop the merging process (number of region to reach). Fig 5 shows the results obtained for a segmentation on a SPOT5 image [tile 1024x1024 pixels] with 1500 regions.
Fig 5. Segmentation algorithm: first image: raw image to be segmented [Enhanced SPOT5 image, 2.5 meter resoltion]. second image: restored image obtained for 1500 regions
Results obtained are promising. Nevertheless, some limitations still remain: In fact, the algorithm depends on the intensity scale. Indeed, the merging will be stronger in low intensity areas than in the high ones. In addition, a final number of regions has to be set to stop the merging process. In our purpose, we prefer to get an under-segmented result, since the possibility to merge regions is possible after the topology analysis process. 4.
Topology extraction
The segmentation algorithm generates a partition Ω(m,n) of the image in a defined number of regions (m,n denote the image dimensions). Nevertheless, the regions extracted are still expressed in a pixel level. In order to deal with the regions expressed as objects, their respective geometry (form, shape) and topology (neighbourhood relations, inclusions…) should be encoded. To achieve topology description, various discrete spatial models were proposed [7, 8, 9]. Among them, the complex cellular approach [7] was choosen since it turns out the connectivity paradox and topological problems in Z² (i.e. with discrete grid). To achieve it, the plane (R²) is decomposed in regular complex cells which replace the pixel model (Z²). A discrete model is generated, which allows to preserve the topology of R² in a similar way as manipulating vectors models. The plane is decomposed in Nodes N (0-cells), Segments S (1 cells) and Regions R (2 cells). These definitions are illustrated in Fig 6b. Several rules are stated, the following system is sketched and follows these axioms with (i, j, k, l) ∈ Z4: Topology: 4 connectivities neighbourhood Nodes: Definition: N ∈ Z² (Ω(m+1, n+1)). N should be the extremity of at least 1 Segment (closed Segment) or maximum 4. Segment: (with i ≠ j) Definition: A segment S is the boundary (and is delimited at each extremity by a starting Node Nbegin and an ending Node Nend) between two adjacent Regions Rright, Rleft. Nbegin, Nend can be the same Node. S is noted S Rright , Rleft , N begin , N end . Unicity: Only one Segment is defined by two adjacent Regions Ri, Rj and two Nodes Nk, Nl. SRi , R j , Nk , Nl and SR j , Ri , Nl , Nk are equivalents.
Intersection: Si ∩ Sj = ∅ || ({Nk}, {Nl},{ Nk, Nl}) Border: (with i ≠ j) Definition: A border Bi is a closed chain of connected Segments. The chain delimitates one Region Ri in Ω and its neighbourhood regions. The direction of the chain is the clockwise turn. For a given Ri Region, Bi denotes its external closed contour: Bi : [∪j S Ri, Rj, Nk, Nl]Ri Unicity: A Region has only one external Border Intersection: Bi ∩ Bj = {S} ⇔ Ri and Rj are adjacent. Bi ∩ Bj = {N} || ∅ ⇔ Ri and Rj are disjoint or (Ri ⊂ Rj || Rj⊂ Ri). Union:
Bi ∪ Bj = Bi + Bj - Bi ∩ Bj
Regions: (with i ≠ j) Definition: A region Ri is defined by one external closed contour Bi and a collection of internal disjoint closed contours Bj: Ri = Bi ⊕ ∪jBj , j denotes an included Region Rj. Its order denotes the vertical position in the tree structure (Fig 6c). Union:
∪j Rj = Ω
Intersection: Ri ∩ Rj = ∅ Inclusion:
Rk ⊂ Ri , Ri = Bi ⊕ ∪jBj ⇔ k ∈ ∪j
orderk ≥ orderi + 1. Rk is a “child” of Ri. Geometry: A pixel represents the smallest possible region and provides the length factor between two adjacent Nodes. For a given S Ri, Rj, Nk, Nl , the shape is described by a Freeman code. Algorithm flowchart: The sequential process is described in three steps: Step1: A merging growing algorithm is performed to retrieve the Regions in the restored image, indexes them (Fig 6a) and initializes the topology process. In the same time, an edge detection is done (Fig 6b). Step2: According to the axioms defined, shape and topology are extracted for each region detected during the Step1 by using both index image and contour map. At this level, all the nodes, segments and borders are stored. Step3: Regions are stored iteratively in a rooted tree structure described in Fig 6c by analysing topology relations (adjacency, inclusion, intersection). New Borders are created to avoid topology inconsistencies (Fig 6c, Region R7+8).
a
Rext
c
b Fig 6. Topology process: a: merging growing algorithm, b: shape extraction, c: Tree structure based on topology analysis
Tree management A root tree structure was choosen to store in a simple and natural manner the objects according to their topology, but also to speed up the data access. In Fig 6c, the root denotes the border of the image processed. It can be seen as an empty Region which covers the whole image where all the Regions are included. In the construction of the tree, the “directed” notion is added. This notion establishes the fact that, in a rooted tree, from a given Region (different of the root) stored in the tree, it exists only one way to access to the root. Given this directed notion of a rooted tree, a rooted subtree can be defined for each region of the tree. As a consequence, for each Region which contains included regions, a sub-tree is recursively generated. In this case, the first child region represents a virtual region R´i without “hole” inside. The level of appeareance in the tree structure is indicated by the level order. It correponds to the vertical distance from the root (order = 0). Therefore, for a given subtree of the region Ri (order k), the direct children (order k+1) have an unique parent (Ri) and constitute “siblings” regions. Hence, for each region stored, we can directly check the included regions (children), the parent region, (i.e. vertical navigation in the tree), or siblings regions (horizontal navigation).
As a consequence, the position of the regions in the tree is important for a further edition of the tree, described in the next paragraph. Edition Once the tree is stored according to a given segmented image, it should be useful for the user to make some modifications; for instance, to merge adjacent regions, or merge child regions with the parent. Merging adjacent regions Ri, Rj in a new Region Rij (Horizontal relation in the tree): Since no information have to be added, Bij = Bi + Bj - Bi ∩ Bj is easely computed. The respective subtrees are merged without changing the depth. The mutual adjacent regions of (Ri Rj) are cheked to verify if one will not be included in the merged region. Merging Ri, Rj with Ri ⊂ Rj (Horizontal relation in the tree): If it exits, the sub-tree of Ri is decremented. With suppression of the R´i , and Ri. The algorithm described allows to build efficiently tree structures from segmented images without any parameters. The interest is to deal directly and to access to the object oriented information and not only to the
pixel oriented way. Shape and form are extracted and the Freeman coding allows to extract various estimators in order to describe more efficiently the objects. The tree architecture enables analyses of tiled images. Many applications are possible such as queries by content or data merging of different types or resolutions.
Table 1 summarizes some of the objects, features, which can be observed in RS images and the constraints to be applied for the DEM regularization. Two types of constraints are modelled. The constraints infered by the object itself, the internal ones, and the constraints related by the neighbouring objects relations/inter-dependencies. 2.
4. DATA FUSION / MERGING 1.
Data content/contraints modelling
In Fig 1 and Fig 7, the interest of 3D visualisation for scene understanding is emphasized. Perspective views are more natural for our perception and complementary details point out. However, in the same time, the artefacts in the DEM mainly leads to unrealistic views. Indeed, our perception associates automatically an object with its corresponding shape, characteristics, etc. The lake example (Fig 2, 5 and 7) is well-suited, intuitively, a lake should be a flat area but also presents an elevation which is inferior to the elevation of all its adjacent objects. The results achieved by segmentation/topology algorithms allow to delimit in 2D the spread of the object to be regularized in the DEM. Registration errors between image and DEM are not taken into account. It remains to set the elevation of the object. Its determination depends on: •
The DEM information
•
The object modelling
Airport / Parking, Football ground, Horse / Car track Street / Motorway River Building/House Forest
Tiled surface (small slope) 2 adjacent tiled surfaces (sym. slope) -
In order to be able to apply these constraints on the DEM, the system should learn the semantic definition of each object (i.e. user interests), such as “forest”, “urban area”, “soil”… To obtain the landscape recognition, an interactive learning step is performed by the user. This process is done in order to link the signal to the image classes. The user selects homogenous and representative thematic coverage which delimit training areas. Therefore, the user’s “knowledge” as well as his specific interests are incorporated in a flexible way. A “naive Bayes’ classifier” is used to link the signal to the image classes. In [10] further details are presented. According to the users interests and the defined label learned, the user selects the thematics coverage to be regularized (i.e. “lake”, “airport”…). From each studied thematic, a set of regions is selected. Adjacent or included regions with the same class are merged by updating the tree structure. Topology relations are preserved/updated during this step. 3.
• The neighbouring relations The last two points can be modeled by relative elevation constraints, while the first one provides observations in the form of absolute elevations. Constraints: Object, border elevation: Hobj Internal Neighbouring Sea border* HobjHneigh (flat) roof Horizontal surface Bridge /dam Hobj>Hneigh
HobjHneigh
4.
Hobj>Hneigh
The first experiments have been done for horizontal surfaces such as lakes or airports (Fig 7) in a dataset composed of multispectral enhanced SPOT 5 images (resolution: 2.5m) and X band SRTM DEM (~22m).
Table 1. Objects with elevation constraints (internal/external) for realistic 3D views. * without taking into account the Earth curvature.
Examples
In the case of the lake, its elevation (Hint) has been computed with a least square adjustement by using Eq 1. for the internal constraints. Neighbouring constraints are formulated through a slope computation for each thematic (Eq 2.). Hint = Hi + vi (1) dsthematic =
H ext j − H int d ext − int
+ vj
(2)
The slopes extracted are used to analyse the elevation computed (for the lake, all the slopes have to be positive) and to generate the transition borders. In Fig 7., the enhancement obtained is presented. The lines over the lake are the borders of the segmented regions before the interactive learning process. Further work will be done to regularize more complex objects (in terms of modelisation).
Fig 7. Merging image content in the DEM, perspective view of the Nice test site (France). A DEM is overlapped by an enhanced SPOT5 image [2.5x2.5m²] used as surface attribute. left: original SRTM DEM [30x30m²], right: regularized DEM.
5. CONCLUSION Potential of SAR data to provide elevation data for ambitious and large virtual reality purposes is emphasised. To enhance/regularize the data for Virtual Reality scenarios, a database preparation line has been presented. An inequality constrained least square adjustement will be added to fully optimize the constraints defined. In addition, the object modelling will be extended in order to be able to deal with more complex objects extracted from very high resolution city images [11]. Such systems constitutes a key step for realistic 3D visualisations of EO data and for the rendering optimisation to manage flight simulations. 3D applications will allow to promote RS images for various applications such as agricultural, urban planning, natural risk monitoring, etc.
3. M. Walessa, M. Datcu, Model based despeckling and information extraction from SAR images, IEEE Transaction on Geo. and RS, vol 38, n°5 p 2258-2269, 2000. 4. M.Walessa, M. Datcu, Enhancement of interferometric DEMs by spatially adaptive model-based filtering in nonstationary noise. Proc. USAR2000, p 695-698, 2000. 5. C. Maire, M. Datcu, P. Audenino, SAR DEM filtering by mean of Bayesian and multi-scale, nonstationary methods, Proc IGARSS´03, 2003. 6. G. Koepfler, C. Lopez, J. M. Morel, A Multiscale Algorithm for Image Segmentation by Variational Method, SIAM Journal on Numerical Analysis, vol 31 n°1, p 282299, 1994. 7. V.A. Kovalevsky, Finite Topology as Applied to Image Analysis, Computer Vision, Graphics, and Image Processing n°46, p 141-161, 1988.
BIBLIOGRAPHY
8. T.Y. Kong, A. Rosenfeld, Digital Topology: Introduction and Survey, Computer Vision, Graphics, and Image Processing n°48, p 357-393, 1989
1. P. Cignoni et al, BDAM - Batched Dynamic Adaptive Meshes for High Performance Terrain Visualization. Computer Graphics Forum, 22(3): p505514, 2003.
9. S. Winter, Topological Relations between Discrete Regions, Advances in Spatial Databases, Proc. SSD´95, vol 951, p 310-327, 1995.
2. M. Datcu, K. Seidel, M. Walessa, Spatial information retrieval from Remote Sensing Images: - Part I : Information theoretical perspectives, IEEE Transaction on Geo. and RS, vol. 36, n°5 p 1431-1445, 1998.
10. H. Daschiel, M. Datcu, Evaluation tools for Image Information Mining Systems, In this procedings, 2004. 11. M. Quartulli, M. Datcu, Stochastic modelling for structure reconstruction from high resolution SAR data, Proc IGARSS´03, 2003.
