Advanced point cloud processing - ITC

CURRENT DEVELOPMENTS IN AIRBORNE OBLIQUE IMAGING SYSTEMS AND AUTOMATED INTERPRETATION OF COMPLEX BUILDINGS Markus Gerke

THE FIRST AIRBORNE PHOTOS

…WERE OBLIQUE

First recorded aerial photograph in the US (Boston), by James Wallace Black, 1860, source Wikipedia

THE FIRST AIRBORNE PHOTOS

…WERE OBLIQUE

 Intuitively the operator shot from a slanted angle  Recognition of buildings  In the 1930’s: USGS and military systematically captured oblique images

First recorded aerial photograph in the US (Boston),

Fairchild T3-A camera, source

by James Wallace Black, 1860, source Wikipedia

Petrie (2009)

CONTENT

1. Properties, configurations and systems

2. Applications, products and research

3. Conclusions and Outlook

CONTENT




PROPERTIES

SCENE OBSERVATION

Vertical/nadir images: good observation of ground features and roof structures, assume constant scale Oblique images: observation of vertical structures, but occlusion more dominant, varying scale

PROPERTIES

SCALE

(symbols from Höhle (2008))

According to Shufelt (1999): Vertical images: t < 5° Low oblique: 5°< t 30° If (t+α) > 90°: horizon visible In this presentation: concentration on high oblique images which do not include the horizon

PROPERTIES

OCCLUSION

visible invisible

“Self-” occlusion

PROPERTIES

OCCLUSION

visible invisible

“Self-” occlusion

Occlusion by other objects

PROPERTIES

OCCLUSION

Mitigation through multiple view and overlap

visible invisible

PROPERTIES

OCCLUSION - EXAMPLE

All images © Slagboom en Peeters


PROPERTIES

OCCLUSION - EXAMPLE



PROPERTIES

OCCLUSION - EXAMPLE


PROPERTIES

OCCLUSION - EXAMPLE


PROPERTIES

OCCLUSION - EXAMPLE


PROPERTIES

CONFIGURATIONS

According to Petrie (2009) we can distinguish between Fan

Some examples will follow

and

Maltese Cross configuration

PROPERTIES

FAN CONFIGURATION

Fan configuration: increase swath width across/along track: more effective area coverage, reconnaissance applications/corridor mapping. Example: IGI Dual CAM

Source: Homepage IGI www.igi.eu

PROPERTIES

FAN CONFIGURATION

Visionmap A3: two stepping frame cameras: increasing the FOV through sweeping.

Source: Homepage Visionmap Source: Pechatnikov et al., 2009

PROPERTIES

MALTESE CROSS CONFIGURATION

Maltese Cross configuration: one vertically pointing camera and 4 highly obliques pointing to the four cardinal directions. Used mainly for visualisation and (urban) data interpretation.

Source: Petrie, 2009

IGI Penta DigiCAM, Source: Homepage IGI

PROPERTIES


Maltese Cross configuration: one vertically pointing camera and 4 highly obliques pointing to the four cardinal directions. Used mainly for visualisation and (urban) data interpretation. Companies offering such systems, eg:  Track‘Air MIDAS  IGI Penta DigiCam  Hexagon/Leica 5 camera head, mid format camera RCD30 (multispectral)

Leica Oblique 5-camera head, source homepage Leica

PROPERTIES


Maltese Cross configuration: one vertically pointing camera and 4 highly obliques pointing to the four cardinal directions. Used mainly for visualisation and (urban) data interpretation. Companies using such a configuration, eg:  Pictometry / BlomOblique

Microsoft Bing maps

PROPERTIES



Microsoft Bing maps

 Google

Google maps

PROPERTIES



Microsoft Bing maps

 Google  Slagboom en Peeters (NL)

Google maps

CONTENT




APPLICATIONS, PRODUCTS AND RESEARCH

Low

Complexity of task

High

COMPLEXITY OF TASK VS. DEGREE OF AUTOMATION

Low

Scene Interpretation

3D city modeling

Supervised classification

Dense Matching/Meshing Monoplotting Manual inspection/visualisation

Degree of automation

High

Complexity of task


Manual inspection/visualisation



MANUAL INSPECTION/VISUALISATION

Purpose: manually check images, also compare to existing (cadastre) maps Slagboom en Peeters GIS interface


MANUAL INSPECTION/VISUALISATION

Purpose: manually check images, also compare to existing (cadastre) maps Pictometry ESRI interface

Complexity of task


Monoplotting Manual inspection/visualisation



MONOPLOTTING

Purpose: 2.5D mapping in images, e.g. for topographic maps, or height measurements, e.g. of houses, trees, lamp-posts etc. Tools e.g.  Pictometry EFS (Electronic Field Study) or Pictometry online  BlomDESKTOP Viewer or BlomWEB Viewer


MONOPLOTTING

Principle of height measurement in oblique images: If a DTM is known, the X-Y location of T can be computed through ray intersection with the DTM. Fixing the X-Y location and measuring dr’ in the image leads to an estimate of dh.


MONOPLOTTING

Tool: EFS Own assessment of EFS (Enschede images): 2D accuracy: RMSE = 0.5m height accuracy: RMSE = 0.6m

Source: IFA Sydney Togarepi, 2009

Problem: DTM specs not disclosed, DTM not exchangeable Camera calibration and orientation parameters cannot be modified

Complexity of task


3D city modeling

Monoplotting Manual inspection/visualisation



3D CITY MODELING/CAD-LIKE RECONSTRUCTION

Purpose: interactive building modeling  Texture mapping (Frueh et al., 2004, Wang et al., 2008)  Pre-compiled collections like BlomUrbex 3D  Interactive image-based modeling  Eagleview



BlomUrbex 3D: Access to 400 city models in different LODs

LOD 2, textures from Blom oblique images. Source: Homepage Blom.



Interactive image-based modeling

Autodesk Imagemodeler. Image ©Blom.



Interactive image-based modeling

Final model.



Roof/Wall report, e.g. from Eagleview/Pictometry

Source: www.eagleview.com

Complexity of task


3D city modeling




DENSE IMAGE MATCHING AND 3D POINT MESHING/TRIANGULATION

Purpose: visualization of 3D structures, simple modeling, derivation of height models  Open source/research:  Semi-Global matching (Hirschmüller 2008), PMVS2 (Furukawa and Ponce, 2010)  Meshlab (meshlab.sourceforge.net)  Commercial software: EADS Streetfactory (http://www.astriumgeo.com/street-factory)



Purpose: visualization of 3D structures, simple modeling, derivation of height models  IGI, Lünen, SGM result from IFP Stuttgart (Fritsch et al., 2012)



Purpose: visualization of 3D structures, simple modeling, derivation of height models  IGI, Lünen, SGM result from IFP Stuttgart (Fritsch et al., 2012)



Purpose: visualization of 3D structures, simple modeling, derivation of height models  IGI, Lünen, PMVS2 point cloud



Purpose: visualization of 3D structures, simple modeling, derivation of height models  IGI, Lünen, PMVS2 point cloud



Purpose: visualization of 3D structures, simple modeling, derivation of height models  IGI, Lünen, PMVS2 point cloud, meshed (Poisson reconstruction)



Purpose: visualization of 3D structures, simple modeling, derivation of height models  Slagboom en Peeters, Enschede, PMVS2 point cloud, meshed



Purpose: visualization of 3D structures, simple modeling, derivation of height models  Commercial system: EADS Street Factory

Source: EADS http://www.astrium-geo.com/street-factory

Complexity of task


3D city modeling





SUPERVISED (3D) CLASSIFICATION

Purpose: semi-automatically detect objects in the scene  Especially vertical structures like walls interesting for object recognition  Clear advantage over nadir images Example: supervised classification in object space: 3D points are classified instead of image pixels information fusion



Purpose: semi-automatically detect objects in the scene  Disaster mapping (here: after 2010 Haiti earthquake)  Facade (health) status important to assess building damage: advantage of oblique images over nadir images

Vertical

original images © Pictometry



Purpose: semi-automatically detect objects in the scene  Disaster mapping (here: after 2010 Haiti earthquake)  Facade (health) status important to assess building damage: advantage of oblique images over nadir images

East Vertical

North

original images © Pictometry

3D point cloud from image matching (PMVS2)



Purpose: semi-automatically detect objects in the scene  Disaster mapping (here: after 2010 Haiti earthquake)  Classifying trees/vegetation, facades, intact roofs, destroyed roofs 3D points with label information Completeness reached in several experiments: Trees/Vegetation: 63% Facades: 82% Roof intact: 80% Roof destroyed: 75% source: Gerke 2011



Purpose: semi-automatically detect objects in the scene  Same approach, but “intact” scene Enschede, overall accuracy ~80%

Complexity of task


Scene Interpretation

3D city modeling





SCENE INTERPRETATION

Purpose: fully automatically interpret the scene. Research at ITC: Exploit façade visibility for a) building detection and b) building verification.


SCENE INTERPRETATION: BUILDING DETECTION

Purpose: fully automatically interpret the scene. Research at ITC: Exploit façade visibility for a) building detection and b) building verification. Building detection: detect and outline buildings only using oblique airborne images Step 1: façade detection

see Xiao et al., 2012



Purpose: fully automatically interpret the scene. Research at ITC: Exploit façade visibility for a) building detection and b) building verification. Building detection: detect and outline buildings only using oblique airborne images Step 2: outlining

see: Xiao 2012



Purpose: fully automatically interpret the scene. Research at ITC: Exploit façade visibility for a) building detection and b) building verification. Building detection: detect and outline buildings only using oblique airborne images Results: -detection(building level): correctness 95% , completeness: 70 to 90% -outlining: accuracy of outline (RMSE): 85cm – 1.10m Quality of detection and delineation depend largely on image resolution and overlap source: PhD thesis Jing Xiao, unpublished yet


SCENE INTERPRETATION: BUILDING VERIFICATION

Purpose: fully automatically interpret the scene. Research at ITC: Exploit façade visibility for a) building detection and b) building verification. Building verification: Given a cadaster map the walls are verified using different evidence measures extracted from oblique images  Eg: lines projected: if wall exists facade lines match in object space (blue)

source: Nyaruhuma et al., 2012


SCENE INTERPRETATION: BUILDING VERIFICATION

Purpose: fully automatically interpret the scene. Research at ITC: Exploit façade visibility for a) building detection and b) building verification. Building verification: Given a cadastre map the walls are verified using different evidence measures extracted from oblique images Results: -correctness of decisions (building still existing/building demolished): 90100% -potential to go for a per-wall check

source: Nyaruhuma et al., 2012

CONTENT




CONCLUSIONS

 Oblique airborne photogrammetry in analogue times was too expensive, but with digital image evolution it comes back into focus  Oblique airborne photogrammetry is becoming mature shift to mid format cameras and sophisticated integrated sensor orientation  Applications in practice so far mainly visualization and (interactive) city modeling, automatic approaches (low level, high level) promising:  Images implicitly contain 3rd dimension; no need for additional data  Facades are very important to identify buildings  However: large image overlap preferred (mitigation of occlusion effects, multiple views per façade)

OUTLOOK

What the future might bring (personal view)  More airborne surveying companies will acquire cameras (see mid format heads from Leica/IGI)  Oblique airborne images will become standard (soon in NL: capturing of entire country by AeroData and Cyclomedia in Spring 2013 http://www.nederlandobliek.nl)  Boost of usage, mostly for urban applications: map update, 3D modeling  Automation: standard software needs to be extended, starting with bundle block adjustment/integrated sensor orientation

LITERATURE

Fritsch, D., Kremer, J., Grimm, A. 2012. A Case Study of Dense Image Matching Using Oblique Imagery – Towards All-in-one Photogrammetry. GIM International. 26(4): 18-23. Frueh, C., Sammon, R., Zakhor, A., 2004. Automated texture mapping of 3D city models with oblique aerial imagery. Proceedings of the 2nd International Symposium on 3D Data Processing, Visualization, and Transmission (3DPVT'04), 396-403. Furukawa, Y., and J. Ponce, 2010. Accurate, Dense, and Robust Multi-View Stereopsis. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 32(8): 1362-1376. Gerke, M., 2011. Supervised classification of multiple view images in object space for seismic damage assessment. Photogrammetric image analysis : ISPRS conference, PIA 2011, Lecture Notes in Computer Science Hirschmüller, H., 2008. Stereo Processing by Semi-Global Matching and Mutual Information. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(2): 328-341. 6952: 221-232.

LITERATURE

Höhle, J., 2008. Photogrammetric measurements in oblique aerial images. Photogrammetrie Fernerkundung Geoinformation, 2008(1): 7-14. Nyaruhuma, A.P., Gerke, M., Vosselman, G., Mtalo. E.G., 2012. Verification of 2D building outlines using oblique airborne images. ISPRS Journal of Photogrammetry and Remote Sensing, 71: 62-75. Petrie, G., 2009. Systematic oblique aerial photography using multi frame cameras. Photogrammetric Engineering & Remote Sensing,75(2): 102-108. Shufelt, J.A., 1999. Performance evaluation and analysis of monocular building extraction from aerial imagery. IEEE Transactions on Pattern Analysis and Machine Intelligence 21 (4): 311-326. Wang, M., Bai, H., Hu, F., 2008. Automatic texture acquisition for 3D model using oblique aerial images. The First International Conference on Intelligent Networks and Intelligent Systems (ICINIS '08), 495-498.

LITERATURE

Xiao, J., Gerke, M. and Vosselman, G., 2012. Building extraction from oblique airborne imagery based on robust facade detection. ISPRS Journal of Photogrammetry and Remote Sensing, 68: 5668. Xiao, J., 2012. Automatic building outlining from multi-view oblique images. XXII ISPRS Congress : Imaging a Sustainable Future, ISPRS Annals, I-3, 2012 ed. by M. Shortis, N. Paparoditis and C. Mallet, 323-328

Thank you for your attention. Questions?

Advanced point cloud processing - ITC

Advanced point cloud processing - ITC

Suggest Documents

Modification of Lidar Point Cloud Processing Methodology - CiteSeerX

CAMPINO â A skeletonization method for point cloud processing

a semi-automated point cloud processing ... - ISPRS Archives

An Overview of Lidar Point Cloud Processing Software - ASPL

a semi-automated point cloud processing ... - ISPRS Archives

Horus: Guide to post-processing of the point cloud

Point cloud densification

Predictive Point-Cloud Compression

Intrinsic point cloud simplification

Cloud-Point Curves

Advanced Signal Processing - SoftBooks

Advanced Digital Signal Processing

Point Cloud Library (PCL) - CiteSeerX

cloud security - Check Point Software

PCL :: Features - Point Cloud Library

smart point cloud - ISPRS Annals

Guided 3D point cloud filtering

Cloud processing in IASI context

Post-processing using Matlab (Advanced)

Advanced XML Processing - SIGMOD Record

(\(Advanced Information and Knowledge Processing

Advanced Laser Processing and Photoluminescence

ADVANCED BIOMATERIALS Fundamentals, Processing, and ...

Advanced Equation Processing for TCAD

Advanced point cloud processing - ITC