Pixel Weighted Average Strategy for Depth Sensor Data Fusion Frederic Garcia*+ Bruno Mirbach* *Advanced Engineering – IEE S.A., Luxembourg
Bjorn Ottersten+
Frédéric Grandidier*
[fga, bmi, fgr, acu]@iee.lu
We present a new multi-lateral filter to enhance the spatial resolution and reduce the depth measurement noise level of ToF depth sensors. Our approach fuses low resolution depth maps with high resolution images taken from the same scene configuration. We extend the joint bilateral upsampling (JBU) filter by a new factor that considers the low realiability of depth measurements along the low resolution depth map edges.
• Given a depth map I´ from a ToF depth sensor and an image standard 2-D camera where: - the resolution of I´ is far below the resolution of
Ángel Cuesta*
+SnT – University of Luxembourg
[email protected]
Idea: Add a new factor (CM) to the kernel in the JBU filter that weights a depth measurement depending on the gradient of a depth edge.
Example 1: Image fusion artifacts.
Approach: Raw Depth Map (61 x 56)px
High Resolution Image (648 x 488)px
D´ from a
Mapped high resolution image D
Mapped low resolution depth map I
Credibility Map CM
I' Raw depth map gradient
D´.
- depth measurements in I´ are strongly affected by noise. • To enhance I´ to the same resolution of D´ and to reduce the noise level in depth measurements.
∇I '
I' Data Alignment
Objectives:
Mapping procedure
Mapping procedure
I
C
D' Mapping procedure
D
• To avoid artifacts such as texture copying or edge blurring.
JBU enhanced depth map
PWAS enhanced depth map
Example 2:
Credibility Map
Intensity value
PWAS
D
I
PWAS result
CM
J
Pixels Depth measurement
Mapped depth map
Mapped 2--D image
CM
Final Depth Map (648 x 488)px
Definition: Pixels
The depth measurement is inaccurate on edge pixels. ⇒ Solution: To assign a lower weight to these pixels in the kernel filter. ⇒ Extension of the joint bilateral filter by the Credibility Map (CM).
Low resolution depth map
PWAS enhanced depth map
Textured enhanced depth map
Pixel Weighted Average Strategy (PWAS):
J (x ) =
∑ Gσ ( x − y )Gσ ( D ( x ) − D ( y ) )CM(y) I (y) ∑ Gσ ( x − y )Gσ ( D ( x ) − D ( y ) )CM(y)
y∈ N ( x )
s
y∈ N ( x )
r
s
Where
CM( x) = Gσ c (C ( x ) )
r
The proposed multi-lateral filter enhances low resolution depth maps with a global noise level reduction. The CM ensures more accurate depth measurements where the depth discontinuities are well defined and adjusted to the guidance image. The filter prevents texture copying and limits edge blurring to the CM boundaries.
2010 International Conference on Image Processing (ICIP’10)
September 26-29, 2010
Hong Kong
