Edge-preserving filtering: denoising, image smoothing/sharpening,. ... D. Detail enhancement for high-dynamic-range infrared images based on guided image.
Real-Time Visualization of High-Dynamic-Range Infrared Images based on Human Perception Characteristics Noise removal, image detail enhancement and time consistency Frederic Garcia, Cedric Schockaert and Bruno Mirbach PTU Optical, IEE S.A., 11, rue Edmond Reuter, Contern, Luxembourg {frederic.garcia, cedric.schockaert, bruno.mirbach}@iee.lu
VISAPP 10th International Conference on Computer Vision Theory and Applications
Introduction Digital Detail Enhancement (DDE) Preserve low contrast targets in HDR scenes To overcome limitations of the human visual system & common display devices 9038
8437
HDR image 255
255
0
0
2
Histogram Equalization
DDE
Temporal DDE (TDDE) Filter Flow diagram/pipeline I
Guided Filter (GF)
B
ai
+ -
λ
D
× D'
Flow diagram of the proposed TDDE filter
I: Input HDR image B: Base image component ai: linear coefficients from the GF to mask D D : Detail image component, i.e., D = I-B D‘: Enhanced detail image component, i.e., D‘ = λ D⋅ai I‘: B+D‘ λ: Gain factor to enhance preserved details O: Output 8bit image 3
I'
Local Adaptive Gamma Correction (LAGC) 12/14 bits → 8 bits
O
Temporal DDE (TDDE) Filter Base image component Similarly to [Zuo et al., 2011] & [Liu & Zhao, 2014] we rely on the advise of [He et al., 2013] and choose the GF to split I into base B and detail D image components Guided Filter: Edge-preserving filtering: denoising, image smoothing/sharpening,... Fast & non-approximative linear time algorithm In contrast to BF, GF does not suffer from gradient reversal artefacts GF has a significantly smaller processing time than BF, i.e., O(1)
[Zuo et al., 2011]
4
Zuo, C., Chen, Q., Liu, N., Ren, J., and Sui, X. Display and detail enhancement for high-dynamic-range
infrared images. Optical Engineering, 50(12):127401:1–9, 2011 [Liu & Zhao, 2014] Liu, N. and Zhao, D. Detail enhancement for high-dynamic-range infrared images based on guided image filter. Infrared Physics and Technology, 67:138–147, 2014 [He et al., 2013] He, K., Sun, J., and Tang, X. (2013). Guided Image Filtering. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(6):1397–1409, 2013
Temporal DDE (TDDE) Filter Base image component B image component is a linear transform of I in a window wk centred at pixel k
Bi = where
1 w
∑ (a I
k |i∈wk
σ k2 ak = 2 , σk + ε
k i
+ bk ) = a i I i + b i
bk = (1 − ak )µ k
are linear coefficients assumed to be constant in wk µk andσk2 are the mean & variance of I in wk |wk| is the number of pixels in wk ε is a regularization parameter penalizing large ak
5
Temporal DDE (TDDE) Filter Base image component
B (k=3, ε=10)
B (k=3, ε=500)
B images illustrating the behaviour of the GF depending on the ε parameter
6
Temporal DDE (TDDE) Filter Detail image component In general, the D image component results from the difference between the input HDR image I and B image component, D = I-B
D (k=3, ε=10)
D (k=3, ε=500)
Resulting detail image D depending on the filter parameters
7
Temporal DDE (TDDE) Filter Detail image component As discussed, Noise is present in D when setting ε too small Noise can be significantly reduced by setting ε big enough This in turn will suppress desired details in D Problem: Tradeoff between noise removal & detail preserve needs to be chosen Retained noise may be identified as details & thus enhanced by mistake in the resulting 8 bit image Solution: Mask the noise depending on the spatial detail Use the linear coefficient ai that results from the GF
8
Temporal DDE (TDDE) Filter Detail image component
ai , (k=3, ε=10)
ai , (k=3, ε=500) Resulting
9
ai depending on the filter parameters
Temporal DDE (TDDE) Filter Detail image component We thus propose to mask D as follows
D' = λ a i ⋅ D with λ a gain factor to increase, if desired, the constrast of the details in D
D‘ (k=3, ε=150) D‘ (k=3, ε=150, λ=1) λ=5) Resulting detail image D‘ after noise masking and detail magnification 10
Temporal DDE (TDDE) Filter 8 bit data representation Local Adaptive Gamma Correction (LAGC) [Liu et al., 2012] Incorporate human visual properties in their design (Weber‘s law) Details hardly seen on darker and/or brighter background became perceptible The final 8 bit image representation results from
I ' ' (i, j ) ( ) O i, j = 2 − 1 ⋅ M 2 − 1
(
N
)
γ (i , j )
with N=8 for data representation within the range [0,255], and
B(i, j ) − 2 M −1 ,1 γ (i, j ) = max exp M −1 2
11
Temporal DDE (TDDE) Filter Time consistency Linear extension within the raw active HDR with a temporal statistical stabilization
(2 I ' ' (i, j ) =
M
)(
t − 1 I ' (i, j ) − I min t t I max − I min
)
with I‘=B+D‘, M the number of bits (12 or 14) to encode I, and Itmin and Itmax the limits of the active data range computed from t I max = µt + k1σ t t I min = µt − k 2σ t
with k1 and k2 defining the active data range to be considered in the 8 bit representation.
µt = α ⋅ µt + (1 − α ) ⋅ µt −1 σ t = α ⋅ σ t + (1 − α ) ⋅ σ t −1
are respectively the adapted mean & standard deviation of I at time t α∈[0,1] defines the speed to adapt Itmin and Itmax to the limits of the active data range of frame t 12
Experimental evaluation Real test cases 2 real test cases have been considered, i.e., Forest scene: fine details to be preserved City scene: high variability of temperature ranges
Forest scene
13
City scene
Experimental evaluation Qualitative comparison Warm objects better perceived Better global contrast/sharpness without noise amplification
14
Comparison between BF&DDE, GF&DDE, and TDDE filters
Experimental evaluation Quantitative comparison EMEE contrast metric (measure of enhancement by entropy)
15
BF&DDE
GF&DDE
TDDE
Forest scene
0.227
0.312
0.377
City scene
0.219
0.216
0.554
Experimental evaluation Time consistency
16
Comparison of global brightness temporal stability
Experimental evaluation Time consistency
BF&DDE
17
GF&DDE
TDDE (with time consistency)
TDDE (without time consistency)
Experimental evaluation Time consistency
Comparison of temporal stability of the histogram mean for BD&DDE, GF&DDE and TDDE
18
Experimental evaluation Running time Running time detail to compute the 8 bit representation of the raw I image
19
B
D
O
Total
BF&DDE
55.26
6.63
4.06
65.95
GF&DDE
4.33
1.95
4.05
10.33
TDDE
4.28
0.47
10.54
15.55
Conclusions New detail enhancement technique for visualization of HDR IR images Independent & dedicated processing to compress the HDR while enhancing quasi nonperceptible details Dedicated noise handling of the D image component Enables to add a gain factor λ to increase the contrast of preserved details Non-perceptible details in front of dark/bright backgrounds are magnified to become perceptible, i.e., LAGC Time consistency Suitable for real-time applications
20
Real-Time Visualization of High-Dynamic-Range Infrared Images based on Human Perception Characteristics Noise removal, image detail enhancement and time consistency
Thank you! Frederic Garcia, Cedric Schockaert and Bruno Mirbach PTU Optical, IEE S.A., 11, rue Edmond Reuter, Contern, Luxembourg {frederic.garcia, cedric.schockaert, bruno.mirbach}@iee.lu
VISAPP 10th International Conference on Computer Vision Theory and Applications
