Visibility Enhancement of Single Hazy Images Using Hybrid Dark Channel Prior Yi-Jui Cheng∗ , Bo-Hao Chen∗ , Shih-Chia Huang∗ , Sy-Yen Kuok , Andrey Kopylov† , Oleg Seredin† , Yury Vizilter§ , Leonid Mestetskiy‡ , Boris Vishnyakov§ , Oleg Vygolov§ , Chia-Ruei Lian¶ and Chi-Ting Wu¶ ∗ Department
of Electronic Engineering National Taipei University of Technology, Taipei 106, Taiwan Email:
[email protected] k Department of Electrical Engineering National Taiwan University, Taipei 106, Taiwan † Department of Automation and Remote Control Tula State University, Tula 300012, Russian ‡ Computational Mathematics and Cybernetics Faculty Moscow State University, Moscow 119991, Russian § State Research Institute of Aviation Systems (GosNIIAS), Moscow 125319, Russian ¶ TPV Technology Limited (TPV), Taipei 106, Taiwan Abstract—Outdoor images captured during inclement weather conditions generally exhibit visibility degradation. Localized light sources often result from activation of streetlights and vehicle headlights and are common scenarios in these conditions. The presence of localized light sources in hazy images may cause the generation of oversaturation artifacts when those images are restored by traditional state-of-the-art haze removal techniques. Therefore, we propose a novel haze removal approach based on the proposed hybrid dark channel prior technique in order to remedy the problems associated with localized light sources during image restoration. The overall results show that the proposed haze removal approach can recover haze-free images more effectively than can the other previous state-of-the-art haze removal approach while avoiding over-saturation. Keywords—haze removal, dark channel prior, localized light.
I.
I NTRODUCTION
The visibility of images captured in outdoor scenarios can frequently experience degradation as a result of phenomena which include absorption and scattering of light by the atmospheric particles such as haze, fog, mist, and so on. Accordingly, image visibility degradation can be problematic to many systems which operate under a wide range of the weather conditions, including intelligent transportation systems [1], surveillance systems [2]–[5], outdoor object recognition systems [6], remote sensing systems [7], [8], and so on. To improve visibility in hazy images, haze removal techniques have been recently proposed for effective restoration. Since the amount of absorption and scattering depends on the depth of scene between the digital camera and the scene point, scene depth information is essential for the recovery of scene radiance by haze removal techniques. As such, these techniques can be divided into two major groups: given depth [9]–[11] and unknown depth [12]–[16]. c 978-1-4799-0652-9/13/$31.00 2013 IEEE
As can be inferred, given depth approaches are based on the assumption that the depth is given [9]–[11], which is then used by these approaches to restore hazy images. Specifically, depth information can be obtained from either additional operations or interactions, such as through the use of altitude and tilt information [9], or via manual approximation of the distance distribution over the sky area and vanishing point in a captured image [10], or through the use of an approximate 3D geometrical model of the captured scene [11]. However, given depth approaches are not suitable for visibility restoration in real-world applications due to a serious limitation: the depth information needs to be provided by the user, yet it is difficult to obtain. Unknown depth approaches have been proposed in order to solve this problem [12]–[16]. They estimate unknown depths and thereby recover scene radiances in hazy images. These approaches use either multiple images [12] and [13] or single images [14]–[16] to obtain depth information. For instance, the haze removal techniques proposed in [12] and [13] restore a hazy image by estimating the unknown depth via multiple images. However, these techniques usually require the use of additional hardware devices and demand more computational complexity. Because of this, recent investigations [14]–[16] have focused on the use of single images to estimate unknown depths and subsequently recover scene radiance. The single-image method proposed by Tan [14] bases its estimation on an observation that captured hazy images have lower contrast than haze-free images, whereupon the image haze formation can be removed accordingly. However, the use of this method usually results in artifact effects along depth edges in the restored images. The method proposed by Fattal [15] deduces the medium transmission by using the assumption that the transmission and surface shading are locally uncorrelated. Nevertheless, this approach often fails in conditions featuring
(a)
(b)
(c)
(d)
(e)
(f) f
(g)
(h)
(i)
(j) () (j
Fig. 1. The restoration results for a hazy image which contains many streetlights via different patch sizes by the method of He et al. [16]. (a) the original hazy image; (b) estimation of the dark channel using a 3 × 3 patch from the hazy image; (c) estimation of atmospheric light using the dark channel produced by 3 × 3 patch size; (d) the recovered image produced by using the 3 × 3 patch; (e) estimation of the dark channel using a 15 × 15 patch from the hazy image; (f) estimation of atmospheric light using the dark channel produced by 15 × 15 patch size; (g) the recovered image produced by using the 15 × 15 patch; (h) estimation of the dark channel using a 45 × 45 patch from the hazy image; (i) estimation of the atmospheric light by using the dark channel produced by a 45 × 45 patch size; (j) the recovered image produced by using the 45 × 45 patch.
heavy haze formation. In contrast to the methods mentioned above, the simple yet effective algorithm proposed by He et al. [16] accomplishes haze removal via the dark channel prior technique by using the observation that at least one channel in the RGB color space is comprised of pixels which have lower intensities within local patches of an outdoor, haze-free image. Due to this observation, the method of He et al. [16] can directly estimate the amount of haze and subsequently recover scene radiance efficiently. Until now, the approach of He et al. [16] has been regarded as the best haze removal technique for single images. However, hazy images which contain localized light sources can present difficulties for the dark channel prior method [16]. Moreover, restoration of hazy images via the dark channel prior method usually results in generation of serious artifacts, such as over-saturation and halo effects. This can be especially problematic in regard to many commonly encountered scenarios, such as those in which drivers turn on their car headlights, or when streetlamps are suddenly lit in order to increase visibility during inclement weather. For example, Fig. 1 presents an illustration of haze removal via the method of He et al. [16] in conjunction with different patch sizes. Fig. 1(a) shows the input hazy image; the second column presents the dark channel prior produced by patches of different sizes; the third column illustrates the estimation of atmospheric light after the operation of the dark channel prior via different patches; the fourth column describes the scene
radiance recovered through the different patches. Note that the estimated atmospheric light is labeled with red circles in the third column of Fig. 1. As can be seen in Fig. 1(b) and Fig. 1(e), the dark channel prior method employed a 3 × 3 and a 15 × 15 patch size to recover scene radiance, respectively. The restored results shown in Fig. 1(d) and Fig. 1(g) indicate that serious over-saturation effects are triggered in the recovered scene radiances. This is due to the use of small patch sizes in the original hazy image featuring localized light, which subsequently lead to the misidentification of the atmospheric light source. This can be seen in the areas labeled with red circles in Fig. 1(c) and Fig. 1(f). In contrast, the dark channel prior method used a large 45 × 45 patch size to restore the hazy image, as can be seen in Fig. 1(h). The restored results shown in Fig. 1(j) indicate that the generation of oversaturated images has been effectively avoided. Moreover, because this operation used a large patch size in the original hazy image featuring localized light, the correct atmospheric light has been identified, as can be seen in Fig. 1(i). However, operation of such large patch usually leads to the generation of halo effects and block artifacts along depth edges of the recovered scene radiances. Therefore, we propose a novel approach for haze removal using the conjunctive utilization of the proposed hybrid dark channel prior module and the proposed visibility recovery
Input Hazy Image
Hybrid Dark Channel Prior Module Hybrid Dark Channel Prior Estimating the Transmission Map Dark Channel Prior using 3x3 patches
Dark Channel Prior using 45x45 patches
Estimating the Atmospheric Light
Visibility Recovery Module
Output Hazy-free Image
Fig. 2.
Flowchart of the proposed haze removal approach for visibility enhancement of single images.
module, as shown in Fig. 2. The proposed approach is capable of concealing localized light sources when the captured image contains them. By doing so, hazy image restoration by the proposed method can effectively avoid generation of serious artifacts. Experimental results and subsequent visual evaluations demonstrate that the proposed hybrid dark channel prior technique can remove haze from single images captured in real-world conditions more effectively than can the dark channel prior technique [16]. II.
P ROPOSED HAZE REMOVAL APPROACH
In this section, we present a new type of haze removal approach which can achieve effective removal of haze formation while avoiding the generation of artifact effects. The proposed approach is composed of two proposed modules: a Hybrid Dark Channel Prior (HDCP) module and a Visibility Recovery (VR) module.
The proposed HDCP module is capable of producing an effective transmission map for circumventing halo effects in the recovered image, as well as accurately estimating the location of atmospheric light in order to avoid oversaturation as mentioned in the previous section. A. Hybrid Dark Channel Prior Module The use of the dark channel prior technique [16] cannot offer satisfactory restoration results due to its tendency to generate serious oversaturation effects when incoming hazy images contain localized light sources. This is because the dark channel prior technique generates an inaccurate estimation of the position of atmospheric light as discussed in the previous section. Thus, in order to conceal localized lights and avoid oversaturation, the hybrid dark channel prior technique will be introduced in the following section. First, we take advantage of small and large patch sizes via
(a)
(b)
(c)
(d)
(e)
Fig. 3. The “Train” image: (a) is the original image with localized light sources; (b) is the estimated atmospheric light using the dark channel prior technique [16]; (c) is the restoration result via the dark channel prior technique [16]; (d) is the atmospheric light estimated via our proposed HDCP method; (e) is the restoration result produced via our proposed HDCP method. The red circles indicate the positions of atmospheric light.
different weights to effectively estimate the haze density of image. In addition, we use a large patch size to acquire the correct atmospheric light during implementation of the dark channel prior technique. The hybrid dark channel prior can be expressed as: α min min J c (i, j) α + β (i,j)∈Ω(x,y) c∈{r,g,b} c min min J (i, j) ,
J dark (x, y) = +
β α + β (i,j)∈µ(x,y)
(1)
c∈{r,g,b}
where J represents an arbitrary image under a wide range of weather conditions, J c represents a channel of color image J, Ω and µ represent a local patch centered at (x, y), min J c (i, j) performs the minimum operation on J c , c∈{r,g,b}
min
(i,j)∈Ω(x,y)
performs a minimum filter on the local patch
centered at (x, y) using the small patch size, and
min
(i,j)∈µ(x,y)
performs a minimum filter on the local patch centered at (x, y) using the large patch size. After calculating haze density, the transmission map can be directly estimated as follows:
ωα I c (i, j) min th (x, y) = 1 − min α + β (i,j)∈Ω(x,y) c∈{r,g,b} Ac (3) I c (i, j) ωβ − min min , α + β (i,j)∈µ(x,y) c∈{r,g,b} Ac where the most optimal small patch size of the image can be set to 3 × 3 experimentally, and most optimal large patch size can be set to 45 × 45 experimentally. Moreover, the value of α and β are set to 20 and 1 experimentally, by which can be acquired the optimum results for single image haze removal. B. Visibility Recovery Module Here, the information provided via the HDCP module is employed in the VR module to effectively recover the scene radiance. As a result, a high-quality, haze-free image can be achieved. The scene radiance J(x, y) can be recoverd by:
J c (x, y) =
I c (x, y) − Ac + Ac , max(th (x, y) , t0 )
(4)
I c (i, j) α min min α + β (i,j)∈Ω(x,y) c∈{r,g,b} Ac (2) β I c (i, j) − min min . α + β (i,j)∈µ(x,y) c∈{r,g,b} Ac
where c ∈ {r, g, b}, J c (x, y) represents the scene radiance, I c (x, y) represents the image captured under different conditions, Ac represents the atmospheric light, th (x, y) represents the transmission map using the HDCP module, and t0 is assumed to have a typical value of 0.1.
Finally, in order to retain a small amount of haze for natural appearance, a constant parameter ω (set to 0.95) is added in the transmission map as follows:
The flowchart shown in Fig. 2 illustrates the proposed haze removal algorithm which is implemented via two proposed modules: a HDCP module and a VR module.
th (x, y) = 1 −
(a)
(b)
(c)
(d)
(e)
Fig. 4. The “Street” image: (a) is the original image with localized light sources; (b) is the estimated atmospheric light using the dark channel prior technique [16]; (c) is the restoration result via the dark channel prior technique [16]; (d) is the atmospheric light estimated via our proposed HDCP method; (e) is the restoration result produced via our proposed HDCP method. The red circles are the positions of atmospheric light.
III.
E XPERIMENTAL RESULTS
In this section, we present the results of a qualitative visual inspection in order to assess and compare the subjective effects of image restoration accomplished by the proposed hybrid dark channel prior method and the dark channel prior method [16] in respect to single images captured in situations of differing localized light conditions. As demonstrated through qualitative evaluation using these test images, the use of the proposed hybrid dark channel prior technique offers significantly improved restoration results when compared to those produced by the dark channel prior technique [16]. As can be seen in Fig. 3(a), the “Train” image was captured at a railway station at which many streetlights exist. Figure 4(a) shows the “Street” image, which features a roadway along which many vehicles are moving with their headlights on due to poor visibility. Restoration results for these situations were generated through use of the proposed hybrid dark channel prior technique and the dark channel prior technique [16], as can be observed in (c) and (e) of Fig. 3 and Fig. 4. The restored images presented in Fig. 3(c) and Fig. 4(c) indicate that unsatisfactory restoration results featuring serious oversaturation were generated by the use of the dark channel prior technique. This is due to the fact that the captured images feature localized light sources that are brighter than the atmospheric light. Consequently, the localized light is misjudged as atmospheric light, as shown in Fig. 3(b) and Fig. 4(b). In contrast to the dark channel prior technique, use of our hybrid dark channel prior technique resulted in superior restoration while generating fewer artifact effects, as shown in Fig. 3(e) and Fig. 4(e). This is because that the proposed HDCP
module can effectively conceal the localized light sources, whereupon the position of atmospheric light can be accurately estimated, as shown in Fig. 3(d) and Fig. 4(d). IV.
C ONCLUSIONS
This paper proposes a new type of haze removal approach based on the proposed hybrid dark channel prior technique for use in realistic scenes captured during inclement weather. Two unique modules make up the structure of the proposed method: the hybrid dark channel prior module and the visibility recovery module. The proposed HDCP module can efficiently conceal localized light sources, subsequently allowing an accurate estimation of the position of atmospheric light. Additionally, the effective estimation of a transmission map is provided by which to avoid the generation of artifact effects in the restored image. By doing so, the VR module can produce a high-quality, haze-free image without any serious artifact effects. The results of simulation experiments using realistic scenes with localized light sources indicate that the proposed hybrid dark channel prior technique achieves the most satisfactory restoration results when compared with the previous dark channel prior technique. In particular, qualitative evaluation of the results produced by the compared methods show that the proposed approach accomplishes a higher degree of visibility recovery for images captured in inclement weather conditions, as well as superior restoration effects. ACKNOWLEDGMENT This research was supported by the National Science Council, Taiwan under Grant NSC 102-2221-E-027-065, NSC 1012923-E-002-016-MY3, and NSC 100-2628-E-027-012-MY3. The corresponding author is with the Department of Electronic
Engineering, National Taipei University of Technology, Taipei 106, Taiwan. (e-mail:
[email protected]) R EFERENCES [1]
[2]
[3]
[4]
[5]
[6]
[7]
[8]
[9]
[10]
[11]
[12]
[13]
[14] [15] [16]
N. Buch, S.A. Velastin, and J. Orwell, “A Review of Computer Vision Techniques for the Analysis of Urban Traffic,” IEEE Trans. Intelligent Transp. Syst., vol. 12, no. 3, pp. 920-939, Sep. 2011. S. C. Huang, “An Advanced Motion Detection Algorithm with Video Quality Analysis for Video Surveillance Systems,” IEEE Trans. Circuits Syst. Video Technol., vol. 21, no. 1, pp. 1-14, Jan. 2011. F. C. Cheng, S. C. Huang, and S. J. Ruan, “Scene Analysis for Object Detection in Advanced Surveillance Systems using Laplacian Distribution Model,” IEEE Trans. Systems, Man, and Cybernetics -Part C: Applications and Reviews, vol. 41, pp. 589-598, Sep. 2011. F. C. Cheng, S. C. Huang, and S. J. Ruan, “Illumination-Sensitive Background Modelling Approach for Accurate Moving Object Detection,” IEEE Transactions on Broadcasting, vol. 57, no. 4, pp. 794-801, Dec. 2011. S. C. Huang, and F. C. Cheng, “Motion Detection with Pyramid Structure of Background Model for Intelligent Surveillance Systems,” ELSEVIER Eng. Appl. Artif. Intel., vol. 25, no. 7, pp. 1338-1348, Oct. 2012. O. Barinova, V. Lempitsky, and P. Kholi “On Detection of Multiple Object Instances Using Hough Transforms,” IEEE Trans. Pattern Anal. Machine Intell., vol. 34, no. 9, pp. 1773-1784, Sep. 2012. S. E. Kemeny, R. Panicacci, B. Pain, L. Matthies, and E. R. Fossum, “Multi-resolution image sensor,” IEEE Trans. Circuits Syst. Video Technol., vol. 7, pp. 575-583, Aug. 1997. D. J. Dailey, F.W. Cathey, and S. Pumrin, “An algorithm to estimate mean traffic speed using uncalibrated cameras,” IEEE Trans. Intell. Transp. Syst., vol. 1, no. 2, pp. 98-107, Jun. 2000. K. Tan and J. P. Oakley, “Enhancement of Color Images in Poor Visibility Conditions,” Proc. IEEE Int. Conf. Image. Proc. (ICIP2000), vol. 2, pp. 788-791, Sep. 2000. S. G. Narasimhan and S. K. Nayar, “Interactive (De)Weathering of an Image using Physical Models,” ICCV Workshop on Color and Photometric Methods in Computer Vision., pp. 1387-1394 Oct, 2003. J. Kopf, B. Neubert, B. Chen, M. Cohen, D. Cohen-Or, O. Deussen, M. Uyttendaele, and D. Lischinski, “Deep Photo: Model-Based Photograph Enhancement and Viewing,” ACM Trans. Graphics., vol. 27, no. 5, pp. 116:1-116:10, Dec. 2008. Y. Y. Schechner, S. G. Narasimhan, and S. K. Nayar, “Polarization Based Vision Through Haze,” Proc. Applied Optics, special issue: light and color in the open air., vol. 42, no. 3, Jan. 2003. S. G. Narasimhan and S. K. Nayar, “Contrast Restoration of Weather Degraded Images,” IEEE Trans. Pattern Anal. Machine Intell., vol. 25, no. 6, pp. 713-724, Jun. 2003. R. Tan, “Visibility in Bad Weather from a Single Image,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, Jun. 2008. R. Fattal, “Single Image Dehazing,” Proc. ACM SIGGRAPH’08, 2008. K. He, J. Sun, and X. Tang, “Single Image Haze Removal Using Dark Channel Prior,” IEEE Trans. Pattern Anal. Machine Intell., vol. 33, no.12, pp. 2341-2353, Dec. 2011.
