Optics & Laser Technology 62 (2014) 73–81
Contents lists available at ScienceDirect
Optics & Laser Technology journal homepage: www.elsevier.com/locate/optlastec
The modified gradient edge detection method for the color filter array image of the CMOS image sensor Yu Zhang a,n, Guangyi Wang a, Jiangtao Xu b, Zaifeng Shi b, DeXing Dong c a
Institute of Electronic and Information, Hangzhou Dianzi University, Hangzhou, PR China School of Electronic and Information Engineering, Tianjin University, Tianjin, PR China c Brigates Microelectronics Co., Ltd., Kunshan, PR China b
ar t ic l e i nf o
a b s t r a c t
Article history: Received 26 December 2013 Received in revised form 24 February 2014 Accepted 26 February 2014 Available online 20 March 2014
The modified gradient edge detection method applied in demosaicing the color filter array image is proposed in this paper. Firstly, the adjacent pixels are ranged from large to small. Then, the absolute differences of sorted pixels are calculated to analyze the distribution of the possible edge. Finally, the arithmetic operators being along the possible edge and that being across the possible edge are designed to estimate the accurate edge information. The experimental results verify that the proposed method gives better performance than the traditional gradient edge detection methods, and can discriminate the accurate edge, even where the gradients in different direction are close. & 2014 Elsevier Ltd. All rights reserved.
Keywords: Demosaicing Edge detection CMOS image sensor
1. Introduction CMOS image sensor is one kind of optoelectronic device. In order to reduce the cost, many CMOS image sensors only use photodiode diode covered with color filter array (CFA) to capture one of the three-primary colors at every pixel location. The process of restoring the CFA image to the full color image is called CFA interpolation or demosaicing. It will generate the color artifacts to interpolate the missing color values across edges. Gradient edge detection is one of the most common methods to distinguish the boundaries of different objects in one CFA image. Due to the mosaic structure of the CFA image, the gradient edge detection methods used in the full color image cannot be used directly in detecting the edges of the CFA image. The existing gradient edge detection methods used in the edge-sensitive demosaicing algorithm can be classified into two categories. The first category of methods directly or indirectly employed the vertical, horizontal or diagonal gradients to orientate the direction of the edge [1–4], or adaptively fused the vertical and horizontal demosaicing values to acquire the missing color values [5,6]. The second category of methods designed the filters or arithmetic operator masks to estimate the direction of the missing color channel [7–10].
n
Corresponding author. Tel.: þ 86 15824482396. E-mail address:
[email protected] (Y. Zhang).
http://dx.doi.org/10.1016/j.optlastec.2014.02.017 0030-3992/& 2014 Elsevier Ltd. All rights reserved.
All the above gradient edge detection methods have made a contribution to reduce the generation of the color artifacts. However, these methods only pay attention to the relative difference values of the same color channel or the same color difference. And they are liable to cause the mistake that considers the edges with the same gradient values as the same edge. In order to overcome this drawback, the proposed method focuses more on analyzing the value distribution of the adjacent pixels, instead of only focusing on their relative difference values. This method firstly ranges the values of the adjacent pixels with the same color from small to large. Then, it calculates the absolute differences between every two sorted pixels to ascertain the possible edge of the current pixel. Finally, the more accurate edge is determined by designing the adaptive arithmetic operators. The rest of the paper is organized as follows: Section 2 presents the proposed method. The experimental results and the performance comparisons with other state-of-the-art gradient edge detection approaches are reported in Section 3. The conclusions are given in Section 4.
2. The proposed algorithm Although there exist many CFA patterns, the Bayer pattern in Fig. 1 is the most widely used pattern in digital devices [1]. Let us take B23 in Fig. 1 as the current pixel for example. The supporting theory of many gradient edge detection methods to acquire the edge information of adjacent green pixels is the intra-channel
74
Y. Zhang et al. / Optics & Laser Technology 62 (2014) 73–81
correction theory, namely the same channel values of the pixels being along the same edge vary less [1–10]. However, according to these edge detection methods, the following mistakes are often made, as shown in Fig. 2. Fig. 2 is extracted from the test image in Fig. 3(b). The images in Fig. 3 are the standard testing images in the RGB pattern, which are supplied by Kodak company. The vertical and horizontal gradients are respectively gradient_v ¼ j99 49j ¼ 50; and gradient_h ¼ j40 90j ¼ 50. With the mean value principle, the missing green value of the current blue pixel is calculated by
Fig. 1. Bayer pattern.
green_current ¼ j99þ 90þ 49þ 40j=4 ¼ 69:5, while the raw green value of the current blue pixel is 40. Thus, the wrong edge detection and the wrong demosaicing value appear. The similar mistake will occur in detecting the edge information of color difference. The reason why the wrong edge detection appears is that the vertical and horizontal gradients are close, and the traditional gradient edge detection method cannot distinguish the accurate edge information. In order to overcome this kind of mistakes, this paper proposed the modified gradient edge detection method applied in demosaicing the color filter array image. The structure of the proposed method is shown in Fig. 4. The supporting theory of the proposed method is still the intrachannel correction theory. In order to distinguish the accurate edge information of the missing pixel with close gradient values, the proposed method adapts three steps. Firstly, the adjacent pixels of the current pixel in CFA image are ranged from large to small. Then, the absolute differences of sorted pixels are calculated to analyze the distribution of the possible edge. Finally, the arithmetic operators being along the possible edge and that being across the possible edge are designed to estimate the accurate edge information. Thus, the accurate edge information of the CFA image is obtained. Since the green channel contributes mostly to the illumination of the image, and the good green demosaicing will benefit the interpolation of red and blue channels, this paper starts with the edge detection in demosaicing the missing green value at blue/red pixel in Bayer pattern. The edge detection in demosaicing the missing blue/red values at red/blue pixels, and in demosaicing the missing blue/red channel values at green pixels, can use the similar method.
2.1. The edge detection in demosaicing the missing green channel values at blue/red pixels Firstly, the values of adjacent pixels are sorted from large to small. Since the number of the adjacent pixels is only 4, no matter what kind of sorting algorithm does not significantly change the complexity of
CFA image
Fig. 2. The data distribution is liable to make wrong edge detection.
Fig. 3. The testing images.
Rang the adjacent pixels from large to small
estimate the possible edge information
ascertain Edge detection the of the CFA image accurate edge information
Fig. 4. The structure of the proposed method.
Y. Zhang et al. / Optics & Laser Technology 62 (2014) 73–81
the edge detection method. This paper utilizes the bubble sort to rank the values of G13, G22, G24 and G33. a, b, c and d mean the maximum value, the second maximum value, the third maximum value, and the minimum value of G13, G22, G24 and G33 respectively. Next, the proposed method calculates the absolute differences between every two pixels of a, b, c and d, let θ1 ¼ ja bj, θ2 ¼ ja cj, θ3 ¼ ja dj, θ4 ¼ jb cj, θ5 ¼ jb dj, and θ6 ¼ jc dj. The possible distribution of the sorted values, as shown in Fig. 5, can be summarized into three states: 1) All the four adjacent green pixels have close values. If ab Z0, then |a þb| ¼|a| þ|b|. The values of a b, b d, a c, c d, and b c are positive values or zero, so 8 a bþ b d ¼ a b þ b d ¼ θ1 þ θ5 > > > : a bþ b c þc d ¼ a b þ b c þ c d ¼ θ1 þ θ4 þ θ6
ð1Þ
θ3 r δ can represent all the situations that the values of four adjacent pixels are close. Where δ is the threshold, and the less δ is, the more accurate the method is. Thinking about θ3 consists of several absolute differences, let δ ¼ 15 in this paper. Thus, the value distribution of the adjacent green pixels is shown in Fig. 5 (k). 2) Three of the four adjacent green pixels have close values.
θ2 ¼ ja cj ¼ ja b þ b cj ¼ ja bj þjb cj ¼ θ1 þ θ4 θ5 ¼ jb dj ¼ jb c þc dj ¼ jb cj þ jc dj ¼ θ4 þ θ6
ð2Þ
75
Finally, in order to improve the accuracy of edge detection, and reduce the error generated by selecting the threshold, two arithmetic operators P1 and P2 are designed. P1 is calculated along the possible edge of the current pixel, while P2 is calculated across the possible edge of current pixel. The smaller the value of arithmetic operator, the larger the probability that current pixel being along the corresponding edge. The accurate edge of the current pixel in (1) is smooth, and P1 ¼ P2 ¼0. In (2) and (3), P1 and P2 are designed as follows: N 9 9 P 1 ¼ ∑ ∑ ∑ ðK i ðm; nÞnAðm; nÞÞ i ¼ 1 m ¼ 1n ¼ 1 N 9 9 P 2 ¼ ∑ ∑ ∑ ðLi ðm; nÞnAðm; nÞÞ i ¼ 1 m ¼ 1n ¼ 1
ð3Þ
where K and L are the 9 9 sparse matrices, which are the high frequency filters along and across the possible edge of current pixel, respectively. A is 9 9 CFA image matrix, and B23 is looked as the center. N is the number of the sparse matrices. Two classical high frequency filters, [1 0 –1] and [1 –2 1] are used to constitute matrices K and L. Thus, arithmetic operators P1 and P2 reflect varieties of the same color channels along and across the possible edge of current pixel. The directions shown in Fig. 5(a)–(d) and (i) and (j) are vertical or horizontal, and N¼2. The direction shown in Fig. 5(e)–(h) is diagonal, and N¼ 5. Fig. 6 shows the patterns of sparse matrices as shown in Fig. 5(a)–(j).
θ2 r δ and θ5 r δ can represent all the situations that the values of three adjacent pixels are close. The value distribution of the adjacent green pixels is shown in Fig. 5(a)–(d). 3) Two of the four adjacent green pixels have close values. The edge corresponding to the minimum value of θ1, θ4 and θ6 is looked as the possible edge of current pixel. The value distribution of the adjacent green pixels is shown in Fig. 5(e)–(j).
2.2. The edge detection in demosaicing the missing blue/red channel values at red/blue pixels The edge in demosaicing the missing blue/red channel values at red/blue pixels can be detected by the similar algorithm as the edge detection in demosaicing the missing green channel values at
Fig. 5. The possible value distribution of the adjacent green pixels.
76
Y. Zhang et al. / Optics & Laser Technology 62 (2014) 73–81
red/blue pixels. Let us take B23 in Fig. 1 as the current pixel for example. After sorting the adjacent red pixel, the value distributions of the adjacent red pixels are shown in Fig. 7. The accurate
edge of the current pixel in Fig. 7(i) is smooth, and P1 ¼P2 ¼ 0. The arithmetic operators P1 and P2 in (a)–(h) can be calculated by (3), and the matrices K and L are shown in Fig. 8. The matrices
Fig. 6. The sparse matrices of two arithmetic operators.
Y. Zhang et al. / Optics & Laser Technology 62 (2014) 73–81
77
Fig. 6. (continued)
in Fig. 8(a) are corresponding to the value distribution in Fig. 7(a) and (c). The matrices in Fig. 8(b) are corresponding to the value distribution in Fig. 7(b) and (d). The matrices in Fig. 8(c) are
corresponding to the value distribution in Fig. 7(e) and (f). And the matrices in Fig. 8(d) are corresponding to the value distribution in Fig. 7(g) and (h).
78
Y. Zhang et al. / Optics & Laser Technology 62 (2014) 73–81
Pixels having not the close value Pixels having the close value Fig. 7. The possible value distribution of the adjacent red/blue pixels.
−
−
Fig. 8. The sparse matrices of two arithmetic operators to Fig. 7.
Y. Zhang et al. / Optics & Laser Technology 62 (2014) 73–81
2.3. The edge detection in demosaicing the missing blue/red channel values at green pixels After the missing green channel values at blue/red pixels and the missing blue/red channel values at red/blue pixels are interpolated, the edge in demosaicing the missing blue/red channel values at green pixels can be detected in the same way as edge detection in demosaicing the missing green channel values at blue/red pixels in part 1. Thus, the edge detection for demosaicing all the CFA missing channel values is wholly completed.
3. Experimental results and discussion Due to the better performance than other methods [4,9], three typical gradient edge detection methods, Jonghwa's method in [1], Se-Hwan's method in [4], and Ibrahim's method in [9], are utilized to compare with the proposed method. The data shown in Fig. 9 is extracted from the image shown in Fig. 3(c), the gradients of the adjacent pixels are close, and the true green value of the current pixel (R¼ 126) is 65. The output of Jonghwa's method in [1] is horizontal gradient¼134, vertical gradient ¼170, and the edge of the current pixel is horizontal. The output of Se-Hwan's method in [4] is horizontal gradient¼343, vertical gradient ¼489, and the edge of the current pixel is horizontal. The output of Ibrahim's method in [9] is horizontal gradient¼ 172, vertical gradient ¼34, and the edge of the current pixel is vertical. The output of the proposed method is shown in Fig. 5(g). The mean value in the estimated direction is calculated to interpolate the missing green
79
value of the current pixel. The interpolated value of Jonghwa's method in [1], Se-Hwan's method in [4], Ibrahim's method in [9], and the proposed method are 103, 103, 89, and 59, respectively. It shows that the edge information obtained from the proposed method is closer to the raw data than other methods. In addition to the above experiment, the standard Kodak images shown in Fig. 3 are used to evaluate the performances of different methods. The reason why these images are chosen for the experiment is that they contain considerable color texture to verify the effectiveness of different methods. First of all, according to the pattern shown in Fig. 1, the original CFA images are extracted from the testing images shown in Fig. 3. Then, the different gradient edge detection methods are employed to recognize the edge. Finally, in order to evaluate different gradient edge detection methods fairly, the missing pixel is interpolated with the method in [4]. Let us take Fig. 5(a) for example. The demosaiced green value is denoted as g_es, g1 and g2 respectively are the adjacent green pixels being along the directions corresponding to P1 and P2. g_esðm; nÞ ¼
g 1 ðm; nÞnðp2 ðm; nÞ2 þ 1Þ þ g 2 ðm; nÞnðp1 ðm; nÞ2 þ 1Þ p1 ðm; nÞ2 þ p2 ðm; nÞ2 þ2
ð4Þ
where g1(m,n)¼ 0.5n(bayer(m,n-1) þbayer(m,n þ1)), and g2(m,n) ¼ bayer(m þ1,n). The missing green values shown in Fig. 5(b)–(k) are obtained in the similar way. And the missing red and blue channel values are also interpolated by the same methods. The value of PSNR [10], between the interpolated values and the original values, is used to evaluate different edge detection methods. PSNR is calculated as follows: PSNR ¼ 10 log 10
2552 M N M
ð5Þ
N
∑ ∑ ðf ði; jÞ f 'ði; jÞÞ2
i¼1j¼1
Fig. 9. The Bayer data in image of Fig. 3(c).
where f(i,j) and f(i,j)’ denote the pixel values of the original image and that of the restored image ,respectively, m and n are the size of the testing images. The more the PSNR values, the more exact the gradient edge detection methods. Table 1 presents the quantitative PSNR values obtained from various testing images by using different methods. Bold fonts are used to highlight the best results acquired from different methods in each row. It can be seen from Table 1 that the
Table 1 PSNR values of different methods. Jonghwa's method in [1]
Se-Hwan's method in [4]
Ibrahim's method in [9]
Proposed
PSNR values of the reconstructed green channel by different gradient edge detection methods Image (a) 34.01 34.58 Image (b) 35.61 36.32 Image (c) 33.94 34.18 Image (d) 35.02 35.45 Image (e) 31.54 31.88 Image (f) 35.55 35.79
33.96 35.58 33.17 34.87 31.31 35.11
34.70 35.98 34.38 35.70 32.33 36.03
PSNR values of the reconstructed red channel by different gradient edge detection methods Image (a) 32.56 33.01 Image (b) 34.04 34.22 Image (c) 32.61 33.07 Image (d) 33.98 34.16 Image (e) 30.33 30.89 Image (f) 34.46 34.88
32.14 33.92 32.27 33.54 30.19 34.21
33.23 34.54 33.23 34.31 31.28 35.01
PSNR values of the reconstructed blue channel by different gradient edge detection methods Image (a) 32.90 33.12 Image (b) 35.01 35.12 Image (c) 32.95 33.08 Image (d) 34.33 34.73 Image (e) 30.81 31.03 Image (f) 34.65 34.89
32.42 34.88 32.36 34.07 30.50 34.31
33.55 35.03 33.32 35.02 31.55 35.21
80
Y. Zhang et al. / Optics & Laser Technology 62 (2014) 73–81
Fig. 10. The processing results of the images in Fig. 3.
proposed method gets more PSNR values than other methods, even when the testing images have complex edges. It is noted that the reported PSNR values are only to validate the effect of the proposed edge detection methods, and are not obtained from the final demosaicing results. So the PSNR values in this paper are less than those reported in the references. Besides the objective criterion appreciated by PSNR, the subjective criterion of the processing results of the proposed method is shown in Fig. 10, and that of the processed visual results of different methods is shown in Fig. 11. The images in Fig. 10(a)–(f) depict the processing results of the test images in Fig. 3(a)–(f) respectively. Fig. 11 shows the enlarged sections of the reconstructed full color image from the testing image in Fig. 3(b). It can be concluded from Fig. 11 that the restored full color image used Jonghwa's edge detection method in [1], Se-Hwan's edge detection method in [4] and Ibrahim's edge detection method in [9] generate various degree of color artifacts, which can be seen from the fences portion in Fig. 11. Due to the adopted proposed edge detection method, the reconstructed full color image can achieve comparable or better visual result than other methods. The better performance of the proposed method is mainly attributed to the analysis of the value distribution of the adjacent
pixels. Through ranking the adjacent pixels with the same color, and calculating the absolute differences of sorted pixels, the value distributions of the adjacent pixels are ascertained, then the possible edge of the current pixel is ascertained. Next, the arithmetic operators being along the possible edge and that being across the possible edge are designed to estimate the accurate edge information. Using the above steps, this method follows the intra-channel correction theory better than the traditional gradient edge detection methods, and can distinguish the edge information even when the gradient values of the missing pixel in different directions are close.
4. Conclusion In this paper, a modified gradient edge detection method applied in demosaicing the CFA image is proposed. Due to analysis of the value distribution of the adjacent pixels, the proposed method gives better performance in comparison with the traditional gradient edge detection methods, which only use the relative difference to identify the edge information, and cannot detect accurately the edge from different directions with close gradient. The objective
Y. Zhang et al. / Optics & Laser Technology 62 (2014) 73–81
81
Fig. 11. The enlarged sections of the reconstructed image of the testing image in Fig. 3(b). (a) The enlarged section of the original image; (b) the image reconstructed by Jonghwa's method in [1]; (c) the image reconstructed by Se-Hwan's method in [4]; (d) the image reconstructed by Ibrahim's method in [9]; (e) the image reconstructed by the proposed method.
experimental results indicate that the proposed method can get the higher PSNR values, even when it processes the images with complex edges. The subjective comparison shows that the proposed method can get better visual quality than other methods. It opens a possibility of improving the accuracy of edge detection of the CFA image.
Acknowledgments This paper is supported by the National Natural Science Foundation of China under Grant no.61002019. References [1] Lee Jonghwa, Jeong Taeu, Lee Chuhee. Edge-adaptive demosaicking for artifact suppression along line edges. IEEE Trans Consum Electron 2007;53(3):1076–83. [2] Tsai ChiYi, Song KaiTai. Heterogeneity-projection hard-decision color interpolation using spectral-spatial correlation. IEEE Trans Image Process 2007;16 (1):78–91.
[3] Jimmy Li Jim S, Randhawa Sharmil. Color filter array demosaicking using highorder interpolation techniques with a weighted median filter for sharp color edge preservation. IEEE Trans Image Process 2009;18(9):1946–57. [4] Yun Se-Hwan, Kim Jin Heon, Kim Suki. Color interpolation by expanding a gradient method. IEEE Trans Consum Electron 2008;54(4):1531–9. [5] Hirakawa Keigo, Parks Thomas W. Adaptive homogeneity-directed demosaicing algorithm. IEEE Trans Image Process 2005;14(3):360–9. [6] Zhang Lei, Wu Xiaolin. Color demosaicing via directional linear minimum mean square-error estimation. IEEE Trans Image Process 2005;14 (12):2167–78. [7] Chung KuoLiang, Yang WeiJen, Yan WenMing, Wang ChungChou. Demosaicing of color filter array captured images using gradient edge detection masks and adaptive heterogeneity-projection. IEEE Trans Image Process 2008;17 (12):2356–67. [8] Hore Alain, Ziou Djemel. An edge-sensing generic demosaicing algorithm with application to image resampling. IEEE Trans Image Process 2011;20 (11):3136–50. [9] Ibrahim Pekkucuksen Yucel Altunbasak. Edge strength filter based color filter array interpolation. IEEE Trans Image Process 2012;21(1):393–7. [10] Chung KuoLiang, Yang WeiJen, Yan WenMing, Fuh ChiouShann. New joint demosaicing and arbitrary-ration resizing algorithm for color filter array based on DCT approach. IEEE Trans Consum Electron 2010;56(2):783–91.
