Improvement of Retinex Algorithm for Backlight Image Efficiency Seongsoo Cho, Bhanu Shrestha, Hae-Jong Joo and Bonghwa Hong
Abstract An image in a dynamic range widened by limitations in an image sensor, which has limited size of dynamic range, causes mobile phones and digital cameras to produce brightly saturated image or dark image with less exposure unlike observed with human eyes. This study, which explores solutions to improve contrast imbalance triggering higher global contrast but lower local contrast in images acquired in an environment with wide dynamic range, uses exposure information and edge information for weighted value mapping. The map is applied to image composition process to compare high-brightness proposed in the algorithm to improve contrast imbalance. The comparison indicates greater contrast improvement in the test image than in the original algorithm. Specifically, average growth rate in the original algorithm declined by roughly 27% generating a very large contrast loss while it declined by just 9% in the proposed algorithm resulting in hardly any loss. This comparison and numerical analysis point to the original
S. Cho (&) B. Shrestha Department of Electronic Engineering, Kwangwoon University, 26 Kwangwoon-gil, Nowon-gu, Seoul 139-701, Korea e-mail:
[email protected] B. Shrestha e-mail:
[email protected] H.-J. Joo Department of HUNIC, Dongguk University, 82-1 Pil-dong 2-ga, Jung-gu, Seoul, 100-272, Korea e-mail:
[email protected] B. Hong Department of Information Communication, Kyunghee Cyber University, Dongdaemun-gu, Seoul 130-701, Korea e-mail:
[email protected]
James J. (Jong Hyuk) Park et al. (eds.), Computer Science and Convergence, Lecture Notes in Electrical Engineering 114, DOI: 10.1007/978-94-007-2792-2_55, Ó Springer Science+Business Media B.V. 2012
579
580
S. Cho et al.
contrast improvement performance to be a better choice as proven by the proposed algorithm, which does not incur contrast loss in the HD side and thus, produce a more balanced contrast improvement performance. Keywords Backlight image efficiency
Retinex algorithm
Contrast
MSR
Image
1 Introduction Driven by advances in image sensor technology and digital image processing technology, digital image devices are everywhere these days in user-friendly mobile phones and digital cameras for everybody to use. In particular, explosive demand for mobile phones and portable digital cameras as well as high expectations for high-quality output image are evolving technologies at a speed of light, which in turn motivate research to search for effective solutions to improve color, brightness as part of photo correction [1–4]. Image sensor, which serves as human eyes in mobile phones and digital cameras, accumulates more pixels thanks to advancing semiconductor technologies, but reduced size of area that gets light per pixel is causing more noise, and area researchers are working to improve. Because dynamic range image, which sensor reacts to, is narrower than the range in the actual input image recognized by human eyes, there is inevitable information loss and this creates discrepancy between the image seen with human eyes and that filed with digital camera. To keep information loss to minimum, many studies are underway at this moment to process digital images to compress dynamic range in the input image and expand contrast. Retinex algorithm founded on Retinex Theory, which defines model on humans’ viewpoint, is known to outperform others with better contrast improvement and color reproduction [5–12]. Retinex algorithm uses Gaussian function based on log computation to estimate image illumination and get rid of the estimated illumination component from input image to acquire reflectance, which portrays an object’s characteristics. Final image is produced by restoring color and applying gains and offsets. Images produced through Retinex algorithm are hugely affected by Gaussian Center/ Surround function parameter setting used for illumination component estimation and by weight added when composing the estimated illumination component images. Since uniform or random weighted values are used when composing estimated illumination components for composition, Retinex algorithm depends hugely on parameter setting related to Gaussian center and surround functions, which are used to estimate illumination components. But, it cannot adaptively respond to all different kinds of input images and cannot maximize the unique characteristics of each illumination component images, which cause local contrast loss. Furthermore, backlight images in which the strong illumination component at subject’s rear side enlarges difference of brightness and subsequently darkens subject cause contrast imbalance showing improved contrast in the darker areas
Improvement of Retinex Algorithm
581
but declined contrast in the brighter areas. This study proposes Retinex algorithm, which uses weight map, to overcome such foregoing drawbacks and improve balanced contrast. The proposed algorithm clears input image’s illumination components and retrieves edge and exposure components, which affect contrast, from each reflection component image to make weight map that covers characteristics of each image concerned. Mutually complementing weight map is then applied to each reflection component image holding different characteristics to compose images, which go through color restoration process and gains and offsets for conversion into output scale and final image.
2 Original Retinex Algorithm Retinex algorithm is based on Land’s Retinex theory, which claims that brightness recognized by humans is a multiplication of illumination components and reflection components as proposed by Jobson etal. Its concept is to make a more compressive dynamic range and better contrast by taking out illumination components that impact subject recognition and reflection components that highlight input image’s characteristics to get rid of illumination components’ effect and emphasize reflection components only [13]. Iðx; yÞ ¼ Rðx; yÞ Lðx; yÞ
ð1Þ
In Eq. (1), which defines Retinex algorithm, I (x, y) is input image, R (x, y) is reflection component and L (x, y) is illumination component. The following equation is a log scale conversion of Eq. (1) according to Weber–Fechner’s Law, which establishes log relation between input image’s actual brightness and brightness recognized with human eyes [14]. log Rðx; yÞ ¼ log Iðx; yÞ log Lðx; yÞ
ð2Þ
Retinex algorithm goes through the process of estimating illumination components using Eq. (2) and removing them from input image to highlight reflection components. There are two types of Retinex depending on image channel. One is single scale retinex (SSR) algorithm applied for individual scale and the other is multi scale retinex (MSR) algorithm applied for RGB’s three scales like color image. The block diagram in Fig. 1, which describes Retinex algorithm processes, shows how image with reflection components are gained after SSR processing in each color component. This process is repeated per size of Gaussian Filter used to estimate illumination components. This is followed by MSR processing in which images with the obtained reflection components are given weighted value for composition for gains, offsets and color restoration before getting MSRCR image in the final stage [15]. Equation (3) defines SSR algorithm equation.
582
S. Cho et al.
Fig. 1 Block diagram of Retinex algorithm
Ri ðx; yÞ ¼ log Ii ðx; yÞ log½Fðx; yÞ Ii ðx; yÞ
ð3Þ
where (x, y) is coordinates of pixels and Ii (x, y) is I th color component. i becomes 1,2,3 in RGB image and Ri (x, y) shows SSR result of ith color. The symbol, * represents convolution computation and F (x, y) is Gaussian center and surround functions to estimate illumination components as defined in Eq. (4). Fðx; yÞ ¼ K exp½ðx2 þ y2 Þ=c2
ð4Þ
In Eq. (4), C is Gaussian center and surround constant and K is calculated with Eq. (5). ZZ Fðx; yÞdxdy ¼ 1 ð5Þ In SSR, image quality is largely determined by the value of C. Local contrast and sensitivity in low brightness areas are enhanced while global contrast witnesses loss due to lessened difference of brightness in High Brightness (HB) and Low Brightness (LB) areas when C is low. When the value of C is high, the difference of brightness in LB and HB areas is appropriately handled to improve global contrast but reduced difference of brightness in HB areas and overall decline in edge components in the image causes local contrast loss. Jobson et al. proposed 80 to be the proper value of C and Ref. [6] proposes MSR algorithm, which makes up for the foregoing shortcomings. MSR adds weighted value, which is Gaussian center and surround constant value of C, to SSR output images that applied Gaussian filters with different sizes and the following Eq. (6) is used to combine all the values gained to produce an output image [7]. RMSRi ðx; yÞ ¼
N X k¼1
Wk Rki ðx; yÞ;
N X k¼1
Wk ¼ 1
ð6Þ
Improvement of Retinex Algorithm
583
Fig. 2 Weighted value map-based Retinex algorithm block diagram
In MSR, random or equal weighted values are used to compose each SSR image but in doing so fails to restore contrasts lost due to Gaussian filter. As a result, quality of output image is over-dependent on the size of Gaussian filter and thus fails to completely clear SSR’s drawbacks. Jobson et al. proposes MSRCR algorithm as shown in Eq. (7), which adds color restoration function in MSR, as an alternative. RMSRCRi ¼ Ci ðx; yÞRMSRi
ð7Þ
Although MSRCR produces more vivid color than MSR by applying color restoration functions covering ratio of each color component, there could be color noise in LB areas, which are more sensitive. Hence, characteristics of output image need to be fully comprehended to set proper value in MSRCR.
3 Weighted Map Retinex Algorithm The original Retinex algorithm is heavily affected by constant values that determine valid area of Gaussian filter used to estimate illuminate components and weighted values added when composing reflection components. This study extracts edge and exposure components that affect contrast in reflection component image for adaptive weighted value mapping in output image in order to remove contrast loss incurred by applying random weighted values and proposes an upgraded algorithm, which is applied to the composition process. Figure. 2 is Retinex algorithm block diagram based on the proposed weighted value map. To get reflection components, RGB color space in input image is converted to YCbCr color space to separate image into brightness component and color difference signal, and apply SSR to brightness component. High-pass filter is used in reflection component image to extract edge components and low-pass filter is applied to extract exposure components to compose both components for weighted value mapping. These weighted value maps organized by size of Gaussian filter
584
S. Cho et al.
Fig. 3 Comparison of test images, a Original image, b Original algorithm, c Proposed algorithm
Fig. 4 Comparison of test images, a Original image, b Original algorithm, c Proposed algorithm
used for illumination component estimation go through regularization process before they compose with each SSR image. Final image is produced by taking gain/offset and color restoration process in the image composed with brightness component followed by color restoration and conversion to RGB color space. In YCbCR color space, computation gets smaller if MSR process is applied only to brightness components since output image is separated between brightness components and color difference components. In addition, restoration process gets simple when change rate of brightness components generated by MSR processing is applied to correcting color difference signal since the color is similar to that in the original image. RGB color space in input image should be converted to YCbCr color space since it is easier to extract edge components and exposure components, which make up weighted value map, from brightness signal. The degree of each edge component and exposure component varies in SSR images that removed illumination components estimated with different-sized Gaussian filter. Weighted value map is then organized to highlight characteristics of each SSR image, which is done by extracting the components.
3.1 Extraction of Edge Components In SSR images, noise contained in illumination components amplifies because they execute log computation over the course of removing estimated illumination components. Gaussian Smoothing is used to reduce noise components and Laplacian of Gaussian (LoG) Filter, which emphasizes edge information via
Improvement of Retinex Algorithm
585
Table 1 Local standard deviation Input image Original retinex Bright Dark Bright Ratio Figure. 3 9.27 Figure. 4 7.15 Average Ratio (%)
8.24 6.97 7.33 5.01 -27.39
Proposed retinex Dark
Ratio Bright Ratio
-24.80 10.88 31.97 7.68 -29.97 10.28 40.18 7.07 36.08 -9.09
Dark
Ratio
-17.13 10.29 24.77 -1.04 8.91 21.55 23.16
Laplacian masking kernel, is applied to edge component extraction process. LoG Filter is described in the following Eq. (8). LoGðx; yÞ ¼
x2 þ y2 2r2 x2 þy2 2 e 2r r4
ð8Þ
where r is Gaussian standard deviation constant and Log (x, y) refers to LoG Filter, which deleted regularization coefficient to simplify computation. Equation (9) is the process of simplifying LoG Filter into 5 9 5 masking kernel and executing SSR image and convolution computation to extract edge components.
3.2 Extraction of Exposure Components For this, brightness components distributed in the middle band out of SSR image’s total brightness distribution is assumed to be the most optimal exposure components and Gaussian function is used to ensure exposure components are properly reflected in the final output image. Equation (9) is used to get exposure components as shown below. ! 0 ðYSSR ðx; yÞ 0:5Þ2 we ðx; yÞ ¼ exp ð9Þ 2r2 where YSSR (x, y) is image obtained by SSR processing on Y component of regularized input image and r is Gaussian standard deviation constant. This study applied 0.2 as the constant. we (x, y) is exposure components acquired. Y component value is regularized in 0 * 1 range and 0.5, which is the intermediate value, is deduced to extract exposure components that highlight brightness in the middle band.
3.3 Composition of Each Extraction component For Eq. (10) is definition of weighted value mapping process of regularizing edge components and exposure components extracted from SSR image with the equal value range and multiplying them.
586
S. Cho et al.
Wk ðx; yÞ ¼ wc;k ðx; yÞ we;k ðx; yÞ
ð10Þ
where wc,k (x, y) and we.k (x, y) are regularized edge components and exposure components, and Wk (x, y) is weighted value map.
4 Experiment In this study, images were acquired in an environment with wide dynamic range and the extent of improvement in the original Retinex algorithm and contrast was compared in order to evaluate performance of the proposed weighted value mapbased Retinex algorithm. 15, 80, 250, which are the Gaussian center and surround constants applied to text images to compare degree of impact on images resulting from weighted value setting, were equally applied to the original Retinex algorithm and the proposed weighted value map Retinex algorithm. In the original Retinex algorithm, weighted value was uniformly applied by 1/3 each and HB areas and LB areas in input image were divided to calculate standard deviation in each area as suggested by Jobson et al. Larger standard deviation means brightness of each pixel is evenly distributed instead of being concentrated to certain values, indicating large contrast. Figures 3 and 4 show original images and images processed with the original Retinex algorithm and the proposed algorithm. Table 1 shows standard deviation per HB and LB areas in the original Retinex algorithm and the proposed algorithm. As presented in Table 1, average growth rate relevant to the original image is roughly 13% greater in the original algorithm than in the proposed algorithm. However, average growth rate of the proposed algorithm is also over 23%, which indicates that subjects can effortlessly be identified in LB areas, too. In HB areas, contrast improved more in the proposed algorithm than in the original algorithm as proven by test images. Average growth rate in the original algorithm fell by approximately 27%, which caused significant contrast loss while loss was roughly 9% in the proposed algorithm indicating hardly any contrast loss. The test result reflects improvements in the original algorithm’s LB areas and performance improvement of a lost and unbalanced contrast in HB areas. Unlike the original algorithm, there is no contrast loss in the proposed algorithm’s HB area, which means it has a more balanced contrast improvement performance.
5 Conclusion Original Retinex algorithm shows much better image contrast and outstanding picture quality. While contrast improvement is fairly good in backlight image’s dark areas, color components deteriorate in the bright areas. This study analyzes characteristics of backlight image to make up for such shortcoming in Retinex
Improvement of Retinex Algorithm
587
algorithm and proposes weighted value map Retinex algorithm to improve the performance. Local standard deviation is calculated to evaluate performance of the proposed algorithm and confirm performance improvement in the global contrast. In HB areas, contrast loss in the original Retinex algorithm is around -27.39% while that in the proposed algorithm is -9.09%, which shows 18% performance improvement compared to the original algorithm. In LB areas, performance of the original algorithm and proposed algorithm improved by approximately 36.08 and 23.16%, respectively. This performance analysis result confirms weighted value map-based Retinex algorithm to have a more balanced contrast improvement performance than the original Retinex algorithm and to be more efficient in improving backlight contrast with wide dynamic range.
References 1. Meylan L, Alleysson D, Sustrunk S (2007) Model of retinal local adaptation for the tone mapping of color filter array images. J Opt Soc Am A 24(9):2807–281 2. Choi DH, Jang IH, Kim NC (2006) Color image enhancement based on an improved image formation model. IEEK SP 4(6):65–84 3. Ke C (2005) Adaptive smoothing via contextual and local discontinuities. IEEE Trans Pattern Anal Mach Intell 27(10):1552–1567 4. Park YK, Kim JK (2007) ‘‘A New methodology of illumination estimation/normalization for robust face recognition,’’ IEEE international Conference image Processing, pp.149–152 5. Land E, McCann J (1971) Lightness and retinex theory. J Optical Soc Am A 61(1):11 6. Jobson DJ, Rahman Z, Woodell GA (1997) Properties and performance of a center/surround retinex. IEEE Trans Image Process 6:451–462 7. Jobson DJ, Rahman Z, Woodell GA (1997) A multi-scale retinex for bridging the gap between color images and the human observation of scenes. IEEE Trans Image Process: Special Issue Color Process 6:965–976 July 8. Rahman Z, Woodell GA, Jobson DJ (1996) A comparison of the multiscale retinex with other image enhancement techniques. Special Issue Color Process 6:451–462 9. Su MC, Guo JH, Lin DT, Wang GC (2002) New compensation algorithm for color backlight images. In: neural networks, Proceedings of the 2002 international joint conference, vol 2, Honolulu, Hawaii, pp 1396–1400 10. Choi DH, Jang IH, Kim MH, Kim NC (2007) Color image enhancement based on singlescale retinex with a JND-based nonlinear filter. In: Proceedings of the IEEE international Symposium Circuits and Systems, New Orleans, USA, pp 3948–3951 11. Jang IS, Park KH, Ha YH (2009) Color correction by estimation of dominant chromaticity in multi-scaled retinex. J Imaging Sci Technol, vol 53(5) 12. Kang BH, Jeon CW, Ko HS (2007) K-Retinex algorithm for fast back-light compensation. J IEEK SP 44(2):126–136 13. Horn BKP (1986) Robot vision. MIT Press, Cambridge 14. Agaian SS, Panetta K, Grigoryan AM (2001) Transform-based image enhancement algorithms with performance measure. IEEE Trans Image Process 10:367–381 15. Sobol R (2004) Improving the Retinex algorithm for rendering wide dynamic range photographs. J Electro Imaging 13(1):65–74
