31
TechArt: Journal of Arts and Imaging Science, Vol. 2, No. 3, May 2015
Minimizing Ringing Artifact of Image Restoration Using Autocorrelation Jinyoung Kang1, Donggyun Kim2 and Joonki Paik2,* 1
LG electronics / Seoul, Korea Chung-Ang University / Seoul, Korea *Corresponding Author (
[email protected]) 2
Abstract: Linear space-invariant image-restoration algorithms often introduce ringing artifacts near sharp intensity transition areas. This paper presents a new adaptive post-filtering algorithm to reduce the ringing artifacts. The proposed method extracts an edge map of the image using autocorrelation. On the basis of the edge information, ringing artifacts are detected and removed by an adaptive bilateral filter. Experimental results show that the proposed algorithm can efficiently remove ringing artifacts with edge preservation. Keywords: ringing artifact reduction, restoration, autocorrelation, bilateral filter Received Aug. 02, 2015; accepted for publication Aug. 26, 2015; published online Aug. 28, 2015. DOI: 10.15323/techart.2015.08.2.3.31 / ISSN: 2288-9248. where h ( m, n ) represents a point spread function (PSF),
1. Introduction
f ( m, n) denotes the input image, and ( m, n) denotes the
Digital image-restoration techniques have been developed for application in military or aerospace programs. However, when the cold war ended after the 1980s, unlimited competition spread globally, and imagerestoration techniques were then applied to industrial machines and consumer electronics. Recently, smart phones and digital cameras have proliferated, and various portable digital video media become the necessities of daily life. Rapid development in the semiconductor technology also provides economical storage and highperformance processors, which are expected to boost dissemination of image-restoration techniques. The goals of digital image restoration are to determine the degradation factors of the image generated during a series of image processing steps and to improve the degraded image to approach the original quality [1]. Many of the image-restoration methods focus on the improvement of processing speed using fast Fourier transform (FFT) for convolution. However, this approach is based on the assumption that the degradation is linear time invariant (LTI). In the degradation caused by the loss of a high-frequency component, the degraded image can be restored by compensating the lost component. In this case, because the whole image is equally compensated with the lost high-frequency component, a side effect such as a ringing artifact could occur. In this paper, an analysis and a removal method for the ringing artifact are proposed. The image-degradation model is expressed as follows: g (m, n) h(m, n) * f (m, n) ( m, n) ,
(1)
additive white Gaussian noise (AWGN). Because h ( m, n ) is not invertible, finding f (m, n) is considered as an illposed problem [2]. Nevertheless, because FFT significantly reduces the calculation complexity of image restoration, a ringing artifact occurs when circular convolution is used based on the assumption that the input image is a periodic signal. In this paper, the proposed method segments the edge area and the ringing artifact using correlation and then reduces the noise and ringing artifact using a bilateral function. This paper is organized as follows: in Section 2, the mathematical model for the ringing artifact is presented. In Section 3, some background and conventional methods are introduced. In Sections 4 and 5, the proposed method and its experimental results are provided. Finally, Section 6 concludes the paper.
2. Degradation Model A ringing artifact is defined as periodic overshoots and/or undershoots near the edges of an image, and it decreases farther from the edges [3]. As mentioned in Section 1, image restoration is an ill-posed problem. Typical solutions are available for the LTI restoration problem, which include the constrained least-square, Wiener, and Kalman filters [4]. Because these typical methods are derived from a unique inverse function, undesired ringing artifact is necessarily generated even if the original image does not have an artifact. The 2D discrete Fourier transform (DFT) version of the restored image under LTI assumption can be expressed as
Fˆ (m, n) Hˆ (m, n) R(m, n) ,
(2)
J. Kang et al.: Minimizing Ringing Artifact of Image Restoration Using Auto-Correlation
where Hˆ (m, n) represents the frequency response of the
ˆ m, n) represents the DFT of the restoration filter and R( degraded image. The DFT of the additive noise can be expressed as Fˆ (m, n) Hˆ (m, n) H(m, n) F(m, n) Hˆ (m, n)V (m, n) , (3) where H ( m, n ) , F ( m, n) , and V ( m, n) represent the DFTs of the degraded image, the original image, and the additive noise, respectively.
32
In this section, a ringing artifact removal method using post-processing after restoration is proposed. Image restoration is performed before the ringing artifact removal while we assume that PSF is given. The image restoration uses a CLS filter as follows:
Gcls (i, j )
H * (k , l ) H (k , l ) C (k , l ) 2
2
,
(5)
Fig. 1 shows the restoration framework, which includes the proposed ringing artifact removal method.
Fˆ (m, n) F (m, n) [ Hˆ (m, n) H (m, n) 1]F (m, n) Hˆ (m, n)V (m, n) , (4) F (m, n) E f (m, n) EN (m, n) where E f (m, n) [ Hˆ ( m, n) H (m, n) 1]F ( m, n) represents the
signal-dependent error term and ˆ E N (m, n) H (m, n)V (m, n) represents the noise-dependent
error term. In this case, if H ( m, n ) periodically have zeroes, E f (m, n) is considered as a ringing artifact. If it is defined as (m, n)
H (m, n) H (m, n) 1 , the DFT of the
degraded image becomes zero when ( m, n) 1 . Therefore, the zeroes in the DFT of the degraded image and their corresponding frequencies cause the ringing artifact, which represents the ringing artifact model.
3. Theoretical Background Because the human vision system is sensitive to the edge of an image, edge is an important factor in image quality. Preserving the edge of an image is meaningful because it represents the shape, position, and size of an object [4]. Therefore, we need to distinguish the ringing artifact from the edge of an image. Because the intensity of the ringing artifact follows the corresponding edge, an adaptable process is needed. Conventional ringing artifact removal methods are classified into two: post processing without considering the edge and adaptable processing using the edges from the original image. The first method is a simple method to reduce ringing artifacts, but it causes degradation at the edge of the restored image. The other approach effectively reduces the artifact but entails more computational complexity because of the iterative process, and noise amplification or divergence can possibly occur [5] [6]. In the next section, a novel ringing artifact removal method is proposed.
4. Proposed ringing artifact removal
Fig. 1. Restoration framework, including the proposed method.
The proposed ringing artifact removal method is performed in three steps: (i) edge extraction using adaptive threshold; (ii) analysis and classification of the image relative to the edge, ringing, and flat area; and (iii) adaptive filtering according to the image classification. A. Edge extraction The edge is extracted from a restored image with ringing artifact using autocorrelation. Next, the edge is filtered using three different filters with different thresholds from one another. Autocorrelation is used for edge extraction. Because the autocorrelation shows different results in the vertical or horizontal direction, the two results are combined for accurate edge extraction. Fig. 2(a) shows a blurred test image using 7 × 7 uniform PSF with an added 40-dB AWGN and restored by the same PSF. Fig. 2(b) shows the adaptive sum of the horizontal and vertical auto-correlated image.
(a)
33
TechArt: Journal of Arts and Imaging Science, Vol. 2, No. 3, August 2015
(b)
(c)
Fig. 2. Restored image with ringing and auto-correlated image: (a) restored image with ringing, and (b) adaptive sum of the horizontal and vertical auto-correlated image.
Fig. 3. Resulting images of the three different threshold autocorrelations: (a) edges, (b) ringing area near the edges, and (c) weak ringing.
Next, image classification relative to the edge, ringing artifact, flat, and noise follows. Three filters with different thresholds are used in this classification. Figs. 3(a)–(c) show the edge area, strong ringing artifact, and weak ringing with noise, respectively. For accurate threshold determination, edge extraction is iteratively performed until the edge area approaches the strong ringing.
B. Image segmentation and determination of filter size The region extension method adaptively finds the neighborhood of the filter to preserve the sharpness of the edge. The center of the region extension is the same as that obtained by the adaptive edge-preserving filter. The three conditions for the region classification can be expressed as follows:
a2 T ,
p e , n(a ) n(amax ) ,
(a)
(6) (7) (8)
Equations (6) and (8) express the conditions for neighborhood decision for edge-preserved smoothing, which indicate that the variation in intensity in a block should be smaller than threshold T . In (7), e represents a cluster of pixels classified according to the edge using autocorrelation, and p represents the candidate for classification. Finally, (8) prevents a computational overload caused by enlarged filter, where n(a) represents the selected block for region extension and n(amax ) represents the maximum limit of the filter size. C. Filtering of selected area
(b)
When the filter size for a selected area is determined, bilateral filter using an appropriate spatial Gaussian kernel is used for smoothing. The bilateral filter is a non-linear weighted-average filter, which means that if the center and its neighbor have less difference, more smoothing is performed [6]. The output of the bilateral filtering on pixel s can be expressed as follows:
J. Kang et al.: Minimizing Ringing Artifact of Image Restoration Using Auto-Correlation
Js where
p
1 f ( p s)g ( I p I s ) I p , k ( s) p
34
(9)
represents the neighborhood and I s represents
the intensity of pixel follows:
s
.
k ( s)
can be regularized as
k ( s ) f ( p s )g ( I p I s ) ,
(10)
p
where
f
represents the Gaussian filter in the spatial
domain and g is the weighted Gaussian following the intensity of the original image. Therefore, the bilateral filter can effectively reduce the ringing artifact while preserving the edge of the restored image. The average of the Gaussian filter is set to zero to represent a zero gain. The variation and size of the Gaussian filter are used for adaptive filtering.
(b)
5. Experimental Result This section presents the experimental results of the proposed method. In this experiment, we assume that the PSF is given. If the PSF is not given, some methods for blind deconvolution are available [8]-[11].
(c)
(a)
(d) Fig. 4. Experimental results. (a) Blurred image with 7 × 7 uniform blur. (b) Restored image. (c) Resulting image with edge-adaptive Gaussian filtering. (d) Resulting image using the proposed algorithm.
35
TechArt: Journal of Arts and Imaging Science, Vol. 2, No. 3, August 2015
Fig. 4(a) shows a degraded image with 7 × 7 uniform blur and 30-dB AWGN. Fig. 4(b) shows the restored image with the same PSF, and 0.01 is used for lambda. Fig. 4(c) shows the resulting image with the ringing artifact removed using the conventional edge-adaptive Gaussian filter [12]. Fig. 4(d) shows the resulting image using the proposed method. The figures show that the ringing artifact is significantly reduced using the existing method.
from Fig. 4(c) cropped to 204 × 155. (d) Image from Fig. 4(d) cropped to 204 × 155.
Fig. 5 shows the cropped and zoomed version of the images shown in Fig. 4, which shows the image details. Fig. 5(b) shows significant ringing artifacts generated by the image restoration. Compared with Fig. 5(c), Fig. 5(d) shows that the proposed method outperforms the conventional method. In this paper, we manually extracted the locomotion concepts to demonstrate the feasibility of the proposed algorithm. More specifically, we call the gait categorization request to insert the random input images into the gait categorization phase, which provides 100% identification rate under ideal condition.
6. Conclusion (a)
In this paper, a ringing artifact removal method has been proposed. The major problems caused by image restoration are twofold: i) computational complexity and ii) artifact caused by space-invariant processing. The proposed method solved both problems. The experimental results demonstrated that the ringing artifact was significantly reduced with low complexity. In our future work, we expect to implement the Markov random model for classification of a degraded image.
Acknowledgement (b)
This work was supported in part by the Institute for Information & Communications Technology Promotion (IITP) Grant funded by Korea; by the Ministry of Science, ICT, and Future Planning (MSIP), Korea, under the Information Technology Research Center (ITRC) support program (IITP-2015-H8501-15-1018) supervised by the IITP; and by the Technology Innovation Program (Development of Smart Video/Audio Surveillance SoC & Core Component for Onsite Decision Security System) under Grant No. 10047788.
(c)
References [1] M. R. Banham and A. K. Katsaggelos, “Digital image restoration,“ IEEE Signal Processing Magazine, vol. 14, no. 2, pp. 24-41, March 1997. [2] A. Tikhonov and V. Arsenin, Solution of ill-posed Problems, Winston, 1977.
(d) Fig. 5. Cropped images in Fig. 4. (a) Image from Fig. 4(a) cropped to 204 × 155. (b) Image from Fig. 4(b) cropped to 204 × 155. (c) Image
[3] M. Tekalp, H. Kaufman, and W. Woods, “Edge adaptive Kalman filtering for image restoration with ringing suppression,” IEEE Trans. Signal Processing, vol. 37, no. 6, pp. 892-899, June 1989. [4] R. Gonzalez and R. Woods, Digital image processing, 2nd ed., Prentice Hall, 2001.
J. Kang et al.: Minimizing Ringing Artifact of Image Restoration Using Auto-Correlation [5] L. Kah, Y. Weimiao, and L. Shay, “Ringing reduction of image restoration,” Proc. ICSP-2002, pp. 1757-1761, 2002. [6] L. Lagendijk, J. Biemond, and E. Boekee, “Regularized iterative image restoration with ringing reduction,” IEEE Trans. Acoustics, vol. 36, no. 12, pp. 1874-1883, December 1988. [7] T. Bakir and J. Reeves, “A filter design method for minimizing ringing in a region of interest in MR spectroscopic images,” IEEE Trans. Medical Image, vol. 19, no. 6, pp. 585-600, June, 2000. [8] S. Reeves and R Mersereau, “Blur identification by method of generalized cross-validation,” IEEE Trans. Image Processing, vol. 1, no. 7, pp. 301-311, July 1992. [9] R. Fergus, B. Singh, A. Hertzmann, S. Riweis, and W. Freeman, “Removing camera shake from a single photograph,” ACM Trans. Graphics, Matting, Deblurring, vol. 25, no. 3, pp. 787-794, July 2006. [10] T. Stocknam, T. Cannon, and R. Ingebretsen, “Blind deconvolution through digital signal processing,” Proc. IEEE, vol. 63, no. 4, pp. 678-692, April 1975. [11] Y. Chung and J. Paik, “Motion analysis in image sequences and its application to image restoration,” IEICE Trans. Fundamentals of Electronics, Communications, Computer Sciences, vol. E82-A, no. 6, pp. 893-898, June 1999.
Biographies Jinyoung Kang was born in Ulsan, Korea, in 1980. She received the B.S. degree in electronic engineering from Ulsan University in 2003 and the M.S. degree in image engineering from Chung-Ang University, Korea, in 2005. She is currently, a researcher with LG electronics, Korea.
Donggyun Kim was born in Busan, Korea, in 1983. He received the B.S. degree in electronic and electrical engineering from Chung-Ang University, Korea, in 2007 and the M.S. and Ph.D. degrees in image engineering from Chung-Ang University in 2009 and 2015, respectively. He is currently working toward the post-doctoral course in image engineering at Chung-Ang University. His research interests include image restoration, digital auto focusing, lens distortion correction, and image super-resolution. Joonki Paik was born in Seoul, Korea, in 1960. He received the B.S. degree in control and instrumentation engineering from Seoul National University in 1984 and the M.S. and Ph.D. degrees in electrical engineering and computer science from Northwestern University in 1987 and 1990, respectively. From 1990 to 1993, he was with Samsung Electronics where he designed image stabilization chip sets for consumer camcorders. Since 1993, he has been with the faculty at Chung-Ang University, Seoul, Korea, where he is currently a Professor in the Graduate School of Advanced Imaging Science, Multimedia and Film. Since 2005, he has been the head of the National Research Laboratory in the field of image processing and intelligent systems. In 2008, he worked as a full-time technical consultant
36
for the System LSI Division at Samsung Electronics, where he developed various computational photographic techniques, including an extended depth-of-field (EDoF) system. From 2005 to 2007, he served as Dean of the Graduate School of Advanced Imaging Science, Multimedia, and Film. From 2005 to 2007, he was Director of the Seoul Future Contents Convergence (SFCC) Cluster established by the Seoul Research and Business Development (R&BD) Program. From 1999 to 2002, he was a Visiting Professor in the Department of Electrical and Computer Engineering, University of Tennessee, Knoxville. Dr. Paik was the recipient of the Chester-Sall Award from the IEEE Consumer Electronics Society, the Academic Award from the Institute of Electronic Engineers of Korea, and the Best Research Professor Award from Chung-Ang University. He has served the Consumer Electronics Society of IEEE as a member of the editorial board. Dr. Paik is currently serving as a member of the Presidential Advisory Board for Scientific/Technical policy of the Korean government and as a technical consultant for the Korean Supreme Prosecutor’s Office for computational forensics.
