A Fast Non-Local Image Denoising Algorithm A. Dauwe, B. Goossens, H.Q. Luong and W. Philips IPI-TELIN-IBBT, Ghent University, Sint-Pietersnieuwstraat 41, 9000 Ghent, Belgium ABSTRACT In this paper we propose several improvements to the original non-local means algorithm introduced by Buades et al. which obtains state-of-the-art denoising results. The strength of this algorithm is to exploit the repetitive character of the image in order to denoise the image unlike conventional denoising algorithms, which typically operate in a local neighbourhood. Due to the enormous amount of weight computations, the original algorithm has a high computational cost. An improvement of image quality towards the original algorithm is to ignore the contributions from dissimilar windows. Even though their weights are very small at first sight, the new estimated pixel value can be severely biased due to the many small contributions. This bad influence of dissimilar windows can be eliminated by setting their corresponding weights to zero. Using the preclassification based on the first three statistical moments, only contributions from similar neighborhoods are computed. To decide whether a window is similar or dissimilar, we will derive thresholds for images corrupted with additive white Gaussian noise. Our accelerated approach is further optimized by taking advantage of the symmetry in the weights, which roughly halves the computation time, and by using a lookup table to speed up the weight computations. Compared to the original algorithm, our proposed method produces images with increased psnr and better visual performance in less computation time. Our proposed method even outperforms state-of-the-art wavelet denoising techniques in both visual quality and psnr values for images containing a lot of repetitive structures such as textures: the denoised images are much sharper and contain less artifacts. The proposed optimizations can also be applied in other image processing tasks which employ the concept of repetitive structures such as intra-frame super-resolution or detection of digital image forgery. Keywords: Non-local image denoising, repetitive structures
1. INTRODUCTION Digital images are often corrupted by noise. The origin of noise is usually found in the acquisition or transmission process. In this paper, we focus on the classic image denoising problem: the objective is to design an algorithm that can remove additive zero-mean white stationary Gaussian noise, while preserving the original image details and fine structures. Numerous and diverse denoising methods have already been proposed in the past decades, just to name a few algorithms: total variation,1 bilateral filter or kernel regression2, 3 and wavelet-based techniques.4–7 All of these methods estimate the denoised pixel value based on the information provided in a surrounding local limited window. Unlike these local denoising methods, non-local methods estimate the noisy pixel is replaced based on the information of the whole image. The motivation to develop non-local methods is to exploit similar patterns and structures in an image. This relatively new class of denoising methods originates from the non-local means.8, 9 Other denoising methods based on this concept are developed in 3d transform-domain filtering10 and in a trainingbased framework.11 Unlike fractal-based methods, we exploit the similarity of small patches in the same scale, i.e. spatially. Besides repetitivity in texture, we can also find this recurrent property in other parts of the image, such as repetition in different but similar objects, along edges and in uniform areas, etc. The non-local method is Further author information: (Send correspondence to H.Q. Luong) H.Q. Luong: E-mail:
[email protected], Telephone: +32 (0) 9 264 79 66 B. Goossens: E-mail:
[email protected] Image Processing: Algorithms and Systems VI, edited by Jaakko T. Astola, Karen O. Egiazarian, Edward R. Dougherty, Proc. of SPIE-IS&T Electronic Imaging, SPIE Vol. 6812, 681210, © 2008 SPIE-IS&T · 0277-786X/08/$18 SPIE-IS&T Vol. 6812 681210-1 2008 SPIE Digital Library -- Subscriber Archive Copy
found perfectly suitable to some applications: some examples are denoising of scanned text images (availability of multiple repeated characters regardless of their font and the scan orientation) and gigantic satellite images (long roads and a lot of texture provide a huge amount of training data). The concept of repetitive structures has also succesfully been employed in other image processing tasks such as interpolation or super-resolution12, 13 and detecting digital image forgery.14 The non-local means algorithm produces state-of-the-art denoising results, however the method is computationally impractical due to the excessive amount of weight calculations between patches. The complexity of the algorithm is O(M 2 ) with M the total number of pixels in the image. This follows directly from the fact that M weights have to be computed for every pixel in the image. Just to give some idea about computation time: denoising a 512 by 512 image on a P4 3.0 GHz with 1024 Mb ram lasts about 5 hours and 20 minutes. Several papers in literature are dedicated to the acceleration of the non-local means based on preclassifying neighborhoods with average gray values and gradients15 and replacing the mean squared deviation (msd) calculations with an efficient summed square image scheme using fast Fourier transform in a limited search window.16 Using a limited search window centered around the processed pixel, can accelerate the denoising process enormously, however, in some applications we wish to exploit the full search space, i.e. the whole image or video sequence. Similar objects can be located randomly in the image after all or images can consists of texture in most parts, e.g. in textile images. Additionally, the non-local property of the algorithm is partially vanished due to the limited search window and is so related back to local denoising methods. This paper will also focus on the acceleration of the non-local means algorithm, while keeping in mind that the proposed techniques can also be applied for other image processing tasks. The remainder of this article is as follows: in Section 2 the structure of the original non-local means algorithm8 is briefly presented. In Section 3 we will present the modifications that accelerate the denoising algorithm and improve the quality of the denoised results. The modifications discussed in this paper consist of preclassication based on statistical moments and the derivation of the optimal thresholds, exploiting symmetry in weight calculations and accelerating mse computations with a lookup table. Finally, the numerical results are presented in Section 4 and Section 5 concludes this paper.
2. NON-LOCAL MEANS Consider a noisy image V on a discrete grid I (which is typically two or three dimensional): V = {vi |i ∈ I}
(1)
where vi denotes the intensity of the pixel at position i in the image. The additive white stationary noise originates from a zero-mean Gaussian distribution. The denoised intensity vˆ(i) of a pixel i is computed as a weighted average of all the pixels in the image, which can be seen as a linear spatially adaptive filter: wi,j vj j∈I
vˆi =
wi,j
(2)
j∈I
Let Ni denote the vector of pixel intensities of a square window of size B × B (e.g. obtained using column stacking), centered at position i. The weights wi,j in equation 2 depend on the similarity between Ni and Nj . In this paper, we use square windows of predefined size (5 × 5) or (7 × 7). The similarity between neighborhoods is computed by means of msd Ni − Nj 2 , i.e. the Euclidean distance. A particular choice of weight function is given by: 9 Ni − Nj 2 − h2 wi,j = e (3) where h is a constant, proportional to the noise variance σ 2 . The weights wi,j decay at an exponential rate, which results in large weights for a small msd (similar windows) and small weights for a large msd (non-similar windows).
SPIE-IS&T Vol. 6812 681210-2
Computation of the M 2 overall weights, for an image of M pixels, makes the algorithm very inefficient and impractical. In order to reduce the computational complexity of the algorithm, we discuss some improvements in the next Section.
3. A FAST NON-LOCAL DENOISING ALGORITHM An improvement of image quality towards the original algorithm is to ignore the contributions from dissimilar windows. Even though their weights are very small at first sight, the new estimated pixel value can be severely biased due to the many small contributions. Using a preclassification technique, only weights for the most meaningful pixels are computed. This preclassification is a fast way to exclude dissimilar windows, which eventually results in a smaller computation time and even in a better overall denoising quality. Our accelerated approach is further optimized by taking advantage of the symmetry in the weights and by using a lookup table to speed up the weight computations.
3.1 Neighbourhood preclassification The bias caused by the many small contributions of dissimilar windows can be eliminated by setting their corresponding weights to zero. Since there is no use to compute the zero-weights of dissimilar windows, we select similar windows before the evaluation of equation 3. In the same spirit of Mahmoudi et al.,15 we will preclassify image blocks but using a statistical moment-based technique. In this paper, we compare consecutively the first three statistical moments of the pixel intensities located in the windows, namely the mean, the variance and the skewness. From the moment a statistical feature exceeds a predefined threshold, the window will be excluded from further calculations. We will derive the thresholds to decide whether a window is similar or dissimilar for images corrupted with additive white Gaussian noise. 3.1.1 First statistical moment Consider an image V corrupted with additive white Gaussian noise n ∼ N (0, σ 2 ), such that Ni also follows a Gaussian distribution Ni ∼ N (µi 1, σ 2 I), with I the B 2 × B 2 identity matrix and 1 is a B 2 × 1 vector consisting of ones. We estimate the first moment or mean µi of the pixel intensities in the square window Ni as: 2
B 1 µ ˆi = 2 [Ni ]k B
(4)
k=1
where [Ni ]k is the k-th component of the vector Ni . The mean can be efficiently computed by a separable moving average filter. Let us define the following two hypotheses H0 : “the neighborhoods are similar according to the first moment” and H1 : “the neighborhoods are dissimilar according to the first moment” precisely as: ˆ 0 : |ˆ H µi − µ ˆj | ≤ ∆µ
ˆ 1 : |ˆ and H µi − µ ˆj | > ∆µ
(5)
ˆ 1 is decided for window Nj , the corresponding weight is set to zero. It with ∆µ a threshold. If hypothese H ˆi − µ ˆj is Gaussian distributed with mean 0 (because δi,j = −δj,i ) and variance can be easily shown that δi,j = µ 2σ 2 /B 2 . Selecting ∆µ clearly involves a trade-off: if chosen too small, too many vectors Nj are disregarded. If ∆µ is chosen too large, dissimilar vectors (in the MSD sense) may be selected and the preclassification is not beneficial anymore. As a compromise, we set the threshold to the standard deviation of δi,j : √ 2σ (6) ∆µ = B which corresponds to the 68% percentile of the Gaussian distribution.
SPIE-IS&T Vol. 6812 681210-3
3.1.2 Second statistical moment The second central moment or variance of the components of Ni is estimated as: B2
1 2 νˆi = 2 ([Ni ]k − µ ˆi ) B −1
(7)
k=1
Similar as for the first statistical moment, we define the two hypotheses as: ˆ 0 : |ˆ H νi − νˆj | ≤ ∆ν
ˆ 1 : |ˆ and H νi − νˆj | > ∆ν
Analogously, we set the threshold to the empirical standard deviation of the distribution: ⎛ ⎞2 1 ⎝νˆi − 1 ∆ν = νˆj ⎠ M −1 M i∈V
(8)
(9)
j∈V
where M is the number of pixels in the image. 3.1.3 Third statistical moment The skewness of the components of Ni is defined as the third central moment divided by the standard deviation to the third power. An unbiased estimate is given by: 2
B
3
([Ni ]k − µ ˆi ) √ B B 2 − 1 k=1 γˆi = ⎤ 32 B 2 − 2 ⎡ B2 2 ⎣ ([Ni ]k − µ ˆi ) ⎦ 2
(10)
k=1
Analogously to the previous cases, the two hypotheses are defined as: ˆ 0 : |ˆ H γi − γˆj | ≤ ∆γ
ˆ 1 : |ˆ and H γi − γˆj | > ∆γ
We set the threshold to the empirical standard deviation of the distribution of γˆi : ⎛ ⎞2 1 1 ⎝γˆi − ∆γ = γˆj ⎠ M −1 M i∈V
(11)
(12)
j∈V
Further classification using fourth and higher order moments is not very beneficial anymore in most cases, it is even counterproductive. The denoising time using the first four statistical moment increases with a couple of seconds compared to the classification using only the first three moments, that means that the computational cost of the fourth check exceeds the gain of the excluded weight computations.
3.2 Exploiting weight symmetry Most distance measures between vectors, among msd, are symmetric: ||Ni − Nj ||2 = ||Nj − Ni ||2
or
wi,j = wj,i
(13)
Due to the symmetry property of the msd, the weights defined in equation 3 are also symmetric. The total number of weight calculations can be roughly reduced by a factor of two. As these computations are the most time consuming part of the algorithm, this will result in a corresponding acceleration of approximately a factor of two.
SPIE-IS&T Vol. 6812 681210-4
First, we initialize the weight normalization matrix W and the accumulated contribution matrix C (both with the same size as V ) with zeros. When processing pixel i with the contribution from a pixel j (j > i), we add the products wi,j vj and wi,j vi to the accumulated contribution matrix C at the pixel positions i and j respectively. Also the weight normalization matrix W is accumulated at the same pixel positions with wi,j . When processing pixel j, we do not need to compute the contributions or weights of pixels i (i < j) anymore. Finally, we only have to normalize the accumulated contribution matrix C via elementwise dividing by W according to equation 2. Note that this strategy also effects the neighbourhood preclassification step: we do not need to compare the already processed window pairs because of the symmetry of the statistical features.
3.3 Speeding up weight calculations Evaluating an exponential function is still a time consuming task for a cpu. To reduce the amount of these evaluations a lookup table can be used for the weight calculations in equation 3. Since the bandwidth parameter h is constant, the weights can be calculated in advance. The gain can be tremendously large as the image size (M pixels) increases (and even more for video applications), because the number of evaluations is fixed when using a lookup table. Note that the time needed for looking up a weight is shorter than the actual weight computation, but still has to be considered in the evaluation of the total computation time. The lookup table is built by rounding the calculated msd to integer, in that way we have only 2562 entries/evaluations instead of M 2 /α evaluations (α is the gain factor obtained from the preclassification and the exploitation of the symmetry of the weights). In case of regular images, e.g. Lena (see Figure 1), α is approximately 50. Remark that it has not been proved yet that the function used in equation 3 is theoretically nor practically optimal. Another possibility to improve and accelerate is to replace the decaying exponential function with another less complex monotonic decreasing function.
4. NUMERICAL RESULTS The proposed algorithm was implemented in c/c++ and is tested on a P4 3.0 GHz with 1024 Mb ram. Table 1 shows the results of the denoising experiments. For every image the computation time and the psnr value are indicated. Despite our proposed method requires much less computation time than the original algorithm,8 current wavelet denoising techniques6, 7 are still much faster. The quality of the Chessboard image is decreased with 0.2 dB compared to the original algorithm, because the image consists of a lot repetitive structures that can be wrongly classified to dissimilar neighborhoods. Additionally, the preclassification with the skewness feature increases the total computation time with a couple of seconds, that is because of the fact that the original noiseless image consists merely of black and white pixels. We can also conclude from Table 1 that the gain in computation time depends on the image content. In Figure 1, we can observe the improvement in denoising quality can for example be visually observed in the hair which is less smeared out than in the original algorithm. This is expressed in a gain of 1.44 dB in psnr, while our algorithm is about 34 times faster compared to the original non-local means algorithm.8 In Figure 2, we illustrate that our proposed method even outperforms state-of-the-art wavelet denoising techniques6, 7 in both visual quality and psnr values for images containing a lot of repetitive structures such as textures. We can clearly see that our algorithm produces much sharper denoised images with less artifacts.
5. CONCLUSION We have proposed several improvements to the non-local means algorithm which results in an acceleration and even an increase in psnr and visual performance. Using the preclassification based on the first three statistical moments, only contributions from similar neighborhoods are computed. This preclassification is a very fast way to exclude dissimilar windows. Our accelerated approach is further optimized by taking advantage of the symmetry in the weights and by using a lookup table to speed up the weight computations. Our proposed method even outperforms state-of-the-art wavelet denoising techniques in both visual quality and psnr values for images containing a lot of repetitive structures such as textures: the denoised images are
SPIE-IS&T Vol. 6812 681210-5
r1v —
(a)
(b)
p
(c)
(d)
Figure 1. (a) Original 512 × 512 Lena image, (b) Lena image corrupted with additive white Gaussian noise (σ = 20), (c) original non-local means8 (psnr: 29.56 dB) and (d) our proposed method (psnr: 31.00 dB).
much sharper and contain less artifacts. The proposed optimizations can also be applied in other image processing tasks which employ the concept of repetitive structures such as intra-frame super-resolution or detection of digital image forgery.
REFERENCES 1. L. Rudin and S. Osher, “Total variation based image restoration with free local constraints,” in Proc. of IEEE International Conference on Image Processing (ICIP), 1, pp. 31–35, Nov. 1994.
SPIE-IS&T Vol. 6812 681210-6
Table 1. Comparison and influence of the proposed modifications on the computation time and image quality (psnr).
Method/modification Noise/parameters Piˇzurica et al.6 Portilla et al.7 original non-local means8 our proposed method preclassification mean preclassification variance preclassification skewness weight symmetry lookup table
Lena (512 × 512) σ = 20, B = 7, h = 15 11s (32.42 dB) 28s (32.78 dB) 5h 21m 11s (29.56 dB) 9m 25s (31.00 dB) 26m 19s 23m 48s 19m 59s 10m 42s 9m 25s
Peppers (256 × 256) σ = 25, B = 5, h = 20 3s (29.30 dB) 6s (29.65 dB) 13m 43s (27.30 dB) 32s (28.55 dB) 1m 32s 1m 18s 1m 11s 37s 32s
Chessboard (256 × 256) σ = 40, B = 5, h = 23 3s (20.81 dB) 6s (21.23 dB) 13m 43s (24.08 dB) 1m 14s (23.88 dB) 3m 34s 3m 7s 3m 13s 1m 42s 1m 14s
2. C. Tomasi and R. Manduchi, “Bilateral filtering for gray and color images,” in Proceedings International conference on computer vision, pp. 839–846, 1998. 3. H. Takeda, S. Farsiu, and P. Milanfar, “Kernel regression for image processing and reconstruction,” IEEE Transactions on image processing 16(2), pp. 349–366, 2007. 4. L. S ¸ endur and I. Selesnick, “Bivariate shrinkage with local variance estimation,” IEEE Signal Processing Letters 9, pp. 438–441, 2002. 5. B. Goossens, A. Piˇzurica, and W. Philips, “Removal of Correlated Noise by Modeling Spatial Correlations and Interscale Dependencies in the Complex Wavelet Domain,” in Proc. of IEEE International Conference on Image Processing (ICIP), pp. 317–320, (San Antonio, Texas, USA), Sept. 2007. 6. A. Piˇzurica and W. Philips, “Estimating the probability of the presence of a signal of interest in multiresolution single- and multiband image denoising,” IEEE Transactions on image processing 15(3), pp. 654–665, 2006. 7. J. Portilla, V. Strela, M. Wainwright, and E. Simoncelli, “Image denoising using scale mixtures of gaussians in the wavelet domain,” IEEE Transactions on image processing 12(11), pp. 1338–1351, 2003. 8. A. Buades, B. Coll, and J. Morel, “A review of image denoising algorithms, with a new one,” SIAM interdisciplinary journal: Multiscale Modeling and Simulation 4(2), pp. 290–530, 2005. 9. A. Buades., B. Coll., and J. Morel, “A non local algorithm for image denoising,” in Proc. Int. Conf. Computer Vision and Pattern Recognition (CVPR), 2, pp. 60–65, 2005. 10. K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3d transform-domain collaborative filtering,” IEEE Transactions on image processing 16(8), pp. 2080–2095, 2007. 11. M. Aharon, M. Elad, and A. Bruckstein, “The K-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation,” IEEE Transactions on signal processing 54(11), pp. 4311–4322, 2006. 12. H. Luong, A. Ledda, and W. Philips, “Non-local image interpolation,” in Proc. of IEEE International Conference on Image Processing (ICIP), pp. 693–696, 2006. 13. D. Datsenko and M. Elad, “Example-based single image super-resolution: a global MAP approach with outlier rejection,” Journal of Multidimensional System and Signal Processing 18, pp. 103–121, Sept. 2007. 14. J. Fridrich, D. Soukal, and J. Luk´ aˇs, “Detection of copy-move forgery in digital images,” in Proceedings Digital Forensic Research Workshop, 2003. 15. M. Mahmoudi and G. Sapiro, “Fast image and video denoising via nonlocal means of similar neighborhoods,” IEEE Signal Processing Letters 12, pp. 839–842, Dec. 2005. 16. J. Wang, Y. Guo, Y. Ying, Y. Liu, and Q. Peng, “Fast non-local algorithm for image denoising,” in Proc. of IEEE International Conference on Image Processing (ICIP), pp. 1429–1432, 2006.
SPIE-IS&T Vol. 6812 681210-7
(a)
(b)
(c)
(d)
(e) Figure 2. (a) Original 256 × 256 texture image, (b) texture image corrupted with additive white Gaussian noise (σ = 70), (c) Portilla et al.7 (psnr: 16.20 dB), (d) Piˇzurica et al.6 (psnr: 16.99 dB) and (e) our proposed method (psnr: 18.36 dB).
SPIE-IS&T Vol. 6812 681210-8
