Watershed-Guided Inpainting for Image Magnification Zhaozhong Wang Image Processing Center Beihang University Beijing 100083, China
[email protected]
Abstract A new algorithm to preserve edges in image magnification is proposed. The algorithm is based on an inpainting technique: Blurry edges in the initially magnified image are erased using automatic edge detections, then the erased regions are filled with the filling priority guided by the watershed concept. The algorithm results in sharp patterns along the edges of the initial image, and the discontinuity across the edges is guaranteed. Experiments demonstrate that the algorithm is superior to some popular methods such as edge-directed interpolations and anisotropic diffusions.
1. Introduction This paper proposes a novel inpainting algorithm for image magnification. The inpainting technique [1] is typically used to recover damaged image regions by propagating reliable information from surrounding pixels into the regions. We shall use such a process to recover sharp edges in magnified images, where blurry edges are automatically erased. The key contribution is that we design the watershed-guided inpainting algorithm, in which the concept of watershed is used to guide the filling priority of inpainting. The filling priority is critical for inpainting methods, especially the exemplar-based inpainting [3]. The typical filling priority is defined based on geometric information on the border of erased regions, aiming to propagate image structures continuously into the regions. Then the filling operation is performed by sampling and copying exemplar pixels. Such a process preserves the original sharpness of object edges, but cannot generate edges with higher resolution. So it fails to produce sharper edges in an enlarged image. In our new inpainting algorithm, the filling priority can be stated as: Pixels with a lower initial gradient
978-1-4244-2175-6/08/$25.00 ©2008 IEEE
Y. F. Li Department of MEEM City University of Hong Kong Hong Kong, China
[email protected]
magnitude will have a higher filling priority, where the initial gradient magnitude is computed from an initially interpolated images. By such a priority, the filling process will be finished when all round propagated pixels meet together at the positions with locally maximal gradient magnitude (i.e. on the watershed lines), where discontinuous edges will be formed naturally, see Fig. 1 for illustration. Watershed lines approximate object edges, so using the proposed algorithm will generate in theory the discontinuous pixel values across edges. A number of image interpolation methods with edgepreserving property have been proposed, e.g. the new edge-directed interpolation [5] and the anisotropic diffusion [7]. There are also many single-frame superresolution algorithms designed for such a purpose [8]. But these methods may still generate a certain degree of edge blur, and most of them are time consuming. Our algorithm results in sharper patterns, and is faster based on the data structure of ordered queue. The work most similar to ours may be the watershedbased maximum-homogeneity filtering [4], which uses the watershed transformation to segment the entire image, then averages together small segmented areas with similar patterns. That algorithm also generates sharp edges, but coupled with block effects due to the image segmentation. We do not use the watershed to segment images, but to guide the filling order. Low-frequency parts of image remain unchanged since they will not be erased, for which no block effects arise.
2. Algorithm formulation We shall process digital color images defined as I : D ⊂ Z2 → Zn , p → I(p). The image domain of definition D is formed by D = Θ ∪ Ω, where Θ denotes the region with known pixels and Ω the erased region, which contains blurry edges to be sharpened. Watersheds can be well defined by immersion simulations [9]. The definition can be further generalized
basins [2], then we have the following recursion:
(a)
(b)
(c)
(d)
Figure 1. Illustrating the watershedguided inpainting. (a) Initial interpolated image. (b) Relief for watershed. (c) Pixels around edges are erased. (d) Sharp edges are obtained by inpainting.
to a marker-based case [2]. The proposed watershedguided inpainting uses the concept of marker-based watershed; the marker region is naturally selected as the known region Θ. We now recall some definitions. The geodesic distance dA (p, q) between two pixels p and q in a set A ⊂ D is the length of the shortest path which joins p and q and is totally included in A [9]. Consequently, the geodesic distance dA (p, B) between a point p and a set B ⊂ A is defined as dA (p, B) = min{dA (p, q) : q ∈ B}. Suppose now that B consists of several connected components B1 , B2 , · · · , Bk , the geodesic influence zone izA (Bi ) of a connected component Bi of B ⊂ A is defined as: izA (Bi ) := {p ∈ A : dA (p, Bi ) < dA (p, Bj ), ∀ j = i}. Then the geodesic influence zone of the set B within A is given by izA (Bi ). IZA (B) := i∈[1,k]
To use the watershed concept, there should exist a relief function in the erased region Ω. We adopt the gradient magnitude of the initially interpolated image as the relief, and define it as g : Ω ∪ Θ → [0, hmax ] ⊂ Z. Denote Th (g) as the threshold of g at level h Th (g) := {p ∈ D : g(p) ≤ h}, and let Xh be the section at level h of the catchment
X−1 = Θ,
(1a)
Xh = IZTh (g)∪Θ (Xh−1 ), ∀h ≥ 0,
(1b)
The watershed of g corresponds to the set D\Xhmax , but in this paper we do not extract the watershed lines. Instead, the inpainting priority that pixels with a lower gradient magnitude have a higher priority can be realized based on the recursion in Eq. (1), with the level h increasing from hmin = 0 to hmax . The pixels to be filled in each level belong to the set Xh \Xh−1 . The initial section X−1 has been set as the known region Θ. When we firstly treat the section X0 , we should compute the geodesic influence zone IZT0 (g)∪Θ (X−1 ), for which only pixels on the boundary of X−1 , i.e. pixels in the set ∂Θ := {p ∈ Θ : ∃q ∈ N (p), q ∈ Ω} need to be used. The N (p) denotes the neighborhood of p containing points with shortest path to p (e.g. the 3 × 3 neighborhood on the 4-connected grid). We pick up from ∂Θ the points p with g(p) = 0, i.e. the points p ∈ ∂X−1 ∩ T0 (g). Then we seek in the neighborhood N (p) the unfilled points q ∈ N (p) ∩ T0 (g) ∩ Ω, thus we have q ∈ IZT0 (g)∪Θ (X−1 ) according to the definition of geodesic influence zones. These points q should be inpainted. After inpainting the points q, we seek again in the neighborhood N (q) the unfilled points s ∈ N (q) ∩ T0 (g) ∩ Ω and fill them. Such a process is repeated until all points in the set IZT0 (g)∪Θ (X−1 ) ∩ Ω are inpainted. Then we set h = h + 1 and handle the higher level. The above paragraph illustrates the inpainting priority: Those points q in {q ∈ N (p) ∩ Th (g) ∩ Ω : p ∈ ∂Xh−1 ∩ Th (g)} (2) with h the current lowest level, should be inpainted first. Now we consider how to inpaint these points using known pixel values. As the simplest case, we can fill the points in (2) using I(q) = I(p), which will rigorously results in a step edge, similar with that in Fig. 1(d). But this may be unsuitable since there are non-step edges in real images, and the inpainting result is probably ”oversharpened”. We use a simple inpainting scheme to overcome the problem: Taking into account the pixel values in the initially magnified image I0 , we formulate the filling for the points in (2) and other erased points as I(q) = αI(p) + (1 − α)I0 (q),
(3)
where α ∈ [0, 1] is a coefficient. If α = 1, we obtain the sharpest edge (a rigorous step edge); if α = 0, the sharpening effect is completely bypassed. Adjusting α between 0 and 1 may obtain inpainting results with more satisfactory subjective qualities.
3. Implementing issues The proposed algorithm consists of four steps: initial interpolation, relief computation, automatic erasion, and pixel inpainting, see Fig. 1. The initial interpolation can be performed by a simple method, such as bilinear or cubic. Final edge locations are affected by the initial gradient magnitude, so it is better to choose an initial interpolation whose largest gradient magnitudes are near object edges. If the image magnification factor is large, we commonly use the cubic interpolation. The relief can be set directly as the initial gradient magnitude. For the color gradient computation, using the eigenvalue decomposition of tensor structures is more proper since it takes into account the coupling among color channels. But this method is time consuming. We simply use in this paper the gradient of the intensity channel to speed up the computation. The step of automatically erasing can be performed by at least two approaches: (1) We extract edges in the initial image using an edge detector such as Canny, then dilate the edge lines to erase surrounding pixels. (2) We directly segment the relief image using an automatic thresholding method such as Otsu’s [6], then erase pixels with reliefs larger than the threshold. We find that the final inpainting results are robust to these erasing operations. Note that points on the magnified grid corresponding to the original pixels should be kept unerased. In the inpainting step, the recursive operation in Eq. (1) can be implemented effectively using the ordered queue structure [2], with a pipeline almost the same as the marker-based watershed transformation, except that we do not segment the image, but fill pixels in the erased image regions using Eq. (3).
4. Results and conclusion We test the algorithm using color images in Fig. 2. Theoretically the magnification factor for the proposed algorithm can be of any number as long as the initial interpolation can. Here we set the magnification factor as 8 (= 23 ) so as to compare with existing algorithms such as [5]. Fig. 3 demonstrates the inpainting steps, where the initial magnification in Fig. 3(a) is performed by the cubic interpolation, and the erased image in Fig. 3(c) is generated from the dilation of Canny edge. The inpainted image in Fig. 3(d) contains sharp edges. In Fig. 4, we compare our algorithm with the bilinear interpolation, Xin Li’s method [5] and the Tschumperle-Deriche method [7]. It can be seen that our algorithm has the best performance on preserving sharp edges among these methods. In the experiments,
we set the coefficient in Eq. (3) as α = 0.8 to reduce the over-sharpness effect. Though there are four steps, our algorithm is much faster than those in [5] and [7]. In conclusion, the proposed algorithm performs well in image magnification to preserve sharp edges. It guarantees in theory the discontinuity across edges. Lowcontrast regions far from edges remain unchanged during the inpainting to reduce computing loads. The algorithm performance is insensitive to automatic erasing operations, but depends on the gradient of the initial interpolated image. Further work to increase the algorithm robustness against the initial interpolation will be studied.
Acknowledgements This work was supported in part by the National High Technology Research and Development Program of China under Grant 2006AA11Z232, and by the Research Grants Council of Hong Kong under Project CityU117106.
References [1] M. Bertalmio, G. Sapiro, V. Caselles, and C. Ballester. Image inpainting. In Proc. of SIGGRAPH 2000, pages 417–424, New Orleans, USA, July 2000. [2] S. Beucher and F. Meyer. The morphological approach to segmentation: the watershed transformation. In E. Doughert, editor, Mathematical morphology in image processing, chapter 12, pages 433–481. Marcel Dekker, 1993. [3] A. Criminisi, P. Perez, and K. Toyama. Region filling and object removal by exemplar-based image inpainting. IEEE Trans. Image Processing, 13(9):1200–1212, September 2004. [4] M. Hansen and W. Higgins. Watershed-based maximumhomogeneity filtering. IEEE Trans. Image Processing, 8(7):982–988, July 1999. [5] X. Li and M. Orchard. New edge-directed interpolation. IEEE Trans. Image Processing, 10(10):1521–1527, October 2001. [6] N. Otsu. A threshold selection method from gray-level histograms. IEEE Trans. Systems, Man, and Cybernetics, 9(1):62–66, 1979. [7] D. Tschumperle and R. Deriche. Vector-valued image regularization with PDE’s: A common framework for different applications. IEEE Trans. Pattern Analysis and Machine Intelligence, 27(4):506–517, 2005. [8] J. van Ouwerkerk. Image super-resolution survey. Image and Vision Computing, 24(10):1039–1052, October 2006. [9] L. Vincent and P. Soille. Watersheds in digital spaces: An efficient algorithm based on immersion simulations. IEEE Trans. Pattern Analysis and Machine Intelligence, 13(6):583–598, June 1991.
Figure 2. Original color images from the Kodak database.
Figure 3. Image magnification (local). From left to right: (a) the initial interpolation, (b) the gradient magnitude, (c) the automatically erased and (d) the finally inpainted images.
Figure 4. Image magnification comparisons. Columns from left to right: bilinear interpolation, Xin Li’s method [5], Tschumperle-Deriche method [7], and the watershed-guided inpainting.
