Image Inpainting Using Wavelet-based Inter- and Intra-scale Dependency Dongwook Cho1,2 and Tien D. Bui1 1. Dept. Computer Science & Software Engineering, Concordia University, Canada 2. Genetec Inc., Montreal, Canada
[email protected],
[email protected] Abstract Image inpainting or completion is a technique to restore a damaged image. Recently various approaches have been proposed. Wavelet transform has been used for various image analysis problems due to its nice multiresolution properties and decoupling characteristics. We propose to utilize the advantages of wavelet transforms for image inpainting. Unlike other inpainting algorithms, we can expect better global structure estimation of a damaged region in addition to shape and texture properties.
1. Introduction One of the major concerns in image processing is estimation of ‘pixel’ values. For example, interpolation or resizing is to estimate plausible pixel values located between known ones while denoising or deblurring is to estimate clean pixel values from corrupted ones. On the other hand, image inpainting, completion, dis-occlusion or object removal is to estimate spatially connected pixels in a region. The main challenge is to estimate the unknown pixel values that satisfy human visual system (HVS). In other words, we may obtain a reasonable estimation by simulating how HVS behaves when an unknown region in an image is exposed. For example, let us consider Figure 1(a). At first, a triangle without edge is perceived as a
front area. Then three small objects with one white contour object can be imagined like Figure 1(b). In order to obtain this result, human brain performs perceptual interpolation based on Gestalt principles such as good continuation, proximity, similarity, closure, symmetry, and so on [1]. Wavelet transform has been used as a good image representation and analysis tool mainly due to its multiresolution analysis, data separability, compaction and sparsity features in addition to statistical properties. In this paper, we propose a wavelet-based approach for image inpainting. Multi-resolution analysis of wavelet transform can be helpful to predict coarse-to-fine image structure. Separable data into low frequency scaling coefficients and high frequency wavelet coefficients make it possible to analyze both structure and texture. In the above example in Figure 1, considering both global and local structure is necessary. Also it is important to analyze texture and detailed patterns for natural images. Wavelet is capable of treating these elements altogether. This paper is organized as follows. In the next section, previous works on this topic is reviewed. In Section 3, our proposed wavelet-based algorithm is described. Experimental results and discussion on the method are discussed in Section 4. Finally we conclude our discussion in Section 5.
2. Problem Definition and Previous Works We first define inpainting problem: given an image u with unknown regions Γ , find an ideal image v :
⎧uˆ ( x, y ) if ( x, y ) ∈ Γ v ( x, y ) = ⎨ ⎩u ( x, y ) otherwise
(a) (b) Figure 1. Modified example of Kaniza [1]: How do we complete a gap (triangular region) for anomalous contours and objects?
978-1-4244-2175-6/08/$25.00 ©2008 IEEE
(1)
Now the goal is to find good estimation uˆ in Γ . Many inpainting solutions are achieved by analyzing image models and utilizing them to obtain probable estimation. In [2], Bertalmio et al. proposed an evolution equation using partial differential equation (PDE) image model inspired by restoration artist’s inpainting process as follows:
u n +1 ( x, y ) = u n ( x, y ) + Δt
∂u n ( x , y ) , ∂t
where ( x, y ) ∈ Γ and ∂u n represents the inner product ∂t
between the propagation direction and the change of propagated information. Also, some variational models such as total variation (TV) [3], Euler elastica [4], or Mumford-Shah [5] models have been adopted for obtaining good approximations from mathematically neat image modeling. The approach can solve an inpainting problem by formulating an energy functional and obtaining corresponding PDE formulation. PDE-based iterative algorithms are usually able to keep good continuation smoothly. However, broken edge estimation with large gap could fail and detailed texture surface is not easily reproducible. Exemplar-based approach, which originates from texture synthesis, is capable of generating detailed texture on an estimated object surface. In addition, Criminisi et al. in [6] considered edge structure by giving high filling priority on a strong edge with isophote direction. The method works well for a large unknown region. Since the estimated region is synthesized with texture patches from known regions, the complete image v is homogeneous both visually and statistically. But it fails with region with complex global structure and curved edges. In addition, overlapped patches may cause visual seams. In [11] a method based on combination of the Mumford-Shah model and the exemplar approach has been introduced which produces relatively good results. Image decomposition-based approaches have been also proposed. These methods can be simply explained by Meyer’s image decomposition model [7], which can be obtained by minimizing the functional ∫ ∇ s + λ w ,
where s is bounded variation component obtained from TV minimization and w is oscillatory part of the image. In other words, s is suited for image structure and w for texture and noise. The image decomposition idea has been used in [8] and [9]. Some other approaches perform graph analysis such as graph-cut or belief propagation to predict better image structure. Texture filling is usually considered to complete the structured image [10].
3. Wavelet-based Inpainting Using Inter and Intra-scale Dependency In our proposed algorithm, wavelet and scaling coefficients are estimated after wavelet decomposition of a given incomplete image. Discrete wavelet decomposition for each scale can be formulated as follows:
s j +1 = h j * h j ∗ s j , w LH j +1 = g j * h j ∗ s j ,
HH w HL j +1 = h j * g j ∗ s j , w j +1 = g j * g j ∗ s j ,
where
h j and g j represent low-pass (scaling) and
high-pass (wavelet) filters,
s j and w j are scaling and
wavelet coefficients for scale j respectively. Wavelet transform is applied to the given original image u and image of the mask m. The mask is a user defined area that covers the inpainting region. Inpainting process is performed in the wavelet domain by predicting both scaling and wavelet coefficients from coarse to fine scales in the unknown regions. As shown in Fig. 2, each coefficient in the wavelet domain is dependent on neighboring coefficients in the same subband and corresponding coefficients located in the other subbands. Therefore, it is important to consider both inter and intra-scale dependency. On the other hand, if the dependency of these coefficients is not considered, it would be difficult to estimate visually meaningful coefficients. Once forward transform has been carried out for both the original image and the image of the mask up to level L (the coarsest level); scaling coefficients of the coarsest scale need to be estimated for global image structure. The coarsest scaling subband is required to be filled first as depicted in Fig. 3. Since the coarsest scaling coefficients have the same properties as the low-pass filtered smooth image, any inpainting algorithm described in the previous section could be applied here. We have simply applied the total variation (TV)-based algorithm used in [4] since the approach works well in smooth image domain. After filling the scaling subband, three wavelet subbands (HL, LH, and HH) in the coarsest level are filled simultaneously to avoid visually annoying artifacts after the inverse transform. In order to complete the wavelet subbands, exemplar-based scheme proposed in [6] has been used. In other words, the unknown areas in the wavelet subbands are filled by example patches of the same subband based on a priority map (see [6]) which is obtained from the scaling subbands of u and m. In this case, example patches in three wavelet subbands should LL3
HL3
LH3
HH3
LH2
HL2 HL1 HH2
LH1
HH1
Figure 2. Inter-scale dependency of wavelet coefficients
be geometrically corresponding to each other to reduce visual artifacts. An example patch should resemble a patch area in the unknown region, i.e. the difference or distance between a patch including the unknown area and an example patch should be minimized. For this, we propose the following distance measure: d e d e d = α w LH − w LH + β w HL − w HL d e + γ w HH − w HH + sd − se e
(1)
d
where wLH and wLH are example patch and destination patch in the LH subband respectively. The destination patch
d wLH includes the unknown area in the LH subband,
similarly for the other subbands.
s e and s d are example
patch and destination patch from the scaling subband. Finally α, β, and γ are parameters to be determined. By default, they are set to 1. Once all the coefficients are estimated, inverse transform for one level is performed to obtain the approximation of the next finer scale. In the subsequent scales, as in the coarsest scale, reconstructed scaling subband is used for the estimation of wavelet subbands. The scaling subband can also be corrected by blending with the example patch corresponding to the patches decided for the wavelet
coefficients. Blending can be done linearly, i.e.
s d ← α s d + (1 − α ) s e where α ∈ [0,1] is a parameter.
(2)
Summary of proposed algorithm is described as follows: 1. Apply wavelet transform to the given image u. 2. Apply wavelet transform to the image of the masks m. 3. Set scale parameter j = L (coarsest level). 4. While j > 0 (from coarse to fine), a. If j is L, fill the unknown area of scaling subband by using PDE or variational inpainting approach (e.g. [4]). Otherwise, scaling coefficients have been already reconstructed in 4.d of the previous scale. b. Estimate wavelet coefficients in the wavelet subbands by finding closest texture patches that can minimize distance d given by Eq. (1). Geometrically corresponding scaling coefficients to the estimated wavelet coefficients are also stored. c. The reconstructed scaling coefficients are blended with the estimated scaling coefficients in 4.b. Blending function is in Eq.(2). d. Perform inverse wavelet transform to obtain scaling subband of the next finer scale j-1 e. j ← j − 1. 5. (Optional) Post-processing (e.g. blending)
4. Experimental Results For the experiments, we assume that masks for the unknown regions are defined manually. The mask image (m) is transformed by wavelet and the scaling subband for each level is used for the inpainting of the subbands of the image at the same level. Different wavelets could be considered. In our experiments, we simply use the Haar basis with L = 2 or 3. The proposed algorithm has been applied to various gray and color images. In Fig. 4, inpainting results of some color images are shown. Both texture patterns and image structure are well-preserved. We also compared some existing algorithms such as PDE-based approach [2], total variation [4], exemplarbased approach [6], and decomposition and sparse reconstruction [8]. Fig. 5 shows inpainting results from these different methods. A narrow region like a golf driver can be smoothly filled by most of the methods.
Figure 3. Inpainting in wavelet domain (one level): scaling and wavelet subbands are estimated and reconstructed.
However, for large regions, PDE-based or variational algorithms result in smooth surface even for textured background in addition to diminution of edge sharpness. Exemplar-based approach mostly produces plausible textures in the unknown area, but some abrupt changes between patched areas are observed. Our proposed method also has slight seam from time to time, but overall quality in terms of image structure and texture looks comparatively better than the other algorithms.
5. Conclusion In this paper, we have presented wavelet-based approach for image inpainting. Wavelet is a decent mathematical tool for estimating global structure of image and texture analysis thanks to its multi-scale analysis and separability features. The proposed method takes advantages of wavelet by utilizing inter- and intra-scale dependency for maintaining image structure and texture quality.
6. References [1] G. Kaniza, “Organization in Vision: Essays on Gestalt Perception”, Praeger, New York, 1979. [2] M. Bertalmio, G. Sapiro, V. Caselles, and C. Ballester, “Image inpainting”, SIGGRAPH, pp. 417-424, 2000. [3] S. Masnou and J.-M. Morel, “Level lines based disocclusion”, ICIP, pp.259-263, 1998.
592, 2002. [5] S. Esedoglu and J. Shen, “Digital inpainting based on the Mumford-Shah-Euler image model”, European J. Appl. Math., 13, pp. 353-370, 2002. [6] A. Criminisi, P. Perez, and K. Toyama. “Region filling and object removal by exemplar-based image inpainting”, IEEE Trans. Image Processing, 13(9):1200-1212, 2004. [7] Y. Meyer, “Oscillating Patterns in Image Processing and Nonlinear Evolution Equations”, vol. 22 University Lecture Series, American Mathematical Society, Philadelphia, 2005. [8] O. G. Guleryuz, “Nonlinear approximation based image recovery using adaptive sparse reconstructions and iterated denoising: Part I – theory”, IEEE Trans. Image Processing, 15(3):539-554, 2006. [9] M. Elad J.-L. Starck and D.L. Donoho, “Image decomposition via the combination of sparse representation and a variational approach”, IEEE Trans. Image Processing, 14(10):1570-1582, 2005. [10] N. Komodakis, G. Tziritas, “Image completion using efficient belief propagation via priority scheduling and dynamic pruning”, IEEE Trans. Image Processing, 16(11):2649-2661, 2007. [11] X. Du, D. Cho, T. D. Bui, “Image Inpainting and Segmentation using Hierarchical Level Set Method”, CRV, 52, 2006.
[4] T. F. Chan, S. H. Kang, and J. Shen, “Euler’s elastica and curvature based inpaintings”, SIAM J. App. Math., 63(2):564-
Figure 4. Inpainting results of proposed method: original (top row), masked images (middle row) and completed (bottom row) images.
(a) given image
(b) PDE-based [2]
(c) total variation [4]
(d) exemplar-based [6]
(e) sparse reconst. [8] (f) proposed Figure 5. Inpainting results of different approaches.
