PROJECTION BASED IMAGE AND VIDEO INPAINTING USING WAVELETS Kedar A. Patwardhan and Guillermo Sapiro Department of Electrical and Computer Engineering, University of Minnesota, Minneapolis, MN-55455, USA.
[email protected] [email protected] ABSTRACT In this paper, we present a technique for automatic color image inpainting, the art of modifying an image-region in a non-detectable form. The main algorithm is based on the theory of projections onto convex sets (POCS). The image and its wavelet transform are projected onto each other after applying suitable constraints in each domain. This technique exploits the frequency-spatial representation provided by wavelets and utilizes the correlation between the damaged area in the image and its neighborhood. The resulting restored area is homogeneous with its surrounding and preserves the aesthetics of the image. The same technique is used for simple video restoration problems. Video frames are stacked and treated as a 3-D volume, making a natural use of inter-frame correlation. 1. INTRODUCTION In recent years there has been a great amount of use of the visual media for commercial and entertainment purposes. This has lead to an increasing emphasis on restoration of old film archives and photographs. With wireless networking and data exchange becoming the ‘in-thing’ today, the need for an automatic and fast technique to restore image blocks lost during transmission, is becoming more and more hard-felt. Film post-production, also involves a lot of image inpainting for removal of artifacts and to generate stunning visual effects. This task is conventionally performed manually, requiring time and skill. These problems have motivated us to search for automatic techniques for still-image and video inpainting. Many useful techniques to address the afore mentioned scenarios have been proposed in recent years, e.g., [1, 2, 3, 4, 5, 6, 11, 12, 13, 14, 15, 17]. The success of these techniques varies according to the type of image and size and characteristics of the lost data. These methods, with the exceptions of [5, 17] and texture synthesis algorithms such as [8, 7, 10, 18, 19], run into trouble when employed on textured images, and only [5] explicitly deals with parts of textured images containing prominent edges.
The main inspiration behind our paper has been the technique employed by Hirani and Totsuka [11] for removal of wires and scratches from still images. It is a hierarchical and prototype based interactive application of the POCS theory [20]. The disadvantage of this technique, though, is the fact that it requires a human operator to select the prototype window, which is demanding in terms of time and skill required. Till now this algorithm has had demonstrable success in case of thin and elongated wires and scratches on still-images. In this work, we combine the POCS ideas of [11] with those in the previously mentioned inpainting and texture synthesis works, and instead of the user identifying the prototype window, we get the required information from nearby pixels. All this is done in the Wavelet domain and not in Fourier domain as in [11], thereby improving the handling of texture.
2. ALGORITHM In this paper, we try to exploit the multi-resolution property of wavelets, whereby, we can dispense with the human operator required in case of [11]. This algorithm also works for larger lost blocks. In the technique proposed by Hirani and Totsuka [11], a human operator takes a look at the neighborhood of the lost block and decides upon the best matching neighborhood which later acts as the prototype window. In our algorithm, wavelets perform this very task, automatically and considerably fast. The basic assumption when restoring any lost image block is that the lost block contained a continuation of data that surrounds it. Wavelets, make it easy to determine the frequency content of the neighborhood. We put frequency domain constraints on the lost block, based upon the frequency content of its neighborhood in the wavelet domain. The spatial domain constraints help us to get a valid image from the inverse wavelet transform. In addition, we also make sure that there is no discontinuity at the border of the newly inpainted area.
To appear in Proc. ICIP2003, Sept 14-17, 2003, Barcelona, Spain
c IEEE 2003
3. IMPLEMENTATION AND RESULTS
Our algorithm is based on Projection Onto Convex Sets (POCS). POCS theory was introduced by Papoulis and Gerchberg [9, 16] to reconstruct band-limited signals by alternate projections onto transform and signal domains and applying constraints in each domain. In a similar fashion, our algorithm iteratively converges to an image which closely resembles the original and preserves its aesthetical characteristics. The new pixels that are generated using the POCS technique are:
The results shown, have been generated using ‘Wavelet Toolbox’ in MATLAB. The algorithm is applied separately on each color channel and the results are combined to get a color image. It should be noted that this does not result in appearance of previously non-existent colors in the image. The algorithm usually converges in less than 25 iterations. On a Pentium II, 400 MHz machine, it takes less than 5 minutes to get the inpainted color image. We have used the simple ‘Daubechies One’ (db1) wavelet for our implementation. Note that in order to improve the results on textured areas, other decompositions can be used, e.g., the steerable pyramid [10]. The image on the left in Fig.1, shows some objects placed on a pool table, while the image on the right shows the inpainted image with the pouring arm of the teapot and the red ball removed. Notice how the curve in the shape of the tea-pot is preserved, and the texture generated in place of the red ball is homogeneous with its surrounding. In Fig.2, the occupants of the boat and the boy on the shore are removed. Fig.3 illustrates the behavior of our algorithm on random textures. For inpainting a video sequence, we select the object or artifact to be removed in each frame and then stack the frames as a 3-D volume. The damaged areas in each frame are then considered to be a hole in the 3-D volume. We then apply our algorithm, and use the 3-D Wavelet Transform and its inverse for the projections onto transform and image domains respectively. The results for video inpainting can be viewed at : www.ece.umn.edu/users/kedar/icip2003/icip results.htm.
(a) as sharp as the surrounding area, (b) maintain continuity of prominent edges running across the scratch and, (c) have a texture matching the surrounding texture. The process of projection onto image and wavelet domains and consequent application of constraints can be summarized as : In+1 = Ci Pwi Cw Piw (In )
(1)
where, In+1 : Inpainted window after n + 1 iterations. In : Current repair window. Piw : Projection from image domain onto wavelet domain (the wavelet transform). Cw : Constraints in the wavelet domain. Pwi : Projection from the wavelet domain onto image domain (inverse wavelet transform). Ci : Constraints in the image domain to get pixel values between 0 and 1.
4. CONCLUSION AND FUTURE SCOPE
The resolution of the image dictates the number of wavelet decomposition levels that can be used. The wavelet ‘approximation’ in the last level of the decomposition is considered to be the DC level and is left unaltered, in order to maintain the overall brightness of the image. For every wavelet coefficient corresponding to the lost or damaged pixel, we find the maximum and minimum of the wavelet coefficients in the known neighborhood. The coefficient corresponding to the lost or damaged pixel is then constrained to be between these maximum and minimum values. We call this the constraint Cw . This constrained wavelet decomposition is then projected onto the image domain (Pwi ). The known pixels in the neighborhood are restored to their original value and the newly grown pixels in the region of inpainting are constrained (Ci ) to get a valid image (In+1 ). The constraints in this algorithm preserve known pixel values and enforce band-limits in the wavelet domain, based upon the coefficients in the neighborhood. This can be thought of as making the least possible change to the image in the signal and transform domain such that it satisfies the above mentioned criteria (a),(b) and (c).
We have demonstrated the effectiveness of using POCS theory in conjunction with wavelets for image restoration. A particular wavelet basis and a projection constraint has been used to illustrate the ideas. The results can be significantly improved using, for example, steerable pyramids and vectorial constraints. This algorithm is automatic and considerably fast. It works well with color images and on relatively large areas of data loss. This algorithm can be used for structure as well as texture inpainting, provided, both can be characterized in the wavelet domain. The speed of this algorithm makes it a possible candidate for restoring images corrupted due to transmission over fading channels. An important application of this algorithm can be in film post production for removing artifacts. It can also be used for restoring old films and photographs. We have got encouraging results when implementing the same algorithm for video inpainting using 3-D wavelets. In conjunction with optical flow data, our algorithm could be a promising technique for video inpainting. 2
Acknowledgments
[11] A. Hirani and T. Totsuka, “Combining frequency and spatial domain information for fast interactive image noise removal,” Computer Graphics, pp. 269-276, SIGGRAPH 96, 1996.
This work was partially supported by the Office of Naval Research, the Presidential Early Career Awards for Scientists and Engineers (PECASE), and a National Science Foundation CAREER Award. GS thanks Prof. Marcelo Bertalmio and Prof. Vicent Caselles, with whom all this work on image inpainting started. We also thank Shantanu Rane for his comments on this paper.
[12] A.C. Kokaram, R.D. Morris, W.J. Fitzgerald, and P.J.W. Rayner, “Interpolation of missing data in image sequences,” IEEE Transactions on Image Processing 11:4, pp. 1509-1519, 1995.
5. REFERENCES
[13] S. Masnou and J. Morel, “Level-lines based disocclusion,” IEEE Int. Conf. Image Processing, October 1998.
[1] C. Ballester, M. Bertalmio, V. Caselles, G. Sapiro, and J. Verdera, “Filling-in by joint interpolation of vector fields and grey levels,” IEEE Trans. Image Processing 10, pp. 1200-1211, August 2001.
[14] M. Nitzberg, D. Mumford, and T. Shiota, Filtering, Segmentation, and Depth, Springer-Verlag, Berlin, 1993. [15] J. M. Ogden, E. H. Adelson, J. R. Bergen, and P. J. Burt, “Pyramid-based computer graphics,” RCA Engineer 30(5), pp. 4-15, 1985
[2] C. Ballester,V. Caselles, and J. Verdera, “Dissoclusion by joint interpolation of vector fields and gray levels,” pre-print, 2002.
[16] A. Papoulis, “A new algorithm in spectral analysis and bandlimited extrapolation,” IEEE Trans. Cir. and Sys. 22:9, 1975.
[3] M. Bertalmio, A. L. Bertozzi, and G. Sapiro, “NavierStokes, fluid dynamics, and image and video inpainting,” Proc. IEEE Computer Vision and Pattern Recognition (CVPR), Hawaii, December 2001.
[17] S. Rane, G. Sapiro, and M. Bertalmio, “Structure and texture filling-in of missing image blocks in wireless transmission and compression applications,” IEEE Trans. Image Processing, to appear.
[4] M. Bertalmio, G. Sapiro, V. Caselles, and C. Ballester, “Image inpainting,” Computer Graphics (SIGGRAPH 2000), pp. 417–424, July 2000.
[18] E. Simoncelli and J. Portilla, “Texture characterization via joint statistics of wavelet coefficient magnitudes,” Proc. 5th IEEE Int’l Conf. on Image Processing, 1998.
[5] M. Bertalmio, L. Vese, G. Sapiro, and S. Osher, “Simultaneous texture and structure image inpainting,” pre-print, July 2002 (mountains.ece.umn.edu/ guille/inpainting.htm).
[19] L. Y. Wei and M. Levoy, “Fast texture synthesis using tree-structured vector quantization,” Computer Graphics (SIGGRAPH 2000), July 2000.
[6] T. Chan and J. Shen, “Local inpainting models and TV inpainting,” SIAM J. Appl. Math. 62:3, pp. 1019-1043, 2001.
[20] D. C. Youla and H. Webb, “Image restoration by the method of convex projections - Part 1: Theory,” IEEE Trans. Med. Imag. 1:2, pp. 81-94, 1982.
[7] J. S. De Bonet, “Multiresolution sampling procedure for analysis and synthesis of texture images,” Proceedings of ACM SIGGRAPH, July 1997. [8] A. A. Efros and T. K. Leung, “Texture synthesis by non-parametric sampling,” IEEE International Conference on Computer Vision, Corfu, Greece, pp. 1033– 1038, Sept. 1999. [9] R. W. Gerchberg, “Super-resolution through error energy reduction,” Opt. Acta, 21, No.9, 709-720, 1974. [10] D. Heeger and J. Bergen, “Pyramid based texture analysis/synthesis,” Computer Graphics (SIGGRAPH 1995), pp. 229–238, July 1995. 3
Fig. 1. Pouring arm of the tea-pot and the red ball from the image on the left have been removed using the POCS algorithm.
Fig. 2. The occupants of the boat and the boy on the shore have disappeared.
Fig. 3. The lost area in the grass-like texture is inpainted.
4
