tion, motion synthesis, compression and film post- production. Based on the pixel-based algorithm proposed in. [10], and inspired by the Image Quilting [4] and.
Image Welding for Texture Synthesis Muath Sabha and Philip Dutr´e Computer Graphics Group Katholieke Universiteit Leuven, Belgium Email: {muath.sabha,philip.dutre}@cs.kuleuven.be
Abstract In this paper, we present a patch-based texture synthesis technique which produces good synthesis results within reasonable time. Part of our work is inspired by existing models, augmented with significant improvements on the timing and quality of results. Our technique reduces the size of the search space to include only the most important patches. A novel criterion for selecting the candidate patch is also exploited according to two measures namely, the cut-path cost and the average difference of the area behind the cut. Matching patches ’welds’ the new patch to the already synthesized part. We upprised our technique by testing it over a wide range of textures ranging from stochastic to regular textures, and also applied it on 3D mapping and worked in an efficient way with stochastic textures. Our technique proved to be easy to understand and implement.
1
Introduction
Texture synthesis algorithms are applied to a sample texture with the aim of generating a large texture, congruous to the source. Computer graphics applications often use textures to decorate virtual objects to increase realism without modifying geometric details. Other applications of texture synthesis in computer graphics include image completion, motion synthesis, compression and film postproduction. Based on the pixel-based algorithm proposed in [10], and inspired by the Image Quilting [4] and Graphcut [7] techniques, our technique achieves good results for a wide range of textures within a shorter time than previous techniques and requires less user intervention. As most of the texture synthesis techniques, our technique follows the Markov Random Field model [14]. It provides an intuitive
Figure 1: An example of texture synthesis on surfaces. framework for the understanding of texture synthesis in general. The proposed technique also produces good results when applied directly to surfaces, Figure 1. In section 2, we briefly analyze previous work, concentrating on the most related algorithms, followed by the motivation behind our technique. Section 4 presents the core of this work in the 2D plane. Applying our technique on surfaces is illustrated in section 5 followed by the results and discussion in which our technique is proved. A comparison section follows, comparing 2D results to those of some recent most related techniques.
2 Related work Texture Synthesis from Example is synthesizing a new texture from a sample texture captured from nature or produced by a human artist. Heeger and
Bergen [6] proposed to analyze textures in terms of histograms of filter responses at multiple scales and orientations. Portella and Simoncelli [15] were able to substantially improve synthesis results for structured textures at the cost of a more complicated optimization procedure. De Bonet [2] scrambles the input in a coarse-to-fine fashion, preserving the conditional distribution of filter outputs over multiple scales. Nicoll et. al [13] separates the regular features with the aid of fractional Fourier analysis in the near regular textures, tiles the regular texture and then adds the irregular texture detail. Efros and Leung [5] developed a method of growing textures using non-parametric sampling. Each pixel to be synthesized is estimated by searching the source texture with a similar neighborhood to its neighbors which are already synthesized. Wei and Levoy [20] used a fixed causal neighborhood size and exhaustively search the sample texture for a best match. They also used a tree structured vector quantization (TSVQ) for accelerating the search. Ashikhmin [1] modified the algorithm to encourage verbatim copying of parts from the input sample and noted that it works best for natural textures. Zelinka and Garland [23] accelerated texture synthesis by pre-calculating references and probability of matching pixels and putting them in a data structure called jump map. In our previous work [10], a framework for a pixel-based texture synthesis is presented. At least, one direct neighbor pixel from the source texture has to match the ”to be synthesized” one in the target texture. The result was avoiding the deterioration of visual quality and speeding up the synthesis. Efros and Freeman [4] point out that the ”thendominant pixel-at-a-time” synthesis algorithms all perform excess computation in the common case of structured textures. The extension that they make to their algorithm for texture transfer is the requirement that each chosen patch satisfies a correspondence map in addition to the texture synthesis requirements. Cohen et. al [3] uses Wang Tiles in their patch based texture synthesis technique. Kwarta et. al [7] suggest synthesizing the new texture by copying irregularly shaped patches from the sample image into the output image. The patch copying process is performed in two stages. First a candidate rectangular patch is selected by performing a comparison between the candidate patch and the pixels already
in the output image. Second, an optimal portion of this rectangle is copied to the output image. The portion of the patch to copy is determined by using a graph cut algorithm. Instead of using the Graphcut technique, Somol and Haindl [18] proposed a fast path search algorithm for image stitching. Lui et. al [8] treat a near-regular texture (NRT) as a statistical distortion of a regular, wallpaper-like congruent tiling, possibly with individual variations in tile shape, size, color and lighting. They defined a measure for regularity in textures, and they dealt with NRT as regular textures. The user spends long time in drawing lattices to define the texels, adjusting the misplaced lattice points to form the distorted lattice, lighting map extraction, and setting the levels of geometry and color regularity. Wu and Yu [22] worked on feature matching for patch-based texture synthesis. They propose to perform texture synthesis using both salient features and their deformation. Some hybrid techniques which combine between pixel-based and patch-based were presented in [11] and [12]. [19], [21] and [24] synthesize directly on meshes pixel by pixel. Praun et. al [16] aim to cover a model with irregular overlapping texture patches oriented according to user defined vector fields. This method works well for stochastic textures but failed with textures which are sensitive to discontinuities. [9] used the quilting technique with some modifications to work with triangular mesh with some preprocessing analysis to store the similar textons together. [25] used pictures taken from different sides of a real model and mapped the patches onto an arbitrary model with the user intervention.
3 Motivation Our work is inspired by the pixel-based algorithm proposed in [10] powered by those of Efros and Freeman [4] and Kwatra et. al [7]. The major advantage of patch-based over pixel-based techniques is that a complete unit of a texture, which may be a complete feature, is copied at a time. This also means that the interior feature details will be maintained without distortion. One of the main goals for our technique is reducing the search space. Instead of testing all the blocks in the sample texture, the most important blocks will be tested to decide which patch to be
4.1 Source Texture Analysis
(a)
(b)
(c)
Figure 2: Sequence of placing patches in a general patch-based technique.
A fast analysis step is needed, in which the sample texture is scanned and each RGB color is stored with all its positions into a suitable data structure (kd-tree). This means that the number of leaves in the kd-tree equals the number of different colors in the source texture, and each color has an array of positions. This information will be used in the synthesis phase where the blocks with common corresponding pixels are needed.
4.2 Texture Synthesis
Source Texture
Target Texture
Figure 3: A schematic drawing shows the blocks that share at least one direct neighbor with the target core block. copied from the sample texture to the target. Resolving the discontinuity between neighborhood patches is the most crucial issue that faces the patch based texture synthesis techniques. In the presented technique, our intention was to take care of the cut-path that separates between the patch to be pasted and the already synthesized part.
4
Patch Welding
In particular, patch based texture synthesis techniques have to make the decision of ”which block to be copied”, ”where to be placed” and ”which part of the block to be transferred and which to be neglected”, for each patch to be copied from source to destination. The process of texture synthesis using our approach consists of two main steps, source texture analysis and texture synthesis.
The synthesis flow in general proceeds as follows: the first block is copied randomly from the source texture and placed in the upper left corner of the new texture. Then, the subsequent blocks are placed from left to right and from top to bottom in a scan line order. The new block is partially overlapping with the previously placed blocks as shown in Figure 2. The overlap between blocks is what we refer as the ”overlap band” as shown in Figure 4. The minimum cost path is figured from one end of the overlapped band to the other end. This path will determine which pixels to be copied from source to destination and which will be neglected, or in other words, which patch to be welded. The Patch Selection A list of positions is retrieved from the kd-tree, for each direct neighbor pixel, by shifting the positions relative to the corner pixel of the block. The union of all the lists is the reduced set of blocks to be tested for finding the most suitable patch. This ensures that each block that shares at least one direct neighbor pixel with the target block will be included. Moreover, this ensures that there are always enough alternatives in the set even for a real number color depth representation. This insures the continuity between previously placed patches and the new candidates. The use of the kd-tree also gives the freedom to use some relaxation around the colors as discussed in [10]. Relaxation also can be used for increasing the probability of including the best patch, but, we found that there is no significant improvement while the timing is severely affected. Figure 3 shows a schematic drawing for the blocks that share at least one pixel with the target block. The direct neighborhood pixels are also shown around the core block. Patch Copying The L2 -norm for the RGB com-
The completed part
Start node The overlapped band
The suggested cut
Arc Node
The core block
The core block
End node
(a)
(b)
(c)
Figure 4: The patch-welding procedure, where (a) represents the difference band of the overlapped area in shades of gray with the suggested cut, (b) is the graph with the added start and end nodes, and (c) the patch to be copied. ponents is used to calculate the difference between the block from the sample texture and the target block as seen in Figure 4. In the overlapped difference band, Dijkstra’s shortest-path algorithm is used to find the best path according to the ”minimum error boundary cut”. First, the L2 differences will be stored in cells (the gray scale cells of the difference band in Figure 4a and b); next, a graph will be generated from this band, where each arc will be calculated by adding the two adjacent cells as: Ms,t (A, B) = Ds (A, B) + Dt (A, B) where Ms,t is the matching quality cost between two adjacent pixels s and t and Dp is the L2 difference of the pixel p in the two overlapped blocks A and B. Dp (A, B) = kAp − Bp k Finally, starting and ending nodes will be added to the graph (Figure 4b). In this graph, the least cost path is calculated, cutting the band in two parts, the inner part (around the core block) and the outer one. The inner part with the core block will be copied from the source to the target which is shown in Figure 4c. The Block-size and the Band-width The recommended block size is as big as the feature size, and that is to include a complete feature, and also to avoid the cut inside the feature. The width of the overlapping band is recommended to be about half the feature size, to enable the boundary cut to circumnavigate around the feature and make the right decision. If the used overlapping band is wide enough, then it works as copying complete features from the source texture to the target. The Overlapping Problem We noticed that sometimes a source of error arises while copying
Figure 5: The overlapping problem. successive patches (from left to right). The error is caused from putting the new patch over part of the pre-synthesized patch. If the upper right part of the previous patch started in the upper part of the overlapping band in the right most part, then there will be a cut (straight vertical line) above the overlapped area as shown in Figure 5. This problem is solved by forcing the path to start from the first pixel in right most part of the direct neighbors in the overlapped area, which ensures that the overlapping of the next patch will cover this part.
5 Synthesis on Surfaces Our technique worked well for surface texture synthesis. Like the previous techniques of texture synthesis over arbitrary surfaces, the user can provide the orientation and scale of the texture synthesis over the mesh by specifying the tangential vector field and the scaling field. For isotropic textures, which are not sensitive to the direction, the technique itself finds the best matched patch following the direction of the previous synthesized neighbor face. A fast synthesis can be done by starting with any polygon over the mesh (whether specified by the user or randomly chosen), and finding a suitable patch in the source texture. After that, going gradually for the adjacent polygons, matching the already synthesized edges of the polygon, the technique finds the most suitable patch in the source texture and using our technique to match edges which are not directly matched. To explain how the procedure goes, let’s take the triangle mesh of four faces shown in Figure 6a. The triangles are flattened on the blue texture (Figure 6b), each triangle follows the pre-textured adjacent one and continuing the texture on the common edge. As a flow, it started with triangle S1, then S2, then S3. When it comes to a triangle (S4) where at least
S2 S1
S2 S3
S1
S4 S1'
(a)
S2 S3 S4
(b)
S1
S3 S4
(c)
Figure 6: Mapping texture over a triangle mesh. one of its adjacent neighbors is already synthesized (S1), a triangular patch with the size and shape of the pre-synthesized adjacent, but rotated to continue the current triangle (triangle S4 in Figure 6) is generated (the red triangle S1’ in 6b). Then, it will be rotated back to exactly match the original triangle S1. The L2 difference between the original triangle texture and the new one is calculated. The best cut is calculated and the area behind the path (the white area) is copied to the triangular texture of S1. This leads to the continuity in isotropic textures. The source texture is theoretically infinite, which means if it needs expansion, the 2D technique expands it and that is to map the triangles on it.
6
Results and Discussion
Our technique has been tested on different types of textures: stochastic, irregular structured, and near regular textures. It gives good results as shown in Figure 7. For the colored textures in Figure 7, the average number of tested blocks is very low relative to the exhaustive search by [4]. As an example, for the apples’ texture, the average number of tested blocks for each patch is about 39 blocks while it is 17978 in the exhaustive search, which means about 460 times less. Speeding up depends on the number of colors in the sample texture, its size, and the dimensions of the block with the overlap band. For the green leaves in Figure 7, the number of colors is 36326, which is high, and because of that, the average number of tested blocks in our technique is 2000 times less than the exhaustive search, while in the last gray texture of the same figure, it is only about 2 times less than the exhaustive search, and that is because it has only small number of colors and it has some dominant color. All the resulting textures are 400 × 400 pixels
and most of the block size used is 40 × 40 pixels with 10 to 20 pixels overlapping band width. The preprocessing and synthesis timings are shown under each texture to give an idea about how fast our technique is, especially in textures of high number of colors. The results in the figures are from different texture categories to show the broad range of textures that our technique succeeded with. We have practiced some false color coding to test the distribution of the synthesized texture and the regions copied from the source texture. Figure 10 shows some examples which prove that our technique works well in copying complete features from the source textures. The sizes of the copied patches are proportional to the block size and to the feature size. It is also clear in some cases that the cut circumnavigate around the feature to include (or discard) a complete feature. In some cases, the technique copies a good part from the source and repeats it in the target, which may leads to regularizing the result. But in general, and from the false colors test, it is also clear that the distribution is good, and the patches are copied from nearly every part of the source texture, which is valid for most of the tested textures. Figure 8 shows some examples for synthesizing some isotropic textures over different meshes. The synthesis can be done on a simplified progressive mesh and to be smoothed after synthesis to ensure big patches and wide overlapping. Sander et. al [17] proposed a suitable technique to avoid deformation artifacts over a progressive mesh. We also applied the false color coding on the 3D surfaces to test the distribution over the mesh, and it is found that it is well distributed as shown in Figure 11. The figure also shows that there are no cuts in the false color image over the mesh.
7 Comparisons In Figure 9, we compare the results of our technique to those from the ”Image Quilting”, ”Graphcut”, and the ”Near Regular Textures”. It shows that our technique works at least as good as the other techniques with an advantage of the timing, simplicity, and minimal human input (which is the block size). In the ”Image Quilting” technique [4], all blocks in the sample texture will be tested as exhaustive search for the best candidate. In the ”Graphcut”
Tp = 2.51 sec Ts = 0.28 sec
Tp = 2.87 sec Ts = 0.25 sec
Tp = 0.65 sec Ts = 20.07 sec
Tp = 1.09 sec Ts = 0.34 sec
Tp = 1.39 sec Ts = 0.36 sec
Tp = 1.79 sec Ts = 0.80 sec
Tp = 2.89 sec Ts = 0.23 sec
Tp = 0.65 sec Ts = 20.39 sec
Figure 7: 2D results with the preprocessing analysis time Tp and the synthesis time Ts in seconds under each figure. The examples were computed on a 1.84 GHz AMD AthlonT M 64 processor.
Figure 8: A surface synthesis example. Source
Image Quilting [EF01]
Graphcut [KSE*03]
NRT [LLH04]
Image Welding
Figure 9: Comparing our results to the reported results from some recent well-known patch-based techniques.
Figure 10: False color test. [7], there are three methods used for selecting the patch, two of them use the complete input image which makes it expensive in calculations, while the third uses a random block which is suitable only for stochastic textures. In our technique, only blocks from the source texture that have at least one direct neighbor pixel matched with the target block will be considered and included in calculation.
8
Conclusion and Future Work
We present a patch-based texture synthesis algorithm which is able to create high quality textures, very fast, and minimal user input. The idea behind our technique is to reduce the search space by including only blocks which share at least one pixel from the direct neighbors with the core block, in which we solved a crucial visual problem which is the discontinuity. Other blocks are discarded because they have a low probability to include the right block. The selection criterion proved that it gives good results. It is also clear from the 3D results that our technique is applicable on synthesis over 3D meshes. While the preprocessing stage is relatively slow, it has to be done only once, this leads to a very fast synthesis process that can be done in real time. From the results section, we noticed that the good results come from patches where the path easily circumnavigates around the feature which leads to copy the right patch. If it becomes possible for the technique to recognize automatically the features size in the sample texture, and the relation between the features
Figure 11: False color coding over 3D models. according to their size, distances, orientation, and illumination, there will be a good chance to generate any texture in real time and in a pleasant way, and that is our future work. The short time taken in synthesis over surfaces enables the user to interactively change parameters to get the required results within an interactive time.
9 Acknowledgments The first author is granted from the BTC (Belgian Technical Cooperation).
References [1] Michael Ashikhmin. Synthesizing natural textures. In SI3D ’01: Proceedings of the 2001 symposium on Interactive 3D graphics, pages 217–226, 2001. [2] Jeremy S. De Bonet. Multiresolution sampling procedure for analysis and synthesis of texture images. In SIGGRAPH ’97, pages 361–368, 1997. [3] Michael F. Cohen, Jonathan Shade, Stefan Hiller, and Oliver Deussen. Wang tiles for
[4]
[5]
[6]
[7]
[8]
[9]
[10]
[11]
[12]
[13]
[14]
[15]
image and texture generation. ACM Trans. Graph., 22(3):287–294, 2003. Alexei A. Efros and William T. Freeman. Image quilting for texture synthesis and transfer. Proceedings of SIGGRAPH 2001, pages 341– 346, 2001. Alexei A. Efros and Thomas K. Leung. Texture synthesis by non-parametric sampling. In ICCV (2), pages 1033–1038, 1999. David J. Heeger and James R. Bergen. Pyramid-based texture analysis/synthesis. In SIGGRAPH ’95: Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, pages 229–238, 1995. Vivek Kwatra, Arno Sch?dl, Irfan Essa, Greg Turk, and Aaron Bobick. Graphcut textures: Image and video synthesis using graph cuts. ACM Transactions on Graphics, SIGGRAPH 2003, pages 277–286, 2003. Yanxi Liu, Wen-Chieh Lin, and James Hays. Near-regular texture analysis and manipulation. ACM Trans. Graph., 23(3):368–376, 2004. Sebastian Magda and David Kriegman. Fast texture synthesis on arbitrary meshes. In EGRW ’03: Proceedings of the 14th Eurographics workshop on Rendering, pages 82– 89. Eurographics Association, 2003. Pieter Peers Muath Sabha and Philip Dutr´e. Texture synthesis using exact neighborhood matching. Computer Graphics Forum, To appear, 2006. Andrew Nealen and Marc Alexa. Hybrid texture synthesis. In EGRW ’03: Proceedings of the 14th Eurographics workshop on Rendering, pages 97–105. Eurographics Association, 2003. Andrew Nealen and Marc Alexa. Fast and high quality overlap repair for patch-based texture synthesis. In Computer Graphics International, pages 582–585, 2004. A. Nicoll, J. Meseth, G. M¨uller, and R. Klein. Fractional fourier texture masks: Guiding near-regular texture synthesis. Computer Graphics Forum, 24(3):569–579, 2005. R. Paget and D. Longstaff. Texture synthesis via a noncausal nonparametric multiscale markov random field, 1997. Javier Portilla and Eero P. Simoncelli. A parametric texture model based on joint statistics
[16]
[17]
[18]
[19]
[20]
[21]
[22]
[23] [24]
[25]
of complex wavelet coefficients. Int. J. Comput. Vision, 40(1):49–70, 2000. Emil Praun, Adam Finkelstein, and Hugues Hoppe. Lapped textures. In Proceedings of ACM SIGGRAPH 2000, pages 465–470, 2000. Pedro V. Sander, John Snyder, Steven J. Gortler, and Hugues Hoppe. Texture mapping progressive meshes. In SIGGRAPH 2001, Computer Graphics Proceedings, pages 409– 416. ACM Press / ACM SIGGRAPH, 2001. P. Somol and M. Haindl. Novel path search algorithm for image stitching and advanced texture tiling. In The 13th International Conference in Central Europe on Computer Graphics, Visualization and Computer Vision 2005, pages 155–218, 2005. Greg Turk. Texture synthesis on surfaces. In SIGGRAPH ’01: Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pages 347–354, 2001. Li-Yi Wei and Marc Levoy. Fast texture synthesis using tree-structured vector quantization. In SIGGRAPH ’00, pages 479–488, 2000. Li-Yi Wei and Marc Levoy. Texture synthesis over arbitrary manifold surfaces. In Eugene Fiume, editor, SIGGRAPH 2001, Computer Graphics Proceedings, pages 355–360. ACM Press / ACM SIGGRAPH, 2001. Qing Wu and Yizhou Yu. Feature matching and deformation for texture synthesis. ACM Trans. Graph., 23(3):364–367, 2004. S. Zelinka and M. Garland. Towards real-time texture synthesis with the jump map, 2002. Steve Zelinka and Michael Garland. Interactive texture synthesis on surfaces using jump maps. In EGRW ’03: Proceedings of the 14th Eurographics workshop on Rendering, pages 90–96, Aire-la-Ville, Switzerland, Switzerland, 2003. Eurographics Association. Kun Zhou, Xi Wang, Yiying Tong, Mathieu Desbrun, Baining Guo, and Heung-Yeung Shum. Texturemontage. ACM Trans. Graph., 24(3):1148–1155, 2005.
