Fast three-dimensional Texture Synthesis

Fast three-dimensional Texture Synthesis Felix Manke [email protected]

Burkhard Wünsche [email protected]

Graphics Group Department of Computer Science The University of Auckland Auckland, New Zealand

ABSTRACT

Three-dimensional (3D) textures are popular for texturing solid objects. They usually achieve superior results over conventional two-dimensional (2D) texture mapping techniques. However, whereas 2D textures can be obtained using for example a photograph, capturing or generating solid textures is not straightforward. Procedural techniques can replicate some types of materials, but are usually difficult to control. An alternative is exemplar-based texture synthesis where 3D textures are generated from a few 2D pictures. Unfortunately, the synthesis from exemplars is very challenging and usually requires very long computation times. In this paper we present a new algorithm for fast solid texture synthesis from 2D input exemplars. Our method extends an existing real-time synthesis approach which has so far been limited to 2D texture synthesis only. The modifications we made allow a hardware-accelerated synthesis utilising the computational capabilities of today’s graphics cards. To our knowledge, our method is the first solid texture synthesis approach that was specifically designed for an execution directly on graphics hardware.

Categories and Subject Descriptors

I.3.3 [Computer Graphics]: Picture/Image Generation; I.3.7 [Computer Graphics]: Three-Dimensional Graphics and Realism—color, shading, shadowing, and texture

General Terms

Algorithms, Computer Graphics, Image Processing

Keywords

Exemplar-based texture synthesis, solid texture, 3D texture, Gaussian pyramids, principal component analysis

1.

INTRODUCTION

In computer graphics textures are commonly used to increase the realism of a 3D scene. Compared to simple colour-

This paper was published in the proceedings of the New Zealand Computer Science Research Student Conference 2008. Copyright is held by the author/owner(s).

NZCSRSC 2008, April 2008, Christchurch, New Zealand.

ing textures are able to model fine details and structures of surfaces. The structural elements of textures can be perceived independently from colour and, hence, textures encode far more information than colour alone. However, a given 2D texture needs to be mapped onto the geometry without introducing too much distortion. Defining a distortion-free mapping can be challenging or even impossible for complex objects. In contrast, using solid 3D texture cubes allows to “carve” the object out of the texture, just like a sculptor carves an object out of a solid block of wood or stone. Moreover, once a solid texture is available it can be used to texture arbitrary many objects of arbitrary shape, because the solid texture carries information about the material in the entire volume it encloses. Unfortunately, obtaining realistic looking solid textures is not straightforward, because imaging devices usually record 2D images. In the past, techniques have been proposed that procedurally define the solid texture, but the covered range of materials is generally very limited. More interesting is the synthesis of 3D textures based on a 2D input exemplar. This enables the generation of a variety of solid textures, because photographs of a real materials can be used as 2D input texture. However, 3D texture synthesis is very complex and the computation time is usually very high. In this paper, we present a new and fast 3D texture synthesis algorithm. By extending Lefebvre and Hoppe’s [14, 15] realtime synthesis method — which is limited to 2D synthesis only — we are able to design an algorithm for solid texture synthesis that can be directly implemented on the graphics processing unit (GPU).

2.

RELATED WORK

Texture synthesis algorithms can coarsely be categorised in several different classes. Procedural approaches for both 2D and 3D texture synthesis use combinations of mathematical functions to define colours in a continuous space [17, 21, 25, 26]. By carefully selecting the functions and adjusting their parameters different effects can be realised. However, procedural methods are often hard to control and a lot of experience is needed for correctly adjusting the parameters. In contrast to procedural methods, exemplar-based synthesis algorithms take images as input exemplar and synthesise a new texture that is visually coherent with the original. Parametric approaches, as proposed in [8, 4, 18, 2], compute a

Fast Three-Dimensional Texture Synthesis

global statistical model from the input exemplar (for example, histograms and their variance or skewness, coefficients of a wavelet or Fourier transform, etc.). These statistical properties serve as constraint functions while matching statistics of the target and the input texture. The drawbacks of parametric synthesis methods are the complexity and the relatively narrow range of mainly homogeneous and stochastic textures that can be synthesised [11]. Patch-based methods paste random patches of the exemplar into the output texture and optimise the transitions between the overlapping patches [19, 6, 13]. While these methods could probably be extended to use 3D texture patches as input, there is no straightforward way to use them to generate 3D textures from 2D input patches. In contrast, pixel-based methods can be used to generate solid textures, because they provide a finer control of the synthesis by synthesising one pixel at a time [7, 24, 1, 14]. In several iterations, the synthesis result is optimised by searching the most similar local neighbourhood in the input exemplar and replacing the current value of the pixel by the value of the found best match. Finally, optimisation-based approaches use the local similarity measures of pixel neighbourhoods to define global texture energy function that is minimised [12]. For synthesising 3D solid textures from 2D exemplars algorithms have to “invent” three-dimensional information out of a two-dimensional input. Heeger and Bergen [8] as well as Dischler et al. [5] proposed parametric methods to synthesise 3D textures. However, like most parametric approaches the presented solutions are only capable of synthesising homogeneous textures and cannot preserve dominant structures very well. The first non-parametric approach was introduced by L. Wei as extension to the previously published pixelbased 2D synthesis algorithm [22, 23]. The work has been very important and inspired later publications. Also our method presented in this paper partly uses ideas introduced by Wei. A specialised solution for synthesising composite materials, based on stereology theories (the analysis of 2D cross-sections of 3D volumes), was proposed by Jagnow et al. [10]. Their method is especially successful for exemplars which show particles surrounded in a binding material. Recently, Kopf et al. presented an algorithm that combines a non-parametric global energy optimisation and a parametric histogram matching [11]. The presented results are impressive for a wide variety of textures. However, the algorithm cannot easily (if at all) be translated onto the GPU and the computation time is with up to 90 minutes rather high.

3.

2D TEXTURE SYNTHESIS

Our texture synthesis algorithm is, as already mentioned, based on the pixel-based method proposed by Lefebvre and Hoppe [14, 15]. Their approach uses the parallel synthesis scheme proposed by L. Wei [23]. The ability to process the pixels order-independent is an essential requirement for Lefebvre and Hoppe’s real-time implementation on the GPU. Because of the close relation we will give a more detailed review of the basic algorithm for 2D texture synthesis in the following. In section 4 we will then show how we extend the original algorithm to perform 3D texture synthesis. As already indicated, pixel-based synthesis approaches perform an iterative optimisation on a per-pixel basis in order to minimise the difference of the synthesis result to the original

85

exemplar. An error is computed for a local neighbourhood around the pixel that is currently updated. To achieve a minimisation of the error, the best-matching pixel (that is, the pixel with the most similar neighbourhood in terms of the given metric) is searched in the original. The pixel value of this best match replaces the previous value of the pixel in the synthesised texture. To improve the synthesis results, a multi-resolution approach is most commonly pursued: For the original exemplar as well as the synthesised target texture a hierarchical image pyramid is computed which consists of coarser copies of the image with the highest resolution. These smaller images, obtained by subsampling and applying a Gaussian-like smoothing, correspond to low-pass filtered versions of the original and, hence, only contain the low frequencies of the image (note that this data structure, called Gaussian pyramid, was introduced by Burt and Adelson in 1983 [3]). Starting at the coarsest level, the synthesis is performed per level and then carried to the next finer level (for example by integrating the coarser level when searching the best match [22], by resampling [11], or in our case by “upsampling” coordinates). In consequence, the synthesis proceeds from first establishing the low frequencies and then determining the finer details in the image.

3.1

Overview of the 2D Synthesis Algorithm

A key difference of Lefebvre and Hoppe’s algorithm to other methods is that it does not work with image colours directly. In order to be able to implement the synthesis on the GPU, the synthesis pyramid stores coordinates of pixels in the original. With these coordinates the input exemplar can be accessed and the pixel colour fetched. It is important to keep this difference in mind. During the initialisation a Gaussian Pyramid E is computed for the input exemplar and a similar synthesis pyramid S is generated that represents the texture to be synthesised. To ensure compatibility it is important that both image pyramids have the same number of levels. However, to be able to enter the synthesis, an additional coarsest level S−1 has to be defined for the synthesis pyramid. It serves as previous level for the first upsampling step described below. Lefebvre and Hoppe propose to initialise S−1 with “zero coordinates” (0 0)T , but we find it more intuitive to initialise S−1 as follows (where P ∈ N2 is a pixel coordinate and s ∈ N2 is the size of the original input exemplar): S−1 (P ) = P mod s,

(1)

as this better reflects how the algorithm proceeds, especially when only a few pyramid levels are used. The synthesis process consists of three steps which are iteratively performed per pyramid level from the coarsest to the finest: coordinate upsampling, coordinate perturbation (jitter), and the correction phase (error minimisation).

Coordinate Upsampling

To fill the next finer level Si based on the results of the previous level Si−1 , each pixel of the smaller level is assumed to point to four pixels of the finer level. The following equation

86

F. Manke and B. W¨unsche

1

0.8

0.8

0.6 0.4 0.2

Randomness r

1

0.6 0.4 0.2

0 -1

-0.8 -0.6 -0.4 -0.2

0

0.2

0.4

0.6

0.8

1

Randomness r

(b) Our proposed Jitter-Function

(a) The original Jitter-Function

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

Value Hash-function H(P)

Figure 1: Plots of the result of different jitter functions Ji (P ). Plot (a): The original jitter function proposed by Lefebvre and Hoppe in [14]. Plot (b): Our improved jitter function. Colour encoding: Blue = −1, Magenta = 0, Yellow = 1 describes the upsampling procedure: Si (2 · P + ∆U ) = (2 · Si−1 (P ) + ∆U ) mod s, ( ∆U ∈

(2)

! ! ! !) 0 0 1 1 , , , . 0 1 0 1

Coordinate Perturbation (Jitter)

The perturbation of coordinates is used to introduce spatial variations. It can be controlled per synthesis level with a “randomness parameter” 0 ≤ ri ≤ 1. For a deterministic randomness Lefebvre and Hoppe introduce a hash function H : Z2 7→ [−1, 1]2 . In [14], the authors define the jitter as: Si (P ) = (Si (P ) + Ji (P )) mod s,

(3)

with the jitter function Ji (P ) defined as: $ !% 0.5 Ji (P ) = H(P ) ri + . 0.5 A plot of the result of function Ji (P ) for different combinations of H and r is shown in figure 1 (a). As can be seen, Ji (P ) always returns zero for randomness r < 0.5. In addition, for r = 1 the probability of Ji (P ) = 0 is twice as high as for Ji (P ) = −1 and Ji (P ) = 1. We therefore propose the following more intuitive equation (see also figure 1 (b)): $ !% kx Ji (P ) = j + , ky where: j kx|y lerp(a, b, t)

= H(P  ) · lerp(0.5, 1, r) lerp(0.5, 2/3, r) = 1 − lerp(0.5, 2/3, r) = a + t(b − a)

Correction Passes

Si (P ) = argmin SSD(NSi (P ), NEi (Q)), Q∈Ei

(4)

0 -1

Value Hash-function H(P)

where:

SSD. The colour (or more precisely, the new coordinate value) for pixel P is hence given by:

if jx|y ≥ 0 otherwise.

In several correction passes neighbourhood matching is performed to optimise the synthesis result. Hence, the correction phase is the most important and computationally most expensive part of the entire synthesis process. As common in pixel-based synthesis, the similarity between neighbourhoods is measured using the sum of squared differences

where NSi (P ) and NEi (Q) are vectors containing the pixels of the neighbourhood around P in the synthesis level and around Q in the exemplar level respectively. To improve convergence, Lefebvre and Hoppe propose to use an interleaved correction scheme which consists of several sub-passes, where each sub-pass only optimises selected pixels. The subsequent sub-passes then benefit from the previous error minimisations. The authors suggest to define a pattern of four sub-passes in a 2×2 square that defines which pixels to process in which sub-pass. Further on, the correction phase can be significantly accelerated by using k-coherence search as it was introduced by Tong et al. [20]: For each pixel Q in each exemplar level Ei a candidate set Cik (Q) is pre-computed. It contains links to the k pixels with most similar neighbourhoods (k-nearest neighbours in terms of the similarity metric). Whenever the best matching pixel for a neighbourhood NSi (P ) is searched, the candidate sets of all locations Q0 given by the 3×3 neighbourhood of P are used. In this way, the number of possible candidates is reduced from all pixels in the exemplar level to 9×k. In our tests, we usually set k = 3.

3.2

Synthesis in the Appearance Space

In their 2006 SIGGRAPH paper [15] Lefebvre and Hoppe discuss an improved synthesis method where the input exemplar is transformed from a three-dimensional colour space into a high-dimensional appearance space. In this space each pixel contains much more information than only colour. As a result of the higher information density the synthesis can be both improved in quality and sped-up significantly. Before computing the Gaussian pyramid E of the exemplar, an appearance vector, containing additional information, is computed for every pixel in the exemplar. Specifically, the authors suggest to integrate a Gaussian-weighted vectorised local 5×5 neighbourhood as well as the signed Euclidean distance to the closest dominant feature in the original texture (an additional binary mask encodes features). In our tests we also found the integration of partial derivatives in x- and y-direction (approximated using a standard Sobel operator) to support the synthesis extremely well. Note that in the described setup the appearance vector for each pixel with three colours is 3×5×5×1×2 = 150-dimensional. We made experiments with other attributes of the appearance space, as discussed in [16]. To make the appearance space practicable the highdimensional appearance vectors are projected into a lowdimensional space defined by the first n principal components obtained from a principal component analysis (PCA). The box plots in figure 2 show how much of the total variance in the appearance space of our input exemplars is explained by the first n principal components (we used about 50 exemplars for the measurements). Selecting the first eight components for the dimensionality reduction proved to be


87

Figure 3: Input exemplar and attributes of the appearance space and the projection onto the first three principal components. From left to right: RGB exemplar, partial derivatives, signed feature distance, projected appearance space. The second to fourth image show mappings to RGB space.

Figure 2: Box plots that show how much of the total variance in the appearance space of input exemplars is explained by the first n principal components. The measurements are based on about 50 exemplars. sufficient, as usually more than 95% of the total variance is explained. As an alternative it is also possible to specify the minimal variance that should be kept when selecting the dimensionality of the projected appearance space. The perhaps most important property of the appearance space is the ability to integrate any meaningful information about the pixel and its surrounding. By reducing the dimensionality, the computation time can be easily controlled. When computing the appearance vectors we normalise all attributes of the appearance space (neighbourhood values, derivatives, feature distances, etc.) so that they are all in the same range [0, x]. The influence of different attributes can be controlled by scaling the values around the centre of the range x/2. For a scaling factor s, the normalisation can be expressed as: v0 =

(v − vmin ) · s · x s·x − , vmax − vmin 2

where v is the value to normalise and vmin and vmax are the minimal and maximal values of the given attribute of the appearance space. Figure 3 shows an example of the transformation from the input exemplar defined in RGB colour space to a projected appearance space that takes the above mentioned attributes into account (local neighbourhood, partial derivatives, and signed feature distance). It can be seen that the projected appearance space of the same dimensionality as the original carries much more information (figure 3, right).

synthesis (see figure 4, left). However, since all four points belong to the same correction sub-pass, the value for each corner point is computed as the average of three adjacent points, as indicated in figure 4, right.

4.

SOLID TEXTURE SYNTHESIS

In the following, we will describe our extensions to the discussed 2D synthesis algorithm in order to generate textures with an additional spatial dimension. We took care that the modifications still allow an implementation on the GPU for hardware-accelerated 3D texture synthesis. For exemplar-based 3D texture synthesis several input exemplars are usually imagined as being different views of the solid texture cube that is to be synthesised [23, 11]. This idea is illustrated in figure 5. During the synthesis, the algorithm tries to generate a 3D texture that reflects the characteristics of all views that have been specified. The views must not necessarily be aligned with the major axes of the solid cube. However, orienting the exemplars orthogonal to the major axes simplifies the synthesis and does not require resampling of the solid texture during the correction phase [11]. Also, it is not always necessary or appropriate to specify all three exemplars. Sometimes, for example when textures exhibit dominant directional features, it is better to define only two views.

4.1

Initialisation

An obvious difference to the basic 2D synthesis algorithm is that now several 2D input exemplars represent different views of the solid cube. However, the individual views are independent from each other and, hence, their initialisation may remain unchanged.

3.2.1 Reduced Neighbourhood

The higher information density of the projected appearance space makes it possible to reduce the neighbourhood used to find the best-matching pixel in the exemplar during the correction phase. Instead of using a complete squared neighbourhood (for example of all 5×5 = 25 pixels) only the four corner pixels around the centre are sufficient for a successful

Figure 4: The reduced neighbourhood used for searching the best match (left). On the right, the 4×3 points are shown that are interpolated. Adopted from [15].

88


a randomness parameter 0 ≤ rij ≤ 1 for every level of each pyramid. A minor difference in equation 3 is that Ji (P ) and H(P ) now take 3D coordinates as input.

4.2.3

Figure 5: Several 2D exemplars define different views of the solid texture being synthesised. Adopted from [22]. Major changes have to be made when initialising the synthesis pyramid. Since the result will be a 3D texture, also the synthesis pyramid S has to consist of solid texture cubes instead of 2D images. Building-up the pyramid is straightforward. However, since S stores coordinates rather than colours, a separate synthesis pyramid S j is required per exemplar view E j . It is important to point out that, because the exemplar is two-dimensional, S j still stores 2D coordinates. The values of the coarsest level of a synthesis pyramid j S−1 can be initialised as follows: j S−1 (P ) = Pu,v mod sj ,

where u and v are the two components of P to which the view E j is parallel (P is now of course defined in R3 ). Because the solid texture is synthesised from multiple input exemplars, each resulting pixel is a mixture between all inputs. In order to allow more controllability we introduce a scalar weight map W j for each exemplar E j that encodes the spatial influence of E j . Note that W j is defined in R3 and is of the same size as the target solid texture. To guarantee a correct synthesis, it is important to normalise the P weight maps so that kj=1 W j (P ) = 1 for all k exemplars.

4.2

Changes in the Synthesis Steps

4.2.1 Coordinate Upsampling

Since the 3D synthesis pyramids still store 2D coordinates, the upsampling cannot be carried out straightforwardly. Thus, we perform the upsampling as defined in equation 2 for every second slice that is oriented parallel to the exemplar view and duplicate the result for each subsequent slice: Si (2 · Pu,v|w + ∆u,v ) = Si (2 · Pu,v|w+1 + ∆u,v ) = (2 · Si−1 (P ) + ∆U ) mod s,

(5)

where w is even (w mod 2 = 0) and depicts the component of P that is orthogonal to the view. ∆u,v describes a 3D vector with the same value in the u- and v-component as ∆U and the w-component set to 0.

4.2.2 Coordinate Jitter

Since the stored coordinates of the synthesis pyramids are jittered and the synthesis pyramids still stores 2D coordinates this step can remain unchanged and is performed for each pyramid level Sij individually, which allows us to define

Correction Phase

A problem that arises for minimising the error of the synthesis result (that is, the difference to the exemplars) during the correction phase is the fact that we now have to compare local neighbourhoods of pixels of a 3D texture Si with neighbourhoods in 2D textures Eij . As a solution we apply an idea previously proposed by Wei as well as Kopf et al. [22, 11]: Each 2D input exemplar is aligned with one of the principal axes of the solid 3D texture cube that is synthesised. The corresponding 2D neighbourhood in the 3D synthesis pyramid is also aligned with the same plane, because the exemplar represents only one view onto the solid cube. In consequence, we need to compute several synthesis neighbourhoods NSi |u,v (P ), one for each possible orientation of the exemplars. There is a second difficulty we have to deal with. To update the colour of a pixel in the synthesis pyramid we search the pixel in the corresponding exemplar with the most similar neighbourhood. However, in the case of 3D texture synthesis the best match for a pixel P in the synthesised texture must represent all input exemplars, though the influence of each exemplar level Eij is given by Wij (P ). In the original 2D synthesis algorithm the neighbourhood of P in the synthesis level Si was given by the set: NSi (P ) = {Ei (Q) | Q = Si (P + ∆N )}, ∆N ∈ N ,

(6)

where ∆N takes the values of offset vectors from the neighbourhood’s centre to all pixels in the neighbourhood, as defined by the set N of all offsets: N = {δ | (P + δ) ∈ N (P )}. In the case of 3D synthesis, however, multiple exemplars are used which are, in addition, oriented with certain planes in the solid cube. Thus, we need to extend the neighbourhood gathering in the synthesis level Si in the following way to account for the more complex situation: NSi |u,v (P ) = {C(i, P + ∆N |u,v )}, ∆N |u,v ∈ Nu,v ,

(7)

where: C(i, P ) =

k X

Wij (P ) · Eij (Q), Q = Sij (P )

j=1

gives the synthesised colour in level i at location P as the weighted average between all exemplar levels, and ∆N |u,v gives the neighbourhood offsets parallel to the current view (and the w-component set to 0). Notice the similarity to the coordinate upsampling which also depends on the orientation of the view. For the 3D texture synthesis k-coherence search can still be utilised. But in fact, the 3 × 3 neighbourhood around P ∈ Si now has to be aligned with the exemplar’s view plane (recall that this 3×3 neighbourhood selects the coordinates


89

Figure 6: Left: Pixels used in our half-reduced neighbourhood. On the right, all pixels are shown that are included when computing the averaged values of the neighbourhood. Q0 around Si (P ) whose candidate sets Cik (Q0 ) are used for searching the best match). This principle is similar to the previously described neighbourhood gathering and can be directly adapted. Furthermore, the interleaved correction in several sub-passes is affected by the extension to 3D synthesis. Instead of defining a 2×2 square that specifies in which order to process the pixels, we now define a 23 cube for the sub-pass patterns. Hence, eight instead of four correction sub-passes are performed per full correction pass.

4.3

A new Neighbourhood for 3D Texture Synthesis

The appearance space makes it possible to use a very compact and sparse neighbourhood for 2D texture synthesis, as discussed in section 3.2.1. We found that in a 3D synthesis, which has to deal with little information that is available as input anyway, this reduced neighbourhood is not capable of preserving meso-structures of the input exemplar. Much better results can be achieved when using a full 5×5 or even 7×7 neighbourhood, though the computation time then increases dramatically.

To overcome this problem, we designed a new “half-reduced” neighbourhood as a compromise between the insufficient fourpixel neighbourhood and a slow full neighbourhood. The new layout is shown in figure 6. We still average several pixel values to compute the values for the individual neighbourhood values (as shown in the right figure). Note that the new neighbourhood, consisting of 9 points, is a superset of the neighbourhood proposed for the 2D synthesis. In figure 7 we compare the synthesis results using different neighbourhoods. As can be seen, our new neighbourhood is much better capable of preserving the coherence of features than the original four-pixel neighbourhood, which fails to produce acceptable results. The synthesis results of our neighbourhood are comparable to the full 5 × 5 neighbourhood, while the computation time is noticeably reduced (see also figure 9 for a comparison of the performance).

5.

RESULTS

In order to speed up implementation and facilitate debugging we first implemented a prototype in a general purpose programming language (C++) as a proof of concept of our new 3D synthesis algorithm before translating it to the GPU. All exemplars used in this paper have a size of 64 × 64 or

Figure 7: Comparison of different neighbourhoods used during the correction phase of the 3D texture synthesis. First column: Reduced four-pixel neighbourhood. Second column: Full 5×5 neighbourhood consisting of 25 pixels. Third column: Our new halfreduced neighbourhood consisting of 9 pixels.

128×128 pixels and the synthesised solid texture has a size of 1283 . All textures were generated using an appearance space that included a Gaussian-weighted 5×5 neighbourhood of the RGB colour, the partial derivatives, and the signed feature distance. The 150-dimensional space was reduced to eight dimensions (the PCA implementation is provided by the Open Source Computer Vision library [9]). For the correction phase, we used two full correction passes with eight sub-passes each. In figure 8, we show some of our synthesis results for different input exemplars. As can be seen, our algorithm is able to synthesise a wide variety of different textures. The generated 3D solid textures coherently reflect the input exemplars. However, a smoothing of the fine details can be observed — a problem that is common in solid texture synthesis algorithms (compare for example to [23, 11]). The chart in figure 9 illustrates how our algorithm in an unoptimised implementation performs using the settings stated TM R above (executed on a 2.13 GHz Intel Core 2 Duo CPU with 2 GB RAM). Even with a full 7×7 neighbourhood our algorithm needs no more than 15 minutes to synthesise a 1283 texture cube. Clearly, our half-reduced neighbourhood is a good compromise between speed and quality. The synthesis is with a little more than 5 minutes very fast. Note that, because we use k-coherence search, the synthesis time does not significantly depend on the size of the input exemplar. With an implementation on the GPU, we can expect an enormous performance gain, because the GPU is specialised for the calculations involved in the synthesis. We compared the computation times of our implementation for 2D texture synthesis with the timings given by Lefebvre and Hoppe in [15] and estimated the difference in performance (the GPU implementation is 70–90 times faster). An extrap-

90


Figure 8: Results of our 3D texture synthesis. The generated solid textures have been used to render different 3D geometries. The input exemplars are shown as well. olation to our 3D synthesis would lead to synthesis times between 2.5 seconds for a reduced neighbourhood and 10 seconds for a full 7 × 7 neighbourhood. Because proportionally more calculations are involved the graphics card is specialised for, the performance gain might be even higher. Our method has some limitations. Examples of failures are shown in figure 10. Problems occur with exemplars that either have very large features (figure 10, left) or where the arrangement of features has a semantic meaning to humans (figure 10, right).

6. CONCLUSIONS AND FUTURE WORK

We presented a new and fast exemplar-based 3D texture synthesis algorithm as an extension of the GPU-based 2D synthesis approach previously proposed by Lefebvre and Hoppe. All modifications are carefully thought-out to allow an implementation on the GPU to benefit from its enormous computational power. We are not aware of other solid texture synthesis algorithms that have been realised on the GPU. Note that even our preliminary implementation on the CPU is much faster than previous methods. The upsampling and correction phases of the original algorithm have been extended to 3D, and we introduced a more intuitive jitter function. Because multiple views of the exemplar are required, we proposed a neighbourhood gathering in the synthesis pyramid that accounts for the more complex situation. As discussed, the reduced neighbourhood of the 2D method is not suitable for 3D texture synthesis. Therefore, a new neighbourhood function has been designed and its performance evaluated. With the examples in figure 8 we are able to show that our method is suitable for a wide range of input textures. The generated solid textures successfully reflect the nature of the input exemplars. In the future, besides implementating our algorithm on the

GPU, we want to further improve the synthesis quality of our method. Alternative half-reduced neighbourhoods could probably lead to improved results without increasing the performance. It might also be possible to incorporate a histogram-matching into our scheme as adaption of Kopf et al.’s synthesis method. However, it is questionable if an implementation on the GPU is possible. In addition, we will extend our approach to allow 2D and particularly 3D texture morphing. That is, we want to generate textures that contain smooth transitions between exemplars that have different characteristics and represent entirely different materials. To our knowledge, 3D texture morphing has not been attempted by anyone so far.

7.

REFERENCES

[1] M. Ashikhmin. Synthesizing natural textures. In Proceedings of I3D ’01, pages 217–226, New York, NY, USA, 2001. ACM Press. [2] Z. Bar-Joseph, R. El-Yaniv, D. Lischinski, and M. Werman. Texture mixing and texture movie synthesis using statistical learning. IEEE Transactions on Visualization and Computer Graphics, 7(2):120–135, 2001. [3] P. J. Burt and E. H. Adelson. The Laplacian pyramid as a compact image code. IEEE Transactions on Communications, 31(4):532–540, Apr. 1983. [4] J. S. De Bonet. Multiresolution sampling procedure for analysis and synthesis of texture images. In Proceedings of SIGGRAPH ’97, pages 361–368, New York, NY, USA, 1997. ACM Press. [5] J.-M. Dischler, D. Ghazanfarpour, and R. Freydier. Anisotropic solid texture synthesis using orthogonal 2d views. Computer Graphics Forum, 17(3):87–95, 1998. [6] A. A. Efros and W. T. Freeman. Image quilting for texture synthesis and transfer. In Proceedings of SIGGRAPH ’01, pages 341–346, New York, NY, USA,


91

900 800

Duration (in seconds)

700

Preparation (64x64) Preparation (128x128) Synthesis (64x64) Synthesis (128x128)

600 500 400 300 200 100 0

Reduced

HalfReduced

Full5x5

Full7x7

Figure 10: Input exemplars where our synthesis algorithm fails to produce an acceptable result.

Figure 9: Preparation and synthesis duration for different neighbourhoods and exemplar sizes. 2001. ACM Press. [7] A. A. Efros and T. K. Leung. Texture synthesis by non-parametric sampling. In Proceedings of ICCV ’99, pages 1033–1038, Washington, DC, USA, 1999. IEEE Computer Society. [8] D. J. Heeger and J. R. Bergen. Pyramid-based texture analysis/synthesis. In Proceedings of SIGGRAPH ’95, pages 229–238, New York, NY, USA, 1995. ACM Press. R [9] Intel Corporation. Open Source Computer Vision Library. Url: http://www.intel.com/technology/ computing/opencv/index.htm [checked: 16/12/2007]. [10] R. Jagnow, J. Dorsey, and H. Rushmeier. Stereological techniques for solid textures. In SIGGRAPH ’04: ACM SIGGRAPH 2004 Papers, pages 329–335, New York, NY, USA, 2004. ACM Press. [11] J. Kopf, C.-W. Fu, D. Cohen-Or, O. Deussen, D. Lischinski, and T.-T. Wong. Solid texture synthesis from 2d exemplars. In SIGGRAPH ’07: ACM SIGGRAPH 2007 papers, pages (2.1)–(2.9), New York, NY, USA, 2007. ACM Press. [12] V. Kwatra, I. Essa, A. Bobick, and N. Kwatra. Texture optimization for example-based synthesis. In SIGGRAPH ’05: ACM SIGGRAPH 2005 Papers, pages 795–802, New York, NY, USA, 2005. ACM Press. [13] V. Kwatra, A. Sch¨ odl, I. Essa, G. Turk, and A. Bobick. Graphcut textures: image and video synthesis using graph cuts. In SIGGRAPH ’03: ACM SIGGRAPH 2003 Papers, pages 277–286, New York, NY, USA, 2003. ACM Press. [14] S. Lefebvre and H. Hoppe. Parallel controllable texture synthesis. In SIGGRAPH ’05: ACM SIGGRAPH 2005 Papers, pages 777–786, New York, NY, USA, 2005. ACM Press. [15] S. Lefebvre and H. Hoppe. Appearance-space texture synthesis. ACM Transactions on Graphics, 25(3):541–548, 2006. [16] F. Manke. Texture Transfer Functions for Direct Volume Rendering. Master’s thesis, The University of Auckland, 2008. [To be published]. [17] K. Perlin. An image synthesizer. In Proceedings of

[18]

[19] [20]

[21]

[22] [23]

[24]

[25] [26]

SIGGRAPH ’85, pages 287–296, New York, NY, USA, 1985. ACM Press. J. Portilla and E. P. Simoncelli. A parametric texture model based on joint statistics of complex wavelet coefficients. International Journal of Computer Vision, 40(1):49–70, 2000. E. Praun, A. Finkelstein, and H. Hoppe. Lapped textures. In Proceedings of SIGGRAPH ’00, pages 465–470, New York, NY, USA, 2000. ACM Press. X. Tong, J. Zhang, L. Liu, X. Wang, B. Guo, and H.-Y. Shum. Synthesis of bidirectional texture functions on arbitrary surfaces. In Proceedings of SIGGRAPH ’02, pages 665–672, New York, NY, USA, 2002. ACM Press. G. Turk. Generating textures on arbitrary surfaces using reaction-diffusion. In Proceedings of SIGGRAPH ’91, pages 289–298, New York, NY, USA, 1991. ACM Press. L.-Y. Wei. Texture Synthesis by Fixed Neighborhood Searching. PhD thesis, Stanford University, 2002. L.-Y. Wei. Texture synthesis from multiple sources. In SIGGRAPH ’03: ACM SIGGRAPH 2003 Sketches & Applications, pages 1–1, New York, NY, USA, 2003. ACM Press. L.-Y. Wei and M. Levoy. Fast texture synthesis using tree-structured vector quantization. In Proceedings of SIGGRAPH ’00, pages 479–488, New York, NY, USA, 2000. ACM Press. A. Witkin and M. Kass. Reaction-diffusion textures. SIGGRAPH Computer Graphics, 25(4):299–308, 1991. S. Worley. A cellular texture basis function. In Proceedings of SIGGRAPH ’96, pages 291–294, New York, NY, USA, 1996. ACM Press.