Depth Painting for Image-based Rendering Applications Sing Bing Kang Compaq Computer Corporation Cambridge Research Lab. One Kendall Square, Bldg. 700 Cambridge, MA 02139 e-mail:
[email protected]
Abstract We describe a system that functions as a painter’s tool for adding a third dimension in single images. In contrast with Horry et al.’s “Tour into the picture” system [5], which adds the third dimension to the single image through polygons and planar regions, our system provides a means for more expressive depth variations that result in more realistic rendering. The key to intuitive “depth painting” is the use of four auxiliary side views in the interface that allows the user to observe direct changes to depth. In addition to a host of local and region-based operations, “depth painting” also involves the option to pad regions within the image so as to remove holes or gaps when viewed at novel camera poses. Finally, the resulting image with depth lends itself to multi-layered imagebased rendering, where new environments can be created from different representations of 3-D scenes and rendered at novel viewpoints in a geometrically-correct manner. Introduction In the video and film industry, the animation process can take any number of various forms: the more conventional cel animation, the use of 3-D models and effects, and image-based rendering. The first two categories of animation techniques are typically laborious and time-consuming, but nevertheless, they create visually-pleasing results. Image-based rendering (IBR) techniques are based on using images as substrates for visualizing in 3-D as well as editing and manipulating 3-D scenes. Techniques that interpolate images in a geometrically consistent fashion are examples of IBR techniques. IBR techniques can obviously also be used to generate special effects that may not be geometrically consistent. It is not unusual for animation techniques to use 2-D layers as independent elements, with more sophisticated techniques employing warping operations to provide illusions of 3-D. However, it is more desirable to assign some reasonable depth distribution to each layer in order to generate more satisfactory 3-D effects. Being able to do this for any single image has the advantage of reusing or repurposing existing images, such as paintings or photos, in the animation process. Work done by Horry et al. [5] helps significantly in this process, though the output of their system is limited to polyhedral and planar approximations only. The proposed idea of depth painting allows users to add depth to single images in an analogous way that painters paint. The “brush,” (whose thickness can be controlled) rather than
adding paint, adds or subtracts depth instead. The interface for depth painting has the main view plus four auxiliary views corresponding to left, right, up, and down side views. This facilitates visualization of the user’s progress, and any change is quickly propagated across these views. In addition to a host of local and region-based operations, “depth painting” also involves the option to pad regions within the image so as to remove holes or gaps when viewed at novel camera poses. Depth painting enables single existing images (paintings, photos) to be more easily reused for animation, video editing, or multimedia authoring purposes. Current approaches for repurposing existing images are limited to simple representations, such as planar, affine warps, or polyhedral. Techniques that could assign arbitrary shapes to single images will be very useful in creating more realistic 3-D effects. Our system provides such a solution. Prior approaches Many of the work that dealt with layered images provide suggestions on rendering them efficiently (e.g., [2, 8]. Others dealt with recovering layers from multiple images (e.g., [10]), or recovering 3-D models from multiple images (e.g., [9]). Recently, an approach that allows the user to generate a 3-D effect for single images has been described [5]. This technique, however, allows the creation of an open polyhedral 3-D model only. Arbitrary shapes are more difficult to create using this approach. The depth painting system Figure 1 shows the overall functionality of our “depth painting” system. The system allows interactive specification of regions within the image as well as interactive change of depth. The regions are specified by tracing the edge of what the user perceives as a separate body within the image. Mistakes in region specification can be easily corrected by “brushing” over the pixels in question to remove from or add onto the current region. Obviously, more sophisticated techniques for interactive boundary extraction, such as the “Intelligent Scissors,” [7], can be used. Critical to the system is the display of four auxiliary side views corresponding to left, right, up, and down side views (right of Figure 1). These views help add the third dimension to the input image. They facilitate visualization of the user’s progress; any change in depth can be quickly propagated across these views.
Focal length/Field of View Control
Camera Viewpoint Control
Region edit, Global Z-shift, Rotate, Bump function, Zip boundary, Region padding, ... Region-based Operations
Interactive Region Segmentation Brush property change, Local Z change, ... Input Image
Local Operations Image with Depth
Figure 1: The depth painting system, with the interface shown on the right. This system allows the user to assign depth to a single image in an intuitive way. The operations used to modify the image depth can be classified as either local or region-based. During local operations, which can affect any part of the image, the user uses the mouse as a “brush” that adds or subtracts depth within the footprint of the brush. Region-based operations, however, affect only the current region of interest. The user can also manipulate the viewpoint of the central image at any time in addition to changing camera parameters such as focal length or fields of view. Local operations The user can adjust the footprint or size of the brush anytime. Moreover, the type of depth change distribution within the brush footprint can be specified (in the current implementation, either constant change, bilinear change, or gaussian distribution). By clicking and dragging the mouse, the user can then modify (adding or reducing, depending on the mouse button used) the pixel depths within the footprint of the brush. Region-based operations Once regions have been cut out, the user can use any of the the following operations on each region:
Figure 2: Result of cutting out bridge region and globally shifting its depth, and cutting out bush region and globally and locally shifting its depth.
• Use local bump function (shift is proportional to intensity of the pixel). This is useful in modeling rough surfaces such as water surfaces or trees.
Zipping boundaries The system allows the user to close up boundary discontinuities that may occur as a result of a combination of edit operations (e.g., shift and rotate operations on adjacent regions). The user can “zip” up these discontinuities by specifying the mode and clicking and dragging the mouse over the discontinuous boundary sections (which the system detects and highlights). An example of applying the “zip” operation is shown in Figure 3. Note, in particular, the circled areas before and after this operation.
After cutting out and manipulating the depths of both the bridge and bush regions, we get Figure 2. The final result after several more rounds of region cutting and depth manipulation is shown in the left of Figure 3.
Padding regions Another key feature to the system is the ability to add “finishing touches” in the form of padded regions to the existing regions. The padded regions are added by the user to the edges
• Shift (add or subtract) depth by the same amount • Rotate about X-axis or Y-axis • Use global bump function (where shift is proportional to distance of the pixel to the region boundary)
Figure 3: The effect of the “zip” operation: (Left) before, and (Right) after “zipping” the boundary below the bridge. The lines in red in the main view indicate depth discontinuities at region borders. Note the circled areas. of the specified regions in order to remove holes or gaps created as a result of depth discontinuities between regions that are exposed at various virtual viewpoints. The user can replicate textures or colors at these “padded regions.” One can use texture synthesis methods such as [1, 4], but for controllability and speed considerations, we use the simple copy and replicate operation. We have found this simple operation to be quite effective. Depth values for the padded region pixels are computed based on local neighborhood averaging.
Figure 4: Three separated regions of the Monet painting. Figure 6: Another example, with padded regions. plication of local and global bump operations, boundary zipping, and region padding. The original image is 240 by 256 pixels. Figure 5: Three separated regions of the Monet painting after padding. Figure 4 show three of the five segmented regions (bridge regions not shown) of the image of the Monet garden painting, while Figure 5 show the same set of regions that have been padded using the interactive technique of replicating textures. Another example Figure 6 shows results of “depth painting” another image, i.e., after applying a combination of region Z-shifting, rotation, ap-
Rendering issues Note that image-based rendering techniques are used to generate novel viewpoints; no 3-D models are explicitly created. This means that no piecewise planar or other global primitive shape approximation is used (unless desired by the user, since regions cut out are originally flat). The depth painting system can be used to generate “image” layers from single images. These “image” layers can then be used to create animations; sample snapshots (one every 20 frames) of an animation clip are shown in Figure 7. We use a variant of the painter’s algorithm originally de-
Figure 7: Snapshots of the short animated clip “Ginger and Willy in Monet’s Garden” (without padded regions). scribed in [6] to render the depth-painted image. The postforward projection step is followed by hole removal; here we use the well-known Elliptical Weighted Averaging technique [3], which employs interpolant kernels with adaptive sizes. Also, the pixel transfer computation is speeded up by separating the computation along a scanline into common and differential components. This is similar to that of [2]. Conclusions and possible extensions We have described our depth painting system that allows users to assign depths to single images in a rather intuitive manner. It is capable of local and region-based operations. This system can be used in conjunction with Horry et al.’s “Tour into the picture” system [5]; the user can use its output as a starting point, for example. Other region-based operations that can be implemented include fitting 3-D templates such as 3-D primitives (cubes, cylinders, ellipsoid, spheres) and 3-D specialized shapes (such as faces). Adobe PhotoShop 5, for example, implemented simple 3-D geometric shapes such as cubes in its image processing repertoire. Another interesting extension to the system is the notion of radiosity painting. According to this idea, the user interactively ascribe radiosity to pixels in the same manner, through local and region-based operations. Local computation of normal can be easily performed, since depth is known everywhere. By controlling the light source direction, the user can then change the appearance of the scene at a given virtual camera pose as desired. References [1] J. S. De Bonet. Multiresolution sampling procedure for analysis and synthesis of texture images. Computer Graphics (SIGGRAPH’97), pages 361–368, August 1997.
[2] S. J. Gortler, L.-W. He, and M. F. Cohen. Rendering layered depth images. Technical Report MSTR-TR-9709, Microsoft Research, Microsoft Corp., March 1997. [3] N. Greene and P. Heckbert. Creating raster Omnimax images from multiple perspective views using the Elliptical Weighted Average filter. IEEE Computer Graphics and Applications, pages 21–27, June 1986. [4] D. J. Heeger and J. R. Bergen. Pyramid-based texture analysis/synthesis. Computer Graphics (SIGGRAPH’95), pages 229–238, August 1995. [5] Y. Horry, K. Anjyo, and K. Arai. Tour into the picture: Using a spidery mesh interface to make animation from a single image. Computer Graphics (SIGGRAPH’97), pages 225–232, Aug. 1997. [6] L. McMillan and G. Bishop. Plenoptic modeling: An image-based rendering system. Computer Graphics (SIGGRAPH’95), pages 39–46, August 1995. [7] E. N. Mortensen and W. A. Barrett. Intelligent scissors for image composition. Computer Graphics (SIGGRAPH’95), pages 191–198, August 1995. [8] J. Shade, S. Gortler, L.-W. He, and R. Szeliski. Layered depth images. Computer Graphics (SIGGRAPH’98), pages 231–242, July 1998. [9] C. J. Taylor, P. E. Debevec, and J. Malik. Reconstructing polyhedral models of architectural scenes from photographs. In Fourth European Conference on Computer Vision (ECCV’96), volume 2, pages 659–668, Cambridge, England, April 1996. Springer-Verlag. [10] J. Y. A. Wang and E. H. Adelson. Representing moving images with layers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 3(5):625–638, September 1994.
