An Intuitional Interface for Invocation of Chinese Painting Henry Been-Lirn Duh* IDMI / ECE National University of Singapore Singapore
Chien-Hsu Chen**
Department of industrial design National Cheng-Kung University Taiwan
ABSTRACT This study establishes a new painting experience that entails an AR-based approach that augments regular freehand 2D sketches into a Chinese painting artwork. Within the study, the realistic diffusion of ink can be interactively generated for users to create works inspired by the Chinese painting style as well as to unveil the opportunity for users to model up their own augmented information without any pre-building processes, potentially spanning the applications of AR in alternate directions. KEYWORDS: Augmented Reality, Interaction design INDEX TERMS: J.5 [ARTS AND HUMANITIES]: and performing 1
Arts, fine
Raymond Koon Chuan Koh+ IDMI / ECE
National Cheng-Kung University
National University of Singapore
Taiwan
Singapore
3
SYSTEM OVERVIEW
3.1 Hardware and Software Apart from having a native pen-based interface, the system is also equipped with cameras. (Fig.1) Two cameras are required; one to detect stroke motions to trigger the algorithm for image processing and the other to recognize the marker for addressing the coordinates of virtual entities. In our study, the widely used HMD was not implemented and we introduced a laptop-based display instead. On the physical input aspect, each page is imprinted with a stamp like marker for the camera to calculate the orientation and distance. The set of coordinate information is generated as soon as the physical sketch works are conceived into AR.
INTRODUCTION
Benefiting from the assistance of technologies, we can sketch up our imagination and present our thoughts in various forms using interesting and interactive tools [1]. Augmented Reality (AR) is an assistive technology in such innovative trends to better introduce narrative content and support the impressive fusion of boundaries between real and virtual worlds. It reveals the benefits of AR technology to possibly help people import certain art styles into the digital realm. However, many AR applications have been pre-built with attractive information. This study hopes to create a flexible environment for users not just only to appreciate the AR works but also to be part of the art creation process. The main goal of this research is to bring Chinese painting into the AR environment, while keeping the creator’s intuitional creative inspiration and the essence of the art. The study presents a sketch-based AR environment that enables user to quickly create sketches that are inspired by Chinese paintings using familiar media such as paper and pen. The depth arrangement and brush effects of the painted object are translated into an AR environment. 2
Chris Chun-Chin Su++ Department of industrial design
RELATED WORKS
Fig. 1. Workplace arrangement and physical setup
The system is entirely based on ActionScript and is separated into three main components, addressing and refreshment of AR coordinates and entity IDs, 3D model visualization and image segmentation and processing of the sketched works. AR recognition utilizes a marker-based tracking algorithm core from FLARToolkit, a derivative of ARToolkit (ARToolworks, Inc.). 3.2 Algorithm In the study, the system provides a sketch platform for the user to draw the simpliest strokes without considering how to control their pen pressure or color to present the painting style.
The motivation of the study was to look for a possible solution for people to recapitulate Chinese painting style in AR form. Digital augmentation has been widely applied in art performances. However, the tendency is also expansive within the AR context. Raphael et al. [2] and Fischer et al. [3] tried to inject more creatively crafted visual effects into AR, where AR information was given a series of rendering effects. Apart from the introduction of a real-time rendered AR environment, Napkin Sketch [4] forwarded the concept of having users to sketch on a physical 2D surface as they would on paper. However, our study introduces an intuitional sketching method of building 3D models based on the concept of AR. Fig. 2. Algorithm block diagram.
e-Mail:
*
[email protected]
[email protected]
**
[email protected]
[email protected]
IEEE International Symposium on Mixed and Augmented Reality 2009 Arts, Media and Humanities Proceedings 19 -22 October, Orlando, Florida, USA 978-1-4244-5465-5/09/$25.00 ©2009 IEEE
The system focuses on the contour brush effect, which reveals the dip appearance in order to address the Chinese painting features. System processing comprises of five parts (Fig.2); shape
53
segmentation, isolated shape contour analysis, stroke definition/filter, positioning and finally integration of PV3D and FLARtoolkit libraries.
counterpart(s) on paper. Photographs, sample hand sketches and software-derived imageries were used to evaluate the system (Fig.5).
3.3 Shape segmentation and Contour analysis Sketches in an image are isolated in accordance to the composition rule of oriental landscape paintings, where individual objects are extracted and recomposed in the painting. Before segmentation, the image’s color information is normalized, converted into binary and threshold bypassed for noise reduction. By threshold clipping, we can extract the object using a selected hue in the image. After luminance and binary operations and noise threshold clipping, shapes are extracted with corresponding component labels (Fig.3). By segmentation, contours are extracted and converted into images of actual brush strokes.
Fig. 5. Preliminary system evaluation. (a)(b) System environment. (c) User manipulation. (d)(e) Users’ analogy works into digital AR environment.
5
Fig. 3. Shape segmentation processes: (a) Original image input (b) Gray filter (c) Binary filter (d) Erosion and dilation operations with calculated color area to thresholding noise. (e) Object segmentation and color labelling. (f) Contour extraction.
3.4 Stroke definition and filter In Chinese painting (the chosen affecter), the subject (figures, landscapes, etc), painting speed and energy dispersion collectively affect stroke performance. We therefore extracted disciplinary traits from the traditional strokes to craft our filter so that the feature presentation can be as authentic as possible. The process is applied in steps; contour and interior renderings are done using two layers generated from the stroke’s varied dapple intensities. The purpose of contour rendering is to present the blur and (ink) overcharging effects which could be influenced by paper material and ink quantity (Fig.4).
CONCLUSION
This study digitizes the process of painting and encourages people to explore the stroke translation process into AR using familiar daily behaviors and props. We have established a user interface to better understand the augmented reality interactions and to depict how such techniques can be used to promote communication through sketches. The system is still limited by technology issues including image processing speed / transformation, brush effect algorithm and failures resulting from concentrated complex strokes by the user. Lastly, notwithstanding technicalities are concerns on the advancements of this system; user evaluation and usability issues will also be required. Having personally stylized virtual objects allows information to hold richer meanings and profounder interactivity. Several interesting AR applications (i.e. game, animation and education) can be extended from this point, providing a novel experience in augmented reality and the revelation of a flexible painting interaction. 6
ACKNOWLEDGEMENT
This project is co-sponsored by National Cheng-Kung University (Taiwan) and IDMI / ECE, National University of Singapore. REFERENCES Fig. 4. Brush effect applications in AR environment: (a) Original sketch (b) Image with applied filter.
3.5 Integration and Rendering After pre-processing, the residue imagery (with stroke effect applied) is ported into AR. It is placed onto a transparent 3D plane generated by PV3D. These planes follow the coordinates for corresponding positioning. FLARToolkit overlays the 3D objects on the printed marker. Hence as users sketch on paper, the image is rendered into stereo in real-time. 4
[2]
[3]
SYSTEM EVALUATION
The system has undergone limited evaluation to create assorted sketches. Initial user feedback indicated that brush effects sometimes did not perform as intended and segmentation failed. HMD should be implemented to directly overlay the AR
54
[1]
[4]
Richards, C. The design of effective ICT-supported learning activities: Exemplary models, changing requirements, and new possibilities. Language Learning and Technology, 9(1), Pages 60-79. 2005. Haller, M., Landerl, F., and Billinghurst, M. 2005. More sketchy, more AR, more fun!. In ACM SIGGRAPH 2005 Posters (Los Angeles, California, July 31 - August 04, 2005). J. Buhler, Ed. SIGGRAPH '05. ACM, New York, NY, 34. 2005. Fischer, J., Bartz, D., and Straβer, W. 2005. Artistic reality: fast brush stroke stylization for augmented reality. In Proceedings of the ACM Symposium on Virtual Reality Software and Technology (Monterey, CA, USA, November 07 - 09, 2005). VRST '05. ACM, New York, NY, Pages 155-158. 2005 Xin, M., Sharlin, E., and Sousa, M. C. 2008. Napkin sketch: handheld mixed reality 3D sketching. In Proceedings of the 2008 ACM Symposium on Virtual Reality Software and Technology. VRST '08. ACM, New York, NY, Pages 223-226. 2008
