Development of K-Touch TM Haptic API for Various Datasets Beom-Chan Lee1
J o n g - P h i l Ki m 2
Jongeun Cha3
Jeha Ryu4
Human-MachineComputer Interface Lab Dept. of Mechatronics, Gwangju Institute of S c i e n c e a n d Te c h n o l ogy(GIST), Republic of Korea
Human-MachineComputer Interface Lab Dept. of Mechatronics, Gwangju Institute of S c i e n c e a n d Te c h n o l ogy(GIST), Republic of Korea
Human-MachineComputer Interface Lab Dept. of Mechatronics, Gwangju Institute of S c i e n c e a n d Te c h n o l ogy(GIST), Republic of Korea
Human-MachineComputer Interface Lab Dept. of Mechatronics, Gwangju Institute of S c i e n c e a n d Te c h n o l ogy(GIST), Republic of Korea
ABSTRACT This paper presents development of a new haptic API (Application Programming Interface) that is called K-TouchTM haptic API. This API can provide users with haptic interaction for various data representations such as 2D, 2.5D (height field), and 3D dynamic as well as static object data. In addition, it supports kinesthetic and tactile interactions simultaneously for more versatile haptic interaction. It is made of scalable software architecture and provides more convenient software interface for users who are not familiar with haptic technology. K e yw o r d s : haptics, haptic API, kinesthetic, tactile, hybrid environments 1
INTRODUCTION 1
In general sense, ‘haptic’ means ‘touch’ that provides users with tactual sensation of virtual, real or augmented environments. Haptic technology allows touching exploration and interactive manipulation of virtual objects through haptic interfaces. Haptic interaction can be roughly classified into two modalities: kinesthetic feedback (force, motion) and tactile display (cutaneous touch). The kinesthetic feedback encompasses perception of large scale details, such as object shape and mechanical properties such as compliance. This is achieved through way of feedback from the muscular and skeletal system. Meanwhile tactile perception is through the mechanoreceptive nerve ending in the skin. Thus, it is primarily means of relaying information regarding small scale details that cause skin stretch, compression and vibration. Kinesthetic feedback and tactile display allow users to have deeper recognition of virtual environments and enhance a sense of immersion in these environments [1]. In various application areas such as medical science, industry design, education and entertainment, therefore, efficient and effective haptic display systems and rendering algorithms have been proposed for supporting immersive experience. Haptic rendering algorithm is a process including collision detection between haptic probe and virtual object and contact force computation corresponding to the collision. In general, geometry-dependent haptic rendering algorithms depend on the geometry representation of the object being touched. These algorithms can be classified into two basic categories: surface haptic rendering algorithms for surface data [2-5] and volumetric haptic rendering algorithms for volumetric data [6-8]. Although various haptic rendering algorithms have been proposed, creation of a haptic application is difficult for users
1 2 3
4
e-mail:
[email protected] e-mail:
[email protected] e-mail:
[email protected] e-mail:
[email protected]
who are less familiar with the technology because the process of collision detection and force computation is fairly complicate for complex virtual and/or augmented environments. Therefore, some tools have been developed to meet the needs of the general or expert users [9-15]. Typically, these haptic tools allow a software engineer to create haptic applications for a certain software package, software framework, hardware platform, computer system, operating system. The GHOST SDK is developed specifically for PHANTOM haptic devices [9]. Although the GHOST SDK can create some haptic applications, there are some limitations for applying it to diverse areas. It is difficult for users to access haptic servo loop because architecture of the GHOST is encapsulated. In addition, since both device and architectural level are interlocked with each other, users cannot access to each level for extending the architecture. To overcome these limitations, they launched another haptic toolkit, OpenHaptics [9]. This has two levels API: HDAPI and HLAPI. HDAPI is a low-level foundational layer
for haptics. It is best suited for developers who are familiar with haptic paradigms and sending forces directly. HLAPI is built on top of the HDAPI and provides a higher level control of haptics than HDAPI. HLAPI is developed on top of the well-established graphical OpenGL API. However it is still difficult to create derivative classes from the OpenHaptics architecture.
The ReachIn API was commercialized as the first haptic and graphic API that is independent of a specific haptic device [10]. The ReachIn API is based on the scene-graph description file format VRML (Virtual Reality Modeling Language). Thus virtual environments, controlled by the ReachIn API, are defined in VRML, but the nodes of the scene-graph are built in C++. The ReachIn API can be used in conjunction with PHANTOM [9] and DELTA [11] haptic devices as well as their proprietary ReachIn display that supports graphic and haptic co-location. However the ReachIn requires high-end components for creating haptic systems such as high performance workstation PC with dual CPU, graphics hardware (over NVIDIA Quadro FX 1400), stereo display components. The e-Touch API is the first API providing a set of open module haptic and graphic libraries [12]. The e-touch API supports both PHANTOM and DELTA devices and allows users to create haptic scenes featuring rigid meshes and simpler implicit-surface haptic objects. Moreover it allows users to create a haptic-based desktop environment where menus can be accessed using the haptic device, and supports the GHOST software toolkit. The e-Touch API includes a full set of commands for creating interfaces, tools, and navigation techniques. The H3DTM API is a C++ implementation of an X3Dbased, open source scene-graph API for the development of haptic applications [13]. It closely follows the X3D design,
and extends graphical rendering with haptic rendering. The H3DTM API has open module architecture and therefore can be expanded and modified as well as uses the C++ Standard Template Library. Haptic rendering is performed by OpenHaptics. In the H3DTM, haptic rendering can be easily disabled, allowing H3DTM to be optionally used as an X3D compliant API. The H3DTM API supports graphic and haptic colocation with three types of workbenches. The CHAI library by Conti et. al. [14,15] is a set of graphic and haptic C++ libraries that allow both high level and low level programming of haptic applications. The CHAI is an open source, freely available set of C++ libraries for computer haptics. It supports several commercially available three and six Degrees-Of-Freedom (DOF) haptic devices, and makes it simple to support new custom force feedback devices. It is especially suitable for education and research purposes, offering a light platform on which extensions can be developed. It also supports developer-friendly creation of multimodal virtual worlds by tightly integrating the haptic and visual representations of objects. The I-Touch haptic frame work by Pocheville et. al. [16] has been designed to provide an open architecture tools. Its purpose is academic and concerns the conception of a generic framework that is able to allow researchers in haptics to prototype quickly their application. Since this framework is designed based on modular structure, no haptic graph-scene is defined and haptics is directly from the dynamic simulation engine which computes contact force. Thus it supports software flexibility that allows creating application with haptic in a simple manner. The I-Touch is especially applicative and concerns a priori virtual prototyping with haptic feedback in industry. The Haptik by Pascale et. al. [17] is a component based SDK that guarantees a binary compatibility of applications with future version of devices and plug-ins. The Haptik library is an open source library with a component based architecture that acts as a Hardware Abstraction Layer to provide uniform access to haptic devices. It does not contain graphic primitives, physics related haptic algorithms or complex class hierarchies, but instead exposes a set of interfaces that hide differences between devices to the applications. The Haptik has a very simple API. Haptik Library is easy to adopt even in already existing projects because it has a simple and absolutely non-invasive API. Current haptic SDKs and APIs have been developed for creation of haptic applications easily. Nevertheless, each SDK or API has some limitations. For example, haptic rendering algorithms implemented in current SDKs and APIs support surface-based models. Thus they cannot support directly other data representation such as volume and height field (2.5D depth image) datasets without converting them into polygonal meshes. In addition, the rapid development of computer graphics domain has made it possible to provide virtual world with hybrid environments. Hybrid environments can contain diverse objects with various data representations at the same virtual world. Current haptic SDK or API, however, cannot support haptic interaction with hybrid environments because surface-based haptic algorithm cannot cover various datasets without preprocessing. In order to allow haptic interaction with diverse data representations by using existing haptic SDKs and APIs, object data should be needed additional efforts like unification and reconstruction of the hierarchical structure in real-time or different types of haptic rendering algorithm should be selectively operated with respect to data representation. For the former case, there are real-time issues of data conversion for large amount of datasets and dynamically changing or loaded objects. The latter case, deterioration of haptic update rate can be occurred especially for hybrid environments. Furthermore, these haptic tools are generally limited only to kinesthetic
feedback. Even though tactile sensation in virtual environments also plays an important role in recognizing surface properties such as roughness and texture, any functionalities or libraries in the architecture of existing SDKs and APIs are not provided for more versatile haptic interaction. This paper presents development of a new haptic rendering API (Application Programming Interface) called K-TouchTM haptic API based on the core haptic rendering algorithm that had been developed by authors [18,19]. It is designed for users to interact with virtual objects by kinesthetic and tactile modalities through haptic interfaces. The K-TouchTM haptic API is a set of haptic and graphic C/C++ classes. Users who are not interested in implementation details of haptic rendering can easily create visual-haptic scenes by using a large set of pre-implemented haptic and graphic algorithms. The architectural design of this API considered the following aspects: 1) supporting hybrid environments with various data representations (polygon, height field (2.5D), volume and image data sets), 2) supporting kinesthetic and tactile haptic interactions simultaneously, 3) efficient haptic and graphic rendering algorithm, 4) convenience of use, and 5) extensible software architecture.
This paper is organized as follows: Section 2 describes the concepts and key features of the proposed haptic API. Detailed hierarchical architecture is described in Section 3. Finally, Section 4 discusses proposed haptic API, and lists possible future works on the K-TouchTM haptic API. 2
CONCEPTS AND KEY FEATURES OF K-TOUCHTM API
The K-TouchTM haptic API is targeted for general users who are less familiar with haptics technology but desire to quickly and easily add haptics to their application. It can also support developers who are familiar with haptic paradigms. It is also designed to serve various application fields with kinesthetic and tactile modalities together, and to support haptic interaction with various data representations. The following key elements were identified as of fundamental importance: i. Independent of haptic rendering contents or data such as conventional polygon model, volumetric datasets, and 2.5D height field data representation. ii. Simultaneous kinesthetic and tactile interaction. iii. Easy to use in creating a visual and haptic scene. iv. Efficient graphic and haptic rendering processing and software architecture. In order to realize the points above the proposed KTouchTM haptic API have been designed with the following features. i) As mentioned in the previous section, all existing haptic rendering SDKs and APIs algorithms are based on surfacebased haptic rendering. Therefore current haptic tools are very difficult to support haptic interaction with hybrid environments with volume, height field (2.5D), and image data. The K-TouchTM haptic API is developed for haptic interaction with hybrid environments with various data representation by the core haptic rendering algorithm that is independent of the type of data representation in the haptic scene [18, 19]. Both collision detection and force computation are based on the graphics hardware. In general, graphics hardware is used to render virtual objects in windows coordinate. Thus all of graphical contents or data should be passed through graphics pipeline. Using graphics hardware, data for haptic interaction is acquired with a uniform representation with the LOMI concept [18]. With these characteristics, therefore, the K-TouchTM API is capable of covering various data representation such as conventional 3D
models, height field (2.5D) data, volume data and captured 2D image. ii) Most current haptic rendering libraries only allow single point contact interaction through kinesthetic paradigms. The KTouchTM haptic API, on the contrary, allows users to simultaneously interact with kinesthetic and tactile sensation. A PHANTOM, for instance, can interact with virtual objects not only by kinesthetic interaction but also by tactile display by combining the force-reflection PHANTOM device with external tactile system. The key element to accomplish this is the distinction between the kinesthetic and tactile rendering classes. A kinesthetic rendering class is a contact force level entity on the muscle. A tactile rendering class, on the other hand, is the entity that supports creating surface properties that can be applied to the skin surface directly. iii) The K-TouchTM haptic API is C/C++ based and is developed using Microsoft Visual C++. The architecture of the K-TouchTM defines basic methods for creating and maintaining a scene graph structure that represents the logical representation of graphic and haptic objects in environments. Therefore user can create a haptic application by using a few lines of C/C++ code easily. A sample code is: world=new KTouchWorld(); // Create and initialize haptic scene Viewport=new KTouchViewport(world); // Initialize Viewport for graphic rendering world->SetViewport(Viewport); // Add to haptic scene graph // Create static object node staticnode = new KTouchStaticObject(world); world->addChild(staticnode); // Add to haptic scene graph // Create static object staticobjpbuff = new KTouchPbuffObj(world); // Load a virtual object from .obj file format staticobjpbuff->CreateObject("top.obj", "top.bmp"); staticnode ->addChild(staticobjpbuff); // Add to haptic scene graph // Start current haptic device and rendering algorithm logically Tool->start(); Viewport->RenderObjects(); // Render virtual objects logically
iv) The K-TouchTM haptic API can be used as a high level tool to easily create a haptic scene. A haptic scene means that specific objects can be rendered haptically as well as graphically. A haptic scene is composed of two main parts: graphical contents and haptic interaction. The base root class of K-TouchTM API can be populated by various virtual objects, haptic device, properties of virtual environments, and haptic rendering algorithms. OpenGL-based graphic rendering of objects and a haptic interaction point is automatically taken care of by the haptic rendering engine. To improve the update rate of the graphic rendering, both conventional and state-of-the-art graphic rendering methods are implemented. Efficient graphic rendering is realized by indexed geometry, display list (using system cash memory), and vertex buffer object (using video
Fig. 1 : The K-TouchTM haptic API architecture.
memory). Since the proposed API is constructed with an Object-Oriented Programming (OOP) methodology, users can easily create or reuse derivative classes from hierarchical structure of the K-TouchTM haptic API. This API also supports major 3 DOF commercially available haptic devices such as the PHANTOM. In addition, K-TouchTM haptic API allows users to easily control and write drivers for other devices. 3
K-TOUCHTM HAPTIC API ARCHITECTURE
The proposed haptic API is created by connecting a chain of class components. The overall architecture of the K-TouchTM haptic API is shown in Fig 1. The KTouchObject class can be populated by various virtual objects. Virtual objects can be classified as a static or dynamic object by its dynamic properties. While the static objects represent backgrounds or three dimensional objects without motion properties, the dynamic objects represent moving objects as well as functional objects with specific properties such as buttons or sliders in a haptic scene. In addition, the haptic scene can be composed of a depth image (2.5D height field datasets) that includes depth image as well as an RGB image. The data representation of depth image is managed in the KTouchImage class. All objects in a haptic scene can be referred to the OpenGL-based graphic rendering and haptic rendering classes. In order to render various objects graphically, OpenGL-based graphic rendering class provides a user with rendering functions to display existing objects and supports virtual environment properties such as camera and light parameters. In case of haptic rendering, both KTouchDepth and KTouchLomi classes are responsible for performing collision detection and calculating forces that should be applied to the user to simulate the desired effect. A virtual object or image data is defined as pixels in the windows coordinates. Each pixel has depth value in addition to its RGB data, which is usually used to cull hidden surfaces and to create an appearance of three dimensions in a perspective plot. The KTouchDepth class acquires the depth information about existing objects. In order to get the depth values of three dimensional objects, six virtual cameras are located around the portion-of-interest. The acquired depth value is used to create and update the LOMI. The LOMI is the entity that contains local geometry information of a touching object corresponding to haptic probe. At the force computation stage, The LOMI is utilized for calculating contact response force [18,19]. When depth information is calculated by the graphics hardware, any object in the haptic scene can be rendered into the frame buffer or pixel buffer. The screen is composed of a rectangular array of pixels, each capable of displaying a tiny square of color at that point in the image. After the rasterization stage in graphic pipeline, the data are not yet pixels, but are fragments. Each fragment has coordinate data which corresponds to a pixel, as well as color and depth values. Then each fragment undergoes a series of tests and operations like blending, anti-aliasing and polygon offset. At the end of graphic pipeline, each object is finally written into the frame buffer and displayed on the screen. Normally, the OpenGL libraries are used for rendering into a window which is displayed on the screen. The depth information is acquired by depth buffer at six sides which is achieved by additional graphic rendering. For the stationary or rigid objects, pixel buffer (KTouchPbuff) is used to store six-side depth image of whole object by the off-line graphic rendering. Note that the pixel buffer rendering is performed only one time before haptic thread begins. For deformable objects, frame buffer
(KTouchFbuff) is used to store six-side local depth image of the object by the on-line graphic rendering. Since the frame buffer rendering is performed at every graphic rendering rate, the depth image of whole object is not required. Actually, both collision detection and force computation procedures of a point interaction are performed in the KTouch3dForceAlgo class by referring the KTouchDepth and KTouchLomi class. Tactile rendering is accomplished by the KTouchTactileAlgo class. These two classes are derived from the KTouchForceAlgo class. The haptic rendering scheduler takes charge of a kinesthetic and tactile rendering procedure and guarantees that the haptic rendering loop will have a 1 KHz update/refresh rate. Finally, both calculated force and tactile effects are applied to the user through kinesthetic and tactile devices. 4
APPLICATION EXAMPLES
Fig 2 shows preliminary application examples based on the KTouchTM haptic API. Fig. 2(a) and Fig. 2(b) show that users can touch real letter captured by a web camera and 2.5D depth image captured by the Z-Cam, respectively. Fig. 2(c) shows haptic interaction with 3D photorealistic datasets of polygons more than two millions. In order to support simultaneous kinesthetic and tactile sensations, a vibrotactile device that equipped simple 2 by 2 pin arrays is attached to a kinesthetic device (See Fig. 2(d)). Fig. 2(e) shows haptic game application.
Fig. 2 : Application examples by using the K-TouchTM.
5
DISCUSSIONS AND FUTURE WORKS
In this paper, the K-TouchTM API is proposed to create haptic interaction applications. Users who are not interested in implementation details of haptic rendering can easily create visual-haptic scenes using a large set of pre-implemented haptic and graphic classes. The proposed K-TouchTM API has the following advantages; firstly, the K-TouchTM API is capable of covering various data representation such as conventional 3D models, height field (2.5D) data, volume data and captured 2D image. That is, this API is independent of data representations, which allows users to interact directly with hybrid environments composed of objects directly. Secondly, architecture of this API is designed for supporting tactile display as well as kinesthetic interaction simultaneously. In order to display haptic texture sensation, tactile display should render a high bandwidth of spatial frequency different from low bandwidth (typically below 30Hz) of force feedback device. Therefore we considered tactile display with kinesthetic interaction to support more versatile haptic interaction. In the KTouchTM haptic API, a high level software architecture which deals with tactile information is designed for generating tactile rendering signals as well as a low level tactile control algorithm is implemented for controlling tactile actuators. Thirdly, the K-TouchTM haptic API can support large virtual environments by using the LOMI [18] concept as well as stateof-the-art graphic rendering methods. The LOMI is a spatiotemporal occupancy map that represents local geometry shape of
graphically rendered object. That is, the LOMI indicates small size of geometry shape corresponding to a portion of interest. Therefore it is not needed to contain whole data of virtual environments. Creation and update of the LOMI structure is independent of data complexity in virtual environments. With efficient graphic rendering and the LOMI concept, the KTouchTM haptic API can support large virtual environments composed of one million polygons under 1KHz haptic rendering and 25~30Hz graphic rendering update rates, respectively. Finally, with respect to easiness of use, users or researchers who are familiar with C/C++ programming can easily make a haptic scene with few lines of code. It may be also utilized for general users who are not familiar with haptics technology in detail to easily create haptic applications. General users can create haptic applications with preimplemented graphic and haptic functionalities like creation of application program using widely spread C languages. With various advantages of the K-TouchTM haptic API, various applications providing kinesthetic and tactile interactions can be created conveniently. The next phase of this work is to implement more advanced functionalities. The important considerations of designing a haptic API are easy contents generation and distribution. It means that a scene description framework for touchable contents, usually virtual environments, is needed. The Reachin[10], GHOST[9] and OpenHaptics[9] adopt VRML and the H3D[13] is based on X3D. However, VRML and the X3D don't consider streaming its data but just adopt download-and-play concept. Thus the user should wait for downloading the contents and then enjoy them. In recent days, since the multimedia available in the Internet is getting diverse and high quality, the scene description framework should be able to extend its data type to deal with and stream high quality, large amount data for the users not to wait for downloading. We are considering BIFS (BInary Format for Scene) in MPEG-4 standards that deal with various media objects and support streaming data for each media object. By adopting the MPEG-4 framework in our API, we are planning to apply the haptic concept to broadcasting. Furthermore, to provide realistic haptic interaction, elaborate graphic and haptic co-location system will be performed in the near future. To support different types of commercialized haptic device, other device classes will be implemented in the architecture of this API. Other important consideration in the proposed haptic API is to include material properties such as friction coefficient by separately assigning different coefficients of friction to different objects. For this purpose, Haptic User Interfaces (HUI) will be implemented by using the proposed haptic API. HUI will support more convenient haptic modeling operations such as edition, modification and creation of the haptic scene in order to support more intuitive haptic interaction. AKNOWLDEGMENT This work was supported in part by the Ministry of Information and Communication through the next generation PC project and the Realistic Broadcasting IT Research Center (RBRC) at GIST and by Ministry of Science and Technology (MOST) through the Immersive Contents Research Center (ICRC) at GIST.
REFERENCES [1]
[2]
[3]
[4]
[5]
[6]
[7]
[8]
[9] [10] [11] [12] [13] [14] [15]
[16] [17]
[18]
[19]
K. Salisbury, F. Barbagli, and F. Conti, “Haptic Rendering: Introductory Concepts”, IEEE Computer Graphics and Applications, vol. 24, no. 2, pp. 24-32, 2004. C. Zilles, K. Salisbury, “A constraint-based god-object method for haptic display”, IEE/RSJ Int. Conf. Proc. Intelligent Robotics and System, vol 3, pp. 146-151, 1995. D.C. Ruspini, K. Kolarov, and O. Khatib., “The Haptic Display of Complex Graphical Environment”, Conf. Proc. ACM SIGGRAPH, vol 1, pp. 295-301, 1997. K. Salisbury and C. Tarr, “Haptic Rendering of Surfaces Defined by Implicit Functions”, Proc. ASME. Dynamic Systems and Control Division, vol. 61, pp. 61-67, 1997. S. Walker and K. Salisbury “Large Haptic Topographic maps: Marsview and the Proxy Graph Algorithm”, Proc. ACM Symp. Interactive 3D graphics, pp. 83-92, 2003. W. McNeely, K. Puterbaugh, and J. Troy, “Six Degreeof-Freedom Haptic Rendering using Voxel Sampling”, Proc. ACM SIGGRAPH, pp. 401–408, 1999. D.A. Lawrence, C.D. Lee and L.Y. Pao, “Shock and Votex visualization Using a Combined Visual/Haptic Interface”, J. IEEE Visualization, 2000. A. Prior, K. Hanies, “The use of a proximity agent in a collaborative virtual environment with 6 degrees-of-freedom voxel-based haptic rendering”, WHC 2005, pp. 631-632, 2000. SensAble Technologies Inc., http://www.sensable. ReachIn , http://www.reachin.se Force Dimension Inc., http://www.forcedimension.com Novint, http://www.novint.com. SenseGraphics Inc., http://www.sensegraphics.com CHAI 3D, http://www.chai3d.org Conti F, Barbagli F, Morris D, Sewell C, “CHAI: An Open-Source Library for the Rapid Development of Haptic Scene”, IEEE World Haptics, Demo presented, 2005. A. Pocheville and A. Kheddar, “I-TOUCH: A framework for computer haptics”, Int. Conf. Prc. Intelligent Robots and Systems (IROS), 2004. M. de Pascale, G. de Pascale, D. Prattichizzo, and F. Barbagli, “The Haptik Library - a Component based Architecture for Haptic Devices Access”, Eurohaptic 2004, Poster, 2004. J.P. Kim and J. Ryu, “Hardware Based 2.5D Haptic Rendering Algorithm using Localized Occupancy Map Instance”, Int. Conf. Proc. Artificial Reality and Telexistence (ICAT), pp. 132-137, 2004. J.P. Kim, B.C. Lee, and J. Ryu, “Haptic Rendering with Six Virtual Cameras”, HCI international 2005, pp. 467-472, 2005.
