www.scichina.com info.scichina.com www.springerlink.com
Haptic rendering for dental training system ¨ PeiJun2 , ZHOU RenGe1 , ZHOU WanLin1 WANG DangXiao1† , ZHANG YuRu1 , WANG Yong2 , LU 1
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing 100191, China;
2
School and Hospital of Stomatology, Peking University, Beijing 100081, China
Immersion and interaction are two key features of virtual reality systems, which are especially important for medical applications. Based on the requirement of motor skill training in dental surgery, haptic rendering method based on triangle model is investigated in this paper. Multi-rate haptic rendering architecture is proposed to solve the contradiction between fidelity and efficiency requirements. Realtime collision detection algorithm based on spatial partition and time coherence is utilized to enable fast contact determination. Proxy-based collision response algorithm is proposed to compute surface contact point. Cutting force model based on piecewise contact transition model is proposed for dental drilling simulation during tooth preparation. Velocity-driven levels of detail haptic rendering algorithm is proposed to maintain high update rate for complex scenes with a large number of triangles. Hapticvisual collocated dental training prototype is established using half-mirror solution. Typical dental operations have been realized including dental caries exploration, detection of boundary within dental cross-section plane, and dental drilling during tooth preparation. The haptic rendering method is a fundamental technology to improve immersion and interaction of virtual reality training systems, which is useful not only in dental training, but also in other surgical training systems. haptic rendering, computer haptics, haptic-visual collocation, dental training, hand-eye coordination
1 Background High performance virtual reality(VR) system should own three features: immersion, imagination and interactivity (3I). The trends of VR system are still progressing towards these three goals. Current VR system usually has strong capability in visual and audio display, and weak capability in haptic feedback. Therefore, the interactivity of the system is not strong enough to meet the requirement of interacting with the virtual world
with touch information. Just like the advent of computer graphics in the 1960s, computer haptics has been playing a more and more important role in high-fidelity VR systems[1−4] . In aerospace industry, flight simulator has become an important tool for training and evaluating a flight pilot before real flight. In medical surgery area, surgical simulator is a goal for researchers. However, most surgical simulators today cannot meet requirement of training surgeons. Compared
Received May 30, 2008; accepted September 17, 2008 doi: 10.1007/s11432-009-0062-4 † Corresponding author (email:
[email protected]) Supported by the National Natural Science Foundation of China (Grant No. 60605027, 50575011), and the National Hi-tech Research and Development Program of China (Grant No. 2007AA01Z310)
Sci China Ser F-Inf Sci | Mar. 2009 | vol. 52 | no. 3 | 1-18
with flight simulator, surgical simulator need to provide realistic force and touch feedback to the trainees, as well as hand eye coordinated environment. In this paper, high fidelity haptic rendering for dental training is studied. The research results are expected to provide fundamental platform to construct “what you see is what you feel” virtual reality training system, including dental training, minimal invasive surgery, eye surgery, etc. Furthermore, the haptic rendering method can be applied to virtual assembly and manual skill training. Traditional dental training methods include extracted tooth from patient, man-made tooth model and clinical practice by the novice. Compared with these traditional training methods, virtual reality training system can both simulate common situation and uncommon cases in clinical environment, which could help the students to learn the diagnosis and operation skills. Furthermore, the three-dimensional motion and force signal can be recorded and played back for analysis, which will be beneficial for teaching and certification. Based on the trends of virtual reality technology, dental training system has been studied in recent years. Dental drilling simulation system was developed in Alpha Tech Inc. in U.S., which can simulate cutting using various shaped drilling tools. However, the output force was not sent to haptic devices. DentX Inc. in Israel developed DentSim system, which can measure the motion of trainees using a photonic motion capture system. The measured data can be compared with that of experts. And the evaluation of the skill level can be determined. SimuLife system Inc. in France developed a dental training system, which provided different visual inspection ways including X-ray, mirror display, transparency display, etc. The dental training system developed by Nagoya University in Japan can demonstrate the righthand position and correct manipulation force during dental operations. The dental training system developed by BioRobotics Lab in Stanford University utilized two haptic devices to train bi-manual operation. The dental training system developed by Illinois University can support multiple view
2
observation during tooth inspection[5] . In our previous work, we developed dental training prototype to simulate dental caries exploration and cavity preparation against single tooth[6] . Current dental training system can provide preliminary haptic feedback to trainees, which cannot maintain the realistic feeling needed by dentists. How to improve the force fidelity and also realize haptic-visual collocation? How to extend the haptic rendering method to complex scenes with a large number of polygons? It is necessary to study haptic rendering algorithms to meet these requirements. Modeling is the fundamental problem in haptic rendering technology, which includes acquisition of physical properties of the simulated objects and organization of the collected data into appropriate digital models. Typical acquisition method utilizes laser scanning and computer vision to get point cloud data on the surface of objects and then makes three-dimensional reconstruction based on the data. The limitation of this method is that physical properties of the manipulated objects cannot be obtained, such as stiffness, damping, frictional coefficients, etc. Therefore, new methods need to be proposed to obtain physical properties of the manipulated objects. For the digital model, typical models used in haptic rendering area include points shell[1] , polygons and volume model. Commercial haptic rendering toolkits such as GHOST SDK and OpenHaptics adopt triangle mesh to represent the virtual objects, and use spring-damping force model to represent the force characteristics[2] . Liu et al.[7] proposed a finite element based modeling method for tooth canal surgery. Multi-resolution modeling method has been extensively studied in computer graphics literatures aiming at simulation of complex environments[8−10] . There are many open challenges in extending these methods into haptic rendering, which include analysis of the requirement of human haptic sensation threshold on the size of the triangle elements, and improvement of the computational efficiency to meet 1 KHz update rate of haptic loop, etc.[1] . Asano et al.[11] proposed levels of detailed rendering method based on
WANG DangXiao et al. Sci China Ser F-Inf Sci | Mar. 2009 | vol. 52 | no. 3 | 1-18
GHOST SDK, which can realize gradual change of visual and force feeling when the user observes and touches the virtual sculpture from a far distance to a close distance. The limitation is that the method cannot support continuous resolution control of the object model. We should explore novel modeling methods that can support both geometric and physical property description. Furthermore, it is valuable to study multi-resolution model based on human perception threshold. Finally, the multi-resolution model should be able to be constructed and updated in real time to meet 1 KHz haptic update rate requirements. Rapid and accurate collision detection algorithm is the second challenge for realistic haptic rendering. Compared with 30 Hz graphical rendering loop, higher update rate up to 1 KHz is re-quired to meet human high bandwidth perception characteristics. Therefore, high efficiency is needed for all algorithms within a haptic loop, including collision detection, force computation, collision response and model update for the manipulated objects[1] . Lin et al.[12] proposed a rapid collision detection method based on spatial partitioning and time coherence. Their method could get accurate collision information; however, it can only handle objects with several thousands mesh elements. In order to meet the high update rate for complex scenes, mesh simplification methods are usually adopted. El-Sana et al.[13] introduced multi-resolution mesh in haptic rendering, while refining mesh is adopted within local area of the probe point and coarser mesh are used in other areas. Otaduy et al.[14] constructed multi-resolution bounding box for convex polygons, which cannot support progressive transmission. Therefore, their method is not suitable for computers with a finite size memory. Liu et al.[15] proposed a local subdivision method to improve the update rate of the collision method. Zhang et al.[16] subdivided interested area on the coarser mesh for deformable object simulation. They limited the number of the mode nodes, thus achieving tradeoff between high update rate for haptic rendering and detailed fea-
ture display for graphical rendering. Force model is the third challenge for haptic rendering. Haptic rendering system is human-inthe-loop system. There exist bi-lateral power and information flow between humans and virtual environment. Stability will be violated when the output signal of the virtual environment does not match the capability of connected haptic device. Vibration and noise of the haptic device will occur when the system becomes unstable, even the human operator will be harmed by the unstable machine. Therefore, haptic rendering algorithms need to provide fidelity of simulation according to physical property of the manipulated objects, and also to ensure stability of the system and computational efficiency. Based on the results of collision detection, there are usually two kinds of methods to compute interaction force between tool and object: geometry-based method and physical-based method. In the latter kind of methods, mass-spring method and finite element method are two typical solutions. Mor et al.[17] summarized the advantage and disadvantage of various force models from the characteristics of surgery operation. Song et al.[18] derived deformation computation equations based on mass-spring sampling and polynomial fit method. Yuan et al.[19] studied deformation and force computation method for soft tissue cutting. Liu et al.[7] computed interaction force based on finite element model for tooth canal operation. The method can provide fairly high force accuracy, but the computation efficiency cannot meet 1 KHz requirement. High efficiency collision detection algorithm and force model are still open problems for complex scene with large number of triangles. Furthermore, the selection criteria of levels of detailed type and the switching method between successive LOD need to be further explored. For example, what is the respective advantage of off-line and online mesh subdivision? How can we determine the switching method based on human perception characteristics so that the rendering could be more natural? These problems are not solved yet in literatures.
WANG DangXiao et al. Sci China Ser F-Inf Sci | Mar. 2009 | vol. 52 | no. 3 | 1-18
3
2 Computational architecture of haptic rendering 2.1 Requirement of dental haptic simulation The haptic rendering in dental surgery simulation should realistically reflect the interaction force between dental tool and teeth. Figure 1 shows the structure of a tooth and Figure 2 shows some dental tools. As shown in Figure 1, a tooth has a tiny geometrical character on its surface and several different tissues in its interior. The functions of a dental surgery training system are summarized as follows: • Simulate typical surgeries, including dental caries exploration, tooth preparation, perceiving physical property of multiple tissues on the cross section of a tooth. • Provide realistic haptic feedback: The system should provide the haptic feeling of the tiny geometrical feature along tooth surface, such as the shape of cusp and fossa on the occlusal surface, and provide the difference of physical properties between carious tissue and healthy tissue in exploration operation. Furthermore, the haptic feedback in dental tooth cavity preparation or drilling process is needed. • Provide multi-resolution haptic display: The training system should not only provide fast haptic interaction against multiple teeth in the whole mouth, but also the refined haptic interaction to local scene, such as a carious area of the tooth.
Figure 2
Typical dental tools (Probe and drill).
The goal of this paper is to develop a hapticvisual feedback dental surgery training prototype to meet the requirement of dental surgery training. The expected effect of the system is shown in Figure 3. The system can provide the training environment with collocated haptic-visual display like the real dental surgery. Human operators can see the virtual teeth, virtual dental tools and their real hands in the system, which can assist them to learn the position and gesture of hands and the required manipulation force in surgery.
Figure 3
The expected effect of dental surgery training system.
(a) Haptic-vistral collocared effect; (b) visual scenario; (c) haptic scenario.
Figure 1
4
Geometric and physical structure of a tooth.
In order to improve the system performance, the dynamic force model between virtual tool and vir-
WANG DangXiao et al. Sci China Ser F-Inf Sci | Mar. 2009 | vol. 52 | no. 3 | 1-18
tual object needs to be formulated based on physical laws, which means to develop a high fidelity haptic rendering technology including modeling of virtual environment, collision detection and dynamic response algorithms, etc. In detail, haptic rendering problems include the motion mapping from hand to virtual tool, the dynamic force models consisting of multi-dimensional components, including contact force, friction force, cutting force and torque. Furthermore, real-time reconstruction of object caused by operation, such as drilling and deformable object is a challenge to tooth preparation simulation. Finally, it is important to reflect the detailed force information, such as carious holes and viscous effect. 2.2 Multi-rate haptic rendering architecture Realistic force rendering faces some challenges. On the one hand, refined representation of an object model is needed to guarantee the force fidelity, which would bring about a large amount of calculation. On the other hand, the calculation has to be reduced in order to ensure real-time performance. Furthermore, calculation differs under different op-
Figure 4
eration conditions (such as tooth surface exploration and tooth-cutting). Accordingly, multirate haptic rendering architecture is proposed to achieve the tradeoff between fidelity and real-time performance. A flowchart of the method is shown in Figure 4. The offline-module includes virtual environment modeling (tooth model and tool model). The online-module includes the tool tracking system, the real-time force rendering system, etc. The architecture shown in Figure 4 has the following functions: • Support multi-resolution force interaction: triangle mesh model with different levels of details (LOD) is built, and a velocity driven LOD haptic rendering algorithm is proposed to show the diversities of haptic feedback accuracy under different operating conditions. On the premise of ensuring real-time performance, which means that the refresh rate of force signal should be more than 1 KHz, the complex degree of the model is a decrease function of allowed maximum interaction velocity. According to this principle, contact determination algorithm based on LOD model is proposed to determine the contact conditions, which will be discussed in detail in section 3.2;
The multi-rate architecture of haptic rendering.
WANG DangXiao et al. Sci China Ser F-Inf Sci | Mar. 2009 | vol. 52 | no. 3 | 1-18
5
• Support multi-rate computing architecture: Different update rates are selected by haptic engine in different simulations according to their respective amount of calculation. Dual-thread is adopted in contact operation such as tooth exploration, and triple-thread multi-rate computing architecture is adopted in cutting operation. More details are to be discussed in subsection 3.3; • Provide uniform modeling for different interactive modes, and thus integrate contacting and cutting operation in one system architecture. Provide an algorithm for cutting simulation based on the conversion of cutting conditions to piecewise contact conditions and avoid building a complex dynamical model in cutting operations.
3 Key technology in haptic rendering 3.1
Digital modeling for virtual objects
Modeling method of virtual objects is a fundamental problem for haptic rendering. Different from modeling method in computer graphics, both geometric and physical properties need to be considered in the haptic model. The data structure should have a high efficiency to match the update rate of haptic rendering. In a dental training system, the goal is to train precise manipulation skill for hand muscle. For example, in surface exploration operation, dentist use a probe to feel the hardness and roughness of tooth surface and to detect the location and size of carious area. Because the interested feature may be very small, the size of the triangle to construct the tooth should be small enough to reflect the detailed feature of the carious part. Therefore, the total number of triangles will be fairly large for multiple teeth. From the scanned data, the numbers of vertices of the triangle mesh for modeling all the teeth in the mandible exceed 143865. Update rate cannot be maintained over 1 KHz for such big data. Therefore, it is necessary to propose a modeling method that can support both manipulation against coarser model for the multiple teeth, and manipulation against finer model of specified 6
tooth to detect the detail. The difference between the resolution of the coarser model and the finer model can be easily controlled by users. Furthermore, the resolution of the model can be changed according to the update of human operator’s viewpoint and position of the tool. In this paper, triangle mesh model is used for tooth and point model is used for tool. In order to solve the contradiction between fidelity and efficiency, multi-resolution haptic rendering method is proposed. Level of detail (LOD) model of the tooth is constructed as shown in Figures 5 and 6. Velocity of human hand is adopted as a criterion to select appropriate level for haptic rendering, which is compatible with human perception characteristics. Original data of the tooth is obtained by laser scanning method. Triangle mesh of the tooth is established by 3D reconstruction. Different physical parameters can be configured on different triangles on the mesh, including stiffness, damping, static and dynamic friction coefficient, etc. These parameters are stored along with geometric parameters in data structure of the triangle mesh.
Figure 5
Figure 6
Finer models of teeth in the mandible.
Coarser models of teeth in the mandible.
WANG DangXiao et al. Sci China Ser F-Inf Sci | Mar. 2009 | vol. 52 | no. 3 | 1-18
3.2
Contact determination algorithm
In haptic rendering loop, the contact determination algorithm needs to compute detailed collision information between virtual avatar and manipulated objects under real-time movement of human operator. In haptic literatures, the position of haptic device is usually denoted by HIP (haptic interface point). The position of the virtual avatar is usually denoted by SCP (surface contact point). Based on the principle of impedance display haptic device[20] , the two points are collocated when the interaction status belongs to free space (no collision between the tool and the object). The two points become separated when the interaction status changes into constraint space The function of contact determination algorithm is to compute corresponding SCP according to HIP information, which constitutes input information for force computation model. Contact determination algorithm can be divided into two procedures: collision detection and collision response. The task of collision detection is to detect whether there is collision between the avatar and the objects, and the task of collision response is to compute detailed collision information such as penetration depth, penetration direction, accurate location of contact point, local normal at contact
Figure 7
point, etc. Because manipulated objects (such as tooth) are usually consisting of geometrical complex surface with concave features, it is a challenge to compute accurate collision information under online movement of tool. It is necessary to solve complex computation and high update rate for the contact determination algorithm. Traditional collision detection methods, such as RAPID, GJK, etc., usually compute intersection using the position of moving objects in current simulation loop[12] . The historical movement information of tool is not considered, which will cause force ambiguity at some special geometrical features such as convex corners, thin objects, etc. These methods cannot reflect the influence of tool moving trajectory on the force result. Contact determination algorithm based on LOD model is proposed to deal with large mesh during multiple teeth exploration task. Velocity of human hand is adopted as a criterion to select appropriate LOD model. As shown in Figure 7, three subalgorithms are included: (1) Real-time collision detection algorithm that exploits time coherence to consider moving trajectory of the tool, and exploits spatial partition to accelerate the search process in a global-local two phase searching manner; (2)
Contact determination algorithm based on LOD model.
WANG DangXiao et al. Sci China Ser F-Inf Sci | Mar. 2009 | vol. 52 | no. 3 | 1-18
7
virtual proxy based collision response algorithm that fully considers the penetration characteristics of impedance display device; (3) SCP mapping algorithm that can maintain stability of the system at LOD model switch time. Uniformed spatial partition method is used to accelerate computation in collision detection algorithm[12] . A trajectory line is constructed to connect the position of the HIP between successive sampling times. Intersection point is computed between this line and the triangle mesh of the manipulated tooth. Therefore, force ambiguity can be avoided at some specific geometric features. In spatial partition method, the bounding box of the tooth is divided into voxel groups. The relationship between voxels and corresponding triangles are established. During the collision detection process, the intersected voxels can be located quickly when the location of the trajectory line of HIP is given. Through this method, the search scope is largely reduced from the whole triangle mesh to local submesh. Collision response algorithm is designed based on iteration of minimal distance within the topology of the triangle mesh. All the vertex, edge and triangle on the triangle mesh are called geometric primitives. A link list data structure is designed to record the neighbouring information among all the geometric primitives. The principle of the collision response algorithm is shown in following steps: 1) Get the intersection results from collision detection algorithm. The geometric primitive where the intersected point is located is defined as the initial active constraint primitive (ACP). 2) Get all the neighbouring primitives of the active constraint primitive and denote these primitives by candidate primitive group (CPG). 3) Compute the generalized distance from HIP to each primitive in the CPG. 4) Extract the minimal distance from the results in step 3), and denote the corresponding primitive by a new ACP. 5) If the new ACP is not the same as the initial ACP, then replace the initial ACP as the new ACP, and go to step 2); else, the intersection point 8
located on the new ACP is de-noted by the surface contact point (SCP). 6) If current HIP is within the boundary of the triangle mesh, then compute interaction force using spring force model and let the collision flag be true. Else let the collision flag be false. In Figure 7, in order to compute collision response, the current SCP has to be searched within the neighbor area of SCP in previous haptic loop. Therefore, an SCP mapping algorithm is proposed to ensure that the SCP search process can be solved along consistent triangle mesh. Otherwise, the haptic device will become unstable when the model is switched between LOD models. The reason is that computed force will change abruptly because the change of the penetration depth between the tool and the LOD model, or even the interaction status will vibrate between free space and constraint space. Therefore, SCP mapping algorithm is needed to maintain stability of the system at model switch time. Take a convex object as an example. When interaction velocity decreases from Va to Vb (as shown in Figure 8(a)), LOD model will be switched from coarser level Li−1 to finer level Li . It can be seen that the interaction status will continue belonging to constraint space at this switching time. Therefore, SCPi−1 should be mapped from coarser level Li−1 to finer level Li , otherwise, the SCP searching algorithm will compute wrong result for SCP. Similarly, when interaction velocity increases from Vb to Va (as shown in Figure 8(b)), LOD model will be switched from finer level Li−1 to coarser level Li . It can be seen that the interaction status will change from constraint space to free space at this switching time, and the collision detection process will be activated to determine the interaction status for next haptic loop. 3.3
Dynamic response
In the architecture in Figure 4, the functions of the dynamic response algorithm include: force computation between the tool and manipulated objects, and the motion or topology change of the objects caused by the interaction force. A corresponding
WANG DangXiao et al. Sci China Ser F-Inf Sci | Mar. 2009 | vol. 52 | no. 3 | 1-18
force model for tooth exploration and preparation is proposed in this paper according to the characteristics of different task. 3.3.1 Force model for tooth exploration In tooth exploration, the interaction force consists of normal force and friction force. Penetration based spring force model is used to compute normal force. Stickslip force model is used to compute friction force. Spring force model is typical in haptic rendering area[20] , which has high efficiency and can reflect the detailed variation of normal force under different contact statuses. Considering the big stiffness of tooth surface, big spring stiffness is utilized in the model to simulate hard contact feeling.
Figure 8
SCP mapping at switching time of LOD model. (a)
Velocity decreases from Va to Vb ; (b) velocity increases from Vb to Va .
With the collision response algorithm, the SCP is computed. The following model can be used for normal force: F n = Ke · (X V − X P ), V 1 · V n 0, (1) V 1 · V n < 0, F n = 0, where F n is normal force, and X P is the point of HIP in current haptic loop, and X V is the point
of SCP in current haptic loop. Ke is virtual spring stiffness to be determined by the physical property of contact material. V 1 is the vector from the HIP in current sampling period to the SCP in current sampling period. V n is the normal vector of the active constraint primitive in current sampling period. V 1 · V n < 0 means that the interaction status belongs to free space. Otherwise, the status belongs to constraint space. The model of friction force needs to consider several influences, including the relationship between the normal force and the friction force, the geometric feature of contact point, the effect of tool velocity on the friction force, etc. Two status force model is proposed to describe the friction force, i.e., the friction force will switch between static and dynamic friction force according to the tangential velocity of tool movement along the tooth surface. Sticky-slip point is defined to describe the status of the friction force. This sticky-slip point is static under the static friction status, and this point is sliding along the surface under the dynamic friction status. The following force model is derived: F St = KS (X S − X V ),
(2)
FD t
(3)
= KD (X S − X V ),
where X S is the position vector of sticky-slip point P ∗ , KS is the equivalent friction coefficient under static friction status, KD is the equivalent friction coefficient under sliding friction status, and F D t and F St are static and dynamic friction force respectively. As shown in Figure 9, when the operator moves the probe across two different tissues, the operator not only needs to feel different physical properties of the two tissues, but also the shape of the boundary. Unilateral virtual wall rendering method is proposed to render the boundary. As shown in Figure 10, a virtual wall is defined along the boundary, and the normal of the unilateral wall points towards softer area. Only when the probe is moved from the softer part to the harder part, will resistance force be produced along the normal of the wall. The stiffness of the virtual wall is the same as that of the harder tissue. 3.3.2 Rendering algorithm for tooth preparation Tooth preparation is a complex dynamic process
WANG DangXiao et al. Sci China Ser F-Inf Sci | Mar. 2009 | vol. 52 | no. 3 | 1-18
9
which involves elastic dynamic, plastic dynamic, fracture dynamic and frictional phenomena, etc. Finite element analysis is usually adopted to model the cutting dynamics, which can model the material property precisely but is time-consuming. For haptic rendering of tooth preparation process, triangle model based cutting simulation algorithm is proposed[6] . The main features of the algorithm includes:
Figure 9
Figure 10
Contact with tooth section.
Definition of virtual wall along the boundary.
• Cutting process is regarded as a piecewise contact process; thereby the problem of modeling cutting dynamics is avoided. • Material removal process is realized by local deformation and reconstruction of the triangle mesh, thus enabling topology change for tooth preparation process. • Local model based multi-rate haptic rendering architecture is proposed as shown in Figure 11, which aims to solve the contradiction between fidelity and computation efficiency. In Figure 11, tooth is modeled using triangle mesh, and tool is modeled using an implicit model, including its spherical and cylinder shapes. In order to compensate for the penetration property of impedance display device, a virtual tool is proposed to compute the parameters of local model.
10
Size scaling is introduced between virtual tool and physical tool to prevent big penetration between the virtual tool and the tooth. Two types of force model are compared in the above force rendering loop. One is spring force model. The relationship between contact force and removed material volume is established by introducing a transition model for contact status. Cutting process is described as transition between adjacent contact statuses. Therefore, cutting force can be computed using traditional contact force model. The stiffness in the force model can be selected by the data in contact process, which can maintain the stability of the system[6] . The other is damping force model. The real force data during tooth preparation is recorded by a force measurement system, and the damping parameters are fitted by least square method[21] . In this paper, the principle of the spring model is introduced and the detailed damping model can be found in ref. [21]. In spring force model, virtual wall is adopted as the local model, which is simple enough to maintain the 1 KHz update loop. With the movement of haptic device, the contact point between the tool and the tooth is evolving. As shown in Figure 12, the local model will be updated at a low update rate by cutting simulation loop. It should be noted that the parameters of the local model remains unchanged until in next cutting simulation loop. In haptic rendering loop, the unilateral spring force model is used: F e = Ke (X V − X S ), DOT < 0, F e = 0,
DOT 0,
where Ke is stiffness of the local model, and X S (xs , ys , zs ) is the virtual proxy (or SCP), which is the projection point of virtual tool point X V (xv , yv , zv ) to the boundary of the local model. Fast collision detection can be achieved with following parameters DOT = nLM · nS ,
(5)
which gives the contact status. is the normal vector of the local model, and nS = Normalize(X V − X S ).
WANG DangXiao et al. Sci China Ser F-Inf Sci | Mar. 2009 | vol. 52 | no. 3 | 1-18
(6)
Figure 11
Architecture of the multi-rate cutting simulation system.
for selected haptic device. The method is shown as follows: Fk = Fk−1 + δ · ΔF/ΔF , ΔF > δ, (7) ΔF δ, Fk = Fk , where δ is a predefined threshold for force change, Fk is the filtered force signal, and ΔF = Fk − Fk−1 ,
Figure 12
AUpdate of local model.
Based on Z-width criteria, there is an impedance range that can be stably rendered for a speci-fied haptic device[22] . A simple force filtering method is proposed to enhance the stability of haptic system
(8)
where Fk and Fk−1 are the virtual force signals in current and previous force rendering loop respectively. 3.3.3 Updating the triangle mesh of tooth Mesh representation of the tooth and tool is shown in Figure 13. In order to reflect topology change of tooth triangle mesh after material removal, the mesh needs to be modified according to the trajectory of the cutting tool. In dental cutting simu-
WANG DangXiao et al. Sci China Ser F-Inf Sci | Mar. 2009 | vol. 52 | no. 3 | 1-18
11
lation, material removed within each computation cycle is small because of brittle characteristic of tooth. Therefore, topology change of surface mesh is small within each simulation cycle. Simplified method combined mesh deformation with local retriangulation is proposed to simulate small material removal volume during progressive dental operation.
tain validity of the triangle mesh. After local mesh re-triangulation, area and size of the deformed triangles on the mesh can be maintained nearly equilaterally. Figure 16 demonstrates the evolvement of a mesh during cutting process. Coordinates of some vertices are changed after re-triangulation; therefore, two kinds of relational mapping need to be updated. One is the relation between triangles and the hash bounding box units, and the other is
Figure 14
Figure 13
Intersection between the tool and the tooth.
Cutting tool is defined as shown in Figure 14 to compute the removed material volume. There is an offset from the center of the cutting tool to the center of the virtual tool. The material removal velocity can be controlled by adjusting this offset. As Figure 15 indicates, parallel projection based vertex deformation method is used to update the tooth triangle mesh model. First, finding those vertices on tooth triangle mesh that intersects with the Cutting Tool, extend all these vertices along vector, until the extending ray intersects with tool’s sphere surface. Thus this intersection point is destination position of the deformed vertex. Via the above procedure, coordinates of these vertices are transformed to surface of the tool. Tooth model is updated and the formed cutting section reflects the shape of the tool. As Figure 16 illustrates, tri-division method is utilized on the abnormal thin triangle, where midpoint of two longer edges is used to divide the triangle. At the same time, adjacent triangles need to be reconstructed using bi-division method to main12
Relationship between cutting tool and virtual tool.
Figure 15
Deformation of the triangle mesh.
Figure 16 Evolvement of a mesh: (a) Before deformation (b) after vertex deformation; (c) local re-triangulation of thin triangles; (d) re-triangulation of adjacent triangles.
the relation between vertices and the hash bounding box units.
WANG DangXiao et al. Sci China Ser F-Inf Sci | Mar. 2009 | vol. 52 | no. 3 | 1-18
4 Experiments on typical dental operations
large range to achieve the simulation of different carious pathologies.
4.1 The configuration of experimental system The configuration of hardware and software of the simulation computer is shown in Table 1. The physical prototype of the co-located haptic-visual augmented reality platform is based on a halfsilvered mirror (Figure 17). The display-space of virtual object can be transferred through the halfsilvered mirror and overlaps with the actual scene, so the operator can get the visual information of the virtual object through the display, feel the force information by the force-feedback device. As is shown in Figure 18, the operator can observe the overlap of hand and virtual tool in the work volume, which augments the reality of operation. In the system, the Phantom desktop is the haptic feedback device, communicating with computer, which is used to model the environment and compute interaction with 1 KHz calculating frequency called back by GHOST SDK, via the parallel port. Table 1
Figure 17
Haptic-visual platform.
Configuration of hardware and software of simulation Hardware CPU
Intel(R) Core(TM)2 duo E4500
Memory
2.00 GB
Graphic card
Figure 18
Co-located haptic-visual display in dental operation.
Geforce FX 5700LE Software
OS
Windows XP
Developing language
Visual C++ 6.0
Developing kit
GHOST SDK, OpenGL
4.2
Teeth examination experiments
Teeth examination experiments include carious areas exploration on tooth surface and tooth section exploration. First, feedback on physical property of different tissues is tested. A carious area is modeled at the top area of a tooth as shown in Figure 19. When operator slides along tooth surface using the tip of probe, he/she can feel the difference between two kinds of tissues. The health area is slick and hardy and the carious area (darker area) is soft and sticky, which can help operator to detect the scope of the carious area. The stiffness and friction coefficient of the carious tissue can be adjusted in a
Figure 19
Tooth surface with carious section.
Second, the sense to tiny geometrical feature is tested. The surface of enamel has several geometrical features, such as smooth curved surface, groove, fossa, cusp, ridge, pit, and so on, which can synthetically test the ability of haptic rendering algorithm to simulate geometrical feature. In the experiment, the operator can feel clearly the change
WANG DangXiao et al. Sci China Ser F-Inf Sci | Mar. 2009 | vol. 52 | no. 3 | 1-18
13
of the direction of contact force at different shape features when the virtual tool contacts enamel surface and slides along different constraint surface including plane to edge, edge to plane, point to plane and point to edge. There is no vibration, and the haptic response is fast enough. The operator can feel the points and edges on the tooth surface, when the tool contacts the enamel surface. And the operator can feel stable interaction and switch between different shapes, when the tool crosses the edges of concave area and the edges of protruding area. When it moves to the point of concave area, the tool is restricted totally, as with the real situation. These results show that the haptic rendering algorithm avoids the ambiguity of the collision detection at the concave area. Then, experiments were carried out to test the stability against change of interaction modes. Human operator is required to contact the surface and leave from the object abruptly, thus causing the frequent exchange between unconstraint space and constraint space. The result can show the rapidity and stability of the rendering algorithm. On the other hand, human operator is required to maintain contact status and to slide the virtual tool along the object’s surface, which can be used to test the stably contacting effect. The experiment results show that haptic device has no apparent vibration under different interactive modes. It can be concluded that the algorithm can maintain stable interaction and provide high update rate. Finally, the effect of the boundary rendering between multi-tissues is validated via the experiment of tooth section exploration. The tooth section, as shown in Figure 20, has a size of 10.5 mm by 17.6 mm, with the thickest size of dentinal 0.2 mm. The stiffnesses of the tissues in a tooth are set at enamel 1 N/mm, dentinal 0.5 N/mm and dental pulp 0.2 N/mm. The force at the boundary of tissues in the tooth section is rendered based on the virtual wall method. It can be felt that the virtual probe can be hooked on the dentin when sliding from pulp cavity to dentin, and can be stuck with the tooth enamel when sliding from dentin to enamel. At the same time, the operator can feel that the stiffness becomes softer from outside to inside like stepping
14
downstairs.
Figure 20
Three kinds of tissues within tooth section.
Based on the results of the exploration experiments, it is shown that the force rendering algorithm can simulate the impalpable surface geometric features, such as the plane, the edge and the cusp in convex domains and concave domains. Furthermore, the algorithm can simulate the different physical attributes of multiple tissues, including carious tissues, normal tissues, and tissue boundary within the tooth section. 4.3 Interactive experiments for large scale data According to the performance verification requirements of force rendering for large scale data, a multi-tooth exploration experiment is designed in order to verify the feasibility of haptic rendering algorithm based on velocity drive LOD. In the oral model as shown in Figure 21, concave shape coexists with convex shape on the tooth surface. There are some special geometric features, such as small dents and salients about 0.1 mm in size. The complex surface is used to verify the adaptive capacity of the algorithm.
Figure 21
Rendering of the multiple teeth in oral model.
Experiments are designed based on contrasting
WANG DangXiao et al. Sci China Ser F-Inf Sci | Mar. 2009 | vol. 52 | no. 3 | 1-18
between two kinds of models: single model and LOD model respectively. The original model for multiple teeth has 143,865 vertexes which are too large and exceeds the memory capacity. Therefore, the most precise hierarchy is selected as shown in Table 2. Table 2
The data of the LOD model for the oral model Level 0
Level 1
Number of Vertice
33140
13306
Level 2 7443
Number of Triangles
66224
26556
14830 Figure 23
In Figure 22, the curves of dynamic signals are obtained from the force interaction experiment in the level 0 model. From that, we find the vibrations of the curves, which cause intense vibration of the haptic device after collision occurs between the tool and the teeth. It fails to meet the requirement of fast interactive display of virtual environment.
Figure 22
4.4
Force signal based on the LOD model.
Cutting experiments
The aim of cutting experiment is to validate the tooth reconstruction efficiency. Based on the local model, two experiments are carried out. One is with the drilling mode, in which the tool’s moving trajectory is vertical to the tooth’s surface approximately. The other is with the tooth preparation mode, in which the trajectory of tool is almost parallel to the tooth’s surface. The relative positions of tool path and tooth model are respectively shown in Figures 24 and 25.
Force signal based on the refined model.
For the interaction based on LOD models, we choose V01 =30 mm/s and V12 =120 mm/s as the switching conditions in the experiment. In Figure 23, the curves of force signals are obtained under variable velocity based on LOD model. From the comparison in the force signals between Figures 22 and 23, we can find that the force curve under LOD model is smoother than that of single model. Furthermore, both fidelity and real-time performances are maintained when the exploration model is switched within the range of normal velocities. From the results, it is clear that the velocity driven LOD force rendering algorithm can meet the condition of stable interaction for the complex surface model.
Figure 24
Tool’s path in drilling.
The virtual force computed with the local model directly in the cutting simulation thread is shown in Figures 26 and 27. The vibration of the device the human operator feels is like striking a rugged surface. To eliminate the vibration, threedimensional virtual force signal is filtered using the formula (7). The filtered force signal under the
WANG DangXiao et al. Sci China Ser F-Inf Sci | Mar. 2009 | vol. 52 | no. 3 | 1-18
15
2.5 2.0 Force(N)
1.5
Fx Fy Fz
1.0 0.5 0
1 701 1401 2101 2801 3501 4201 4901 5601 6301 7001 7701 8401 9101 9101 9801
drilling mode and the burring mode are shown in Figures 28 and 29. After force filtering, the operator can feel the continuously changeable virtual force and the device works stably. As shown in Figure 30(a), the cutting sections are quite smooth. Scraped along the tooth’s surface with a cylindrical tool, the sections are shown as Figure 30(b).
−0.5
Time(ms) Figure 28
Force signal in drilling process (after Filtering).
2.5
Force(N)
2.0 1.5
Fx Fy Fz
1.0 0.5 1 701 1401 2101 2801 3501 4201 4901 5601 6301 7001 7701 8401 9101 9101 9801
0 −0.5
Time(ms) Figure 25
Tool’s path in burring.
Figure 29
Force signal in burring process (after Filtering).
Figure 30
Section shape during tooth preparation process.
2.5
Force(N)
2.0 1.5 Fx Fy Fz
1.0 0.5 1 701 1401 2101 2801 3501 4201 4901 5601 6301 7001 7701 8401 9101 9101 9801
0 −0.5 −1.0
Time(ms) Figure 26
Force signal in drilling process.
3.5
Force(N)
3.0 2.5 2.0 1.5
Fx Fy Fz
1.0 0.5 1 701 1401 2101 2801 3501 4201 4901 5601 6301 7001 7701 8401 9101 9101 9801
0 −0.5
Time(ms) Figure 27
16
Force signal in burring process.
The result shows that the local model under the multi-rate architecture can simulate tooth preparation and the stable drilling with small drilling depth. Force computing efficiency could meet the requirements of update rate of force rendering loop. In addition, the user can achieve appropriate cutting sections using simple shaped tools such as spheres and cylinders. However, there is a shortcoming that the triangle-mesh deformation may not be suitable for simulating the drilling operator with a big depth.
WANG DangXiao et al. Sci China Ser F-Inf Sci | Mar. 2009 | vol. 52 | no. 3 | 1-18
5 Conclusion and future work In order to enhance the immersion and interaction of virtual reality dental training system, the haptic rendering method is studied. Haptic-visual collocated training prototype is established, which roughly demonstrates that the “what you see is what you feel” mode is beneficial for training hand eye coordination. The main conclusion of the paper can be summarized as follows: • The challenges of haptic rendering are identified: the first challenge is to provide 1KHz update rate for haptic loop; the second is to simulate the dynamic force and motion response with enough resolution; finally, the characteristics of virtual environments should be compatible with that of haptic device so that the stability of the haptic interaction system can be maintained; • Velocity driven levels of detailed haptic rendering algorithm is proposed. Experimental results show that the proposed algorithm can not only reflect the detailed force information at small geometric feature, but also can meet the 1KHz update rate under complex scenes owing multiple teeth made of thousands of triangle elements; • Local model based multi-rate cutting rendering algorithm is proposed. A deformation and reconstruction method of triangle model is proposed to simulate the topology change induced by material 1 Kenneth S, Francois C, Federico B. Haptic rendering: introductory concepts. IEEE Comput Graph, 2004, 24(2): 24–32 2 Srinivasan M A, Basdogan C. Haptics in virtual environments:
removal during cutting process. Experimental results demonstrate that the proposed algorithm can maintain the stability and computation efficiency under small volume removing during tooth preparation tasks. • Real-time collision detection algorithm based on consistent spatial segmentation and time coherence is proposed. Combined with proxy-based collision response algorithm, the contact determination algorithm can maintain the consistency and high update rate for contact status computation, which proves effective for objects with various geometric features. Furthermore, SCP mapping algorithm is proposed to re-compute SCP at the switch time between successive levels of detail models. Experiment results proved that the SCP mapping method can maintain the stability during complex model exploration. In future, we plan to improve the fidelity of cutting simulation by using voxel-based modeling and rendering method, which will be flexible to form various holes with high geometric accuracy. Furthermore, haptic rendering against deformable objects is necessary for extending the haptic training method to more kinds of dental operations. Last but not least, it will be necessary to invite dozens of dentists and dental students to practice and evaluate the performance of the system. 6: 671–683 7 Li M, Liu Y H. A virtual endodontics test bed for training root canal skills.
In: Proceedings of 2004 IEEE Interna-
taxonomy, research status, and challenges. Comput Graph,
tional Conference on Robotics and Automation, ICRA’04.Vol
1997, 21(4): 393–404
1. HongKong, 2004. 841–846
3 Saddik A E. The potential of haptics technologies. IEEE Instru Meas Mag, 2007, 10(1): 10–17
8 Hoppe H. Progressive meshes. In: Proceedings of Siggraph’96. 1996. 99–108
4 Hollerbach J. Some current issues in haptics research. In: Pro-
9 Zhou K, Pan Z G, Shi J Y. Smooth transition between levels
ceedings of the 2000 IEEE International Conference on Rob-
of detail of models. J Comput Aid Des Comput Graph, 2000,
otics and Automation. Salt Lake City: IEEE, 2000. 757–762
12(6): 463–467
5 Luciano C, Banerjee P, Florea L, et al. Design of the immersiveTouch: a high-performance haptic augmented virtual reality system. In: 11th International Conference on HumanComputer Interaction. Las Vegas, 2005. 22–27 6
Wang D, Zhang Y, Wang Y H, et al. Cutting on triangle
10 He H G, Jian J, Zhang X P, et al. A Survey on mesh simplification. J Softw, 2002, 13(12): 2215–2224 11 Asano T, Ishibashi Y. Adaptive display control of exhibits in a distributed haptic museum. In: The 3rd IEEE International Workshop on Haptic, Audio and Visual Environments
mesh: local model based haptic display for dental preparation
and Their Applications. Washington: IEEE, 2004. 19–23
surgery simulation. IEEE Trans Visual Comput Graph, 2005,
12 Lin M C, Gottschalk S. Collision detection between geomet-
WANG DangXiao et al. Sci China Ser F-Inf Sci | Mar. 2009 | vol. 52 | no. 3 | 1-18
17
ric models: a survey. In: Proceedings of IMA Conference on Mathematics of Surfaces. 1998. 37–56 13 El-Sana J, Varshiney A. Continuously-adaptive haptic rendering. Virt Envir, 2000, 135–144 14 Otaduy M A, Lin M C. Sensation preserving simplification for haptic rendering. ACM Trans Graph, 2003, 22(3): 543–553 15 Liu P, Shen X J, Georganas N. Multi-resolution modeling and
model and real-time force reflection algorithm. In: Proceedings of International Conference on Information Aquisition. 2004. 327–330 19 Song A G, Yuan K, Fu Y J. Haptic modeling and rendering based on neurofuzzy rules for surgical cutting simulation. Acta Automat Sin, 2006, 32(2): 193–199
locally refined collision detection for haptic interaction. In:
20 Zilles C B, Salisbury J K. A constraint-based god object
Proceedings of the Fifth International Conference on 3-D Dig-
method for haptics display. In: Proceedings IEEE/RSJ Inter-
ital Imaging and Modeling (3DIM’05). IEEE, 2005. 581–588
national Conference on Intelligent Robots and Systems. Pitts-
16 Zhang J, Payandeh S, Dill J. Haptic subdivision: an approach
burgh, 1995. 146–151
to defining level-of-detail in haptic rendering. In: Proceedings
21 Colgate J E, Brown J M. Factors affecting the Z-width of a
of the 10th Symposium. Haptic Interfaces for Virt Environ
haptic display. In: Proceedings IEEE International Confer-
Teleop Syst(HAPTICS.02). IEEE, 2002. 201–208
ence on Robotics and Automation. Los Alamitos, 1994. 3205–
17 Mor A B. Progressive Cutting with Minimal New Element Cre-
18
18 Wu J, Song A, Li J Q. A quick physics-based deformation
3210
ation of Soft Tissue Models for Interactive Surgical Simula-
22 Liu G, Zhang Y, Townsend W T. Force modeling for tooth
tion. PhD dissertation. Pittsburgh: The Robotics Institute,
preparation in a dental training system. Virt Real, 2008, 12(3):
Carnegie Mellon University, 2001. 19–26
125–136
WANG DangXiao et al. Sci China Ser F-Inf Sci | Mar. 2009 | vol. 52 | no. 3 | 1-18
