Virtual Reality (2006) 9: 118–132 DOI 10.1007/s10055-005-0012-4
O R I GI N A L A R T IC L E
Daniel Wang Æ Yuru Zhang Æ Chong Yao
Stroke-based modeling and haptic skill display for Chinese calligraphy simulation system
Received: 29 July 2005 / Accepted: 7 October 2005 / Published online: 11 January 2006 Ó Springer-Verlag London Limited 2006
Abstract The goal of this paper is to study haptic skill representation and display in a Chinese calligraphy training system. The challenge is to model haptic skill during the writing of different strokes in Chinese characters and to achieve haptic rendering with high fidelity and stability. The planning of the writing process is organized at three levels: task, representation and device level to describe the haptic handwriting skill. State transition graph (STG) is proposed to describe switches between tasks during the handwriting. Chinese characters are modeled using 39 typical strokes, which are further grouped into basic and compound strokes. The compound stroke is considered to be sequential combination of the basic strokes. Straight and curve strokes are modeled using line segment and the Bezier curve, respectively. Information from STG is used for real-time collision detection and haptic rendering. Ambiguity of the collision detection at stroke-corner points is prevented using active stroke combined with local nearest point computation. A modified virtual fixture method is developed for haptic rendering. The approach is tested on a prototype training system using Phantom desktop. Initial experiments suggest that the proposed modeling and rendering method is effective.
1 Introduction Learning to write Chinese characters by hand is difficult for beginners to learn because various skills are involved in order to co-ordinate motor output with sensory (visual and haptic) feedback for producing perfect D. Wang (&) Æ Y. Zhang Æ C. Yao Robotics Institute, Beihang University, 100083 Beijing, People’s Republic of China E-mail:
[email protected] Tel.: +86-10-82338023 Fax: +86-10-82338017
strokes. The sensory-motor skills of wrist and fingers need to be trained over a long period before the structure and sequence of strokes become familiar. It is valuable to explore ways to accelerate the learning process. Traditionally, there are mainly two training methods. The first is hands-on training by a teacher. The second is to practice on character templates sculpted on wood or printed on paper. Each method has its disadvantages. The first method is highly dependent on the skill of the teacher and is very time-consuming. The second method tends to be boring to the student who has to spend a long time repeating the same word over and over. Therefore, a haptically enhanced virtual reality system is proposed as an alternative for handwriting skill training. Solis et al. developed a Japanese character training system [1]. The emphasis of their system was to identify the intent of a user in real-time and to compare the user’s trajectory with those in a pre-defined character library to find the best match. Henmi and Yoshikawa recorded the motion and force information of a teacher during the writing of Japanese characters and used the recorded data to train students [2]. Teo et al. established a Chinese calligraphy training system using a six degree of freedom Delta haptic device [3] and a mirror that collocated visual and force information. The learning process was decomposed into motion guidance and path constraining. To develop a stroke-trajectory based training system of Chinese characters, it is necessary to study the kinesthetic cues that can lead to realistic force sensations. The kinesthetic cues should be quantified and be incorporated into an impedance controller of a haptic device. The modeling of haptic handwriting skills is still an open research problem. There currently exists no model of Chinese characters for collision detection between a virtual pen and virtual characters. Hsi-Ming Yang et al. adopted piecewise Bezier curves to model the contour of Chinese brush pen [4]. Chu and Chiew-Lan Tai utilized a skeleton model to simulate handwriting effects [5]. Information in these
119
models are not suitable for computing virtual forces during handwriting [6]. Henmi and Yoshikawa focused on skill transfer of hands-on teaching from teacher to students [2, 7–9]. Because modeling issue in hands-on teaching is different from standard character based training, therefore, modeling and training of writing skill based on standard characters is an unsolved problem. High efficient memory cost data structure is expected to store large number of Chinese characters and access speed for characters model information should meet 1 KHz update rate for stable haptic rendering [6, 10]. How to prevent ambiguity of collision detection at corners of complex strokes and maintain stability between interaction switches among different strokes, are still open problems. The goal of this paper is to model a user’s haptic skills during the Chinese handwriting process and to develop a training system for studying motor skill learning. It is expected that the system can be used to train young Chinese pupils and non-Chinese students to learn handwriting. Such a system can also be used as a platform for studying human haptic skills. The remainder of the paper is organized as follows. Section 2 describes a model of haptic skills and a threelevel task planner. Section 3 presents the modeling of Chinese strokes and characters. Section 4 proposes a haptic rendering architecture and algorithm for collision detection and force modeling. Section 5 discusses an experiment using a force-feedback haptic device. Section 6 presents conclusions and future work.
2 Task planning 2.1 Architecture In a real-time force-feedback Chinese calligraphy simulation system, a task-planning model is the fundamental component to ensure realistic force interaction. The goal of task planning is to establish a representation model of human haptic skills and fidelity evaluation criteria through an analysis of human actions during a Chinese hand writing process, and furthermore to decompose the skills into meaningful control commands for a haptic device. As Fig. 1 shows, a hierarchical task planning method is proposed to describe the hand writing process. In Fig. 1, the term task space refers to an abstract space for describing the handwriting process, and the term device space refers to the physical workspace of a haptic device. At the task level, the haptic skills involved in handwriting are decomposed into perceptible elements, that is, independent element perceptible by human operator. The inputs to the task level are Chinese characters and the outputs are perceptible elements of the interaction process. At the representation level, the perceptible elements are divided into interaction status to realize dynamic simulation and status switching control during the
Fig. 1 A hierarchical task plan method
interaction process. The inputs are task language units and the outputs are interaction status series between a virtual pen’s tip and a virtual paper. At the device level, the interaction status series are mapped into control signals in the device space that includes motion commands and control goals for the haptic device. Its inputs are interaction status series between a virtual pen’s tip and a virtual paper. The output is the control goals of the haptic device. The result supplies the planning goal for force rendering and device control. The task and representation levels are independent of the haptic device. Haptic skills and the interaction status, are respectively represented by these two levels. The device level is dependent on the specific device to enable stable simulation on the device. There are two advantages of using the three-level task planner. The first is that task planning is independent of the haptic device and extensible to different devices. The second is ease of debugging and control of the entire simulation, because the effect of each component on system performance can be tested independently and therefore, the overall system performance can be optimized. 2.2 Task level The task level focuses on the factors related to the fidelity of haptic simulation. Based on the characteristics of human haptic skills, a factor combination and status switching model is proposed to represent haptic skills of the handwriting process. Standard Chinese character templates are predefined as the goal and the reference of haptic simulation. Force display is divided into the display of structural infor-
120
mation and the display of sequences among strokes. Task planning and status series during handwriting are established based on the status switching method. Evaluation criteria are proposed for each perceptible element. The combined precision of all perceptible elements displayed on the haptic device is defined as the fidelity of the entire simulation system. As Fig. 2 shows, the perceptible elements during haptic simulation of Chinese calligraphy using a pen include a character contour element and a stroke sequence element. The character contour element refers to the force instructions along each stroke to reflect the trajectory of the stroke. As Fig. 3 shows, a user can feel the shape of each stroke and the writing direction within each stroke; therefore, haptic memory of the character’s shape can be enhanced. The stroke sequence refers to the movement patterns made by the user while writing a character according to the sequence of different strokes for the character. Following the standard order of strokes for each Chinese character is an important feature of writing skill. Two kinds of sequence should be classified. One belongs to the external sequence; i.e., the sequence ordering among different strokes. The other belongs to the internal sequence; i.e., the movement direction within a single stroke. Figure 4 shows the internal sequence by a horizontal stroke (the gray arrow). When the virtual pen is moving along the stroke from left to right, it is regarded as in the right direction. Otherwise, resistance force (the blue arrow) will be imposed on the user’s hand if the pen is moving from right to left (the red arrow). The internal sequence is realized by comparing the moving direction of the virtual pen with the predefined internal sequence stored in the model.
Fig. 3 Force feedback of shape of a curve stroke
The simulation procedure of the perceptible elements simulation process is as follows: Step 1: Input the Chinese character; Step 2: Load the required character data, including contour and sequence data. Parametric points describing the shape of the character is stored in the contour data. Identity (ID) of each stroke and index of each feature points are stored in the sequence data; Step 3: The standard character is displayed graphically and the haptic loop is started; Step 4: The currently active stroke is decided according to collision detection between virtual pen’s tip and the virtual character. Feature point array of active stroke is loaded and the user is guided by computed force of this stroke. The next stroke is set to be active when the current stroke is finished and all other strokes are disabled. The aforementioned computation is repeated until all strokes in the character have been written.
2.3 Representation level The representation level discusses the interaction types between virtual pen’s tip point (VTP) and virtual paper’s surface, or between VTP and each stroke. The switching condition that leads to a change of the status will also be described at this level. Action of virtual pen’s tip during writing of a single character includes moving the pen to contact the paper, moving the pen to retreat from the paper, writing on the paper, moving the pen’s tip with the help of visual feedback, moving the pen’s tip with the help of haptic feedback. Possible interaction status resulting from the actions includes virtual pen’s tip in contact or not in contact with the paper.
Fig. 2 Perceptible elements of haptic skill
Fig. 4 Force feedback of internal sequence
121
Fig. 6 State transfer graph of interaction state
Fig. 5 Bounding box and neighbor area hint
2.4 Device level
Figure 5 shows the bounding box of a reference character and the neighboring area of the start point of the first stroke of the reference standard character. Because the goal of haptic simulation is to guide the user along the strokes of a standard Chinese character, only movements within the bounding box of a standard Chinese character are regarded to be valid writing attempts. Interaction status during the contact phase can be further classified according to the distance from the VTP to start point of the first stroke of the reference standard character. Table 1 gives definition of all interaction states, where ‘‘system behavior’’ refers to expected result of haptic device and graphical device under each interaction status. State transition graph (STG) in Fig. 6 is proposed to describe the real-time interaction status between virtual pen and the virtual paper. The STG can be used to describe information flow among different status and switch condition between these states.
Device level discuss mapping of motion and force signal from task space to device space and focus of this level is to realize representation of interaction status and realtime haptic simulation. Two aspects are included in this level: (1) internal description of each status about motion and force series; (2) description of switch performance between every two status about stability and bandwidth of such switch. Considering the impedance display characteristic of haptic device, virtual pen’s tip will deviate from haptic interface point when the haptic probe enters constraint space [11]. In force rendering algorithm, mapping from haptic interface point to virtual pen’s tip should be established to transfer interaction status from device coordination system to virtual co-ordination system. Device’s behavior under each interaction status can be derived based on status of representation level. Table 2 gives characteristic of device’s output force and mapping of position signal. In order to describe concisely, HIP is used to represent haptic interface point and VTP is used to represent virtual pen’s tip.
Table 1 Definition of all interaction state State name
State definition
System behavior
Separation state (SS) Adjusting state (AS)
VTP is non-contact with the paper VTP contacts the paper, contact point is not within bounding box of reference character VTP contacts the paper, contact point locates within bounding box of reference character, but is not within neighboring area of start point of first stroke in reference standard character VTP contacts the paper, contact point locates within neighboring area of start point of first stroke in reference standard character
No force feedback VTP moves toward bounding box of reference character guided by visual feedback VTP moves toward neighboring area of start point of first stroke in reference standard character guided by feedback force perpendicular to the paper’s plane Force feedback is computed according to trajectory of operator’s hand movement and graphical display is real-time updated
Calibration state (CS)
Writing state (WS)
122 Table 2 Behavior of haptic device under each status Status name
Output force
Mapping of position signal
SS AS
Zero force Zero force
CS
Force perpendicular to the paper is not zero, and force within the plane is zero Both force perpendicular to the paper and force in the plane is not zero
VTP is same to HIP VTP deviates from HIP and HIP penetrate into surface of virtual paper VTP deviates from HIP and HIP penetrate into surface of virtual paper and VTP moves within the paper’s plane VTP deviates from HIP and HIP penetrates into virtual paper and VTP moves within the paper’s plane
WS
Stability problem will occur on the switch point of two states such as contact or non-contact of pen’s tip with the virtual paper, writing from current stroke to next stroke, or writing from current character to next character, and writing on stroke with large curvature or at corner point of strokes, etc. Output force of haptic device will increase or decrease violently and will mismatch with human’s active force. Therefore, the device will manifest vibration. Special force rendering algorithm will be discussed in Sect. 4 to ensure stability and continuity during transform between strokes.
It is well known that Chinese character is composed of strokes; naturally, modeling of character is transferred into modeling of strokes. Modeling method of Chinese character based on combination of sequential strokes and additional physical property is proposed in this paper. Within geometrical property subgroup, Chinese character is classified into combination of basic and complex strokes, while basic strokes is described by control points and complex strokes are modeled using sequential series of feature points. Within physical property subgroup, attraction factor is defined to reflect force magnitude difference on different control point.
3 Stroke-based character modeling 3.1 Typical strokes library Digital model of Chinese character is the fundamental component to simulate realistic and real-time handwriting process. Challenge of modeling is to meet requirement of fidelity and high force computation rate, and to meet adaptability for writing various characters.
Chinese character is combined from sequence of strokes. In order to improve storage efficiency and maintain arbitrary character modeling requirement, 39 typical strokes are selected to form arbitrary Chinese characters and stroke structure is modeled using stroke ID and stack sequence. The typical strokes library is shown in Fig. 7. Each typical stroke is assigned a unique ID. For a specified typical stroke, the ID is constant no matter the variation of the size and no matter the location of the stroke within the character. The ID is corresponding to inherent structural characteristic of the stroke instead of its location. The ID will be used to quickly check out each stroke in proper sequence for collision detection and force rendering. 3.2 Stroke combination method
Fig. 7 List of typical strokes
In this system, graphical rendering program is developed based on OpenGL, and force rendering is based on GHOST SDK and Phantom desktop device. Memory cost will be a large burden if different models are used for graphical and force rendering respectively. Therefore, it is needed to propose a unified model for both graphical and force rendering. Figure 8 gives modeling hierarchical logic of character. Chinese character is modeled as sequential combination of typical strokes and each typical stroke can be modeled using combination of stroke elements. There
123
Fig. 10 Model of basic strokes Fig. 8 Modeling hierarchical logic of character
are two kinds of stroke element: line segment stroke element (LSE) and curve segment stroke element (CSE). According to complexity level, each typical stroke can be classified into two groups, basic stroke or compound stroke. Each one is defined as follows: (a) Basic stroke means LSE and CSE. The model of the former is a line segment, and the model of the latter is third order Bezier curve. (b) Complex stroke is described by combination of several basic strokes using geometric transformation.Within geometrical property subgroup, modeling data of a character can be divided into two sets: structural data and sequence data in order to reflect the different property of the character. Structural data means relative co-ordinate of each control point within its parent character, and sequence data means sequence of control points within each character, which include index of each feature point and ID of each child stroke. Group of control points are used for model the structural data of each stroke, which also decide position, size, and orientation of each stroke within its parent character. All the strokes are defined to be constant width. Feature points are defined to describe sequence among the control points within their parent stroke. Feature point is special control points, which connect basic strokes into a compound stroke. Therefore, all typical strokes can be modeled to form arbitrary Chinese character while maintaining compact data structure. 3.3 Modeling of basic stroke
strokes, basic strokes can be classified into line segment and curve segment strokes. Line segment stroke include points, horizontal stroke, vertical stroke, etc. Figure 9a–c give model of these strokes. Control points are defined to describe geometric property of the strokes and force attraction factor is defined to describe physical property of the strokes. In Fig. 9, control points are show using circle dot and red dot means that the control point is also a feature point. Curve segment stroke include left-hand and righthand arc, etc. Figure 10a–b give the model of these strokes. The third order Bezier curve is used to describe curve segment strokes in order to enable arbitrary curve strokes modeling requirement. Mathematical model is *
P ðuÞ ¼
4 X i¼0
* 4! ð1 uÞ4i ui V i ð4 iÞ!i!
u 2 ½0; 1;
*
*
where P ðuÞ is co-ordinate of each point on the curve, V i is control vertex of Bezier curve, and dotted line means the convex bounding polygon of the Bezier curve. 3.4 Modeling of compound stroke Compound strokes are modeled using combination of several stroke elements and feature points are defined to connect all the stroke elements. Figures 11, 12 gives example of two compound strokes, respectively. Each one consist several basic strokes and is defined by sequence of feature points and control points. The first stroke is combined through two stroke elements and the second stroke is combined through five stroke elements. It can be found that feature points are some special points, which are usually locates at special places, such
Basic strokes include points, horizontal stroke, vertical stroke, etc. According to shape difference between
Fig. 9 Model of basic strokes
ð1Þ
Fig. 11 Model of compound stroke (ID=16)
124 Table 3 Data structure of character ‘‘ ’’ Data Item
Data Value
Number of typical strokes ID array of typical strokes Number of control points Number of feature points Sequence of typical strokes
5 2, 1, 10, 31, 4 15 13 2, 1, 10, 31, 4
Fig. 12 Model of compound stroke (ID=20)
4 Haptic rendering
as the start point, the end point of a stroke and connection point between two basic strokes within a typical stroke. Because the location of feature point in its parent stroke is constant, feature points can be stored using its sequence index in the control points instead of their coordinate. The co-ordinate can be loaded through search in the control points co-ordinate information. Feature point locates at the connection points between basic strokes, and the goal is to achieve two-order continuity between two strokes. Not only position of connection point but also derivative of two adjacent strokes is same. Therefore, the change of curvature and change of gradient do not have negative influence on system performance.
4.1 Architecture
3.5 Sample of character
Haptic rendering is a key element in haptic display systems, which directly decides fidelity and stability of the simulation. Strokes-sequence based rendering architecture shown in Fig. 14 is proposed to enable haptic Chinese calligraphy training. Position of the virtual pen’s tip is computed according to mapping from position of haptic device. Collision detection is carried out according to relative position between virtual pen’s tip and virtual paper, which is used to find the currently activated stroke in current character. Line segment model and Bezier curve model is adopted to ensure rapid collide detection for straight and curve stroke, respectively. Virtual force can be computed using modified virtual fixture method and predefined force attraction factor at feature points along each stroke.
Hierarchical stroke combination method is used to describe Chinese character. Model of character ‘‘ ’’ is shown in Fig. 13. Each feature point is marked by combination of ID of typical stroke with index of the feature point. Table 3 gives detailed data structure of this character.
Fig. 13 Digital model of character ‘‘ ’’
Fig. 14 Architecture of calligraphy simulation system
125
Challenge of Chinese calligraphy is detection of various interaction status and corresponding system control under different interaction between virtual pen’s tip and virtual paper and under switch point between these statuses. Furthermore, compound strokes exist in character and virtual pen will move in different area of the character and will be near to several strokes at same time at some stroke-clustered areas. Therefore, it is the responsibility of haptic rendering algorithm to maintain consistent result of collision detection within stroke clustering areas and at some corner points. All the above requirements propose challenge for system’s stability and fidelity. Based on above requirement, simulation procedure for haptic rendering is proposed as Fig. 15 shows. In this architecture, there are several main components: motion mapping from device’s position to virtual pen’s tip, collision detection, force computation model and force rendering algorithm for ensuring stability. Stroke stack based force rendering architecture is proposed to ensure consistent result of collision detection. When the haptic device is moved by human operator, position of the virtual pen’s tip is computed by motion mapping algorithm. Collision detection is carried out according to position of virtual pen’s tip and decide the active stroke of active character, thus to ensure uniqueness of collision detection result within neighboring area of several clustered strokes or at corner point of compound strokes.
Fig. 16 Signal mapping flow from human to virtual pen
4.2 Motion mapping Two spaces are defined to describe motion mapping from human’s hand to virtual pen, which are device space and virtual space. Figure 16 gives signal mapping flow from human operator to virtual pen. Motion scale KS is defined to describe velocity transformation between two*spaces. Velocity of human’s hand in device space is V h ; and velocity of haptic * interface point in virtual space is V HIP : Transformation between these two velocities is defined: *
*
V HIP ¼ KS V h :
ð2Þ
In this system, constant motion scale is adopted and * the scale is set to be 1. Furthermore, X HIP is position of haptic interface point (Avatar) and it can be computed by numerical integration: Z * * X HIP ¼ V HIP dt ð3Þ *
X p is the position of virtual pen’s tip in virtual space and it is used to be graphically displayed for writing trajec* tory. Based on collision response algorithm, X p can be decided according to dynamic response computation in free space and constraint space:
Fig. 15 Procedure for real-time simulation
Fig. 17 Location of feature plane
126
*
Xp ¼
8 < :
*
*
XHIP X HIP * 62 Rres * ; gCR X HIP X HIP 2 Rres
ð4Þ
where gCR ðÞ is collision response function that transform from position of HIP to position of VTP. Rres means constraint space, which will be defined in Sect. 4.3 according to difference strokes. In this system, following simplification is adopted: *
*
X p X HIP :
ð5Þ
4.3 Collision detection Feature points are used to maintain smooth transition between different basic strokes during handwriting of a character. The index of the feature points in current active stroke is stored in an array before simulation. The co-ordinate of each feature point is loaded through its index during the writing process. In each stroke, first feature point is always the start point of this stroke and its child basic stroke. As Fig. 17 shows, feature plane is defined at each stroke’s ending feature point, which is always perpendicular to current basic stroke. Real-time collision detection is computed according to relative position between the virtual pen’s tip and each feature plane. When a feature plane is popped through, it means that the writing on first basic stroke finished and writing on second stroke begins. As a result, the control points of the second basic stroke will be loaded into the memory and collision detection will be carried out between the pen and the second basic stroke. The above process will be repeated until the virtual pen’s tip stepped over the last feature plane and reached the last feature point, and the stack for feature point’s array will be cleared.
Fig. 18 Relative positions of HIP and linear segment
Handwriting on one compound stroke can be simulated using above process. If virtual pen’s tip retreats from virtual paper’s surface during writing state, it means operator attempts to give up writing of current stroke. Therefore, all feature points of this stroke will be loaded into the stack again. The graphical trajectory of the unfinished stroke will be cleared and the operator can re-write the stroke. 4.3.1 Collision detection against LSE When the virtual pen’s tip contact with line segment stroke, difference between operator’s real-time trajectory and referenced standard character’s trajectory is computed based on collision detection algorithm. The first step is to compute the perpendicular foot of virtual pen’s tip projected on the linear segment. Figure 18 shows the relative position of HIP and linear segment. Because each line segment stroke is limited length, valid collision detection area should be defined to avoid force feedback when writing on adjoining strokes. For every line segment stroke, constraint space is defined in * Fig. 19. where P h is possible position of virtual pen’s tip, ~ P1 is vector from start point of the stroke to the position of virtual pen’s tip, ~ P2 is vector from end point of the stroke to the position of virtual pen’s tip, ~ PEdge is vector from start point of the stroke to the end point of the stroke. Constraint space is computed according to following procedures: (a) If dot product of ~ P1 and ~ PEdge is less than zero, it means HIP is not within constraint space; (b) If dot product of ~ P1 and ~ PEdge is bigger than 0, but norm of ~ P1 is bigger than norm of ~ PEdge ; it means HIP is not within constraint space; (c) If dot product of ~ P1 and ~ PEdge is bigger than 0, and ~ norm of P1 is less than norm of ~ PEdge ; it means HIP is within constraint space; (d) If start point of the stroke is the same with the end point of the stroke, it means HIP is within constraint space.Only when the virtual pen’s tip is within the constraint space, force feedback for the stroke will be enabled. Otherwise, it is regarded as the virtual pen’s tip is within the free space.
Fig. 19 Constraint space definition
127
Fig. 20 Definition of control points
4.3.2 Collision detection against CSE Spherical neighboring area around the start point of each stroke is set as valid area of collision detection. Nearest point on the Bezier curve from the virtual pen’s tip is searched by the collision detection. Force feedback is computed based on distance between the solution point and the virtual pen’s tip. Based on subdivision algorithm of Bezier curve shown in Figs. 20 and 21, middle point of Bezier curve is computed by first division and two child segments can be produced. Middle point of each child segment can be got and distances from HIP to the three points on each child segment are compared. The child segment that owns smallest distance will be selected as new parent segment. Above subdivision is repeated until the subtraction of the distances from HIP to middle point of each child segment is less than e. Collision detection is finished and middle point of the segment that connects final two middle points is set as the solution point. 4.3.3 Collision detection against compound stroke For compound strokes, collision detection needs to consider relative position between virtual pen’s tip and strokes with various shapes. Furthermore, collision detection ambiguity needs to be avoided within strokeclustered area and at switch point between two adjoining strokes while maintaining high update rate requirement. Compound stroke is described as combination of line and curve segment stroke and feature point is used to connect basic strokes. During real-time writing, relative position between virtual pen’s tip and feature point need to be computed. Figures 22 and 23 gives collision detection against feature plane on LSE and CSE,
Fig. 21 Subdivision algorithm of Bezier curve
Fig. 22 Feature plane detection against line segment stroke element (LSE)
respectively. Feature plane is defined to decide the interaction result and mathematical model is: ( * * /i > 0V t V f > 0 ; ð6Þ * * /i\0V t V f 0 *
where V t is tangential vector of the stroke at current * feature point, V f is vector from current feature point to virtual pen’s tip. /i means whether current feature point has been written or not and /i>0 means current feature point has been written already.
5 Force model In order to achieve realistic force simulation, force model during handwriting process need to be analyzed. All the active and resistance forces on a pen are given in Fig. 24, where human’s force are divided into tangential * * and normal components, i. e. F t and * F n : Normal * resistance force F nr and frictional forces f between pen and paper is passive force. In our system, new virtual force within the paper’s surface is added to constraint operator from deviation *from the reference character template. Furthermore, f is ignored to simplify force-
Fig. 23 Feature plane detection against curve segment stroke element (CSE)
128
Fig. 24 Active and resistance forces on a pen
rendering algorithm. Therefore, feedback force in simulation system can be defined: *
*
*
F e ¼ F en þ F et ; *
Fig. 26 Minimum distances at stroke’s corner
ð7Þ *
where F e is total*feedback force, F et is force within the paper’s surface, F en is force *perpendicular to the paper’s * surface and F en equals to F nr : Feedback forces are computed based on result of real-time collision detection and typical virtual fixture force model is adopted [7]. However, modification has been made to improve the linear spring force model. For the constraint force within paper’s surface, different force attraction factors can be set at different feature point to reflect characteristic of each stroke as follows: ( * * * * Ket ðX p X S Þ X p 2 Rres F et ¼ ; ð8Þ * 0 X p 62 Rres where K*et is force attraction factor at current feature * point, X p is position of virtual pen’s tip, X S is perpendicular projection of virtual pen’s tip on current
Fig. 25 Minimum distances on big curvature strokes or stroke’s corner
active stroke, Rres means constraint space, which is defined in Sect. 4.3 according to difference strokes. Force perpendicular to the paper’s surface can be computed: * Ken ðzP z0 Þ zP z0 F en ¼ ; ð9Þ 0 zP\z0 where Ken is stiffness of the virtual paper’s plane, zp is position of device’s avatar along normal of the paper’s plane, z0 is position of virtual paper along Z-axis.
6 Stability issues According to spring based force model, instability will arise because of two possible cases, initial distance from pen’s tip penetrating neighboring circle of stroke is too big, and big stiffness in the force model. Furthermore, unsuitable character model will lead to ambiguity of virtual force during collision detection process. Figures 25, 26, 27, 28 gives several possible
Fig. 27 Minimum distances at parallel strokes
129
Fig. 28 Minimum distances at crossed strokes
cases of force ambiguity. In Figs. 25 and 26, multiple minimum distances will arise when virtual pen’s tip moves to neighboring area of big curvature strokes or stroke’s corner. Furthermore, in Figs. 27 and 28, multiple minimum distances will also arise when virtual pen’s tip moves to neighboring area of two closely located strokes. Multiple minimum distances will lead to multiple force result, therefore lead to instability. Finally, transition among different interaction status will also lead to instability of haptic device. Such transition includes: pen’s tip contact or separate from the paper’s plane, and transition from writing along one stroke to next stroke, transition from writing along one character to next character, etc. For the above three kinds of instability, control methods are proposed, respectively. For the first kind of force magnitude step change, constraint space of each stroke is proposed to maintain the step change of force magnitude within allowed limit. For the second kind of instability, i.e., the force ambiguity problem, strokes element and feature point are proposed to determine active stroke during collision detection between pen’s tip and current character. Active stroke element is determined through collision detection to avoid force ambiguity. For the third kind of instability, i.e., status transition problem, active character and active stroke are proposed to ensure smooth force signal between different statuses. All other characters will be masked when active character is been written with. Furthermore, the written strokes cannot be written twice to ensure force result consistency.
Experiment platform is constructed using Phantom desktopÒ, and Chinese calligraphy simulation software is developed using the configuration in Table 4. Figure 29 gives the scenario of handwriting operation. Wrist of the operator is required to be fixed on the desk and pen-grasp is required during writing process. Experiments are designed based on two modes: vision feedback only, vision feedback with haptic feedback. Five native (Skilled and unskilled) students and one foreign student are invited to write using the system. Before formal trial and evaluation, participants are allowed to write a specified character for a few times to get familiar with the system. When formal trials begin, new characters different from the learning character are loaded into the system. Participants are required to write each character for several times and average score will be computed. 7.2 Result As Fig. 30 shows, specified Chinese character group—‘‘ ’’ is selected to validate system performance. Trajectory of reference template is displayed in blue color. Current active character is displayed in yellow color and actual trajectory result of the virtual pen is displayed in light green color. Graphical display of the writing result is shown in Fig. 31, where error of actual trajectory and referenced trajectory can Table 4 Hardware and software configuration CPU
Intel P4 2.4G
Memory Graphical Card Operating system Language Packages
Kinmax DDR 512M Geforce3 Ti4200 Windows 2000 Visual C++ 6.0 GHOST SDK, OpenGL
7 Experiment 7.1 Configuration There are two goals for the experiment. The first goal is to validate stability and fidelity of simulation architecture, task planning method, collision detection and haptic rendering algorithm under various strokes. The second goal is to compare the handwriting effect with or without haptic feedback. Effect on human’s motor skill will not be studied in this paper, which will be carried out on large number of participants in our future work.
Fig. 29 Scenario of handwriting operation
130
Fig. 32 Force signal of stroke ‘‘ ’’
Fig. 30 Specified Chinese character group
be seen clearly. In addition, a red ellipse is drawn at beginning control point of current active stroke to give a hint for the operator. Force signals within the virtual paper plane during writing on basic stroke element are recorded to analyze stability of the system. The plane of virtual paper is defined to be XOY plane. As Fig. 31 shows, there are line and curve shaped stroke element within character ‘‘ ’’. Figures 32 and 33 gives force signal of stroke ‘‘ ’’ and ‘‘ ’’, respectively. Collision detection is proved to be rapid enough to deal with interaction status computation within 1 ms. Effectiveness of modified virtual fixture force model to train handwriting is also validated. Force signals within the virtual paper plane during writing on compound stroke are recorded to test the function of feature point in haptic rendering algorithm. As Fig. 31 shows, there is one compound stroke ‘‘ ’’ within character ‘‘ ’’, where four feature points exist in this stroke. Figure 34 gives force signal of writing process on the compound stroke. Ambiguity of collision detection could be avoided by using active stroke method. Stability can be ensured when feature point is passed.
Fig. 31 Error of actual trajectory and referenced trajectory
Fig. 33 Force signal of stroke ‘‘ ’’
Figure 35 gives writing result on a single character ‘‘ ’’. There are five strokes in the character, which will lead to five times status transition between pen and the virtual paper. No matter the pen contact or separate from the virtual paper, or the pen transit from a stroke to next stroke, the haptic device always works smoothly. Figure 36 is detailed view of Fig. 35 to display force within paper’s plane clearly. Stability is ensured at interaction status switch time during writing process. Writing experiments on multiple characters are also been carried out and stability can be maintained when avatar of haptic device change among characters.
Fig. 34 Force signal of compound stroke
131
Fig. 37 Result of writing without haptic feedback
Fig. 35 Writing result on a single characters ‘‘ ’’ Fig. 38 Result of writing with haptic feedback
Figure 37 give results of writing without haptic feedback and Fig. 38 give result of writing with haptic feedback. Table 5 gives the time difference of four characters between two modes. The unit of the time is second. It can be seen clearly that the trajectory error is reduced with help of force feedback. The same results can be observed on other Chinese characters.
Experiment results preliminarily indicate validity of the proposed task planning, collision detection and haptic rendering method. Writing on typical character illustrates validity of the model to reflect force display of calligraphy of character shape and strokes sequence while maintaining small storage memory and high computation rate. Experiment on typical character illustrates validity of the rendering algorithm to achieve expected collision
detection result within neighboring domain of strokes clusters and corner points of complex strokes, and stability can be maintained even when frequent switch occur between the virtual pen’s tip and the virtual paper, or between the virtual pen’s tip and the strokes. Advantage of haptic feedback is apparent according to experiment results. Written time can be reduced for each character with help of haptic feedback. The time difference is especially apparent for foreign student. From the experiment, some limitations are also found. Friction force between virtual pen and virtual paper should be added to prevent slippery sensation. Furthermore, haptic rendering algorithm cannot cope with larger deviations, which might be generated by a less skilled novice. Under this case, magnitude of guided force within the virtual paper’s plane need be enlarged, which is now limited by stiffness threshold of Phantom desktop. It is necessary to discuss new algorithm to enlarge force magnitude while maintaining stability.
Table 5 Time difference between two modes
8 Conclusion
7.3 Discussion
Test mode
C1
C2
C3
C4
No force feedback With force feedback
50.8 37.5
30.3 26.6
31.0 27.3
32.2 26.4
Fig. 36 Detailed view of Fig. 35
A prototype of haptic training system for Chinese calligraphy is developed, which can be used as a platform for studying human haptic skill representation and skill transfer. Chinese characters are modeled as a sequential array of typical strokes consisting of compound and basic strokes elements. In the process of task planning, the haptic skill is decomposed as perceptible elements at the first level and further transformed into control signals of the haptic device in the following two levels. In the haptic rendering, collision detection algorithm based on nearest point is implemented successfully for various strokes and the modified virtual fixture method is introduced to obtain high stability. This combines the strategy of activating the stroke under writing makes a stable haptic simulation. Stability of the rendering algorithm operating on different characters and strokes is validated by experi-
132
ment. Writing simulation with/without force feedback has been compared and the results indicate using haptic feedback in Chinese calligraphy training system is helpful to reduce writing error and to improve writing speed. Our future work will investigate how to evaluate training effect of the system quantitatively. Large number of human subjects will be invited to use the system and their writing information will be recorded and analyzed using statistical methods. Acknowledgements This work is supported by the Science Foundation of China Post-doctoral Research and the National Science Foundation of China under the grant No. 50275003. Their support is greatly appreciated. Authors would also like to thank Professor Hong Z. Tan from Purdue Univ. for her proof reading and suggestions during revision phase of the paper.
References 1. Solis J, Avizzano CA, Bergamasco M (2002) Teaching to write Japanese characters using a haptic interface. In: Proceedings of the 10th international symposium on Haptic interfaces for virtual environment and teleoperator systems, HAPTICS, pp 255–262 2. Henmi K, Yoshikawa T (1998) Virtual lesson and its application to virtual calligraphy system. In: Proceedings of the 1998 IEEE international conference on robotics and automation. 2:1275–1280
3. Teo CL, Burdet E, Lim HP (2002) A robotic teacher of chinese handwriting. In: Proceedings of 10th international symposium on Haptic interfaces for virtual environment and teleoperator systems, HAPTICS, pp 335–341 4. Yang H-M, Lu J-J, Lee H-J (2001) A bezier curve-based approach to shape description for chinese calligraphy characters. In: Proceedings of the 6th international conference on document analysis and recognition, pp 276–280 5. Chu NS-H, Tai C-L (2002) An efficient brush model for physically-based 3D painting. In: Proceedings of the 10th Pacific conference on computer graphics and applications, pp 413–421 6. Yeh J-S, Lien T-Y, Ouhyoung M (2002) On the effects of haptic display in brush and ink simulation for Chinese painting and calligraphy. In: Proceedings of the 10th Pacific conference on computer graphics and applications, pp 439–441 7. SagaS, Kawakami N, Tachi S (2005) Haptic teaching using opposite force presentation. In: Proceeding of the 1st World Haptics conference, Pisa, Italy, pp 18–20 8. Hennion B, Gentaz E, Gouagout P, Bara F (2005) Telemaque, a new visuo-haptic interface for remediation of dysgraphic children. In: Proceedings of the 1st World Haptics conference, Pisa, Italy, pp 18–20 9. Sakuma M et al (1999) System for Japanese calligraphy lesson with force feedback. In: Proceedings of the 8th annual symposium on Haptic interfaces for virtual environment and teleoperator systems 10. Srinivasan MA, Basdogan C (1997) Haptics in virtual environments: taxonomy, research status, and challenges. Comput Graph 21(4):393–404 11. Adams RJ, Hannaford B (1999) ‘‘Stable Haptic interaction with virtual environments’’. IEEE Trans Rob Autom 15:3
