Proceedings of the 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems October 9 - 15, 2006, Beijing, China
Evaluation of 3D Pseudo-Haptic Rendering using Vision for Cell Micromanipulation Mehdi Ammi, Hamid Ladjal and Antoine Ferreira Laboratoire Vision et Robotique ENSI de Bourges - Universit´e d’Orl´eans 10 bd Lahitolle, 18020 BOURGES Cedex, France Email:
[email protected]
Abstract— This paper presents a new three-dimensional biomicromanipulation system for biological object such as embryo, cell or oocyte. As the cell is very small, kept in the liquid, and observed through a microscope, the two-dimensional visual feedback makes difficult accurate manipulation in the 3-D world. To improve the manipulation work, we proposed an augmented human-machine interface. A 3-D visual information is provided to the operator through a 3-D reconstruction method using visionbased tracking deformations of the cell embryo. In order to stable injection tasks, the operator needs force feedback and haptic assistance during penetrating the cell envelop, the chorion. The proposed human-machine user’s interface allows real-time realistic visual and haptic control strategies for constrained motion in image coordinates. Virtual haptic rendering allows to constrain the path of insertion and removal in the 3-D scene or to avoid cell destruction by controlling adequately position, velocity and force parameters.
I. I NTRODUCTION Living cells are increasingly used in drug discovery, functional genomics, toxicology and many other fields. In fact, biological micromanipulations are currently performed manually and efficiency is relatively low. Various cell injection systems have been developed to provide more controllable manipulation of biological cells. Prior published works focus on using teleoperated micromanipulators in combination with vision methods to improve guidance of the injection tasks [1][2][3]. In order to realize successful autonomous injection, image based segmentation and visual tracking of cells have been attempted by several groups [4][5][6]. If segmentation of the egg and the holding pipette is relatively easy, the injecting needle is completely in focus prior to the micro-injection or completely occluded by the biomembrane during insertion. It increases considerably the segmentation procedure leading to unstable visual tracking of the automatic micro-injection task. Furthermore, when manipulating deformable biological objects, it is often useful to have knowledge of the force applied to the object via a force sensor in order to prevent damage. However, force sensor will provide only the local forces at the pipette puncture point which limits strongly the operator haptic rendering. In order to improve the haptic rendering of the cell micromanipulation system, vision feedback could be used to estimate the force applied to the biological object through observation of object deformations. This technique is referred to visionbased force estimation [7]. If an accurate model defines the
1-4244-0259-X/06/$20.00 ©2006 IEEE
elastic behavior of the cell, the interaction force field can be measured using computer vision. Various models have been implemented to demonstrate the concept using finite element method (FEM) [8], boundary element method (BEM) [9] or neural networks (NN) [10]. These approaches are mainly based on a 2D physics-based modeling which limits their use in haptic applications. The proposed user’s interface provides 3D visual and haptic feedback in order to guide and to assist the teleoperator during cell injection. Using the visual tracking information, vision-based biomembrane force estimation strategy is investigated through a 3D nonlinear mass-spring-damper (MSD) model. The proposed model takes into account the real time calculation constraint of the various nonlinear parameters necessary for visual and haptic feedback. The fusion of these different sensing modalities in an intuitive user interface allows realistic cell injection tasks to be performed. The paper is summarized as follows. Section 2 presents the micro-world data extraction and visual tracking deformation for 3D reconstruction of the cell. Section 3 presents the nonlinear MSD model of the cell for 3D haptic rendering. Finally, Section 4 presents the results of an experimental evaluation carried out on several non-trained candidates. II. S YSTEM DESCRIPTION The experimental biological micromanipulation system is shown in Fig.(1). The microrobotic cell manipulation system is composed of a cell holding unit, a vacuum unit and a Nikon T E −300 inverted microscope. The holding pipette was mounted on a oil hydraulic micromanipulator, and the injection pipette was mounted on the Eppendorf 5171 piezoelectric micromanipulator. The viewing camera is a Coolpix 990 from Nikon. The imaging unit includes an inverted microscope, a charge-coupled device (CCD) camera, a peripheral component interconnect (PCI) frame grabber, and computer image. Optical microscope images through the CCD camera is used to reconstruct the micro level 3D images. The reconstruction results are rendered to the user as virtual reality scene through a virtual graphic engine. A computational model of interaction between the injection pipette and the biological cell is used to provide displacement information to the mathematical model for computing interaction forces. Haptic feedback is provided to the user (PHANToM haptic interface) based on vision information and the material properties of the biological cell.
2115
3D reconstruction Pseudo-Haptic Rendering
Cell Micromanipulation
Image acquisition
Injection Pipette manipulator
Cell Handling Unit
Geometric 3D Reconstruction
Image recognition and tracking
Visual Rendering
Overlaying real data and physical based modeling
Position
Inverted microscope
Teleoperation
Position
CCD camera
Haptic Rendering
Rendered force Manipulator controller
Virtual guiding system
Virtual Coupling
Applied force Gesture planning
Haptic interface
Fig. 1.
Microrobotic system with vision and haptic feedback for cell micromanipulation.
In order to inject the cell with minimal damage, the microinjector must be guided and withdrawn as quickly as possible. To facilitate this, a virtual guiding system based on virtual fixtures to constrain the path of insertion and removal is integrated in the bilateral teleoperation system. The setup of the visual rendering system consists of four parts: (i) vision (optical microscope and CCD camera), (ii) image grabbing and processing board, (iii) image recognition and tracking and (iv) 3D microscopic image reconstruction. In this study, the total magnification of the microscope is fixed at 10. At this magnification, the valid area of grabbed image is about 600 × 400 microns with a resolution of 0.825 micron by pixel. The position information is then transferred to the main 3-D virtual engine module to realize 3D image reconstruction of the biological environment. So, the speed of vision system depends strongly on the speed of localization and recognition procedure. The living cell being a deformable body, it is necessary to represent in real time the deformation of both biomembranes along the penetration phase. In order to reconstruct the biomembrane geometry deformations in a three-dimensional virtual space, a robust deformable visual tracking algorithm (namely, snakes) has been developed in a previous work [12]. On the basis of the control points returned by the snakes, we generate a 3D mesh corresponding to the external and internal biomembranes. This meshing is obtained by applying a rotation operator to the membrane’s contours (snakes’s control nodes) around the symmetry axis (Fig.2(a) and Fig.2(b)). Since ideally the living cell is in spherical shape, so it is intuitive to represent the cell mesh as a pseudospherical shape also. III. 3D P SEUDO -H APTIC R ENDERING In order to provide force feedback to the user during the microinjection task, we have chosen vision to estimate the interaction forces rather than using a force sensor integrated on the micromanipulator. The main reason is due to the fact that a force sensor will provide only the local forces at a point rather
(a) Meshing for a rotation of 30 (b) Meshing for a rotation of 60 degree. degree. Fig. 2.
Pseudo-3D visual representation of biological cell.
than force feedback for an area of interest. Another advantage of using vision is the non-invasiveness of the displacement estimation method. Finally, the majority of the existing biological micromanipulation systems are not equipped with force sensor due to cost and integration problems. Towards this end, we have developed a mechanical model of interaction with the biomembrane using nonlinear mass-spring-damper (MSD) elements whereby vision is used to provide displacement information to the MSD model for computing the interaction forces. A. Vision-based force measurement approach The basic idea is to compel a mechanical model using nonlinear mass-spring-damper (MSD) components with respect to the cell’s volume measured from visual feedback. The mechanical model assumptions are the same as those proposed in [14]: (i) the biomembrane encapsulates the liquid that exerts a uniform hydrostatic pressure on the biomembrane, (ii) the cell volume does not change, (iii) the biomembrane is linearly elastic and (iv) the cell is free of initial membrane stress or residual stress. Figures (3(a)) and (3(b)) represents the 3D-reconstructed cell volume encapsulating the biological liquid. The model starts with a planar circular area where the volume of the mechanical model corresponds to the cell’s volume at rest
2116
(a) Cell volume at rest
(c) MSD model at rest
(b) Cell volume during punction
(d) MSD model during punction
Fig. 3. Cell’s volume and mechanical system evolution during the microinjection operation.
spring and damper system. We choose an elementary structure that presents a set of X-shaped connectors (Fig.4(a)). This configuration reduces greatly the abrupt jumps of the nodes in the vicinity of the contact point. Several elementary structures are assembled according to the spherical volume (cell’s initial volume) of the biological cell. We assigned the same mass for all the nodes and the connectors are also characterized by the same stiffness and damping parameters. A typical model of a 3D cell object is shown in (Fig.4(b)). The MSD system is based on an iterative updating method. The procedure consists to calculate for each iteration and for all nodes the resultant interaction force. 1) Force Calculation: Eq.(1) indicates the force Fi that acts on the node i with a mass mi . kij and bij are respectively the stiffness and damping coefficients ; Lij is the connection’s length at rest. Finally, pij and vij are respectively the relative position and the relative velocity of the node i with respect to the node j. The dynamic behaviors of the modelled cell including their deformations and motions are determined by the motions of these particles, which is effectively dictated by the Newtonian law of mechanics. Fi =
X½ i
(a)
(b)
Fig. 4. Geometrical configuration of the MSD mechanical cell model: (a) X-shaped spring-damper connector and (b) model of deformable spherical cell.
(Fig.3(c)). The mechanical system being at equilibrium, no force feedback is returned to the operator. During the injection task, the cell’s volume is updated according to the control points provided by the membrane’s contours (snakes’s control nodes). The external surface of the mechanical system is forced according to this new volume distribution (Fig.3(d)). After updating, it is possible to obtain the interaction force field on all nodes of the mechanical system and more particularly at the puncture point between the injection pipette and the cell. The numerical force is then felt by the operator through the haptic interface as an interaction force.
µ kij
¶
Lij −1 |pij |
− bij
vij · pij pij · pij
¾ pij (1)
where the stiffness coefficients kij are defined by a second order equation such as kij = k12ij + k2ij . The constant parameters k1ij and k2ij are identified numerically following the node location ij. 2) Numerical solution of dynamic equations: We can rewrite Eq.(1) in order to express the position of the nodes and velocity derivative in the following form : d2 − d − − → → → → − X (t) + B X (t) + K X (t) = fa (2) dt2 dt where M is the mass matrix, B is the damping matrix and − → → → d − d2 − K is the stiffness matrix ; X (t), dt X (t) , dt 2 X (t) are the displacement, velocity and acceleration vectors of the particles − → at the node points; fa is the external force vector applied to each node. The solution of the system of Eq.(2) is found by numerical integration using the Implicit Euler method. In the current application, linear acceleration method is used due to its simplicity, good accuracy and efficiency such as: Z t+h q(t + h) = q(t) + q(q, ˙ t) dt (3) M
t
B. MSD Mechanical Model Among the several existing mechanical models of cells (finite element method (FEM) [8], boundary element method (BEM) [9] or neural networks (NN) [10], etc.), we choose the nonlinear mass-spring-damper model (MSD). The motivations of this choice are mainly due to the modularity and the flexibility of MSD model when considering different cell geometries, mechanical interactions at the pipette/cell interface and real time interaction. The cell under consideration is modelled as a set of primitives particles of masses interconnected via
where h is the time step. The second order differential equation of the node states is then implemented in two firstorder equations as follows : + ∆t) vi (t + ∆t) = vi (t) + Fi (tm ∆t i (4) xi (t + ∆t) = xi (t) + vi (t + ∆t) ∆t The time step ∆t in the simulation should be small enough to achieve reasonable precision in the force prediction. The
2117
TABLE I M ECHANICAL CHARACTERISTICS OF THE ZP MEMBRANE OF THE OOCYTE IN CASE OF MODEL PROPOSED BY [14]. Membrane type
Maximum deformation
Punction force
Young modulus
Poisson ratio
Oocyte
44µm
7.5µN
17.9kP a
0.5
Embryo
53µm
13µN
42.2kP a
0.5
larger the step size, the more inaccurate and unstable the simulation tends to be. The force errors decrease almost linearly as the step time size is reduced. Beyond certain extent, however, this trend saturates and the effect of further reduction in the force error becomes insignificant.
(a) Tested configurations
C. Identification of Model Parameters
(b) Finite element model (FEM) (c) Mass-spring-damper (MSD)
model
(d) Comparison betwwen FEM and MSD models for parameter identification. Fig. 5.
Geometrical configuration for MSD parameter identification.
Rendered pseudo-force Force (µN)
Actually, the relationship between geometrical and mechanical properties of the cell still remained largely unknown. Studies of mechanical responses from the cell model with a series of applied stresses/forces had been performed recently by [14] on mouse oocyte ZP (see Tab.(I)). Based on these mechanical properties of the cell, mechanical responses from finite element model (FEM) has been carried out in order to identify the parameters of proposed MSD cell model. As an approximation, we extended the planar model parameters of Tab.(I) in order to estimate the volumic model parameters. The identification procedure of the parameters k, b and m consists to adjust by simulation the mechanical response of the MSD model to the mechanical response of the FEM model. Several pipette injection tasks according to different penetration axis have been carried out in order to validate the model.The tested configurations are described in the following (Fig.5(a)) : 1) Configuration 1 : The penetration axis is aligned with the cell’s geometrical center. It presents a zero angle with the holding pipette. 2) Configuration 2 : The penetration axis is aligned with the cell’s geometrical center. It presents an angle of 45degree with the holding pipette. 3) Configuration 3 : The penetration axis is aligned with the cell’s geometrical center. It presents an angle of 90degree with the holding pipette. 4) Configuration 4 : The penetration axis is slightly shifted according to the cell’s geometrical center. It presents a zero angle with the holding pipette. As illustration, Fig.5(b) and Fig.5(c) shows a snapshot of the injection task when the penetration axis is aligned with the cell’s center (case1). The used identification method allows us to simulate the mechanical behavior of the cell in a relatively precise way. The registered errors compared to the finite elements model are lower than 5% (Fig.5(d)). As example, Fig.(6) shows a comparison study between the experimental force-deformation data performed on a mouse oocyte and the rendered force provided to the user through the MSD model. We can see clearly the influence of the parameter estimation error on the rendered force (for a model identification error of 10% and 3%, respectively).
Error=3%
Error=10%
Experimental force
Displacement (µm)
Fig. 6. Comparison between experiments and rendered force: Forcedeformation curves of mouse oocyte.
2118
IV. E XPERIMENTAL R ESULTS In order to evaluate and to validate the proposed approach we carried out a series of experiments. This section presents a study carried out on 13 persons coming from different fields (experts, students, technicians). In these experiments, we were only interested by micro-injection phases during cell penetration. The operator introduces the injector in the nucleus (from the contact point defined by the pipette and the cell) until to reach the predefined injection point inside the cell. Fig.(7(a)) and Fig.(7(b)) show the master’s arm velocity evolution along the penetration axis without and with the haptic rendering. We noticed that when the operator has no haptic feedback the movement is abrupt and is characterized by high accelerations at the beginning and at the end of the penetration task. When we provided to the operator a haptic feedback, the gesture of the operator is dampened along its motion leading to lower velocity values. The same observation is valid for the deceleration phase where the operator decreases progressively his velocity while approaching the cell punction point. In this case the operator takes into account the increasing counteracting effort exerted by the cell during the penetration phase, as shown in Fig.(7(c)).
(a) Velocity without pseudo-haptic (b) Velocity with pseudo-haptic renrendering. dering.
(c) Cell injection forces with pseudohaptic rendering. Fig. 7. Master arm’s velocity and injection force evolution according to the penetration axis.
Fig.(8(a)) shows the improvement of the execution time when considering or not haptic rendering. The appreciations given by the operators are shown in figure Fig.(8(b)). The results indicate that the operators mainly prefer micromanipulation with a pseudo-haptic feedback. In order to evaluate the level of pseudo-haptic interaction during the micro-injection task, we tested different visual and/or haptic modalities. The results given in Fig.9 shows different trajectories experienced by a teleoperator. Fig.9(a) gives the trajectory of the injector when considering only the 2D vision-based tracking deformations provided by the active
(a) Execution time.
(b) Operators’s appreciations Fig. 8.
Execution time and appreciations for the pseudo-haptic feedback.
countour algorithm. As we can see the trajectory is not optimum and the completion time is important (mean completion time: 9sec). When considering 3D pseudo-visual rendering (Fig.9(b)), the operator improves its trajectory inside the cell but the success rate is quite low (30 percent) even by skillfull operator after a long training period. When considering the 3D pseudo-visual and -haptic rendering (3D vision and 3D force feedback modalities), it should be noted (see Fig.9(c) that the trajectory is improved and the error of positioning is reduced. The operator is able to counteract easily the deformations of the cell during the injection process by controlling the injection forces and improves his dexterity and manipulability through the 3D virtual environment. However, the manipulation task requires much attention efforts of the operator and does not eliminate hand tremor after a long training process (mean completion time: 6sec). Figure 9(d) shows the linear trajectory of the pipette injector when considering different modalities (3D pseudo-visual and pseudo-force renderings and rectilinear virtual fixture). In order to analyse the influence of the virtual haptic guide on the pseudo-haptic feedback provided to the operator, Fig.(10(a)) and Fig.(10(b)) show the results when considering the virtual fixture without and with a pseudo-haptic feedback. When the operator does not use the pseudo-haptic feedback we noticed mainly two phases in the operator’s gesture. The first phase corresponds to the displacement of the injection pipette from the initial configuration (in contact with the cell) to the final configuration (punction configuration). The second phase of the movement corresponds to the damping gesture in the vicinity of the punction point. This step is characterized by weak velocity fluctuations. When we enhanced the virtual guide by the pseudo-haptic feedback (Fig.(10(b))), we
2119
(a) Operators’s time execution
Fig. 9. Master arm’s position evolution according to penetration axis for different visual and haptic renderings: when considering (a) 2D imaging, (b) 3D pseudo-visual rendering, (c) 3D pseudo-haptic rendering and (d) 3D pseudo-haptic rendering and rectilinear virtual fixture. (b) Operators’s appreciation
observed a better gesture control when the pipette approaches the punction configuration. The damping mouvement is widely damped leading to a better control of its final position.
Fig. 11.
Execution time and appreciation for the virtual fixture.
R EFERENCES
(a) Pseudo-haptic feedback only. Fig. 10.
(b) Pseudo-haptic feedback with virtual fixture
Master arm’s velocity evolution according to penetration axis.
Additionally, Fig.(11(a)) shows the execution time obtained with the proposed virtual fixture. We noticed the important gain brought by the use of the guide only (without pseudohaptic feedback) but there was no significant contribution when we combined both modalities. On contrary, the appreciations of the operators (Fig.(11(b))) where mainly for the fusion of both modalities showing their preference for a better quality rather than a fast execution micromanipulation task. V. C ONCLUSION This paper has reported preliminary experiments to improve the realism of visual and haptic modalities perceived by the operator during cell microinjection. Using the visual tracking information of cell deformations, vision-based biomembrane force estimation strategy is investigated through a 3D nonlinear mass-spring-damper (MSD) model. The proposed model takes into account the real time calculation constraint of the various nonlinear parameters necessary for visual and haptical feedback. Finally, an experimental investigation has shown the efficiency of the proposed 3D pseudo-haptic rendering during practical micro-injection tasks.
[1] S. Yu, B.J. Nelson, Microrobotic Cell Injection, IEEE International Conference on Robotics and Automation,Seoul, Korea, 2001, 620-625. [2] R. Kumar, A. Kapoor, R.H. Taylor, Preliminary Experiments in RobotHuman Cooperative Microinjection, IEEE International Conference on Intelligent Robots and Systems, Las Vegas, Nevada, 2003, 3186-3191. [3] X. li, G. Zong, S.Bi, Development of Global Vision System for Biological Automatic Micro-Manipulation System,IEEE International Conference on Robotics and Automation,Seoul, Korea, 2001, 127-132. [4] J. Koshwanez, M. Holl, M. McMurray, D.Gottschling, D. Meldrum, Automation of Yeast Pedigree Analysis, IEEE International Conference on Robotics and Automation,New Orleans, LA, 2004, 1475-1480. [5] X. Li, G. Zong, S. Bi, W. Zhao, Automatic micromanipulating system for biological applications with visual servo control, Journal of Micromechatronics, Vol.1, N0.4, pp.345-363, 2002. [6] M. Gauthier, E. Piat, Control of a Particular Micro-Macro Positioning System Applied to Cell Micromanipulation, IEEE Transactions on Automation Science and Enginnering, 2006. [7] M. Greminger, B. J. Nelson, Vision-based Force Measurement, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol.26, N0.3, 2004, 290-298. [8] C. W. Kennedy, J. P. Desai, Force Feedback using Vision, IEEE International Conference on Advanced Robotics, Coimbra, Portugal, 2003, 179-184. [9] M. Greminger, Y. Sun, B. J. Nelson, Boundary Element Deformable Object Tracking with Equilibrium Constraints, IEEE International Conference on Robotics and Automation, New Orleans, LA, 2004, 38963901. [10] M. Greminger, B. J. Nelson, Modeling Elastic Objects with Neural Networks for Vision-based Force Measurement, IEEE International Conference on Intelligent Robots and Systems, Las Vegas, Nevada, 2003, 1278-1283. [11] M. Kass, A. Witkin, D.Terzopoulos, Snakes: Active Contour Models, International Journal of Computer Vision, 1, pp. 321-331,1987. [12] M. Ammi, A. Ferreira, Biological Cell Injection Visual and Haptic Interface, Advanced Robotics, Vol.20, N0.3, 2006. [13] L. Piegl and W. Tiller , The NURBS book, Springer pres, Berlin, 1996. [14] Y. Sun and K. Wan and K. P. Roberts and J. C. Bischof and B. J. Nelson, Mechanical property characterization of mouse zona pellucida, IEEE Transactions On NanoBioscience, Vol. 2, No. 4, 2003.
2120
