Autonomous Micromanipulation Using a New Strategy of ... - CiteSeerX

Autonomous Micromanipulation Using a New Strategy of Accurate Release by Rolling Fabien Dionnet, D. Sinan Haliyo and Stéphane Régnier Laboratoire de Robotique de Paris (LRP) Université Pierre et Marie Curie (Paris 6) - CNRS FRE 2507 18 route du Panorama, BP61, 92265 Fontenay-aux-Roses, France Email: [dionnet, haliyo, regnier]@robot.jussieu.fr

Abstract— This paper presents our work in developing an autonomous micromanipulation system. The originality of our system is that it takes advantage of adhesion to grip microobjects by using a single fingered gripper. This is in fact a tipless cantilever previously designed for Atomic Force Microscopy applications. We describe vision techniques employed to process images provided by an optical microscope, allowing to position accurately the end-effector for a gripping task. A theoretical study of the direct force measurement device and an experimental validation show how we can improve the measurement of impact and contact forces. Then we explain the strategy used to bring the gripper into contact with the object, based on force control and kinematic redundancy. Finally, a simplified model of the release task is proposed in order to determine conditions that allow to roll the object, and then to place it with precision.

I. I NTRODUCTION Due to recent development of MEMS and biotechnology, there is a great demand for original micromanipulation techniques. Many approaches have been proposed to manipulate microscopic objects. The principal obstacle specific to this scale is that the force of gravity becomes negligible in comparison with adhesion forces. Consequently, any microsystem based on the miniaturization of conventional macroscopic robots encounters a lot of difficulties in releasing a gripped object as it adheres to the gripper. Complex techniques are thus necessary to reduce adhesion [6]. An original approach has been developed at LRP which consists in taking advantage of adhesion to manipulate objects with a single fingered gripper. [5] shows that pick-up under quasi-static conditions is possible by choosing the material of the gripper so that adhesion between object and end-effector is greater than between object and substrate. [4] describes a way to dynamically release an object. The gripper is excited to induce high object accelerations. Contact is broken if the inertial force is greater than the adhesion force. This system is not only an operational micromanipulation station, but also a measurement platform, as explained in [3]. Whatever the employed manipulation techniques (with contact or not), and the environment (in air, liquid or vacuum) [1], due to specific mechanical and physical laws which govern the microworld, micromanipulation systems often suffer from a lack of reproducibility. That is why micromanipulation tasks need complex and robust control based on sensor feedback. Section II briefly describes the experimental configuration

of the effector, the actuators and the vision and force sensors composing our micromanipulation system. Section III focus on the techniques, in particular focusing and image-based visual servoing, used for safe and accurate positioning the gripper close to the object to be gripped, so that the system has an optimal configuration for performing a quasi-static pickup. Section IV presents the strategy developed for the pickup task. The force measurement device is described in detail and results of impact and contact force control are shown. A macroscopic simplified model, describing the system for the release task by rolling, is introduced in Section V. The goal is to find out conditions on the contact force and the gripper motion allowing to roll the object. Finally, first experiments on release by rolling and potential applications for micromechanical characteristic measurement are presented. Future evolutions of our micro-manipulation system and applications are presented in conclusion. II. E XPERIMENT CONFIGURATION 1) System description: Figure 1 shows the overall experiment which takes place on air environment in a controlled clean-room. Objects are lying on a fixed horizontal specimen carrier which is placed under a camera-equipped microscope with a vertical degree of freedom, allowing to focus alternatively on the different features in the workspace. Indeed, due to the shallow depth-of-field of the microscope, features located at different horizontal planes can not be focused at the same time in a single image. The single fingered gripper is actuated by five serial linked translators, allowing a wide motion range: a Cartesian DC motor micro-stage for large displacements, a piezoelectric nano-stage for more accurate vertical motion, and a piezoelectric ceramic to produce impulses required for dynamic release. Moreover, the gripper has a device able to quantify its deformation, to detect contact and to measure adhesion and contact forces. 2) Task definition: A full-manual grip operation is decomposed in six steps: (1) focus on the planar substrate containing the objects, (2) select the object to grip, (3) focus on the endeffector, (4) position the end-effector in the horizontal plane, (5) move the end-effector down until contact and (6) move the end-effector up. These steps are highly ticklish and require a lot of practice for operator. Obviously, the touching step

Focus criterion 16

(i)

12

8

(ii) 4

0 −200

−100

0

100

(iii)

(iv)

(v)

200

300

400

Distance from the optimal position (µm)

(i)

Fig. 1.

III. A PPROACH TASK A. Vertical positioning by focusing The goal is to have a rough estimation of the vertical gap between the gripper and the object, in order to move the end-effector to the object neighborhood. Some image properties are affected by good or bad focus and can be used to build suitable criteria. Focused images have more high frequency components, more localized histograms, higher contrast, higher peaks and deeper valleys than blurred images. With a criterion based on one of these properties, the focusing problem can be processed as an optimization problem. Some robust focusing techniques exist, as in [9]. In the current case, images have two types of features: end-effector and objects. Moreover, recording conditions -mainly the lightingare steady because of working in a controlled environment. Thus an adaptive criterion is not needed. Some simple criteria have been tested, the chosen one being to quantify high frequency components of the image. For a N × M image Iz , taken by camera at position z, the criterion is written as N −2 M −2 q 1 X X 2 2 |Gx (i, j)| + |Gy (i, j)| , N M i=1 j=1

(iii)

Fig. 2.

Micromanipulation system

is not only the most critical for task achievement, but also the most perilous for the system, as the end-effector is very fragile. It requires an accurate positioning of the end-effector, and in spite of operator precautions, the task is only successful for half of the attempts. Thus, it would be useful to automate these steps both to achieve the manipulation and to not damage either gripper or object.

f (z) =

(ii)

(iv)

(v)

Focus criterion measure

visual servoing using an external camera, as described by Figure 3. In our case, the problem is simplified for two reasons. First, the robot has no rotational joints, and the plane defined by the X and Y axes of the micro-stage is almost parallel to the microscope image plane. It is then enough to process only a single point. Thus, the goal is to reduce the error between the end-effector contact point desired position (u∗ v ∗ ) and its actual position (u v) in the current image. 2) Template extraction: Before processing, a sub-image taken from a focused image, containing the end-effector is extracted. Contours are detected using Sobel masks as explained in Section III-A. The output image is the template of the gripper. The operator must specify the desired contact point along the main axis of symmetry. 3) Real-time detection: The gripper contact point has to be known in each acquired image. This is achieved by a template matching method. Assuming that the algorithm knows the position of the template in the previous image, the template will be searched for an extended area compared to the template sub-image. This area of interest is first processed like the template, and then swept and compared pixel by pixel to determinate its best matching location according to a vote process. Knowing the template position in the image, it is obvious to deduce the current position of the contact point. 4) Performance: Even if static error of this servoing is null, there is a remaining uncertainty about the absolute position of the gripper, which depends on the real area size covered by a pixel (i.e. on the zoom). In our case, this uncertainty is about

(1)

where Gx and Gy are the thresholded convolution of the initial image Iz with respectively horizontal and vertical Sobel masks. Figure 2 illustrates a focus measure performed on the extremity of the gripper.

(²u ²v )

(u∗ v ∗ )

C(s)

+ (u v)

−

B. Horizontal positioning by image-based visual servoing 1) Issue statement: The next step is to position the gripper above the desired object. This is achieved by imaged-based

(UX UY )

Fig. 3.

(x y z) M (s)

Image Processing

Image

Image based visual servoing scheme

2µm, and constant with vertical motion of the end-effector. Indeed, if the optical and motion axes are not perfectly aligned, the induced deviation is permanently corrected. Hence without the servoing, an alignment error of 1 degree and for a vertical motion of 1mm, would induce a deviation more than 17µm, which is great compared to both gripper and object size.

FL vL

Fig. 4.

IV. P ICK - UP TASK In order to pick-up the desired object using adhesion forces, the gripper has to be brought into contact with it. This step is critical because both gripper an object are fragile. On the other hand, a “high” force (at this scale) is required to ensure a firm contact. Indeed, the end-effector might bend under non contact adhesion forces (van der Waals, electrostatic, . . . ). Hence the contact force must be accurately controlled [10] [8] [2]. A. Force measurement 1) Modeling: Our end-effector is an AFM (atomic force microscopy) tipless cantilever beam. It is a built-in silicon rectangular prism of size 600 × 140 × 10µm3 . Micro strain gages are fixed at the built-in end, forming a Wheatstone bridge and measuring the local strain. According to its dimensions, the end-effector can be modeled as a bending beam. Generally speaking, a force Fl is applied to the beam at a distance l from the built-in end. Assuming Bernoulli hypothesis, we know that the deflection vl is given by Fl d 2 vl = (l − x) , for 0 ≤ x ≤ l. (2) dx2 EI where E is the Young modulus and I the moment of inertia of the beam. By integrating (2) twice and according to limit conditions stated by the built-in end, we deduce the rotation ωl and the deflection vl as µ ¶ dvl x 2 Fl ωl (x) = = lx − , (3) dx 2 EI µ 2 ¶ x 3 Fl x − . (4) vl (x) = l 2 6 EI The output voltage of the Wheatstone bridge is an instantaneous measure proportional to the rotation ωl (δ) of the first section located at distance δ from the built-in end. It is then amplified to produce a measurable voltage. Using (3) and assuming that δ ¿ L, the ultimate measured voltage U is µ ¶ Fl lδ U = AU + U0 , (5) EI where AU is the amplification gain and U0 the measure offset. 2) Calibration: Thereafter, vx (x) is noted vx where x can be either the supposed or real contact point. Calibration of the device is done by touching the end of the gripper to a fixed and rigid substrate. It is obvious that the deflection is directly the opposite of the nano-translator position measured from the initial contact position. According to specification data, theoretical study and dynamical experiment, the equivalent

Calibration configuration

stiffness of the beam is estimated as Ks = 21, 06N m−1 . Writing (3) for l = L, (4) and (5) give the deflection as ¶ µ FL 3EI FL L3 (6) = ⇒ Ks = 3 , vL = 3EI Ks L and that the measured voltage is µ ¶ 3AU δ U= vL + U 0 , L2 | {z }

(7)

KU

where coefficients KU and U0 have to be experimentally obtained. Finally, we have the deflection of the beam from the voltage measure U and the applied force FL , vL =

U − U0 , KU

FL = K s

U − U0 . KU

(8)

3) Practical conditions: In manipulation conditions, objects are gripped in l < L in order to ensure contact. In this case, (8) actually gives the equivalent theoretical applied ∗ force FL∗ , and the induced theoretical deflection vL , that would produce the same real measure U , were the contact at the extremity of the beam. Writing (5) for both ideal calibration and real cases, knowing the contact point l, and measuring a voltage U , the real applied force is ¶ µ L U − U0 L Fl = ⇒ Fl = K s . (9) FL∗ l l KU In the same way as for (9), but using (6), the real deflection at contact point l is vl ∗ = vL

¶ µ ¶2 µ ¶2 µ l U − U0 l . ⇒ vl = L L KU

(10)

4) Validation: In order to validate (10) and the theoretical results of Section IV-A.3, we did the calibration process at different contact points and measured the variation of the nanotranslator position between the initial contact position and the position required to produce a fixed voltage output. Results are collected in Table I. Obtaining the ratio l/L using vision, we are able to improve the precision obtained by (9) and (10). B. Force servoing To bring the gripper in contact with the desired object, an accurate interaction force control is needed. Vertical motion of the gripper can be done by both micro and nano-stages.

TABLE I

Contact force (µN )

Impact force (µN ) 60

V ERIFICATION OF E QUATION (10)

80 consign

Contact point ratio l/L Measured deflection vl

¡ v ¢ ± ¡ l ¢2 l

Ratio

vL

(µm)

3/4

1

0.65

1.50

2.48

1.048

L

40

1/2

1.075

consign

1.000

F =

0, if zn ≥ zn0 ¢ ¡ L ¢3 ¡ 0 . zn − zn , if zn ≤ zn0 Ks l

(11)

Owing to its limited travel range of 12µm, the nano-stage may easily reach its bounds. Hence, in order to ensure a large force range, the end-effector should be very close to the object (a few micrometers) before starting the servoing and keep as much travel range as possible. However, this is not possible due to the inaccuracy of the position information given by focusing. 2) Enhanced loop: The way to solve this problem is to take advantage of the redundancy of the manipulator on the vertical axis, associating an auxiliary loop controlling the micro-stage with the basic one, whose goal is to maintain the nano-stage in the middle of its travel range. Due to their different resolutions, it is important to define a dead zone around the desired nanostage position where the auxiliary loop has no effect in order to avoid undesirable oscillations. Figure 6 illustrates the overall servoing scheme. When a contact force F ∗ is desired, and the measured force F is null, the nano-stage reaches its bottom bound due to the basic loop, and moves out of the dead F∗

²n +

−

Cn (s)

Fig. 5. ∗ zn

+ F∗

N (s)

Um

zn

F

G(s)

zm

−

zn Un Cn (s)

− Fig. 6.

measure

0

2

4

Time (s)

Fig. 7.

6

0

0

2

4

6 Time (s)

8

10

Impact and contact force measurement

zone. Consequently, the micro-stage moves down thanks to the auxiliary loop. When the gripper reaches the object, a non null contact force is measured and the former basic loop works in linear mode. The secondary loop will stop when the nanostage is in the dead zone and activate again if a new force command makes the nano-stage go out of the dead zone. 3) Impact and contact force experiment: Figure 7 shows practical results of the enhanced force servoing in two cases. In the left plot, the end-effector has been placed above an object using vision focusing and servoing. Then 50µN contact force is required from the servoing. The observed overshoot depends on the speed limit of the micro-stage and the length of dead zone. It could be reduced by reducing the speed and increasing the dead zone, but consequently, the system would be slower and less reactive. In the right plot, the gripper and the object are in contact and several desired contact forces are required. V. R ELEASE TASK Most of the applications require the precise release of the manipulated objects. The dynamic release strategy is limited in its ability for accurate positioning. The alternative solution proposed in this article is to release an object by rolling it between the gripper and the substrate. A. Macroscopic approach modeling 1) Initial state: Micro-physical phenomenons are highly complex [7] [10]. Let’s consider a simplified macroscopic approach of the problem, i.e. without taking in account complex effects of microscopic adhesion forces (local deformation, delayed effects, . . . ). The spherical object is placed between two horizontal planes formed by the gripper and the substrate. At both interfaces, each couple of bodies are mutually attracted. adh adh for the top interface and Fos These adhesion forces are Fog for the bottom one. In order to be able to perform static gripping, substrate material has been chosen such that

M (s)

Cm (s)

²n +

Un

Basic force servoing loop scheme

²m

40

0

1) Basic loop: A basic force servoing loop has been implemented to control the nano-translator stage, thus the gripper is able to apply a precise contact force. Figure 5 shows the force control scheme. F ∗ and F are desired and measured force, ²n is the servoing error, Un the input voltage, zn the nano-translator position, N (s) the transfer function of the low level controlled nanometric stage, G(s) the transfer function of the beam and Cn (s) the corrector term. Assuming that the deflection of the beam vl is the opposite of the nano-stage position zn measured from its initial contact position zn0 , and using (9) and (10), we deduce that ½

measure

20

+ N (s)

zn

+

zb

F G(s)

Enhanced force servoing loop scheme

adh adh kFog k > kFos k.

(12) norm

Moreover, the end-effector applies a contact force F to the object, which is transmitted to the substrate, so that total normal forces at both interfaces are norm adh Fog = F norm + Fog ,

norm adh Fos = F norm + Fos . (13)

TABLE II F

F norm

ext

G ENERAL FRAMEWORK OF MODE OBSERVATION

F ogadh

Mode

(a)

(b)

(c)

Gripper vertical speed

+

−

−

Normal applied force Fig. 8.

w w w w norm w ? wF ext w > µog wFog

gripper slides on object

in dynamical release have too short a travel range to allow to release the object. Hence, it is not possible to experiment mode (a). To observe mode (b), we must have, according to the equations in Figure 9, w w w norm w norm w. k < wF ext w < µog wFog µos kFos

Mode (b): object adheres to gripper and slides on substrate

w norm w norm w, kFos k < wFog

object rolls

(15)

which is always true according to (12). However the greater the normal applied force F norm is, the closer both bounds of (14) are, and even if F norm is null, the allowed range is narrow, so that, we should mainly observe the mode (c). 3) Mode detection: Although a lot of approximations are made to foresee the behavior of the system, summarized in Table II, the analysis of such an experiment could be interesting for estimating micro-mechanical properties of the manipulated object. This requires to know the motion of the object during the release task. Let’s rewrite (10), omitting the voltage offset, considering that U is null at no-load and assuming that the deflection at contact point vl is directly the opposite of the nano-stage position zn , measured from the initial contact position zn0 . Let’s introduce the variable z = zn0 − zn . The contact point is then given by U (t) = KU

no

Mode (c):

(14)

Here again, coefficients µos and µog are badly known. If we make the hypothesis that µos ' µog , the condition (14), corresponding to the mode (b), is possible only if

no

w w norm k ? wF ext w > µos kFos

yes Mode (a):

+

Simplified approach

Friction coefficients for both interfaces are defined as µog and µos . Figure 8 illustrates the release configuration and the forces applied to the end-effector. 2) Expected system behavior: The goal is to roll the object by moving the gripper in the horizontal plane. Considering the system at the initial state previously described, if the endeffector moves along the X axis under a horizontal force F ext , a tangential force is transmitted to the object. According to the roll/slide condition at both interfaces summarized in Figure 9, we can differentiate between three expected modes: (a) the gripper slides on the object, (b) the object adheres to the gripper and slides on the substrate, and (c) the object rolls between the end-effector and the substrate. Two decoupled control parameters are available to perform rolling: the tangential force F ext and the normal applied force F norm . It is difficult to quantify F ext , but we know that it depends on the initial acceleration of the gripper, thus on its desired horizontal speed. Consequently, mode (a) is theoretically possible only with very high speed motion. This issue is the vertical equivalent to the dynamical release problem described in [5]. The required high accelerations can not be produced by the conventional micro-stage used to actuate the end-effector, and piezo-ceramics such as employed

yes

−

µ

L l(t)

¶2

z(t) ⇒ l(t) = L

s

KU

z(t) . U (t)

(16)

Comparing the variation of l with the gripper vertical motion, we are able to detect the current mode, (a), (b) or (c), as in Figure 9. B. Results

Fig. 9.

Conditions of mode observation

Experiment show that it is easy to roll the object, as expected. The observed rolling is not perfect, but it is efficient to release objects. Moreover, knowing by vision the initial contact point on the beam and the size of the object, it is possible to estimate the distance required to release, and thus to release the object accurately. Figure 10 gives experimental results. The gripper and the object are in contact and the basic force servoing loop is running. The gripper starts to move along its X axis at time t1 . Due to the servoing loop, the measured voltage is constant. According to (16), the fact that the nano-stage moves down proves that the contact point l moves to the extremity of the beam, i.e. the object rolls.

Nano-stage position zn (µm)

Measured voltage U (V )

6 5

0.5

4

0.4

(i)

(ii)

(iii) (i) focus on object

3

(ii) focus on gripper

0.3

t1

(iii)position the gripper

t2 2

0.2

1

0.1

0

0.0

(iv)

(v)

Fig. 11. t3 0

2

4

6

8

10

12

14

(iv) touch object (v) release object

Top view of a micromanipulation task (ii)

(i)

(iii)

16

Time (s) l Estimated contact point ratio L 1.0

Fig. 12. 0.9

Side view of a release task

t1

0.8

master haptic device and bilateral control scheme are being included in this system, which is evolving toward a telemicromanipulation platform.

t2

0.7

R EFERENCES 0

Fig. 10.

2

4

6

8 Time (s)

10

12

14

16

Rolling measures and contact point estimation

Between t2 and t3 , the end-effector follows the curve of the object and after t3 , there is no more contact. An estimation of the ratio l/L, using (16) is given in the bottom plot. Figure 11 shows an autonomous manipulation task of a glass micro-ball with a 50µm-diameter. The rolling step is detailed by the side view in Figure 12. VI. C ONCLUSION In this paper, we have described the micromanipulation system developed at LRP. Strategies of pick-up using adhesion forces and release by rolling task have been proposed. This system uses vision and force feedback techniques that allow to achieve accurate and safe autonomous micromanipulation. The distance between both the gripper and object plane is estimated from focusing informations. The gripper is accurately placed in the horizontal plane by image-based visual servoing. A force servoing loop allowing the precise control of the impact and contact forces is designed taking advantage of the kinematic redundancy. Moreover, a new strategy for the release task by rolling is proposed. Experimental results of autonomous micromanipulation validate employed strategies and prove the efficiency of developed vision and force techniques. Future work will be focused on three points. First we want to improve the performance of the system by developing more robust vision and force techniques. We want also to extend the study of the rolling experiment to be able to measure micromechanical properties, such as friction coefficients. Finally, a

[1] T. Fukuda and F. Arai, “Prototyping design and automation of micro/nano manipulation system,” Proc. of 2000 IEEE Int. Conf. Robotics and Automation, pp.192–197, San Francisco, USA, April 2000. [2] C. K. M. Fung, I. Elhajj, W. J. Li and N. Xi, “A 2-D PVFD force sensing system for micromanipulation and micro-assembly,” Proc. of 2002 IEEE Int. Conf. of Robotics and Automation, pp.1489–1494, Washington, USA, May 2002. [3] D. S. Haliyo, F. Dionnet and S. Régnier, “Advanced micromanipulation applications,” Proc. of 2003 IEEE/RSJ Int. Conf. Int. Robots and Systems, pp.1870–1876, Las Vegas, USA, October 2003. [4] D. S. Haliyo, Y. Rollot and S. Régnier, “Manipulation of micro-objects using adhesion forces and dynamical effects,” Proc. of 2002 IEEE Int. Conf. Robotics and Automation, pp.1949–1954, Washington DC, USA, May 2002. [5] Y. Rollot, S. Régnier and J.-C. Guinot “Dynamical model for the micromanipulation by adhesion: experimental validations for determined conditions,” Int. J. of Micromechatronics, vol.1, no.4, pp.273–297, 2002. [6] S. Saito, H. Himeno and K. Takahashi, “Electrostatic detachment of an adhering particle from a micromanipulated probe,” J. of Applied Physics, vol.93, no.4, pp.2219–2224, February 2003. [7] S. Saito, H. T. Miyasaki, T. Sato, K. Takahashi and T. Onzawa, “Analysis of micro-object operation based on the dynamics considering the adhesion under a SEM,” Proc. of 2001 IEEE/RSJ Int. Conf. Int. Robots and Systems, pp.1349–1357, Maui, Hawaii, USA, 2001. [8] T. Tanikawa, M. Kawai, N. Koyachi, T. Arai, T. Ide, S. Kaneko, R. Ohta and T. Hirose, “Force control system for autonomous micro manipulation,” Proc. of 2001 IEEE Int. Conf. Robotics and Automation, pp. 610–615, Seoul, Korea, 2001. [9] G. Yang and B. J. Nelson, “Micromanipulation contact transition control by selective focusing and microforce control”, to appear Proc. of 2003 IEEE Int. Conf. of Robotics and Automation, Taipei, Taiwan, 2003. [10] Y. Zhou and B. J. Nelson, “The effect of material properties and gripping force on micrograsping,” Proc. of 2000 IEEE Int. Conf. Robotics and Automation, pp.1115–1120, San Francisco, USA, April 2000.