IEEE TRANSACTIONS ON ROBOTICS, VOL. 29, NO. 2, APRIL 2013
373
Three-Dimensional Visual Servo Control of a Magnetically Propelled Microscopic Bead Zhipeng Zhang, Member, IEEE, Fei Long, and Chia-Hsiang Menq, Senior Member, IEEE
Abstract—This paper presents the development of a visual servo control system that is capable of trapping and steering a microscopic magnetic bead in a 3-D workspace using a hexapole magnetic actuator. The magnetic actuator employs six sharp-tipped magnetic poles, arranged in a hexagonal configuration, and six actuating coils to achieve 3-D magnetic actuation. Real-time 3-D visual tracking is realized through processing video images acquired by a complementary metal-oxide semiconductor (CMOS) camera to achieve nanometer-scale resolution for particle tracking. A model-based nonlinear control law is developed to achieve two objectives: 1) to stabilize the magnetically propelled microscopic bead and control its Brownian motion and 2) to rapidly steer the microscopic bead within the 3-D workspace. Specifically, 3-D visual servo control of a 2.8-μm magnetic bead is employed to validate the capability of the developed control system. The control performances in terms of variance minimization, 100 nm stepping, and large range steering are experimentally demonstrated. Index Terms—Magnetic actuation, minimum variance control, motion control, visual servoing, visual tracking.
I. INTRODUCTION ROBING biological samples and manipulating biological processes using magnetically propelled microscopic beads have become an important technique in the study of cell mechanics and biophysics. By attaching the two ends of a macromolecule, such as DNA, chromatin, etc., to a magnetic bead and a coverslip, respectively, researchers applied stretching forces or rotating torques to the tethered molecule to study its behavior at single-molecule scale [1]–[5]. Microscopic beads could be attached onto cell membrane through surface functionalization and then employed to apply mechanical stress to cell membrane and study the cellular responses to mechanical forces [6]–[8]. Furthermore, microscopic beads could also be introduced into the interior of live cells to investigate intracellular properties [9]–[13].
P
Manuscript received September 9, 2012; accepted November 13, 2012. Date of publication December 11, 2012; date of current version April 1, 2013. This paper was recommended for publication by Associate Editor Y. Sun and Editor B. J. Nelson upon evaluation of the reviewers’ comments. This work was supported by the National Science Foundation under Grant CMMI-1067962. Any opinions, finding, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation. Z. Zhang was with the Ohio State University, Columbus, OH 43210 USA. He is now with the GE Global Research Center, Niskayuna, NY 12309 USA (e-mail:
[email protected]). F. Long and C.-H. Menq are with the Department of Mechanical and Aerospace Engineering, The Ohio State University, Columbus, OH 43210 USA (e-mail:
[email protected];
[email protected]). Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/TRO.2012.2229671
Magnetic tweezers are typical micromanipulation tools that are being used for biological applications. They employ electromagnets and sharp-tipped magnetic poles to generate magnetic field gradient and exert noncontact magnetic forces on microscopic magnetic beads [14]. Placing a magnetic object in a magnetic field without feedback control is, however, inherently unstable [15]–[17]. In order to achieve stability, the magnetic beads were, therefore, either functionalized and anchored to target biological samples or physically confined inside cells. In those applications, magnetic tweezers were employed as simple force appliers with open-loop control, wherein magnetic forces of desired magnitudes and directions were applied using magnetic actuators, and the induced motions of magnetic beads were recorded by appropriate measurement systems [18]–[20]. One development of magnetic tweezers did use feedback control to achieve stabilization of a 4.5-μm magnetic bead suspended in water by employing a six-pole magnetic apparatus placed above the bead [21]. The system could generate an upward magnetic force, whereas the downward force was caused by the gravity. According to the spatial configuration of the six poles employed, the direction of the resulting magnetic force was qualitatively estimated and used to determine constant feedback control gain. Although position stabilization was demonstrated, the approach was heuristic in nature. Moreover, it would be very difficult to employ this design to trap beads in smaller size, as gravity would no longer dominate over random thermal forces. There have been successful developments of millimeter- and micrometer-scale swimming robots propelled in liquid using external magnetic fields [22]–[25]. Specifically, swimming locomotion of untethered artificial bacterial flagella, consisting of a thin square head (4.5 μm × 4.5 μm × 200 nm) and a helical tail (47 μm), was achieved using low-strength, rotating magnetic fields [25]. Although prior developments focused on propelling swimming robots, an electromagnetic system was designed to control intraocular microrobots for delicate retinal procedures [26]–[28]. The untethered microrobot was an axially symmetric body with the long axis ranging from 500 to 2000 μm. Mathematical modeling of torque actuation and force actuation [26] led to rational design and arrangement of multiple standalone electromagnets to enable five-degree-offreedom (DOF) magnetic actuation of the microrobot [27]. In the case where the magnetic moment of the microrobot is independent of the input currents, the pseudoinverse method was employed to determine the needed input currents for a desired torque/force [27]. This approach greatly simplified the realization of real-time control. Moreover, a visual sensing system was employed to locate the position of the microrobot to enable
1552-3098/$31.00 © 2012 IEEE
374
IEEE TRANSACTIONS ON ROBOTICS, VOL. 29, NO. 2, APRIL 2013
visual servo control. An intraocular projection model was developed to improve localization precision. An error less than 0.5 mm was achieved when controlling the microrobot in a position-based visual-servoing framework [28]. The authors of this paper had developed a planar quadrupole magnetic tweezers, employing four electromagnets and four tipshaped magnetic poles to generate magnetic field gradient and exert 2-D controllable magnetic force on microscopic magnetic particles [29]. Active feedback control had also been realized to enable stable positioning and precise steering of a 2.8-μm magnetic bead in a 2-D workspace [30]. In this paper, we present the development of a 3-D visual servo control system that is capable of stable positioning and precise steering of a microscopic magnetic bead suspended in water. The details of design and modeling of the magnetic actuation system can be found in [31]. It is a hexapole magnetic actuator, which is an extension of the planar quadrupole magnetic actuator. The hexapole magnetic actuator is a fully actuated 3-D system, wherein six magnetic poles are placed on two parallel planes and employed to generate controllable magnetic forces in a 3-D workspace between the two planes. It is integrated with an inverse microscope along with a complementary metal-oxide semiconductor (CMOS) camera. Real-time 3D visual tracking is realized through processing video images acquired by the camera to provide the magnetic bead’s position with nanometer-scale resolution for particle tracking [32]. Since the actuator is an overactuated system, current allocation is necessary. In addition to being nonlinearly position dependent, the resulting magnetic force is nonlinearly related to the applied currents; the application of the pseudoinverse approach is, therefore, not valid. An analytical inverse model is developed to lay the foundation for the development of a model-based nonlinear control law to achieve two objectives: 1) to stabilize the magnetically propelled microscopic bead and control its Brownian motion and 2) to rapidly steer the microscopic bead within the 3-D workspace. Specifically, 3-D visual servo control of a 2.8-μm magnetic bead is employed to validate the capability of the developed control system. The control performances in terms of variance minimization, nanometer stepping, and large range steering are experimentally demonstrated. The content of this paper is organized as follows. Section II describes modeling and realization of 3-D magnetic actuation. Specifically, the implementation of a hexapole magnetic actuator is summarized. Section III presents the measurement principle, calibration, measurement range and resolution, and time delay of the visual tracking system. Visual servo control, including stabilization, calibration, minimum variance control, and experimental investigation, is presented in Section IV. Finally, conclusion is made in Section V. II. THREE-DIMENSIONAL MAGNETIC ACTUATION Three-dimensional actuation is accomplished using a hexapole magnetic actuator [31]. The actuator consists of six sharp-tipped magnetic poles placed on two parallel planes, six actuating coils, and a magnetic yoke. Each magnetic pole has its own actuating coil for magnetic flux production. The mag-
netic yoke is used to connect all six magnetic poles together to minimize the magnetic flux loss in the magnetic actuator and, hence, maximize the efficiency of magnetic field generation. A. Magnetic Actuation When electric current is applied to the coil, magnetic flux is produced and permeated through the magnetic pole to its tip. The flux then strongly diverges outward from the tip into the proximity of the tip, resulting in a high magnetic field with large field gradient. Magnetic gradient forces can, therefore, be exerted on microscopic magnetic beads placed in the field. Since all six magnetic poles are connected together using the magnetic yoke, the magnetic field generated by one magnetic pole magnetizes all other magnetic poles. The resultant magnetic field is, therefore, not the simple linear summation of fields generated by individual magnetic poles, like in [27]. In [31], magnetic circuit analysis was performed to solve this problem. The magnetic field model of the hexapole actuator is a six-pole model, which is considered as an extension of the magnetic dipole model. The magnetic field B in the workspace is position dependent, whereas without considering hysteresis and saturation, it is proportional to the input current vector I = [I1 I2 I3 . . . I6 ]T . The soft magnetic bead is magnetized in a magnetic field, and it, in turn, varies the magnetic field around it. Therefore, to calculate the magnetic force experienced by the bead, one needs to know its magnetization and the actual magnetic field around it. To simplify this calculation, effective magnetization is defined such that the magnetic force experienced by the bead can be calculated using its effective magnetization and the original external magnetic field. Details of the derivation can be found in [33]. The effective magnetization of a superparamagnetic microscopic bead when placed in the field and without being magnetized to saturation is proportional to B, i.e., m = (3V /μ0 ) · [(μ − μ0 )/(μ + 2μ0 )] B, where μ is the permeability of the particle and V is its volume, and μ0 is the permeability of vacuum. The gradient force, i.e., Fm = (1/2)∇(m · B), can then be cast into a quadratic form p, I) = kI IT KTI Li (ˆ p)KI I = kI IT Ni (ˆ p)I Fi (ˆ
(1)
where Fi is the magnetic force for the ith DOF, and kI is the force gain of the actuator 2 2 μ − μ0 Nc 3V km (2) kI = 2μ0 l5 μ + 2μ0 μ0 Ra in which km = μ0 /4π = 1.0 × 10−7 N/A2 , Nc is the turns of the coils, Ra is the lumped magnetic reluctance from the pole tip to the workspace center in the air, and is the effective radius ˆ is the bead’s position coordinate, of the workspace. Moreover, p normalized with respect to , in the workspace, KI is the 6 × 6 magnetic flux distribution matrix, which characterizes the coupling among the six magnetic poles, and Li and Ni are two 6 × 6 position-dependent matrices, characterizing magnetic gradient. p) for a multipole magnetic actuator can The derivation of Li (ˆ be found in [29] and [30], whereas KI for a hexapole magnetic actuator can be found in [31]. The two parameters in the force
ZHANG et al.: THREE-DIMENSIONAL VISUAL SERVO CONTROL OF A MAGNETICALLY PROPELLED MICROSCOPIC BEAD
375
Fig. 1. Specific hexagonal configuration of the magnetic actuator. (a) Six pole tips placed on three orthogonal axes: two on each axis. (b) Six pole tips placed on two parallel planes after a coordinate rotation is performed.
model, i.e., Ra and , need to be precisely determined. They can be quantitatively obtained first using finite-element analysis and later experimentally calibrated. With the two parameters determined, this force model accurately characterizes the nonlinearity of the magnetic force exerting on the magnetic particle with respect to the currents applied to the coils and the position dependence of the magnetic force in the workspace. B. Realization of the Hexapole Magnetic Actuator The hexapole magnetic actuator is to be assembled on an inverted microscope. The specific hexagonal configuration of the magnetic actuator is illustrated in Fig. 1, in which the solid dots indicate the spatial locations of the pole tips surrounding the workspace. The reason that six magnetic poles are employed to achieve 3-D magnetic actuation is due to the fact that each magnetic pole can only generate attractive magnetic force toward the pole tip. Hence, two opposing magnetic poles are usually placed along one axis to produce back-and-forth magnetic actuation. By employing three pairs of magnetic poles along the three orthogonal axes, as shown in Fig. 1(a), 3-D magnetic actuation can be realized. It is evident that P5 and P6 are added to a planar quadrupole magnetic actuator to form a 3-D hexapole magnetic actuator. The two magnetic poles along the z-axis would, however, block the optical path when integrated to an inverted microscope. A coordinate rotation is applied to the arrangement of the six magnetic poles to solve the problem. After the rotation, the hexapole configuration is shown in Fig. 1(b). The six poles are placed on two parallel horizontal planes, and the optical path is free of blockage. Since the actuation coordinate frame {O; xa , ya , za } is not aligned with the measurement coordinate frame {O; xm , ym , zm } of the microscope, it is necessary to establish a rotational matrix, which is denoted as amR, between the two coordinate frames. Realization of the hexapole magnetic actuator is illustrated in Fig. 2. Three 178-μm-thick magnetic poles, fabricated from nickel-iron-molybdenum foils [34], are glued onto a No. 1 coverslip (thickness 130 –150 μm), as shown in Fig. 2(a). The three pole tips are assembled to form an equilateral triangle whose side length is about 840 μm, as shown in Fig. 2(b). Then, the other three magnetic poles are fixed on a 1.2-mm-thick glass slide, as shown in Fig. 2(c), and their pole tips also form an equilateral triangle of the same size. The two pieces are then fixed face-to-face, as shown in Fig. 2(d). The distance between
Fig. 2. Realization process of the hexapole magnetic actuator. (a) Three magnetic poles on the top layer, (b) zoomed-in view of the three pole tips, (c) three poles on the bottom layer, (d) two layers assembled, (e) whole actuator setup with the magnetic yoke and six actuating coils, and (f) sample chamber.
the two layers is determined by the thickness of the spacers [see Fig. 2(c)], which is 508 μm. Therefore, the distance from the geometric center to each magnetic pole can be calculated to be = 594 μm. The magnetic yoke has the shape of a square loop, and it has six protrusions, on which six actuating coils are assembled. The yoke is made of cold-rolled steel, and the coils are wound from American wire gauge (AWG) #24 magnetic wire (diameter = 0.51 mm) with 50 turns each. The entire actuator setup is shown in Fig. 2(e). When experiments being conducted, the microscopic magnetic beads suspended in water are introduced to the workspace using a sample chamber, as shown in Fig. 2(f). The chamber is formed by stacking one piece of No. 1 coverslip, a 120-μm-thick seal ring, and one more piece of No. 1 coverslip. The actuating coils are driven by six linear power amplifiers (Model BTA-18 V-6 A, Precision Micro Dynamics). These linear power amplifiers are configured to work in current mode. The input voltages are generated by an analog-output board (PCI-6733, National Instruments, Austin, TX) in a workstation (Dell Precision workstation T7400, dual Quad Core Xeon E5405 2.00 GHz, 2 GB RAM). Therefore, controllable currents can be applied to the actuating coils through computer commands. Since saturation and hysteresis of magnetic materials degrade the performance of the hexapole actuator, they should be avoided as much as possible in the process of design and implementation. Materials with low hysteresis loss are preferred for the fabrication of the yoke and poles. The input current applied to the coils is limited to a predetermined maximum value to prevent magnetic saturation. C. Inverse Model and Force Generation The resulting magnetic force is nonlinearly position dependent and nonlinearly related to the applied currents. Moreover, since the actuator is an overactuated system, current allocation is a challenge. For simplicity, input current and the resulting magnetic force are normalized. To prevent magnetic saturation caused by excessive current, the input current is limited to Im ax . By normalizing the current vector with respect to Im ax , i.e.,
376
IEEE TRANSACTIONS ON ROBOTICS, VOL. 29, NO. 2, APRIL 2013
ˆI = I/Im ax , the force model can be normalized as p, ˆI) = Fˆi (ˆ
Fi = ˆIT Ni (ˆ p)ˆI. 2 kI Im ax
(3)
2 The normalized force Fˆi is dimensionless, and kIˆ = kI Im ax is the lumped force gain that has the unit of Newton. Specifically, kIˆ is related to the particle properties, magnetic circuit properties, and the maximum input current. The force model enables the determination of the magnetic force exerted on the microscopic magnetic bead at a specified spatial location when the input currents applied to the coils are known, whereas an inverse force model that determines the input currents to produce a desired magnetic force is necessary for the practical use of the actuator. Since the six poles are arranged symmetrically in the actuator coordinate frame, the normalized force at the center of the workspace can be simplified significantly from (3): ⎧ˆ F (0, ˆI) = 2(Iˆ1 − Iˆ2 )[2(Iˆ1 + Iˆ2 ) − (Iˆ3 + Iˆ4 ) − (Iˆ5 + Iˆ6 )] ⎪ ⎨ x Fˆy (0, ˆI) = 2(Iˆ3 − Iˆ4 )[2(Iˆ3 + Iˆ4 ) − (Iˆ1 + Iˆ2 ) − (Iˆ5 + Iˆ6 )] . ⎪ ⎩ Fˆz (0, ˆI) = 2(Iˆ5 − Iˆ6 )[2(Iˆ5 + Iˆ6 ) − (Iˆ1 + Iˆ2 ) − (Iˆ3 + Iˆ4 )] (4) This result accords with the fact that (I1 , I2 ) is a pair of actuation currents applied to the two poles facing each other and aligned in the x-direction; (I3 , I4 ) in the y-direction; and (I5 , I6 ) in the z-direction, respectively. Redundancy is removed by imposing three linear constraints, i.e., Iˆ1 + Iˆ2 = cx , Iˆ3 + Iˆ4 = cy , and Iˆ5 + Iˆ6 = cz . Letting δ Iˆx = Iˆ1 − Iˆ2 , δ Iˆy = Iˆ3 − Iˆ4 , and δ Iˆz = Iˆ5 − Iˆ6 , an exact linear relationship between the effective actuation current, i.e., δˆI = [δ Iˆx δ Iˆy δ Iˆz ]T , and ˆ the force at the center, i.e., F(0, δˆI) = [Fˆx Fˆy Fˆz ]T , can be cast as
ˆ F(0, δˆI) = JδˆI δˆI
(5)
where JδˆI = 2 · diag[(2cx − cy − cz ), (−cx + 2cy − cz ) (−cx − cy + 2cz )].
(6)
The effective actuation current δˆI required for force generation at the center can be analytically determined according to (5), whereas it is not straightforward to obtain the exact inverse solution for the entire workspace. Nonetheless, by keeping the first-order terms in Taylor expansion, a linear approximation of the magnetic force around the center can be derived ˆ p, δˆI) ≈ J ˆ δˆI + Jpˆ p ˆ F(ˆ δI
(7)
where Jpˆ = 2 · diag[(2cx − cy − cz )2 , (−cx + 2cy − cz )2 (−cx − cy + 2cz )2 ].
(8)
The effective actuation current required to produce the desired force is derived as ˆ (ˆ ˆ. δˆI = J−1 p) − J−1 Jˆ p F δˆI d δˆI p
(9)
Fig. 3. Calibrated model established by acquiring all radius vectors as the bead moves along the axial direction.
The effective actuation current along with the three linear constraints can be employed to calculate the normalized current ˆI and then the required actuation current I: I = Inverse(Fd , p).
(10)
It is worth noting that the desired force Fd and the spatial location p are associated with the measurement coordinate frame. When employing (9) to realize inverse model calculation, the rotational matrix amR between the two coordinate frames needs to be applied to both vectors. III. THREE-DIMENSIONAL VISUAL TRACKING When imaging the motion of a microscopic particle, it is relatively straightforward to determine the lateral motion, whereas the axial localization is much more challenging. A 3-D visual tracking scheme is implemented and employed to acquire the absolute position, with respect to the microscope’s coordinate system, of microscopic beads. Since the magnetic field has significant spatial heterogeneity in the workspace and the force exerted on the magnetic bead is position dependent, position detection serves at least two purposes. First, it determines the bead’s deviation from the target equilibrium point to enable active control for stabilization. Second, it enables model-based nonlinear control, e.g., feedback linearization, to achieve rapid steering of the magnetic particle. A. Measurement Principle The xyz directions referred in this section are the three main axes of the measurement coordinate frame. The lateral motion (x, y) of the microscopic bead is estimated from the acquired image using a widely accepted centroid method [35], [36]. Since the image of a microscopic particle is circularly symmetric, the centroid of the 2-D image of the bead represents the center location of the bead projected to the xy plane. However, the axial position estimation is not as obvious. It is based on the image variation when the bead moves along the z-axis. The measurement concept is explained in Fig. 3.
ZHANG et al.: THREE-DIMENSIONAL VISUAL SERVO CONTROL OF A MAGNETICALLY PROPELLED MICROSCOPIC BEAD
377
of the stage vanishes and then takes 100 images at a frame rate of 200 frames/s in the remaining 0.5 s. After converting the acquired images into radius vectors, a single radius vector is obtained for each z-position by averaging 100 vectors to reduce the effect of noise. These low-noise vectors are then combined into a discrete model and, finally, synthesized to a continuous model. C. Measurement Range and Resolution Fig. 4. Two-dimensional image converted to a radius vector through radial projection.
On the left-hand side of Fig. 3, there are three bead images acquired at three distinct z locations under a microscope. They are, from top to bottom, the above-focus, the in-focus, and the belowfocus images. The in-focus image of the microscopic bead has sharp edges and high contrast, whereas the off-focus images are blurry with concentric diffraction rings. The above-focus and the below-focus images are different due to the inherent aberration errors of the objective lens. Therefore, once an object-specific model that relates the varying images to the axial positions of the bead is established through calibration prior to undertaking the experiments, it can be utilized to estimate the axial location of the particle. The axial position measurement consists of three steps [32]. First, through radial projection, the circularly symmetric 2-D image is converted to a radius vector, as illustrated in Fig. 4. During the process of projection, each pixel of the image is projected to the radius vector according to its distance to the bead center, which has already been determined using the centroid method. By projecting all the pixels in the image this way, the 2-D image is compressed to the radius vector with no information loss. Second, automated calibration is conducted to generate a discrete model by acquiring all the radius vectors as the bead moves along the z-axis, as shown on the right-hand side of Fig. 3. The details of the calibration will be explained in more detail in the next section. The calibrated discrete model is further synthesized to a continuous object-specific model by spline fitting along the z-axis, under the assumption that the image intensity profile changes smoothly. Third, during real-time measurement, when an image is acquired, it is first converted to a radius vector. Then, this radius vector is compared with the calibrated model, and the axial position is determined using a least-squares matching algorithm. B. Automated Calibration Since the lateral motion (x, y) of the bead can be measured directly from the acquired image, 2-D measurement and, thus, 2-D motion control in the xy plane can be established. At the same time, a constant downward magnetic force is applied to pull the bead downwards against the top surface of the bottom coverslip of the sample chamber to stabilize the motion in the z-axis. Once the motion of the bead is stabilized, an automatic routine is executed to move the piezoelectric stage along the z-axis at constant step (200 nm) every 0.6 s. The routine first waits for 0.1 s right after each step so that the transient response
The measurement range along the x- and y-axes is limited by the field of view of the microscope, e.g., the range is around 200 μm in our current implementation with a 60× objective lens. The calibration range, and thereby the measurement range along the z-axis, is limited by the depth of field of the optical system. When the bead goes too far from the focus, the contrast of the off-focus image will be too low to distinguish the bead from the background. In practice, the motion range of the piezoelectric stage (which is 20 μm) along the z-axis is another limiting factor to the calibration range. In the setup employed for visual tracking, the condenser’s aperture can be tuned. Reducing the aperture of the condenser will increase the depth of field and, therefore, the measurement range along the z-axis, whereas increasing the measurement range along the z-axis will inevitably compromise its measurement resolution. The measurement range along the z-axis in current setup is 18 μm. The measurement resolution of the computer-vision algorithm has been experimentally determined using a stationary microscopic bead fixed to the piezoelectric stage. The measurement resolution along the x- and y-axes is around 3-nm RMS, regardless of z-position, and, along the z-axis, 5–12-nm RMS, depending on the axial location of the bead, as the measurement resolution along the z-axis varies with respect to the z-position of the bead. The variation in measurement resolution along the z-axis is a result of the off-focus image variation of the bead. When the bead is near focus, the bead image profile is more sensitive to the changes in z-position than when the bead is farther from the focus. Therefore, higher measurement resolution in z-axis can be realized when the bead is near focus. D. Measurement Time Delay The photodetectors of a camera accumulate light during the exposure time interval to produce image signal. The exposure time of the CMOS camera employed in our experiments is set to be equal to the sampling time ts of the feedback control loop. The Fourier transform of a rectangular time interval, from 0 to ts , is a sinc function with a time delay of ts /2. The sinc function has amplitude of one at zero frequency and smoothly reduces to zero amplitude as the frequency increases to 1/ts . It can, therefore, be considered as a low-pass filter. The low-pass filtering effect is, however, not significant, as we are mostly interested in the signals band-limited within the Nyquist frequency 1/(2ts ) of the digital control loop. By ignoring the low-pass filtering effect, the image sampling process can be simply considered as a pure time delay of ts /2. In other words, a 0.5-step time delay is introduced from image sampling. The acquired image is then transferred
378
IEEE TRANSACTIONS ON ROBOTICS, VOL. 29, NO. 2, APRIL 2013
Fig. 6. Visual servo control system. (a) Block diagram showing feedback linearization. (b) Simplified block diagram.
Fig. 5.
Physical setup of the visual servo control system.
from the camera to the vision processor board, and this process introduces a one-step delay. Afterward, on-board processing and computer algorithms are performed to calculate the 3-D positions of the bead, which brings in another 0.5-step delay when operating at a 200-Hz sampling rate. The total time delay introduced by the visual measurement system is, therefore, two steps of the sampling period.
by incorporating the inverse force model (10). Once the force model employed is accurate, feedback linearization effectively cancels the nonlinear relationship in 3-D magnetic actuation, and the control block diagram is greatly simplified, as shown in Fig. 6(b). Subsequently, a linear controller can be designed to deal with linear dynamics of the microscopic magnetic bead. B. Stabilization The motion of the microscopic bead in water can be described by the Langevin equation [37]. As the motion dynamics is equivalent in the three directions, for simplicity, the dynamic and control analyses are expressed in the x-direction. The equation of motion along the x-axis is expressed as
IV. VISUAL SERVO CONTROL SYSTEM
m¨ x(t) + γ x(t) ˙ = FT x (t) + Fx (t)
Three-dimensional magnetic actuation and 3-D visual tracking have been introduced in the previous two sections. In this section, visual feedback control of the microscopic magnetic bead is presented. The physical setup of the visual servo control system is shown in Fig. 5, wherein the hexapole magnetic actuator is assembled on an inverted microscope (TE2000-U, Nikon), between a high numerical aperture (NA) 60× dry objective lens (CFI Super Plan Fluor ELWD 60XC, 0.70 NA, Nikon) and a high-NA condenser (CLWD, 0.72 NA, Nikon). The thickness of the actuator is less than 10 mm. It is directly fixed on the top of a three-axis piezoelectric stage (P-517.3CL, Physik Instrumente, Bedford, U.K.), which is assembled on top of the microscope’s manual planar stage. The piezoelectric stage is capable of providing subnanometer precision motion in all three translational directions in a 100 μm × 100 μm × 20 μm workspace.
where m is the mass of the magnetic bead, γ is the bead’s drag coefficient in water, FT x (t) is the random thermal force in x-direction, and Fx (t) the applied magnetic force in x-direction. In addition to feedback linearization, a proportional-gain controller is employed to stabilize the motion of the bead. The control law is to be implemented in a digital computer, whereas for simplicity, continuous time analysis is presented. The proportional control law can be expressed by
A. Feedback Linearization and Current Allocation The control block diagram is illustrated in Fig. 6(a), wherein pd is the desired position, Fd is the desired magnetic force, I is the vector of currents applied to the coils, FT is the random thermal force, and p and pm are the actual and measured positions of the magnetic bead, respectively. The digital controller is divided into two blocks in the diagram, i.e., linear controller and feedback linearization along with current allocation. The latter block is employed to deal with three issues in 3-D magnetic actuation, namely, nonlinearity, position dependence, and redundancy. It calculates the six input currents I to produce the desired 3-D magnetic force Fd
Fx (t) = Kp (xd − x(t − td ))
(11)
(12)
where Kp is the proportional feedback gain, xd is the set point, and td is the total time delay in the control loop. It is worth noting that Kp in (12) is the product of kp , the control gain programmed in the linear controller, and kIˆ is the force gain of the actuator. For a 2.8-μm-diameter magnetic bead, the inertia force m¨ x(t) is negligible when compared with the drag force γ x(t) ˙ at the sampling rate of 200 Hz. Therefore, taking Laplace transforms of the equation of motion with feedback control, the motion of the magnetic bead can be derived as 1 {FT x (s) + Kp Xd (s) + δFx (s)} γs + Kp e−τ d s (13) where δFx is the force generation error due to the imperfection of the force model. It can be seen that by increasing the control gain x tends to track xd better, the Brownian motion caused by thermal noise decreases, and the effect of modeling error is reduced, whereas high gain lead to instability due to X(s) =
ZHANG et al.: THREE-DIMENSIONAL VISUAL SERVO CONTROL OF A MAGNETICALLY PROPELLED MICROSCOPIC BEAD
Fig. 7. PSDs of the bead’s Brownian motions with proportional gain feedback control. The experimental results are in blue, whereas the theoretical calculations are in red. The proportional feedback gains are 0.32, 0.64, 0.96, 1.28, 1.44, 1.68, 1.92, and 2.16 μN/m, respectively, for the curves from top to bottom near lowfrequency range.
379
Fig. 8. Standard deviation of the bead’s Brownian motion versus proportional control gain.
delay. The application of the Nyquist stability criterion yields a fundamental tradeoff between delay and permissible feedback gain πγ . (14) Kp < 2td C. Calibration When xd is kept constant, the bead undergoes Brownian motion caused by random thermal forces FT x (t). As the power spectral density (PSD) of the thermal force is known, SF T x (f ) = 4γkB T [38]–[40], where kB = 1.38 × 10−23 J · K−1 is the Boltzmann constant and T is the absolute temperature, the PSD of the controlled Brownian motion of the bead is derived from (13) Sx (f ) =
4γkB T . |iγ2πf + Kp exp(−i2πf τd )|2
(15)
The total time delay in (15) is the combination of the two-step measurement delay and the equivalent 0.5-step delay due to the zero-order hold in digital implementation, i.e., td = 2.5ts . It is evident from (15) that the Brownian motion can be reduced by increasing the control gain, i.e., Sx (0) = 4γkB T Kp2 . However, when the control gain is too large, the PSD exhibits a resonance peak at f ≈ 1/4τd due to time delay. The result of this theoretical analysis was verified experimentally. With different digital gain kp , increased from low to high until the trap went unstable, the PSDs of the bead’s Brownian motion were obtained and compared with those calculated using (15). This comparison enabled the calibration of the force gain kIˆ along the x-axis, which was estimated to be 0.53 pN when the maximum input current Im ax is 1.2 A. The experimental PSDs are plotted in blue dotted line in Fig. 7, whereas the theoretical PSD curves are plotted in red. The standard deviations of the Brownian motion with increasing proportional gains are plotted in Fig. 8. As illustrated in this figure, there exists an optimal proportional gain that minimizes the bead’s Brownian motion. When the gain is higher
Fig. 9.
Calibrated force gain versus Im a x employed for normalization.
than the optimal gain, the bead starts to oscillate at the resonant frequency, and a peak appears in the PSD plot, as illustrated in Fig. 7. The control performance thus degrades. When the feedback gain reaches its optimal value, i.e., Kp = 1.68 × 10−6 N/m, the standard deviation of the bead motion reaches a minimum value of 78 nm for proportional-gain feedback control. This optimal gain corresponds to the fifth curve in Fig. 7, counting from top near the low-frequency range. It is worth noting that the optimal feedback gain is proportional to γ/td and that the minimum variance of the associated Brownian motion is inversely proportional to the optimal feedback gain. Since γ is proportional to the radius of the bead, the variance is inversely proportional to the size of the bead. 2 As the ideal force gain kIˆ is proportional to Im ax until saturation occurs. Using the same calibration procedure with increasing values of Im ax , magnetic saturation of the specific magnetic actuator was investigated. The experimental data illustrating the relationship between kIˆ and Im ax for each of the three axes are plotted in Fig. 9. It can be seen that the three sets of data, associated with the x-, y-, and z-axes, respectively, are in close resemblance. A dashed parabola is drawn in the figure to illustrate the ideal quadratic relationship. It is evident from the figure that the experimental data accord with the parabola very well until Im ax reaches 1.5 A. After exceeding 1.5 A, the increase of
380
IEEE TRANSACTIONS ON ROBOTICS, VOL. 29, NO. 2, APRIL 2013
kIˆ is reduced, indicating magnetic saturation of the actuator’s magnetic poles. D. Minimum Variance Control Although a simple proportional controller can be employed to achieve stabilization and enable calibration of the actuator’s force gain, an optimal controller is designed and implemented from the point of view of Brownian motion control. The design is based on the exact discrete-time model of the bead dynamic in the sampled-data control system ts z −1 X −1 (z ) = Fx γ(1 − z −1 )
(16)
and the fact that the visual sensing and data conversing introduce two-step time delay. The difference equation in terms of the measured bead motion can be derived: ts (17) xm (k) − xm (k − 1) = z −2 Fx (k − 1) + ε(k) γ
Fig. 10. Standard deviation of the bead’s Brownian motion versus detuned control gain of the minimum-variance controller.
where ε(k) is the incremental random motion induced by the thermal force with E{ε(t)} = 0 and E{ε2 (t)} = 2kB T ts /γ [38]. The design objective is to minimize the variance of the microscopic bead’s Brownian motion. Since it is a well-established design technique [41], detailed derivation is not presented here. Only the transfer function of the designed controller is given: C(z −1 ) =
Fx −1 γ (z ) = E ts (1 + z −1 + z −2 )
(18)
where E(z −1 ) is the z-transform of the error signal e(k) = xd (k) − xm (k). With this controller, it can be shown that when xd = 0, the measured bead motion is only affected by the thermal noise xm (k) = ε(k) + ε(k − 1) + ε(k − 2).
(19)
This controller enables the magnetic actuator to propel the microscopic bead to follow the reference input in three time steps. When the bead deviates from its set position, it takes two sampling periods to obtain the position from the visual tracking system, and then, it takes one additional step for the control effort to move the bead back to its set position. Hence, the tracking error is a consequence of the thermal noise within three sampling periods, i.e., var(xm ) = 3var(ε) = 6
kB T ts . γ
(20)
The advantage of minimum variance control is evident. The optimal controller can be directly designed. Similar to the experiments presented for gain calibration using a proportional controller, the control gain of the controller, i.e., C(z −1 ) = 3Km v /(1 + z −1 + z −2 ), was deliberately detuned in the experiment to illustrate the performance of minimum variance control. Specifically, the control gain was varied from low to high until the magnetic trap went unstable. The standard deviations of the bead’s Brownian motion versus control gains are plotted in Fig. 10. It can be seen that optimal performance is achieved when the control gain is near Km v = 1.9 × 10−6 N/m, which accords well with the result of optimal design γ/(3ts ) =
Fig. 11. Positioning performance of the visual servo control system: The position is drawn in the measurement coordinate frame. The data points are intentionally drawn with large marker for display purpose. The real Brownian motion is within ±200 nm.
1.76 × 10−6 N/m. The slight difference can be explained by the ignored low-pass filtering effect of the visual measurement. The standard deviation at the optimal gain is 63 nm. Since γ is proportional to the radius of the bead, the variance of the Brownian motion is inversely proportional to the size of the bead. E. Visual Servo Control Three experiments were conducted to illustrate visual servo control of the developed system. First, the microscopic magnetic bead was controlled to raster scan and stop at 75 locations, arranged in a grid pattern of 5 × 5 × 3 with lateral step 20 μm and axial step 5 μm. The bead was controlled to stay at each location for 10 s before moving to the next one. Since the position dependence of magnetic actuation is compensated through feedback linearization, uniform control performance is expected near the center of the workspace. The result is shown in Fig. 11, wherein at each location, after the transient response has vanished, the bead’s positions over the last 5 s are plotted. The magnitudes of the bead’s Brownian motion at different locations are of little difference, whereby the effectiveness of feedback linearization is partially validated. Second, the magnetic bead was controlled to perform 100-nm stepping along the x-axis. Large motion fluctuation exists due to the effect of Brownian motion, as shown in Fig. 12. A 100-point
ZHANG et al.: THREE-DIMENSIONAL VISUAL SERVO CONTROL OF A MAGNETICALLY PROPELLED MICROSCOPIC BEAD
381
development of a more general inverse force model enables more accurate feedback linearization over the entire workspace and yields greater force generation capability of the magnetic actuator. REFERENCES
Fig. 12. Performance of 100-nm stepping in the x-axis: The raw position data are drawn in cyan and the filtered data in black.
Fig. 13.
Performance of tracking the outline of a 10-μm cube.
moving average filter is, therefore, applied to condition the raw position data in offline analysis to reduce motion fluctuation, and the filtered result is overlaid in the figure. The 100-nm stepping can be clearly seen. Third, the magnetic bead was controlled to scan the outline of a 10-μm cube at a speed of 5 μm/s. The result is shown in Fig. 13. V. CONCLUSION A visual servo control system has been developed to magnetically trap and propel a microscopic magnetic bead in 3-D workspace. Three-dimensional magnetic actuation was achieved using a hexapole magnetic actuator, whereas a 3-D visual tracking scheme was employed to provide position feedback. An inverse force model was derived and employed to overcome three challenges in 3-D magnetic actuation: 1) redundancy in current allocation; 2) nonlinearity and coupling in force generation; and 3) position dependence of the magnetic force exerting on the magnetic bead. A model-based nonlinear control law, combining feedback linearization and minimum variance control, was subsequently developed to enable 3-D visual servo control of a magnetically propelled microscopic bead suspended in water. There are at least two needed improvements. First, shortening measurement delay is the most important key to improving the performance of Brownian motion control. Second, the
[1] C. Haber and D. Wirtz, “Magnetic tweezers for DNA micromanipulation,” Rev. Sci. Instrum., vol. 71, no. 12, pp. 4561–4570, 2000. [2] J. Zlatanova and S. H. Leuba, “Magnetic tweezers: A sensitive tool to study DNA and chromatin at the single-molecule level,” Biochem. Cell Biol., vol. 81, pp. 151–159, 2003. [3] C.-H. Chiou and G.-B. Lee, “A micromachined DNA manipulation platform for the stretching and rotation of a single DNA molecule,” J. Micromech. Microeng., vol. 15, pp. 109–117, 2005. [4] C.-H. Chiou, Y.-Y. Huang, M.-H. Chiang, H.-H. Lee, and G.-B. Lee, “New magnetic tweezers for investigation the mechanical properties of single DNA molecules,” Nanotechnol., vol. 17, pp. 1217–1224, 2006. [5] J. K. Fisher, J. Cribb, K. V. Desai, L. Vicci, B. Wilde, K. Keller, R. M. Taylor II, J. Haase, K. Bloom, E. Timothy O’Brien, and R. Superfine, “Thinfoil magnetic force system for high-numerical-aperture microscopy,” Rev. Sci. Instrum., vol. 77, pp. 023702-1–023702-9, 2006. [6] A. R. Bausch, F. Ziemann, A. A. Boulbitch, K. Jacobson, and E. Sackmann, “Local measurements of viscoelastic parameters of adherent cell surfaces by magnetic bead microrheometry,” Biophys. J., vol. 76, pp. 2038–2049, Oct. 1998. [7] F. J. Alenghat, B. Fabry, K. Y. Tsai, W. H. Goldmann, and D. E. Ingber, “Analysis of cell mechanics in single vinculin-deficient cells using a magnetic tweezer,” Biochem. Biophys. Res. Commun., vol. 277, pp. 93–99, 2000. [8] H. Huang, C. Y. Dong, H.-S. Kwon, J. D. Sutin, R. D. Kamm, and P. T. C. So, “Three-dimensional cellular deformation analysis with a twophoton magnetic manipulator workstation,” Biophys. J., vol. 82, pp. 2211– 2223, 2002. [9] E. H. C. Crick and A. F. W. Hughes, “The physical properties of cytoplasm: a study by means of the magnetic particle method,” Exp. Cell Res., vol. 1, pp. 36–80, 1949. [10] A. R. Bausch, W. Moller, and E. Sackmann, “Measurement of local viscoelasticity and forces in living cells by magnetic tweezers,” Biophys. J., vol. 76, pp. 573–579, 1999. [11] B. G. Hosu, K. Jakab, P. Banki, and F. I. Toth, “Magnetic tweezers for intracellular applications,” Rev. Sci. Instum., vol. 74, no. 9, pp. 4158– 4163, 2003. [12] A. H. B. de Vries, B. E. Krenn, R. van Driel, and J. S. Kanger, “Micro magnetic tweezers for nanomanipulation inside live cells,” Biophys J, vol. 88, no. 3, pp. 2137–2144, 2005. [13] J. S. Kanger, V. Subramaniam, and R. van Driel, “Intracellular manipulation of chromatin using magnetic nanoparticles,” Chromosome Res., vol. 16, pp. 511–522, 2008. [14] P. Kollmannsberger and B. Fabry, “High-force magnetic tweezers with force feedback for biological applications,” Rev. Sci. Instrum., vol. 78, p. 114301, 2007. [15] S. Earnshaw, “On the nature of the molecular forces which regulate the constitution of the luminiferous ether,” Trans. Cambridge Philosph. Soc., vol. 7, pp. 97–112, 1842. [16] Z. Zhang and C.-H. Menq, “Six-axis magnetic levitation and motion control,” IEEE Trans. Robot., vol. 23, no. 2, pp. 196–205, Apr. 2007. [17] S.-K. Kuo and C.-H. Menq, “Modeling and control of a six-axis precision motion control stage,” IEEE/ASME Trans. Mechatronics, vol. 10, no. 1, pp. 50–59, Feb. 2005. [18] A. Pralle, M. Prummer, E.-L. Florin, E. H. K. Stelzer, and J. K. H. Horber, “Three-dimensional high-resolution particle tracking for optical tweezers by forward scattered light,” Microsc. Res. Tech., vol. 44, pp. 378–386, 1999. [19] V. Levi, Q. Ruan, and E. Gratton, “3-D particle tracking in a two-photon microscope: Application to the study of molecular dynamics in cells,” Biophys. J., vol. 88, pp. 2919–2928, 2005. [20] Z. Zhang and C.-H. Menq, “Best linear unbiased axial localization in threedimensional fluorescent bead tracking with subnanometer resolution using off-focus images,” J. Opt. Soc. Amer. A, vol. 26, no. 6, p. 106262, 2009. [21] C. Gosse and V. Croquette, “Magnetic tweezers: Micromanipulation and force measurement at the molecular level,” Biophys. J., vol. 82, pp. 3314– 3329, 2002.
382
[22] T. Honda, K. I. Arai, and K. Ishiyama, “Micro swimming mechanisms propelled by external magnetic fields,” IEEE Trans. Magn., vol. 32, no. 5, pp. 5085–5087, Sep. 1996. [23] K. Ishiyama, K. I. Arai, M. Sendoh, and A. Yamazaki, “Spiral-type micromachine for medical applications,” J. Micromechatronics, vol. 2, no. 1, pp. 77–86, 2003. [24] K. B. Yesin, K. Vollmers, and B. J. Nelson, “Modeling and control of untethered biomicrorobots in a fluidic environment using electromagnetic fields,” Int. J. Robot. Res., vol. 25, no. 5/6, pp. 527–536, 2006. [25] L. Zhang, J. J. Abbott, L. X. Dong, B. E. Kratochvil, D. Bell, and B. J. Nelson, “Artificial bacterial flagella: Fabrication and magnetic control,” Appl. Phys. Lett., vol. 94, pp. 064107-1–064107-3, 2009. [26] J. J. Abbott, O. Ergeneman, M. P. Kummar, A. M. Hirt, and B. J. Nelson, “Modeling magnetic torque and force for controlled manipulation of softmagnetic bodies,” IEEE Trans. Robot., vol. 23, no. 6, pp. 1247–1252, Dec. 2007. [27] M. P. Kumar, J. J. Abbott, B. E. Kratochvil, R. Borer, A. Sengul, and B. J. Nelson, “OctoMag: An electromagnetic system for 5-DOF wireless micromanipulation,” IEEE Trans. Robot., vol. 26, no. 6, pp. 1006–1017, Dec. 2010. [28] C. Bergeles, B. E. Kratochvil, and B. J. Nelson, “Visually servoing magnetic intraocular microdevices,” IEEE Trans. Robot., vol. 28, no. 4, pp. 798–809, Aug. 2012. [29] Z. Zhang, K. Huang, and C.-H. Menq, “Design, implementation, and force modeling of quadrupole magnetic tweezers,” ASME/IEEE Trans. Mechatronics, vol. 15, no. 5, pp. 704–713, Oct. 2010. [30] Z. Zhang, Y. Huang, and C.-H. Menq, “Actively controlled manipulation of a magnetic microbead using quadrupole magnetic tweezers,” IEEE Trans. Robot., vol. 26, no. 3, pp. 531–541, Jun. 2010. [31] Z. Zhang and C.-H. Menq, “Design and modeling of a 3-D magnetic actuator for magnetic microbead manipulation,” IEEE/ASME Trans. Mechatronics, vol. 16, no. 3, pp. 421–430, Jun. 2011. [32] Z. Zhang and C.-H. Menq, “Three-dimensional particle tracking with subnanometer resolution using off-focus images,” Appl. Opt., vol. 47, no. 14, pp. 2361–2370, 2008. [33] S. Chikazumi, Physics of Magnetism. New York: Wiley, 1964. [34] The MuShield Co., (2012). [Online]. Available: http://www.mushield. com/ [35] G. Cao and X. Yu, “Accuracy analysis of a Hartmann-Shack wavefront sensor operated with a faint object,” Opt. Eng., vol. 33, pp. 2331–2335, 1994. [36] B. F. Alexander and K. C. Ng, “Elimination of systematic error in subpixel accuracy centroid estimation,” Opt. Eng., vol. 30, pp. 1320–1331, 1991. [37] W. T. Coffey, Y. P. Kalmykov, and J. T. Waldron, The Langevin Equation: With Applications to Stochastic Problems in Physics, Chemistry and Electrical Engineering, 2nd ed.. Singapore: World Scientific, 2004. [38] A. Papoulis and S. U. Pillai, Probability, Random Variables, and Stochastic Process, 4th ed. Boston, MA: McGraw-Hill, 2002. [39] J.-Q. Sun, Stochastic Dynamics and Control. New York: Elsevier, 2006. [40] F. Gittes and C. F. Schmidt, “Thermal noise limitations on micromechanical experiments,” Eur Biophys. J., vol. 27, pp. 75–81, 1998. [41] K. J. Astrom, Introduction to Stochastic Control Theory. New York: Academic, 1970.
IEEE TRANSACTIONS ON ROBOTICS, VOL. 29, NO. 2, APRIL 2013
Zhipeng Zhang (S’08–M’09) received the B.S. degree from the Department of Precision Instrument and Mechanology, Tsinghua University, Beijing, China, in 2003 and the M.S. and Ph.D. degrees, both in mechanical engineering, from The Ohio State University, Columbus, in 2005 and 2009, respectively. He is currently a Lead Mechanical Engineer with GE Global Research Center, Niskayuna, NY. His research interests include 1) the design, development, and control of precision electromechanical and robotic systems; 2) optical metrology, signal processing, and computer vision; and 3) cell mechanics and bioprocess automation. Dr. Zhang is a member of the American Society of Mechanical Engineers.
Fei Long received the B.S. degree from the Department of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, China, in 2009. He is currently working toward the Ph.D. degree with the Department of Mechanical and Aerospace Engineering, The Ohio State University, Columbus. His research interests include modeling, control, signal detection, and estimation of electromechanical systems.
Chia-Hsiang Menq (M’03–SM’08) received the B.S. degree from National Tsing-Hua University, Hsinchu, Taiwan, in 1978 and the M.S. and Ph.D. degrees from Carnegie Mellon University, Pittsburgh, PA, in 1982 and 1985, respectively, all in mechanical engineering. Since 1985, he has been with the Ohio State University, Columbus, where he is currently a Professor and Ralph W. Kurtz Chair in Mechanical Engineering. His research interests include 1) control, sensing, and instrumentation; 2) metrology, precision engineering, and manufacturing; 3) imaging and mechanical characterization of live cells; and 4) visual sensing and visual servo control. Dr. Menq is a Fellow of the American Society of Mechanical Engineers, the Association for the Advancement of Science, and the Society of Manufacturing Engineers.
