Co-Located 3D Graphic and Haptic Display using Electromagnetic Levitation Peter Berkelman∗
Muneaki Miyasaka†
Juaquin Anderson‡
Department of Mechanical Engineering University of Hawaii-Manoa
A BSTRACT This paper presents a novel system in which 3D graphics and highfidelity haptic feedback can be generated seamlessly at the same physical location, without using mirrors or any other hardware other than a tool handle and the user’s hand in the line of sight of the user. A thin flat screen is placed directly above an array of coils so that electromagnetic forces and torques can be generated through the display to act upon magnets inside the tool handle held by the user. An array of 16 coils generates forces and torques upon 3 magnets embedded in the tool handle. A detailed model of the forces and torques generated on a single magnet from the current through a single coil was calculated analytically. The coil currents required to generate desired haptic interaction forces and torques on the tool handle are calculated from this model at each control update using redundant control methods. Forces of several Newtons gan be generated on the handle in any orientation within approximately 25 mm from the display. Tool position and orientation feedback is provided by an optical motion tracking sensor and infrared LEDs on the tool handle. The design and analysis of the prototype system is presented with results from a simple interactive environment. 1
I NTRODUCTION
The overall aim of haptic interaction technology is to enable users to physically interact with a simulated environment. In practice, this interaction generally requires the user to manipulate a haptic device with the hand to control the representation of a tool in a graphical simulated environment, while feeling the force and torque response calculated from the simulation environment and exerted on the user’s hand through the haptic device. The haptic device and graphical display are generally separate and forces and motions of the device and simulation are mapped to one another by control system software. In our system described here, haptic force and torque feedback is generated from an array of coils behind a thin flat panel display, onto magnets contained in the handle held by the user. Thus the haptic device grasped by the user and the 3D graphical environment observed by the user can occupy the same physical space. The system is shown schematically in Figure 1 and is pictured in use in Figure 2. The graphical display and the tool handle are both simultaneously in the field of view of the user, with a seamless transition between the physical and graphical environments, provided that the display is correctly aligned and scaled to the haptic device. The user perceives the real tool handle inside or immediately in front of a virtual environment. ∗ e-mail:
[email protected] [email protected] ‡ e-mail:
[email protected] † e-mail:
Figure 1: Co-located electromagnetic haptic and graphic display concept
2
BACKGROUND
Several systems have previously been developed to provide colocated graphics and haptics, either by using a video camera to capture a real-time image of the user’s hand to be inserted into the graphical environment, or by using a half-silvered mirror to superimpose a haptic device and the user’s perception of a 3D graphical environment. Graphics and haptic feedback can also be colocated by placing a haptic interface device in front of a 3D display and generating 3D imagery in front of the screen, however this arrangement is likely to be tiring to observe and the links of the haptic device remain in view, obscuring the display and disturbing the illusion of direct interaction with the graphic imagery. Head-mounted stereo graphics displays can also provide co-located haptics and graphics when used with a head-tracking position sensing system. The WYSIWYF system developed by Yokokoji [14] used a small video camera behind a flat panel display to capture a real-time image of the user’s hand while interacting with a haptic device and place the image of the hand into the displayed simulated environment using a chroma key technique. This system requires the user to reach behind the display to interact with an unseen haptic device, and a uniform blue or green background seen by the camera. Position marker lights on the haptic interface device enable the haptic and graphical environments to be registered to one another using the video image. ImmersiveTouch [8], Reachin, and SenseGraphics have commercialized display systems which use half-mirror displays to superimpose a 3D graphical simulation environment onto the physical space of the user’s hand and a haptic interaction device, typically a Sensable Technologies Phantom [9] or Novint Falcon [11]. These systems can provide 3D environments so that the focal point and convergence of the eyes are at the same distance, which is more comfortable for the user. The haptic device and the user’s hand do not occlude the 3D graphics behind them, but rather the real and virtual environments are superimposed and semitransparent due to
3.1 Electromagnetic Actuation The system produces haptic feedback from an array of 16 electrical coils which generate forces and torques on a set of 3 magnets embedded in the end of the user handle. The forces and torques produced on each magnet from each coil are proportional to the coil current, provided that the coil cores are non-ferrous. The forces and torques are highly nonlinear with respect to magnet position and orientation however, and cannot be calculated in real time during haptic interaction, which typically has an update rate near 1000 Hz for stability and realism. A precomputed 6-dof interpolated lookup table model of the forces and torques between a single magnet and coil as functions of position and orientation is used to calculate coil currents to produce desired haptic forces and torques. The forces and torques between each magnet and each coil as a function of coil current are combined into a matrix transformation at each feedback update. The pseudoinverse of this transformation matrix is then used to find the least-squares set of coil currents which generates the required force and torque at each update. Design and control methods for desired force and torque generation between any sets of magnets and coils are described in further detail in [3]. Magnetic levitation systems have previously been developed using these methods for 5 degree-of-freedom controlled levitation of a single disk magnet [2], and 6 degree-of-freedom controlled levitation of a two-magnet platform over large ranges of translation and rotation [4]. The magnet and coil parameters used in this system are as follows: Magnets: Diameter 25.4 mm Thickness 9.5 mm Material N52 NdFeB Figure 2: Co-located haptic and graphic display in use
Coils:
Inner Diameter Outer Diameter Thickness Windings
12.7 mm 25.4 mm 30 mm 1000
the half-silvered mirror, which may be a distraction to the user. NTT DoCoMo has also demonstrated a system which uses an electromagnetic coil behind a 3D display to generate a haptic force on a magnetic stylus held by the user. The only haptic feedback force generated was magnetic attraction to a single fixed point however. Other systems have been developed to provide colocated haptic feedback directly in front of a screen using built-in mechanisms [13], a cable-driven pen [7], or a linear induction motor [10]. These systems can only provide planar haptic feedback in the plane of the screen or in fixed locations, whereas our system described here can produce full rigid-body force and torque haptic feedback in 6 degrees of freedom. User studies [12, 6] have shown convincing benefits from haptic and graphic co-location in interactive task-based environments. 3
S YSTEM D ESCRIPTION
The present prototype system is as shown in Figures 1 and 2. For a simple, static environment, the virtual environment simulation and graphical rendering can be executed on the single PC performing the position sensing, control, and electromagnetic actuation calculations. More detailed environments and more complex simulation physics can execute on a separate PC, with a dedicated communication link to the haptic device controller. The position tracking system (Northern Digital OptoTrak Certus) can locate 3 infrared LED position markers at 0.01 mm resolution at a 1000 Hz update rate to track rigid-body motion sufficient for stable levitation and haptic interaction. The LED position markers can be used without a wired connection to the position sensing system, but the additional mass and bulk of the battery and electronics required in wireless mode would be much more cumbersome than the lightweight wiring used.
Figure 3: Model of user handle
A detailed model of the 3-magnet tool handle is shown in Figure 3. The three cylindrical magnets touch at their edges and their central axes are mutually perpendicular. The three magnet axes intersect the axis of the handle at a single point, at an approximately 55 degree angle to the handle axis. Three infrared LEDs are mounted on the back end of the handle for the motion tracking system. The Radia electromagnetic analysis software package [5] from the European Sychrotron Radiation Facility was used to obtain the single coil and magnet actuation model. Force and torque in x, y, and z directions produced from a 1 Ampere current were calculated over ranges of 10-50 mm vertical and 0-75 mm horizontal separations between the coil and magnet, sampled at 1 mm intervals, and
for all magnet tilt (altitude) and tilt direction (azimuth) angles sampled at 20 degree intervals. A small subset of the single coil and magnet actuation model data calculated is shown in Figures 4 and 5. The coils are arranged in a 4x4 array with their centers spaced 35 mm apart in the 0, 60, and 120 degree directions, producing a usable actuation area of approximately 100x120 mm, up to a height of 30-40 mm above the tops of the coils. The transformation from the coil currents to force and torque must be invertible and wellconditioned for any tool position and orientation within the motion range, as the pseudoinverse of the transformation is used to calculate the coil currents for the required feedback forces and torques [1]. The condition numbers of these transformation matrices calculated over 1 mm intervals across a single horizontal plane with the tool at a fixed height and with the tool handle in the vertical orientation are shown in Figure 6. Previous magnetic levitation systems using the same methods have produced adequate performance with typical condition numbers up to the 15-20 range, indicating that feedback forces and torques can be produced in all directions without requiring excessive coil currents. 3.2 3D Display The 3D display of our prototype system is provided by an NVIDIA Quadro 4000 graphics processor, the NVIDIA 3D Vision kit with active shutter glasses and an infrared synchronization emitter, and Linux software drivers for OpenGL graphics. The present system requires a fixed viewing position, as a head tracking system would otherwise be required to maintain correspondence between the haptic device and the 3D graphical environment if the user observes the display from different viewpoints. A disassembled flat panel monitor is being used for the display. The LCD panel is approximately 2 mm and the backlight assembly approximately 13 mm thick, so that the separation between the haptic device coils and magnets cannot be less than 15 mm, limiting the achievable haptic forces and torques without overheating the coils. Therefore we are seeking to replace the current display backlight assembly with a thinner one, to increase the achievable haptic forces and torques. Any user discomfort due to the mismatch between eye convergence and focal distances in 3D displays is minimal with this system, as virtual objects are displayed close to the plane of the screen, directly behind the actual user tool handle. The magnetic fields of approximately 1 Tesla from the magnet assemblies were not found to interfere with the function of the display in any way, and no disturbances to magnetic levitation control were observed from any attraction to ferrous materials in the LCD panel or backlight. 4
Figure 4: Single magnet and coil force and torque, with 1 Ampere current and magnet at 25 mm height and 20 degree tilt, with variable tilt direction and horizontal offset
P RELIMINARY R ESULTS
The present system can levitate the tool handle for only a few seconds at a time if it is not grasped by the user due to rapid overheating of the actuation coils. A previous two-magnet design could levitate for several minutes without overheating, however, due to larger magnets and less vertical separation from the coils. The system can generate haptic feedback forces of approximately 1.5 Newtons and 0.2 Newton-meters on the 200 g handle for long periods and peak forces up to 5 Newtons, which is adequate for realistic haptic interaction with simulated environments. To demonstrate and test the use of the system for haptic and graphic interaction, a very simple dynamic environment simulation with Newtonian physics, collision detection, and contact forces and torques from rigid-body interpenetration was programmed to execute on the haptic control PC in real time. This environment consisted of a single spherical object with a 10 mm radius free to move horizontally on a 100x120 mm plane with perfectly elastic boundaries, and a 5x10x40 mm block-shaped virtual extension of the haptic tool handle, so that the spherical ball can interact with
Figure 5: Single magnet and coil force and torque, with 1 Ampere current and magnet at 25 mm height and 60 degree tilt, with variable tilt direction and horizontal offset
and be manipulated by the tool. The contact stiffness was set to 0.2 N/mm. A damping gain was added to the haptic feedback force and torque on the handle and a small amount of Coulomb friction was added to the motion of the ball in the virtual environment to improve the realism and stability of the haptic system. The graphical environment is visible in the display underneath the haptic tool handle in Figure 2. The graphical display and the coil array can easily be aligned by hand provided that the displayed graphics are appropriately scaled, however it would be straightforward to register the haptic and graphical spaces automatically through the use of additional LED markers attached to the corners of the display panel. Tool position and the generated contact force feedback results while striking and pushing the ball with the block tool are given in Figure 7. Contact forces are zero except when the block is pushed into the ball by the user, when the contact forces are proportional to interpenetration between the two shapes.
Figure 6: Coil current to force and torque transformation matrix condition numbers for 3-magnet 16-coil system, with vertical handle orientation at 15 mm height and variable x and y horizontal position
x y z
Haptic Tool Position (mm)
120 100 80 60 40 20 0 0
Haptic Feedback Force (N)
5.0
5
10
15 fx f y fz
2.5
0.0
6 C ONCLUSION The basic concept and design of a novel system for co-located haptic and graphic interaction was described. The advantages of this system are that there is no additional linkage mechanism between the haptic handle and the displayed environment, no ghost images due to the use of half-silvered mirrors, and no user discomfort due to significant differences between convergence and focal distances for the display of 3D environments. Haptic feedback was demonstrated for simple dynamic environments and work is in progress to improve the haptic performance and the detail and sophistication of the simulated environments. ACKNOWLEDGEMENTS This work was supported in part by National Science Foundation grants IIS-0846172 and CNS-0551515. Sebastian Bozlee provided graphical simulation and device interface programming assistance.
−2.5
−5.0 0
5 R ESEARCH P LANS We are currently seeking to improve and optimize the design of the electromagnetic actuation system, to obtain greater forces and torques with less massive and more compact magnet assemblies on the user tool handle. A head tracking system can be implemented using the present optical motion tracker and additional position markers, to allow the user to move around and observe the 3D displayed environment from different directions and see around the tool handle held during haptic and graphic interaction. We aim to integrate the described graphic and haptic interaction system with more detailed environments and more sophisticated physics engines, incorporating friction, texture, and deformable soft bodies. Work is currently in progress to develop complex detailed deformable body simulations to be executed at realtime haptic rates by the graphical processing unit (GPU) of the PC. The ergonomics of our system are well adapted to simulation of medical procedures: The haptic tool is in the shape of an instrument handle and the graphical display is horizontal, so it is well suited for simulations of medical procedures on a patient lying on a table. When the system is improved to a sufficient degree of realism, performance, and reliability, we will conduct user studies to validate and quantify any improvements in perception and/or task performance due to the co-located haptics and graphics provided by our system.
5
Time (sec)
10
15
Figure 7: Tool handle position and force during electromagnetic colocated haptic and graphic interaction
R EFERENCES [1] J. Baillieul. A constraint oriented approach to inverse problems for kinematically redundant manipulators. In IEEE International Conference on Robotics and Automation, pages 1827–1833, Raleigh, March 1987. [2] P. Berkelman and M. Dzadovsky. Magnet levitation and trajectory following motion control using a planar array of cylindrical coils. In ASME Dynamic Systems and Control Conference, pages 767–774, Ann Arbor, October 2008. [3] P. Berkelman and M. Dzadovsky. Novel design, characterization, and control method for large motion range magnetic levitation. IEEE Magnetics Letters, 1, January 2010. [4] P. Berkelman and M. Dzadovsky. Magnetic levitation over large translation and rotation ranges in all directions. IEEE/ASME Transactions on Mechatronics, 2011. in press. [5] O. Chubar, P. Elleaume, and J. Chavanne. A three-dimensional magnetostatics computer code for insertion devices. Journal of Synchrotron Radiation, 5:481–484, 1998. [6] G. Jansson and M. Ostrom. The effects of co-location of visual and haptic space on judgements of form. In EuroHaptics, pages 516–519, Munich, June 2004. [7] L. Lin, Y. Wang, Y. Liu, and M. Sato. Application of pen-based planar haptic interface in physics education. In International Conference on Computer-Aided Design and Computer Graphics, pages 375–378,
September 2011. [8] C. Luciano, P. Bannerjee, L. Florea, and G. Dawe. Design of the ImmersiveTouch: a high-performance haptic augmented virtual reality system. In 11th International Conference on Human-Computer Interaction, Las Vegas, August 2005. [9] T. Massie and K. Salisbury. The PHANToM haptic interface: A device for probing virtual objects. In Proceedings of the ASME Winter Annual Meeting, Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, Chicago, Illinois, November 1994. [10] H. Noma, S. Yoshida, Y. Yanagida, and N. Tetsutani. The proactive desk: A new haptic display system for a digital desk using a 2-DOF linear induction motor. Presence: Teleoperators and Virtual Environments, 13(2):146–163, April 2004. [11] Novint Technologies Inc. Novint Falcon User Manual, 2007. [12] D. Swapp, V. Pawar, and C. Loscos. Interaction with haptic feedback and co-location in virtual reality. Presence, 10(1):24–30, April 2006. [13] C. Swindells, M. Enriquez, K. MacLeand, and K. Booth. Co-locating haptic and graphic feedback in manual controls. Technical Report TR2005-28, Computer Science, University of British Columbia, 2005. [14] Y. Yokokoji, R. Hollis, and T. Kanade. WYSIWYF display: A visual/haptic interface to virtual environment. Presence, 8(4):412–434, August 1999.
