Interactive Force Sensing Feedback System for Remote Robotic Laparoscopic Surgery lan Mack Stuart Ferguson and Karen McMenemy
Stephen Potts
i\listair l)ick
Royal College of Surgeons in Ireland Dublin, Republic of Irelend
Royal Belfast Hospital for Siek Children Belfast, United Kingdom Email:
[email protected]
School of Electronics, Electrical Engineering and Computer Science The Queen's University of Belfast Belfast, United Kingdom Email:
[email protected] Email:
[email protected] Email:
[email protected]
Abstract-This paper presents hardware and software systems which have been developed to provide haptic feedback for teleoperated laparoscopic surgical robots. Surgical instruments incorporating quantum tunnelling composite force measuring sensors have been developed and mounted on a pair of Mitsubishi PA-IO industrial robots. Feedback forces are rendered on pseudosurgical instruments based on a pair of PHANTOM Omnis, which are also used to remotely manipulate the robotic arms. The paper describes the measurement of forces applied to surgical instruments during a teleoperated procedure, in order to provide a haptic feedback channel. This force feedback channel is combined with a visual feedback channel to enable a surgeon to better perform a two-handed surgical procedure on a remote patient by more accurately controlling a pair of robot arms via a computer network. I. INTRODUCTION
In open surgery a relatively long incision is made in the patient's body to allow the surgeon to have sufficient access to perform the desired procedure. The incision is made large enough to allow the use of various surgical instruments and to provide good visual feedback for the surgeon. The surgeon can also feel the tissue by either touching it directly, albeit with a gloved hand, or via the surgical instruments. In minimally invasive surgery (MIS) small incisions are made either side of the target tissue to allow long slender surgical instruments to be inserted via trocars into a carbon dioxide filled cavity. A camera and light source, an endoscope, is inserted through another trocar, and the surgeon performs the procedure by watehing the tips of the surgical instruments inside the body cavity via a video monitor. This requires excellent hand-eye coordination. However, with MIS the surgeon can only feel the patient's tissue by touching it with the long surgical instruments which also diminish the tactile feedback. In some spinal procedures a form of MIS is used where instruments are inserted into tissue through incisions, but no gas cavity is created. Instead the operation is observed by repeatedly pausing to take, or continuously taking, low power
978-1-4244-4218-8/09/$25.00 ©2009 ©2009 IEEE IEEE 978-1-4244-4218-8/09/$25.00
x-ray images of the target area. This could result in the surgeon receiving a higher dose of radiation than recommended by safety standards. In order to remove the surgeon from the hazardous x-ray environment this research proposes the use of teleoperated surgery. Teleoperated robots allow the surgeon to be remote from the patient during a procedure. The surgeon performs the operation by moving pseudo-surgical instruments which send position data over a computer network to remote robots with surgical instruments attached. The remote robots mimic the surgeon's hand movements. Visual feedback is obtained by watehing the remote scene via a video link. However, there is a serious limitation to this surgical method in that the surgeon cannot feel the forces exerted by the robots on the patient. In order to create a more realistic immersive experience for the surgeon and reduce the chance of iatrogenic injury to the patient, our research emphasis has been focused on creating a force feedback channel from the surgical instruments to the surgeon's hands. 11. FORCE SENSORS
The human body has five major senses, namely sight, touch, hearing, taste and smell. The senses take information from the surrounding environment and the interpretation of this information by the brain is known as human perception. In an ideal telerobotic surgery world the surgeon would be unaware of the fact that there was any physical separation between himself and the patient. Therefore, in order to create the perception of performing surgery directly on a patient the surgeon's senses have to be deceived into believing there is no patient/surgeon separation. The important senses for perception of the surrounding environment during a surgical procedure are vision, touch, smell and hearing. Feedback paths, or channels, for two of these senses, namely vision and touch, are implemented in this research.
A. Visual Feedback During an operation visual feedback (VF) provides data pertaining not only to the colour and shape of the tissue, but also to the velocities of objects with respect to the surgeon and to other objects. Visual feedback for the surgeon appears on a video monitor or computer screen. When performing surgery visual feedback provides the most complete set of tissue quality information compared to tactile and force feedback which only provide information about tissue features local to the surgical instruments [15]. A video camera attached to a computer views the remote robots and the resulting video is streamed over a network to a local PC which displays it on a monitor for the surgeon . B. Haptic Feedback
Haptic feedback relates to touch, and comprises both tactile feedback and force feedback information. Tactile feedback (TF) is perceived by cutaneous receptors in the skin which can sense, for example, texture or temperature. Force feedback (FF) is perceived by receptors in muscles, tendons, joints and skin through direct contact with an object [2]. It provides force, position and velocity information about objects . There is some crossover however, as vibration and stretching, for example , are a combined perception between force feedback and tactile feedback [3]. Haptic feedback is obtained from the skin, which is the sensory organ for touch, and the limbs, which have the sensors for force detection. Tactile and force feedback , being so closely linked, make efficient use of the human body's automated motor responses . This reduces hand-eye coordination errors by speeding up reaction times, as shown by Hale and Stanney [6]. After visual feedback , the most important type of feedback is haptic information returned from sensors on the remote robot to the local pseudo surgical instruments. One of the main disadvantages of the current generation of teleoperated master-slave systems is the lack of haptic feedback . The surgeon performing the procedure using remotely controlled robots is not able to feel the patient [5]. Lindeman et al [8] showed that the accuracy achieved when performing intricate user interface manipulation tasks could be greatly improved when feedback channels, such as haptic and visual, are combined. By using a combination of visual and force feedback the surgeon can assess tissue structure . For example, by applying force and observing the deformation of a tissue, the surgeon can estimate the elasticity of a tissue . It has been shown that the combination of visual and force feedback is more reliable than using only visual or force feedback alone to assess tissue elasticity [19]. Unlike visual and force feedback, tactile feedback is entirely lost with the use of laparoscopic surgical instruments. Thus the surgeon cannot make use of tactile feedback when performing a laparoscopic procedure. However, the insertion of a robot between the surgeon and the patient meant that there was a physical separation between the surgeon 's hands and the surgical instruments,
resulting in a loss of force feedback. At present there are no laparoscopic robots in clinical use which can provide force feedback from the patient-instrument interface to the surgeon [14]. The surgeon has only position and velocity information to help with the surgical procedure. When no force feedback is available to the surgeon there is a possible risk of damage to the patient's tissue [7]. As is the case with laparoscopic surgery, robotic minimally invasive surgery has no feedback channel for tactile feedback [17]. C. Force Measurement
In order to provide a force feedback channel the forces of interest have to be measured by sensors which rely on direct contact with tissue [4]. Force sensors have been developed incorporating Quantum Tunnelling Composite (QTC) pills which are 3.4mm x 3.4mm x Imm in size. QTC is a relatively new type of composite which is produced in the UK by Peratech Ltd [9]. When the composite is compressed, bent, twisted or stretched the resistance falls from 10 12 - 10 13 n to less than I n. QTC has been used in our earlier research into providing low-cost instruments for a virtual reality simulator [II] [10]. They have been used to measure forces in the NASA Robonaut glove [12]. They have also been used in a robotic teddy bear which interacts with humans as a therapeutic companion [16]. D. Force Rendering Locally, the forces have to be rendered for the surgeon, so that he can experience the forces on the surgical instruments attached to the teleoperated robots . Vibrating motors, like those in pagers, can be used to provide low-cost vibro-tactile feedback [I] . Force reflecting interfaces such as the PHANTOM Omni from Sensable Technologies provide useful feedback [18]. However, their utilization is often limited by their cost and the fact that it is difficult to customize them to fulfill a specific task. In this research a pair of PHANTOM Omnis have been modified with the addition of surgical instrument handles which are held in a framework which can be used to simulate laparoscopic surgery.
III. MECHANICAL D ESIGN OF INSTRUMENTS Surgical instruments have been designed for attachment to robotic arms. The instruments incorporate force sensors, and the force data is sent over a network to be rendered on two PHANTOM Omnis. From an early stage of the design process the QTC force sensors were mounted at the handle/robot end of the surgical instruments, and not at the patient/tool-tip end. If force sensors are mounted at the instrument tips the tips become rather bulky as it is very difficult to produce miniaturized sensor mechanisms. Mounting the sensors at the base of the instruments means that size restriction is considerably reduced . This has the advantage that the forces are measured
at a position where the surgeon 's hand would normally be holding the instrument. Additionally, this would simplify the encap sulation of the instruments, allowing them to be utilized in a real surgical procedure and sterilized after use. A QTC sensor is formed by sandwiching a QTC pill between two electrodes. The sensitivity was increased by using a domed electrode on one side of the pill with a flat plate electrode on the other side. The sensitivity of the QTC sensors was further increased by the use of a lever mechani sm which uses principl e of mechanical advantag e: . M echam calA dvan t age
Load
= E f f ort
Figure 2.
(I)
CAD model showing sensor tubes.
A. CAD Model of the Surgical Instrum ent
A CAD model of the surgical instrument was developed which incorporated fourteen sensors. Figure I shows a view of the instrument with all of the outer casings removed. On the left is the jaw open/close sensor, with the up/down/left/right sensor on the right of the figure on a shortened grey instrument shaft.
Figure 3.
Figure I .
CAD model showing robot end of surgical instrument.
CAD model showing positions of QTC sensors. Figure 4.
Two sensors are used to measure the opening and closing forces on the jaws of the instrument, which are controll ed by a servomotor. Also, torque forces around the axis of the main shaft are measured by four sensors. The complete sensor tubes are shown in Figure 2. Figure 3 shows the sensor tubes and servomotor in relation to the rigid backing plane and socket for attachm ent to the robot arm. The complete CAD model for the surgical instruments is shown in Figure 4. Once a design had been completed that fulfilled the functional requirements of the basic surgical instrum ent a working model was built for evaluation and testing before progressing to build the prototype surgical instrument. Figure 5 shows the working model of the surgical instrument. When the model proved the concept a prototyp e instrument was produc ed and this is shown in Figure 6. The surgical instrument incorporates the shaft and jaws from an actual medical instrument.
Figure 5.
CAD model of surgical instrument.
Working model of surgical instrument.
IV. EL ECTRO NIC S D ESI G N OF I NSTRUM E NTS
Each surgical instrument has fourteen QTC force sensors mounted at various points on the mechani sm. A Microchip PIC microcontroller with an integral USB interface is used to connect the QTC sensors to a PC. A/D converters are used to
algorithm must be used to derive the requ ired joint angle s from the position and orientation data. WeIman developed an inverse kinem atic algorithm called the Cyclic Coordinate Descent (CCD) method , which is quite simple to implement in real time [20]. An 2-D example of an iterativ e cycle to position the arm is shown in Figure 7, where the arm can be seen to move towards the target position.
T+.:).,...... .... Figure 6. The prototype surgical instrument.
mea sure the voltage s produced by the force s being exerted on QTC sandwich sensors mounted on the surgical instrum ent. The full y populated PCB has a minimal component count with only the microcontroller, crystal, capacitors , resistors and USB conn ector. Descriptors define the USB device as a COM port which prov ides a convenient way to input data to the PC via the Universal Serial Bus .
V. KINEMATIC CONTROL It is essential in robotics to continuously know the location of all relevant objects in 3-D worksp ace . The se objects are the sections or links of the robot ic arm, the surgical instrument attached to it, and any other objects in the robot 's work space . The location of an object can be defined by its position and orientation. These quantiti es must be represented in a form that allow s them to be mathematically manipulated.
.
2
-v .
'
,.j
....
Figure 7. Cyclic coordinate descent: Ist iteration. VII. SOFTWARE AND OVERALL CONTROL Communication between the local and remote sites is made via a standard computer network. Data is sent from servers and received by client s based on the virtual reality periph eral network (VRPN) libraries [13].
A. Robot Positioning At the local site the surgeon can perform two-h anded procedures by mo ving two PHANTOM Omni s which are connected to a PC by Firewire interfaces . A PHANTOM Omni is shown in Figure 8.
A. Forward Kinematics Kinematics is concerned with position, velocity and acceleration in geometrical and time-related motion . Robot arms con sist of almost rigid links connected by joints which enable adjoining links to move. Position sensors on the joints allow the relative positions of these adjoin ing sections to be measured. In the case of the rotary joints on the PA-IO robot these mea surements are called joint angle s. Seven independent joint angles need to be specified to position the PA-IO arm as it has 7 degrees of freedom . Forward kinematic s is the calculation of the position and orientation of the end-effector relative to the base fram e when given a set of joint angles. This is in effect a conversion from joint angles to Cartesian space.
B. Inverse Kinematics In order to position and orientate the end-effector relative to the fixed base fram e, it is necessary to calculate the joint paramet ers which will enable this pose to be achieved. This can be con sidered a conversion from Cartesian space to joint angles, and is known as Inverse kinematics. VI. POSITIONING USING CCD ALGORITHM The target pos ition and orientation of the surgical instrum ent s will be known from data received from the PHANTOM Omnis . However, in order to posit ion the robot the joint angles must be specified. Therefore, an inverse kinem atic
Figure
8.
A PHANTOM Omni.
The PHANTOMs are norm ally controlled by hold ing the pen and moving it to the desired position and orientation . However, this does not look or feel like a surgical instrument. A mechanism was therefore developed which allow ed the handles from real surgical instruments to operate the PHANTOMs, and this apparatus is shown in Figure 9. Three coordinates specify the position of a PHANTOM's pen and three angles specify its orientation. A VRPN server application on the PC strea ms this position data from the PHANTOMs. At the remote location a PC runs a VRPN client which is logged onto the local server. The client extracts the pos ition and orientation data , which is then used in the CCD algorithm to determine the angles required to position the
Figure 9.
PHANTOM Omnis with instrument handles attached .
robots with the force sensing surgical instruments attached. As the surgeon moves the instrument handles attached to the PHANTOMS so the surgical instruments attached to the remote robots mimic his actions . A robotic simulator was developed to ensure that the PHANTOMs would properly control the robots, and is shown in Figure 10.
Figure 10.
Robotic simulator.
Figure I I shows a pair of force sensing surgical instruments attached to the pair of Mitsubi shi PA-IO robots which were used in this research . B. Force Feedback At the remote site the surgical instruments can be moved to touch objects . The integral QTC force sensors are connected to another PC by the USB interfaces. A VRPN server application on the PC streams this force feedback data from the surgical instruments . At the local site a PC runs a VRPN client which is logged onto the remote server. The client extracts the force feedback data, which is then used by the PHANTOMs to render the feedback forces for the surgeon to experience. C. Visual Feedback
Similarly, a VRPN server streams video data from a webcam placed between the two robots to a local PC running a client which displays the video on a monitor. This enables the surgeon to view his actions as executed by the remote robots .
Figure II.
Instrument s attached to Mitsubishi PA-IO robots.
The remote camera has to be positioned carefully so that it produces a perspective which promotes intuitive operation by the surgeon. The research has reached the point where almost all of the systems have been tested individually. The video feedback server and client can display the scene at the remote site. The PHANTOMs, with their instrument handle mechani sm, can position the remote robots. The force sensing surgical instruments have been developed and a workshop version produced . The force feedback server and client have been tested. The CCO algorithm proved to be more reliable than the integral positioning algorithm when used over a period of time, as it did not allow the arm links to rotate in an incremental way and reach their end limits. The PA-IO robots have 7-00F and therefore do not have limited dexterity. Robotic surgery is restricted by size limitations. However the PA-IO with seven OOF has redundancy and therefore the elbow can be positioned to avoid causing an obstruction or being obstructed in the operating theatre . VIII. CONCLUSION
All of the systems have been tested individually prior to moving into user testing . The video feedback server and client can display the scene at the remote site. The PHANTOMs, with their instrument handle mechanism , can position the remote robots. The force sensing surgical instruments have been developed and a workshop version produced. The force feedback server and client have been tested. What remains to be completed is the integration of the force feedback from the surgical instruments and the force rendering on the PHANTOMS. The CCO algorithm proved to be more reliable than the integral positioning algorithm when used over a period of time, as it did not allow the arm links to rotate in an incremental way and reach their end limits.
The aim of this research has been to add a force feedback channe l to a teleoperated robotic surgical system which allows the surgeon to control the forces being applied to the remote patient , and at the same time having the benefit that the surgeon is no longer exposed to x-radiat ion. The two-handed robotic system allows the surgeon to perform intricate dextrous positioning movemen ts as though he was performing the procedure directly on the patient. Custom surgical instruments have been designed with integral QTC sensors to measure forces being applied to the surgical instruments. Prior to this researc h the measurement of forces on surgical instrumen ts using QTC had not previous ly been reported. Because the forces are not measured at the tips of the instruments, but at the handle end, shafts from actual surgical instrume nts can be used. This leaves open the possibi lity of encapsu lating the sensor end in latex and therefore specialized sterilization would not be required. Indeed disposable instrument shafts could be emp loyed, or alternative instruments could be selected during a procedure by the use of a robot tool changer. A CCD algorithm was developed to enable the positioning angles of the robots to be calculated . This proved to be a more reliable method than using the supplied algorithm . The CCD algorithm also kept the arm in a vertical plane which is more aesthetically pleasing to the human eye . Teleoperated robotic surgery, by the appropriate use of filtering, can have the advantage of reducing the hand shake of surgeons . Although this research has been aimed at surgery, it is equally valid in other fields. Teleoperated robots with force feedback could prove to be of value in bomb disposal, in nuclear plants handling radioactive waste, in underwater wreck exploration, controlling a telescope in space, or indeed a rover vehicle on another planet. R E F ER E NC ES [ I] L. T Cheng, R. Kazman and J. Rob inson, Vibrotactile feedback in delicate virtual rea lity operat ions, in Proceedings of the 1996 4th ACM International Multimedia Conf erence, pag es 243-251 , 1996. [2] J. C. Cra ig and G. B. Roll man , Som esthe sis, Ann ual Review of Psychology , 50 :305- 331 , 1999 . [3] A. Csi llag, Atlas of the Sensory Organs, Humana Press, Totowa , New Jersey, 2005 . [4] M. E. H. Eltaib and J. R. Hewit , Tactile sensing techn ology for minimal access surgery- a review, Mecha tronics, 13(10): 1163- 1177, 2003 . [5] I. Font, S. Weiland, M . Franken, M . Steinbuch and L. Rover s, Haptic feedback designs in teleoperation sys tems for mini ma l invasive surgery, in IEEE International Conference on Systems, Man and Cybernetics, volume 3, page s 2513-2518, 2004 . [6] K. S. Hal e and K. M . Stann cy, Deriving haptic design guidelines from human physiological, psychophysical , and neurological foundation s, IEEE Computer Graphics and Applications, 24(2) :33-39, 2004 . [7] M. Hashizume, M . Shim ada, M . Tornikawa , Y. Ikeda, I. Takahashi, R. Abe , F. Koga, N. Gotoh , K. Konishi, S. Maeh ara and K. Sugimachi , Early experienc es of endosco pic procedures in general surge ry assisted by a computer-enhanc ed surgical system, Surgical Endoscopy, 16(8):1 187-1191 ,2002. [8] R. Lind eman , J. Templ eman , J. Sibert and J. Cutl er, Handlin g of virtual cont act in immersive virtual environments: beyond visu als, Virtual reality, 6(3):130-139, 2002 .
[9] Perat ech Ltd ., Force Pressure Touch Sensing Solutions, 2008 , URL http : //www.peratech.com. [10] l. W. Mack , K. R. McM enem y, R. S. Ferguson , S. Potts and A. Dick , Measuring the Force s Applied to a Virtual Re ality Laparo scopic Surgical Simul ator with Qu antum Tunnell ing Composite Sensor s, in Proceedings of the First International Conference on Biomedical Electronics and Devices, page s 29- 34, 2008 . [II] I. W. Mack, S. Potts, K. R. McMenemy and R. S. Ferguson , Adding tact ile reali sm to a virtu al realit y laparo scopic surgical simulator with a cost -effective human interface device, in Stereoscopic Displays and Virtual Reality Systems XIII, volume 6055 , page 60551 , 2006 . [12] T. B. Martin , R. O. Ambrose , M . A. Diftl er, R. Platt Jr. and M . J. But zer, Tactile gloves for autonomous grasping with the NASA/DARPA Robonaut, in IEEE Internati onal Conference on Robo tics and Automati on, pages 1713-1718,2004. [13] University of North Carolina, Virtual Reality Peripheral Net work , 2009, URL http ://www.cs.unc.edu/Re search/vrpn/. [14] J. Rasswe iler, T. Frede , O. Seemann, C. Sto ck and L. Sentker, Telepresence surgery : first expe rience s with laparoscopic radica l pro statecto my, Minimally Invasive Therapy and Allied Technologies, 10(6):261-270, 2001. [15] M . Re iner, Seeing through touch: The role of haptic information in visualization, pages 73-84, Visualization: Theory and Practice in Science Edu cation, Springer, 2008 . [16] W. D. Stieh l, J. Liebe rman, C. Brea zeal , L. Basel, L. Lalla and M . Wolf , Design of a therapeutic robotic companion for relational, affective touch , in 14th IEEE Workshop on Robot and Human Interactive Communication. RO-MAN 2005, volume 2005 , pages 408-415 , 2005 . [17] M . Tavakoli , A. Azirninejad, R. V. Patel and M . Moallern , Methods and mechanisms for cont act feedb ack in a robot -assisted minimally invasive envi ronment, Surgical Endoscopy , 20(10): 1570-1579, 2006 . [18] SensAble Technologies, PHANTOM Omni Hapti c Device, 2009 , URL http ://www.sensable .com/haptic-phantom- omn i.hun. [19] G. Tholey, J. P. Desai and A . E. Castellan os, Force feedb ack pla ys a significant role in minimally invasive surgery: result s and analy sis, Annals of Surgery, 241( 1):102-109, 2005 . [20] C. Welman, Inverse Kinematics and Geom etric Constraints for Articulated Figure Manipulat ion , Ma ste r thesis, 1993.
