Modeling and Control Aspects of Long Distance Telesurgical Applications Tam´as Haidegger, Levente Kov´acs, Stefan Preitl, Radu-Emil Precup, Adalbert Kov´acs, Bal´azs Beny´o and Zolt´an Beny´o Abstract— There is an increasing need to support telemedicine with advanced technology. Currently, long distance telesurgical systems are only used on Earth for research purposes, but the military and the space agencies are interested in the spatial extension of effective surgical treatment through remote teleoperation. One of the major limitations of future systems is the communication lag time. In the case of the envisioned human exploratory space missions to Moon and Mars, the signal latency can be several seconds or even minutes. This paper deals with theoretical and practical aspects of the problem. Modeling approaches are discussed, proposing simplified human and machine representations to accommodate long distance telesurgical applications. Further, classical control methods are presented based on cascade control, focusing on teleoperation. A suitable controller can be designed with an empirical controller of the inner loop, based on Kessler methods, while the outer loop can be based on predictive control.
I. INTRODUCTION Effective teleoperation has been a keen interest for many applications from industrial manufacturing to military campaigns. Robots have been able to explore remote planets, hazardous sites, the depth of the oceans and many other places, relying mainly on the control of a distant human operator. Different control schemes have been developed and tested to facilitate the guidance of these devices, to enhance telepresence and to deal with latency from passivity based control to wave variables and soft computing methods [1]. Technology enabled telepresence has great scientific and commercial potential in health care as well. The new field— telemedicine— is defined as “The use of medical information exchanged from one site to another via electronic communications for the health and education of the patient or healthcare provider and for the purpose of improving patient care. Telemedicine includes consultative, diagnostic and treatment services.”[2] The advantages of telemedicine are various: in the case of short-distance operations, the technology involved can mean great added value, such as an externally controlled tool holder or surgical robot [3]. In long distance telementoring, the time/cost effectiveness T. Haidegger, L. Kov´acs, B. Beny´o and Z. Beny´o are with Dept. of Control Engineering and Information Technology, S. Preitl and R. E. Precup are with ”Politehnica” University of Timisoara, Dept. of Automation and Applied Informatics, Timisoara, Romania, A. Kov´acs is with ”Politehnica” University of Timisoara, Dept. of Mathematics, Timisoara, Romania {haidegger, lkovacs, bbenyo, benyo}@iit.bme.hu, {spreitl, rprecup}@aut.utt.ro,
[email protected] This work was supported by the National Office for Research and Technology (NKTH), Hungarian National Scientific Research Foundation grant OTKA CK80316.
and the provided higher level of medical care are the most important benefits, while in extreme telemedicine, such as space exploration it may be the only available form of adequate medical aid. Telemedicine can be broken down to three main categories based on the timing and synchrony of the connection. Storeand-forward telemedicine means there is only one way communication at a time, the remote physician evaluate medical information offline, and sends it back to the original site at another time. Next, remote monitoring enables medical professionals to collect information about patients with different modality sensors from a distance. Finally, interactive telepresence services provide real-time communication between the two sites, which might be extended with different forms of interactions, achieving real teleoperation, including telesurgery. The latter category can be applied to telesurgery, allowing physicians to invasively treat patients geologically separated from themselves. Instant and unlimited remote access to the medical site means that the physician can actually be capable of performing operations through robots and other teleoperated devices. Fig. 1 shows the integration of different modalities to the control diagram of telesurgery concept [4]. Currently, the dominant form of sensory feedback is visual, as that provides the highest density of information. To ensure smooth control of the remote system, advanced control schemes are required, as described later in this paper. When the connection is not reliable enough, or the technical tools are not given, a remote surgeon can direct the local one based on almost real-time video and voice feed from the operating room. This technique is called telementoring, the spatial extension of classical mentoring and professional
Fig. 1. Integration of different modality feedback information in telesurgery. Interaction is only possible through a system of sensors and human–machine interfaces. (Modified from [4].)
guidance. Further, when communication quality or latency does not even allow semi-real-time connection, consultancy telemedicine (or telehealth consultancy) can be used. This only requires limited access to the remote site, however, as a result, the distant group cannot use real-time services or information updates. Telemedicine and especially telementoring have been widely used around the world. The aim of the paper is to give a general idea of long distance telesurgical applications and to present the most important problems to be solved by an automatic controller to ensure the high-quality health care support of remote units. The structure of the paper is the followng: Section II gives a general view of telesurgery problems for space applications, Section III and IV presents a human and a robot model for teleoperation scenarios, respectively and Section V discusses classical control possibilities, proposing a cascade control structure for a telesurgical robot. Finally, conclusions are drawn at the end of the paper.
Fig. 2. Concept of telehealth support to provide maximum level of available medical care to astronauts during interplanetary exploration missions. (Reprinted with permission from [5].)
II. TELEMEDICINE AND TELESURGERY FOR SPACE APPLICATIONS While advanced internet-based communication enables telesurgery all over the Earth, serious technological problems arise in the case of long-haul space exploration missions. Currently, it seems inevitable to have a flight surgeon on board of the spacecraft, as robots do not have enough autonomy to adapt to any unforeseeable events. To cope with the difficulties of endoscopic surgery in weightlessness and complex system requirements, a three-layered mission architecture has been proposed earlier by the authors to achieve the highest degree of performance possible by combining robotic and human surgery [5]. Depending on the physical distance between the spacecraft and the ground control center, different telepresence technologies may provide the best performance (Fig. 2). Basically, with the accession of latency, real-time control strategies and communication techniques’ effectiveness decreases significantly. Mainly within the range of 380,000 km (the Earth–Moon average distance), pseudo real-time telesurgery techniques can be used to provide medical support in the case of emergency. Leaving the Earth orbit, special control engineering algorithms have to be applied (e.g., virtual coupling of the remote environment [6], predictive displays projecting the intended motion of the tools ahead in time [7]) to extend the feasibility of telesurgery up to a maximum of 2 s of delay. With robot assisted surgery, a shared control approach should be followed, integrating high-fidelity automated functions into the robot to extend the capabilities of the human surgeon through image processing and force sensing. This concept could be most beneficial for long duration onorbit missions, primarily on board of the International Space Station (ISS). Presently, there is no other option than the immediate evacuation of the affected astronaut, which poses bigger health risk and huge costs. Teleoperation controller design has a huge role in providing the high quality control signals and sensory feedback to facilitate the surgery over the time delay network.
Flying further from the Earth, and reaching the limits of pseudo real-time communication, the procedures should be performed by the flight surgeon under the detailed telementoring guidance of master surgeons on the ground. Telementoring transforms into consultancy telemedicine above a certain signal delay. Around one minute of delay, it is inconvenient and impractical for the crew to wait for the guidance of the ground after every procedural step, and in some cases, it would endanger the success of the operation. For these missions, the flight surgeon must be trained to conduct the operation and make decisions on its own. However, the ground staff can provide high value support by patient specific simulations and thorough consultancy beforehand the operation. III. HUMAN MODEL FOR TELEOPERATION SCENARIOS A. Human operator models In order to design an optimal control scheme for the teleoperation scenarios introduced above, it is necessary to derive the applicable model of the human operator (master) and the robot (slave). Teleoperation through a human–machine interface based on visual information has been shown to correlate to an “equalization type” or “look and move” task, where the human subject needs to reach to a prescribed point in space. In the 1960s, several research projects were undertaken to develop accurate models of human pilots [8]. The results revealed that while the human body shows a variety of nonlinear and time-varying behavior, it is possible to determine a quasi-linear equivalent model [9]. It has been shown in human experiments already in the ’60s that people can adapt to latency through a “move and wait” strategy [10]. This means, the operator stops while waiting the round-trip delay time for sensory confirmation of the performed action. It is possible to apply a small, incremental position change open loop (without feedback),
Fig. 3. Simplified model of a human from the control aspect, in the case of a simple tracking (equalization type) task performed in teleoperation mode through a delayed channel. (Modified from [8].)
however, the larger the latency, the slower the movement gets, with a bigger inherent risk of causing some damage due to the prolongation of the unmonitored periods. Experiments clearly supported the prior assumptions that teleoperation task performance is a direct function of the lag time, the ratio of the movement’s length/accuracy and some other minor aspects [9]. While this strategy has successfully been applied to e.g., exploration rovers, it has serious limitations in the case of telesurgery, where processes are time-critical, and prompt reaction is required. Therefore the human operator should be supported with augmented reality, predictive displays, model based predictive control and similar technologies. B. The crossover model The physiology driven block scheme of the human control—originally developed for pilots—is given in Fig. 3. The different aspects of the human sensory system’s behavior (retinal excitation, nerve conduction, processing in the central nervous system, etc.) have been aggregated to lumped equivalent perceptual time delay τ . The H(·) transfer function (t.f.) represents the human operator’s attempts to optimize its control strategy. Naturally, there are other remnant terms that determines the general behavior of a human operator, however, these may be better represented in a statistical manner, and will be omitted from further considerations due to their minor effect on the overall motion. Observation and task execution (motor) noises are also neglected [11]. The generally used form with a first-order neuromuscular lag time model (containing simplifications and neglecting high frequency terms) is: WP = k p
(τL s + 1)e−sτ , (τI s + 1)(τN s + 1)
(1)
where kp is the pilot’s (the operator in more general) static gain, the e−sτ term reflects the pure time delay caused by the human sensory system limitations, τL is the lead time constant (relative rate-to-displacement sensitivity), τI is
the lag time constant, τN the neuromuscular and activation mechanism lag time. The remaining terms represent the human’s equalization characteristics. They are adjustable according to the task requirements and are chosen such that the closed-loop characteristics will approximate those of a good feedback control system [12]. In many cases, the amplitude ratio data is best approximated with (1), called the crossover model. The open loop t.f. with one pole can be derived in the form: ωC −sτD , (2) e GP = kp s where kp is the static gain, ωC is the crossover frequency (meaning the limitation of the human operator’s reaction based on the information feedback) and τD is the delay between the observation and the reaction of the motor system [13]. With zero time delay, (2) yields to Fitts’ law [14]. This is a widely accepted model that describes the time taken to acquire a visual target using some kind of manual input device. In the most generic (Shannon) form, the average time—T —taken to complete the movement is: D , (3) T = a + b log2 1 + W where a represents the start time, b stands for the inherent speed of the device, D is the distance from the starting point to the center of the target and W is the width of the target measured along the axis of motion (practically, the allowable error). The crossover model leads to the following form of Fitts’ law, determining the task completion time for moving an object over a distance D and placing it within the W/2 proximity of the target 1 D TC = . (4) ln 1 + ωC W More recent experiments explicitly showed the validity of Fitts’ law in the case of da Vinci type telerobotic surgery [15]. As the delay increases, the exponential motion becomes oscillatory, therefore looses stability. This also leads to the increase of the overall completion time, and should be avoided. The simple model can be extended with additional terms representing short and long term human learning [13]. More realistically, in the case of unknown delays, human reaction is a function of: ∗ τD = τD0 + (τD − τD0 ) e−t/τlt ,
(5)
where τD0 represents the time it takes to the human operator ∗ to learn the amount of delay, τD is the assumed delay and τlt (τD , N ) is the effect of long term learning over N repetitions of a task. It might be possible to acquire the proper τlt parameters in a simulated experiment, while it is a complex function of personal skills and experience in the case of a complex, real-life surgery. The effect of learning has been shown to follow an exponential pattern. The human operator’s kp gain is also affected by learning over time: ∗ KD = KD0 + (KD − KD0 ) e−t/τlt .
(6)
IV. ROBOT MODEL FOR TELEOPERATION SCENARIOS It can be assumed that the robot is a series of rigid links, with general mechanical properties, and the servos are directed by the local robot controller according to the control commands from the master side. In telesurgery, it is desirable to minimize the load to the patient’s tissue, therefore force control may be used. It is common to model the impedance characteristics of both the master and the slave devices separately; even though, the master controllers are very sophisticated and compliant nowadays. A simple dynamic model of the manipulator is: FR = M R x ¨R + BR x˙ R ,
(7)
where MR and BR are the inertia and damping coefficients of the robot, respectively, FR is the force applied to the manipulator and xR is the displacement in position [16]. This form is suitable to describe a force-controlled robot, where the commands to the local robot controller are given as position (velocity) increments. It may be adequate to incorporate the deviation of the tool from the master controller’s position. This yields to the extension of (7): FR = KS (xS (t) − xM (t − Tlat )) + BS (x˙ S (t) − x˙ M (t − Tlat )) + MS x ¨S (t),
(8)
where xS is the Cartesian position of the slave, xM is the Cartesian position of the master, KS is the stiffness of the slave manipulator and Tlat is the latency of the communication network [17]. Tissue characteristics are considered through the following equation FT = p eqxS (t) − 1 , (9) where p and q are constant, tissue-specific parameters. This represents the exponential resistance of soft tissue towards external forces applied on it. Notation G(s) will be used to refer to the linearized, frequency domain equivalent of (9), therefore, the slave robot can be modeled with t.f. WR =
(KS + BS s) G(s) . s (MS s2 + BS s + KS + G(s))
Fig. 4. Model of a teleoperated slave robot with an Observer to determine tissue force reactions. (Modified from [17].)
A. Cascade controller for a telesurgical robot Out of the many criteria towards telesurgical systems, stability and transparency have been considered to be the most important. Due to the fact that the robot–human–sensor triplet can be well delimited an adequate method for control could be the design of a cascade controller. A realistic teleoperation system—as it was presented in Section III— suffers from time delays during communications between the master (controller) and slave side (effector system). Unless the process is significantly slower than the latency, the control lag time can cause the deterioration of the control quality and even general instability due to unwanted power generation in the communications. Time-varying delay poses further difficulty to classical PID controllers. Cascade control can improve control system performance over single-loop control whenever disturbances affect a measurable intermediate/secondary process output that directly affects the primary process output, or if the gain of the secondary process (including the actuator) is nonlinear. Advantages of cascade control have been widely studied and published [18]: •
(10)
•
This is shown in Fig. 4, together with an observer to determine FT . Deviation originating from the physical realization of the robot’s mechanical structure (imperfections and frictions) have been omitted from the model.
•
V. CONTROLLER DESIGN SOLUTIONS FOR LONG DISTANCE TELESURGICAL APPLICATIONS Based on the above discussed problems, the pre-control context can be well defined, hence, different control methods can be formulated. In the current paper the authors focus only on classical control options, while in the future, other algorithms like soft-computing or modern control paradigms may also be investigated.
allow faster secondary controller to handle disturbances in the secondary loop; allow secondary controller to handle complex non-linear problems; allow operator to directly control secondary loop during certain modes of operation (such as startup).
Requirements for cascade control can be summarized as follows: • • •
secondary loop process dynamics must be faster than primary loop process dynamics; secondary loop must have influence on the primary loop; secondary loop must be observable and controllable.
A possible cascade controller scheme for a telesurgical application is illustrated in Fig. 5. It can be seen that the above listed requirements are all satisfied [19].
Fig. 5.
Cascade controller block diagram for a telerobotic space exploration application [19].
B. Empirical design approach—Kessler methods Based on (3) it can be seen that the inner part of the cascade control scheme (robot) can be described in a simple form. However, it is well known that empirical methods can provide a solution for the class of plants characterized by t.f. [20]: kp HP (s) = , (11) s (1 + sT1 ) (1 + sTΣ ) or kp HP (s) = , (12) s (1 + sT1 ) (1 + sT2 ) (1 + sTΣ ) where TΣ is a small time constant or aggregated time constant corresponding to the sum of parasitic time constants, TΣ < T2 < T1 ). The use of a PI or PID controller having the t.f.: kc HC (s) = (1 + sTC ) (1 + sTC0 ) (13) s can ensure acceptable control system performance [21]. TΣ can also include the time constants used to approximate the time delay. In (12), the process pole (p1 = −1/T1 ) may be compensated by controller zero (z1 = −1/TC0 ) in order to obtain the desired open-loop t.f.—H0 (s)—in the form: H0 (s) = HC (s)HP (s) =
k0 (1 + sTC ) , s2 (1 + sTΣ )
(14)
with k0 . However, in certain practical applications the performance—that can be specified a priori for a given process—is rather unacceptable [20]. Therefore, the t.f. of the control system can be expressed as: HW (s) =
H0 (s) b0 + b1 s = , 1 + H0 (s) a0 + a1 s + a2 s2 + a3 s3
(15)
with b0 = a0 and b1 = a1 , due to the expression of H0 (s). For the tuning of controller parameters kc and TC (TC0 = T1 through compensation) the literature recommends the Modulus Criteria (CM) or Symmetrical Optimum (SO) method introduced by Kessler [22] that proves to be advantageous in practice because it provides well established tuning relations. CM can be applied for plants with t.f given by (12) (with no integrator term). SO may be used for plant described in the form (11). However, the performance of Kessler methods become unacceptable due to a large sensitivity with respect to the modification of kp accompanied by an alleviation of the phase margin φr . This shortcoming can be much stronger if TΣ corresponds to the sum of parasitic time constants. To compensate for the above mentioned limitations, extensions of the Kessler methods were proposed in the literature, both for the SO method [23] and for the CM [24]. These newer methods can be well used in a teleoperational scenario, as they are fast and reliable. C. Predictive control In the case of the control methods described above, the information of the inner loop gives feedback to the outer loop, but no a priori knowledge about the inner loop’s dynamics is required to design of the outer controller. On the other hand, it is possible to explicitly consider the remote dynamics into the outer controller in order to predict the inner behavior [25]. This can be based on the well-known Smith predictor scheme [18]. Smith predictor can be used in the outer loop in order to anticipate computation of the delayed information from the
inner part, whereas a simple controller is implemented at the inner part. This is a crucial and effective observation as in case of teleoperating a robot in a spacecraft delays appear. Hence, the cascade control scheme presented in Fig. 5— combined with predictive control—gives a suitable solution of the presented problem [19]. In this way, the human at the remote workstation may be able to control the operation by monitoring the predicted model of the robot. The control commands given to the robot-simulator are sent to the remote robot using time-delayed links. Consequently, the ground workstation contains a model of the uplink and the downlink delay lines as well as a model of the actual states of the real robot and its environment. Hence, several alternatives to superimpose the predicted robot model exist as presented in [19], such as the use of predictive video displays [26], or augmented sensory fusion solutions. This structure is suitable to later include more complex models of both the human (e.g. fuzzy control model [27], or optimal control model [28]) and the robot to acquire a complete representation of the whole telesurgical scenario. VI. CONCLUSION In the past two decades, the premature concept of telesurgical support over long distances gradually became feasible. Several experiments have been conducted to learn about effective human teleoperation. The paper gave an overview of the possible concepts of a human space flight support, analyzing the mission from the control point of view. One of the major challenges still remains communication lag time, therefore adequate control algorithm has to be chosen to compensate for the disturbing effects of latency. To achieve this, a simple model for teleoperator and the distant robot had to be derived first. Further, different control methods have been revised to determine an adequate solution for the control of a remote telesurgical robot. The proposed cascade loop relying on empirical methods and predictive control may be a good solution to support future teleoperational mission design. R EFERENCES [1] J. Cui, S. Tosunoglu, R. Roberts, C. Moore, and D. Repperger, “A review of teleoperation system control,” in Florida Conference on Recent Advances in Robotics, FCRAR, Boca Raton, Florida, 2003. [2] R. W. J. Pease, Ed., Medical Dictionary. USA: Merriam-Webster, 2003. [3] B. Herman, “On the Role of Three Dimensional Visualization for Surgical Applications in Interactive Human Machine Systems,” Master thesis, The Johns Hopkins University, 2005. [4] G. Hager, A. Okamura, P. Kazanzides, L. Whitcomb, G. Fichtinger, and R. Taylor, “Surgical and Interventional Robotics: part III,” IEEE Robotics & Automation Magazine, vol. 15, no. 4, pp. 84–93, 2008. [5] T. Haidegger and Z. Benyo, “Surgical robotic support for long duration space missions,” Acta Astronautica, vol. 63, no. 7-10, pp. 996–1005, 2008. [6] J. M. Thompson, M. P. Ottensmeyer, and T. B. Sheridan, “Human factors in telesurgery: effects of time delay and asynchrony in video and control feedback with local manipulative assistance.” Telemedicine Journal (American Telemedicine Association), vol. 5, no. 2, pp. 129–37, January 1999.
[7] R. Rayman, K. Croome, N. Galbraith, R. McClure, R. Morady, S. Peterson, S. Smith, V. Subotic, A. V. Wynsberghe, and S. Primak, “Long-distance robotic telesurgery: a feasibility study for care in remote environments ,” Int J Med Robotics Comput Assist Surg, no. 2, pp. 216–224, 2006. [8] D. McRuer, D. Graham, E. Krendel, and W. Reisener Jr, Human pilot dynamics in compensatory systems. (AFFDL-65-15), Wright Air Development Center, USAF., 1965. [9] S. Baron, “Application of the Optimal Control Model for the Human Operator to Reliability Assessment,” IEEE Transactions on Reliability, vol. R-22, no. 3, pp. 157–164, August 1973. [10] T. B. Sheridan, “Space teleoperation through time delay: review and prognosis,” IEEE Transactions on Robotics and Automation, vol. 9, no. 5, pp. 592–606, 1993. [11] D. Kleinman, S. Baron, and W. Levison, “An optimal control model of human response. I- Theory and validation,” Automatica, vol. 6, pp. 357–369, 1970. [12] D. McRuer and H. Jex, “A Review of Quasi-Linear Pilot Models,” IEEE Transactions on Human Factors in Electronics, vol. 8, no. 3, pp. 231–249, September 1967. [13] R. Rayman, S. Primak, R. Patel, M. Moallem, R. Morady, M. Tavakoli, V. Subotic, N. Galbraith, A. van Wynsberghe, and K. Croome, “Effects of latency on telesurgery: an experimental study,” Lecture Notes in Computer Science (LNCS), vol. 3750, p. 57, 2005. [14] P. M. Fitts, “The information capacity of the human motor system in controlling the amplitude of movement. 1954.” Journal of experimental psychology. General, vol. 121, no. 3, pp. 262–9, 1992. [15] J. H. Chien, M. M. Tiwari, I. H. Suh, D. Oleynikov, and K.-c. Siu, “Accuracy and speed trade-off in robot-assisted surgery,” Int J Med Robotics Comput Assist Surg, no. April, 2010. [16] S. D. Eppinger and W. P. Seering, “Introduction to Dynamic Models for Robot Force Control,” IEEE Control Systems Magazine, vol. 7, pp. 48–52, 1987. [17] K. Kawashima, K. Tadano, G. Sankaranarayanan, and B. Hannaford, “Model-based passivity control for bilateral teleoperation of a surgical robot with time delay,” 2008 IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, pp. 1427–1432, September 2008. [18] B. Lantos, Theory and design of control systems I-II., (in hungarian ed. Budapest: Akademia Press, 2001. [19] G. Hirzinger, J. Heindl, and K. Landzettel, “Predictive and knowledge-based telerobotic control concepts,” in Proceedings, 1989 International Conference on Robotics and Automation. IEEE Comput. Soc. Press, pp. 1768–1777. [20] S. Preitl, R.-E. Precup, L. Kovacs, and Z. Preitl, “Control Solutions for Electrical Driving Systems. Tuning Methodologies for PI and PID Controllers,” in Proc. of the Kand´o Conference 2002—60 years in Electrical Training. BMF, 2002. [21] K. Astrom and T. Hagglund, “PID controllers: theory, design, and tuning,” in Instrument Society of America, Research Triangle Park, 1995. [22] C. Kessler, “Das Symetrische Optimum,” Regelungstechnik, vol. 6, pp. 395–400, 432–436, 1958. [23] S. Preitl and R. Precup, “An extension of tuning relations after symmetrical optimum method for PI and PID controllers,” Automatica, vol. 35, no. 10, pp. 1731–1736, 1999. [24] D. Vrancic, S. Strmcnik, and J. Juricich, “A magnitude optimum multiple integration tuning method for filtered PID controller,” Automatica, vol. 37, no. 9, pp. 1473–1479, 2001. [25] P. Arcara and C. Melchiorri, “Control Schemes for Teleoperation With Time Delay: A Comparative Study,” Robotics and Autonomous Systems, vol. 38, pp. 49–64, 2002. [26] W. Kim and a.K. Bejczy, “Demonstration of a high-fidelity predictive/preview display technique for telerobotic servicing in space,” IEEE Transactions on Robotics and Automation, vol. 9, no. 5, pp. 698–702, 1993. [27] R. Hess, Human-in-the-loop Control. College Park: CRC Press, 1999, ch. 12. [28] A. Phatak, H. Weinert, I. Segall, and C. N. Day, “Identification of a modified optimal control model for the human operator,” Automatica, vol. 12, no. 1, pp. 31–41, January 1976.
