Via E ~}~: virtual contact. Vs E NJ: velocity set. Table 1: The explanation of the various variables in the teleoperation systems. literature [12]. In addition; the ...
Proceedings of the 2000 IEEE International Conference on Robotics & Automation San Francisco, CA • April 2000
R e a l - T i m e C o n t r o l of I n t e r n e t B a s e d T e l e o p e r a t i o n w i t h Force Reflection * Imad Elhajj and Ning Xi
Yun-hui Liu
Department of Electrical and Computer Engineering
Department of Mechanical and Automation Engineering The Chinese University of Hong Kong Shatin; N.T., Hong Kong
Michigan State University East Lansing, MI 48824, U.S.A.
Abstract The use of the Internet is no longer limited to the transmission of data. In the past few years many successful attempts have been made to use the Internet as a command transmission media; through which control can be sent to remote systems and feedback can be obtained. But with this media come several limitations: delay, lost packets and disconnection. All of these limitations may cause instability in the system especially if the system loop is closed. All the previous work addressing these problems assumed several conditions; for example, time delay is constant or has an upper bound, control is not in realtime. A new real-time control approach is presented that deals with these limitations without any assumptions made regarding delay. The approach is based on Event Based Control, which was implemented on a mobile robot over the Internet. The commands sent to the robot are velocity and the feedback is force based on the environment. It will be shown that this approach results in a stable system. In addition, a new force feedback generation method is used.
1
Introduction
The Internet has changed from a simple data transmission media, to a virtual world where you buy, sell, learn, explore and "control". Control over the Internet can be characterized by several properties and takes different forms. Regardless of the system used all realtime controlled systems over the Internet share many similar difficulties and advantages. A major characteristic and limitation of teleoperation is delay. Data Packets transmitted between 2 points will be delayed. This delay, although not desired, does not constitute *Research Partially supported under NSF Grant IIS-9796300 and IIS-9796287. 0-7803-5886-4/00/$10.00©
2000
IEEE
3284
a major problem for d a t a being passed. But when we start considering the network (Internet) as an action super-highway, via which we send control commands and receive sensory feedback in real-time, time delay becomes an important factor in the stability and efficiency of the system. Several Internet based robots have been developed and studied. Tarn and Brady [1] implemented a semiautonomous telerobot, where the teleoperator supplies a trajectory for the robot to execute. Then the operator intervenes only in case of unexpected circumstances. Pai [2] made an expensive experimental facility, the ACME, available via the Internet, where a user can conduct customized measurements and built accurate models from anywhere. Stein [3] had introduced a P u m a manipulator that can be controlled to paint over the Internet. Simmons [4] has an autonomous mobile robot (Xavier) that can be commanded to go to different locations and do different tasks. Other web robots have been introduced, and currently the NASA Space Telerobotics Program web site 1 lists more than 20 of them. However, except for visual information, these systems have no other form of sensory feedback. A very common form of sensory feedback is force reflection. This force information is obtained and may correspond to different environmental parameters. The most common way of generating this force information is using force/torque sensors, where the operator feels a force corresponding to the actual physical force the robot is subjected to. However, the force reflected can also be virtual force representing certain status of the remote robot. This reflection of force provides important haptic information to the operator which can significantly improve teleoperation [15]. This improvement is the result of coupling the operator with the environment. With force reflection the operator has a better feeling about the environment I htt p://ranier.oact.hq.nasa.gov/
and the status of the system. However, note that force feedback enhances the system but does not replace visual feedback. Many studies have been made relating to force reflection and force reflection under time delay in specific [5]-[9]. Each of these approaches has its limitations: delay is taken to be fixed and not random, or the same in both directions, or having an upper bound. These assumptions are not true when dealing with teleoperation over Internet. To the best of our knowledge, no experiments were done over the Interact only simulation on teleoperation with force reflection. The limitations of the existing approaches become critical when the Internet is used as a communication media. When it comes to the Internet no assumptions should be made regarding the time delay, because it can be random (No specific simple statistical model has been developed) and there is no upper bound on them. Because of the lack of simple models the inter-arrival times of packets can not be estimated. Therefore any assumption made about them would be limiting. The main effects of these delays are instability and desynchronization. The real challenge is to develop a new control method for the teleoperation via the Internet with force reflection. This paper describes a new approach for teleoperation with force feedback that overcomes several problems, especially those associated with delay.
2
Non-Time
Based
Teleoperation
Control
With
Force
vo() 4 v,GI
I
G
I
GI .......
Figure 1: Detailed block diagram of our teleoperation system. Block H u m a n Operator Master Communication Link Slave Environment
Our Variables Fh E ~3: applied force Xm E ~ : Joystick position Vm E ~ : velocity desired
Va E N3: Actual velocity Tm E N3: desired force Via E ~}~: virtual contact Vs E NJ: velocity set
Table 1: The explanation of the various variables in the teleoperation systems. literature [12]. In addition; the human can compensate for certain machine instabilities making the coupled human-machine system stable [14]. So once the operator feels a force, he will generate a new joystick position according to the following:
For
Xm(s) - F h ( S - 1)
(1)
IKh
Re-
where Kh is a constant, s is the event and s E {1,2,--.}. Xm and Fh are:
flection The instability and desynchronization in traditional control systems, which are caused by delays in communication links, result from the use of time as reference variable. Therefore, if a non-time based reference (an event) is used, the system would become immune to delay. The event-based controller design was first introduced by [10]. Then several studies and applications followed [11].
2.1
xo()
System Model
A detailed block diagram modeling the system is shown in Fig.1. Each block will be discussed in detail and all terms are explained in Table1. Human Operator: This was the most difficult to model, but eventually a spring-like behavior was assumed, shown in [13] and used in several places in the
3285
X (s) =
xm (8) Xme(S)
Fh(s) =
[ G(s)] G (s) Fhe(s)
(2)
As Table1 indicates Fh(s) is the applied force, which means the force that the operator feels. Thus, the x and y components are due to the force fed back and to the additional force required to get the joystick to the new location. Since force is not fed back in the direction, this component is just a result of getting the joystick to the next location. As seen in eq.1, Xm(s) is related to Fh(s -- 1), so Xm(s) at event s is generated by the previous force at event s - 1. This results in an event-based system where each event is triggered by the previous one. Master (Joystick):The dynamics of the joystick are:
M ~ V ~ m ( s + 1) = Fh(S) + Tm(S)
(3)
where Mm is the mass of the joystick handle, Vim is velocity of joystick movement, Fh is as described before and Tm is the feedback from the robot, which would be the force played by the joystick. The result from these dynamics is the joystick getting to a new position Xm(s + 1). From this position the desired velocity Vm is derived according to:
Vm (s) = K l X m (s)
(4)
Slave (Mobile robot): Once the robot receives Vs(s), it will be commanded to move with that velocity but would actually have Va(s) as its velocity. T h e n the robot will calculate the velocity tracking error with respect to original desired velocity Vm (s) and send that back to the master to be played as the virtual force due to the environment. So TIn(S) and the dynamics of the robot are:
~m(s) MsYo(s) ~.(8)
where Km is a scaling constant, Xm(s) is as before and Vm(s) is the desired velocity of the robot. The different vectors are:
v~(s) =
Vm,(s)
~(S) =
Vmo(s)
~,(S)
=
F0(s) +~s(s)
=
- T Y . ( s ) + KY~r~(s) - ~1r~(s)
Y.(s) - y~(s)
(S) (9) (10)
o
v~(s) = V~(s) - y~.(s)
(6)
where V~(s) is the velocity set to the robot, Vin(s) is the effect of the environment and Vm (s) as before. So now the desired velocity that the robot gets is less than the actual one, this slowing down will be used to generate the virtual force feedback. Keep in mind that no force is generated in the rotational direction t h a t is why Vino(S) : 0 and Vso(s) = Vlo(s), as seen in eq.7. v~(s) v~o(s)
g~(Ym(s) - Yo(s))
V~(s) =
(5)
From eq.5 we see t h a t TmO(S) = O, since force is not fed back in the rotational direction. Communication Block (Internet): Resulting from event based control, the communication link is simply a delay element that plays no role in the modeling of the system. It is assumed that the Internet is simply supplying the link and in case this connection is lost, the system will simply stop awaiting the connection. Since the advance of time does not affect the system and only the advance of the event s will, when the connection is lost the system will remain stable and will resume action after the connection is re-established. This makes the system very robust since no initialization or synchronization is required. Environment: Interaction with the environment was done in a new way. Contact is not required to generate force. Instead different sensors around the robot are used to detect objects. Based on the distance between the object and the robot a virtual force is generated. This is done according to a function of distance f(d), which is discussed in the next section, that would give a velocity value Vin(s) to subtract from the desired velocity Vm (s).
V.(s) =
=
y~.(s) =
yi~(s)
(7)
o
3286
Va(s) =
V~,(s)
(11)
Wins(s) K~, 7, K and a are constants, M8 is mass of robot,
Fe is the actual environment forces if any and usually assumed very small. Ts and V~r~ are robot internal controller terms. Eq.8 shows that what the operator is feeling is actually the velocity tracking error, which could be a result of the robot getting close to an object. This implies t h a t from the direction and magnitude of the force, the operator feels the location and the distance of the obstacle to the robot. The important point here is t h a t the operator will still feel force whenever the robot comes in contact with an object. In actual contact t h a t results in a decrease in Va(s) will result in a proportional increase in TIn(S). Moreover; we take Vao (s) = Vlo (s) since Tin0 = 0. 2.2
System
Stability
Several methods such as scattering theory [5] and wave variables [9]have been used to analyze the stability of teleoperation systems. The main concept used in these analysis is based on proving that the system is passive. A passive system can not create energy, and thus from a control point of view a passive system can not go unstable [5]. Our approach is slightly different because of the use of event based control. Concerning the stability of the system under event-based referencing, the following theorem was proven in [11]: T h e o r e m 1 If the original robot dynamic system
(without remote human~autonomous controller) is asymptotically stable with time t as its action reference; and the new non-time action reference, s : I ~ (y) is a (monotone increasing) nondecreasing function of time t, then the system is (asymptotically) stable with respect to the new action reference s.
F~
The only assumption needed is that the robot is a stable system, which means that the original robot dynamic system (without remote human operator) is asymptotically stable with t as its action reference. This would allow the use of T h e o r e m l and proves the (asymptotical) stability of the system with respect to the new action reference s, simply by proving that the new non-time action reference is (monotone increasing) nondecreasing function of time t. The advantage of this approach is that stability is proven independent of the human model or the statistics of time-delay. The major difficulty in selecting s was the many uncertainties in the system. The main uncertainties were the path (trajectory) and the environment. Usually the event s is taken to be a function of the physical output, for example, the distance to the origin, the angle, the absolute position. But, in teleoperation, typically none of these parameters are defined. In this case the selection of the event will be clear after the flow of commands and feedback on the system are explained. This is the first time this specific approach has been used for event-based teleoperation. This is how the control and feedback commands are communicated; the operator places the joystick in a certain position that corresponds to velocity vector. This vector is sent to the sensing unit on the robot. The Sensing unit scans the environment and based on the position of obstacles, the velocity is reduced and sent to the robot motors to be executed. The motors will execute the command, after t h a t the actual velocity is calculated. Then the actual velocity is subtracted from the original velocity required by the operator. This difference is sent back to the joystick motor to be played as force. The important point is that none of these steps can be executed out of order. Each event is triggered buy the end of the previous one. It is important to note that, although the operator could move the joystick, the commands will not be sampled by the joystick until feedback is received and played. Based on this control algorithm a natural selection of the event variable s was some kind of a counter. In our case, it is taken to be the number of forces felt by the operator since the connection was established.
3
Implementation
The implementation can be divided into two parts; hardware and software. As we see in Fig.2, the system consists of different operating systems and different configurations. So the problem of interconnection
3287
X~
V~
V~
V~
LkL___J Mobile Robot
Figure 2: Hardware structure of system.
had to be studied carefully. The joystick used is a programmable Microsoft SideWinder Force Feedback Pro. with 3 degrees of freedom. The mobile robot, called Arone, is a Nomadic XR4000. The software developed can be divided into two main parts: motion server, which runs on the robot, and client, which runs on the local machine. Motion Server: In this case the service is moving the robot and receiving feedback. However the server does not execute requests blindly; it first checks the sensors and based on that makes a decision according to an obstacle avoidance algorithm, which is discussed later. After the velocity to be set is decided, it is sent to the motors for execution. T h e n the server checks the actual velocity of the robot, subtracts it from the original desired one Vm, and sends it back to the client as force feedback. To overcome the problem of disconnection, the server would execute the velocities for 250ms; after which, if no new command is received, it would stop moving and wait for the next command. Client: The client sends commands to the motion server and relays force commands back to the joystick once feedback is received. Communication with the joystick is achieved with DirectX technology, and that with the server over the Internet. A client implementation "trick" was to delay the sampling of the next command, so when feedback is received and force played, the client would wait 250ms and then sample the position of the joystick. This way the operator is given time to feel the force and plan the next step. The obstacle avoidance and force generation algorithm is as follows: the different sensors on the robot are used to detect the distance to the closest object in the direction of motion. If a sensor detects an object closer than de, which is a predefined programmable critical distance, the motion is stopped. Otherwise this closest distance d is used to calculate the velocity to be set. Velocity is set to Vs, which is given in eq.6, where Vi,~ = f(d)Vm. Thus the velocity set would be
a fraction of the desired one based on the predefined programmable function f(d). In our case the robot will slow down linearly when an object is detected between 0.5m and lm, then it would stop if an object is detected closer than 0.5m. Now the force is generated by measuring the actual velocity of the robot, and taking the difference between that and the desired one, and sending it back to the joystick.
x I0 e
1000
,oo ~9
3080
0
•
!
50
100
150
Experimental Results
~oo
....
2
0
.
.
.
.
..i.
-400 ~
.
-
.
3288
flO
0
100
150
.
.ooL . . . .
50
150
0
.
4.00
.
.
.
: . . . . .
o
100
" 1 150
,
i- -
- :. . . . . .
50
ls°°l
500
50
200
......
:. . . . .
i. . . . . .
0
"4000
•
- - -
.
0
~
~ _ ?
0 Ti l .
200
-2O0
0
0f !
i. . . . . . .
'. . . . .
--1 O0
--~
0
Described here is one of the several experiments done, where the robot(Robotics and Automation lab, Michigan State University) and the operator (Robot Control lab, Chinese University of Hong Kong) were connected via the Internet. To understand the system operation let us consider Fig.3. In the first row a plot of time versus s, the event, is given. Its clear that s is a nondecreasing function of time, which was crucial for proving the system stability. The other plot in the first row shows the desired rotational velocity. The second row shows the desired velocities in x and y directions, which we call Vm. The third row displays the actual velocities in both directions, which we call Va. Next we illustrate the force that is played by the joystick in both directions, called Tin. The last row displays the same plot, which is the closest detected distance to obstacles in the environment. To understand the behavior of the system, let us consider all these plots together. Actual velocity changes according to the distance detected; for exmmple, if the distance decreases the actual velocity also decreases, but the force increases, as expected. Thus a close object causes the robot tracking error to increase, thereby telling the operator that an obstacle exists. Whenever the robot gets to the critical distance of 0.5m the actual velocity becomes zero, in both x and y directions. When the robot gets further away from obstacles the actual velocity starts tracking the desired one again, thereby reducing the tracking error, and hence the force. The sum of the actual velocity and the force plots should result in the desired velocity. This is illustrated in Fig.4: the first row shows the desired velocities and the second row the sum of the force and the actual velocities. It is clear t h a t the plots are similar although not identical. The difference is due to the fact t h a t the force we play is a filtered version of the tracking error. In Fig.5 we show the tracking error and the actual force felt, which is clearly a low pass filtered version of the error. The force felt is thus more smooth, without abrupt changes.
i ......
>, 400
-
4
......
- -
....
0
100
150
'
I
0
50
-
100
~1~oo[
150
:
]
1O0
150
'-
-, O0
50
-4110
150
~
o,
.
0
50
,
Figure 3: The behavior of the system during the control from Hong Kong.
4OO
4OO ~
200
i
200
!
-
0
-:
50
-
•
:
. . . . . . . . . . .
. . . .' . ~. i. . . .
o
~
~+~ 200. .. . . . . . . . . . . . . . . . .
: . . . .
1O0
150
0
Figure 4: Comparison between the desired velocities and the sum of the actual velocities and the force felt.
x 400~
~
2t~O . . . . . .
.....
.
[4] R. Simmons, "Xavier: An Autonomous Mobile Robot on the Web", IROS Workshop on Robots on the Web, Canada, 1998.
•
2O0 . . . . . . .
"~200 -4OO
100 . . . . . . : . . . . . :
...... i ¸ -2oo t ....
i ......
i .... -t
0
~"°°t • ....
[5] R. Anderson, M. Spong, "Asymptotic Stability for Force Reflecting Teleoperators with Time Delay", The Int. Journal of Robotics Research, 1992.
~0
•
-
: v~:
/-
[6] W. Kim, B. Hannaford, and A. Bejczy, "ForceReflection and Shared Compliant Control in Operating Telemanipulators with Time Delay", IEEE Trans. on Robotics and Auto., Vol. 8, April 1992.
i
Figure 5: Comparison of tracking error and force felt. 5
Conclusions
This study presents a new event-based control method for teleoperation systems with force feedback, which solves the problem of delay. This approach is immune to any kind of time delay, no upper bound is assumed, and the delay can be random or fixed; therefore, it is ideal for Internet applications. The method results in a stable and synchronized system; it is also independent of the system model, and specifically the network. In addition, a new force generation method was used, where the force felt is actually a function of the distance to obstacles in the environment of the robot. The operator is thus able to deduce the position of objects in the robot's vicinity simply by moving the robot around and feeling the force, getting force feedback with obstacle avoidance built in. It was demonstrated experimentally and theoretically that the system obtained is asymptotically stable, and that it is safe due to the built-in obstacle avoidance. In addition, the system is reliable and robust, since a disconnection does not affect it, and the operation can be resumed once a connection is re-established.
[7] G. Leung, B. Francis, J. Apkarian, "Bilateral Controller for Teleoperators With Time Delay via/zSynthesis", IEEE Trans. on Robotics and Automation, Vol 11, No. 1, February 1995. [8] ,M. Otsuka, N. Matsumoto, T. Idogaki, K Kosuge, T. Itoh "Bilateral Tele-manipulator System With Communication Time Delay Based on Force-SumDriven Virtual Internal Models", IEEE Int. Conf. on Robotics and Automation, pp. 344-350, 1995. [9] G. Niemeyer, J. Slotine, "Stable Adaptive Teleoperation", IEEE Journal of Oceanic Engineering, Vol 16, No. 1, January 1991. [10] N. Xi, "Event-Based Planning and Control for Robotic Systems", Doctoral Dissertation, Washington University, December 1993. [11] N. Xi, T. J. Tam, "Action Synchronization and Control of Internet Based Telerobotic Systems", IEEE Int. Conf. on Robotics and Auto., May 1999. [12] Y. Zheng, "Human-Robot Coordination for Moving Large Objects", Workshop Note, 1997 ICRA. [13] N. Hogan, "Multivariable Mechanics of The Neuromuscular System", IEEE Eight Annual Conf. of Engineering in Medicine and Biology Society, 1986. [14] T. Milner, "Human Operator Adaptation to Machine Instability", Advances in Robotics, Mechatronics, and Haptic Interfaces, ASME 1993.
References
[1] K. Brady, T. J. Tam, "Internet-Based Remote Teleoperation", Proceedings of the 1998 IEEE Int. Conf. on Robotics and Auto., Belgium, May 1998. [2] D. Pai, "ACME, A Telerobotic Measurement Facility for Reality-Based Modelling on the Internet", IROS Workshop on Robots on the Web, 1998. [3] M. Stein, "Painting on the World Wide Web: The PumaPaint Project", IROS Workshop on Robots on the Web, Canada, 1998. 3289
[15] B. Hannaford, L. Wood, B. Guggisberg, D. McAffee, H. Zak, "Performance Evaluation of a sixaxis generalized force-reflecting teleoperator", Jet Propulsion Lab. pub., 1989.
