Internet-Based Telerobotics of Mobile Manipulators: Application on RobuTER/ULM B. Khiter1, A. Hentout1, E. Boutellaa2, M.R. Benbouali1, and B. Bouzouia1 1
Division of Computer-Integrated Manufacturing and Robotics (DPR) 2 Division of Multimedia and Systems Architecture (ASM) Centre for Development of Advanced Technologies (CDTA) BP 17, Baba Hassen, Algiers 16303, Algeria {bkhiter,ahentout}@cdta.dz,
[email protected]
Abstract. Few works only deal with telerobotics of mobile manipulators via the Internet. This paper consists of a contribution in this research field and describes an Internet-based multi-agent telerobotic system of such robots. The developed system provides the operator with a human/robot interface, accessible via the Internet, for remote control of mobile manipulators. This interface displays all sensors data and video images delivered by the eye-inhand IP camera. In addition, the interface allows the operator to perform primitive tasks, either separately by the manipulator or by the mobile base, or in cooperation by both of them. The proposed telerobotic system is implemented on the RobuTER/ULM mobile manipulator. The validity of the system is demonstrated through telerobotic experiments of four primitive tasks via the Internet over a long distance. Keywords: Mobile Manipulators, Internet-based Telerobotics, Multi-agent Architecture, RobuTER/ULM.
1
Introduction
Telerobotic systems are traditionally implemented by using dedicated communication channels. Moving the local/client and/or the remote sites requires the movement of all the equipments and the network reconfiguration of the communication channel connecting the two sites. Another disadvantage is that operators are, also, forced to move at the client site in order to work and to prepare tasks. Moreover, if the client machine (where the human/robot interface is implemented) breaks down, it becomes difficult (or very expensive) to continue accomplishing the task successfully. In recent years, with the popularity of the Internet, Internet-based telerobotic systems are becoming a very interesting and promising field of researches in robotics. Telerobotics via Internet involves control of remote robots within a web browser [1]. Such a system eliminates most of the traditional problems mentioned previously. Furthermore, it allows easy and low-cost relocation of operators, and does not depend on the location of the equipment that controls the robot (located on a remote site). The development of an Internet interface, provides an opportunity for researchers to work and to cooperate together to control the robot from anywhere in the world [2]. C.-Y. Su, S. Rakheja, H. Liu (Eds.): ICIRA 2012, Part II, LNAI 7507, pp. 635–644, 2012. © Springer-Verlag Berlin Heidelberg 2012
636
B. Khiter et al.
The literature on telerobotics over the Internet is numerous. A brief survey on the main researches is given in what follows. In September 1994, a six-dof manipulator was put online in the University of Western Australia [3]. This system allowed the operators, via a web page, to control the robot located above a table with wooden blocks placed on it [4]. Mercury Project [5] was put online in August 1994 in the University of Southern California. This project consists of a SCARA manipulator over a semi-annular workspace containing sand and buried artifacts. The CINEGEN project [6] seeks to facilitate simulation aspects and offline programming of manipulators for non-specialists. The PUMAPaint Project [7] is an online manipulator located at Roger Williams University, and allowing any operator with a web browser to control a PUMA760 manipulator in order to paint on white paper with real brushes and paint. Other mobile base projects have been interested in Internet-based telerobotics. These systems allow the operator to control a mobile base, either in a static environment or in the exploration of dynamic ones. KhepOnTheWeb system [8] allows operators to control a Khepera mobile base in a static environment. Carnegie Mellon University developed Xavier autonomous indoor mobile base on the web [9]. This robot accepts commands to travel to different offices within a building and broadcasts camera images as it travels. Another web system is the WebPioneer project [10] where the operator drives a Pioneer mobile base in a dynamic environment. Few researchers only have been interested in telerobotics of mobile manipulators via the Internet. In RISCbot project [2], the authors implemented a web-based teleoperated mobile manipulator via sensor fusion. The RISCbot consists of a wheelchair mobile base with an end-effector capabilities for manipulation. Carelli and colleagues [11] proposed a combination of autonomous control and teleoperation command that gave more flexibility to the entire system. Their prototype consists of a three-wheeled mobile base with a five-dof standard manipulator mounted on it. As few works only deal with telerobotics of mobile manipulators via the Internet, this paper consists of a contribution in this field. It describes an Internet-based multiagent telerobotic system of such robots. The paper is structured into seven sections. The first section proposed a brief state-of-the-art on Internet-based telerobotic systems. The different control modes for telerobotics are analyzed in the next section. The third section presents the hardware structure of the experimental robotic system. Section four describes the proposed Internet-based telerobotic system of mobile manipulators. The fifth section describes the human/robot interface developed for telerobotics of the RobuTER/ULM over the Internet. Experiments and obtained results are presented in section six. Finally, conclusions and future work are presented.
2
Control Modes for Telerobotics
Chong [12] proposed a taxonomy for telerobotic systems (i) Single Operator Single Robot (SOSR), (ii) Single Operator Multiple Robot (SOMR), (iii) Multiple Operator Single Robot (MOSR) and, finally, (iv) Multiple Operator Multiple Robot (MOMR) [13]. Fong and colleagues [14] studied SOSR systems, which is the case of our
Internet-Based Telerobotics of Mobile Manipulators: Application on RobuTER/ULM
637
developed telerobotic system, where cooperation occurs between a single operator and a single robot. In this case, telerobotic control modes can be separated into four types ranging from “no assistance provided to the operator by the robot” to “no assistance provided to the robot by the operator” [15]. Direct Mode. The operator specifies all the robot motion by continuous input. He uses, thus, a suitable input device to control the movement of the robot such as a joystick, a mouse, etc. The robot does not take any initiative except to stop when it recognizes that communications breakdown [16]. Traded Mode. This mode provides alternative control of the robot. The competence of the robot includes capabilities to choose its own path, to respond intelligently to the environment, and to accomplish local goals. The operator assumes direct control in case of critical situations [15]. Shared Mode. The operator and the robot control, concurrently, different aspects of the telerobotic system. This mode relieves the operator from controlling details and lets him concentrate on the goals of telerobotics [17]. Autonomous Mode. The operator performs high-level planning and monitors the execution of the robot. He may have to interrupt the execution of the robot in dangerous situation or help it to execute tasks [17].
3
Hardware Structure of the Experimental Robotic System
The experimental robotic system, given by Fig. 1, consists of the RobuTER/ULM mobile manipulator. The robot is controlled by an on-board industrial PC and by four MPC555 microcontroller cards communicating via a CAN bus.
Fig. 1. Structure of the experimental robotic system (RobuTER/ULM)
638
B. Khiter et al.
RobuTER/ULM is composed of a six-dof ultra-light manipulator (ULM) with twofingered electrical gripper, mounted upon a rectangular non-holonomic differentiallydriven mobile base (RobuTER). The mobile base is equipped with an odometer sensor on each driven wheel, a laser measurement system at its front and a belt of 24 ultrasonic sensors. The manipulator is equipped with an incremental position sensor for each articulation, a six-dof effort sensor and an eye-in-hand IP camera installed on the gripper. The robot has also a wireless communication system in order to communicate with an off-board PC.
4
Internet-Based Telerobotic Architecture of Mobile Manipulators
4.1
Internet Constraints
The time delay depends on the distance separating the local/client from the remote site. It depends, also, on the processing time required for coding and data transmission, the processing speed and the load of nodes, the connection bandwidth and the transmission speed, the amount of data, etc. [18]. Such delays may be constant in case of direct connection, but may be variable according to the load of the network servers and to the dynamicity of the network structure (which is the case of the Internet) [19]. It is clear that direct control mode needs a high-speed network to achieve online direct control of the robot. The previous constraints make direct telerobotics unsuitable for time critical, constrained and dangerous interactions (for example collision between the robot and its environment). In this case, we refer to a shared/traded control mode of telerobotics, with the remote robot executing a set of primitive (or more complex) tasks in a completely autonomous mode [20]. 4.2
Internet-Based Multi-agent Telerobotic Architecture
The Internet-based telerobotic architecture of mobile manipulators is shown by Fig. 2. It consists of four local agents (SA, LMRA, LARA and VSA) and two remote agents (RMRA and RARA). The roles of each agent are given here below: •
Supervisory Agent (SA): it receives the task to be carried out and, decides on its feasibility according to the status and the availability of the required resources and sensors. If the task is accepted, SA distributes it on the local agents. • Local Mobile/Manipulator Robot Agent (LMRA/LARA): it cooperates with the other local agents (LARA/LMRA, VSA) in order to build an operations plan. In addition, this agent receives information on the environment of the mobile base/manipulator and feedback (from RMRA/RARA) on the execution of operations. Moreover, LMRA/LARA sends requests to RMRA/RARA for execution. • Vision System Agent (VSA): it observes the environment of the robot by the camera and, extracts useful and required information for the execution of the task from the captured images (images processing, recognition of objects, etc.).
Internet-Based Telerobotics of Mobile Manipulators: Application on RobuTER/ULM •
639
Remote Mobile/Manipulator Robot Agent (RMRA/RARA): it scans the sensors equipping the mobile base/manipulator and sends useful information to LMRA/LARA. In addition, RMRA/RARA controls the movement of the mobile base/manipulator in order to move to the desired situation.
Fig. 2. Internet-based multi-agent telerobotic architecture of mobile manipulators
4.3
Implementation of the Proposed Architecture
This work was performed on a mixed environment (i) Windows XP for the off-board PC (client) and (ii) Linux Redhat for the on-board PC (robot). The implementation of the Internet-based telerobotic architecture consists of hosting the four local agents (SA, LMRA, LARA and VSA) into a IIS Web server that allows to publish the control interface (the web application developed in ASP.net). The remote agents (RMRA and RARA) are installed on the on-board PC of the robot. The client can be any device running a web browser (PC, Laptop, PDA, Smartphone, etc.) in order to access the web page of the control application.
5
Human/Robot Interface
Following the connection of the operator to the server, an authentication page appears. The operator must enter his username and his password in order to access the control web page. Often, there are many clients making requests at the same time. Because our current system accepts one request at a time, only one client can have access to the control web page at the same time. The other clients (requests) are all ignored. Once the username and the password are verified, the control page of Fig. 3 appears. It consists mainly of six parts:
640
B. Khiter et al.
First Part. The first part concerns the LMRA agent. It displays all the data relative to the sensors of the mobile base (i) its current situation (New_X, New_Y, New_θ) and (ii) data of both right and left encoders (E_R, E_L). This part enables, also, activating US and LMS sensors, and computing the odometry (localization) of the mobile base (Odo_X, Odo_Y, Odo_θ). Finally, it allows to send requests to the mobile base in order to move to a given situation (X, Y, θ) while avoiding possible obstacles. Second Part. This part is reserved to LARA agent. It allows to move the manipulator, either axis by axis (Axis1, …, Axis6), or by specifying a given situation (x, y, z, ψ, θ, ϕ). It permits, also, to open/close the gripper. This part displays data of the sensors (i) six-dof effort sensor (Fx, Fy, Fz, Tx, Ty, Tz), (ii) position sensors of the joints (Axis1, …, Axis6) and (iii) the current situation of the end-effector (x, y, z, ψ, θ, ϕ). Third Part. This part concerns the VSA agent. The operator can visualize, in realtime, video images on the environment of the robot delivered by the eye-in-hand IP camera placed on the gripper of the manipulator. This part offers the ability to the operator to capture an image and to perform the necessary processing. Fourth Part. This area displays the different messages exchanged between the agents of the architecture and shows the result after each operation/task (success/failure). Fifth Part. This button is used to stop completely the mobile base and the manipulator while keeping the operator connected to the robot. Last Part. The operator can stop completely and disconnect from the robot by clicking on Logout.
4
1
5
3
6
Fig. 3. Human/robot interface for Internet-based telerobotics of RobuTER/ULM
2
Internet-Based Telerobotics of o Mobile Manipulators: Application on RobuTER/ULM
6
641
Experimental Results R
In this section, we describee the manipulation progress (via the developed applicatiion) from the client connection n to the complete execution of the task by the robbot. Experiments have been conducted c between two different locations in Algeeria equipped with connection to the Internet and separated by more than 200km. T The RobuTER/ULM was installeed in the CDTA Research Center and two clients conneccted to the robot (i) the first waas in Constantine (400km) in the east and (ii) the secoond client was in Ain Defla (200 0km) in the west. Considering that all the tasks described in the following sub-sections are execuuted autonomously by the robo ot in the same manner and independently of the typee of connection, the execution accuracy a and the positioning errors are, therefore, the same. Consequently, these experriments have been performed in order to evaluate the developed telerobotic systtem in terms of (i) average connection times and (ii) execution times of the taskss. Both of these times are given in seconds (s). 6.1
Average Connectio on Times
There are many Internet no odes between the local/client site and the remote site/roobot which introduce a network k time delay. The average connection time is given ass an indicator of such delays. The connection to the co ontrol application (both RMRA and RARA agents) has bbeen tested in three different cases depending on the type of connection (i) dirrect connection via a network cable c (ii) the LAN of the CDTA (iii) the Internet. In all the experiments, LMRA agent connected first to RMRA agent. The average connecttion times are summarized in Fig. 4. As it can be seen from these results, the connecttion time increases as the operattor moves from direct to LAN to Internet connection.
14 12 10 8 6 4 2 0
8.9
10.6 8.2
12.3 8.9
9.4
Direcct connection Local network Internet Mobile base Manipulator
Fig. 4. Average connection n times to the remote agents for the different connection types
6.2
Execution Times off the Tasks
Four basic types of tasks,, illustrated in this experimental sub-section, have bbeen performed by the mobile manipulator. The initial situation of the mobile bbase BaseInit(xBInit, yBInit, θBInit)= =(0mm, 0mm, 0°). The initial configuration of the
642
B. Khiter et al.
manipulator ConfigurationInit I (Q1Init, Q2Init, Q3Init, Q4Init, Q5Init, Q6Init)=(0°, 0°, 0°, 0°, 0°, 0°). The initial situation of o the end-effector EffectorInit(xEInit, yEInit, zEInit,ψEInit, θEEInit, ϕEInit)=(-432mm, -108.49mm m, 164mm, -180°, -180°, -180°). First Task (T1). Moving g the manipulator from its initial situation to the fi final situation given by EffectorFin mm, F (xEFin, yEFin, zEFin, ψEFin, θEFin, ϕEFin)=(-330mm, -630m 1080mm, -135°, -88°, 5°). The final configuration n of the manipulator corresponding to this situatiion, computed by using the Inverse I Kinemetic Model (IKM) of the manipulator, is ConfigurationFin(Q1Fin, Q2Fiin, Q3Fin, Q4Fin, Q5Fin, Q6Fin)=(-60°, 61°, 30°, 95°, -15°, 0°°). Second Task (T2). Mo oving the manipulator from its initial configurattion ConfigurationInit successiveely to three different configurations as follows: • Configuration1(Q11, Q21, Q31, Q41, Q51, Q61)=(0°, 40°, 28°, 0°, 0°, 0°). • Configuration2(Q12, Q22, Q32, Q42, Q52, Q62)=(20°, 32°, 28°, 0°, 0°, 0°). • ConfigurationFin(Q1Fin, Q2Fin 2 , Q3Fin, Q4Fin, Q5Fin, Q6Fin)=(0°, 45°, 45°, 0°, 0°, 0°). Third Task (T3). Moving the mobile base from its initial situation to a final situattion given by BaseFin(xBFin, yBFinn, θBFin)=(-1920mm, 2mm, 15°) in presence of one obstaacle at the position Obstacle((xOb, yOb, zOb)=(-1000mm, 0mm, 50mm) of a size of (800x200x100)mm. Fourth Task (T4). Moving the mobile base from its initial situation to a fi final situation given by BaseFin(xxBFin, yBFin, θBFin)=(-3440mm, 13mm, 12°). Two obstaccles are present in the environm ment. The first one is located at the position Obstacle1(xxOb1, yOb1, zOb1)=(-1000mm, 400mm, 50mm) of a size of (800x200x100)mm. The secoond obstacle is at the position Obstacle O of a 2(xOb2, yOb2, zOb2)=(-2000mm, -400mm, 50mm) o size of about (600x250x100 0) mm. For each primitive task considered in this sub-section (T1, T2, T3, T4), ten ttests have been performed by thee mobile manipulator for all the types of connection (dirrect connection, LAN connectiion and Internet connection). Fig. 5 gives the averrage execution times of all these tasks. 172 174 175 180 160 140 120 100 80
65 67 68
67 69 70
71 73 74
60 40 20 0 T T1
T2 Direct connection
T3 Local network
T4 Internet
Fig. 5. Average execution n times (in seconds) for the considered tasks (T1, T2, T3, T4)
Internet-Based Telerobotics of Mobile Manipulators: Application on RobuTER/ULM
7
643
Conclusion
The objective of this work is to develop an Internet-based telerobotic system for mobile manipulators. Such a system allows using mobile manipulators in order to accomplish complex tasks in dangerous, inaccessible and hostile environments via the Internet. We described the control interface, available via the Internet, of the RobuTER/ULM mobile manipulator. Through this interface, the operator has a panel for telerobotics of the robot. The operator has all the sensors information of the RobuTER/ULM displayed on its web interface. He has, also, control mechanisms of the mobility, the manipulation, the sensors and the end-effector of the robot. In addition, the interface allows the operator to visualize the environment of the robot through the video images delivered by the eye-in-hand IP camera. This latter can be controlled via the web interface allowing more flexibility to view the environment. The connection to the RobuTER/ULM mobile manipulator (RMRA and RARA) has been tested in three distinguished cases (i) Direct connection via a network cable, (ii) connection through the local network of the CDTA and, finally, (iii) connection via the Internet. It is obvious that connection over the Internet takes much more time compared with direct connection or via the CDTA's LAN. While performing the tasks, we noticed a difference of less than 03 seconds (01 to 03 seconds much more) in the average total execution times of all the tasks. This augmentation is acceptable regarding all the other advantages of using the Internet. Once the operator connected to the web control interface via a browser, he must authenticate to access the various tasks to be performed by the robot. These latter are of three categories (i) control of the manipulator, (ii) control of the mobile base and (iii) cooperative control of the manipulator and the mobile base. Our system suffers by forcing operators to work sequentially and wait in a queue. Consequently, the next step is to emigrate from Single Operator Single Robot (SOSR) current system onto Multiple Operator Single Robot (MOSR) system.
References 1. Monroy, C., Kelly, R., Arteaga, M., Bugarin, E.: Remote Visual Servoing of a Robot Manipulator via Internet2. Journal of Intelligent Robotic Systems 49(1), 171–187 (2007) 2. Elkady, A., Sobh, T.: Web-Based Control of Mobile Manipulation Platforms via Sensor Fusion. In: Tzafestas, S.G. (ed.) Web-Based Control and Robotics Education, Intelligent Systems, Control and Automation: Science and Engineering, pp. 297–312. Springer Science + Business Media B.V. (2009) 3. Dalton, B., Taylor, K.: A Framework for Internet Robotics. In: The International Conference on Intelligent Robots and Systems (IROS 1998): Workshop on Web Robots, Canada (October 1998) 4. Dalton, B.: Techniques for Web Telerobotics. Ph.D. Thesis in Mechanical and Materials Engineering, University of Western Australia, Perth, Australia (2001) 5. Goldberg, K., Gentner, S., Sutter, C., Wiegley, J.: The Mercury Project: A Feasibility Study for Internet Robots. IEEE Robotics & Automation Magazine 7(1), 35–40 (2000)
644
B. Khiter et al.
6. Fluckiger, L., Baur, C., Clavel, R.: CINEGEN: a rapid prototyping tool for robot manipulators. In: Schweitzer, G., Siegwart, R., Cattin, P. (eds.) The Fourth International Conference on Motion and Vibration Control (MOVIC 1998), Zurich, Switzerland, vol. 1, pp. 129–134 (1998) 7. Stein, M.R.: The PumaPaint Project. Autonomous Robots 15(1), 255–265 (2003) 8. Simmons, R.: Xavier: An Autonomous Mobile Robot on the Web. In: The International Conference on Intelligent Robots and Systems (IROS1998): Workshop on Web Robots, Canada (October 1998) 9. Burgard, W., Cremers, A.B., Fox, D., Lakemeyer, G., Hâhnel, D., Schulz, D., Steiner, W., Thrun, S.: The interactive museum tour-guide robot. In: The 15th National Conference on Artificial Intelligence, USA (July 1998) 10. Grange, S., Fong, T., Baur, C.: Efective Vehicle Teleoperation on the World Wide Web. In: The International Conference on Robotics and Automation (ICRA 2000), USA (April 2000) 11. Carelli, R., Forte, G., Canali, L., Mut, V., Araguas, G., Destefanis, E.: Autonomous and teleoperation control of a mobile robot. Mechatronics 18(1), 187–194 (2008) 12. Chong, N., Kotoku, T., Ohba, K., Komoriya, K., Matsuhira, N., Tanie, K.: Remote coordinated controls in multiple telerobot cooperation. In: The International Conference on Robotics and Automation (ICRA 2000), San Francisco, USA (April 2000) 13. Goldberg, K., Song, D., Khor, Y., Pescovitz, D., Levandowski, A., Himmelstein, J., Shih, J., Ho, A., Paulos, E., Donath, J.: Collaborative Online Teleoperation with Spatial Dynamic Voting and a Human Tele-Actor. In: The International Conference on Robotics and Automation (ICRA 2002), Washington, USA (May 2002) 14. Fong, T., Thorpe, C., Baur, C.: Safeguarded Teleoperation Controller. In: The International Conference on Advanced Robotics (ICAR 2001), Budapest, Hungary (August 2001) 15. Ong, K.W., Seet, G., Sim, S.K.: An Implementation of Seamless Human-Robot Interaction for Telerobotics. The International Journal of Advanced Robotic Systems 5(2), 167–176 (2008) 16. Bruemmer, D.J., Dudenhoeffer, D.D., Marble, J.L.: Dynamic-Autonomy for Urban Search and Rescue. In: AAAI Mobile Robot Competition, Canada, July 28-August 1 (2002) 17. Lin, I.-.S., Wallner, F., Dillmann, R.: An Advanced Telerobotic Control System for a Mobile Robot with Multisensor Feedback. In: The Intelligent Autonomous Systems (IAS4), Karlsruhe, Germany, March 27-30, pp. 365–372 (1995) 18. Yu, R., Huang, X.G.: Robot Remote Control Internet Architecture. In: Qi, L. (ed.) ISIA 2010. CCIS, vol. 86, pp. 514–518. Springer, Heidelberg (2011) 19. Wang, S., Xu, B., Liu, Y., Zhou, Y.: Real-time mobile robot teleoperation via Internet based on predictive control. Frontiers of Mechanical Engineering in China 2008 3(3), 299–306 (2008) 20. Tzafestas, C.S.: Web-Based Laboratory on Robotics: Remote vs. Virtual Training in Programming Manipulators. In: Tzafestas, S.G. (ed.) Web-Based Control and Robotics Education, Intelligent Systems, Control and Automation: Science and Engineering, pp. 195–225. Springer Science + Business Media B.V. (2009)
