Web-Based Robot Programming Environment and Control Architecture

2012 Brazilian Robotics Symposium and Latin American Robotics Symposium

Web-based robot programming environment and control architecture Rafael V. Aroca1 , Renato Q. Gardiman2 and Luiz Marcos G. Gonc¸alves3 Abstract—We propose a novel web-based programming architecture for robots. The proposed system offers a web interface that allows users to type their robot control programs, so no development environments and tools need to be installed in the user’s computer. From the web control panel, the user can also save, load and execute programs stored in the robot’s memory. For the implementation we use a smartphone as the robot’s main control computer, so the web server is embedded in the phone, which is fixed to the robot. The system allows different programming languages to be used and also takes advantage of sensors available in the phone.

control system, such voice recognition and synthesis [3]. Moreover, as phones are produced in a very large scale, it is cheaper to buy a mobile phone with the mentioned sensors built-in than buying a microcontroller, all these sensors separately and integrating them all. Furthermore, it has been already shown that mobile phone processors are capable of executing complex robotics algorithms [4]. The main contribution of this paper is a robot programming architecture based on the web, allowing users to directly edit, run and control robots’ programs remotely using web browsers. The robot can be be programmed in several mature programming languages such as Python, Perl and others. The system also offers an abstraction layer which allows programs to be executed in different types of robots without source code modification. It can be used for research or educational purposes in undergraduate classes or even for young teenagers. We will show the architecture details and experimental setup consisting of a mobile robot and its programming that has validated it, so validation is done through a proof of concept. Several applications can be built based on this proposal, including educational robotics, remote manipulation and navigation, reconfigurable robots, among others. A standard HTML browser is the only requirement of the development station, that needs few hardware resources.

I. INTRODUCTION Robot programming typically needs specific software tools to write a program source code for a given robot, to compile it and send it to the robot. Many of these tools are proprietary [1] and tied to a specific programming language. Thereby users need to have access to a computer with administrator rights in order to install and sometimes even to run development software. Another approach consists in using an embedded computer installed in the robot with an operating system and development tools, so the user can remotely connect to the robot and compile/run programs directly inside the robot’s main computer. One interesting solution consists in deploying a web server with web applications that allow users to remotely edit, compile and execute programs in the robot using only a simple web browser. With this solution, any computing device that has a web browser can be used to develop, execute and control robot programs. We propose in this paper the use of such approach embedded into a mobile phone that is used as the main computer of the robot. The proposed architecture can be used to easily control different types of robots and the same control program can be used in different robots thanks to a Robot Hardware Abstraction Layer (RHAL). Also as a novelty, we use a smart phone as the control computer of the robot using an approach based on Dual Tone Multi Frequency (DTMF) for the coding of control data. A recent report from IEEE Robotics and Automation magazine points that smartphone-based robots is one of the top robotics trends for the year of 2012 [2]. In fact, these phones have several features that are useful for a robot, such as Global Positioning System (GPS) receiver, camera, 3 axis compass, accelerometer, wifi, bluetooth, speakers, microphone, light sensor and software support that fits perfectly on a robot

II. STATE OF THE ART The main concept of our work is commonly used in remote and virtual laboratories setups that allow students and researchers to remotely program, execute, monitor and control experiments. In such systems, the expressions “remote programming” and “tele-programming” are frequently used. The origin of remotely programmed robots came naturally as the next step from remotely controlled robots that first appeared in the 1940s [5]. Nonetheless one of the first works dealing with remote robot programming environments appears in the year of 2001 [6] with the aim of simplifying the problem of remote robot control. In 2002, Lowe and Cambrell [7] presented a system for controlling mechatronic devices that allows users to send manual movement commands to a robot using a Java Applet. In order to program the robot in such system, the user has to connect to a server using telnet. In their setup, the robot has a proprietary language and interface available via a RS-232 serial port, so the server only redirects telnet connections to the serial port, allowing the robot’s proprietary interpreter to be used remotely. Mootien et al propose a educational remotely operated robot to allow students to write their programs and run them at any time [8]. Their system is based on running a Virtual Network Computing (VNC) server in a computer connected to the robot, allowing remote users to access and control this computer’s

1 NatalNet Laboratory, Federal University of Rio Grande do Norte (UFRN), Natal-RN, Brazil rafaelaroca at ieee.org 2 NatalNet Laboratory, Federal University of Rio Grande do Norte (UFRN), Natal-RN, Brazil renatoqg at gmail.com 2 NatalNet Laboratory, Federal University of Rio Grande do Norte (UFRN), Natal-RN, Brazil lmarcos at natalnet.br

978-0-7695-4906-4/12 $26.00 © 2012 IEEE DOI 10.1109/SBR-LARS.2012.12

27

applications such as Apache HTTPd and Microsoft Internet Information Services (IIS). In the following sections we describe an approach with substantial improvements over these systems, including the possibility of executing programs of different languages and running several programs in parallel in the robot’s computer. Furthermore, our proposed solution also offers an abstraction layer so the same program can be executed in several robots of different types, which is an important aspect of our work due to the fact that robot’s commonly present a wide variation of hardware and interfaces [1].

graphical user interface from their own computers with a VNC desktop client software. Lopez et al describe a system that allows remote access to write, compile and execute programs using Secure Shell (SSH) remote access [9]. Their system have a web interface with live camera preview, an instant messaging system for the connected users and a Java applet to access the SSH terminal from the browser. Their robot can be programmed with Ada or C programming languages. Another project with live video preview uses a Java client application. The client application allows users to edit programs and locally compile their programs on the remote development machine. Their system support only V+ scripts, a proprietary language for a specific robot brand. After compilation, the user submits the program to the server, which is connected to the robot, and execute it [10]. Marin et al describe a virtual environment with remote programming feature [11]. Their system implement a TCP/IP network server, so programs can be written in any language to connect and exchange control data with the main robot control software, so only high-level pre-defined commands can be executed from the remote program. They also compare seven tele-operation projects and argue that theirs is the only one with remote programming capability. Fernandez and Casals present an On Line Laboratory concept and implementation that allows users to edit, publish and execute programs remotely [12]. Their system use a web server that allows clients to write program using a web browser, check its syntax, and execute it. The client needs only a web browser with Java enabled in order to run a Java Applet. One advantage of such approach is that the programming environment can be used in any computer, independently of operating system. A related project is the Mindstorms Internet Control Environment (MICE) [13]. It consists of a web interface that also uses a Java Applet which runs on the user’s web browser. This Applet allows users to write source codes for Lego Mindstorms Intelligent bricks using the “Not Quite C” (NQC) language. After the code is written, it is possible to transmit it to the Robot, run it and stop it. The system also offer a camera showing live information from the robots workspace. Popescu et al propose a custom made differential drive mobile robot with a web based remote programming environment [14]. The robot’s sensors and actuators are managed by a low end microcontroller which is connected to a tablet computer also installed on the robot. The microcontroller and tabletPC computer are connected with a standard serial port. The PC also has a wireless wifi connection to allow remote access from the Internet and runs a dedicated web server so the robot can be accessed from any web browser with Java Applet support. Users can open, save and write their programs in the microcontrollers. As described in this short review, several remote programming projects have already been done, but these projects are tied to specific programming languages. Moreover most of them use setups with fixed robots and traditional web server

III. PROPOSED ARCHITECTURE As mentioned in Section I, the proposed system uses smartphones as the robot’s main computer, so our architecture relies on Android, a Linux based operating system developed by Google for mobile devices such as phones and tablet computers. It offers a rich application programming interface (API) that allows users to execute sophisticated tasks with little programming effort. Our first experiment consisted in using reflection mechanism. Reflection allows a program that is being executed to load a class or a method from disk and dynamically call methods available in this file. In that way, users can write and compile algorithms for the robot and the main program can list these algorithms, letting the user choose one of them to be dynamically loaded into the running code and executed, interacting with the rest of the system. Although flexible, unfortunately we found this approach not practical because of the need of a Dalvik compiler in the phone. Current smartphones have processors faster than 1GHz and they have computing power to run compilers, but this approach would make the system’s deployment more complex and error prone. As a solution, we decided to use Scripting Layer for Android (SL4A) [15], [16], which allows several script programming languages to be edited and executed on Android devices. Moreover, it offers access to the Android API resources from these languages. The languages currently supported are Python, Perl, Ruby, Lua, BeanShell, JavaScript, Tcl and Shell scripts, but SL4A offers guidelines to add support for more languages, so this list can increase with time. Our system automatically takes advantage of these features, so if new languages are added to SL4A, robots will be able to be programmed with these new languages automatically. Figure 1 shows a general overview of the architecture. Its main component is an embedded webserver that we developed. It runs on the mobile phone and accepts connections from any device connected to the same network in which the phone is connected. Any device with a standard web browser can access and interact with the system without the need of Flash or Java Applet support, so thin clients and devices with constrained resources can be used, as only basic HTML capacity is needed. The main actions that the users can do after accessing the embedded webserver from a web browser are: 1) Source code management a) List programs

28

Fig. 2.

Fig. 1.

System’s architecture layers

programs to be executed at the same time, so users can build modular and cooperative systems. For example, a program to control a PID loop can be executed while another program recognizes voice commands and another program recognizes objects viewed by the camera. The programs can exchange information and synchronization using standard Android, Linux and Unix inter process communication support. “Monitor and manual control” allows the user to send simple manual commands to the robot such as move forward or backwards. Each manual command can be executed with a single click, and is directly mapped to the execution of a user program. It also offers the possibility of viewing a live preview of the phone’s camera and a flexible way to show dynamic results from user programs, which can generate data that is shown in a status page with 1 second automatic refresh. Figure 1 also shows that user programs can use sensors both from the mobile phone and the robot. As said before, smartphones commonly have accelerometer, GPS, compass, camera and other sensors. The connection from the mobile phone to the robot can be done with any hardware interface supported by the phone, such as Bluetooth, USB, serial or even using the analog audio channel [3]. An overview of the system hierarchy is shown in figure 2. User programs can access operating system facilities using the provided API, while robot specific commands are handled by a Robot Hardware Abstraction Layer (RHAL), which maps user program’s commands to robot’s commands according to each robot. So, in order to run the same program in different robots, only the RHAL block has to be modified or configured to a specific robot. As the mentioned languages are mature and have plenty of libraries and documentation, their usage offers easy development with several possibilities such as network access, image processing, numerical computing, access to robot’s sensors, among other advantages. Distributed computing among several robot’s or computers and cooperation of robots can also be implemented with the networking support of these languages.

Overall system architecture

b) c) d) e)

Edit a program Start the execution of a program Save or “Save as” a program Share a program with another person or robot using Bluetooth or email 2) Running processes a) List running programs b) View the output of running programs c) Stop a running program 3) Monitor and manual control a) Live view from robot’s camera b) Execution of manual commands (Eg.: move forward, stop) c) Exhibition of output dynamically generated by user’s programs From the given list, “Source code management” allows users to list all user programs available in the phone. The programs are actually stored on a SD Card as text files, so these source codes can be copied from and to the phone and edited with any text editor. The edit option opens the source code for editing in the web browser, so all tasks and development can be done using the browser. The user can also start the execution of a selected program or share it with another person or robot. “Running processes” allows the user to manage all the programs that were started from the web. A web page listing all programs being executed is shown and users can view the standard output and errors of these programs, or choose to stop the program. As can be noted, the architecture allows several

29

if code[’extras’][’SCAN_RESULT’] == "7891219026195": droid.ttsSpeak("I’m home") else: droid.ttsSpeak("Unknow landmark")

IV. IMPLEMENTATION AND EXAMPLES Our implementation was tested in several Android mobile devices, but we focused the tests in the HTC G1 mobile phone, an early Android based phone with an ARM processor of 528MHz, 192MB of RAM memory and Android 2.1. As all newer devices are more powerful, we made efforts to make the system work in this platform, so it will easily run in newer systems. Furthermore, although the focus here is in smartphones, any platform that can run Android will be able to run our system. In fact, there are Android versions for x86 PC computers and for ARM processor based embedded boards commonly used in robots such as the BeagleBoard, PandaBoard and Gumstix. The connectivity is quite flexible under Android. Systems can be connected to the Internet using 3G/GPRS services provided by network operators infrastructure. The robot can connect to a wifi access point or also use direct wifi connection. In the latter case, if there are no access points available, the user can directly connect to the robot using an ad-hoc wifi connection. For the software, when the user asks for the execution of a certain program, a wrapper script launches the desired program in the background. The program’s output is stored in log files for on line or later inspection. Next, we list some examples to show the easiness of using Python language and Android to program robots.

Example 3: Connect to a Lego NXT robot, move it forward when the user says ”Move”, and finally speak the distance measured by Lego’s ultrasound sensor import android, time import nxt.locator from nxt.motor import * droid = android.Android() def move(s) m_left = Motor(s, PORT_B) m_left.update(100, 3600) m_right = Motor(s, PORT_C) m_right.update(100, 3600) sock = nxt.locator.find_one_brick() if sock: voiceCmd = droid.recognizeSpeech().result; if voiceCmd == "move": move(sock.connect()) else: droid.ttsSpeak("Command not understood") else: print ’Lego NXT unavailable’ D = UltrasonicSensor(sock, PORT_4).get_sample() droid.ttsSpeak("Ultrasound distance is " + str(D)) sock.close()

A. User program examples Figures 3, 4 and 5 show three different robots under the control of the same program. Only the RHAL functions are updated to map the control calls from the user software to the robot’s hardware. The robot of figure 3 uses the analog audio channel [3], [18] from the mobile phone to control the robot, offering an easy, affordable and practical solution to control the robot with a standard connection (the earphone output of the phone). Commands are encoded as audio tones and a tone decoder module in the robot’s circuit decodes the audio to execute the desired motor movements. The Lego robot of figure 4, on the other hand uses a bluetooth wireless link between the mobile phone and Lego NXT intelligent brick and the robot of figure 5 uses a wireless wifi connection. More details about the robot shown in figure 3 and its control architecture based on the audio channel can be found in a previous work [3], with detailed explanation on how to build a flexible and powerful robot system with low cost (under US$ 30 without the phone’s costs). Although we only depict simple examples, any resource provided by the mentioned languages could be used. Control loops can be closed using the mobile phone built-in sensors or the robot sensors. It is possible, for example, to use the compass of the mobile device or the gyroscope to control the robot heading direction and curves [3]. Accelerometer and computer vision can be used to implement visual odometry and Simultaneous Localization and Mapping (SLAM) techniques. Figure 6 shows a screenshot of the web browser view

We note that these examples are full listings of fully functional programs and not parts of the source code. Example 1 shows how to read the phone’s sensors, specifically the compass. One possibility for robot localization is using barcode labels as landmarks [17], so example 2 shows how to use the phone’s camera to recognize barcode labels and do some action. Example 3 recognizes a voice command and then moves a Lego NXT mobile robot. In order to read the GPS, the function droid.getLastKnownLocation() can be used. Further information about these commands can be found in SL4A API manual. Example 1: Read the phone’s compass and verbally say it with the text to speech system import android, time droid = android.Android() droid.startSensing() time.sleep(1) results = droid.readSensors().result R = results[’zMag’]; droid.ttsSpeak("My heading is " + str(R) + " degrees")

Example 2: Recognize a barcode label with the camera and speak ”I’m home” if the barcode number is 7891219026195 import android, time droid = android.Android() code = droid.scanBarcode().result

30

of the main page when the robot embedded webserver is accessed. On the left it can be seen the camera alive view and a dynamically generated compass showing the heading of the robot. The compass view is updated automatically with a 1 second refresh rate and is fully generated by a user code, so even graphical output with shapes can be done with user programs.

Fig. 3. A custom built robot under the control of the proposed system using audio tones from the phone to encode motor commands

Fig. 6. Screenshot of a web browser accessing the robot’s embedded server

As the developed webserver can also access static HTML documents, users can customize menus, pages, add documentation and other information as in any other standard web server.

Fig. 4. A Lego Mindstorms robot (TriBot) under the control of the proposed system using Bluetooth wireless link

V. CONCLUSIONS

Fig. 5. system

We proposed a novel approach for web based remote programming of mobile robots that was validated with a proof of concept implementation. A much more general architecture is provided, comparing to the ones found in the literature. All the development is done by accessing a webpage embedded on the robot that has all the editing and execution functions. As it uses only standard HTML, it can be accessed from any browser in any device, even the ones with low hardware resources. Our approach is suitable for applications that need remote programming of robots, including educational robotics, remotely reconfiguration of robots, in any level, research or even to build advanced robotic toys. It allows users to develop, execute and monitor robot programs in different languages without the need of installing any software in their computers. The architecture also provides an abstraction layer that allows users to write high level control programs and use them in different programs without the need of modifying the code for each robot.

A pioneer P3AT research robot under the control of the proposed

31

[13] A. Garrett and D. Thornton, “A web-based programming environment for lego mindstorms robots,” in Proceedings of the 43rd annual Southeast regional conference - Volume 2, ser. ACM-SE 43. New York, NY, USA: ACM, 2005, pp. 349–350. [Online]. Available: http://doi.acm.org/10.1145/1167253.1167333 [14] D. Popescu, D. Selisteanu, I. Dinulescu, and L. Popescu, “Web based telematics application for robotics,” in Computing in the Global Information Technology, 2008. ICCGI ’08. The Third International MultiConference on, 27 2008-aug. 1 2008, pp. 19 –24. [15] L. Jordan and P. Greyling, Practical Android Projects. Apress, 2011. [16] P. Ferrill, Pro Android Python with SL4A. Apress, 2011. [17] A. Sousa, C. Santiago, P. Malheiros, P. Costa, and A. P. Moreira, “Using barcodes for robotic landmarks,” in Fourteenth Portuguese Conference on Artificial Intelligence, Aveiro (Portugal), oct 2009. [18] R. Aroca, L. Gonc¸alves, and P. Oliveira, “Towards smarter robots with smartphones,” in Robocontrol 2012, Bauru - SP (Brasil), jun 2012.

One of the limitations of our implementation is that it is based on the Android system, so mobile devices with other operating systems cannot be used in this current implementation. On the one hand, the proposed architecture is generic enough to be implemented in other languages and operating systems. The implementation is open source and is available under request. On the other hand we consider that this is not a limitation but a good feature, thanks to the Android Scripting Layer, that allows to the system the usage of several programming languages. Currently supported languages are Python, Perl, Ruby, Lua, BeanShell, JavaScript, Tcl and Shell scripts. We listed several examples using Python language to show the ease of reading sensors and controlling robots in the proposed environment. Finally, we have not evaluated real time performance of the proposed system, so a future work should analyse real time features of the mentioned scripting languages running on a smart phone. ACKNOWLEDGMENT The authors would like to thank the support from the National Research Council (CNPq), Brazilian Sponsoring Agency for Research. R EFERENCES [1] L. Gumbley and B. MacDonald, “Development of an integrated robotic programming environment,” 2005, cited By (since 1996) 0. [2] E. Guizzo and T. Deyle, “Robotics trends for 2012 [the future is robots],” Robotics Automation Magazine, IEEE, vol. 19, no. 1, pp. 119 –123, march 2012. [3] R. V. Aroca, A. F. Burlamaqui, and L. M. G. Gonc¸alves, “Method for reading sensors and controlling actuators using audio interfaces of mobile devices,” Sensors, vol. 12, no. 2, pp. 1572–1593, 2012. [Online]. Available: http://www.mdpi.com/1424-8220/12/2/1572/ [4] A. C. Santos, L. Tarrataca, and J. a. M. Cardoso, “The feasibility of navigation algorithms on smartphones using j2me,” Mob. Netw. Appl., vol. 15, no. 6, pp. 819–830, Dec. 2010. [Online]. Available: http://dx.doi.org/10.1007/s11036-010-0236-8 [5] K. Goldberg and R. Siegwart, Beyond Webcams: An Introduction to Online Robots. MIT Press, 2001. [6] I. R. Belousov, R. Chellali, and G. Clapworthy, “Virtual reality tools for internet robotics,” in ICRA. IEEE, 2001, pp. 1878–1883. [7] G. S. Lowe and A. Cambrell, “Web system for control of mechatronic devices,” in ICARCV. IEEE, 2002, pp. 1464–1469. [8] S. Mootien, R. T. F. A. King, and H. C. S. Rughooputh, “A webbased interface for the Gryphon robot,” INTERNATIONAL JOURNAL OF ELECTRICAL ENGINEERING EDUCATION, vol. 43, no. 2, pp. 134–149, APR 2006. [9] D. Lopez, R. Cedazo, F. Sanchez, and J. Sebastian, “Ciclope robot: Web-based system to remote program an embedded real-time system,” Industrial Electronics, IEEE Transactions on, vol. 56, no. 12, pp. 4791 –4797, dec. 2009. [10] M. Alifragis, A. Mantelos, and C. Tzafestas, “Web-based remote and virtual programming console of the v+ robotic system,” in Industrial Electronics, 2008. IECON 2008. 34th Annual Conference of IEEE, nov. 2008, pp. 3551 –3555. [11] R. Marin, P. Sanz, P. Nebot, and R. Wirz, “A multimodal interface to control a robot arm via the web: a case study on remote programming,” Industrial Electronics, IEEE Transactions on, vol. 52, no. 6, pp. 1506 – 1520, dec. 2005. [12] J. Fernandez and A. Casals, “Open laboratory for robotics education,” in Robotics and Automation, 2004. Proceedings. ICRA ’04. 2004 IEEE International Conference on, vol. 2, 26-may 1, 2004, pp. 1837 – 1842 Vol.2.

32