Blender for Robotics: Integration into the Leuven Paradigm for Robot Task Specification and Human Motion Estimation Koen Buys, Tinne De Laet, Ruben Smits, and Herman Bruyninckx Department of Mechanical Engineering Katholieke Universiteit Leuven, Belgium
[email protected] http://www.mech.kuleuven.be/robotics
Abstract. A growing number of robotics labs cooperate in the “Blender for Robotics” project [4], to integrate the open source computer animation program Blender into their research. In the first place as a visualisation tool, but gradually more and more as a full-fledged component in a robot simulation and programming environment. The lab in Leuven focuses on improving Blender support for the task specification and control of complex, sensor-based robot tasks, in which also the motion of the human(s) interacting with the task must be tracked.
1
Introduction
A lot of tools already exist in the robotics community to simulate robots in virtual environments and to visualise actual data in such a virtual world. Most of them were created for a specific research project or to interface a specific robotics framework. One of the most popular free and open-source software (FOSS) projects is Player, with the 2D Stage [13] simulator, and the 3D Gazebo [14] simulator; both have been accepted as core modules in the “Robot Operating System” ROS [27], which is rapidly becoming the number one software framework in robotics. So, why spend efforts to build yet another visualisation and simulation tool, with Blender [19] at its basis? The major reasons are the following significant differences with most robotics software frameworks: Blender is a well-established project (with a development history of over 15 years) with larger user and (professional!) developer communities than any robotics project; Blender runs on all major operating systems; its code base is, by design, extremely modular, so that it can accomodate different solutions for the same functionality, such as rendering, physics engines (Bullet [6], ODE [21], and others), or kinematics; its Python language bindings are extensive, and allow for flexible interfacing with external programs; the currently developed version 2.5 provides extensive support for hardware-accelerated computations (graphics in the first place), for the creation of application-specific interface panels, and for state of the art inverse kinematics of redundant tree-structured robots. N. Ando et al. (Eds.): SIMPAR 2010, LNAI 6472, pp. 15–25, 2010. c Springer-Verlag Berlin Heidelberg 2010
16
K. Buys et al.
Section 2 gives an overview of K.U.Leuven’s contributions. They are illustrated by an example in Section 3. Section 4 concludes the paper with a discussion and a look at future work.
2
Contributions
The first tangible result of K.U.Leuven’s contributions has been the work of Benoit Bols´ee, who implemented the iTaSC (instanteneous Task Specification using Constraints) framework [8,24,23] in Blender 2.5. This contribution lays the foundations of a complete physics simulator (and not just a visualisation!) of kinematic chains and their interactions with objects in the environment and has already shown to outperform the inverse kinematics coming with Blender (http://vimeo.com/groups/blenderandrobotics/videos/5857397). In this paper we show how to use Blender not only as a computer animation package but also as a robotics toolbox. Both as a visualisation tool (Section 2.1), that is, making real world experimental data visible in a graphical way, but also as a simulator (Section 2.2) that can create virtual sensor data. In addition, we also show the middleware interaction possibilities (Section 2.3) of Blender with various robotic frameworks as Open Robot Control Software (Orocos) [5], Robot Operating System (ROS) [18], GenoM [11,15],YARP [10],. . . 2.1
Visualisation Tool
Visualising in a graphically appealing way the large amount of data that is generated during typical robotics experiments is a major requirement of most robotics projects. To this end, the visualisator must provide the same interface as the robot controller. To this end, the interface should ideally provide a similar interface as the robot controller to allow for off-line visualisation of algorithms. The visualisation feature of Blender can be used to get a graphical impression of the motion of robots. Figure 1 shows a snapshot of a rendering of one of K.U.Leuven’s industrial robot arms. To visualise the motion execution by the robot, the kinematic chain (or armature) underlying the industrial robot is modeled using bones, and the mesh defining the robot’s appearance is attached to this kinematic chain (Figure 2). In this example we use an external control program that calculates the robot’s joints using inverse kinematics (via the Kinematics and Dynamics Library [22]). These robot joint values are sent to a Python using a specific middleware. The Python script is attached to a Blender-native sensor-controller-actuator system, and to each “sample time” of the Blender Game Engine (BGE) rendering loop. This Python script moves the bones (and hence the attached mesh) of the Blender-native kinematic chain. The visualisation feature of Blender is also illustrated in the markerless human motion capture example of Section 3, where the estimated joint angles coming from a particle filter are passed on to the Blender visualisation for visual verification. A video illustrating this can be found here: http://people.mech. kuleuven.be/~u0062536/files/movies/simpar/visu/
Blender for Robotics at K.U.Leuven
17
Fig. 1. Visualisation of an industrial robot
Fig. 2. The internal kinematic chain (of the Blender armature of the robot in Fig. 1)
2.2
Simulator
In addition to being a mere visualisation tool, the BGE can also be used as a simulator environment. This requires a full-duplex communication over the middleware as shown in Figure 3: the software algorithm running outside of Blender gets simulated sensor data from the simulator and sends its results back in order to let the BGE update the virtual environment within Blender. At the moment the following sensors can be simulated by the BGE (thanks to the hard work at the LAAS and ONERA research labs, for videos see [9]): a
18
K. Buys et al.
Fig. 3. Simulator blockdiagram. External programs must be given a bi-directional communication with Blender.
Fig. 4. On the left PR2 with its internal left arm kinematic chain and camera frames shown and on the right showing its camera output (Arm cameras and stereo head) to the ROS middleware
camera with configurable intrinsic and extrinsic parameters (Figure 4), a laser range finder, an inertial measurement unit (IMU), and a gyroscope; other sensors are in development. The user has different possibilities to control a kinematic chain. First he has to decide whether to send end-effector positions or joint angles. In case endeffector positions are send, the user has the choice between the Blenders internal inverse kinematics (IK) solver functions or his own IK-solver and control the armature based on joint angles and translation vectors. For the first option we added a new IK-solver to Blender 2.5 which is able to solve based on the iTaSC framework [8,24,23]. For the second option we illustrated two different ways in our code: the first based on armature control, the second based on mesh control. The second choice the user has to make concerning the control of a kinematic chain is on how to obtain the control signals. First he can make use of the BGE control structure, which allows him to contol a robot with a keyboard, mouse, . . . For this he needs to use the logic bricks that Blender provides (figure 5). Second, the user can also control the kinematic chain from a python module that connects over the middleware to an external program. This possibility is handled in section 2.3.
Blender for Robotics at K.U.Leuven
19
These accomplishments are illustrated in the markerless human motion capture example in which Blender is used as a 3D render engine to render the simulated expected measurements. In this example, the sensor feedback consists of a virtual camera image. A second example shows the Willow Garage PR2 robot simulated in Blender and gathering sensor data. A video illustrating this can be found here: http://people.mech.kuleuven.be/~u0062536/files/ movies/simpar/simu/ 2.3
Middleware
Because of the good integration of Python with the BGE, it is rather easy to connect the BGE to external programs via different types of middleware. This variety of middleware choices allows users to stay within the frameworks they are used to (as long as there are Python bindings for their prefered middleware). Python scripts are attached to the logic block of a BGE controller in the BGE logic panel which is shown in Figure 5. This panel is executed once in each BGE process loop, and allows programmers to trigger reactions based on some sensor input. These inputs are basically joystick, keyboard, distance, collisions,. . . within the original Blender context; but in the “Blender for Robotics” context, all of the above-mentioned robotics sensors can now also be used to trigger reactions of logic blocks. The latter contain logical relationships (like AND, OR, XOR,. . . ), and even full Python scripts, who contain the core robotics functionality. The activated reactions can be movements of robots, or game/task functionality (like deciding to switch to another part of the task execution). The current implementation of Blender runs the BGE in a single thread, hence the logic panel can provide only limited middleware connectivity. This
Fig. 5. Blender logic bricks, allowing to “reason” and make decisions from within Blender Game Engine’s native sensor, controller and/or actuator blocks
20
K. Buys et al.
hinders applications that require continuous synchronisation. Currently, the following middleware interfaces (in addition to the universal TCP/UDP stream and datagram connection) have been tested, at K.U.Leuven and other labs: – YARP [10] (Yet Another Robot Platform) allows the user to implement a large variety of transport protocols: TCP, UDP, Mcast, ShMem, Local,. . . Every connection has a specific type of carrier associated with it. By default YARP uses a TCP based connection. To allow various transport protocols to work with YARP, the specification has a carefully choosen data representation format, the NetType format. This NetType format chooses data types that can be represented easily in both binary and text format. YARP uses a name server to maintain a list of all YARP ports and how to connect to them. YARP also allows the programmer to implement his own carrier protocol. This means that if there is a python implementation available of the protocol, the user can implement it in the BGE by implementing the yarp::os::impl::Carrier interface. Examples that use Blender visualisation via YARP can be found at [9]. – CORBA (Common Object Request Broker Architecture) [20] uses an interface definition language (IDL) that specifies how the interface to the network looks like. On top of this it has a mapping from a IDL file to the specific programming languages. An application has to initialise a Object Request Broker (ORB) that manages the network requests. The language mapping of the IDL format is highly dependent on the object-oriented architecture of the programming language. For the BGE Python interface this is not a problem, when the user program at the other side is using the C programming language however, this can be less straight forward. For the moment, we only use the remote procedure call (RPC) functionality of CORBA in the BGE, other services that CORBA provides (like a time service) could be a future extension of the middleware interface. An example that uses Blender visualisation via CORBA is shown in [7], which illustrates our work on multiple target tracking and localisation/detection algorithms. In this example the BGE is connected over CORBA to an Orocos Real-Time Toolkit (RTT) [25] component containing the multiple target tracking and localisation algorithm. A video illustrating this can be found on: http://people.mech.kuleuven.be/~u0062536/ files/movies/simpar/mttl/ – ROS [27] (Robot Operating System) uses a remote procedure call (XMLRPC [1]) approach, using HTTP as the transport and XML [26] as the encoding. ROS’s master service provides the API for nodes to register themselves as a data publisher, a subscriber to a topic, and/or providing a service. The parameter server is part of the master and stores configuration parameters also based on XML-RPC. By using a Python script it is possible to implement a ROS slave node in the BGE. Instead of using the normal callback handlers the BGE scheduling forces the programmer to use an alternative method and lets the programmer use a wait for message type of routine. This will get the latest message out
Blender for Robotics at K.U.Leuven
21
of the queue or waits for a message within a certain time-out range. This way the programmer can keep the Blender loop deterministic without interfering the communication with the ROS master. An example that uses Blender via ROS is described in Section 3)
3
Example: Markerless Human Motion Capture
This section explains the current developments at K.U.Leuven to use the Blender Game Engine as, both, a visualisation tool and as a virtual lab (simulator), for the purpose of markerless human motion capturing as shown in Figure 6. That is, to estimate a human pose (i.e. joint angles in a skeleton model of the human) on the basis of video input from a real or simulated camera in a Blender environment. The estimated joint angles of the human skeleton model can then be used by a post-processing algoritm (e.g. for intent estimation, or gait classification, or gesture recognition, etc.). The elaborate human model of Figure 7 is imported from the MakeHuman project [3] into the simulation part of the algorithm (external to Blender). A virtual lab is set up in Blender, in which a similar human model is imported and moved around. A particle filter estimator (using the Bayesian Filter Library (BFL) [12]) tracks the person’s movements, by estimating it’s pose and sends the results to the Blender visualisation.
Fig. 6. Block diagram of the markerless human motion capture setup
All middleware communication is done using ROS. The measurements consist of camera images of the human. They can come from the real-world lab or the virtual lab. The images are converted into an OpenCV [17] structure, and then sent to the ROS estimator node over a time-stamped image ROS topic. For its operation the estimator has to calculate the expected measurements, i.e. the expected camera images, for each of its particles. To this end the estimator sends the particle state, which is a combination of position of the human centroid and the joint angles of the whole kinematic chain, using a ROS message to the BGE simulation. The BGE provides the 3D rendering, and returns either a black&white or a full color image. The binary image allows to evaluate poses based on a background-subtracted image to do simple pixel match counting. The full color image allows more complex color and feature matching algorithms.
22
K. Buys et al.
Fig. 7. Human model with internal kinematic chain
In our example, in order to evaluate each particle, the particle filter uses a pixel-based image correspondence calculation: K p(yk |˜ yk ), p(y|˜ y ) = Πk=1
(1)
with K the number of pixels, yk and y˜k the expected and the measured pixel values. In this calculation all pixels are assumed to be conditionally independent given the human pose. For each of the pixels, a 2D Gaussian pattern is used to calculate the local probability: y ) = Π1 p(yk |˜ yk ) + Π2 p(yk |˜ yk−1 )+ p(yk |˜ yk+1 ) + Π4 p(yk |˜ yk−w ) + Π5 p(yk |˜ yk+w ), Π3 p(yk |˜
(2)
with Π the Gaussian probability pattern and w the image width (see Figure 8).
Fig. 8. Pixels used in Gaussian pattern
Blender for Robotics at K.U.Leuven
23
The code is available from: http://svn.mech.kuleuven.be/repos/orocos/ trunk/kul-ros-pkg/human_motion_capt/; videos demonstrating the code are available on: http://people.mech.kuleuven.be/~u0062536/files/movies/ simpar/mhmc/
4
Discussion and Future Work
A common need in all robotics labs is the use of Blender as a flexible simulation environment, capable of connecting to different robotics frameworks by a multitude of middleware solutions. The use of Blender has some advantages: the first is that it is a Open Source project, providing the user with lot of freedom. Furthermore it has the physics engine Bullet integrated, it allows for photo-realistic rendering, has a structured code layout and has a large community supporting it. However, there are currently some disadvantages of using Blender, some of which will be handled in future work: (i) Blender has a steep learning curve for beginners and a complicated graphical user interface (GUI) expecting some effort from the beginner to learn it; (ii) Blender has no external time synchronisation support; (iii) the Collada interface doesn’t support the full 1.5 standard yet; and (iv) the Blender API documentation is often incomplete. This paper showed how Blender can be used as a visualisation tool, how it can be extended to a simulation environment, and by what middlewares robotic frameworks can connect to it. To help overcome the disadvantages ongoing shared work at the major “Blender for Robotics” labs is focused on the following: first writing a wrapper around the different types of middlewares, providing a generic interface to robotic frameworks on the one hand, and to Blender and its game engine on the other hand. A second major missing feature when doing realistic simulation in a system built from various “legacy” components, is the need to let the activity of all components be triggered by one single global reference clock. In principle, Blender allows to run its internal graphical rendering loop in an event-triggered way, which is more than what most current robotics frameworks offer to their own components. Another shared development effort is the implementation of the Collada 1.5 standard [2,16] to import/export scenes, robots, kinematic chains, etc. The Blender developers have already realised a partial integration of Collada 1.5 into Blender 2.5, but this implementation does not support armatures, which is an important features in robotics applications. The last disadvantage of Blender we are trying to change is the GUI, we are adding a ’Robotics Panel’ to Blender allowing users to place, configure and use robots, sensors and environments without having to know the extensive interface that Blender offers.
24
K. Buys et al.
Acknowledgements The Blender for Robotics project is mainly driven by the research labs of ONERA and LAAS in Toulouse, France. The authors acknowledge the Concerted Research Action project on “Global real-time optimal control of autonomous robots and mechatronic systems” funded by K.U.Leuven [GOA/2010/011].
References 1. XML-RPC, http://www.xml-rpc.com/ 2. Barnes, M., Finch, E.L.: COLLADA—Digital Asset Schema Release 1.5.0 (2008), http://www.collada.org (last visited March 2010) 3. Bastioni, M., Flerackers, M.: Make human: Open source tool for making 3d characters (2007), http://www.makehuman.org/ (last visited 2010) 4. Bruyninckx, H.: Blender for robotics, http://wiki.blender.org/index.php/ Robotics:Index (accessed online June 20, 2010) 5. Bruyninckx, H.: Open Robot COntrol Software (2001), http://www.orocos.org/ (last visited 2010) 6. Coumans, E.: Bullet physics library (2005), http://bulletphysics.org/ (last visited March 2010) 7. De Laet, T.: Rigorously Bayesian Multitarget Tracking and Localization. PhD thesis, Department of Mechanical Engineering, Katholieke Universiteit Leuven, Belgium (May 2010) 8. De Schutter, J., De Laet, T., Rutgeerts, J., Decr´e, W., Smits, R., Aertbeli¨en, E., Claes, K., Bruyninckx, H.: Constraint-based task specification and estimation for sensor-based robot systems in the presence of geometric uncertainty. The International Journal of Robotics Research 26(5), 433–455 (2007) 9. Echeverria, G., Lemaignan, S.: Blender for robotics video channel, http://vimeo.com/groups/blenderandrobotics/videos (last visited June 2010) 10. Fitzpatrick, P., Metta, G., Natale, L.: YARP: Yet Another Robot Platform (2002), http://eris.liralab.it/yarp/ (last visited 2010) 11. Fleury, S., Herrb, M., Chatila, R.: GenoM: a tool for the specification and the implementation of operating modules in a distributed robot architecture. In: Proceedings of the 1997 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 1997, Grenoble, France, pp. 842–848 (1997) 12. Gadeyne, K., De Laet, T.: BFL: Bayesian Filtering Library (2001), http://www.orocos.org/bfl (last visited 2010) 13. Gerkey, B., Vaughan, R., Howard, A., Koenig, N.: The Player/Stage project (2007), http://playerstage.sourceforge.net/ 14. Koenig, N., Howard, A.: Design and use paradigms for gazebo, an open-source multi-robot simulator. In: Proceedings of the 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2004, Sendai, Japan, vol. 3, pp. 2149–2154 (2004) 15. Mallet, A., Fleury, S., Bruyninckx, H.: A specification of generic robotics software components and future evolutions of GenoM in the Orocos context. In: Proceedings of the 2002 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2002, Lausanne, Switzerland, pp. 2292–2297 (2002) 16. NetAllied Systems GmbH. OpenCOLLADA (2009), http://opencollada.org (last visited March 2010)
Blender for Robotics at K.U.Leuven
25
17. OpenCV. Open computer vision, opencv (2001), http://www.intel.com/research/mrl/research/opencv/ 18. Quigley, M., Conley, K., Gerkey, B., Faust, J., Foote, T.B., Leibs, J., Wheeler, R., Ng, A.Y.: Ros: an open-source robot operating system. In: ICRA Workshop on Open Source Software (2009) 19. Roosendaal, T.: Blender (2002), http://www.blender.org (last visited March 2010) 20. Schmidt, D.C., Kuhns, F.: An overview of the real-time CORBA specification. IEEE Computer special issue on Object-Oriented Real-time Distributed Computing 33(6), 56–63 (2000) 21. Smith, R.: ODE: Open Dynamics Library (2000), http://www.ode.org (last visited March 2010) 22. Smits, R.: KDL: Kinematics and Dynamics Library (2001), http://www.orocos.org/kdl (last visited 2010) 23. Smits, R., Bruyninckx, H., De Schutter, J.: Software support for high-level specification, execution and estimation of event-driven, constraint-based multi-sensor robot tasks. In: Proceedings of the 2009 International Conference on Advanced Robotics, Munich, Germany (2009) 24. Smits, R., De Laet, T., Claes, K., Bruyninckx, H., De Schutter, J.: iTaSC: a tool for multi-sensor integration in robot manipulation. In: IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems, MFI 2008, Seoul, South-Korea, pp. 426–433 (2008) 25. Soetens, P.: A Software Framework for Real-Time and Distributed Robot and Machine Control. PhD thesis (May 2006), http://www.mech.kuleuven.be/dept/ resources/docs/soetens.pdf 26. W3C. XML: Extensible Markup Language (2008), http://www.w3.org/XML/ 27. Garage, W.: Robot Operating System, ROS (2009), http://www.ros.org (accessed online Januari 26, 2010)
