Development and experimentation of a real-time ...

Simulation Applications

ROSUnitySim: Development and experimentation of a real-time simulator for multi-unmanned aerial vehicle local planning

Simulation: Transactions of the Society for Modeling and Simulation International 2016, Vol. 92(10) 931–944 Ó The Author(s) 2016 DOI: 10.1177/0037549716666683 sim.sagepub.com

Yuchao Hu and Wei Meng

Abstract In this paper, we present a novel real-time three-dimensional simulation system, ROSUnitySim, for local planning by miniature unmanned aerial vehicles (UAVs) in cluttered environments. Unlike commonly used simulation systems in robotic research—e.g., USARSim, Gazebo, etc.—in this work our development is based on a robot operation system (ROS) and with a different game engine, Unity3D. Compared with Unreal Engine, which is used in USARSim, Unity3D is much easier for entry level developers and has more users in the industry. On the other hand, as we know, ROS can provide a clear software structure and simultaneous operation between hardware devices for actual UAVs. By developing a data transmitting interface, a communication module and detailed environment and sensor modeling techniques, we have successfully glued ROS and Unity3D together for real-time UAV simulations. Another key point of our work is that we propose an efficient multi-UAV simulation structure and successfully simulate multiple UAVs, which is a challenging task, running 40Hz LIDAR (Light detection and ranging) sensing and communications in complex environments. The simulator structure is almost the same as real flight tests. Hence, by using the developed simulation system, we can easily verify develop flight control and navigation algorithms and save substantial effort in flight tests.

Keywords Unity3D, simulator, unmanned aerial vehicle, navigation, guidance, robot operating system

1 Introduction Recently, small-scale unmanned aerial vehicles (UAVs) have attracted great interest and attention from academic research groups worldwide because of their great potential in both military and civil applications. Many groups have constructed their own UAV platforms.1–6 Especially, navigation and guidance of UAVs in GPS-denied environments is still a challenging task.7 Nevertheless, generally speaking, the implementation of UAVs is costly and timeconsuming. Simulation technologies have been widely used by academic research as a cost-effective way to accelerate processing procedures. They are especially important in the case of UAV development due to the complexity involved in outdoor environments and network communications. Therefore, simulation is an effective way to detect and prevent unnecessary malfunctions of hardware, software and automatic flight control systems.1 Simulation technologies can provide great help to verify the algorithms and identify potential problems before flight tests and to make physical implementation smooth and successful.8,9 Some well-known recent works have

demonstrated the capabilities of three-dimensional (3D) simulations, such as OpenSim,10 the Virtual robot experimentation platform (V-REP),11 Delta3D,12 USARSim,13 and Gazebo,14 etc. OpenSim is one of the earliest attempts, in 2001, at a 3D robotic simulator. It is a 3D simulator for autonomous robots that uses OpenGL for real-time rendering of the robot environment. But the development is no longer active since October 2008 and also it is not good at real-time rendering of robot environments. V-REP is a commercial product and not open-source.11 Gazebo is a mature and good open-source simulator but it is developed and supported mostly on Linux and not meant for hybrid simulation.14 To further address the challenges of hybrid simulation, the unified system for automation and robotics simulator (USARSim), a high-fidelity simulation tool Temasek Laboratories, National University of Singapore Corresponding author: Wei Meng, Temasek Laboratories, T-Lab Building, 5A, Engineering Drive 1, Singapore 117411. Email: [email protected]

Downloaded from sim.sagepub.com at NATIONAL UNIV SINGAPORE on October 11, 2016

932

Simulation: Transactions of the Society for Modeling and Simulation International 92(10)

based on Unreal technology, gained more researchers’ attention, as it represents robot automation and behavior, and also renders user interface elements precisely.13 However, USARSim-based simulation tools have not been updated regularly recently. According to our experience, the available simulation systems are not suitable for multiUAV real-time tests, especially when the 3D environments are large-scale and complex. As you can see, there are many 3D simulators already available. So, why do we still need to develop a new UAV simulation system? The main reason is that most open-source simulation systems find it difficult to handle multiple UAVs simultaneously, sensing in complex and high fidelity environments. In addition, it is hard to modify the developments in USARSim. Another reason is that, for industrial applications, it is required to develop a complete simulation system, including environment sensing modeling, flight control and navigation algorithms. The UAVs are expected to cooperatively and autonomously operate in GPS-denied foliage or urban environments. The LIDAR-based simultaneous localization and mapping (SLAM) algorithm needs to be simulated in real-time. According to our knowledge, most simulation systems are not ‘‘real’’ enough. Firstly, the environments—especially, the collision level of the trees and buildings—needs to be modeled as realistically as possible. Secondly, UAV sensors, such as LIDAR and cameras, are required to be modeled as real and need to run in a high-frequency level in a dense and harsh environment. In this paper, a novel hybrid 3D simulator is developed based on robot operating system (ROS) and a game engine, Unity3D. ROS15 is an open-source framework designed to provide an abstraction layer to complex robotic hardware and software configurations. It provides libraries and tools to help software developers create robot applications and has found wide use in both industry and academia. Gazebo, which was mentioned above, also originates from ROS. Unity3D is a more flexible and powerful development platform for creating multiplatform 3D and two-dimensional (2D) games and interactive experiences. This game engine, which supports almost every platform, is chosen to be the simulation server. Besides game developments, Unity3D could also be employed by academic research.16 It has been developed as a simulator in geographic information systems,17 the communication system for Moon base simulated scenarios, and wind energy development.18 These implications prove that Unity3D can also be used in academic area with vivid and highly interactive performance. The main reason of choosing Unity3D as the visualization tool is that it is easier for entry-level researchers to develop their UAV applications, especially, to build real-time sensor models, physical models, etc. In addition, since the programing language for Unity3D is C#, it is easier for the researchers to understand and develop further.

The main contribution of this paper is that we have developed an application driven multi-UAV simulator based on ROS and Unity3D. To the best of our knowledge, our work is the first to involve the Unity3D technique in UAV sensing and planning simulation research. We focus particualarly on a novel and effective way to simulate multiple UAVs simultaneously. In our work, the interface between ROS and Unity3D has been developed based on the TCP/IP protocol. The LIDAR sensor is one of the most popular sensors used in GPS-denied environments. In this work, we focus on sensor modeling and local planning in GPS-denied environments. GPS does not function perfectly with our simulator. The key component for the SLAM algorithm is modeled in detail in the Unity3D script. Environment modeling—including trees, terrain, buildings, etc.—is also developed and will be introduced in our simulation technique section. The preliminary version of our simulator was presented in Meng et al.19 In this earlier work, we gave a brief description of the structure of the simulator: ROSUnitySim. In the current version of the work, we have made extensive extensions and new contributions: (1) The structure of the simulator has been improved to support efficient multi-UAV simulations. (2) Detailed technical issues, such as the communication protocol, client/server structure, the interface between ROS and Unity3D, have been added. (3) Performance analysis of the simulator has been provided. (4) More experiment results and analysis have been included to verify the efficiency and success of the simulator. The remainder of the paper is organized as follows. The main structure of the developed ROSUnitySim is stated in Section 2. Some key technical issues involved are presented in Section 3. Simulator performance analysis is addressed in 4. In Section 5, both simulation results and flight test results have been reported. Section 6 addresses the impact of our work. Section 7 concludes the paper.

2 ROSUnitySim development Before going on to illustrate the detailed techniques in our developed ROSUnitySim, the structure of the simulator is briefly introduced. As addressed above, in this work, ROS and Unity3D have been combined to work out a real-time high fidelity simulator closed as much as possible to its real-world case. The main difficulty of UAV simulation is how to simulate multiple UAVs efficiently in a real-time manner. In Figure 1, the main structure of ROSUnitySim is presented. As seen from the figure, a game development structure, server and client mode has been adopted. By using this kind of game mode, our simulator can easily handle multiple UAVs if the workstation CPU is powerful enough. Each UAV has a ROS section and a Unity3D section. The Unity3D server and clients will serve as sensing

Downloaded from sim.sagepub.com at NATIONAL UNIV SINGAPORE on October 11, 2016

Hu and Meng

933

Figure 1. Structure of ROSUnitySim. GCS: Ground Control Station.

environments in real cases. The communication server section will serve a communication media function, such as MANET and other commercial products for ad hoc communication. Figure 2 shows a more detailed system level structure, which is used by our project. Network and

virtual machine technology are used to support the whole simulation system. This structure is mainly for the purposes of easy expansion. In one workstation/PC, one or more UAV can be simulated. And if necessary, more workstations/PCs can be added easily to provide a more powerful computation resource. The virtual machines and Unity3D server/client also can be set up in different workstations/PCs, because all the communication systems between them are based on a network. Hence, the developed simulator system is scalable. Users can set up their own structure based on their requirements. In the following, we will give brief introductions to each portion of ROSUnitySim.

2.1 Unity3D section Unity3D is a game development ecosystem: a powerful rendering engine fully integrated with a complete set of

Figure 2. System level structure.

Downloaded from sim.sagepub.com at NATIONAL UNIV SINGAPORE on October 11, 2016

934

Simulation: Transactions of the Society for Modeling and Simulation International 92(10)

intuitive tools and rapid workflows to create interactive 3D and 2D content, easy multi-platform publishing, thousands of quality, ready-made assets in the Asset Store and a knowledge-sharing community.20 Unity3D allows game developers to create games that delight players on any platform. The high fidelity model, live environment and ability to adapt to any platform which can be achieved by Unity3D make it the best choice of our 3D high-fidelity simulator. Moreover, natural environment and weather conditions can be simulated in Unity3D too, to monitor the possible complex outdoor influence on UAV in physical test through simulations. Though this may still be far from natural light, we can continue to employ the potential of Unity3D in environmental simulation. Another crucial advantage of Unity3D compared to our existing simulator is that the collision regions of objects can be as narrow and detailed as the shape of the objects themselves. 2.1.1 3D environment modeling. In order to improve the confidence level of the simulation system, 3D virtual environments need to be modeled as realistically as possible. There are two parts to consider when modeling objects in the simulator. The first is mesh, which determines the visual shape of the object: The more meshes used, the more accurate the depiction of the object. But meshes require computer resources, so a balance should be considered based on all requirements. For visualization purposes, the corresponding textures will be added to the surface of the mesh. Figure 3(a) shows the tree’s meshes (in blue) and the rendered results with textures in Unity3D. The other is a collider, which is used for raycasting, collision detection and other physic simulations. Just like the mesh, the detail of the collider should be modeled according to requirements. Reducing the detail level of the mesh or the collider may be an alternative choice if Unity3D takes up too much computer resource. In the case of the UAV model shown in Figure 3(b), which we developed for simulation purposes, components such as the propeller, the body frame and the motor have been implemented. This model can be easily adapted into a test of physical control if required. In Figure 4, one UAV is flying autonomously in our developed 3D virtual urban environment. We can also see a camera view of the UAV, along with a 2D sketched map on the left and right bottom of the figure, respectively. 2.1.2 UAV sensor modeling. All hardware sensors mounted in our UAVs are modeled in the simulator, including the laser scanner, camera and so on. LIDAR The LIDAR sensor used in our UAV system is Hokuyo UTM-30LX. The main features modeled in the simulator are: (1) the detectable range is .1–30m; (2) the scanning angle is 270 degrees; (3) the scan speed is 25ms; (4) at

Figure 3. Tree and UAV modeling. (a) Modeling of tree. (b) UAV 3D physical model.

Figure 4. UAV flies in city.

W60 x D60 x H87mm it is compact; (5) and at 370g weighs little. There are two LIDAR sensors mounted on our UAV platform. One of them is placed statically on top of the platform. The main purpose of this LIDAR sensor is used by SLAM to estimate the location of the UAVs. Another LIDAR sensor is mounted on the front part of the UAV

Downloaded from sim.sagepub.com at NATIONAL UNIV SINGAPORE on October 11, 2016

Hu and Meng

935

Figure 6. Camera render.

Figure 5. LIDAR sensor. (a) LIDAR sensor in operation. (b) Real LIDAR sensor. Figure 7. Fast rendering.

platform. Specially, this LIDAR sensor continuously tilts (nods up and down) at a certain speed to obtain 3D obstacle information. These two LIDAR sensors’ data is quite important, since it will be used for localization and mapping, and also for 3D path planning. As seen in Figure 5(a), two LIDAR sensors on one UAV are operating simultaneously in a forest environment, where red color represents the nodding LIDAR and blue represents the horizontal static LIDAR. For comparison purposes, the placement of the two LIDAR sensors mounted on our real UAV platform is shown in Figure 5(b). According to a LIDAR sensor’s specifications, there is a total of 1081 pieces of data for each 270 degree scan. The scanning frequency is around 40Hz. In Unity3D, one straightforward technique is to use the Physics.Raycast scripting API to develop a customized LIDAR sensor model. In our Raycast script, scanning range and angles are calculated based on the UAV’s condition. Both LIDAR sensors developed can be run simultaneously. Usually, Raycast in Unity3D runs in the main thread. When we run several UAVs from the same workstation, Raycast calculation for each UAV can be done in different instances. According to our experience, if the simulated 3D map is complex, then the Raycast will take up almost all of the CPU’s resources. In our development, we have implemented some novel ideas: for example, for Raycast, the calculation only considers the environment information near each UAV, and the detailed level of the collider may be reduced to speed up the calculation.

To allow the LIDAR sensing to be as real as possible, in Unity3D we can set the sampling frequency to a fixed value—for example, 40Hz—even though sometimes the workstation can support higher frame rates. For measurement errors, we can add noise to the scan data to simulate a real laser’s data. Also, collider detail level should be considered. Camera Camera sensor development is easy in most game engines. In our system, one camera is mounted statically on the front of the UAV platform. The camera model is from the original camera of Unity3D. The field of view (FOV) is 60 degrees, and perspective projection is used. The main purpose of the camera is target detection. Figure 6 shows the rendering of a camera. To accelerate processing, only objects near the camera are rendered. So, the camera’s output only includes these rendered objects: objects out of range are ignored. 2.1.3 Fast rendering. For real-time simulation tests, rendering speed is a very important requirement, which determines if the whole system can touch the frequency of the real system. To increase rendering speed, some modifications are required. For example, if the performance of laser-based algorithms needs to be tested, unrelated elements can be removed: Textures, small detailed objects, light, shadows, and so on. Figure 7 shows one example

Downloaded from sim.sagepub.com at NATIONAL UNIV SINGAPORE on October 11, 2016

936

Simulation: Transactions of the Society for Modeling and Simulation International 92(10) During the actual simulations, the server and the client both allow three kinds of user view: Walkthrough, flythrough and UAV. Users can control the view position in walkthrough and flythrough modes to monitor the moving speed and trajectory of the UAV from a macro perspective. As stated in the top-left corner of Figure 4, it illustrates what the walkthrough view looks like: Similar to that of a pedestrian—blocked by objects. On the other hand, the flythrough mode is not affected by such environments. The UAV view is from the back of the activated UAV and follows its movements. Users cannot adjust the view angle and position for the UAV view. It is appropriate to investigate the movements of the UAV more specifically in terms of stability and smoothness. Furthermore, as shown in the bottom-right of Figure 4, the server has a map which allows users to investigate the global position of the UAV; indicated by the red circle in the map.

2.2 ROS section

Figure 8. Server (a) and client (b) configuration interfaces.

where not all the leaves and textures are rendered, and the objects out of laser range have also been removed. Avoidance of rendering unnecessary objects can boost rendering speed, which can save CPU resources. If the camera-based algorithms need to be tested, the textures should remain, but the level of detail can be reduced for objects far from the camera. 2.1.4 User interface. Figure 8(a) shows the server configuration interface for Unity3D simulations. There are some environment types for users to select. The number of UAVs should be set for initialization. In the client interface, shown in Figure 8(b), ROS data and the image port should be set for communication between Unity3D and ROS; the parameters of the IP address and the port should point to the server. After a connection has been established, the whole simulation will be running.

ROS is an open-source operating system for robots, which was originally developed in 2007 at the Stanford Artificial Intelligence Laboratory.15 The modular architecture and easy-to-sync threads make ROS flexible and convenient. Same nodes can be multiplied for use in different operations. For ease of development and set-up, we choose ROS as a high level algorithm developing and running environment. On the hardware side, the on-board x86 computer MasterMind is powered by ROS Ubuntu. In simulation environments, MasterMind is replaced by a virtual machine, which can create an isolated environment in workstations running on Ubuntu. Except for some hardware drivers, this virtual environment can be considered the same as that in real hardware. In ROS section of ROSUnitySim, we can build all the algorithms which will be implemented in the CPU sections of the UAVs, i.e., MasterMind, Gumstix, etc. In our case, we developed ROS nodes, such as: SLAM, path planning, Octomap conversion, a UAV dynamic model, Control logic, Communicator, etc. Here we give a brief introduction to several nodes which play important roles in our simulation system. 2.2.1 Simulated UAV dynamic model and control. In order to simulate UAV control sections, including inner and outer loop control laws, we have used our identified quadrotor model, including: Roll/pitch dynamics, yaw dynamics, and heave dynamics. For the outer loop control, we implemented a robust and perfect tracking (RPT) control law.21 We put all control parts together to serve as a simulated UAV dynamic model and control. The output of this node is the UAV’s reference pose. Actually, in the most gameengine simulators, the UAV dynamic model is not

Downloaded from sim.sagepub.com at NATIONAL UNIV SINGAPORE on October 11, 2016

Hu and Meng

937

provided by their physical engines. If users want to use these simulators, they have to build up their own UAV model for simulations. Unity3D does not have an aerodynamic physical engine either. For consideration of easy modification and extension, the UAV dynamic model is executed in ROS with the controller as a single node. 2.2.2 Control logic. The control logic node is important, especially when UAVs are handling complex tasks. In our case study, it will mainly serve as a task manager: i.e., global planning, multi-UAV collision avoidance, information sharing, etc. 2.2.3 Communicator. When the control logic node wants to share information with other UAVs, the communicator is the only path. In the simulations, it will connect to a communication server, which is like a virtual router simulated by software, to exchange data with other UAVs’ communicators. A more detailed technique will be discussed below. In real hardware environments, the communicator and communication server would be replaced by MANET or similar products. 2.2.4 Unity socket. The unity socket node, which connects to the Unity client via TCP/IP, mainly focuses on data exchange between ROS and Unity3D. All the sensor data—such as, laser, Inertial Measurement Unit (IMU) and sonar—generated by Unity3D is sent to the Unity socket, which then transfers the coordinates to the ROS standard frame with x forward, y left and z up. The node of the UAV dynamic model and control is responsible for generating the position and orientation of the UAV and sending it to the Unity client via the Unity socket node. Hence, the Unity socket can be seen as a bridge between ROS and Unity3D. A customized driver, connected to Gumstix or Pixhawk, would take its place when the system is working in a real hardware environment. Additionally, to ensure the data is transferred correctly, we use the checksum technique.

2.3 Communication section 2.3.1 Communication protocol. The purpose of the communication protocol is to allow communication between UAVs and the ground control station using a variation of the TCP/IP communication protocol implemented in both C and C++ . At any time any station can begin transmission of data to any other station. The data transmitted will be in one of two pre-defined formats: Command data or sensing information. The communication server module is stand-alone and uses Linux to connect with ROS. The basic structure of the developed communication module is presented in Figure 9 where, in the ROS section, each UAV has a

Figure 9. Basic structure of a communication module.

communication node to send and receive information to and from GCS and other UAVs. Simulation of a real-world scenario brings us to the addition of the ‘‘Communication Server’’ module, which acts as a router and connects to all stations. The main functionality provided by this module is to be able to receive data packets from any source station, and retransmit them to the intended destination station. 2.3.2 Communication server module. Handshake protocol When any station attempts to connect with a communication server, the server expects to receive an ‘‘identifier’’ packet containing the station’s unique ID, before allowing the socket to remain open for further communication. The unique ID is in the format of a character array of size two, as it requires two separate variables to identify its batch and plane numbers. However, it is not practical to store the ID based on the data format of a two-dimensional array, which is why we use a basic hash algorithm: liner identity = 4 3 (batch num) + plane num

ð1Þ

to reduce the 2D arrays, based on their maximum values, to a corresponding linear identity. Finally, we will need to store an internal map of which socket is open with which station. As such, we store the socket identifier for the open connection together in an array by setting the hashed (linear) identity of the station as the array index. 2.3.3 Multi-thread programming. The connection (socket) between the server and each station should hold equal priority. As such, to ensure fairness, we allocate each connection to a single thread in the process, and schedule each thread using a round-robin scheduling protocol. 2.3.4 Preventing socket clash. While the re-transmission of data follows the ‘‘store and forward’’ principle—the server stores individual packets until the entire message is received, before forwarding it to the destination—the data itself is not actually backed-up on the server. As such, a failure scenario would be if two or more stations try to

Downloaded from sim.sagepub.com at NATIONAL UNIV SINGAPORE on October 11, 2016

938

Simulation: Transactions of the Society for Modeling and Simulation International 92(10)

transmit a message to the same destination within a short timespan. The solution we have implemented to prevent this is to use MUTEX to lock the transmission process within each individual socket. Now, we do not lock the entire transmission function itself—this is not optimal as it forces the system to only be able to transmit to one station at a time. For example, consider the scenario that, within a short time span, these two transmissions are attempted:  

Station 1 transmits to Station 3 (slightly sooner than below); Station 2 transmits to Station 4 (slightly later than above).

In this case, if we have a single MUTEX on the transmit function, then Station 1 will lock the MUTEX before it transmits to Station 3. Then, when Station 2 attempts to lock the MUTEX before it transmits to Station 4, it will be blocked because the MUTEX has already been locked by Station 1. This is not ideal because it will not cause problems if there are simultaneous ongoing transmissions to different destinations. Instead, what we do is to declare multiple MUTEX variables that can be locked, and each station has its own dedicated MUTEX. This way, only when there is an ongoing transmission to a specific station will its corresponding MUTEX be locked.

called in the main thread. As this only uses one of the CPU core’s resources, the simulation will be slow in a multiple-UAV environment. The other solution is to use a client/server structure, as in network games. In this structure, the main task of the server is information receiving, storage and distribution via a network: Each client can share its UAV state with the other clients through the server. The server is also responsible for rendering the environment and showing the state of the UAVs. From the client’s point of view, just one UAV is handled in one client instance, so it only processes two lasers’ data and one CPU core’s resource, which is sufficient for computing raycasting in 40Hz by using Unity3D built-in APIs. To further improve the performance, the client view is not rendered in detail at the server—it just retains basic environment information. For example, textures, light and shadow are removed in teh client, and colliders are kept for raycasting computation. This multiple instance structure fully uses the CPU’s resources. It is also very easy to extend two or more workstations to support more UAV simulations as all the communication functions are based on a network instead of local memory. Due to these advantages, we chose the second solution: It can handle as many as three UAVs in a workstation with six CPU cores.

3.2 Interface between ROS and Unity3D 3 Key technical issues 3.1 Client/server structure based on Unity3D During development of the simulator, we found that only the Raycast API of the Unity3D engine can be used in the main thread. This means that, if all the UAVs’ laser data is generated in one Unity3D instance, just one CPU core can be used to handle the data generation process. Because the Raycast API of Unity3D is inefficient, this traditional structure will cause the simulation to run slowly. In addition, the computation resource of other CPU cores is also not fully used. To overcome these issues, we found two potential solutions. One is the Unity3D technology known as compute shaders, which uses the GPU to handle computing tasks. In this solution, each raycast will be parallel-computed in different cores of the GPU so it can reduce the total processing time of all 1081 laser ray lines. However, to use this technology we must transfer the environment information from the CPU’s memory to that of the GPU, then return the results from the GPU’s memory to the CPU’s. This is very slow, taking about 20ms. There is not enough time to process other things—such as the raycasting itself— because we want the frequency to touch 40Hz. Moreover, when multiple UAVs are added to the simulator, the situation deteriorates. Because Unity3D limits the number of threads, all APIs, including compute shaders, must be

As described above, the whole environment part of the simulation is implemented in Unity3D, which can be separated into client and server. For controlling the UAVs, a ROS is chosen to process the high level algorithms which control the UAVs’ behavior: Such as, SLAM, path-planning, task management, and so on. So, ROS needs to receive sensor information from the Unity3D part and sends the states of UAVs to Unity3D. For this communication structure, the client part of Unity3D is considered to be the server and ROS part is the client. The exchange of data packets is achieved through the TCP/IP protocol. To make sure the data is transferred correctly, a checksum algorithm CRC16 is used to verify every package of data. This interface between ROS and Unity3D has been developed to support multiple UAV applications. ROS node and the Unity socket will communicate with the preset IP address, which should be that of the Unity3D server. For every spin, the ROS client receives laser scan data (a 1081 data array of 270 degrees, the same as a real 30m URG laser scanner) and publishes it as a LaserScan message on the topic ‘‘laser/horizontal/scan.’’ It also publishes the transformation required by ROS based on the state of the Unity3D UAV model. This node computes the transformation of a vertical laser scanner based on the x, y and z rotation of the Unity3D laser model. With the help of the TCP/IP protocol, wireless communication between different Unity3D servers and clients is possible.

Downloaded from sim.sagepub.com at NATIONAL UNIV SINGAPORE on October 11, 2016

Hu and Meng

939

Figure 10. Unity3D server CPU usage.

Figure 11. Unity3D client CPU usage.

Originally, we implemented a local interface which allowed Unity3D and C++ control logic to access the same memory for data exchange. However, this interface can only be used on a Windows platform, which means it can’t communicate with ROS, which usually runs in Ubuntu. Using a network protocol instead not only solves this issue, but also realizes communication between a remote server and its client. Currently, we have managed to develop a ROS/Unity3D interface, which can support multiple UAV data transmissions. Additionally, for images captured from the simulator’s camera transmissions, another channel is created, which means the camera data processing is standalone. The Unity client listens to a special port, and when an image receiver connects to it from ROS, it will send images at a particular frequency. We know that image transmission and processing is a high load task, so this design separates the image task from other tasks.

3.3 Distributed computation Thanks to the network-based client/server structure, the whole simulation task can be separated into several subtasks, whic can be assigned to any PC or workstation, as long as they are in the same network. For Unity3D, the server and each client are designed as standalone items, so they can be set up either on several common PCs or on a powerful workstation. And in ROS each node can also be put in a different computer, which is more flexible—useful when simulating multiple UAVs. From the results of our experiment, a six-core workstation can only handle three UAVs. If we want to process more UAVs at the same time, more PCs or workstations can be added easily without any

software modification. Furthermore, in most situations, three common dual-core PCs have more powerful parallel computation ability than a six-core workstation, and their total price is less than a workstation’s. So, for considerations of economy, distributed computation support provides more choice when simulating multiple UAVs.

4 Performance analysis 4.1 CPU Usage Figures 10 and 11 show the performance of each Unity3D instance, when the FPS is not limited. We can see the CPU proceeding time of each frame of both the Unity3D server and the client is below 16ms, which means that the sensors’ data generation speed can be as fast as 60Hz, or even more. To reduce CPU usage, the FPS in Unity3D is limited to 40Hz, which meets the frequency of a simulated laser scan. In a workstation, one server and three clients are executed, the graphics configuration of each instance is 640 x 480—the fastest speed. When idle (with ROS not started), the CPU usage of the server is 2%, and that of each client is 7%. With ROS in operation, each ROS instance takes about 20% of CPU usage, while Unity3D’s CPU usage remains the same. The total CPU usage is about 85%. Note that the CPU usage of ROS is based on the complexity of the algorithm: We can see the different CPU usages in the multi-UAV experiment below, which uses another algorithm. 4.1.1 Performance for multiple simultaneous UAVs and computers. To test the performance for multiple UAVs

Downloaded from sim.sagepub.com at NATIONAL UNIV SINGAPORE on October 11, 2016

940

Simulation: Transactions of the Society for Modeling and Simulation International 92(10)

Table 1. Configuration of workstations. #

CPU type

Freq.

#1 #2

E5-2630 v2 E5-2630 v3

2.6 GHz 2.4 GHz

Cores

Cache

6 8

15M 20M

Table 2. One unmanned aerial vehicle (UAV) in Workstation #1.

Table 5. Three UAVs in Workstation #2.

Program

CPU Usage (%)

Remark

Program

CPU Usage (%)

Remark

Unity-Server Unity-Client #1 Virtual Machine #1

1.4 2.6 13.3

Server UAV #1 UAV #1

Unity-Client #1 Unity-Client #2 Unity-Client #3 Virtual Machine #1 Virtual Machine #2 Virtual Machine #3

2.1 2.2 2.1 8.5 8.1 8.2

UAV #1 UAV #2 UAV #3 UAV #1 UAV #2 UAV #3

Table 3. Two UAVs in Workstation #1. Program

CPU Usage (%)

Remark

Unity-Server Unity-Client #1 Unity-Client #2 Virtual Machine #1 Virtual Machine #2

2.3 3.1 3.2 13.5 12.5

Server UAV #1 UAV #2 UAV #1 UAV #2

workstation #2 has lower CPU usage, because of its more powerful CPU. Note that the load of each virtual machine is based on the complexity of the high level algorithm program, so it will not be the same for different tasks and algorithms. On the other hand, for the Unity server and its clients, performance will not be much different in different situations.

4.2 Different proceeding between simulation and real systems

Table 4. Three UAVs in Workstation #1. Program

CPU Usage (%)

Remark

Unity-Server Unity-Client #1 Unity-Client #2 Unity-Client #3 Virtual Machine #1 Virtual Machine #2 Virtual Machine #3

2.6 3.7 3.8 3.6 13.0 12.5 12.4

Server UAV #1 UAV #2 UAV #3 UAV #1 UAV #2 UAV #3

and computers, two workstations are set up, as shown in Table 1. Three virtual machines are created on each workstation. The configuration of each virtual machine is three cores and 3GB of memory. We use the Windows built-in program Performance Monitor to log CPU usage. The duration of each test is 60 seconds. First we start the Unity server, then add the first, second and third UAV into the simulator on workstation #1. Finally, another three UAVs in workstation #2 are added into the simulator. From Tables 2, 3 and 4 we can see that the more UAVs, the more CPU usage is needed in each instance. That’s because every time a UAV is added into the simulator, the communication module for each instance has to handle another UAV’s data. Tables 4 and 5 show performance when simulating six UAVs on two workstations. It’s obvious that

Keeping the algorithm code the same between simulation and real systems is important for testing: It can reduce errors caused by modified codes and increase the simulation system’s reality. The code of high level algorithms running in our simulation system can be easily moved to a real system without modification. The different proceeding paths between the simulation and the real system are shown in Figure 12: Black represents high-level algorithms, which are the public part of both the simulation and real systems. Actually, it is the part of interest, which needs to be tested in the simulation and in real environments. As the special parts of a real system—the ranger driver, the laser driver and the FCU (flight control unit)—gain height, they scan IMU data from the real sensors and then send it to the high level algorithms. Additionally, the FCU receives the UAV’s current position and orientation and reference waypoints from the high level algorithms and controls the UAV’s position and orientation based on them. For the simulation system, sensor data is generated by Unity3D, which simulates the environment state and gets the UAV’s state from its dynamic model. The simulator has its own controller to replace the FCU’s function. Like the FCU, the dynamic model and controller collect the UAV’s current position

Downloaded from sim.sagepub.com at NATIONAL UNIV SINGAPORE on October 11, 2016

Hu and Meng

941

Figure 12. Simulation and real system.

and orientation and reference waypoints, and generate the UAV’s state. In both the real system and the simulation system, the high level systems are the same—only some data links are changed. Thanks to ROS communication structure, this modification is quite small, so only some topics’ names need to be changed. In the simulator, different launch files are created to allocate these topics’ names to different purposes: Such as, simulation, flight test, ground test, and so on. For each situation just use the related launch file with no need to modify the source codes. This feature is very useful in real projects.

the two onboard LIDAR sensors, the UAV needs to run the SLAM algorithm to work out its relative location information. In addition, a 3D path-planning algorithm is run for navigation and guidance. By solving the technical issues in Unity3D and developing the UAV’s model and control algorithms in ROS, our hybrid simulation system can be used to test and verify various UAV navigation and guidance methods in cluttered environments.

5.1 Simulator demonstration 5 From simulation to a real flight test Here we provide a demonstration example using our developed simulator. One UAV is supposed to search the forest autonomously without GPS information. Using

For our forest search application, the main purpose of a flight test is to verify whether ROS and Unity3D can work together smoothly. Important issues include: Sensor data sampling frequency, communication latency between ROS and Unity3D, etc.

Downloaded from sim.sagepub.com at NATIONAL UNIV SINGAPORE on October 11, 2016

942

Simulation: Transactions of the Society for Modeling and Simulation International 92(10)

Figure 15. Flight test in a foliage environment. Figure 13. Unity3D view of forest search.

Figure 14. ROS view of forest search.

Figure 16. Trees in the simulator environment.

The workstation we used for our simulation system is a six-core, E5-2630v2 processor. The graphic card of this PC is GTX 780 Ti. Currently, according to our test, this workstation can handle three UAVs simultaneously running 40Hz LIDAR sensing in a complex 500m 3 500m forest map developed in Unity3D. In Figures 13 and 14, UAV views in Unity3D and ROS are shown respectively. In ROS view in Figure 14, we can also see three UAVs’ local path planning and also a reconstructed 3D octomap. In Figure 13, we can see that three UAVs are running simultaneously on the same workstation. In some cases, for a multi-agent system, we need to test the cooperative control logic. Hence, how many UAVs can run on the same workstation is an important evaluation factor for the developed simulator. The simulation demonstration videos can be watched and downloaded from the following YouTube links: www. youtu.be/CIrKFFXL5Xk, www.youtu.be/06UFkBluICo.

environments. Finally, we also succeeded in the flight test. In the near future, we will conduct multi-UAV cooperative forest search flight tests in a 500m 3 500m forested area. The flight test demonstration video can be watched and downloaded from the following YouTube link: www.youtu.be/rLggamUHuf4.

5.2 Flight test Currently, after successful tests in ROSUnity3D, we have finished flight tests for a single UAV in a real foliage environment. The test scenario is shown in Figure 15. The test field is around 60m 3 60m and near to our campus. The tree density there is sparser than that in our simulation 3D

5.3 Analysis of real and simulation flight tests To compare real and simulator flight data, real and simulation experiments were conducted in the same area of the forest. Figure 16 shows the appearance of the forest in Unity3D, and Figure 17 provides a more clear view of each tree’s position. Figures 18, 19 and 20 present the results of this experiment. In the experiment the UAV started from the zero point, flew clockwise, and returned to the origin. From these figures we can see that the UAV’s moving trajectories and reconstructed maps in real flight and the simulator are close enough, indicating that the system has the ability to simulate real-time UAV flight. Some interesting findings: Because of the unavoidable differences between a dynamic model and a real UAV environment, the dynamic behavior of simulated flight has a certain difference from that of real flight. This can be observed in Figures 19 and 20. One interesting finding is that, in the simulator environment, the UAV completes its

Downloaded from sim.sagepub.com at NATIONAL UNIV SINGAPORE on October 11, 2016

Hu and Meng

943

Figure 17. Simulated tree trunk.

Figure 20. Y axis performance.

6 Impact

Figure 18. Real and Sim trajectory and map.

ROSUnitySim has the capacity to help UAV researchers to evaluate their developed algorithms, UAV models, inner loop control, control logic, etc. The researchers can always challenge their developed method in extreme conditions. For example, in Unity3D, a challenging environment can be built in high fidelity. Another important issue is that both ROS and Unity3D are free software, so no additional cost occurs. This should be useful in the academic arena.

7 Conclusion and future work In this paper, we have given a brief introduction to our newly developed ROS/Unity3D hybrid UAV simulation system. By developing the sensor modeling— especially the LIDAR sensor, and the interface between ROS and Unity3D—we have managed to make the simulation system run smoothly when there is more than one UAV. Since the simulation structure is almost the same as the real system, the developed simulator is quite useful for real UAV application tests, especially for multi-UAV applications. In the near future, we will release the whole system to the public for research development. Acknowledgements Figure 19. X axis performance.

task faster and slightly smoother than in real flight. One of the reasons is that the simulator environment is ‘‘cleaner.’’ Due to the limited memory of our workstation, it is not possible to model the leaves of the trees, and other obstacles, in much detail.

The authors would like to thank Veldis Experience Pte Ltd for helping us build 3D urban and forest maps.

Funding This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Downloaded from sim.sagepub.com at NATIONAL UNIV SINGAPORE on October 11, 2016

944

Simulation: Transactions of the Society for Modeling and Simulation International 92(10)

References 1. Cai G, Chen B, Lee T, et al. Design and implementation of a hardware-in-the-loop simulation system for small-scale UAV helicopters. Mechatronics 2009; 19: 1057–1066. 2. Bortoff SA. The University of Toronto RC helicopter: a test bed for nonlinear control. In: Control applications, 1999. Proceedings of the 1999 IEEE international conference on, 1, Hawai’i, USA, 22–27 August 1999, pp. 8.C.1-1–8.C.1-9. pp.333–338. IEEE. 3. Dittrich JS and Johnson EN. Multi-sensor navigation system for an autonomous helicopter. In: Proceedings of the 21st Digital avionics systems conference, Irvine, USA, 27–31 October 2002, pp. 8C1-1. 4. Gavrilets V, Shterenberg A, Dahleh M, et al. Avionics system for a small unmanned helicopter performing aggressive maneuvers. In: Proceedings of the 19th Digital avionics systems conference, Philadelphia, USA, 7–13 October 2000, pp.1.E.2-1–1.E.2-7. 5. Cai G, Chen BM, Peng K, et al. Modeling and control of the yaw channel of a UAV helicopter. IEEE Trans Ind Electron 2008; 55: 3426–3434. 6. Meng W, He Z, Teo R, et al. Integrated multi-agent system framework: decentralised search, tasking and tracking. IET Control Theory Appl 2015; 9: 493–502. 7. Zhang X, Xian B, Zhao B, et al. Autonomous flight control of a nano quadrotor helicopter in a GPS-denied environment using on-board vision. IEEE Trans Ind Electron 2015; 62: 6392–6403. 8. Cicirelli F, Furfaro A and Nigro L. An agent infrastructure over HLA for distributed simulation of reconfigurable systems and its application to UAV coordination. Simulation 2009; 85: 17–32. 9. Javaid A, Sun W and Alam M. Uavnet simulation in uavsim: A performance evaluation and enhancement. In: Proceedings of the 9th International Conference on Testbeds and Research Infrastructures for the Development of Networks and Communities, Guangzhou, China, 5–7 May 2014, pp.107–115. Springer International Publishing. 10. Opensim. www.opensimulator.sourceforge.net/. 11. Freese M, Singh S, Ozaki F, et al. Virtual robot experimentation platform v-rep: A versatile 3D robot simulator. In: Second International Conference, SIMPAR 2010, Darmstadt, Germany, 15–18 November 2010, pp.51–62. Springer International Publishing. 12. McDowell P, Darken R, Sullivan J, et al. Delta3D: A complete open source game and simulation engine for building military training systems. J Def Model Simul 2006; 3: 143–154. 13. Lewis M, Wang J and Hughes S. USARSim: Simulation for the study of human-robot interaction. J Cogn Eng Decis Making 2007; 1: 98–120. 14. Koenig N and Howard A. Design and use paradigms for gazebo, an open-source multi-robot simulator. In: IEEE/RSJ

15. 16.

17.

18.

19.

20. 21.

international conference on intelligent robots and systems (IROS 2004), Sendai, Japan, 28 September–2 October 2004, pp.2149–2154. Robot operating system. www.wiki.ros.org. Liao H and Qu Z. Virtual experiment system for electrician training based on Kinect and Unity3D. In: Mechatronic sciences, electric engineering and computer (MEC). Proceedings 2013 international conference on, Shenyang, China, 20–22 December 2013, pp.2659–2662. IEEE. Wang S, Mao Z, Zeng C, et al. A new method of virtual reality based on Unity3D. In: 2010 18th international conference on geoinformatics, Beijing, China, 18–20 June 2010, pp.1–5. Falcone A, Garro A, Longo F, et al. Simulation exploration experience: A communication system and a 3D real time visualization for a moon base simulated scenario. In: DSRT’14 Proceedings of the 2014 IEEE/ACM 18th International Symposium on Distributed Simulation and Real Time Applications, Toulouse, France, 1–3 October 2014, pp.113– 120. Washington DC: IEEE Computer Society. Meng W, Hu Y, Lin J, et al. ROS + unity: An efficient highfidelity 3D multi-UAV navigation and control simulator in GPS-denied environments. In: Industrial electronics society, IECON 2015. 41st annual conference of the IEEE, pp.002562–002567. Unity3d. www.unity3d.com. Cui JQ, Lai S, Dong X, et al. Autonomous navigation of UAV in foliage environment. J Intell Robot Syst. Epub ahead of print 29 October 2015. DOI: 10.1007/s10846-015-0292-1.

Author Biographies Yuchao Hu received a BE in Automation and a ME degree in Pattern Recognition and Intelligent Systems from Northeastern University in 2012 and 2014, respectively. Since December 2014, he has been working as an associate scientist in Temasek Laboratories, National University of Singapore. His research interests include unmanned systems, SLAM, 3D robot simulation systems, etc. Wei Meng received BE and ME degrees in Automation from Northeastern University in 2006 and 2008, respectively, and a PhD degree from Nanyang Technological University, Singapore, in 2013. From August 2008 to July 2009, he worked as a research associate in the School of Electrical Electronic and Engineering, Nanyang Technological University, Singapore. Currently he is a Research Scientist in Temasek Laboratories, National University of Singapore. His research interests include source localization and tracking, coverage control, unmanned systems, wireless sensor networks.

Downloaded from sim.sagepub.com at NATIONAL UNIV SINGAPORE on October 11, 2016