Indoor 3D Object Model Obtained using Data Fusion ...

Indoor 3D Object Model Obtained using Data Fusion from Laser Sensor and Digital Camera on a Mobile Robot Razvan Solea, George Veliche, Daniela-Cristina Cernega, Madalina-Roxana Teaca Faculty of Automatic Control, Computers, Electric and Electronic Engineering “Dunarea de Jos” University of Galati, 47 Domneasc Street, 800008, Romania, Email: [email protected], [email protected], [email protected], [email protected] Abstract—This paper proposes a framework to build a 3D model for an indoor object from partial data using a mobile robot. The purpose of the paper is to obtain photorealistic 3D models for objects from raw mobile laser scanning data and digital camera information (to improve the appearance information). The proposed method integrates and analyses the relationships between the visual data and raw mobile laser scanning data. This method is capable to recover the complete density range and the shape of the investigated object. In order to obtain 3D mapping using the laser sensor system (called Lidar), this is positioned to a vertical scan orientation and the tilt actuator provides a rotation movement to the Lidar in order to perform a 3D scan. In this paper, the mobile system acquires 3D Lidar Data only while it is stationary. A 3D map of an object is created by acquiring overlapping scans from multiple locations and viewpoints using local co-ordinates for each scan and merging the scan into a single global co-ordinate system. Video registration is used to texture the scan points in order to provide photo realistic 3D maps. Some techniques are also implemented to improve the resolution of the Lidar Scan with video data. Experiments on real-world data are given to illustrate the suitability of this approach.

I.

INTRODUCTION

For an autonomous mobile robot it is a natural task to create a 3D model from local sensor data collected as it moves in an unknown environment. This 3D model must contain geometric and photometric details as well as knowledge concerning empty spaces. The final representation of this knowledge will lead to the ability of the autonomous system to interact with its environment and accomplish its tasks. On March 11, 2011, a earthquake and accompanying tsunami hit Japan, claiming many lives. The number of collapsed houses totaled more than 350 000, and over 20 000 persons were confirmed as either missing or dead. The Fukushima Daiichi Nuclear Power Plants also sustained extensive damage in the disaster, which resulted in total loss of power, and meltdown accidents during which radioactive materials were released.

Therefore, exploration using mobile robot technology was crucial for this and subsequent missions [1]. Today the mobile robots are used in Nuclear Power Plants for: debris clearing; surveillance and mapping outside and inside of the buildings (images, radiation, temperature, humidity, oxygen concentration, etc.); instruments setup; shield and decontamination; material transportation; construction of pipes and equipments; etc. Also they are used in other disasters for: victim search and rescue; inspection, diagnosis and recovery of plants and facilities; surveillance of coast underwater; mapping of the damaged area; power assist for heavy load tasks, etc. Laser rangefinders have multiple applications in mobile robotics. They can be used for object tracking ([2], [3]), obstacle avoidance ([4]), feature extraction ([5], [6]), map building or self-localization ([7]). With additional hardware, laser rangefinder can be used for surface scanning or 3D modeling. Having a mobile robot able to build a 3D model of the environment (like Fig. 1) is particularly appealing as it can be used not only for planning robot actions but also for several important applications. For example, virtual exploration of remote locations, either for safety, automatic rescue and inspection of hazardous or inhospitable environments (e.g. the reactor of a nuclear power plant). All these applications depend on the transmission of meaningful visual and geometric information. A typical 3D model acquisition structure is composed from

In this emergency situation, the objective of the first disaster response mission was to assess the damage to the target environment, including the taking of dose measurements at the disaster site. However, the site, including the outside and inside the buildings housing the nuclear reactors, proved very dangerous for humans because of the potential for high radiation exposure due to the release of radioactive materials. Fig. 1. Pioneer 3DX mobile robot with laser rangefinder and digital camera

978-1-4799-2228-4/13/$31.00 © 2013 IEEE

497

a 3D scanner to acquire precise geometry, and a digital camera to capture appearance information. The techniques used for surface depth recovery are essential in multiple applications involving robotics and computer vision [8], [9]. We explore and analyze the relationships between characteristics of photography and data range. Our purpose is to demonstrate that the surround (context) statistics on both the characteristics of photography and laser range data can provide information to infer the 3D layout of object (the range information and intensity of the images are correlated). The map of the environment and the robot positions are incrementally estimated and updated through the integration of the new observations performed during the robot evolution. A class of localization methods considers the mapping of the environment and the robot localization simultaneously in a same framework is known as “Simultaneous Localization and Map building (SLAM)” [10] - [12]. A good survey of some methods of objects reconstruction from images and laser scanning can be found in [13] - [15]. II.

THE MECHATRONIC SYSTEM

A. Mobile Robot The Pioneer 3DX is the world’s most popular and versatile robot for research and teaching university robotics. Pioneer has a 45.5𝑐𝑚 x 38𝑐𝑚 x 23.7𝑐𝑚, aluminum body. Its 19cm diameter tires handle nearly any indoor surface. The two motor shafts hold 1000-tick encoders. This differential drive platform is nonholonomic.A proprietary 𝜇ARCS transfers sonar readings, motor encoder information and other I/O via packets from the micro controller server to the PC client and returns control commands. Pioneer can be run from the client or users can design their own programs under Linux or under Windows using C/C++ compiler. B. Laser Rangefinder System In this paper, Hokuyo URG-04LX-UG01 (see Fig. 1) laser rangefinder will be used for scanning. The Hokuyo URG04LX-UG01 scanning laser rangefinder is a compact, small, lightweight and accurate laser scanner that is perfect for mobile robotic applications [16]. The URG-04LX-UG01 is used for area scanning and localization of autonomous robots and automated material handling systems. Operation principle of the Hokuyo LRF is based on a light source given by an infrared laser of wavelength 785nm, a rotating mirror system motor-operated and lens. HokuyoURG04LX-UG01 key features: ∙

USB 2.0/1.1 interface

∙

Measuring area: 20 to 5600 mm

∙

Accuracy: 30 mm

∙

Scan area is 240°, with an angular resolution of 0.36°

∙

Scan rate is about 100 msec

∙

Power source 5VDC (USB bus power)

Fig. 2.

Tilt servo controller used in real experiments

C. Tilt Servo Controller The SSC-32 (serial servo controller) is a small preassembled servo controller (see Fig. 2). It has high resolution (1𝜇𝑆) for accurate positioning, and extremely smooth moves. The range is 0.50𝑚𝑆 to 2.50𝑚𝑆 for a range of about 180°. The motion control can be immediate response, speed controlled, timed motion, or a combination. The servo’s position or movement can be queried to provide feedback to the host computer. The HS-311 is the perfect servo for inexpensive applications. The following Matlab code sequence is a demonstration on how to use the tilt servo system %laser data aqcuisition loop while pCrt˜=pFin aux=pCrt/10; a= mod(aux,grad); if(a==0) motor elevation command fprintf(S,strcat(’#20P’,num2str(pCrt),’T300’)); pause(.1); %horizontal offset for servomotor(in degrees) elev=(pCrt-pStBy)/10; %effective data aqcuisition from sensor range=LidarScan(lidar); pause(.1); %adding sensor data to the bidimensional array scanare=[scanare; range]; end %incr. current servomotor position by 1 deg. pCrt=pCrt+10; end

D. Digital Camera HTC Desired X Smartphone is used to take photos of the objects. This type of smartphone have a 5 MP VGA camera. III.

ENVIRONMENT MODELING

A mobile robot exploring and mapping an unknown environment generate maps by accumulating sensor information

498

80

100

120

140

160

180

200

220 60

Fig. 3. Grid elements are divided into four regions depending on their position relative to the laser rangefinders. 𝑅 is the laser maximum range and 𝛽 is half of the sensor cone angle width (top view)

while exploring the environment. Unfortunately readings from range sensors are only useful for the map making process if the position of the sensor at the time the reading took place is known. Encoders are the most commonly used source of odometry information. However it usually suffers from both systematic errors like inaccurate wheel measurements, and non-systematic errors like wheel slippage. These errors cause the odometry error to quickly accumulate over time. When a robot is not being localized in the environment, the quality and accuracy of the generated map will be degraded. In this chapter is presented the modeling of the environment measurement used on the localization and mapping process. The measurement of the environment is provided by laser sensors mounted on the robot. It is introduced the concept of the local occupancy grid that is an extension of the laser model. A local geometric map, is extracted from the local occupancy grid by performing a weighted Hough transform. The occupancy grid is composed by a set of cells that contain a probability value in the [0...1] interval. The 0 value represents a free cell, the 1 value represents an occupied cell. Intermediate values represents different levels of occupancy, in particular, the value 0.5 represents a completely unknown level of occupancy. Each cell is a reference to a region on the Cartesian space around the robot. All cells on the laser range are updated using a Bayes rule. In this work the implementation of such update process is subdivided in two phases: a Response Grid update and an Occupancy Grid update, as described in ([17], [18]). The Pioneer robot used is equipped with Hokuyo URG04LX-UG01 laser rangefinder sensor. Most roboticists have converged to a model of laser sensor uncertainty which looks like in Fig. 3. The model can be divided into four different regions. The first three regions are inside the laser sensor’s field of view and the last one is outside: Region 1 - elements in this region are probably all empty; Region 2 - elements in this region are probably at least partially occupied; Region 3 - the state of elements in this region is unknown because they are located behind whatever reflected the laser rangefinder

499

Fig. 4.

80

100

120

140

160

180

200

Example of map using laser rangefinders

(shadow region); Region 4 - nothing is known about elements outside the laser rangefinder of view. In the Bayesian approach sensor models convert sensor measurements into conditional probabilities of the form 𝑃 (𝑠∣𝐻), i.e. the probability that a sensor reading 𝑠 will occur, given that a certain grid element is occupied (𝑂𝑐𝑐). The laser sensor model used in this paper uses the following functions to calculate the probability for each grid element located at a distance 𝑟 and an angle 𝛼 for the laser position (axis): ∙

for elements in region 1: 𝑃 (𝐸) =

𝛽−𝛼 ( 𝑅−𝑟 𝑅 ) +( 𝛽 )

2

(1)

𝑃 (𝑂𝑐𝑐) = 1.0 − 𝑃 (𝐸) ∙

for elements in region 2: 𝑃 (𝑂𝑐𝑐) =

𝛽−𝛼 ( 𝑅−𝑟 𝑅 )+( 𝛽 )

2

⋅ 𝑀 𝑎𝑥𝑜𝑐𝑐𝑢𝑝𝑖𝑒𝑑

(2)

𝑃 (𝐸) = 1.0 − 𝑃 (𝑂𝑐𝑐) These probabilities are projected onto an occupancy grid at the position and direction of the sensor, and are merged with the grid’s existing spatial knowledge using Bayesian sensor fusion. The sensor model provides conditional probabilities of the form 𝑃 (𝑠∣𝐻) but the form 𝑃 (𝐻∣𝑠), the probability that a grid cell is occupied given sensor reading 𝑠, is of more interest when updating an occupancy grid. Baye’s rule states that:

𝑃 (𝑂𝑐𝑐∣𝑠) =

𝑃 (𝑠∣𝑂𝑐𝑐)𝑃 (𝑂𝑐𝑐) 𝑃 (𝑠∣𝑂𝑐𝑐)𝑃 (𝑂𝑐𝑐) + 𝑃 (𝑠∣𝐸)𝑃 (𝐸)

(3)

Where 𝑃 (𝑂𝑐𝑐) and 𝑃 (𝐸) probabilities represent the initial unconditional belief about the state of a grid element (initial 𝑃 (𝑂𝑐𝑐) = 𝑃 (𝐸) = 0.5). Using equation (3) it is possible to populate an occupancy grid with probability values from a single sensor reading. With multiple sensor readings the equation becomes:

Fig. 5.

Real environment used in example

Fig. 7.

𝑃 (𝑂𝑐𝑐∣𝑠1,2,...,𝑛 ) = =

𝑃 (𝑠1,2,...,𝑛 ∣𝑂𝑐𝑐)𝑃 (𝑂𝑐𝑐) 𝑃 (𝑠1,2,...,𝑛 ∣𝑂𝑐𝑐)𝑃 (𝑂𝑐𝑐) + 𝑃 (𝑠1,2,...,𝑛 ∣𝐸)𝑃 (𝐸)

(4)

Equation (4) has one software implementation problem. It requires the program to keep track of all sensor readings while updating an element in order to be able to update it in the future. This is problematic because it is not known how many times each grid element will be updated. Fortunately, after clever use of the relation 𝑃 (𝐻∣𝑠)𝑃 (𝑠) = 𝑃 (𝑠∣𝐻)𝑃 (𝐻), a recursive version of equation (4) can be created: 𝑃 (𝑂𝑐𝑐∣𝑠) = =

𝑃 (𝑠𝑛 ∣𝑂𝑐𝑐)𝑃 (𝑂𝑐𝑐∣𝑠𝑛−1 ) 𝑃 (𝑠𝑛 ∣𝑂𝑐𝑐)𝑃 (𝑂𝑐𝑐∣𝑠𝑛−1 ) + 𝑃 (𝑠𝑛 ∣𝐸)𝑃 (𝐸∣𝑠𝑛−1 )

(5)

With this equation, only the previous grid value 𝑃 (𝐻∣𝑠𝑛−1 ) needs to be saved in order to be able to update the element in the future. The Continuous localization (see as shown in Fig 6) technique proposed by Schultz et al. [19] takes advantage of the fact that positioning errors from shaft encoders usually accumulate over time. Continuous localization tries to perform small positioning corrections often instead of large corrections far apart in time.

Example of 3D object used for scanning

contain recent sensor information. Given that, these maps are constructed during a relatively short period of time. It is assumed that they contain relatively small amounts of odometry errors. IV.

T HE P ROPOSED F RAMEWORK

The camera is attached to the laser in such a way that their center of projections (optical center for the camera and mirror center for the laser) are aligned to the center of projection of the tilt unit. This alignment facilitates the registration and corelation between the photo and range data. The mobile robot first realises the map of the surroundings using the algorithm presented in section 3 (as shown in 4). After this step, it can start the scanning of the object of interest (see 7). The scanning of the object is done from four different positions (north, east, south, and west). This operation is performed in an autonomous way (without human operator intervention). This is important in case of use in a potential dangerous situation when there is neither a human operator nor a communication line available. One side of the object once scanned in displayed by the graphical interface together with the robot coordinates and with the photo camera information (as 8 shows). In the left down corner of Fig. 8 one can see the scanning is done at the distance of 1.5 meters of the object.

Continuous localization is an exteroceptive method that builds temporary occupancy grids called short-term maps that

Fig. 6.

Continuous localization process

Fig. 8.

500

Matlab graphical user interface of application

Z [mm]

600 400 200

0 −2000

0 −2000

0

−1000 2000 1000

3000

2000

0

1000 0

Y [mm]

1000

−1000 Y [mm]

X [mm]

3000

1000 2000

X [mm]

4000

2000 4000

Fig. 9.

Fig. 10.

Result of 3D scanned object using only laser informations

The object scanning is realised on 24 lines with the tilt servo system. When the whole scanning is done, the 3D object model is obtained. In Fig. 9 and Fig. 10 the shape of the object is obtained through the read laser data. In Fig. 11 the algorithm steps, implemented in Matlab for the whole mobile system used for 3D scanning, are presented. In the algorithm is easy to observe that after data acquisition from the lase sensor, they are filtered and after that 3D plotted. On this plot is applied the information read from digital photo camera (more precisely, the photo is overlapped on the data already read from the laser and filtered).This overlap is accomplished at the pixel level (to each point read from the laser it corresponds more than one pixel). This leads to the texture of the scanned object. The result of a complete scanning consists in four data sets (corresponding to the four scans: north, east, south, and west) read from the laser. In order to accomplish the correct link between the four data sets, the precise mobile robot position must be known. For this reason, before the beginning of the 3D object scanning, it is performed a surrounding scan with a SLAM type algorithm. This stage provides information concerning the surrounding “state” and information used to correct the errors from the mobile robot (position). The SLAM type algorithms are known as using large amount of time and resources, but in case the time is a critical resource, performant correction systems (SLAM) using landmarks cannot be used. V.

C ONCLUSION

The 3D object model identification presented in this paper is a low cost one and it can be interconnected with other systems used in our day by day life: notebook, smart phone, a small laser system. In catastrophically situations, information about the sensible objects, who are in different location and potentially can harm people, are necessary. The 3D object model identification presented in this paper is completely autonomous and does not necessitate teleoperation. The ability to reconstruct a 3D model of an object or scene greatly depends on the type, quality and amount of information

501

Top view of scanned object

available. The data acquisition framework described here was designed to speed up the acquisition of range data by obtaining a relatively small amount of range information from the scene to be modeled. By doing so, we compromise the accuracy of our final representation. As a future work, this system can be improved. The software part can improve the SLAM algorithm and the 3D scanning algorithm. New sensors to detect some additional features (like temperature, radiation level, etc.) of the object can improve the hardware architecture of this system. Experiments on real-world data are given to illustrate the suitability of our approach. R EFERENCES [1] K. Nagatani, S. Kiribayashi, Y. Okada, K. Otake, K. Yoshida, S. Tadokoro, T. Nishimura, T. Yoshida, E. Koyanagi, M. Fukushima, and S. Kawatsuma, Emergency response to the nuclear accident at the Fukushima Daiichi nuclear power plants using mobile rescue robots, Journal of Field Robotics, Vol.30(1), pp 44-63, John Wiley and Sons Ltd. Chichester, UK, January 2013. [2] D. Brscic and H. Hashimoto, Tracking of Objects in Intelligent Space Using Laser Range Finders, IEEE Int. Conf. on Industrial Technology, Mumbai, pp. 1723-1728, 2006. [3] T. Sasaki and H. Hashimoto, Calibration of Laser Range Finders Based on Moving Object Tracking in Intelligent Space, IEEE Int. Conf. on Networking, Sensing and Control, Okyama, pp. 620-625, 2009. [4] Y. Negishi, J. Miura and Y. Shirai, Mobile Robot Navigation in Unknown Environments Using Omnidirectional Stereo and Laser Range Finder, IEEE Proceedings Int. Conf. on Intelligent Robots and Systems, Vol.3, pp. 2737-2742, 2004. [5] C. Premebida, O. Ludwig and U. Nunes, LIDAR and vision-based pedestrian detection system, Journal of Field Robotics, Wiley, Vol.26(9), pp. 696-711, 2009. [6] S.M. Mavaei, Adaptive feature extraction and mapping for mobile robots in unknown environment, International Journal of the Physical Sciences Vol.6(29), pp. 6741-6750, 2011. [7] N. Bahari, M. Becker, H. Firouzi, Feature Based Localization in an Indoor Environment for a Mobile Robot Based on Odometry, Laser, and Panoramic Vision Data, ABCM Symposium Series in Mechatronics, Vol. 3, pp. 266-275, 2008. [8] P. Gurram, H. Rhody, J. Kerekes, S. Lach, E. Saber, 3D Scene Reconstruction through a Fusion of Passive Video and Lidar Imagery, 36th IEEE Applied Imagery Pattern Recognition Workshop, pp. 133138, 2007.

Fig. 11.

The algorithm used in application

[9] L.A. Torres-Mndez and G. Dudek, Statistics of Visual and Partial Depth Data for Mobile Robot Environment Modeling, MICAI 2006: Advances in Artificial Intelligence, Lecture Notes in Computer Science, Springer, Vol.4293, pp. 715-725, 2006. [10] A. Eliazar and R. Parr, DP-SLAM: fast, robust simultaneous localization and mapping without predetermined landmarks, In Proceedings of the 18th International Joint Conference on Artificial Intelligence (IJCAI’03), San Francisco, pp. 1135-1142, 2003. [11] J.G. Kang,S. Kim, Su-Yong An and Se-Young Oh, A new approach to simultaneous localization and map building with implicit model learning using neuro evolutionary optimization, Applied Intelligence, Vol.36(1), pp. 242-269, 2012. [12] B. Vincke, A. Elouardi and A. Lambert, Real time simultaneous localization and mapping: towards low-cost multiprocessor embedded systems, EURASIP Journal on Embedded Systems- a SpringerOpen Journal , Vol.5, 2012. [13] C. Brenner, Building reconstruction from images and laser scanning, International Journal of Applied Earth Observation and Geoinformation, Vol.6(3-4), pp. 187-198, 2005.

502

[14] Shi Pu, Knowledge based building facade reconstruction from laser point clouds and images - PhD thesis, Publications on Geodesy 75, Delft, 133 pages, 2010. [15] L. Zhu, J. Hyyppa, A. Kukko, H. Kaartinen and R. Chen, Photorealistic Building Reconstruction from Mobile Laser Scanning Data, Remote Sensing, Vol.3, pp. 1406-1426, 2011. [16] L. Kneip, F. Tache, G. Caprari and R. Siegwart, Characterization of the compact Hokuyo 𝑈 𝑅𝐺 − 04𝐿𝑋 2D laser range scanner, IEEE International Conference on Robotics and Automation, Japan, pp. 14471454, 2009. [17] L.S. Dinnouti, A.C. Victorino and G.F. Silveira, Simultaneous Localization and Map Building by a Mobile Robot Using Sonar Sensors, ABCM Symposium Series in Mechatronics, Vol.1, pp. 115-123, 2004. [18] J. Borgstr¨om, ARIA and Matlab Integration With Applications, Ume˚a University, Department of Computing Science Master’s Thesis Project, Supervisor: Thomas Hellstr¨om, 66 pages, 2005. [19] A. C. Schultz and W. Adams, Continuous localization using evidence grids, Proceedings of the IEEE International Conference on Robotics and Automation, Belgium, pp. 2833-2839, 1998.