Development of On-Line Simulation System for Multi Camera Based ...

2008 IEEE/RSJ International Conference on Intelligent Robots and Systems Acropolis Convention Center Nice, France, Sept, 22-26, 2008

Development of On-line Simulation System for Multi Camera based Wide Field of View Display Naoki MIDORIKAWA, Kazunori OHNO, Satoshi SAGA, Satoshi TADOKORO Abstract— IRS Soryu is rescue robot which used for victim search in disaster area. We aim at development of display method for wide view angle and high definition images for controlling the rescue robots and searching for victims. In our study, we developed small size multi camera system and displayed a wide view angle and high definition image. However, it is too hard for an operator to recognize the robot and environment intuitively. We need to develop a display method of these multi camera images for intuitive recognition. In this paper, we develop the on-line simulation system for the verification of multi camera based wide field of view display. This system has some requirements that; a. Simulation of IRS Soryu dynamics, b. Simulation of multi camera system, c. Flexibility of camera setting. We developed such simulator based on USARSim. Using the simulator, we examined some experiments for the verification of of the display methods.

I. I NTRODUCTION

Fig. 1.

Various rescue robots have been studied to search for victims in disaster area. Especially, it is important to develop the rescue robots that can search in narrow space (e.g. collapsed house, collapsed elevated bridge, broken train etc). A snake-like robot, “IRS Soryu” is one of them (Fig.1). It has been developed by Tokyo Institute of Technology, International Rescue System Institute (IRS), Kobe University, Tohoku University and the University of Electro- Communications [1]. Its size is 170(W)*110(H)*1200(L) [mm]. IRS Soryu is composed of three box-shaped segments linked to each other. Each segment has two crawlers, and front segment has two on-board cameras. An operator at remote place has controlled the robot. For the remote operation, we need a wide view angle and high definition image. So, we aim at development of wide view angle and high definition display method based on multi camera system. The authors have developed a small size multi camera system[3], and installed 32 cameras on the IRS Soryu. Now we can get wide view angle and high definition images around of the robot at the same time. However, it is hard for an operator to recognize the situations from the 32 independent camera images (Fig. 5) intuitively. Therefore, we have to develop a display method of multi camera images. For this purpose, we need a on-line simulation system that can simulate robot motion and multi camera system during remote control. Using the simulation we can examine various camera settings. The on-line simulation system was developed using USARSim. The authors construct the IRS Soryu robot model and multi N. MIDORIKAWA, K. OHNO, S. Saga and S. Tadokoro are with Graduate School of Information Sciences, Tohoku University, 6-6-01 Aramaki Aza Aoba, Aoba-ku, Sendai, 980-8579, Japan, (midorikawa,

kazunori, saga, tadokoro)@rm.is.tohoku.ac.jp

978-1-4244-2058-2/08/$25.00 ©2008 IEEE.

Fig. 2.

IRS Soryu

Experimental Victim Search System using On-line Simulation

camera system in USARSim and we examine the method of information display. In this paper, we introduce related works for wide view angle images in Sec.II and describe the display method for wide view angle and high definition images using the multi camera system in Sec.III. Our developed on-line simulation system is explained in Sec.IV. We examined whether the simulator can be used for the evaluation of display method. The experiment and its result are explained in Sec.V. Conclusions and future works are described Sec.VI. II. R ELATED W ORKS In study of Yanco, it takes 30 percent of the whole operating time for an operator to recognize environment[2]. Visual information is very important for the remote operation. Especially, wide view angle and high definition camera images are important for the purpose. Major camera systems that can get such wide view angle images are fish-eye camera and Omni directional camera [4] [5]. In these methods, we can easily recognize relative between the robot’s body and objects around the robot. It

2097

frame robot model is displayed according to the movement of the robot. using many images[9]. They make image-history database and wire frame model of the robot moves on the image. It enable operators to use bird’s eye view information. These studies show that human can recognize much information efficiently using composite information (ex. robot’s model and image information) in the imaginary world. III. D ISPLAY OF W IDE V IEW A NGLE AND H IGH D EFINITION I MAGE USING M ULTI C AMERA S YSTEM

Fig. 3.

Multi Camera System and One Camera Module

seems to be unsuitable for IRS Soryu that searches in narrow space because collision against broken bricks may damage lens of fish-eye camera or Omni directional camera. In addition, these methods cannot capture high definition image. In other camera systems, composite image is made from multi camera images [6] [7]. In this case, lens of cameras can be built inside the robot’s body. The high definition and wide field of view images are gotten at the same time. However, this camera system cannot be used for these small size rescue robots because its sizes are too large. These small size robots need small size camera devices. In our study, we developed small size multi camera system it can be equipped in narrow space. We need to develop a display method for multi camera images. multi camera images are important information to recognize environment around the robot, but it is too complex for human to recognize environment intuitively. To solve this problem, there is the method of using bird’s eye view information that is often used in racing games. Nelsen’s proposed a bird’s eye view user interface that is composed of map information and robot model[8]. In our research, virtual bird’s eye view is made from multi camera image (Fig. 10). We can recognize 32 camera images intuitively using the composite image. This display method is better than the camera image aligned display method (Fig. 5). Sugimoto also proposed a method that bird’s eye view user interface is made of camera images and wire frame robot model[9]. Wire

Figure 2 shows the victim search system construction. We should verify the display method to fill the system. Our research objectives are acquirement of wide view angle and high definition image using the multi camera system. For intuitive recognition, we need to develop a method for image composition. The authors developed a simulator that can simulate the robot movement and multi camera system. Display method can be evaluated on various camera settings using the simulator. Figure 2 shows our proposed system construction. Figure 3 illustrates our developed multi camera system and one camera module [10]. This multi camera system consists of some multi camera modules that have four small size CMOS cameras and small size CPU board. By connecting some multi camera modules, we can easily increase the number of cameras. For controlling the robot that searches in narrow space like IRS Soryu, an operator mainly needs to check following three information from camera images: 1. Interference between robot’s body and environment, 2. Contacts between crawlers and the ground, and between joints and environment, 3. Detailed information about victims. The authors had proposed camera layout that can obtained above information for Soryu[10]. In this paper, 44 cameras were used for the purpose. We had installed 32 camera into the IRS Soryu. Front and rear of the robot, left-side and right-side of the robot, low bird’s eye view point of front and rear segments, upper-side and lower-side of the robot. We performed a remote control experiment using IRS Soryu in our laboratory. Figure 5 illustrates a screen shot of 32 camera images at the experiment. In the experiment, we could get and display wide view angle and high definition camera images. Frame rate of these images were 0.9[fps]. However, it was hard for an operator to recognize the robot and environment from the images intuitively Fig.5 because the display method lacked spacial information between each image. Therefore, we develop the composite method to make the multi camera images easy to recognize. Concretely, a pseudo ellipsoid screen is put around the robot, and multi camera images are projected on the screen like Fig.7. Thus, we make the composite image which we can recognize spatial connections. IV. O N - LINE S IMULATION S YSTEM A. Required Specification In our study, we develop an on-line simulation system for IRS Soryu and the multi camera system. We need the fol-

2098

lowing three requirements when we develop the simulation. a. Simulation of IRS Soryu Dynamics. b. Simulation of Multi Camera System and the socket communication. c. Flexibility of camera setting (e.g. camera position, number, method of composite images) We solve these requirements as follows: To solve a, we use USARSim that is a game engine, “Unreal Engine” [11][12]. To solve b, we develop the function which emulate multi camera system on USARSim. To solve c, we use simulator instead of the real robot and apparatus. We constructed online simulation system by using USARSim. B. Framework of Simulation System

Fig. 4. On-line Robot Simulation System for Wide View Angle and High Definition Image Display

Fig. 5.

Display Result of 32 Cameras

Figure 4 illustrates the system construction image of our developed on-line simulation system. In this system, we use 2 PCs:One is a server PC for the simulator, the other one is a client PC for user interface (controller and display). We explain the simulation that simulates the movements of the robot and multi camera server on Server-PC. USARSim(Unified System for Automation and Robot Simulation) is used in the simulation system. By using USARSim, we can simulate the robot behavior and the sensor measurements (e.g. cameras, laser range finders etc) in the imaginary world that is made from polygon models. We can also construct new models of the robot and environments for the examination. Using the simulator, multi camera images can be obtained at the same time. The wide view angle image is constructed with these images on the Client-PC. The details of multi camera simulation is described in the next section. In order to evaluate the intuitiveness of the interface, it is necessary to conduct the evaluation experiment repeatedly while changing various conditions. The evaluation experiment can be repeated easily while changing the settings by verifying it with the simulation base. In addition, because neither the preparation for the experiment nor the damage of the equipment by the experiment are caused, it can be expected to evaluate it from the experiment efficiently in a short time more than the case of using real machines. C. Multi Camera Simulator

Fig. 6.

Simulated Images of 32 Cameras

We created the multi camera simulator using camera models in USARSim. The installation position and the angle are same to a real machine’s. Images from each camera models are acquired individually. Figure 6 shows the 32 images layout. It is same to the real machine’s layout(Fig.5). We can confirm that the simulator can simulate the real multi camera system. In our system, Windows is used as server PC and Linux as used in client PC. Socket communication is used for the control instruction of the robot and the communication of the image data. All commands can be lodged from client PC, All images are transmitted from server PC to client PC via TCP/IP communication. The data format of the transmitted image can be selected Raw or JPEG. Resolution of the display image depends on this selection.

2099

Environment Projective screen Projecte j e ct ct Projected perspective images

Soryu Pseud viewpoint

Fig. 7.

Rounded Pseudo Screen of IRS Soryu and Camera Images

Fig. 9.

Fig. 8.

Image of the On-line Simulator

Wide Field of View Display (Proposed UI)

In our study, a lot of image data are continuously communicated by the network. Sequentially sending each of one image data causes a great delay of the display of the image on client PC, and we cannot operate the robot on real time. So, the simulation image is sent from server PC to client PC as a JPEG compressed image of 480*360 sizes. The data size is about 20000 bytes. For this one image, the time until the image data is stored in the client-PC’s memory shows about 162 ms from the average of the measurement value. If 32 images are sequentially transmitted, the camera image is updated every about 5 seconds. This result shows that the simulator cannot display the wide view angle image in real time. Therefore, in this study, 32 camera images are compressed into an image. This image is sent from server PC to client PC. client PC updates all image information at high frame rate. The image is divided into 32 pieces again in client PC. Wide view angle image is composed these images. In addition to the advantage of the communication, this technique simplifies the processing of the image presentation software that uses the texture mapping in OpenGL. This value can secure real time that the verification and the evaluation of the image display method.

Fig. 10.

Side View Images and Bird’s Eye View Images

V. T RIAL E XPERIMENT FOR E VALUATION OF W IDE V IEW A NGLE D ISPLAY M ETHOD USING S IMULATOR A. Proposed method for Wide field of view Display The author proposed an on-line simulator for displaying wide view angle images using 32 independent camera images. The composite image of remote area was made by projected images on the pseudo ellipsoid screen around the IRS Soryu model (Figs. 7 and 8). Fig. 9 shows image of the on-line simulator displayed to an operator. The positions of 32 camera images in composite image have to be changed according to movement of IRS Soryu. In proposed method, we can move the viewpoint by dragging it with the mouse and can see the entire surroundings or detailed part of the image(Fig.10). The positions of 32

2100

Fig. 11.

Environment of the Experiments

camera images are calculated from the pseudo field and the direction of the optical axis based on the perspective projection method. The joint’s angle of the simulated IRS Soryu in server PC and of the wire frame model in client PC are synchronously changed.

Fig. 12.

Results of Experiment

B. Experiments We performed victim search experiments to evaluate the effectiveness of our on-line simulation system. In this experiment, an operator controlled the robot and searched for victims in an unknown environment using camera image. An operator uses two kinds of user interface: 1.Simple UI displays composite image of two images from front camera and bird’s eye view camera, 2.Proposed UI displays composite image of 32 camera images. We verified the effectiveness of our on-line simulation by comparison of two user interfaces. In the both methods, the field of view and the resolution of the camera are same. Figure 11 shows the environment in the victim search experiment. The operator controls IRS Soryu by using Simple UI or Proposed UI, and searches for five victims in this field. The task of operator are drawing the rough map, the route of the robot, and the position of victims. The operator finished the experiment when it exceeded the time limit (15 minutes) or all five victims were found. After the experiment, each interface were evaluated as five ranks by questionnaire. The questionnaire asked the ranks of three points: “Easiness to operate”, “Easiness to search”, and “Easiness to recognize the environment”. C. Results Three operators controlled IRS Soryu and searched for victims. Figure 12 and Fig.13 show the results of the experiment. Figure 12 shows the time to spent searching, the times of rolling over, and the number of discovered victims. In the case of simple UI, the time to spent searching is 15 minutes because all three operators could not find five victims for 15 minutes. In the case of proposed UI, one operator could find all five victims. There is no difference in the times of rolling over. The number of discovered victims was 2 in the case of Simple UI, and 3.3 in the case of Proposed UI. These results

Fig. 13.

Questionnaire’s Answers of Each Interface

indicated that the effectiveness of proposed UI is higher than that of simple UI. The answer of questionnaire also shows that Proposed UI is easier to use than Simple UI (Fig. 13). Operators said that the operation using Simple UI was very difficult because they could watch only a narrow angle view. On the other hand, the operation using Proposed UI was easy because they could watch all surroundings including behind of the robot. They also said that they could confirm robot’s position by finding landmarks when the robot was rolling over in the case of Proposed UI. The rough map drew by operators was more correct in the case of Proposed UI than that in the case of Simple UI. As a result, operators could understand the environment efficiently from the map. However, Proposed UI has some problems: displaying overlapped images, and low resolution of composite image. Operators said that they cannot recognize the environment easily from overlapped images, especially when the IRS

2101

Soryu was bent. When IRS Soryu was bent, the image positions became disorder, and the area of overlapped images increased. To solve this problem, we need to consider not only put the image but also the method of composing the overlapping part and the depth change of image according to distance. Operators also said that it was difficult to understand the environment because the resolution of image was low. Low resolution is attributed to using compressed 32 images. VI. C ONCLUSIONS AND F UTURE W ORKS The authors aim at development for display method of wide view angle and high definition image using multi camera system. In this paper, we developed on-line simulation system for the evaluation of the display method. We examined the display method using simulated IRS Soryu and multi camera system. The experiment shows that our developed simulation system is effective to evaluate our display method. We also confirmed that our proposed user interface is more efficient to control IRS Soryu and to search for victims than our previous method. However, we found some problems of our proposed user interface: displaying overlapped images, and low resolution of composite image. In future works, we will improve the robot movement in the simulation system, and develop more efficient user interface. R EFERENCES [1] M. Arai, “Improved Driving Mechanism for Connected Crawler Vehicle “Soryu-IV” for In-Rubble Searching”, SSRR2006, TUE-AM1. [2] H.A.Yanco and J.Drury, “Where Am I ” Acquiring Situation Awareness Using a Remote Robot Platform, In Proc. of SMC ’04, Vol.3, pp.2835-2840, Oct, 2004. [3] Masaki Minobe, “The Development of Embedded Compound Vision System for Rescue Robot”, SI2005. [4] Kazumasa Yamazawa, Yasushi Yagi and Masahiko Yachida, “Obstacle Detection with Omnidirectional Image Sensor Hyper- Omni Vision”, Proceedings IEEE the International Conference on Robotics and Automation, pp. 1062-1067,1995.5. [5] Kousuke Nakashima, Takashi Machida, Kiyoshi Kiyokawa and Haruo Takemura: “A 2D-3D Integrated Tabletop Environment for Multi-user Collaboration”, Computer Animation and Virtual Worlds, Vol. 18, No. 1, pp. 39-56, Feb. 2007. [6] S. Ikeda, T. Sato, M. Kanbara, and N. Yokoya, “Immersive telepresence system with a locomotion interface using high-resolution omnidirectional videos”, Proc. IAPR Conf. on Machine Vision Applications 2005, pp. 602-605, May 2005. [7] Viewplus ASTRO Sensor Series: http://www.viewplus.co.jp/products/sos/astro-e.html [8] C.W.Nielsen and M.A.Goodrich: “Comparing the Usefulness of Video and Map Information in Navigation Tasks”, In Proc. of HRI 2006, pp.95-101, Mar, 2006. [9] M.Sugimito, G.Kagotani, H.Nii, N.Shiroma, M.Inami and F.Matsuno: Time Follower’s Vision: A Tele-Operation Interface with Past Images, IEEE Computer Graphics and Applications, Vol.25, No.1, pp.54-63, Jan/Feb, 2005. [10] Naoki Midorikawa, “Development of Small-Size Multi Camera System for Snake-like Robot, and Display of Wide View-Angle Image”, SSRR2007. [11] S. Balakirsky, C. Scrapper, S. Carpin, M. Lewis, “USARSim: Providing a Framework for Multi-robot Performance Evaluation” [12] USARSim - Unfied System for Automation and Robot Simulation: http://usarsim.sourceforge.net/

2102