Proc. 1994 IEEE Int. Conf. on Robotics and Automation pp.3368-3373, San Diego, CA, May 1994
Modeling Obstacles and Free Spaces for a Mobile Robot using Stereo Vision with Uncertainty Jun MIURA
Yoshiaki SHIRAI
Dept. of Mech. Eng. for Computer-Controlled Machinery, Osaka University, Suita, Osaka 565, Japan
[email protected]
Abstract
This paper describes a new method of modeling an environment in terms of obstacles and free spaces from a set of 3D segments obtained by stereo vision. Since the stereo vision provides only the position of segments, it is necessary to determine whether a region formed by the segments is an obstacle or a free space. The ambiguities and the uncertainties in the obtained data must be considered in modeling. The nal output of the proposed method is a set of possible situations of the environment and their probabilities; each situation consists of the description of obstacles and critical regions between the obstacles. Experimental results for a real scene are described. 1
Introduction
Detection of obstacles and free spaces is an essential function of the vision system for a mobile robot. Even if a robot is given a map of the environment, this function is indispensable to cope with unknown obstacles or errors of the map. There have been many research works on geometric modeling of the environment from sensory data. Most of them use a laser range nder [8] [10] or an ultrasonic sensor [3] [5]. Stereo vision is a passive ranging method and is important in many situations where active ranging methods are not feasible [1]. There are few works on environment modeling using stereo vision. Since most stereo systems provide sparse range data, it is necessary to determine object regions. Faugeras et al. [4] proposed a method of obtaining polyhedral surfaces by interpolating position data of 3D segments. The method is based on the Delaunay triangulation and a simple visibility constraint. Echigo [2] proposed a method of calculating free areas using occlusion information and the assumption of polyhedral objects.
Figure 1: A pair of stereo images.
These methods, however, do not consider the uncertainty of stereo data. The most important information when a mobile robot moves among obstacles is the passability of the spaces between obstacles. If the objective of a robot is to reach the goal point, the precise description of a whole environment may be unnecessary. In many cases, the topological structure of possible paths and the description of critical (narrow) regions on the paths are sucient. This paper describes a new method of modeling an environment in terms of obstacles and free spaces from a set of 3D segments obtained by a segment-based stereo vision. The target scene is non-trivial indoor scenes; for example, a scene shown in Fig.1. Given a set of 3D segments, the system calculates possible situations of the environment and their probabilities. Each situation consists of the description of obstacles and critical regions between obstacles. An obstacle is represented by a set of 3D segments. A critical region is a region which a robot need to pay attention to in passing between obstacles. This modeling is performed considering the positional uncertainties of segments and the ambiguities of the stereo matching.
impassable
group 1
obstacle 1
undecided obstacle 2
group 2 group 3
critical region obstacle 3
(a) Initial data set.
(b) Classi ed relations.
(c) Modeling result.
Figure 2: From segment data to obstacles. 2
Basic Idea
All 3D segments above the ground plane are projected onto the oor. Thus, the segments are represented by 2D segments (see Fig.2(a)) and obstacles and free spaces are modeled in a 2D space. The positional distribution of each endpoint is also provided from stereo. Let us consider a mobile robot passing between two points. From the positional distribution of the points, the distribution of the distance between the points can be calculated. According to the relation between the distribution and the robot width, possible relations between two points are classi ed into three cases: passable, impassable, and undecided (see Fig.3). By using impassable relation, endpoints are clustered into point groups so that inside a group, every point is reachable from one point via only impassable relations (see Fig.2(b)). Since a robot cannot pass between any two points of a group, each group is regarded as an obstacle. Then, critical regions between obstacles are determined (see Fig.2(c)). A critical region is set to cover all undecided relations between two obstacles; this region is critical for determining the passability of the space. If there are ambiguous matchings, an actual situation depends on the actual state of the matchings, and is undecided in advance. In such a case, possible situations and their probabilities are calculated. 3
Uncertainty Modeling of SegmentBased Stereo
This section describes the model of uncertainty of our stereo method [7]. We use a straight line segment
prob.
able
pass
(a)
d : distance
between points
d
Wr
prob.
(b)
Wr : width of
le
ssab
impa
d
Wr
the robot
prob.
d
cide
unde
(c) Wr
d
Figure 3: Three possible relations between the robot width and the distance between two points. as a primitive because there are many line segments in indoor scenes. 3.1
Matching Process
Potential match of segments are rst detected based on the epipolar and the directional constraint; a pair of segments can be matched if their vertical positions overlap each other to a certain extent and they have similar directions. If two segments have almost opposite directions, they are accepted as a matched pair because the direction of an occluding edge may be opposite in the left and the right image. In order to decide the matching among the candi-
dates, we employ a local disparity histogram [9]. In this method, we assume that the disparity is almost constant in a local region. A disparity histogram is calculated from the possible matching pairs of segments in the region; each candidate disparity is weighted by the sum of the length of segments which have that disparity. If the histogram has one prominent peak, this peak indicates the disparity of the local region. The actual matching in the region is established based on the determined disparity. From each matching of segments, a 3D line segment is calculated by triangulation. 3.2
Fig.4, for example, shows a top view of 2D segments calculated from a pair of images shown in Fig.1. Each circle indicates the mean of the 2D normal distribution of an endpoint of a segment. A thick line connecting two circles is a 2D segment. Each thin line connecting 2D segments indicates the ambiguous matching of a segment. Positional uncertainties are also indicated.
Model of Quantization Error
The positional distribution of a segment in a real space depends on edges used by stereo matching. The horizontal position of a segment in the image is calculated from edges in the vertically overlapping part of the segment. The distribution of the horizontal position is calculated from the positional distributions of the edges by the least squares method. Assuming that the horizontal position of each edge is normally distributed, the horizontal position of each segment also follows a normal distribution. By linearizing the equation of the image projection, the positional distribution of an endpoint of a segment in a real space is represented by a 2D normal distribution. 3.3
field of view
Figure 4: Calculated 2D positions with matching ambiguity and positional uncertainty.
Model of Ambiguous Matching
If the disparity histogram of a local region has multiple prominent peaks, there are multiple candidate disparities in the region. If a segment in the left image has multiple matching candidates corresponding to some of these disparities, we cannot decide which one is true. In such a case, the possible matchings are all kept in the model, and the ambiguity is resolved by the subsequent observations if necessary. For each matching candidate, a 3D segment in a real space is calculated. The 3D segment is then projected into a 2D segment. In case of multiple matching candidates, we assign a probability to each 2D segment. Let n denote the number of possible 2D segments, and ak denote the size of the peak corresponding to the k -th segment. The probability P k of the k -th segment is determined by
4
Modeling Algorithm
This section describes the modeling algorithm in detail. We divide 2D segments in a real space into two categories: unambiguous segment and ambiguous segment. One ambiguous segment has multiple possible positions corresponding to possible matchings. In modeling, we rst use only unambiguous segments, and then add ambiguous segments into the model. The nal output of the modeling process is a set of possible situations with their probabilities. 4.1
Modeling ments
using
Unambiguous
Seg-
(1)
Modeling using unambiguous segments is performed by the following four steps:
Each 2D segment has the positional uncertainty calculated from the model of the quantization error.
Eliminate isolated short segments: Since isolated short segments possibly are obtained from false matchings, such segments are eliminated.
P
k
=
ak
Pn
j =1
aj
:
Classify relations between endpoints of segments and cluster endpoints: Relations between
endpoints are classi ed into three categories: passable, impassable, and undecided. The relation between two endpoints of a segment is assigned to be impassable. Endpoints are then clustered into obstacles. Each obstacle could be approximated by a plane gure such as a polygon. Determine critical regions: As mentioned above, a critical region is critical for determining passabilities between obstacles. Since any undecided relations are critical, a critical region is set to cover all undecided relations as shown in Fig.5. In application to planning, the distribution of the distance between two obstacles is necessary [6]. We use the most critical (narrowest) undecided relation in a critical region in order to calculate the distribution. Determine occluded areas: Considering the visibility of segments from a current observation point, the area behind the obstacles is determined as an occluded area. If an area is fully occluded, the passability of the area is completely unknown. If an area is partially occluded (e.g. the area between the bookshelf and the chairs at the right part of Fig.1), the passability of the area is conjectured to a certain extent. Thus, the passability of an occluded area could be decided according to the degree of occlusion. At present, an occluded area is treated as a free space. Fig.6 shows the modeling result using unambiguous segments in Fig.4. Obstacles, critical regions, and occluded regions are indicated. 4.2
Add Ambiguous Segments into the Model
After modeling obstacles and free spaces using unambiguous segments, ambiguous segments are added into the model. If there are n ambiguous segments and each ambiguous segment has mn candidate matchings, Q it is, in principle, necessary to investigate ni=1 mn situations. Since this computation may cause combinatorial explosion, we want to reduce both n and mn as many as possible. For this purpose, we perform the following operations: Eliminate isolated short segments: Similar to the case of unambiguous segments. Grouping ambiguous segments: If several ambiguous segments may have almost the same combination of candidate positions, the segments are considered to belong to one object. Instead of investigating every combination of possible states, the ambiguity of one unambiguous segment is considered (see Fig.7).
critical region undecided relation
Figure 5: Calculation of a critical region.
occluded region obstacle−4
obstacle−3
critical region between 2 and 3
obstacle−2 critical region between 1 and 2 obstacle−1
Figure 6: Modeling result using only unambiguous segments.
Eliminate segments which do not a ect the passability: If the ambiguity of a segment does not
a ect the passability of any space (i.e. does not change the topological structure of possible paths), the segment is eliminated. Such segments are shown in Fig.8. In the gure, every position of segment q makes space A-C impassable. Thus, obstacles A and C are merged and segment b is eliminated. Although the lower position of segment p may change the cost of the path through space B -C , we neglect it for reducing the cost of planning. After the above operations, the remaining ambiguous segments are divided into mutually-independent groups by considering the e ects of the segments against the passability; all ambiguous segments in a group should be considered at the same time. In Fig.9, for example, segments p and q , segments r and s, and segments s and t are responsible for the passability of space A-B , space B -C , and space A-C ,
p
A p
q
r
r
respectively. Therefore, there are two groups: fp; q g and fr; s; tg. For each group, possible combinations of segment positions are enumerated, and the situation and its probability is calculated for each combination. If the passability of a space is undecided, a critical region including both unambiguous and ambiguous segments is calculated. By combining the results calculated from the segment groups, the nal set of situations is obtained. The probability of a situation is given by the product of probabilities of selected positions of the segments. Suppose the k (i)-th position is selected for the i-th segment in a situation. The probability P (k (i)) of the selection for the i-th segment is given by Eq.(1). The probability of the situation is given by
P
0
k
(1); k (2); : : : ; k(n)
1
=
n Y P
0
1
()
k i
i=1
( ) = 1; : : : ; m(i)
k i
C q
Figure 8: Eliminate segments which do not a ect the passabilities.
Figure 7: Group similar ambiguous segments into an object.
s
B
C
B
t
A
Figure 9: Grouping WTA segments.
obstacle−4
b2 c2
a2
obstacle−3
obstacle−2
c1 b1
obstacle−1 a1
(2)
;
;
where n is the number of the segments, m(i) is the number of positions for the i-th segment. Fig.10 shows remaining ambiguous segments. Possible positions of the segments are indicated by large circles. Since there are three ambiguous segments and each segment has two possible positions (a1 -a2 , b1 -b2 and c1 -c2 ), there are eight possible situations. Two of them are shown in Fig.11. The probability and the passability of the space between obstacles are indicated. Thin dashed lines in spaces 1-2 and 2-3 indicate undecided relations in modeling using unambiguous segments (cf. Fig.6). By adding ambiguous matchings, impassable relation (bold line) and undecided relation (bold dashed line) are newly detected. The topological structure of possible paths is easily extracted from the result.
Figure 10: Remaining ambiguous segments
4
4 3
2 1
3
2
impassable undecided
1
(a) prob.=0.015 (b) prob.=0.036
Figure 11: Two of possible situations: (a) c1 are true; (b) a 2 , b2 and c2 are true.
a2
,
b2
and
5
Application to Vision-Motion Planning
In our previous paper [6], we formulated a visionmotion planning under uncertainty based on probabilistic decision theory, where uncertainty of properties of the environment are modeled by probabilistic distributions and the optimal (expectation-minimum) sequence of vision and motion operations is recursively determined. In solving a planning problem, possible alternatives at each decision point should be provided. For the problem that a mobile robot goes to the goal point among obstacles, important decisions are made on path selection and viewpoint selection. Our modeling method can provide sucient alternatives for both of the decisions: the topological structure of possible paths and critical regions to be observed. If some of paths are not related to the problem, we can neglect them, thereby reducing the computational cost. 6
Conclusions and Discussion
We have proposed a new method of modeling obstacles and free spaces from a set of 3D segments obtained by a segment-based stereo. By considering the uncertainties and the ambiguities of data, possible situations and their probabilities are calculated; each situation consists of the description of obstacles and critical regions between the obstacles. The proposed method does not generate a precise description of the environment but eciently extracts sucient information for vision-motion planning of a mobile robot. In practical applications, it is unusual that a robot moves in a completely unknown environment. In most cases, a robot might be given a rough map of the environment; \rough" means that there may be unknown objects or the map may have an error. In such cases, by matching the modeling result with the map, more correct modeling is achieved. Developing an ecient matching method is a future work. It is also necessary to extend the proposed method so that it can incrementally model the environment using data from a sequence of observations. Acknowledgement
This work is supported in part by a research grant from Kumahira Security Foundation, Japan.
References
[1] U.R. Dhond and J.K. Aggarwal. Structure from stereo: A review. IEEE Trans. on Systems, Man, and Cybernet., Vol. 19, No. 6, pp. 1489{1510, 1989. [2] T. Echigo. Segmentation of a 3d scene into free areas and object surfaces by using occluded edges of trinocular stereo. In Proceedings of 1991 IEEE/RSJ Int. Workshop on Intelligent Robots and Systems, pp. 863{868, 1991. [3] A. Elfes. Sonar-based real-world mapping and navigation. Int. J. of Robotics and Automat., Vol. 3, No. 3, pp. 249{265, 1987. [4] O.D. Faugeras, E. Le Bras-Mehlman, and J.D. Boissonat. Representing stereo data with the delaunay triangulation. Arti cial Intelligence, Vol. 44, pp. 41{ 87, 1990. [5] J.J. Leonard, H.F. Durrant-Whyte, and I.J. Cox. Dynamic map building for an autonomous mobile robot. Int. J. Robotics Res., Vol. 11, No. 4, pp. 286{298, 1992. [6] J. Miura and Y. Shirai. Vision-motion planning with uncertainty. In Proceedings of 1992 IEEE Int. Conf. on Robotics and Automat., pp. 1772{1777, 1992. [7] J. Miura and Y. Shirai. An uncertainty model of stereo vision and its application to vision-motion planning of robot. In Proceedings of the 13th Int. Joint Conf. on Arti cial Intelligence, pp. 1618{1623, 1993. [8] P. Moutarlier and R. Chatila. Incremental free-space modeling from uncertain data by an autonomous robot. In Proceedings of 1991 IEEE Int. Workshop on Intelligent Robots and Systems, pp. 1052{1058, 1991. [9] O. Nakayama, A. Yamaguchi, Y. Shirai, and M. Asada. A multistage stereo method giving priority to reliable matching. In Proceedings of 1992 IEEE Int. Conf. on Robotics and Automat., pp. 1753{1758, 1992. [10] F. Nashashibi and M. Devy. 3d incremental modeling and robot localization in a structured environment using a laser range nder. In Proceedings of 1993 IEEE Int. Conf. on Robotics and Automat., pp. 1:20{ 27, 1993.
