Proceedings of the 8th International IEEE Conference on Intelligent Transportation Systems Vienna, Austria, September 13-16, 2005
TA2.5
Detection of Pedestrian Crossing and Measurement of Crossing Length - an Image-Based Navigational Aid for Blind People Mohammad Shorif Uddin and Tadayoshi Shioyama Abstract— Pedestrian crossings are dangerous places for a visually impaired person to cross safely. This paper describes a simple method for the detection of the location of a pedestrian crossing as well as measurement of its length to enhance the safety and mobility of blind people while crossing a road. A crossing is characterized by zebra pattern i.e. by evenly spaced white stripes on a usual black road surface. Detection of the crossing location as well as measurement of its length is done using this criterion from the image captured by a single camera. The method at first checks for the existence of a crossing; if there exists a crossing then it goes to measure the crossing length. Experimental results on a large number of street scenes with and without crossings demonstrate the effectiveness of the proposed technique.
I. INTRODUCTION
W
ORLD Health Organization reported that approximately 40 million people are blind all over the world [1]. This paper intends to develop a safe road crossing system for these visually impaired people. Usually, blind people use a white cane as a travel aid. The range of detection of special patterns or obstacles using a cane is very narrow. To improve the versatility of the white cane, various devices have been developed such as the SONICGUIDE [2], the Mowat sensor [3], the Laser cane [4] and the Navbelt [5]. However, these devices are not able to assist the blind at a pedestrian crossing where information about the location of a crossing, an idea about its length and the state of traffic lights is important. Some traffic lights have beepers, which prompt the blind person to cross the road, when it is safe to do so. However, such equipment is not available at every crossing; perhaps, it would take too long for such equipment to be installed and maintained at every crossing. Blind people obviously can not see, but can hear. The arrival of fast and cheap digital portable laptop computers with multimedia computing to convert audio-video streams in real time opens new avenues to develop an intelligent navigation system for blind people. In this paper, we concentrate on the detection of the location of a pedestrian crossing as well as measuring crossing length. Our aim is to develop a device with which the blind can autonomously detect important information for safely negotiating a crossing using a single camera. A single camera M. S. Uddin is with the Department of Electronics and Computer Science, Jahangirnagar University, Savar, Dhaka 1342, Bangladesh. He has been pursuing this research as a Research Fellow with the Department of Mechanical and System Engineering, Kyoto Institute of Technology, Matsugasaki, Sakyo-ku, Kyoto 606-8585, Japan
[email protected] T. Shioyama is with the Department of Mechanical and System Engineering, Kyoto Institute of Technology, Matsugasaki, Sakyo-ku, Kyoto 6068585, Japan
0-7803-9215-9/05/$20.00 ©2005 IEEE.
method has the advantage over a stereo method. Because it leads to a device with a simple structure and does not need camera calibration since the necessary information can be obtained using only a camera coordinate system rather than a world coordinate system. A crossing is characterized by zebra pattern i.e. by evenly spaced white bands on a usual black road surface, which can be treated as a bipolar pattern. In Japan, the width of each white or black band of crossing is 45 cm. At first, candidate crossing areas are extracted from an image using bipolarity feature. Then, the existence of crossing is detected by careful evaluation of the candidate areas on the basis of crossing position, crossing direction, number of crossing bands as well as band width trend. If there exists a crossing, then the technique goes for determination of its length. The crossing length is calculated using the number of white and black bands in the crossing region. In order to evaluate the performance of the proposed method, an experiment is performed using real road scenes with and without pedestrian crossings. The rest of this paper is organized as follows. Section II describes the previous works related to this research. Section III explains the principle of the proposed technique. Method for the detection of the existence of crossing as well as measurement of crossing length is presented in section IV. Section V shows the experimental results, followed by the conclusions and future research directions in Section VI. II. PREVIOUS WORKS From the viewpoint of an image-based blind navigation system, Stephen Se [6] first proposed a pedestrian crossing detection by grouping lines and checking for concurrency using the vanishing point constraint. However, a thorough evaluation of this technique has not yet performed and the technique works slow and is far from real time. Meijer’s [7] “vOICe” consisting of a head-mounted camera, stereo headphones and a laptop, is the only commercially available vision-based travel aid that uses 1-to-1 image-to-sound mapping. Though it can recognize the walls, doors etc., a pedestrian crossing detection technique is still absent in it. Recently, we proposed a crossing detection system [8] based on the Fisher criterion. The method is very simple, but as the feature points are extracted on a single vertical line in an image, its reliability is not upto the mark. Compared to that technique the present method has multiple checking criteria and hence its reliability is high. Previously, Shioyama et al. [9] reported a crossing length measurement technique, which is based on edge
578
detection, Hough transformation and Susan feature detection techniques. However, it is complicated, computationally inefficient and needs many thresholding parameters to adjust. Recently, we presented a crossing length measurement technique [10] assuming that the image contains a crossing using the information of crossing bands, which is simple, stable and computationally faster than Shioyama’s method. However, we did not use the segmented area based on bipolarity, which we incorporated in the present method. One of the goals of the present paper is to unify the two steps – detection of existence of crossing and measurement of crossing length.
µ1 ) and p2 (x) = δ (x − µ2 ). So, γ = 1 corresponds to perfect bipolarity and γ = 0 represents the absence of bipolarity. Let, |µ1 − µ2 | |µ1 − µ2 | and σ2 ≤ , n n where n is a real number. Then, (3) implies d = |µ1 − µ2 | , σ1 ≤
γ≥
α (1 − α )d 2 1 = 1− . 1 + α (1 − α )n2 d 2 n12 + α (1 − α )
(4)
Therefore, using (4), one can estimate a lower bound of γ taking α as a parameter. For example, γ must be greater than 0.8 when there are p1 and p2 such that α = 0.5 and n ≥ 4.
III. CROSSING REGION EXTRACTION PRINCIPLE Some images of real road scenes containing pedestrian crossings are shown in Fig. 3(a)-(d). The crossing pattern can be treated as a bipolar region. To extract the crossing region in the image, first, candidate regions are identified based on the bipolar nature of their image intensities. These regions are then corrected for viewing angle. This correction allows a final labeling based on the number and nature of black-white and white-black transitions within a region. As we have used bipolarity as the main feature in extracting candidate crossing areas, in the following subsection A, first we define bipolarity and then show a way to estimate it. As a blind person is interested in detecting a frontal crossing not a sided one, estimation of crossing direction is important. We present the principle of crossing direction estimation in subsection B. The Crossing direction estimation principle is also used in the crossing length measurement process. In subsection C, we describe the crossing length estimation model.
If there is a pedestrian crossing in the image, then θ will closely correspond to the direction of crossing bands. The differential of i(x, y) along v direction includes alternate peaks and valleys on the edges of crossing bands (i.e. from black to white and vice versa). Therefore, ∂∂ vi has local extremes about direction v on the edges of crossing bands. For simplicity, we used i instead of i(x, y). Since the edges of crossing bands are straight lines, the integration along u direction emphasizes the local extremes. Consequently, one can find crossing bands by analyzing the projection as
A. Bipolarity We denote the intensity distribution of an image block as p0 (x). If the block contains only black and white pixels, then p0 (x) can be written as p0 (x) = α p1 (x) + (1 − α )p2 (x), where 0 ≤ α ≤ 1, p1 (x) is the intensity distribution of black pixels and p2 (x) is the intensity distribution of white pixels. Let define some variables as:
du, (6) ∂v which is a one-dimensional function about v. Then, the integral of the square of (6) becomes a good measure of closeness to true crossing direction. Accordingly, we use θ that maximizes ∞ ∞ ∂ i 2 (7) du dv. −∞ −∞ ∂ v
∞
∞
B. Principle of the Extraction of Crossing Direction Let i(x, y) be the intensity of an image, where x and y are the horizontal and vertical spatial coordinates, respectively. If the image rotates by an angle of θ , then we can write expressions for the rotated coordinates u and v as follows. u = + x cos θ + y sin θ , v = − x sin θ + y cos θ .
(5)
∞ ∂i −∞
(i = 0, 1, 2), Using Parseval’s formula, one can derive the following equation. (1) ∞ ∞ 1 ∞ 2 ∂ i 2 where µi and σi 2 represents mean and variance, respectively. dv = ζ |I (−ζ sin θ , ζ cos θ )|2 d ζ , du 2π −∞ Using the above relations, we can write σ0 2 as −∞ −∞ ∂ v (8) σ02 = ασ12 + (1 − α )σ22 + α (1 − α )(µ1 − µ2 )2 . (2) where I(ξ , η ) is the Fourier transform of i(x, y) and is given ∞ ∞ Equation (2) shows that the total variance consists of the by I(ξ , η ) = i(x, y)e− j(ξ x+η y) dxdy, (9) weighed sum of variances and the difference of means. If −∞ −∞ 2 2 σ0 ≈ α (1− α )(µ1 − µ2 ) , then p0 (x) can be said to be almost j is a complex operator and ζ = −ξ sin θ + η cos θ . bipolar. So, we define bipolarity γ as Although the left hand side of (8) includes two integrations 1 2 depending on θ , the right hand side includes only one (3) γ ≡ 2 α (1 − α )(µ1 − µ2 ) . σ0 integration depending on θ . Therefore, we find θ that Equation (3) implies that 0 ≤ γ ≤ 1. If γ = 1, there are α , maximizes the right hand side of (8). p1 and p2 such that σ1 = σ2 = 0. This means p1 (x) = δ (x −
µi =
−∞
xpi (x)dx, σi2 =
−∞
(x − µi )2 pi (x)dx,
579
Color to gray scale conversion
C. Crossing Length Estimation Model We use the projection model shown in Fig. 1 in estimating the crossing length. Let yn and y0 be the positions of the first and the end lines of crossing in the image plane, respectively, f is the focal length of the camera, h is the camera position from the road surface, d is the crossing length (i.e. the horizontal distance from the observer to the end line of crossing) and w is the width of a crossing band. If there are n number of bands in the crossing area, then we may easily find the following relations from Fig. 1. y0 =
fh d
and
yn =
fh . d − nw
Segmentation Extract highly bipolar and wide candidate regions Fill small areas inside the candidate and refine
Y
Is the location of candidate appropriate?
N
Y Y Crossing Direction > 15 deg
(10)
Y
N
Then, we get yn − y0 = f h ×
nw . d(d − nw)
Crossing Y bands number >= 8
(11)
N
Is there any more candidate?
Y
We can derive the expression for crossing length from (11) as 1 4nw f h 2 nw + (nw) + . (12) d= 2 yn − y0
Check band width trend
White bands number >= 4
N N
Y
In (12) w, f and h are known.
There is a crossing
There is a NO crossing
Determination of length using crossing region
end line
Fig. 2.
Flow diagram of the proposed method.
2nd line 1st line
First, determine the largest region that has bipolarity higher than 0.80. Then extract the candidate regions, which have bipolarity greater than 0.80 and areas more than 50% of the largest region’s area. The above values are chosen based on experimental data.
d y0 yn
image plane
f h camera center optical axis
Fig. 1.
ground plane
Crossing length estimation model.
IV. EXPERIMENTAL METHOD A flow diagram of the proposed method is presented in Fig. 2. A. Method for Detection of Existence of Crossing At first, the color image is converted to a gray scale image. Then the image is partitioned into equal-size rectangular blocks sized (16 × 16) pixels. Detection of the existence of a crossing is performed according to the following procedure. We follow the following two steps to find the crossing region candidates. 1) This step identifies homogenous bipolar regions. Segmentation is carried out by merging neighboring blocks of similar pattern on the basis of intensity distributions. 2) Only keep regions that are sufficiently bipolar. Calculate the bipolarity of each segmented region.
580
Then, we follow the following steps in checking each candidate region. 3) This section refines the segmentation. First, check the largest area candidate region. If there exists a small region of a different label within this candidate region, then fill it with same label if its area is less than 5% of the candidate area. Then refine the bottom boundary region of the crossing area. The refinement is done by assigning same label to the pixels on a different label horizontal line if its leftmost and rightmost pixels lie in the candidate region. 4) Make sure the region is centered in the image. Check the location (either it is in appropriate position or too left or too right or too far w. r. t. the observer) of the candidate region. If the total candidate region lies in a position, which is situated within 20% of image width from the left or right boundaries, then the position of the candidate region is treated too left or too right, respectively. If y-coordinate of the bottommost point of the candidate region is situated more than 40% of image height from the bottom boundary, then the position of the candidate region is treated too far from the observer. If the candidate region is not in
an appropriate location then decide that there is no crossing for this candidate and go to examine the next candidate region (if any). 5) This section corrects for viewing angle. Calculate the crossing direction using the original image content at the location of the candidate region using (8) with the help of the two-dimensional fast Fourier transform. If the crossing direction is greater than a threshold (15◦ is used here) then decide that there is no crossing for this candidate, as the crossing direction is too steep and go to examine the next candidate region (if any). 6) Remove regions with too few crossing bands. Binarize (0, 1) the original image content at the location of the candidate region using the mean. Use median filtering of window size (5 × 5) pixels on the binarized image to eliminate sporadic noise. Take the mean value of binarized pixels of the candidate region along the u-axis direction (i.e. along the crossing direction) for each v-axis position. From this result determine the number of crossing points when the mean value becomes 0.5. If the number of crossing points (i.e. number of black and white bands) is less than eight then decide that there is no crossing for this candidate and go to examine the next candidate region (if any). 7) Remove regions with too few white bands. Calculate the band-width pattern of existing bands. Usually, the band width is decreasing in image plane from nearer to more far locations (w. r. t. the observer). Let, d0 , d1 , d2 , · · · , dn are the widths of 0th, 1st, 2nd, · · · , nth bands, respectively. Starting from the nearest location, determine the largest continuous set of bands for each location where (dn−1 + 4) ≥ dn and (dn−2 + 2) ≥ dn .
1) Integrate the crossing region of the image along the crossing direction (i.e. along the u-axis direction) for each v-axis position. To find the location of edges of white to black or black to white crossing bands, we use the differentiation along v direction of the integral data. There will be local extremes (maximum or minimum) at the position of edges. 2) Extract important extremes (true band edges) by comparing the absolute value of the differentials. First, find a local extreme that has the maximum absolute value of the differential. Then search a new local extreme whose value lies in the interval from 0.4 to 6.0 times of the previously selected extreme. Each newly selected extreme becomes reference for the next search of local extreme selection. In this way, the search and selection are continued for all local extremes. Omit the local extremes that are not selected. The parameters used in this step are also selected on the basis of experimental data. 3) Remove the local adjacent extremes of the same sign. Starting from the farthest local extreme (on the basis of y position in the image), if there exists two adjacent extremes of the same sign, remove the extreme that has lower value. 4) Count the number of local extremes and determine y positions of the first and the last local extremes. Then calculate the crossing distance using (12).
(13)
In real road scenes, there are noises and also the white paintings on crossings are not 100% perfect, therefore, disturbances may occur in band widths. To cope with these, we add 4 and 2 pixels in the first and the second conditions of (13). Choose the maximum number of bands that satisfy the band width pattern among all locations. A band is taken as a white band if the integration result at the center of the two successive crossing points is greater than 0.5. If there exists at least four successive white bands that satisfy the band width pattern of crossing then decide there is a crossing in the considered image; otherwise, decide no crossing for this candidate and go to check the next candidate region (if any). White band number 4 is taken on the basis that there are some road marking (not crossing) consists of three painted stripes. This will safeguard against false positive detection result. An image of this type is shown in Fig. 3(e). B. Method for Estimating Crossing Length We use the following steps in estimating the crossing length.
(a)
(b)
(c)
(d)
(e)
(f)
Fig. 3. Some experimental images: existence of crossing is detected in (a) and (b), false negative is obtained in (c) and (d), no crossing is detected in (e) and (f).
581
TABLE I C ROSSING D ETECTION R ESULT S UMMARY.
V. RESULTS AND DISCUSSIONS To evaluate the performance of the proposed method we used 113 real images with different backgrounds taken by a commercial digital camera. Among them, 79 images include crossing and the rest 34 are of no crossing. Fig. 3 shows some samples of experimental images. The size of the images is (width × height) = (640 × 480) pixels. Fig. 4 presents the obtained results at different steps in detecting the existence of crossing for the image of Fig. 3(a).
Image with
Image without
Decision
crossing
crossing
crossing
74
0
(Ok)
(False positive)
No crossing
5
34
(False negative)
(Ok)
79
34
Total number of images
200 projection imp_local_extremes
150
(a)
100
(b) 1 integration crossing points
0.9
50
0.8 0.7 0.6
0
0.5 0.4 0.3 0.2
-50
0.1
0
0 0
(c)
50
100
150
200 250 300 y-position [pixel]
350
400
450
500
(d)
50
100
150
200 250 300 v-position [pixel]
350
400
450
(a)
Fig. 4. Obtained results at different steps in detecting the existence of crossing for the image of Fig. 3(a): (a) bipolarity of each segmented regions, (b) original image (crossing area) at the location of candidate region, (c) binarized crossing area, (d) mean value of integration along the crossing direction of the binarized image.
The complete result of the detection of the existence of crossings is summarized in Table I. From this table, we see that the proposed algorithm is quite successful in detecting the existence of crossings from real road images. The method has not made any dangerous (false positive) error such that it decides the existence of a crossing for a scene without crossing. For a scene containing a crossing where the white paintings on the crossing are damaged or the scene contains too few crossing bands - very short crossing (less than 4 white bands) then it decides nonexistence of a crossing (i.e. false negative). This has happened only in 5 cases among 113 experimental images. Among them, 4 cases contained less than four white bands (an image of these type is shown in Fig. 3(c)) and in the rest case (shown in Fig. 3(d)) the white paintings are not perfect, as these are done on an unusual pattern of road surface. Extremely high accuracy is required for detection of the existence of crossing, as a false positive error is life threatening for a pedestrian. If there is a crossing in the vicinity, the algorithm is also able to give a message to the user on the basis of position (such as left, right or more front) and direction of the crossing. Crossing length measurement is done for 74 images. Fig. 5 shows the extraction process of crossing bands, which are used in estimating crossing length for the image of Fig. 3(a). For this image, the true crossing length is 16.4 m and the
(b) Fig. 5. Experimental result of the image of Fig. 3(a) for estimation of crossing length: (a) extraction process of band edges using crossing region only (”projection” indicates mean value of integration and “imp local extremes” indicates local maximum or minimum of differentials i.e. positions of band edges), (b) straight line marks are drawn at the positions of extracted band edges. The true crossing length is 16.4 m and the estimated length is 16.7 m.
estimated length is 16.7 m. The technique correctly estimated the crossing length. The average estimation error is 9.9% and the r.m.s. error is 2.0 m for the total 74 images. The maximum relative error is 30%, which is relatively high. This high error occurred as the segmentation technique could not extract the full crossing region on the basis of bipolarity. The crossing stripes at the end portion of the crossing region are not clear in this crossing image. The accuracy depends on the clear white paintings on the crossings. A realistic device utilizing the proposed algorithm can
582
be constructed for visually impaired persons using a small digital camera that would be clipped onto a pair of glasses and connected to a small wearable computer (PDA, like a cell phone set) that analyzes the images. A voice speech system relays information and instructions to the user through a small speaker placed near the ear. VI. CONCLUSIONS AND FUTURE WORKS In this paper, a simple and real-time method for the detection of the existence of a crossing as well as measurement of crossing length has been proposed. Experimental results with real road scenes confirmed its effectiveness. In detecting crossings, the method has not made any dangerous (false positive) error such that it decides the existence of a crossing for a scene without a crossing. For a crossing scene where the white paintings on the crossing are damaged or the scene contains too few crossing bands, then it decides nonexistence of a crossing (i.e. false negative). This has happened only in 5 cases among 113 experimental images. The method is successful in measuring the length of the crossing with reasonable accuracy if the image resolution is good and the white paintings on the crossing are fine as well as there is no obstruction on the crossing. We hope this technique will help in improving the mobility of blind people. We have not tested our algorithm under various illumination conditions yet. In the future, we would like to include illumination invariant strategy to make it robust to illumination changes. Beside this, we would like to include a technique for detection of the state of traffic lights and a preprocessing vehicle detection technique for making a complete road crossing system.
Matsuo of Kyoto Institute of Technology, Japan. They are also grateful for the support of Japan Society for the Promotion of Science under Grants-in-Aid for Scientific Research (No. 16500110 and No. 03232). R EFERENCES [1] World Health Organization, “Blindness and visual disability: Seeing ahead-projections into the next century,” WHO fact sheet no. 146, 1997. [2] L. Kay, “A sensor and to enhance spatial perception of the blind, engineering design and evaluation,” Radio Electron. Eng. 44, pp. 605629, (1974). [3] D. L. Morrissette et al., “A follow-up study of the Mowat sensor’s applications, frequency of use, and maintenance reliability,” J. Vis. Impairment and Blindness 75, pp. 244-247, (1981). [4] J. M. Benjamin, “The new C-5 laser cane for the blind,” in Carnahan Conf. on Electronic Prosthetics, pp. 77-82, (1973). [5] S. Shoval, J. Borenstein, Y. Koren “Auditory guidance with the navbelt-a computerized travel aid for the blind,” IEEE Trans. Syst. Man Cybern. C28, pp. 459-467, (1998). [6] Stephen Se, “Zebra-crossing detection for the partially sighted,” Proc. IEEE Computer Society Conf. Computer Vision and Pattern Recognition (CVPR), South Carolina, USA, Jun. 2000, vol. 2, pp. 211-217. [7] P. B. L. Meijer, “Vision technology for the totally blind,” [Online]. Available: http://www.seeingwithsound.com/ [8] T. Shioyama and M. S. Uddin, ”Detection of pedestrian crossings with projective invariants from image data,” Meas. Sci. Technol. 15, no. 12, pp. 2400-2405, (2004). [9] T. Shioyama, H. Wu, N. Nakamura, S. Kitawaki, “Measurement of the length of pedestrian crossings and detection of traffic lights from image data,” Meas. Sci. Technol. 13, pp. 1450-1457, (2002). [10] M. S. Uddin and T. Shioyama, “Measurement of the length of pedestrian crossings - a navigational aid for blind people,” Proc. IEEE Intl. Conf. on Intelligent Transportation Systems (ITSC2004), Washington, USA, Oct. 2004 pp. 690-695.
ACKNOWLEDGMENTS The authors gratefully acknowledge the suggestions and comments of Professor Yasuo Yoshida and Mr. Tadashi
583
