Proceedings of the 2005 IEEE International Conference on Mechatronics July 10-12, 2005, Taipei, Taiwan
Automatic Speed Measurements of Spherical Objects Using an Off-the-Shelf Digital Camera Huei-Yung Lin and Chia-Hong Chang Department of Electrical Engineering National Chung Cheng University 160 San-Hsing, Min-Hsiung, Chia-Yi 621, Taiwan
[email protected],
[email protected]
algorithms use reference information in the scene, such as the distance traveled across image frames, to compute the speed of the moving object. In this work, we propose an approach for speed measurements of spherical objects based on a single image taken by a hand-held digital camera. The basic idea of our approach is as follows. Due to the relative motion between the camera and the moving object during the finite exposure time of the camera, motion blur will appear on the dynamic regions of the image. For any fixed shutter speed, the moving distance of the object is proportional to the amount of motion blur caused by the imaging process. Thus, if the direction and extent of the motion blur can be identified, it is possible to recover the physical displacement of the moving object using the intrinsic and extrinsic parameters of the camera. Since the projection of a sphere-like object on the image plane is circular, more accurate motion extent of the object can be obtained by applying a circle fitting algorithm on a sequence of deblurred images created with the initial blur extent estimation. Finally, the speed of the object can be calculated by means of displacement divided by the camera exposure time. To make the system fully automatic, a target object detection algorithm using motion blur information is proposed. The identified target region is extracted from the original image and used for speed measurement. The proposed method also has the advantages of reducing the computation and increasing the robustness of the proposed speed measurement approach. Generally speaking, image degradation caused by motion blur can be classified as either spatially invariant or spatially variant distortions based on the formation of the imaging process. Spatially invariant distortion corresponds to the cases that the image degradation model does not depend on the position in the image. This type of motion blurred images is usually a result of camera movement during the imaging process. Restoration of spatially invariant motion blurred images is a classic problem and several approaches have been proposed in the past few decades [3], [4]. The goal is to find the point spread function (PSF) of the blurring system and then use deconvolution techniques to restore the ideal image. The motion blur we have to deal with appears in the image containing a fast moving object and recorded by
Abstract— A method for automatic speed measurements of spherical objects using a digital camera is presented. Conventional speed measurement techniques use radar or laser based devices, which are usually more expensive compared to a passive camera system. In this work, a single image captured with object motion is used to estimate the speed of a spherical object (such as baseball). Due to the relative motion between the camera and a moving object during the camera exposure time, motion blur occurs in the dynamic region of the image. By identifying the motion blur parameters, the speed of a moving object can be obtained. Automatic target identification and motion estimation are first done by motion blur analysis, followed by more accurate blur identification using circle fitting of the spherical object. Finally, the object speed is calculated according to the imaging geometry, camera pose, and blur extent in the image. Index Terms— speed measurements, motion blur, image analysis, computer vision
I. I NTRODUCTION In recent years, because of advances in the development of various aircraft, baseball exercise, and electronic devices, people are becoming more interested in knowing the speed of flying objects, the pitching speed of sports players, and the speeding of motor vehicles detected by law enforcement agencies. Currently, most speed measurements of moving objects are achieved by microwave radar or laser devices. Radar-based devices bounce a radio signal off to a moving object, and the reflected signal is picked up by a receiver. The radar receiver then measures the frequency difference between the original and reflected signals, and converts it into the speed of the moving object. A laser-based device times how long it takes a light pulse to travel from the laser gun to the moving object and back. Based on this information, it can quickly find the distance between the gun and the object. By making several measurements and comparing the distance the object traveled between measurements, a laser-based device can accurately determine the speed of the object. Due to the availability of low-cost imaging hardware in the past few years, video-based methods have been proposed for speed estimation of moving vehicles [1], [2]. These methods use reference information in the scene, such as the distance traveled across image frames, the speed of the vehicle is then estimated by the distance divided by the inter-frame time. Most of the video-based speed estimation
0-7803-8998-0/05/$20.00 ©2005 IEEE
66
detector
−1 0 −1 0 −1 0
is extended to an n × n mask of the following form −1 · · · − n−1 2 −1 · · · − n−1 2 .. .. . . −1 · · · − n−1 2 −1 · · · − n−1 2
Fig. 1.
1 1 1
(where n is an odd number) 0 0 .. .
n−1 2 n−1 2
0 0
n−1 2 n−1 2
.. .
··· 1 ··· 1 .. . ··· 1 ··· 1
to emphasize the response on the ramp edge region. Typically, a 9 × 9 mask is used and strong response of the ramp edge spreads a few pixels, as shown in Fig. 2. Different from sharp edges or regions without texture, motion blur caused by an object moving in front of a static background gives a symmetric response on the width of the ramp edges. To identify the target region, a binary image is first generated from the gradient image with nonzero values set as 1, and the run-length code is created for each image scanline [12]. Some of the run-length codes are shown below for the marked regions in Fig. 2, where the first half corresponds to the red region and the second half corresponds to the yellow region. 7, 6, 5, 44, 41, 6, 1 5, 8, 5, 44, 41, 5, 2 8, 5, 5, 44, 41, 4, 3 0, 1, 8, 4, 4, 45, 41, 4, 3 0, 2, 6, 5, 3, 2, 3, 41, 41, 4, 3 0, 15, 9, 3, 4, 7, 8, 3, 6, 5, 6, 16, 15, 1, 2, 4, 6 0, 16, 15, 7, 8, 2, 7, 5, 5, 17, 19, 2, 7 0, 16, 15, 7, 15, 1, 2, 5, 4, 2, 1, 14, 19, 2, 6, 1 0, 17, 15, 6, 18, 5, 5, 1, 1, 14, 19, 2, 7 0, 6, 1, 10, 14, 8, 14, 14, 2, 12, 20, 2, 7 A unique pair of 1 and 0 runs which are both large and comparable in size is then identified for each runlength code. The motion blur region is then obtained by
Motion blurred image taken with a fast moving object.
a static camera. The corresponding PSF is a function of position in the image, and thus it is classified as spatially variant distortion. Image restoration of this type of distortion is considered as a more difficult problem than the spatially invariant case and has been addressed by only a few researchers [5]–[7]. In this work, we focus on blur parameter estimation and its application on speed measurement, rather than developing sophisticated image restoration algorithms. II. TARGET O BJECT D ETECTION AND I DENTIFICATION The first step of a vision-based method for speed measurement of a moving object is to identify the target automatically. One simple solution for object detection is using image subtraction with an additional background image. However, for most situations, it is not possible to obtain exactly the same scene for two image captures with a hand-held device, even without the presence of noise. Thus, registration of the images has to be performed prior to image subtraction, which complicates the object detection algorithm [8]. In this work, the target object for speed measurement is identified using only a single image with motion blur information. As shown in Fig. 1, the moving object generates unidirectional ramp edges along the motion direction in its neighborhood, while the static background regions are either focused with sharp edges or slightly blurred with isotropic defocus (caused by optical defocus). Thus, the target object can be identified by detecting the specific ramp edge regions of the image. We first assume the object is moving parallel to the horizontal image scanlines, and then deal with motion blur along any direction. Commonly used edge detection masks such as Sobel or Prewitt operators are designed for strong responses on sharp edges [9]. Although some of the previous work has been done for ramp edge detection [10], [11], these studies basically dealt with both the blurred and sharp edges. To detect the horizontal ramp edges exclusively (while suppressing the sharp edges), the horizontal Prewitt edge
Fig. 2.
67
Gradient image from horizontal ramp edge detection.
considering the vertical continuity of the above pairs of 1 and 0 runs for all image scanlines and their corresponding locations in the horizontal direction. Since only the target is used for speed measurement, a subimage centered at the detected region is extracted with typically 3 - 5 times of its height and width. Under the assumption that the object is moving in front of a uniform background (such as the sky or ground), the experiments have shown that the proposed method for target identification using motion blur is very robust. If the approximate motion direction of the object is unknown, three additional images are created by rotating the original image by 45◦ , 90◦ , 135◦ , respectively. The above algorithm is then applied on these four images to identify the target region.
Fig. 3. The original image (left), after image rotation (middle), and after Sobel detection (right).
It is well known that the response of a sharp edge to an edge detector is a thin curve, whereas the response of a blur edge to the same edge detector is spread over a wider region. This suggests that the result of edge detection can be used as an initial estimate of the motion blur extent. To obtain the blur extent along the horizontal direction, a Sobel edge detector is first applied on the subimage to find the left and right blur regions. Ideally, there will be two edges with the same width which correspond to the left and right partially motion blurred regions for each image scanline. Thus, the blur extent of a moving object can be obtained by finding the mode of the edge widths in the edge image. However, as shown in Fig. 3, the blur extents along the image scanlines are not all identical due to the circular shape of the spherical object. A good estimate can be given by taking the average of the edge widths which are larger than 75% of the largest edge width. Since the intensity variations inside the object region affect the blur extent estimation, they should be suppressed while searching for the best estimate. The steps of the algorithm for robust estimation of motion length is thus given as follows: 1) Calculate the summation of the edge widths for each row, and find the mode of the summations for the whole image. 2) Set Edgewidth as half of the edge width corresponding to the mode derived in Step 1). Set the number of iterations. 3) Compare the summation of the edge widths for each row with 2 · Edgewidth . Record the summations which are larger than 75% of 2 · Edgewidth . 4) Find the mode of the above summations for the whole image and replace Edgewidth with half of the corresponding summation. 5) Go back to Step 2) and repeat until Edgewidth converges or pre-defined number of iterations is reached.
III. M OTION E STIMATION To estimate the affine motion of a spherical object from a motion blurred image, the direction and extent of motion blur have to be identified first. A sequence of deblurred images is then obtained according to those parameters, followed by a circle fitting process to select the image with the best motion deblurred result and the corresponding blur extent. A. Blur Parameter Identification It is clear that the motion direction of a moving object (appeared in the image) is the same as the direction of motion blur. Thus, the intensity of high frequency components along this direction is decreased. As a result, the derivative of the image in the motion direction should suppress more of the image intensity compared to other directions. To identify the motion direction relative to the horizontal image scanlines, either spatial or frequency domain approaches can be used [4]. However, most of those methods consider the case that the whole image is blurred, which is different from the motion blur caused by an object moving in front of a static background. To solve this problem, a gradient based method is applied only on the subimage (containing the target object), as derived in the previous section to estimate the motion direction [13]. For simplicity of blur extent identification, the angle between the identified motion direction and image horizontal scanlines is used to rotate the image, so that the direction of object motion is reduced to a one-dimensional case. In this case, a commonly used spatially invariant model for image blur is given by g(x) = h(x) ∗ f (x)
B. Image Deblurring and Circle Fitting Due to noise, non-uniform intensity of the background, and other artifacts in real images, the blur extent estimates obtained in the previous section usually contain several pixels of error. To make the estimation more robust, a circle fitting algorithm is employed to identify the object and estimate the blur extent more accurately. First, a sequence of motion deblurred images is created by Wiener filter or blind deconvolution [15] using the estimated blur parameters. The number of deblurred images to be created depends on the size of the object appeared in the image and
(1)
where h(x) is a linear PSF, and f (x) and g(x) are the original and observed images, respectively [14]. Furthermore, the one-dimensional PSF for the uniform linear motion can be modeled as 1/R, |x| ≤ R/2 h(x) = (2) 0, otherwise where R is the extent of motion blur. That is, a step edge becomes a ramp edge with a width of R due to the uniform linear motion of the scene.
68
Fig. 4.
Motion deblurred images (top), edge images (middle), and circle fitting results (bottom).
the initial blur extent estimation. In the implementation, typically 9 deblurred images (each image with one pixel difference of motion deblurring) are generated for circle fitting. For example, if the estimated blur length is K pixels, then the images are deblurred with K − 4 to K + 4 pixels. After a sequence of deblurred images is created, a circle detection algorithm [16] is used to select the image with the most circle-like object. The selected image is referred to as the best deblurred image, and the corresponding motion blur extent is considered as an accurate motion length of the object appeared in the image. Fig. 4 shows five motion blurred images and the circle fitting results. In the experiments, the difference between the initial blur extent estimation and the result after an additional circle fitting procedure is less than 3 pixels in most cases. This further ensures that 9 deblurred images are sufficient for the identification of blur extent used for circle fitting.
is half of the angles between the rays passing through the edges of the object. Moreover, x and y are given by zx ˆ f
x=
and
y=
z yˆ f
(4)
from perspective projection, where f is the focal length of the camera. If x ˆl and x ˆr represent the left and right edge points of the object in the image, respectively, then θ and α are given by x ˆl tan θ = (5) f and tan(θ + 2α) =
x ˆr f
(6)
Thus, the 3-D coordinates of an object can be identified from the captured image if its actual size is available. z
IV. O BJECT S PEED M EASUREMENT
Q d 2
The proposed method for speed measurements is based on a pinhole camera model. 3-D coordinates and motion direction of the target object in the camera coordinate system are identified and then used for speed calculation.
P
φ
A. Camera Pose Estimation z
As shown in Fig. 5, suppose the center of a spherical object with diameter d is located at P = (x, y, z) in the camera coordinate system and the corresponding image point is p = (ˆ x, yˆ). Then the depth of the object can be derived as d cos(α + θ) (3) z= 2 sin α
θ
α
α Image Plane
p q f x
O
where θ is the angle between the optical axis and the ray passing through the left edge point of the object, and α
Fig. 5.
69
Camera pose estimation.
given by (7) have to be used to find the 3-D point P′ = (x′ , y ′ , z ′ ) since the depth z ′ cannot be recovered by Eq. (3). Suppose the angle φ is available, i.e.,
P′ d
P
tan φ =
z
p
u
z′ =
p′
β
f (z − x tan φ) , f − xˆ′ tan φ
x′ =
z′x ˆ′ f
and y ′ =
z ′ yˆ′ f (10)
where f is the focal length of the camera. If we consider a special case that the object is moving in a direction parallel to the image plane, i.e., φ = 0 in Figs. 5 and 6, then the displacement can be obtained from similar triangles. Thus, Eq. (8) can be simplified as
x f
O
v= Fig. 6.
(9)
then the point P is given by
y
φ
z′ − z x′ − x
Camera model for speed measurement.
zKsx Tf
(11)
where K is the blur extent in pixel and sx is the CCD pixel size of the camera. In this case, the position of the object in the image is not required for speed measurement. Similarly, the distance z can be derived as
To estimate the motion direction φ with respect to the image plane, two image captures are used to identify the locations of the object moving on the path. Suppose the 3-D coordinates of the objects are (x1 , y1 , z1 ) and (x2 , y2 , z2 ) from the two images, then the angle is given by z2 − z1 −1 φ = tan (7) x2 − x1
z=
fd W sx
(12)
where d and W are the diameters of the object measured physically and observed in the image, respectively. From Eqs. (11) and (12), the speed v can be written as
It should be noted that the estimation of the angle φ can be thought as a calibration stage and should be done prior to speed calculation. At this stage, the shutter speed of the camera should be set as fast as possible to avoid motion blur.
Kd (13) WT without the knowledge of the focal length and CCD pixel size of the camera. v=
V. E XPERIMENTAL R ESULTS
B. Formulation of Speed Measurement
The proposed speed measurement algorithms for spherical objects have been tested on real images from outdoor scenes with softball pitching. A system flowchart is depicted in Fig. 7. For the first set of experiments, the camera was placed approximately perpendicular to the motion direction of the softball. Eq. (13) is used for speed measurements with camera shutter speed T = 1/320 sec. and softball diameter d = 97.45 mm. For the images shown in Figs. 3 and 4, the blur extent and the size of the softball are identified as 26 and 72 pixels, respectively. Thus, the speed is found to be 40.54 km/hr. For the second set of experiments, the camera was installed such that the image plane is approximately 30 degrees off the moving direction of the softball. The same camera parameters are adopted, except that the shutter speed is first set as 1/800 seconds for motion direction estimation. To make the motion direction estimation more robust, instead of using Eq. (7), the angle φ is given by line fitting on the xz-plane projections of the 3-D coordinates obtained from five different images. The angle φ is found as 32.5 degrees in the experiment. For each speed measurement, Eqs. (3) - (10) are used to compute the speed of the softball with the CCD pixel size of 4.4 µm. Some results (16 out of 25 pitches) for both
In this work, we consider both a special case that the object is moving parallel to the image plane and the general case that the object is approaching or leaving the camera at a certain angle. Speed measurement is formulated as the object displacement in 3-D space divided by the camera exposure time. As illustrated in Fig. 6, suppose the initial and end positions of the object are P and P′ for a single image capture, then the speed v is given by d ||P′ − P|| = (8) T T where T is the shutter speed. Thus, if the motion blur is identified and denoted as a displacement vector u, and the center of the object in the deblurred image is denoted as p, then the corresponding image points of P and P′ are given by p and p′ = p + u, respectively. Suppose the identified motion direction is an angle β with respect to the image scanlines and blur extent is K pixels along the motion direction, then we have u = (Ksx cos β, Ksx sin β), where sx is the CCD pixel size of the camera. For a given image point p = (ˆ x, yˆ), the 3-D coordinates of P = (x, y, z) can be obtained from Eqs. (3) and (4). However, Eq. (4) and the constraint on the motion direction v=
70
TABLE I S OFTBALL SPEED MEASUREMENT ( IN KM / HR ) FROM MOTION BLURRED IMAGES (B1, B2) AND A VIDEO - BASED APPROACH (V1, V2).
No. B1 V1 B2 V2
1 40.5 40.9 38.9 37.1
2 34.0 37.2 34.7 39.1
3 36.9 38.9 34.8 36.5
4 33.1 34.2 52.8 48.1
5 40.1 40.4 42.2 35.4
6 36.4 38.2 50.3 44.3
7 41.7 41.0 49.2 51.5
8 45.5 45.3 46.6 46.6
13 42.1 39.5 54.8 50.2
14 41.0 43.0 40.3 37.0
15 39.3 43.5 51.6 46.3
16 42.1 46.2 42.8 41.0
R EFERENCES [1] S. Pumrin and D.J. Dailey, “Roadside camera motion detection for automated speed measurement,” in IEEE 5th International Conference on Intelligent Transportation Systems, 2002, pp. 147– 151. [2] A.L. Harvey and H.A. Cohen, “Vehicle speed measurement using an imaging method,” in International Conference on Industrial Electronics, Control and Instrumentation, 1991, pp. 1730–1733. [3] M.M. Sondhi, “Image restoration: The removal of spatially invariant degradations,” Proceedings of IEEE, vol. 60, no. 7, pp. 842–853, July 1972. [4] Y. Yitzhaky, I. Mor, A. Lantzman, and N.S. Kopeika, “Direct method for restoration of motion blurred images,” Journal of the Optical Society of America, vol. 15, no. 6, pp. 1512–1519, June 1998. [5] H.J. Trussell and S. Fogel, “Identification and restoration of spatially variant motion blurs in sequential images,” IEEE Trans. Signal Processing, vol. 1, no. 1, pp. 123–126, January 1992. [6] M.K. Ozkan, A.M. Tekalp, and M.I. Sezan, “Pocs-based restoration of space-varying blurred images,” IEEE Trans. Signal Processing, vol. 3, no. 4, pp. 450–454, July 1994. [7] S.K. Kang, Y.C. Choung, and J.K. Paik, “Segmentation-based image restoration for multiple moving objects with different motions,” in International Conference on Image Processing, 1999, pp. 376–380. [8] Lisa Gottesfeld Brown, “A survey of image registration techniques,” ACM Comput. Surv., vol. 24, no. 4, pp. 325–376, 1992. [9] R.C. Gonzalez and R.E. Woods, Digital Image Processing, 2nd Edition, Prentice Hall, 2001. [10] Z.Q. Wang, K.R. Rao, and J. Ben-Arie, “Optimal ramp edgedetection using expansion matching,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 18, no. 11, pp. 1092–1097, November 1996. [11] M. Petrou and J.V. Kittler, “Optimal edge detectors for ramp edges,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 13, no. 5, pp. 483–491, May 1991. [12] R.C. Jain, R. Kasturi, and B.G. Schunck, Machine Vision, McGrawHill, 1995. [13] H.Y. Lin and K.J. Li, “Motion blur removal and its application to vehicle speed detection,” in International Conference on Image Processing, 2004, pp. 3407–3410. [14] M.R. Banham and A.K. Katsaggelos, “Digital image restoration,” IEEE Signal Processing Magazine, vol. 14, no. 2, pp. 24–41, March 1997. [15] M. Cannon, “Blind deconvolution of spatially invariant image blurs with phase,” IEEE Trans. Acoustics, Speech, Signal Processing, vol. ASSP-24, no. 1, pp. 58–63, 1976. [16] T.C. Chen and K.L. Chung, “An efficient randomized algorithm for detecting circles,” Computer Vision and Image Understanding, vol. 83, no. 2, pp. 172–191, August 2001.
Object Speed Two Images
Target Identification Environment Parameter Estimation Image Segmentation Speed Measurement
no Image Rotation Circle Fitting
yes
Fig. 7.
12 50.2 49.1 53.4 34.0
The support of this work in part by the National Science Council of Taiwan under Grant NSC-93-2218-E-194-024 is gratefully acknowledged.
Most commonly used methods for object speed measurement adopt radar or laser based devices. They are active devices and usually more expensive compared to a passive camera system. In this paper, we have presented an automatic speed measurement method of spherical objects using a single image. Blur parameters of the motion blurred image are estimated and then used to calculate the speed of the object according to a pinhole camera model. Since our approach is an image-based method, it can be implemented with low cost, off-the-shelf digital cameras. Furthermore, the camera settings (such as shutter speed, image resolution, focal length) can be easily adjusted to improve the
Initial Blur Length Estimation
11 39.2 43.8 48.6 43.1
ACKNOWLEDGEMENTS
VI. C ONCLUSION
Horizontal Motion Blur
10 40.5 41.1 49.3 41.2
accuracy of the system. In contrast to the commercially available speed measurement devices, our approach has virtually no limitation on the speed test range. In the future work, more complicated camera model and different methods of blur parameter estimation will be considered to increase the robustness of the speed measurement results.
cases are shown in Table I. B1 and B2 represent the first and second set of the experiments, respectively. To find the relative error of our method compared to a video-based approach, a video camera was used to record an image sequence of each experiment. The speed is calculated by the distance traveled between two fixed locations divided by the time difference. The results are shown in Table I for both cases (V1 and V2). Most of the results from our method and the video-based approach have less than 2% difference for the first set of experiments (where the ball was moving parallel to the image plane), and less than 10% difference for the second set of experiments (where the ball was moving about 30 degrees off the image plane). Some results with large error for the second case could be caused by incorrect angle φ for the motion direction, since this angle might be different for each pitch.
Motion Blurred Image
9 45.5 44.2 42.5 47.0
Image Deblurring
System flowchart for speed measurement.
71
