The issues involved in tracking a ball in free- ight through an uncontrolled environment are ex- plored. ..... 5] Chaw-Bing Chang and John A. Tabaczynski.
PREDICTING BALL TRAJECTORIES IN UNCONTROLLED ENVIRONMENTS Robin Hosie
Geo West
Computer Science Department School of Computing Curtin University of Technology PO Box U1987 Perth, WA 6001 Australia
ABSTRACT The issues involved in tracking a ball in free- ight through an uncontrolled environment are explored. While previous work involving ball tracking has had success in speci c, structured domains, complexity is increased when more general cases are considered. A generalised behaviour model is developed to enable prediction, and a discrete Kalman lter is used to re ne the parameters of ight in accordance with information gained from each image. Extrapolating the determined trajectory is achieved by iterative prediction using the last known position and velocity. Results of the analysis of a sample image sequence are presented and discussed, and constraints for improved accuracy as well as future enhancements are suggested.
1
INTRODUCTION
The problem of tracking is one that has been explored in recent years, entailing examining a sequence of images to locate a target and predict its position throughout the sequence. Some areas of tracking address situations where elements of the target's behaviour are known a priori. For example, a ball in motion will follow a ight path that is determined by factors such as its mass, speed, and the angle from which it was initially projected. Computerisation of this task in structured, well-de ned situations has been achieved in previous research [1, 3, 2, 8, 7, 4]. However, in unstructured situations, where factors such as variable lighting, air pressure and wind speed come in to play, the task is more complex. Without prior knowledge of these factors, the method used to track the ball needs to adjust the parameters of behaviour as a result of information obtained from each new image in the sequence being analysed. To this end, we have explored the problem of tracking a ball in uncontrolled situations, and determining its ight path in order to predict uncaptured movement with a degree of con dence.
2
PREVIOUS RESEARCH
Current research applications that include the analysis of balls in ight include a robot ping-pong player [1] and a robot juggler [3, 2, 8, 7, 4]. Both of these systems run in real time under structured conditions, where the background is static and of a uniform colour, and the lighting is di use and reasonably uniform. These conditions enable the intensity of the ball to remain regular throughout the image sequence, contrasting well against the background to allow easy segmentation. In addition, both of these systems operate in a laboratory environment so external factors, such as wind and additional lighting, do not a ect the ight of the ball or the appearance of the image sequence. While these systems perform e ectively, their domains are quite limited and speci c. This research provides the basis for a more generalised ball tracking system that extends the exibility of other tracking methods by allowing situations where a greater number of unknown factors can a ect the
ight of the ball (such as ball features, and environmental and trajectory factors). To cater for these unknowns, we use a discrete Kalman lter [6] to adjust model parameters in light of information obtained from each new image in the sequence. The Kalman lter allows for a more general case, and will adjust its coecients to optimise its performance as a function of the statistics of the incoming image data [5]. This enables reasonably accurate results with a minimum of input or a priori knowledge.
3
GENERALISED BALL TRACKING SYSTEM Real World Locate Ball Image Capture Calculate Search Window
Kalman Filter
Image Processing
Update Stage Locate Ball in Initial Image(s)
Determine Velocity
Extrapolate
Initialise Kalman Filter
Boot Stage
Figure 1: Generalised ball tracking system. 3.1
Boot Stage
The generalised ball tracking system consists of several stages as depicted in Figure 1. The boot stage is necessary to obtain positional information from the rst few images in the sequence by locating the centroid of the ball using area moments. The initial velocity of the ball is then calculated by the di erence in ball position in the rst two images, and both position and velocity data are used to initialise the state parameter prediction x^?k in the Kalman Filter. An estimate of state parameter error is also calculated for Pk? . The measurement error Rk assumes a Gaussian distribution with a standard error equivalent to a one pixel displacement. k contains information of a model of a point
mass in ight:
= xk + x_ k t 2 = yk + y_k t ? t2 = x_ = y_k ? t
xk+1 yk+1 x_ k+1 y_k+1
(1)
where xk ,yk is the current position of the ball, xk+1 ,yk+1 is the next predicted position, x_ k and y_k are the current velocities in the x and y directions, x_ k+1 and y_k+1 are the next estimated velocities, and t is the time di erence between images. Hk is initialised to enable conversion of the state parameters to the same basis as the measurement data, which in this case was an identity matrix. Kk
Compute Kalman gain = Pk? HkT (Hk Pk? HkT + Rk )?1
* Project ahead
x^? k+1 = k x^k ? Pk+1 = k Pk Tk + Qk
HHH Y HHH
HHH
HHHj
Update estimate with measurement ? x^ = x^? k + Kk (zk ? Hk x^k
Compute updated error Pk = (I ? Kk Hk )Pk?
Figure 2: Kalman lter loop. 3.2
Update Stage
The update stage begins with the initial information gained from the boot stage, and iterates for every remaining image in the sequence. A window to bound a region in the image in which to search for the ball is determined, and the ball is located within this window in a similar manner to the boot stage. The ball position detected in the image is used in the Kalman lter to re ne the current estimate of ball parameters. Initially, when the model parameters are only rough estimates, the gain matrix ensures that the measurement data is highly in uential in estimating the state parameters. Then, as con dence in the accuracy of the parameters grows with each iteration, the gain matrix values decrease, causing the in uence of the measurement data in updating the parameters and associated error to lessen. 3.3
Extrapolation Stage
Once all the images in the sequence have been analysed, the extrapolation stage predicts the ball locations for subsequent time instances using the nal information on ball parameters and associated error produced in the update stage. The extrapolation is an process consisting of iterative predictions from the Kalman lter, initialised to the last values of the state parameter and error from the update stage.
Figure 3: Image at time 15
X Velocity
Y Position
Velocity (m/s)
3.5 3 2.5 2 1.5
2
4
6
8 10 12 14 16 18 20 22 24 26 28
Time Instance AAAA AAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAA AAAA AAAA AAAA Est. Upd. AAAAAAAAAAAA AAAAAAAAAAAALoc. AAAAAAAAAAAAAAAA AAAA AAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAA
2
4
6
8 10 12 14 16 18 20 22 24 26 28
Time Instance AAAA AAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAA AAAA AAAA AAAA Est. Upd. AAAAAAAAAAAA AAAAAAAAAAAALoc. AAAAAAAAAAAAAAAA AAAA AAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAA
Figure 5: Ball Position
9 8.5 8 7.5 7 6.5 6 5.5 5 4.5 4
Y Velocity
Velocity (m/s)
4 Position (metres)
Position (metres)
X Position 7.5 7 6.5 6 5.5 5 4.5 4 3.5 3 2.5 2 1.5 1 0.5 0
Figure 4: Flight of ball
2
4
6
8 10 12 14 16 18 20 22 24 26 28
Time Instance AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAA AAAA AAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAA Est. Loc. Upd.AAAA AAAA AAAA AAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAA
10 8 6 4 2 0 -2 -4 -6 -8 2
4
6
8 10 12 14 16 18 20 22 24 26 28
Time Instance AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAA AAAA AAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAA Est. Loc. Upd.AAAA AAAA AAAA AAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAA
Figure 6: Ball Velocity
For simplicity, only balls in free ight are considered (i.e., bounce is not catered for).
4 4.1
RESULTS Soccer Image Sequence
A Silicon Graphics Indy R4000 workstation, with a 1280 1024 resolution screen and 8-bit graphics board was used. Images were captured with standard video at 25 frames/sec, digitised to Silicon graphics greyscale format and smoothed to reduce noise. Di erence pictures were obtained and thresholded to produce the nal binary image, containing the ball as an area of white intensity. The image sequence captured a soccer ball being kicked gently, so the ight path formed an arc of approximately eight metres in the horizontal direction. The camera was placed orthogonal to the direction of movement, and the conditions were sunny, with little wind. As the ball moved relatively slowly compared to the capture rate, and was large compared to the image size, the image sequence was easily segmented and the ball was able to be located correctly in most images. Figure 5 shows that the ball moved at a relatively constant speed in the x direction, due to minimal wind interference. The y positions form a parabola that is roughly representative of the ight path the ball took. In this image sequence the velocity information was not updated through measurement in images processed after the boot stage. Consequently, although Figure 5 shows a reasonably constant x velocity (as the position changed in a linear fashion), Figure 6 shows that the estimated and updated velocity appears to increase before stabilising. This e ect was caused by the initial velocity estimate in the boot stage being calculated below the true velocity. Kalman ltering in the update stage allowed the velocity to be steadily increased until equilibrium was met about half-way through the
2 4 6 8 10 12 14 16 18 20 22 24 26 28 Time Instance AAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAA AAAA AAAA AAAA AAAAAAAAAAAA Est. AAAAAAAAAAAAUpd. AAAAAAAA AAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
2 4 6 8 10 12 14 16 18 20 22 24 26 28 Time Instance AAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAA AAAA AAAA AAAA AAAAAAAAAAAA Est. AAAAAAAAAAAAUpd. AAAAAAAA AAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
Figure 7: Position errors
2.8
7.8 7.6 7.4 7.2 7 6.8 6.6 6.4 6.2 6
Position (metres)
Position (metres)
0.3 0.25 0.2 0.15 0.1 0.05 0 -0.05 -0.1
Y Extrapolated Position
X Extrapolated Position
Y Position Error
Error (metres)
Error (metres)
X Position Error 0.2 0.15 0.1 0.05 0 -0.05 -0.1 -0.15 -0.2
2.5 2.2 1.9 1.6
25
26
27
28
29
Time Instance AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAA AAAA AAAA AAAA AAAA H E L LocAAAA AAAA AAAA AAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAA
25
26
27
28
29
Time Instance H
Ext.
L
Loc
Figure 8: Extrapolated position
image sequence. The slightly declining velocity after that can be attributed to the e ect of air drag on the ball (which was not considered in this research). The velocity in the y direction was also initialised too low (based on the location estimates from the rst two images), but processing via the Kalman lter allowed it to increase quite quickly to re ect the true velocity. The steady decline after that was due to the e ects of gravity on the ball. The velocity calculated from the images is shown as Loc. in Figure 6. Anomalies in the apparent velocity, particularly at time instances 12, 18 and 26 for x, were due to calculating from images where the ball's centroid was located incorrectly. The velocity for y is a ected similarly for instances 16, 22, and 26. The di erences between the estimated and updated positions from the position located in the image are given in Figure 7. It can be seen that the x error was initially high due to incorrect initialisation, but quickly adjusted to a reasonable level. The steady decline shown was caused by the prediction model overestimating the ball position by not considering the air drag e ect. The error in the y direction converged quite quickly to be around zero, but was also a ected by spurious ball location data. The deviations at time instance 12, 18 and 26 (x) and 16, 22, and 26 (y ) were a result of a misinterpretation of the centroid of the ball, due to a less than optimal segmentation of these images. As they appear further on in the image sequence, the gain matrix had reduced allowing only a small portion of the error to be used to update the prediction. This meant that the e ect of this anomalous image data was minimised for subsequent predictions. Figure 8 shows the position of the ball extrapolated after time instance 25 for four more time instances. The x and y positions were predicted to a reasonable accuracy, however it appeared the predictions slowly deviated from the located positions. The cause of this was attributed to the predictor model not catering for air drag on the ball. The anomalous located position at image 26 was removed to highlight the steady increase in error.
5
CONCLUSION
The research presented here has explored the area of tracking an unknown ball moving in free ight in an uncontrolled environment by using a generalised Kalman lter. Previous applications work in controlled environments to provide good image segmentation, and rely on knowledge of the speci c
situation to achieve accurate tracking of a ball in ight. In this research, exibility is increased by considering outdoor scenes and di erent ball speeds and sizes, however accuracy can diminish as a result of inconsistent image segmentation due to light e ects or background interference. In addition, factors such as wind and spin can a ect the ight of the ball in the real world and should be considered if adequate prediction is to be achieved. Prior knowledge of these factors could be incorporated in the tracking model so that it suits the particular situation, but this would reduce the ability to track the general case. To maintain exibility and allow for automatic adjustments to each new situation, anomalies in the expected ight path must be detected in each image and used to update the parameters of the predictive model. The system developed provides a basis for a general tracking system, by allowing a degree of exibility in the camera angle, the environment, and ball speeds and sizes, and is able to track a ball within an area of approximation in most cases, giving an indication of the con dence of prediction. However, increased accuracy can be gained by using improved segmentation techniques, and incorporating knowledge of the ball's attributes to cater for air drag.
References [1] Russell L. Andersson. A Robot Ping-Pong Player: Experiment in Real-Time Intelligent Control. MIT Press, Cambridge, Mass, 1988. [2] M. Buhler, D. E. Koditschek, and P. J. Kindlmann. A family of robot control strategies for intermittent dynamical environments. IEEE Control Systems Magazine, pages 16{22, Feb 1990. [3] M. Buhler, D. E. Koditschek, and P. J. Kindlmann. Planning and control of robotic juggling tasks. In Hirofumi Miura and Suguru Arimoto, editors, Robotics Research: The Fifth International Symposium, pages 321{332. MIT Press, Cambridge, Mass., 1990. [4] M. Buhler, D.E. Koditschek, and P.J. Kindlmann. Planning and control of robotic juggling and catching tasks. International Journal of Robotics Research, 13(2):101{118, April 1994. [5] Chaw-Bing Chang and John A. Tabaczynski. Application of state estimation to target tracking. IEEE Transactions on Automatic Control, 29(2):98{109, February 1984. [6] R. E. Kalman. A new approach to linear ltering and prediction problems. Transactions of the ASME { Journal of Basic Engineering, pages 35{45, March 1960. [7] A. A. Rizzi and D. E. Koditschek. Progress in spatial robot juggling. In Proceedings of the 1992 IEEE International Conference of Robotics and Automation, pages 775{780, May 1992. [8] Alfred A. Rizzi, Louis L. Whitcomb, and Daniel E. Koditschek. Distributed real-time control of a spatial robot juggler. Computer, pages 12{24, May 1992.
