Proceeding of the IEEE International Conference on Robotics and Biomimetics (ROBIO) Shenzhen, China, December 2013
Object tracking based on the improved particle filter method using on the bionic eye PTZ Jun Luo, Juqi Hu, Hengyu Li, Hengli Liu, Hao Wang, Shaorong Xie*, Jason Gu Abstract—Our bionic eye PTZ requires the tracking target at the central field of view of the camera, which means it is so important to realize the target tracking well in the first step. The particle filter method is famous for its robust tracking performance in cluttered environments. However, most methods are in the mode of moving object and stationary camera and they are not utilizing so well on the bionic eye PTZ since the camera in our project needs real-time motion. In this paper, we proposed an improved particle filter based on the SKL (Symmetric Kullback-Leibler divergence) similarity measure to realize object tracking and a closed-loop control model based on speed regulation to keep the target at the centre of the camera. The experiment results show that our system can track the moving object well and can always keep the object in the middle of the field of the view. I.
A
INTRODUCTION
s one of the core research topics in computer vision, target tracking is generally based on processing the image sequence, trying to identify targets from a complex background and to predict the movement of the targets, and to achieve a continuous, accurate tracking. This emerging technology combines image processing, pattern recognition, artificial intelligence, automatic control, and many other areas of advanced computer technology. More specifically, the so-called tracking includes the detection, extraction, identification and tracking of moving targets in the image sequence, thus obtaining motion parameters, such as position, velocity, acceleration, and trajectory to achieve understanding of the behavior of the moving objects. Numerous approaches have been proposed to track moving objects in complex backgrounds. Among the tracking methods, the most commonly used can be divided into two categories: one is deterministic tracking method, such as
Manuscript received Sep 12, 2013. This project is supported by the Key Projects of the National Natural Science Foundation of China (Grant No. 61233010), Shanghai Municipal Science and Technology Commission Project (No.12111101200, 12140500400). Jun Luo, Shaorong Xie are with the Mechatronic Engineering department, Shanghai University, Shanghai, China. (e-mail:
[email protected],
[email protected]). Juqi Hu is with the Mechatronic Engineering department, Shanghai University, China (e-mail:
[email protected]). Hengyu Li is with the Mechatronic Engineering department for his post-doctoral degree, Shanghai University, Shanghai, China (e-mail:
[email protected]). Hengli Liu is with the Mechatronic Engineering department for his doctoral degree, Shanghai University, Shanghai, China (e-mail:
[email protected]). Hao Wang is with the Mechatronic Engineering department, Shanghai University, Shanghai, China (e-mail:
[email protected]). Jason Gu is a professor with the Department of Electrical & Computer Engineering, Dalhousie University, Halifax, NSB3J2X4 Canada. (e-mail:
[email protected]).
978-1-4799-2744-9/13/$31.00 ©2013 IEEE
Mean Shift, Camshift and Kanade-Lucas-Tomasi tracking method; the other type is random tracking method, such as Kalman filter, extended Kalman filter [1] and particle filter. The industry on the particle filter tracking algorithm for visual tracking [2-3] keeps growing since the particle filter does not require the system to be linear or the noise to be Gaussian and it has good robustness, anti-blocking, interference resistance [4-5]. The basic idea of the particle filter is that the posterior density is approximated by a set of discrete samples (called particles) with associated weights. For target tracking of a fixed installation of the camera, the conventional tracking methods such as background extraction, time difference could achieve good tracking results due to the limited changes of the background. However, for the bionic eye PTZ in this project, we need the bionic eye PTZ to obtain stable dynamic image information even in the rough environment. Therefore, the camera vibration as well as the interference of the similar colors in the background has greatly increased the difficulty of the tracking. This paper aims to propose an improved particle filter method to obtain better color-coded in order to achieve more accurate target tracking of bionic eye PTZ. II. PARTICLE FILTER BASED ON THE SKL A. Particle Filter Particle Filter is a Monte Carlo simulation [6-7] based on recursive Bayesian estimation and approximate numerical solution method [8] that can effectively deal with non-linear, non-Gaussian filtering system problem and it is a global optimal filtering method. Its core idea is to use a weighted sum of a series of random samples to approximate the state of the entire posterior probability density which can be expressed as follows: N
p ( x0:k | z1:k ) ≈ ∑wki δ ( x0:k − x0:i k ) i =1
(1)
Where { xki } is also known as the “particles”, is a series of samples that are obtained by the Monte Carlo random sampling at the moment k, {wki } is the normalized weights of the particles, N is the number of particles and δ ( ⋅) is Dirac-Delta function. So we can use the particle set and its weight { xki , wki }iN=1 to approximately describe the system posterior probability density p ( x0:k | z1:k ) according to (1). The principle is as shown in the figure 1. [9]
2213
B. Spatiogram Color histogram is one of a common target models which is just a statistics of different colors in the entire picture in proportion without concern for spatial location of each color. Therefore, it is not rather sensitive to rotation but suitable for non-rigid or prone to deformation modeling target objects. Targets based on this model are vulnerable to backgrounds which have similar color distribution or other interference, thereby causing the target tracking failure. In this paper, we let the particle filter algorithm based on a new target model-spatiogram which adds the pixel coordinate information to the traditional color histogram [11-13]. The spatiogram is proposed by Birchfield and Rangarajan in 2005 [14]. The second-order spatiogram can be described as the followings:
p ( x0:k | z1:k )
Fig. 1.
The principles of particle filter
h ( b ) = {nb , ub , Σb } , b = 1,..., B
(7)
Where B is the total number of the intervals, nb , ub , Σb is the probability of each interval, coordinate mean and covariance matrix, respectively. They can be calculated using the formula as follows: Fig. 2.
The BD and SKL distance under different t
According to the importance sampling, importance density q ( x0:k | z1:k ) can be described by [10]: N
q ( x0:k | z1:k ) ≈ ∑ δ ( x0:k − x0:i k ) i =1
{w } i k
w ∝w
p ( zk | xki ) p ( xki | xki −1 ) q ( xki | xki −1 , zk )
( 3)
It is usually chosen a priori probability density which is easy to be implemented as the importance density function:
q ( xki | xki −1, zk ) = p ( xki | xki −1 )
( 4) (5)
while δ jb = 0 indicates the j-th pixel is quantized to other
C. Particle Filter based on SKL In order to apply the spatiogram to target tracking, we need to select a method to measure the similarity metrics of the spatial histogram between the targets and the candidate targets. We select the SKL-based coefficient of similarity metrics to measure the similarity of the target spatiogram h( b) = {nb , ub , Σb} and candidate target spatiogram
h' ( b) = {nb' ,ub' , Σb'} .
N
i =1
Given a spatiogram h ( b ) = {nb , ub , Σb } , b = 1,..., B , we use
a Gaussian distribution to describe the spatial distribution of all the pixels in each section. The distribution of the b-th section can be described as:
If the normalized weights get further processed, the posterior probability density can be expressed as: p ( xk | z1:k ) ≈ ∑ w ki δ ( xk − xki )
(8)
area, x j = ⎡⎣ x j , y j ⎤⎦ is the coordinate position of the j-th pixel, δ jb = 1 denotes the j-th pixel is quantized to the b-th interval,
Substituting equation (4) into the equation (3), we can get: the posterior probability density can be expressed as:
wki ∝ wki −1 p ( zk | xki )
1 ΣNj=1 (x j − ub )(x j − ub )T δ jb N * nb −1
intervals.
according to Bayesian estimation methods as follows: i k −1
Σb =
T
density function depends only on xk −1 and zk , therefore we
i k
1 N 1 Σ j=1δ jb , ub = ΣNj=1x jδ jb , N N * nb
B is the total number of pixels within the target
( 2)
If q ( xk | x0:k −1 , z1:k ) = q ( xk | xk −1 , zk ) , then the importance can derive the updating formula of the weights
nb =
( 6)
When the number of particles N → ∞ , the equation (6) can be guaranteed by the Law of Large Numbers to be approximate to the real posterior probability p( xk | z1:k ) .
fb ( x ) =
1 2π Σb
1/2
exp[ −
1 T ( x − ub ) Σ b−1 ( x − ub ) ] ( 9 ) 2
Where u b is the mean value of all coordinates of the pixels of the b-th interval Σ b is the mean covariance matrix of all coordinates of the pixels of the b-th interval.
2214
The KL distance between the Gaussian distribution fb ( x )
BD ( f , g ) =
and the Gaussian distribution fb ' ( x ) can be obtained by a closed form solution which is calculated using the following formula: ⎛ Σ 'b ⎞ −1 1 ⎟ + Tr ( Σ 'b ) Σ b − d KL ( f b & fb ' ) = [ log ⎜ ⎜ Σb ⎟ 2 (10 ) ⎝ ⎠
)
(
+ ( ub − ub ' ) ( Σ 'b ) T
−1
(u
b
− ub ' ) ]
Where d is the spatial dimension (for spatiogram, d is equal to two). Similarly, we can get the KL distance between the Gaussian distribution fb ' ( x ) and the Gaussian ⎞ ⎟ + Tr ( Σ b−1Σ b' ) − d ⎟ ⎠
SKL ( f , g ) =
(11)
+ ( u b ' − u b ) Σ b−1 ( u b ' − u b ) ]
Then, the SKL distance of the two Gaussian distribution of fb ( x ) and fb ' ( x ) is: (12)
Substituting the equation (10) and (11) into the equation (12) and simplifying, we can get: −1 1 SKL ( fb , fb' ) = [Tr ( Σ 'b ) Σ b + Tr ( Σ b−1Σ 'b ) − 2d ] 4 (13) 1 ' T −1 ' −1 ' + [ ( ub − ub ) Σ b + ( Σ b ) ( ub − ub ) ] 4
(
(
)
1 2 −2 (t + t − 2) 4
(18 )
where t = σ 1 / σ 2 . Fig. 2 shows the changes of the two different distance measurement, it is clear that the change ratio of the SKL distance is much quicker than that of the BD distance, and thus the distinction capacity of the similarity measure based on the SKL distance is much stronger. III. BIONIC EYE PTZ SERVO CONTROL MODEL
T
1 SKL ( f b , f b ' ) = [KL (f b & f b ' )+KL(f b ' & f b ] 2
(17 )
While
distribution fb ( x ) : ⎛ Σb 1 KL ( f b ' & f b ) = [ log ⎜ ' ⎜ Σb 2 ⎝
1 ⎛ 1+ t2 ⎞ ln ⎜ ⎟ 2 ⎝ 2t ⎠
)
Generally, the ranges of the similarity are [ 0,1] ， the
Bionic eye PTZ is actually a spherical parallel mechanism that consists of the upper platform (eye), the base and the branched-chain which is made up of the three pairs of connecting rods. The eye can rotate to achieve three degrees of freedom with respect to the base co-driven by three motors. The bionic eye PTZ used in our system is shown in Fig. 3, featuring a high resolution of effective pixels. As we want to realize the object tracking, so the camera should follow closely to the movement of the target, when the target moves fast, the camera should response quickly and accurately in order to keep the target in the field of view (FOV). In order to keep the target in the middle of the FOV, the coordinates of the image centre and size (xc, yc, zc ) become the input of the control model, and the feedback of the control model is the location coordinates ( x, y, z ) of the moving
similarity ψ b of each pair of intervals on the spatiogram can be described as: ψ b = exp ⎡⎣ − SKL ( fb , fb ' ) ⎤⎦ (14) Thus, the similarity of the spatiogram based on SKL distance can be calculated as: B
ρ ( h, h' ) = ∑ nb nb 'ψ b exp[− SKL ( fb , fb' )] b =1
(15 )
According to equation (12), we can get: 1 4
ρ ( h, h ' ) = Σ bB=1 nb nb' ψ b exp[ − (Tr (Σ b−1Σ 'b ) +Tr (Σ b−1Σ 'b ) − 2 d + 0)]
(16 )
Fig. 3. The bionic eye PTZ
1 = Σ bB=1 nb exp[- (d + d - 2 d )] = Σ bB=1 nb = 1 4
target. The error signal e = (ex , e y , ez ) is the input of the
This indicates that this section of the KL distance based on symmetric spaces histogram similarity measure has ensured the most similar to the target itself. If the two histogram distribution are f , g , especially when f , g are one-dimensional Gaussian distribution, f = N ( x; u1 , δ 12 ) ,
g = N ( x ; u 2 , δ 22 ) , then we can calculate their SKL distance
( if u1 = u2 )according to the equation (13):
Fuzzy PID control. So the vector (ω x , ω y , ω z ) which is the rotation angles of the three motors becomes the output of the control model. And the vector (ω x , ω y , ω z ) is decided by the following function (19 ) , where K is the proportional gain decided by a complex transformation which is decided by the Fuzzy PID control model and F is the local length of the camera.
2215
⎡ω x ⎤ ⎡ ex ⎤ ⎡ x − xc ⎤ ⎢ ⎥ K⎢ ⎥ K⎢ ⎥ ⎢ω y ⎥ = F ⎢ ey ⎥ = F ⎢ y − yc ⎥ ⎢ω ⎥ ⎢e ⎥ ⎢⎣ z − zc ⎥⎦ ⎣ z⎦ ⎣ z⎦
distinguishing the target in complex background. The results demonstrate that the improved algorithm has achieved a better real-time, continuous and reliable tracking. In order to make comparison, we have also used the Camshift algorithm (including simple prediction) and the
(19 )
The camera servo control model is given as Fig.4. IV. EXPERIMENTAL RESULTS The particle filter method based on the SKL distance has been described in detail previously, and in this section we will use the algorithm to do some experiment to demonstrate the effectiveness and robustness of the proposed tracking scheme.
Fig. 4.
Bionic eye PTZ camera servo control block diagram
Our algorithm is developed in Microsoft Visual Studio 2010 and runs on a 2.13 GHz Pentium Dual-Core CPU, 2Gbyte
Fig. 6.
Fig. 7.
Fig. 5. Tracking results of a rectangular block using the improved particle filter
DDR memory, using a video image size of 658× 494 . The camera is GUPPY F-033C from AVT. The image frame rate is 30 frames per second and the number of particles is 64.
A. Tracking for the rectangle block In this experiment, the tracking results of rectangular block using the improved particle filter method are shown in Fig.5. As we can see from the figure, the improved method based on the similarity measurement of SKL distance is effective at
Tracking results of a rectangular block using the Camshift algorithm (including simple prediction)
Tracking results of a rectangular block using the general particle filter
general particle filter algorithm to track the moving rectangle block, respectively. As seen in Fig.6 and Fig.7, the results show that the two algorithms (including simple prediction) failed when the rectangular target reach into the background of similar color, and the tracking performance is unstable when the rectangular target is moving in the area where there are similar colors in the background. This is due to the fact that the traditional algorithm is based on Bhattacharyya coefficient which can ensure the searching area mostly similar to the targets. However, Bhattacharyya coefficient is
2216
not so distinctive about similar colors [15], thereby reducing the stability and accuracy of target tracking.
B. Tracking for the circular model (tennis ball) The second experiment used the red tennis ball as the tracking object, as seen in Fig 8. From the experiment results, one can tell that our improved particle filter algorithm is also effective in tracking a circular object and the process was accurate not only in position, but also in the object size. In order to evaluate the effectiveness of the proposed algorithm, pixel deviation has been used to analyses the experiment results. Pixel deviation is the pixel distance
between the position of the object and the centre of the image and is used to evaluate the tracking position accuracy. The results are as shown in Fig 9. The Fig.9 shows the average distance of pixel between the real positions of the target and the image centre. You can clearly see that the new algorithm has achieved a good tracking function of the bionic eye PTZ. In order to get a comprehensive understanding of the robust tracking performance of our improved particle filter algorithm, we provide the pitch and roll angles of both the robot and bionic eye during our experiment. The results are as shown in Fig.10.
Fig.10. Roll & Pitch angle of both the robot and the bionic eye while tracking targets
V. CONCLUSION
Fig. 8.
Tracking results of a tennis ball using the improved particle filter
This paper presents a novel particle filter algorithm based on distance SKL which has a stronger distinction and better tracking stability compared to the Bhattacharyya Distance. The algorithm proposed in this paper is particularly suitable for working in a complex background under bionic eye PTZ environmental requirements. We discussed the principle theory of particle filter, spatiogram, similarity measurement based on SKL, and the detailed information of the proposed servo control model. The experimental results demonstrate that our algorithm is effective and robust in dealing with moving objects, and can always keep the target at the centre of the camera. REFERENCES [1] [2]
[3]
Fig.9.
X&Y pixel deviation of the tennis ball
[4]
2217
Y. Bar-Shalom, X. R. Li and T. Kirubarajan. Estimation with Applications to Tracking and Navigation: Theory, Algorithm, and Software. New York, Wiley, 2001. Y. Li, H. Ai, T. Yamashita, S. Lao, M. Kawade, “Tracking in low frame rate video: A cascade particle filter with discriminative observers of different life spans,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 30, no. 10, pp. 1728-1740, 2008. K. Nummiaro, E. Koller-Meier, and L.V. Gool, “An adaptive color-based particle filter,” Image and Vision Computing, vol. 21, no.1, pp. 99-110, 2002. M. S. Arulampalam, S. Maskell, N. Gordon and T. Clapp, “A tutorial on particle filters for on-line non-linear/non-Gaussian Bayesian
tracking,” IEEE Transactions Signal Processing, vol. 50, no. 2, pp. 174-188, 2002. [5] Z. H. Khan, I.Y.H. Gu, and A.G. Backhouse, “A Robust Particle Filter-Based Method for Tracking Single Visual Object Through Complex Scenes Using Dynamical Object Shape and Appearance Similarity,” Journal of Signal Processing Systems, vol. 65, no. 1, pp. 63-79, 2011. [6] J. C. Spall. Estimation via Markov chain Monte Carlo. Proceedings of the 2002 American Control Conference, 2002, 4: 2559-2564. [7] J. C. Spall. Estimation via Markov chain Monte Carlo. IEEE Control Systems Magazine, 2003, 23(2): 34-35. [8] J. S Liu, R. Chen, “Sequential Monte Carlo methods for dynamic systems,” J. American Statistical Association, 1998, vol. 93 pp: 1032-1044. [9] Z. Chen. Bayesian Filtering: From Kalman Filters to Particle Filters, and Beyond. Hamilton: McMaser University, 2003. [10] N. Gordon, D. J. Salmond and A. F. M. Smith. Novel Approach to Nonlinear/Non-Gaussian Bayesian State Estimation. IEEE Proceedings-Radar, Sonar & Navigation. 1993, 140(2): 107~113. [11] S. T. Birchfield and S. Rangarajan, “Spatiograms versus histograms for region-based tracking,” in Proc. CVPR, June 2005, pp. pp. 1158-1163. [12] S. T. Birchfield and S. Rangarajan, “Spatial histograms for region-based tracking,” in ETRI Journal, 2007. vol. 29(5):, pp. 697-699. [13] A. F. Smeaton C. O’Conaire, N.E. O’Connor, “An improved spatiogram similarity measure for robust object localization,” in Proc. ICASSP, 2007. [14] Birchfield S.T., Rangarajan S. Spatiograms versus histograms for region-based tracking. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. San Diego: IEEE, 2005: 1158-1163. [15] Wang H., Suter D., Schindler K., et al. Adaptive object tracking based on an effective appearance filter. IEEE transactions on Pattern Analysis and Machine Intelligence, 2007, 29(9): 1661-1667.
2218
