2015 International Conference on Industrial Instrumentation and Control (ICIC) College of Engineering Pune, India. May 28-30, 2015
Robust Target Tracking Algorithm for MAV Navigation System S. Sankarasrinivasan∗ , E. Balasubramanian† , F. Y. Hsiao‡ , L. J. Yang§ ∗ † Center
‡§
for Autonomous System Research, Research Park, Vel Tech University, Chennai, India Email:
[email protected] Department of Mechanical and Electromechanical Engineering, Tamkang University, Tamsui, Taiwan
Abstract—Micro Aerial Vehicles (MAV’s) are becoming ubiquitous with its ever increasing applications in defense, space and environmental sectors. In real time scenario, MAV’s are expected to perform autonomously and development of intelligent algorithms meant for pattern recognition and object tracking are most demanding. This work concentrates on the development of vision based navigation system for real time target tracking using MAVs. The target is identified based on its color feature and various color models namely RGB, Normalized RGB, HSI, YUV, YIQ, YCbCr, CIELAB and CIELUV are considered for thresholding analysis. The idea is to frame an effective image processing algorithm concerning thresholding time and accuracy. In addition, the robustness of the color models for various noises such as fast fading, gaussian blur, jpeg, jp2k and white noise are also investigated. Simulation results suggests that, Y based color models exhibits less thresholding time, good accuracy and robust to noise. The target tracking algorithm is developed using optimum color model and justified through real time experimentation. A MATLAB (Matrix Laboratory) based navigation system is developed encompassing micro camera, A/V transmitter and receiver unit, flight controller, image processing system and other interfacing circuits. The navigation system is successfully tested in our lab environments and it is proven to be a realizable and a cost effective solution.
I.
I NTRODUCTION
A lot of research work have been carried out in the development of vision based system for MAV’s [1],[2],[3], [4] and [5]. In specific, exploration of image processing algorithms for real time tracking of objects using MAV’s are emerging very popular [6], [7],[8]. The real time target identification and tracking is a primary requirement for applications such as surveillance, reconnaissance and so on. This necessitates development of efficient image processing algorithm for on-line navigation. In the literature many strategies such as fuzzy logic,neural network, feature based and region based methods are suggested to achieve target identification and tracking. However, identifying the target through its color feature can be faster and easy to implement. Target detection based on color feature mainly depends on the ability of the color model in distinguishing color [9]. As each color model is unique in the way to represent a color, the selection of an optimum color model for a specific application and environment is still a challenging task. There has been lot of work carried out in optimizing the color model for medical applications([10]), skin classification([11]), automated detection algorithms [12] and etc. In this paper, color thresholding for target tracking is carried out on eight different color models. The main analysis 978-1-4799-7165-7/15/$31.00 ©2015 IEEE
269
TABLE I: Color Spaces derived from RGB Linear Transforms
YIQ, YUV, YCbCr
Non Linear Transforms
HSI, Normalised RGB, CIELAB, CIELUV
are on the ability of the color model to differentiate colors, its thresholding time and robustness to noises. The identified target is tracked under optimal color space for enhanced navigation. A MATLAB GUI(Graphical User Interface) is developed to achieve real time vision based navigation system. The organization of the paper is as follows. Section II give brief introduction to color spaces considered under this work and its thresholding parameters. Section III elaborates on target tracking algorithm. Section IV and V describes the real time experimentation and MATLAB navigation system implemented in our work. II.
C OLOR S PACE
Human visual system perceive colors as a mixture of tristimuli such as red(R), green(G) and blue(B) colors. Based on this fact, primary color model(RGB) is formulated. Other color models are derived by linear and non linear transformations of RGB space which is given in Table 1. RGB model is widely used in video monitors, picture acquisition and television systems. However, it is less preferred for image processing application because of its high sensitivity to illumination [13]. It occurs due to the high dependency among R,G,B components i.e. if the intensity changes, all the components will change. Hence, in general, Hue based color models are widely preferred among color segmentation algorithms. In this paper, several color models are exploited to examine their performance and optimizing it for real time navigation. The table II shows the considered color spaces and the parameters available for thresholding. III.
TARGET EXTRACTION AND TRACKING
The primary objective is to develop an image processing algorithm for fast detection and tracking of an object based on its color feature. As the attributes of the target object can be found only in real time, thresholding values cannot be computed in advance. The proposed algorithm provides for an user to select the target in real time and subsequently thresholding is performed based on the color information of the
TABLE II: Color space and its thresholding parameters Color Spaces
Thresholding Parameters
RGB
R,G,B
HSI, HSV
H
YCbCr
Cb,Cr
YIQ/YUV
I,Q/U,V
Nomalised rgb
r,g,b
CIELAB
A,B
CIELUV
U,V
(a) Orgininal Image
(d) Ignore Small Regions
selected object. The proposed algorithm automatically compute the thresholding parameters and its values. In order to perform effective color thresholding, the ability of the color space to extract a specific color of interest is a crucial factor. If the color of the object is more distinguishable in a particular color space, the extraction would be easier and accurate. This property can be mathematically expressed as the inner product between the histogram vectors of actual and the extracted object. [14].
(b) HSV Image
(c) Thresholded Image
(e) Erosion or Dilation
Fig. 1: Color Thresholding
(a) Test
(b) Target
(c) RGB
(d) HSV
(e) YCbCr
(f) YIQ
(g) YUV
(h) CIELAB
(i) CIELUV
(j) N-RGB
Let ’R’ be a random real time image and Rt be the identified portion from the real time image. The histogram vectors of the target and real time images are Ht and HR respectively. As color images are three dimensional, histogram vectors exist for all the components of an image. For example, in RGB, histogram vectors for R, G and B exist for both target and real time image.
Fig. 2: Target Identification and Extraction
PR = HRr .HT r
A. Optimum color space
PG = HRg .HT g
The main factors considered for selecting the optimum color model are thresholding time and accuracy. In addition, noise analysis are also carried out to ensure the robustness of the color space in real time.
PB = HRb .HT b where HRr , HRg , HRb correponds to histogram vectors of red, green and blue components of a real time image. HT r , HT g , HT b refers to histogram vectors of red, green and blue components of a target image. The minimum inner product signifies least match between the target and other visuals present in the same frame. The minimum among PR , PG and PB decides the optimal parameter for thresholding. In the similar way, the thresholding is carried out for all other color models. In addition, the thresholded image undergoes image processing techniques such as filtering, erosion or dilation to enhance the binary output [15]. For example, the Fig. 1 illustrates the color thresholding based on HSV space. Figure 2 shows a thresholded output in different color spaces for a synthetic test image. It can be seen that dark blue color is chosen as object of interest by the user (indicated by dotted black lines). Based on the target, the developed algorithm computes the optimal thresholding parameter and its value. The output obtained in different color models are shown in Fig 2c to 2j. The results validates the reliability of our method in extracting a target. The next issue is to find the optimal color model for fast and effective thresholding. 270
The experimental results are shown in box plots for better visualization in Fig. 3 and Fig. 4. Box plot is a graphical representation to show the distribution of output data points with respect to its median value. The 25th and 75th percentiles of the data set are represented by the horizontal edges of the box. The outliers are the values which are about twice the standard deviation from the data set mean. The median is indicated by the red horizontal line placed inside the box. In order to evaluate the performance of the algorithm, time and noise analysis is carried out. 1) Time Analysis: The overall thresholding time consist of conversion time from actual acquisition hardware color space to processing color space (tas ), thresholding parameter selection time(tp ) and thresholding time(th ). Total time = tas + tp + th a) Conversion time: RGB color based devices are used predominantly in many acquisition systems. In order to perform thresholding in other color spaces, conversion from actual RGB model have to be performed through the linear or non linear transformations. To study the significance of conversion time, the following experiment is performed. For
several random real time images (of size 320x240) conversion time from RGB to other color spaces has been done. From the tableIII, it can be observed that average conversion time for N-RGB and Y based color models are considerably less. (a) Target Identification
TABLE III: Avg. conversion time from RGB to other color spaces Color Space
Avg. Conversion Time
Norm RGB
0.0025
HSV
0.0455
YCbCr
0.0136
YIQ
0.0135
YUV
0.0133
CIELAB
0.1673
CIELUV
0.1213
(b) img1
(c) img2
(d) img3
(e) img4
(f) img5
(g) img6
(h) img7
(i) img8
(j) img9
(k) img9
b) Thresholding Time: In this experiment, thresholding is performed considering several random images from LIVE image database [16], real time pictures and few synthetic images. The test images are resized to 320 x 240 resolution for simplicity in execution. The average thresholding time (tp and th ) taken by each color model to identify a specific color is given in tableIV. From the table IV it is clear that, average thresholding time for RGB, N-RGB, YCbCr, YUV, YIQ are approximately 0.03 seconds, whereas for HSV and CIE color spaces it is 0.05 and 0.1 seconds respectively. Even though RGB and N-RGB takes less thresholding time, their accuracy is less compared to Y based color spaces for low illumination images. As our acquistion system is hardware modeled to deliver YCbCr information, it gains an additional advantage over YUV and YIQ. This made YCbCr as an optimal color space for real time implementation. 2) Noise Ananlysis: In vision based systems, navigation commands are issued by ground station based on the received real time image. In this process, noises may get introduced in several ways due to environment condition, acquisition hardware, transmission lines and processing. Addition of noises may result in either false detection or complete failure. In order to examine the color space robustness for various noise models, the following analysis are performed. The test images were taken from live image database version 2. It is an image database which consists of 29 reference images along with five distorted versions such as jpeg, jp2k, fast fading, gaussian blur and white noise. In further, each distorted version contains images with different level of degradation is available. For testing, the considered images are converted to different color models and then thresholding is performed. The purpose of this experiment is to investigate the influence of noises in deciding the thresholding efficiency. Mean Square Error (MSE) is considered as an analysis paramater and it measures the square of the difference between desired and actual system output[17]. It is calculated between the thresholded binary output from reference image and its distorted versions for different color models. The color model which corresponds to less average MSE are more robust towards noise. The results shown in table V concludes that the error is maximum for images containing white noise and less 271
(l) trakcing curve
Fig. 5: Real time experimentation
for gaussian blur noises. YIQ, YUV and Ycbcr color spaces exhibits lowest error rate among all. IV.
R EAL T IME E XPERIMENTATION
In order to examine the performance of the developed algorithm, real time experiments are conducted in our lab environment and the results are shown in Fig. 5. The image processing hardware system comprises of Intel Core i3-4030U CPU, 1.90Ghz processor with RAM of 4 GB. Initially the target is identified by the user (indicated by black rectangular box) as shown in Fig.5a and the motion of the target at successive time interval is shown in Fig. 5b to 5f. The gaussian noise of mean zero and standard deviation 0.01 is added to incoming image frames to validate the robustness of the algorithm. The thresholding is carried out in optimal color space and the corresponding output binary images are shown in Fig. 5g to 5k. Here, the objective is to find the centroid of the binary images and subsequently utilizing the same to track an object. The developed algorithms is implemented in real time and the Fig 5l shows the tracking curve (tracking points are taken from frame buffer domain) generated with an incorporation of noise and without noise. It can be inferred from the results that, addition of noises does not affect our tracking curve significantly, and thus our algorithm is more robust and adaptable for noisy environments. Similar experiments are conducted for different noises, and the test results indicates that proposed algorithm performs with an accuracy of more than 75 % and the processing time for each frame is approximately 0.05 seconds. The navigation signals are issued based on the location of object with respect to real time space. The real time
TABLE IV: Thresholding time for different color spaces Colors
Thresholding Time RGB
N-RGB
HSV
YCbCr
YUV
YIQ
CIELAB
CIELUV
Red
0.0311
0.0307
0.0461
0.0357
0.0361
0.0359
0.0926
0.0744
Light Blue
0.0306
0.0304
0.0459
0.0357
0.0361
0.0362
0.0878
0.0686
Brown
0.0318
0.0311
0.0478
0.0368
0.0371
0.0369
0.0898
0.0711
Green
0.0317
0.0310
0.0477
0.0364
0.0366
0.0367
0.0891
0.0717
Yellow
0.0310
0.0305
0.0477
0.0360
0.0364
0.0361
0.0916
0.0721
Pink
0.0312
0.0306
0.0491
0.0359
0.0364
0.0362
0.0924
0.0742
Blue
0.0315
0.0311
0.0477
0.0373
0.0370
0.0369
0.0915
0.0743
Orange
0.0337
0.0324
0.0472
0.0367
0.0366
0.0369
0.0860
0.0691
Fig. 3: Average thresholding time for different color spaces
TABLE V: Average Mean Square Error Distortion Types Jpeg Jp2k Fast Fading Guassian Blur White Noise
Average Mean Square Error YCbCr YUV YIQ 0.0298 0.0292 0.0200 0.0178 0.0177 0.0191
RGB 0.0293 0.0175
N-RGB 0.0348 0.0205
HSV 0.0458 0.0347
0.0476
0.0399
0.0448
0.0264
0.0262
0.0358
0.0378
0.0250
0.0128
0.0853
0.0827
0.0882
0.0846
setup for sending the guidance signal to MAV is elaborated in next section. V.
MATLAB NAVIGATION S YSTEM
Our navigation system comprises of micro camera and its transmitter and receiver module, flight control unit, MATLAB based image processing and navigation system, and Microcontroller interface circuitry. MATLAB supported GUI (Fig. 8) for target tracking is developed. GUI is interfaced to the real time camera and provides an option for the user to identify the target. The controls for real time tracking and navigation are incorporated in the GUI. In addition, by executing the functions of deploy tool, standalone executable *.exe package is made. It can be executed in any computers where MATLAB compiler runtime is installed. The complete navigation system 272
CIELAB 0.0569 0.0392
CIELUV 0.0277 0.0167
0.0250
0.0595
0.0236
0.0127
0.0125
0.0165
0.0112
0.0890
0.0880
0.1168
0.1153
block diagram and real time experimentation setup are shown in Fig. 6 and 7.
The real time images are sent as an input to the developed navigation system. The algorithm detects the centroid of the desired target and send the navigation commands accordingly. The navigation commands from the MATLAB GUI are sent to Microcontroller unit which acts as interface between MAV transmitter system and MATLAB platform. Throughout our real time experimentation, we faced lot of problems in attaining stabilised flight and control, this is due to adoptation of basic flight controllers for MAV. In addition, even though our algorithm performs well, we have been faced with undesirable image translation and rotation effects and that will be addressed in the future.
Fig. 4: Mean Square Error
Fig. 6: MAV Navigation System
VI.
C ONCLUSION
on, vol. 53, no. 6, pp. 1713–1725, Nov 2004.
A vision based navigation algorithm is developed to track an object of interest from the real time video. The color feature is used to identify the target and its thresholding is performed using eight different color models. Simulation and analysis is carried out to determine the optimal threshold space based on thresholding time, accuracy and robustness to various noises. It is concluded that, YCbCr,YIQ and YUV performances are significantly better than other color space models and suitable for MAV applications. The real time experimentation for the optimum color model confirms that, the successful rate is more than 75 % with processing time of 0.05 seconds for the image resolution of 320X240. A MATLAB GUI is designed in such a way that it provides an option for an end user in selecting the target, tracking and navigating MAV. The basic flight test results confirms that our navigation system is found to be consistent, robust and suitable for practical implementations.
[2] F.-Y. Hsiao, L.-J. Yang, S.-H. Lin, C.-L. Chen, and J.-F. Shen, “Autopilots for ultra lightweight robotic birds: Automatic altitude control and system integration of a sub-10 g weight flapping-wing micro air vehicle,” Control Systems, IEEE, vol. 32, no. 5, pp. 35–48, Oct 2012. [3] S. Todorovic and M. Nechyba, “Intelligent missions for mavs: visual contexts for control, tracking and recognition,” Robotics and Automation, 2004. Proceedings. ICRA ’04. 2004 IEEE International Conference on, vol. 2, pp. 1640–1645 Vol.2, April 2004. [4] S. Lin, F. Hsiao, C. Chen, and J. Shen, “Altitude control of flappingwing mav using vision-based navigation,” in American Control Conference (ACC), 2010, June 2010, pp. 21–26. [5] F. Hsiao, H. Hsu, C. Chen, L. Yang, and J. Shen, “Using stereo vision to acquire the flight information of flapping-wing mavs,” Journal of Applied Science and Engineering, vol. 15, no. 3, pp. 213–226, 2012. [6] A. L. Gilbert, M. K. Giles, G. M. Flachs, R. B. Rogers, and U. Y. Hsun, “A real-time video tracking system,” Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. PAMI-2, no. 1, pp. 47–56, Jan 1980.
R EFERENCES
[7] K. Schauwecker and A. Zell, “On-board dual-stereo-vision for autonomous quadrotor navigation,” in Unmanned Aircraft Systems (ICUAS), 2013 International Conference on, May 2013, pp. 333–342.
[1] S. Todorovic and M. Nechyba, “A vision system for intelligent mission profiles of micro air vehicles,” Vehicular Technology, IEEE Transactions
[8] M. Munderloh, H. Meuel, and J. Ostermann, “Mesh-based global motion compensation for robust mosaicking and detection of moving
273
[13]
[14]
[15]
[16] (a) MAV System
[17]
(b) Ground Navigation System
Fig. 7: Realtime Navigation Setup
Fig. 8: MATLAB Navigation GUI
[9]
[10]
[11]
[12]
objects in aerial surveillance,” in Computer Vision and Pattern Recognition Workshops (CVPRW), 2011 IEEE Computer Society Conference on, June 2011, pp. 1–6. J. Bruce, T. Balch, and M. Veloso, “Fast and inexpensive color image segmentation for interactive robots,” in Intelligent Robots and Systems, 2000. (IROS 2000). Proceedings. IEEE/RSJ International Conference on, vol. 3, 2000, pp. 2061–2066. M. Osman, M. Mashor, H. Jaafar, R. Raof, and N. Harun, “Performance comparison between rgb and hsi linear stretching for tuberculosis bacilli detection in ziehl-neelsen tissue slide images,” in Signal and Image Processing Applications (ICSIPA), 2009 IEEE International Conference on, Nov 2009, pp. 357–362. S. Phung, A. Bouzerdoum, and S. Chai, D., “Skin segmentation using color pixel classification: analysis and comparison,” Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 27, no. 1, pp. 148–154, Jan 2005. S. Ben Chaabane, M. Sayadi, F. Fnaiech, and E. Brassart, “Color image segmentation using automatic thresholding and the fuzzy c-means
274
techniques,” in Electrotechnical Conference, 2008. MELECON 2008. The 14th IEEE Mediterranean, May 2008, pp. 857–861. H.-D. Cheng, X. Jiang, Y. Sun, and J. Wang, “Color image segmentation: advances and prospects,” Pattern recognition, vol. 34, no. 12, pp. 2259–2281, 2001. Y. Chen, F. Hsiao, J. Shen, F. Hung, and S. Lin, “Application of matlab to the vision-based navigation of uavs,” in Control and Automation (ICCA), 2010 8th IEEE International Conference on, June 2010, pp. 877–882. R. C. Gonzalez, R. E. Woods, and S. L. Eddins, “Digital image processing using matlab,” Upper Saddle River, N. J: Pearson Prentice Hall, 2004. H. R. Sheikh, Z. Wang, L. Cormack, and A. C. Bovik, “Live image quality assessment database,” Release 2, 2003. A. Papoulis, Signal analysis. McGraw-Hill New York, 1978, vol. 191.
