Multisensor-Multitarget Tracking Testbed

Proceedings of the 2009 IEEE Symposium on Computational Intelligence in Security and Defense Applications (CISDA 2009)

Multisensor-Multitarget Tracking Testbed David Akselrod, Ratnasigham Tharmarasa, T. Kirubarajan, Zhen Ding, and Tony Ponsford Abstract— In this paper we present a multisensor-multitarget tracking testbed for large-scale distributed scenarios. The objective is to develop a testbed capable of handling multiple, heterogeneous sensors in a hierarchical architecture for maritime surveillance. The testbed consists of a scenario generator that can generate simulated data from multiple sensors including radar, sonar, IR and ESM as well as a tracker framework into which different tracking algorithms can be integrated. In the current stage of the project, the IMM/Assignment tracker, and the Particle Filter (PF) tracker are implemented in a distributed architecture and some preliminary results are obtained. Other trackers like the Multiple Hypothesis Tracker (MHT) are also planned for the future. Index Terms— Testbed, tracking, distributed, fusion.

I. I NTRODUCTION HE objective of the presented work is to develop a stateof-the-art distributed estimation, tracking and fusion system testbed with advanced algorithms for multisensormultitarget surveillance of air, littoral and sea targets. One crucial aspect of defense capability is the surveillance of mobile targets, in the air, ground or sea, within and near the borders. A large number of sensors, for example, radar, ESM or imaging, are used to gather information about the targets of interest. The gathered information or sensor reports are processed in one or more fusion centers by combining the data from multiple sensors. Typically, a realistic surveillance system is both hierarchical and distributed (or networkcentric); that is, information fusion is carried out in a number of fusion centers and at different fusion levels (e.g., raw data, state estimates or decisions). In this work we propose to develop algorithms that solve some specific problems in a network-centric surveillance scenario. The proposed testbed will assist in developing 1) different architectures for distributed surveillance with heterogeneous sensors and ways to quantify their performances. 2) algorithms for preprocessing (e.g., registration, debiasing, out-of-sequence measurements, etc.) heterogeneous multisensor data. 3) estimators of multitarget states using distributed, heterogeneous sensor data. 4) fusion algorithms for combining raw sensor data, state estimates and target features in distributed and hierarchical architectures. 5) resource management algorithms in heterogeneous, multisensor scenarios.

T

David Akselrod, Ratnasigham Tharmarasa, and T. Kirubarajan are with the Department of Electrical and Computer Engineering, McMaster University, ON, Canada (email: {akselrd, kiruba}@mcmaster.ca); Zhen Ding is with Radar Systems Section, DRDC Ottawa, ON, Canada (email: [email protected]); A. M. (Tony) Ponsford is with Raytheon Canada Ltd., Waterloo, ON, Canada (email: tony [email protected])

978-1-4244-3764-1/09/$25.00 ©2009 IEEE

6) a simulation environment to implement and evaluate the new algorithms developed in this project. The proposed work enables a better utilization of existing sensor resources to extract the maximum information available in multisensor data about the targets of interest. Different aspects of multisensor-multitarget tracking, the process of estimating the states of moving objects using the data from multiple sensors like, for example, radar, ESM or optical, has been handled by a number of researchers (see [1], [2], [3] for theory and applications). Typical applications of multisensor-multitarget tracking are in ground target tracking [8], air traffic control (ATC) [12], coastal monitoring [11], space surveillance [1], radar tracking in the presence of electronic countermeasures [4], [7], image-based tracking of tissue cells [9] and visual tracking [10], to name a few. Most practical systems consist of a number of heterogeneous and asynchronous sensors (i.e., different types of sensors resulting in measurements at different times) that may be geographically distributed. Then the task is to combine or fuse the noisy information from these different sensors and find the best possible state estimates of the targets of interest that are mobile. Multisensor tracking and fusion systems can be broadly categorized as centralized and decentralized (or distributed). In the centralized architecture, all raw data from individual sensors are sent to a single fusion center where they are statistically combined to obtain a single set of track estimates. In distributed scenarios multiple airborne or shipmounted sensors, the centralized architecture may not be practicable [1]. In contrast, in the decentralized architecture, each sensor processes its own data and sends only the resulting estimates (and its covariances or uncertainties) to the fusion center. The fusion center carries out a track-totrack association to combine the estimates from different sensors. In some cases, the fusion center may send its fused estimates back to the individual sensors, in which case one has a centralized fusion system with feedback [1]. While the decentralized architecture requires less communication bandwidth, its performance is sub-optimal to that of the centralized architecture even if the track-to-track fusion is done optimally by accounting for all possible errors [5], [6] (results are available only for the case of homogeneous sensors). Simulation studies using homogeneous sensors have validated these results [6]. In a networked environment, as in many practical systems today, the architecture is hierarchical; that is, fusion may be carried out in more than one fusion center and in more than one level (e.g., raw data, estimates and decision) - the basic fusion architectures (centralized and decentralized) are just the building blocks in the overall fusion topology. Distributed sensor networks may require

Fig. 2. Fig. 1.

Scenario View mode

Testbed block diagram

modification to the standard state estimation algorithms to handle the underlying fusion architectures. In addition, the hierarchical nature of a sensor/fusion system has to be taken into account. Another important application of the presented testbed, especially for multisensor-multitarget tracking, is the evaluation of various algorithms on realistic scenarios. In [4] a realistic benchmark system was developed for evaluating different algorithms for single target tracking and single (multifunction) sensor resource management. It was later extended to two sensors and two targets [13]. In [6] a homogeneous multisensor-multitarget benchmark problem was used to evaluate tracking and fusion performances without sensor resource management. A realistic simulation or prototype environment for heterogeneous multisensor-mulitarget tracking and sensor resource management performance evaluation is highly desirable. II. S YSTEM D ESCRIPTION Being a distributed Data Fusion System (DFS) emulator, the testbed is composed of one or more of Data Fusion Centers (DFC) which may reside on distinct computer systems, together forming a common emulating system interconnected via a network. A DFC thus presents one of the main elements which compose the emulated system and interconnection which define its final architecture. Fig. 1 shows the block diagram of the testbed as well as interaction between its main modules. A DFC consists of the four main operational modules: Simulator, Tracker, Resource Manager, Tracking Performance Evaluator, and three auxiliary ones: System Manager, Human-Machine Interface (HMI), and Utilities module. A. System Design Principles The system is compounded in Object Oriented (OO) form. It consists of separated blocks with maximal independent functionality and minimal data flow between these blocks. The system is flexible in reference to further changes of its components at later stages without substantial system reorganization. Many of the system components are simulated in either simplified or a full form, in reference to the fullness

of physical phenomena simulation. For example, probability of target detection may be defined as a sensor parameter or calculated taking into account target size, environment specific, sensor operational regimes, etc. Target parameters and trajectories are also defined by different ways: with fixed route points or leg kinematic model. The system supports joint work of different Data Fusion Centers, processing output data of various sensors, with arbitrary sensor to DFC and DFC-to-DFC inter-connections. B. Human-Machine Interface (HMI) Module The description of the testbed is started from auxiliary modules as they are the interface between the user and the testbed itself. The HMI module is responsible for definition of system work scenario and results visualization. It includes Scenario Generator (SG) and DFC Connection Module. 1) Scenario Generator: The Scenario Generator is built by using the “Lego” principle. This user-friendly interface allows creating scenarios from different elements, in a way similar to how it is done in the popular game. Fig. 2 shows Scenario View Mode, where on the right side all the various elements of the defined scenarios and their components are defined, whereas on the left side we can see the graphical representation of the chosen scenarios including the trajectories of all the targets and sensors. C. Simulator Module The Simulator module provides simulation of defined scenarios of a given DFC, generating measurements based on simulative sensors, targets, target emitters, and other elements of the scenario. The module also simulates different physical phenomena of environments in which the simulation takes place, affecting signal propagation. The scenario simulation consists of the following components: • Target Scenario simulation • Sensor Scenario simulation • Environment Scenario simulation The simulation process is organized as follows. Initially, upon receiving the simulation scenario, the following actions are performed: sensors and targets are generated, corresponding routes are transformed into target and sensor trajectories, defined environment regions are transformed into the internal

format. Further, each simulation step, the following actions are performed: • For each target a current target location and emissions are calculated. • Each sensor’s current state is evaluated and the lists of targets, which can be detected by this sensor, are obtained. • For each target, which can be detected by this sensor, the signal on the sensor input (including noise and interference accompanying the signal) is evaluated, taking into account the current sensor and emitter parameters, and current environment conditions on the signal path. • For each signal received by this sensor, the detection occurrence is randomly generated depending on the signal characteristics, noise level and sensor parameters. If the detection occurs, corresponding measurements are generated. • For each sensor, the false alarms, arising due to clutter, internal sensor noise, as well as other reasons, are generated. The simulator has tracking information extrapolation capability which is automatically provided for the cases when the time which it takes to a sensor or a group of sensors to provide a new portion of measurements is greater than the maximal time after which either true tracking information or extrapolated results should be obtained in order to update the system’s database and output the tracking data to the screen or other output channels. Trackers which may be used in a system might already have that capability but in case they have not, the testbed provides it. 1) Target Scenario Generation: Target Scenario generation includes calculation of the location and kinematic parameters for each target as well as emission and reflection parameters. It consists of a target list and of a list of Target Scenarios. Each of the targets includes Target Type, Route (or trajectory) and optionally the list of Emitters, corresponding to the particular target. Target Scenario generation is also responsible for determining emitters’ instant states according to their operational scenario. Target Scenario generation includes route generation, which produces a route according to its parameters and calculates the target kinematic parameters (velocity, acceleration, climb rate, etc.) for the given time instant. It includes generation of the leg list. A route thus consists of a sequence of items composed in such a way that the end of the previous leg serves as a start for the current leg. Three different types of routes may be defined: • Fixed-point (FP) route, consisting of FP legs • Fixed-kinematic (FK) route, consisting of FK legs. • Stationary route, designating a non-moving object. FP Leg calculation FP leg kinematic parameters at a given time instant are calculated based on its given parameters as well as the kinematic parameters at the end of the previous leg. (X0 , Y0 , Z0 ) and (X1 , Y1 , Z1 ) denote the leg start and end point coordinates respectively, and V0 and V1 denote the corresponding target speed values at the start and end of

the leg respectively. At the first stage, the trajectory calculation is decomposed into 3 sequential parts: • Calculating a curved line, a part of the total horizontal trajectory • Straight-line segment with constant target acceleration • Straight-line segment with constant target velocity At the second stage, the target trajectory in z axis is evaluated, which is divided into 4 sequential segments: • Vertical acceleration • Constant climb rate • Vertical deceleration • Zero climb rate FK Leg calculation The kinematic model of the target in FK leg modeling is given by xk+1 = Fk xk + Gk vk

(1)

where xk is the target state vector at time step k, vk is process noise vector, Fk is the state transition matrix from time tk to time tk+1 and Gk is the noise gain. The target motion can follow different kinematic models, among those white noise acceleration model, Wiener process acceleration model, and coordinated turn model. We assume motion in x-y plane and in z axis to be independent. A choice is given between a number of models for both x-y plane and z axis. The target state for x-y and for z will be denoted as x and x respectively. In a similar way there will be represented the transition matrix F and the noise gain matrix G. The covariance matrix for the noise components will be denoted by Q. Thus the combined state vector, transition matrix, noise gain, and noise covariance matrices are obtained by combining their components as follows         F 0 G 0   ,F = ,G= , x= x x 0 F  0 G    0 Q Q= 0 Q 2) Sensor Scenario: The Sensor Scenario contains a list of sensors, which have different physical principles and performance regimes. Based on the Target Scenario, environment conditions and the Resource Management module commands, the sensors generate simulative measurements, which are the input for the further Tracker processing. The measurement generation is performed based on the following factors: • Parameters of the signal, received by a sensor • The sensor type and parameters, characterizing the sensor operational ability D. Peer-to-Peer Implementation of Data Flow for NetworkCentric Architecture Simulation As shown in Fig. 1, the testbed implements a decentralized multisensor tracking and fusion architecture. Each DFC thus represents one of the nodes which can interchange information with other DFC nodes. The information can be

Fig. 3.

DFC Connections View Fig. 4.

measurements or tracks. Each DFC acts independently and has its tracker to process the measurement data information which is obtained from the sensors pertaining to the specific DFC. Also, each DFC is actually an independent standalone instance of a testbed application running on the same or remote computer. One of the DFC nodes, depicted in Fig. 1, is defined as a main DFC node which designates that it is the node belonging to the current testbed application instance. There can be created an arbitrary configuration of DFC connections using graphic interface of the DFC Connection Module. In a specific testbed application, a specific desired data fusion configuration should be first defined. Fig. 3 shows an example of a possible data flow configuration between different data fusion centers. DFC “C”, which is the DFC of the current node (main DFC center), should receive track data from DFC “A” and DFC “B”. Both DFC “A” and DFC “B” designate remote DFCs running as standalone testbed applications on the same or remote locations. A specific tracker type as well as the scenario to be run on the main DFC should be selected. After that, a connection with remote stations, where DFC “A” and DFC “B” are running, should be established. The communication scheme of testbed instances can be classified as peer-to-peer type to distinguish it from the client-server model, where one can identify a fixed division into clients and servers. In our case, every testbed instance can communicate with any other testbed. When considering a specific one-direction communication channel between two testbeds, then it is convenient to see it as a client-server type connection. Each testbed contains both server and client mechanisms in order to be able to receive information from the multiple clients (server side) and to send information to multiple servers (client side). 1) Server Side: As mentioned, though the overall communication scheme is peer-to-peer, as micro level we consider each instance of data transfer between two testbeds as clientserver connection. The server waits for the incoming client calls. When the incoming client call is received and accepted, the server thread will not start the communication with this client. Instead, it will create a new thread and will pass the client connection to this thread letting the communication between the server and this new client become the full-time job for this newly created thread. Then the server will wait

DFC Connection properties

for another potential incoming communication. By doing so, the server can handle multiple clients. Each DFC has the following properties, as shown in Fig. 4: • • •

Name. IP address. Port number.

2) Client Side: The part of a particular testbed mechanism that is responsible to transmitting the relevant tracking data to remote testbed or testbeds is called client. Each client contains two tables - ”New connections” and ”Established connections”. ”New connections” table contains the parameters (IP address and the port number) of the remote DFC connection to which should be established. After the successful connection with the remote DFC, the responsibility for maintaining the connection is transferred to the ”Established connections” table, and the corresponding record in the ”New connections” table is cleaned. 3) Time Synchronization: Each testbed has its own local time which is to be synchronized with the rest of the testbeds participating in the peer-to-peer communication. The example of the synchronization scheme between two testbed instances is as follows: • • •

Both testbeds equalize the simulation speed. Round-trip-time (RTT) of data communication is measured. If the local time difference between two testbeds is within a pre-defined threshold, taking into account the measured RTT, data transfer is initiated. Otherwise the following steps are taken: – The testbed with more advanced local time will pause. – The testbed where the local time lags behind will run and notify the other one ahead of time before both local times are equalized.

In case of more than two testbeds, all the testbeds are paused and continue running after testbeds that lag behind in time equalize their local time with them according to the scheme above.

Fig. 6.

Fig. 5.

RMSE for two particular targets

IMM/Assignment tracker simulation. One sensor and nine targets

E. Tracker Module The Tracker Module analyzes sensor measurements and presents the results graphically in Real Time Mode, reflecting the dynamics of observed targets. The main purpose of this module is to create and support the dynamic picture of moving and stationary targets based on the measurements, arriving from the different sensors and external information sources. The tracker module is realized as a dynamic library, which may be selected from the available tracker realizations and connected to the testbed “on-the-fly”. Thus a library of trackers is maintained, all of them having a unified interface with the testbed and ability to be dynamically connected to it. Currently there are two implemented trackers in the library - the IMM/Assignment tracker, and the Particle Filter (PF) tracker. Fig. 5 shows simulation, involving IMM/Assignment tracker, where there is one sensor (a radar) and nine targets. The small window in the middle of the main simulation screen gives an opportunity to independently view an arbitrary part of the monitored area. F. Tracking Performance Metrics Module This module receives the actual data regarding all the target and sensor locations as well as the results from the Tracker module after a pre-defined number of Monte-Carlo runs of a given scenario. It analyzes the provided information and evaluates the quality of the Tracker module performance. The required performance evaluation method can be chosen from a list of available ones and its properties can be defined. The following metrics are currently implemented [14]: • Track Accuracy: At each time step, a 2D assignment is performed between true targets and estimated tracks. Root mean sum squared error (RMSE) history in position and velocity are calculated for the associated targets. • Completeness History: This is the proportion of the targets that should be tracked which are declared as tracks at each time in the scenario. • Timeliness: Time at which a particular object that should be tracked has a valid declared track. • Ambiguity: – Redundant track mean ratio: the number of declared

Fig. 7.

Simulation import/export menu options

tracks that are assignable to real objects that should be tracked, divided by the number of declared valid tracks. – Spurious track mean ration: the number of declared tracks that are unassignable to real objects that should be tracked, divided by the number of declared valid tracks. • Track continuity: – Mean cumulative swap of tracks: the cumulative number of swaps of tracks for particular objects and averaged across all objects by time t into the scenario. – Mean cumulative broken tracks: the cumulative number of breaks of tracks for particular objects and averaged across all objects by time t into the scenario. Fig. 6 shows position RMSE for two particular targets. G. External Data Support The testbed employs an Extensible Markup Language (XML) standard for storing its internal data and information exchange between different instances of the testbed. The testbed uses XML for two different tasks. The first one is storing configuration information, created objects and their relations in a database. The second task, is a feature of importing and exporting simulation data of specific scenarios with ability to import them to any particular testbed installations. Two modes are provided for importing and exporting simulation data: •

Playback mode: in this mode all the data related to a specific simulation that is stored as an external XML file is imported to allow exact repeat of this simulation with consequent analysis of its performance.

Fig. 8. Simulation import dialog options: (a) Playback mode (b) Simulation mode

•

Simulation mode: in this mode tracking/measurement information of the recorded simulation is fused in the current simulation, similar to how the track/measurement information arriving from another instance of the testbed is fused. III. C ONCLUSIONS

In this paper we have presented a multisensor-multitarget tracking testbed for large-scale distributed scenarios. This development gives an opportunity to research into various aspects of distributed tracking systems. Possible application areas are tracking algorithms, performance evaluation methods, target movement models, sensor models, environmental influence on tracking process, noise models pertaining to all the testbed elements, distributed data fusion architectures, security, actual tracking systems implementation, and many others. R EFERENCES [1] Y. Bar-Shalom, and X. Li, Multitarget-Multisensor Tracking: Principles and Techniques, Storrs, CT: YBS Publishing, 1995. [2] Y. BarShalom, X. R. Li and T. Kirubarajan, Estimation, Tracking and Navigation: Theory, Algorithms and Software, John Wiley & Sons, New York, June 2001. [3] S. Blackman, and R. Popoli, Design and Analysis of Modern Tracking Systems, Dedham, MA: Artech House, 1999. [4] W. D Blair, G. A. Watson, T. Kirubarajan and Y. Bar-Shalom, “Benchmark for radar resource allocation and tracking in the presence of ECM”, IEEE Trans. Aerospace and Electronic Systems, Vol. 34, No. 3, pp. 1015-1022, October 1998. [5] K. Chang, R. Saha and Y. Bar-Shalom, “On Optimal Track-to-Track Fusion”, IEEE Trans. Aerospace and Electronic Systems, 33(4), pp. 1271-1276, October, 1997. [6] H. Chen, T. Kirubarajan and Y. Bar-Shalom, “Centralized vs. Distributed Tracking Algorithms for Air to Air Scenarios”, Proc. of SPIE Conf. on Signal and Data Processing of Small Targets, Vol. 4048, April 2000. [7] T. Kirubarajan, Bar-Shalom, W. D. Blair and G. A. Watson, “IMMPDA Solution to Benchmark for Radar Resource Allocation and Tracking in the Presence of ECM”, IEEE Trans. Aerospace and Electronic Systems, Vol. 34, No. 3, pp. 1023-1036, October 1998. [8] T. Kirubarajan, Y. Bar-Shalom, K. R. Pattipati and I. Kadar, “Ground Target Tracking With Topography-Based Variable Structure IMM Estimator”, IEEE Trans. Aerospace and Electronic Systems, AES36(1):2646, Jan. 2000. [9] T. Kirubarajan, Y. BarShalom, and K.R. Pattipati, “Multiassignment for Tracking a Large Number of Overlapping Objects”, IEEE Transactions on Aerospace and Electronic Systems, AES37(1), pp. 221, January 2001. [10] A. Kumar, Y. Bar-Shalom and E. Oron, “Image Segmentation Based on Optimal Layering for Precision Tracking”, Partitioning Data Sets, Series in Discrete Mathematics and Theoretical Computer Science, Vol. 19, pp. 155-168, AMS, 1995. [11] L. Sevgi, A. Ponsford and H. C. Chan, “An Integrated Maritime Surveillance System Based on High-Frequency Surface-Wave Radars”, IEEE Antennas and Propagation Magazine, Vol. 43(4), pp. 28 - 43, August 2001.

[12] H. Wang, T. Kirubarajan, and Y. Bar-Shalom, “Large Scale Air Traffic Surveillance Using Imm Estimators With Assignment”, IEEE Trans. Aerospace and Electronic Systems, Vol. 35, No. 1, pp. 255-266, Jan. 1999. [13] G. A. Watson and G. H. McCabe, “Benchmark Problem with a Multisensor Framework for Radar Resource Allocation and Tracking of Highly Maneuvering Targets, Closely-Spaced Targets, and Targets in the Presence of Sea-Surface-Induced Multipath”, Technical Report NSWCDD, Dahlgren, VA, 1999. [14] R. L. Rothrock and O. E. Drummond, “Performance Metrics for Multiple-Sensor, Multiple-Target Tracking,” Signal and Data Processing of Small Targets 2000, Proceedings SPIE, Vol. 4048, pp. 521-531, Jul. 2000.