Multi-approach Strategy for Multi-Sensor Data Fusion

Multi-approach Strategy for Multi-Sensor Data Fusion Enhancement Unified analysis of advanced and unconventional techniques, targeted to Airport Surveillance enhancement

Carlo A. Vertua, Luca Saini

Olivier Baud, Nicolas Honoré, Peter E. Lawrence

THALES Italy - Air Operations Division Milan, Italy [email protected]

THALES - Air Operations Division Rungis, France [email protected]

Abstract — Target identification and tracking in Airport Surveillance are heavily influenced by the performance and limitations of the surveillance sensors used. The measurement process of such sensors is adversely affected by environmental clutter and dense traffic conditions and this may result in MultiSensor Data Fusion (MSDF) not being able to adequately manage the associated errors. The scope of this study is to identify a number of techniques pertinent to the enhancement of MSDF performance and reliability, and to elaborate on the analysis of some such techniques. Keywords — Multi Sensor Data Fusion, Target Tracking, Surveilllance, Identification.

I.

INTRODUCTION

Airport Surveillance very often involves situations of dense traffic conditions and extraneous sensor and environmental clutter noise (in terms of spurious radar echoes, of reflections and multipath for multistatic sensors, of signals stability due to structures shadowing, etc.). Under such conditions the close proximity of targets with each other and with environmental structures of large Radar Cross Section (RCS) leads to problematic discrimination and/or tracking results for airport surveillance; factors such as reduced sensor coverage of the airport area due shadowing and multipath also negatively impact the surveillance process. The task of a Multi-Sensor Data Fusion (MSDF) process is to estimate target characteristics, such as position, from surveillance data provided by an ensemble of sensors. This entails the assessment, comparison and merging of such measurement data based on (often probabilistic) multihypothesis decisions made on parameters such as target spatial proximity, target speed and heading, and target identification and classification [1]. In the case of dense traffic and/or adverse environmental conditions, the aforementioned parameters used in the MSDF decision methodology are difficult to identify and/or extract from surveillance data and this degrades the performance of such a surveillance estimation process. A possible circumvention to this may lie in estimating the individual

parameters used by the MSDF via auxiliary state-of-the-art techniques. The main features and components to be considered in this paper for an MSDF global process are as follows: •

MSDF architecture (including the design of pre/postprocessing);

•

Correlation/association algorithms;

•

Estimation algorithms;

•

Target classification algorithms;

•

On-line computation of sensor characteristics.

For the analysis component of this paper the following criteria have been taken into account: •

Tracking continuity and integrity;

•

Tracking accuracy;

•

Ability to interface with legacy sensor sources;

•

Adaptability to the sensor quality and environment conditions;

•

Complexity of algorithm “tuning” (required for lowcost solution and quick installation).

The first section of this paper deals with an overview of recurring sensors and MSDF miss-behaviors in a high-density airport traffic environment. The second section provides elaboration on proposed MSDF architectures. Finally, in the third section a discussion and critique is provided on the choice of data fusion algorithms for MSDF used in airport surveillance. II.

AIRPORT SURVEILLANCE MAIN ISSUES

A. Tracking continuity and integrity issues In airport surveillance, the level of false target reports depends on factors such as land-clutter and multipath for radars

such as Surface Movement Radar (SMR), relative target-sensor positioning and orientation, as well as environmental conditions such as rain which typically leads to high probability of false alarms for SMR sensors ([2]). Other contributors to false target reports are those encountered at the MSDF level. These include the creation and maintenance of duplicate tracks for the same target, target splitting, target merging, same identity-code for different targets, to name but a few. The correlation/association algorithms used in an MSDF process are key to ensuring against such target estimation pathologies.

•

Following figure is showing the breakdown of chosen MSDF architecture. Air Surveillance Approach/En Route MSDF

B. Tracking accuracy issues Depending on the type of reference position taken for detections arising from an extended target, the estimated target position provided by a tracking system may appear erratic over time; this is typically encountered when a target is stationary. Depending on the type of sensor the detections arise from, such reference positions may be taken as: •

The center of mass of position for detections of an extended target from a given radar (such as an SMR);

•

The Mode S transponder location (typically located in the nose of an aircraft), for detections provided by an Multi-LATeration (MLAT) system;

•

MSDF ARCHITECTURE

The basic architecture of an MSDF system dedicated to airport traffic is commonly “distributed”, in the sense that each sensor processes its own measurements before passing them to the MSDF. Such an architecture leads to reduced computational overhead for the core data fusion processes, however may lead to poor target estimation performance due to loss of information from sensor measurements due to the sensor-based data pre-processing. It would be advantageous to the MSDF system to provide raw/unprocessed data from such sensors in order that all available data is used; in fact, the next generation of surveillance sensors, such as MLAT and ADS-B, do not typically use any such data pre-processing due to their improved detection and target positioning accuracy. In conclusion, the following hybrid architecture is proposed for improved MSDF-based airport surveillance. This consists of : •

Local tracking for sensor report consistency checks with MSDF feedback;

ADS-B FP

MLAT

MLAT FP

SMR

SMR Extractor

Cameras

Camera FP

MSDF core tracking Integrated Target Data (Track)

ADS-B reports, for their part, may suffer from loss of detection, position biases and time-stamping offsets related to associated GPS functioning.

III.

ADS-B

Ground Surveillance

The GPS antenna location, (often located near aircraft’s tail), for an Automatic Dependent System Broadcast (ADS-B) report.

By way of consequence, these erratic track positions lead to speed-vector instabilities. Such position “jumps” may also be caused by older data fusion techniques based on weightedsums local tracks estimated from the individual sensors.

A Multiple Report Variable-Update technique for the multi-sensor tracking including use of contextual information.

Figure 1. Proposed MSDF architecture

A. ADS-B Front Processing Also known as an ADS-B server, the ADS-B Front Processor (ADS-B-FP) is a module that supports the following key functions: •

Sensor report format verification and possible data discarding;

•

Combination of reports from several ADS-B ground stations and suppression of duplicated reports. This is required to be achieved with little time bufferization. As the ADS-B reports are supposed to be the best quality reports that feed the SMGCS system, it is important to process them at the MSDF core tracking algorithms level as soon as possible. Moreover, if the A-SMGCS system is re-broadcasting the ADS-B data in the scope of a Traffic Information Service – Broadcast (TIS-B) application, the latency must be reduced;

•

Filtering of sensor reports according to a given Domain/Area Of Interest;

•

Filtering of sensor reports according to a required level of accuracy based on, say, the Navigation Integrity/Accuracy Category (NIC/NAC) information;

•

Filtering of sensor reports according to their latency;

•

Provides data output in multiple formats.

B. MLAT Front Processing Also known as an MLAT server, the MLAT Front Process or (MLAT-FP) is a module that supports the following key functions: •

Sensor report format verification and possible data discarding;

•

Filtering of sensor reports according to a given Domain/Area Of Interest;

•

Filtering of sensor reports according to a required level of accuracy based on, say, the reported Dilution Of Precision (DOP);

•

Filtering of sensor reports according to their latency.

C. SMR extractor Target tracking of extended targets, or groups of targets, can be achieved by use of a Bayesian multiple tracking methodology on sequences of either video images or highresolution radar profiles. Although it is desirable to use the raw data from an SMR, the SMR local tracking data may also be advantageous in providing a means to validate the raw data that will be used in an MSDF. The collection of SMR data used in the Bayesian methodology is required to be extracted from the raw radar detections via thresholding related to background-noise estimates, and by use of a clustering scheme such as single-link nearest-neighbor method. These clusters are referred to as blobs. This is depicted in next figure.

Figure 3. Track segmentation and guard regions used for blob-to-track conflict

As multiple blobs may be associated to a single target owing to image irregularities, shadows, occlusions, etc., a blobto-track multi-assignment method is adopted. By modeling each target region as a Gaussian mixture of target parameters (such as extent and orientation), a collection of weights is calculated and attributed to all the available targets at each frame update of a video sequence. A weighted sum of target estimates using these weights yields a pseudo-measurement that is used in a tracking filter, such as a 2D Cartesian Constant-Velocity (CV) Kalman filter, to predict the track position. D. Camera Front Processing The need for pre-processing or a gateway preceding the MSDF core functionalities depends on the way the camera data is collected.

Image capture

Figure 2. Extracting target blobs from sensor data

Each extracted blob is contained in the smallest bounding rectangle in order to simplify the MSDF processing. Target tracking filter states/positions (having been initialized at the start of such a sequencing) are also bounded by a rectangular box, and moreover an auxiliary guard-rectangle further surrounds this target box in order for later blob-to-track conflict analysis ([3]). This is depicted in following figure.

Detector and Blob extractor

Association

Background calculation

Track initiation

Track update Track Data Base Track prediction

Figure 4. Camera local tracking

If the camera data is received from a group of cameras that use some form of local tracking, then no gateway function is

required to be implemented. Otherwise, when no local tracking is present in the camera data processing chain, a gateway is put in place to perform such local tracking. The track output is to be considered as the Camera FrontProcessor (FP) level if it used in this way. Otherwise, a local track database different from the multi sensor track database is implemented within the MSDF module. The local tracks are then correlated and associated with the multi sensor tracks. The estimation is performed taking into account that the object used are tracks rather than reports. IV.

MSDF CORE TRACKING ALGORITHMS

The multi-sensor tracking receives sensor reports and performs the following tasks according to sensor reports type. All tasks are processed before another sensor report is received and processed. Data fusion architecture includes: •

SMR report processing;

•

MLAT report processing;

•

ADS-B report processing;

•

Camera local tracks processing;

•

Multisensor tracking;

•

ADDs management;

•

On-line assessment of sensor characteristics (including errors).

The sensor reports (associated to the same target) updates a unique surveillance track. The sensor processing depends on the characteristics of the type of sensor in order to optimize correlation, association and tracking-update functions. Depending on the nature of the sensor data, the correlation / association / or track update processing can be fed with either sensor reports or sensor local tracks.. A. SMR report processing SMR report processing aims to solve potential association conflicts between SMR reports detected in the same or adjacent sectors. A buffer of SMR reports is memorized by the tracking function to correlate and associate the reports with surveillance tracks within a rotating azimuth gate (rotating about the SMR at the rate of this radar’s antenna). The purpose of this gating process is to perform a many-to-many correlation of SMR reports to tracks. The association is then based on best likelihood of correlated report-track pairs in an attempt for a one-to-one association (a complex assignment algorithm). B. MLAT (ADS-B) report processing The accuracy, latency and the attributes of the MLAT/ADS-B reports are used in the association stage of target tracking. Upon reception of a MLAT/ADS-B report, a one-to-many correlation is attempted against existing surveillance tracks. This type of correlation method allows individual processing of MLAT/ADS-B reports as soon as they are received in order to avoid introducing extra processing

delays (no bufferization). The association stage is then based on the best likelihood of correlated report-track pairs to in order to attempt a one-to-one association (a simple assignment algorithm). C. Multisensor Tracking The consistency of the different surveillance sources, updating the surveillance track, is assessed using tracking results from independent Multi-SMR, MLAT, ADS-B, Multi Radar and WAM tracking systems. Afterwards, correlation and association based on multi-sensor tracking is performed to confirm the association of tracking data to the same surveillance track. Once associated, the sensor report contributes to the multi-sensor track state in accordance with the estimated level of consistency of the related sensor type (inconsistent sensor reports are not used to update the surveillance track but are instead sent to the multi-sensor initialization process to create a new tentative track). The multi-SMR stand-alone track state vector is used to: •

Check the position consistency of any MLAT report;

•

Check the position consistency of any ADS-B report;

•

Estimate the distance between ADS-B and Multi-SMR stand-alone track states.

In case of position inconsistency of an ADS-B report, the ADS-B report does not update the track. This report is either sent to the initialization process to create a new tentative track or discarded, according to an offline criterion. If the estimated distance between the multi-SMR and the ADS-B stand-alone track states is greater than a pre-defined threshold, the ADS-B report will still update the track and the track will be distributed with a flag signaling that the information is inconsistent with other surveillance sources. Similarly, stand-alone MLAT and ADS-B filters are used to verify the consistency of such sensors’ information. Inconsistent reports from such sensors are not used to update the track but are instead sent to the multi-sensor initialization process to create a new tentative track. The tracking continuity with approach/en-route tracker is insured by using its tracks for quicker initialization. The MSDF tracker is designed to process SMR reports, MLAT reports, ADS-B reports, WAM reports, and Mode S ADD in the following way: •

Multi-SMR Stand-Alone Tracking: dedicated processing is performed to check the reliability and integrity of received SMR sensor data;

•

This multi-SMR stand-alone tracking filter is only updated by SMR sensor report (range, azimuth);

•

MLAT/ADS-B Stand-Alone Tracking: a dedicated processing is performed to check the reliability and integrity of received MLAT/ADS-B reports (via a “blunder-detection” process which entails time stamping and position jump issues);

•

•

Aircraft Derived Data (ADD) checking: each ADD related to the aircraft kinematics tracking information is checked for reasonableness according to the multi-sensor stand-alone track state vector that is not refreshed with any ADD; The data from the different sensors are then processed in the Multi-Sensor Kernel Tracking function, taking into account their consistency status. This Multi-Sensor Kernel Tracking function is split into two kind of tracking filters: o o

Horizontal tracking filter for ground surface tracking refreshed by all kind of sensors; Vertical tracking filter for Mode C tracking when a barometric altitude is available.

•

FS-IMM-C : Fixed Structure - Interacting Multiple Model with constraints;

•

VS-IMM : Variable Structure - Interacting Multiple Model without constraints;

•

VS-IMM-C : Variable Structure - Interacting Multiple Model with constraints;

•

PF-C : Particle Filter with constraint;

•

VS-IMM-PF-C : Variable Structure - Interacting Multiple Model – Particle Filter with constraint.

For each constrained filter, both measurement projection and state projection have been investigated. A mixed projection gives the best results. Figure 5. Tracking estimates comparison from several tracking filters

D. Correlation/association algorithms As soon as a one-to-many assignment problem arises (ADS-B, MLAT, …), a simple Nearest Neighbor Standard Filter (NNSF) is recommended as the complexity of the association is reduced. Such data are coming without any bufferization and then asynchronously, so they are used as soon as possible in the fusion processing: based on the fact that they have identification parameters such as a 24 bit Mode S address or a SSR code, the association is also less complex and do not require enhanced correlation/association solutions. When a many-to-many assignment problem is encountered (SMR radar reports buffer) then the preferred approach to be considered is the Jonker-Volgenant-Castanon (JVC) algorithm. This solves the assignment optimization problem into two stages: the first one ensures the feasibility of the assignment problem by an appropriate conditioning of the assignment matrix and is similar to the auction algorithm; the second stage is similar to a sparse version of the Munkres algorithm and considers only the finite values of the assignment matrix. As soon as a bufferization is made, then track cluster and sensor report buffer needs to be compared, correlated; the association is then decided at the end using an enhanced assignment algorithm. This is moreover true when the incoming sensor data have no identification parameters (non cooperative sensors) and when the corresponding sensor is providing a great number of false reports. E. Estimation algorithms The global estimator is based on a multiple report variable update technique, allowing to update the fused track as soon as possible with incoming reports. The estimation methods that have been investigated for the central fusion estimator of various kind of sensors are the following: •

KF/EKF: Standard Kalman filter (used for benchmark only);

•

FS-IMM : Fixed Structure - Interacting Multiple Model without constraints;

The estimator that will be finally chosen has been benchmarked between several solutions with simulation data. The best solution seems to be the PF tracking filter but the VS-IMM-C solution seems to be the best compromise between tracking performance versus CPU load; this will be further assessed in an operational environment. ADDs such as ground speed and true track angle can slightly enhance tracking performance as soon as they are used as measures rather than as triggering state noise or target model switch. The local estimators per kind of sensor reports that are recommended are the following: •

multiple SMR track update: 2 model based IMM tracking filter;

•

MLAT only track update: 2 model based IMM tracking filter;

•

ADS-B only track update: standard Kalman filter,

These tracking filters are not constrained according to the topology as they are considered as stand alone and provide, per kind of sensor, a specific situation picture. Moreover they do not make use of ADDs as this responsibility is deferred to the central fusion tracking filters. However, each track has at its disposal these particular tracking filters that can be initiated

and used, or not, according to the location of the target at one time and to the sensor coverage.

sensor-related and aircraft-related systematic or random errors, it helps in improving the continuity and accuracy of the tracking function ([5]).

F. Target classification In comparing the use of Neural Networks (NN) and Kernel Method classifiers, both have a similar degree of arbitrariness in the choice of defining parameters such as the neuron activation functions for NN and the kernel function for the multidimensional data separability in the Support Vector Machine (SVM). Both are capable of over-training. The recommendation will be in favor of the SVM classification scheme over the NN due to a relative simplicity in its structure due to the fact that there is no clear criterion for defining the NN topology and although one hidden layer is generally sufficient to yield a solution to any classification problems, there is no proof that this is the “most optimal” - this seems to be an added level of arbitrariness with the NN setup ([4]).

The bias registration processing is scheduled as follows:

As an aside note, one should proceed with some degree of caution when using image moments as features in classification. The fact that such metrics exhibit perspective invariance does not warrant their use for images of targets captured at aspect angles exceeding the reasonable bounds of rotation, translation and projective scaling. Moreover, use of images with a high degree of variability in the background will degrade the discernability power of using these features in a classification scheme. Such metrics should be used when targets of interest have been separated from the image background/clutter, or when very similar conditions are maintained when capturing test and reference images.

The reference sensor to be used is the SMR.

When restricting classification input data to the sensor genus of radar, the complexity of implementation of the Dempster-Shafer (DS) method has little justification over use of a priori defined conditional probability distributions used in a Bayesian context. However, in the case for highly heterogeneous sensors there is strong support for use of the DS method, albeit for the vague criterion in literature of which mass functions to use. Classification via use of optical camera/video data is given the highest recommendation due to its level of data integrity and robustness in comparison with other sensors, especially in regions where range is not a limiting factor such as in an airport area. It is also recommended to use classification aided tracking of optical video features for providing a reference to other sensor sources; for instance, for bias processing of radar sensors with respect to a reference. Moreover, emphasis should be placed on the use of optical flow features more than on image feature parameters such as template matching and/or moments, and this being due to the inherent link of target dynamics with objects in a sequence of video frames. Having said this, there is much merit in the automated registration of tail numbers and such aircraft identifiers, using a method such as SVM or a Bayesian word-pattern classifier. G. On-line sensor/aircraft characteristics computation This component has became one of the most important component of every MSDF process. Able to compute

•

Macro-errors computation during initiation phase (first computations of biases before steady state is reached): a reference sensor that makes use of the contextual information for own calibration is used, other sensors being aligned afterwards;

•

Macro-errors computation during converged phase: it uses a standard non-recursive method (helps in reducing CPU loading), but, still uses the reference sensor alignment according to the context for bias values computation check;

•

Micro-errors computation (aircraft dependent errors).

The sensor stochastic errors, also known as noises, are estimated on-line for ADS-B sensors and approach radars. ADS-B noise is assessed per target, while the approach radar noises in range and azimuth are assessed per radar resolution cell. The estimation method of these noises is achieved as follows. Firstly distances computed between each sensor report and the associated reference multi-sensor tracks (the reference track being interpolated/extrapolated to the time of the report); these values distance are stored. Secondly, statistics are derived from the stored data for which the outlier reports have been discarded. These noise values are then retrofitted for further use during the various tracking filter updates. V.

CONCLUSION

In this paper we have identified algorithms that could be used in the scope of a global improvement to the airport surveillance data fusion process. Initial results on simulated sensor data have indicated the ability of newly developed algorithms to reach a strong level of performance. Future work will entail testing such data fusion methodology against recorded and live airport sensor data, results of which will be provided in a future publication. REFERENCES [1] [2]

[3]

[4]

[5]

T. E Fortmann., Y. Bar Shalom, “Tracking and data association”. Academic-Press, Boston, 1988. G. Galati, M. Leonardi, A. Cavallin, G. Pavan. Airport Surveillance processing chain for high resolution radar. IEEE Transactions on Aerospace and Electronic Systems, Volume 46, Number 3, July 2010. J. Garcia, G. De Miguel, A. Berlanga, J. M. Molina, A. Soto. ASDERADAR Data Association for Robust and Efficient Tracking on airport surface. International Radar Symposium, 2003. H. Xiao, F. Sun, Y. Liang. Support Vector Machine Algorithm Based on Kernel Hierarchical Clustering for Multiclass Classification. International Conference on Electrical and Control Engineering (ICECE), 2010. J. A. Besada Portas, J. G. Herrero, G. De Miguel. New approach to online optimal estimation of multisensor biases. IEE Proceedings Radar, Sonar and Navigation, Volume 151, Number 1, 2004.