Multisensor Tracking and Fusion for Maritime Surveillance

Multisensor Tracking and Fusion for Maritime Surveillance Craig Carthel, Stefano Coraluppi and Patrick Grignan NATO Undersea Research Centre (NURC) Viale S. Bartolomeo 400, 19126 La Spezia, Italy [email protected], [email protected], [email protected] Abstract – Over the past several years, the NATO Undersea Research Centre has conducted extensive research in multisensor networks for undersea surveillance, culminating in the development of the DMHT tracker. In this paper, we discuss upgrades to this technology and its application to maritime surveillance. Keywords: Maritime Surveillance – Automatic Identification System (AIS) – Coastal Radar – SAR Imagery – Multisensor Fusion and Tracking – Track Fusion – Anomaly Detection

1 Introduction In recent years, global political changes have generated significant interest in surveillance applications to combat terrorism, smuggling activities, and illegal immigration. An important theater for these activities is the maritime domain. This has led to a number of national and multinational initiatives in maritime surveillance with the goal or having knowledge of all coastal and high-seas activities relevant to national security.

target Fig. 1. An example of a mobile surveillance network (bistatic detection in blue, monostatic in red).

As part of these efforts, NATO is pursuing research activities to exploit existing sensor systems in support of maritime surveillance. Recent progress is documented in [1]. To date, these efforts have focused on exploitation of Automatic Identification System (AIS) transponder data, which is available in principle for all large vessels [2]. Consequently, the NATO Undersea Research Centre (NURC) is conducting preliminary activities to assess how to contribute to maritime surveillance. NURC has significant experience in multisensor tracking and fusion systems for undersea surveillance. Active sonar surveillance is generally based on one of two system concepts: mobile LFAS-equipped platforms (suitable for expeditionary tasks), and fixed/drifting deployed fields (suitable for surveillance of ports, harbors, choke points, etc.). An example of the first system concept is illustrated in Figure 1, which shows the monostatic and bistatic sourcereceiver combinations that were used in a 2003 sonar sea trial jointly conducted by NURC and TNO. Figure 2 illustrates the second system concept: deployable sonar equipment used in a moored configuration.

Fig. 2. An example of a fixed surveillance network. Active sonar signal and information processing includes processing of raw hydrophone data into contact-level data [3], and multi-sensor fusion and tracking [4]. Application of NURC technology to sea-trial data is discussed in [5]. In the multi-national maritime domain, the primary sensor assets include AIS tracks, coastal radar contacts, and SAR imagery detections. In this paper, we will discuss the use of distributed multi-hypothesis tracking (DMHT) technology for generating a real-time consolidated surveillance picture. This is a vital requirement for maritime surveillance, but it is not sufficient: anomaly detection algorithms must be developed, based on the analysis of single and multi-vessel kinematic patterns [6-

7], as well as on inference-engines that exploit web-based sources (e.g. Lloyd’s databases) and reason over vessel identities.

1.

Multi-sensor signal and information processing that generates contact-level data for all available sensors (coastal radar, SAR, video, IR, etc.);

A recent application of multi-sensor fusion for port surveillance is described in [8]. Our work differs from port-protection activities in a number of respects. We are addressing a global surveillance problem, leading to a number of challenges: (1) we exploit networked AIS, leading to corrupt-data issues; (2) we must address coverage-gap issues; (3) we employ target-centric tangent-plane coordinate systems rather than the single 2D coordinate frame that is a reasonable approximation for local surveillance problems. Also, compared to [7], our suite of sensor differs, as we employ SAR-based detections in addition to coastal radar and AIS.

2.

Global multi-sensor fusion and tracking that is capable of processing (potentially anomalous) AIS tracks and contact-level or track-level data from other sensors to produce a single, consolidated surveillance picture;

3.

Anomaly detection algorithms that drastically reduce the number of multi-sensor tracks and identify only those with kinematic or other anomalies;

4.

Potentially anomalous tracks require further operator investigation and a combination of operator-assisted and automated re-tasking of available assets. The latter is relevant in the face of scarce surveillance assets that must be used optimally to address numerous data-collection needs.

This paper is organized as follows. In Section 2, we illustrate an overall vision for maritime surveillance that includes semi-automated sensor management. Section 3 describes our upgraded DMHT tracker that is the basis of our contribution to maritime surveillance activities. Simulation-based validation of the tracker is discussed in Section 4. Section 5 describes our experimental activities, and Section 6 presents our preliminary findings and a discussion of the way ahead.

In this paper, we focus on the second of these four technology areas, a high-performance multi-sensor fusion that provides the basis for anomaly-detection work.

2 Vision for Maritime Surveillance

A high-level block diagram that includes the elements described above is illustrated in Figure 3.

An overall surveillance system for the maritime domain should include the following components:

Fuser and Tracker: • multi-source & multi-rate RAW • distributed MULTI-SENSOR • multi-hypothesis DATA • high precision • robust & computationally efficient S1-R1 Contacts

MHT Tracker

S1-R2 Contacts

MHT Tracker

S2-R1 Contacts

MHT Tracker

S2-R2 Contacts

MHT Tracker

Tracks

MHT Tracker

GIS User Interface: • alert notification • drill-down capability • asset override

SIGNAL AND INFORMATION PROCESSING

Fused Tracks

SURVEILLANCE ASSETS U.S. AIR FORCE

GIS USER INTERFACE

MULTI-SENSOR CONTACT DATA MARITIME DATA FUSER AND TARGET TRACKER AUTOMATED SENSOR MANAGER

Sensor Manager: • Constrained optimization • Monte Carlo techniques • Information flow evaluation

ALERTS

Anomaly Detector: CONFIRMED MULTI-SENSOR TRACKS

ANOMALY DETECTOR

Fig. 3. A notional closed-loop maritime surveillance system

• track classification • network analysis

3 DMHT Tracking Data fusion and target tracking are critical components of a multi-sensor surveillance capability. Contact-level data, particularly in multi-sensor settings, provides far too much information to a human operator. Rather, the output of an effective multi-sensor tracker provides a unified surveillance picture with a relatively small number of confirmed tracks and improved target localization.

approach is scan-based, contact-based, includes multihypothesis tracking (MHT), and is Kalman-filtering based, and has culminated in the development of the distributed MHT (DMHT) tracker. Our performance modeling and sea trial-based evaluation of the DMHT leads to a number of fundamental conclusions about the state-of-the-art in multi-sensor automated tracking:

Automated tracking provides a significant contribution to the active sonar processing chain, with a two orders of magnitude reduction in false objects at comparable detection levels, with improved localization accuracy and limited track fragmentation;

Effective multi-sensor tracking requires (1) likeperforming sensors or, at a minimum, comparable single-sensor data rates; (2) well registered data, i.e. removal of bias errors through prior system calibration; (3) an informed selection of centralized or distributed processing, recognizing the strengths and weaknesses in the two approaches [10].

With current technology, there are fundamental performance limits in target tracking. High-rate contact data due to sub-band processing (or to a large field of sensors) does not always lead to enhanced performance: ultimately, improved detection probabilities come at the cost of increased false object rates [11].

Numerous approaches to multi-sensor fusion and tracking are documented in the literature; see [9] and references therein. These approaches fall into a number of classes as summarized below.

Scan-based vs. batch processing. The former processes contact files in increasing time sequence, and produces a current surveillance output at each step; the latter operates on the full set of contact files, and then provides a single consolidated surveillance output. Scan-based processing is applicable to real-time surveillance tasks, while batch processing is applicable to operations such as area clearance, where results are required after a specified time interval. Contact-based vs. unified detection and tracking. The former takes detection-level data resulting from sequential contact-level processing, while the latter includes techniques such as trackbefore-detect, whereby detection and tracking do not constitute separate functions. The latter approach, often referred to as Bayesian tracking, is less mature and not amenable to networkbased processing, though it does hold high potential for dim targets in heavy clutter. Hard vs. soft data association. Hard association includes immediate or deferred (multihypothesis) assignment of contacts to tracks; soft association involves weighed assignments of multiple contacts across multiple tracks, to reflect data association uncertainties. Kalman filtering based vs. numerical filtering. Associated contacts must be processed to result in a sequence of numerical estimates of target location and velocity. Analytical filtering methods include extensions of the well known Kalman filter: the Extended Kalman filter (EKF), interactive multiple-model (IMM) filtering, and variable-structure IMM (VS-IMM) filtering. Numerical filtering includes grid-based approaches as well as recently developed particle filtering techniques.

Over the past several years, the NATO Undersea Research Centre (NURC) has investigated multi-sensor tracking as part of its research into multistatic active sonar undersea surveillance networks, with fixed sonobuoy networks and multiple towed arrays. Our

The following upgrades to the DMHT tracker have been implemented to address our maritime surveillance objectives:

Positional measurement data. Rather than (time, bearing) contact data, here we have (x, y) measurement data, and, correspondingly, simpler recursive filtering (linear Kalman filtering);

Track-breakage functionality. Network AIS track data contains anomalies that result from erroneous use of the technology, e.g. improperly entering ship identification information in the AIS unit. In almost all cases, these are unintentional errors that nonetheless lead to erratic track data. AIS tracks fed through the DMHT module will be essentially unaltered, except that track updates that fall outside a validation gate lead to track breakage.

Target-specific coordinate system. Each target has its own coordinate system centered on the current positional estimate for the target, and tangent to a spherical earth model. As the target moves, its coordinate system slowly translates and rotates. The modification allows for consolidated global tracking.

Future required upgrades include the following:

AIS-specific logic. There is a need to manage effectively AIS track IDs. We may possibly be required to reason over AIS tracks that reappear after long periods of time.

Advanced filtering algorithms. Information on coastal constraints and traffic flow patterns should be exploited in the prediction stage of recursive filtering computations.

Data registration. There is a need for automated alignment of imagery-based detections for subsequent fusion with coastal-radar and AIS tracks.

The input data is illustrated in Figure 4; radar contact are in blue, AIS track are in red, and the corresponding surveillance regions are denoted by blue/red boxes. Note the northward-heading track that lacks a corresponding AIS track, and the portion of track for the maneuvering target that lacks a corresponding AIS track.

4 Simulation-Based Multisensor Fusion In this section, we provide a simulation-based example that provides initial validation of multi-sensor fusion as a foundation for anomaly detection capabilities, in support of maritime surveillance. We consider the following scenario :

10hr surveillance period, (200km)2 surveillance area;

20 targets with mixed stochastic/deterministic trajectories (i.e. piecewise-constant speed and heading, impacted by random perturbations as well as target feedback that seeks to keep the vessel on course – details on this motion model are in [12]);

2 anomalous tracks (one with AIS turned off, the other with AIS temporarily off to coincide with target maneuvers);

Two areas of radar coverage and a single area of AIS coverage.

Fig. 4. The input to the DMHT: radar contacts and AIS tracks. The result of DMHT processing is shown in Figure 5. As the input data is quite “clean”, the tracker has no difficulty in generating exactly tracks corresponding to all targets in the field of coverage of the sensors. Additionally, note that the fused track that lacks corresponding AIS information, as well as the fused track that executes maneuvers while invisible to AIS, are clear examples of anomalous behavior.

anomalies

Note that neither anomaly can be detected with AIS-only analysis. The simulated data is as follows:

Contact-level radar data with 2min scan rate, 50 contacts per scan (false contacts are uniformly distributed in measurement space), PD=0.75, and 100m standard deviation of positional measurement error in both x and y dimensions; AIS tracks with 1min update rate and 10m standard deviation of positional measurement error in both x and y dimensions;

The DMHT processing scheme includes radar tracking followed by scan-based track fusion of radar and AIS tracks. As noted in Section 3, scan-based fusion is essential in real-time applications.

Fig. 5. The DMHT output: fused tracks that provide a basis for effective anomaly detection. While the example illustrated here is quite simple by design, it serves the objectives of validating the proper execution of the DMHT algorithm, as well as identifying its importance to an effective maritime surveillance.

5 Experimentation Activities in Support of Multisensor Surveillance As previously discussed, an important asset for the maritime surveillance sensor suite is track data based on an AIS network. In Figure 6, we illustrate a 60hr window of AIS network data. The network is continuing to increase in size as additional receivers are brought online. With the existing receiver suite, we are already faced with voluminous civilian traffic and the anomalies associated with misuse of AIS equipment.

Fig. 9. Processed AIS network data (English channel).

6 Conclusions

There is still a need further to refine AIS track information, to reflect coastal constraints. Figure 10 illustrates the infeasibility that some processed AIS tracks still exhibit.

Fig. 6. An example of raw AIS network data. Applying our recently developed track breakage functionality to this data, we obtain the result shown in Figure 7. A further example is illustrated in Figures 8-9.

Fig. 10. Processed AIS tracks and coastal constraints. Recently, we have achieved real-time connection of the NURC AIS receiver to our global experimental AIS network. This is illustrated in Figure 11.

Fig. 7. Processed AIS network data.

Fig. 11. Display of live AIS traffic.

Fig. 8. Raw AIS network data (English channel).

Having live network AIS data provides a starting point for the development of multi-sensor experimental datasets. Currently, NURC is pursuing two activities for multi-sensor experimentation in 2007. We now describe these in turn.

6.1 Local coastal surveillance We plan soon to have coastal radar data networked to NURC. This data will provide overlapping coverage with our local AIS data, allowing for immediate testing of our DMHT capabilities for local surveillance. Additionally, there is the potential to include IR-based detections,

leveraging collaborative efforts with the Italian Navy research lab in Livorno (MARITELERADAR).

6.2 Surveillance in open waters We will leverage future NURC sea-trial activities in the Mediterranean, adding ship-based AIS and radar data to AIS network data. The dataset will be augmented with SAR-based imagery acquisitions through the NURC satellite ground station.

7 Conclusions This paper summarizes initial research activities conducted at NURC in support of maritime surveillance. In particular, recent global maritime surveillance activities have focused on extensive analysis of AIS data. Multi-sensor fusion and tracking can significantly enrich the consolidated surveillance picture, and provide an effective basis for anomaly detection processing. To support this fusion and tracking work, NURC has leveraged its DMHT tracker that was originally designed for undersea surveillance networks. Relevant enhancements to this technology are identified in this paper. We illustrate the effectiveness of the enhanced DMHT algorithms with simulated test data. Next, we illustrate its performance in processing the AIS track data to remove overt kinematic anomalies. The task of combined live network AIS track data with additional sensor feeds will be the focus of our forthcoming experimentation, both in the local La Spezia area as well as in Mediterranean-based sea-trial activities. In addition to these experimental activities, we will focus on DMHT algorithmic refinements (AIS-specific logic, advanced filtering, and data registration). A longer-term effort will focus on the development of effective anomaly-detection algorithms. This is a particular challenge, particularly to detect criminal activities hidden in benign traffic patterns. We expect that kinematic anomaly detection algorithms will need to reason over multi-vessel behavior, and will provide only a partial solution that complements web-based inference engines.

8 Acknowledgements The authors thank the NATO Undersea Research Centre (NURC) for providing full funding support for the research activities documented in this paper.

9 References [1] M. Balci and R. Pegg, Towards Global Maritime Domain Awareness – “Recent Developments and Challenges”, in Proceedings of the 9th International Conference on Information Fusion, July 2006, Florence, Italy.

[2] B. Tetreault, Use of the Automatic Identification System (AIS) for Maritime Domain Awareness, in Proceedings of OCEANS 2005, September 2005, Washington D.C., USA. [3] A. Baldacci and G. Haralabus, Signal Processing for an Active Sonar System suitable for Advanced Sensor Technology Applications and Environmental Adaptation Schemes, submitted to the 14th European Signal Processing Conference, September 2006, Florence, Italy. [4] S. Coraluppi and C. Carthel, Distributed Tracking in Multistatic Sonar, IEEE Transactions on Aerospace and Electronic Systems, vol. 41(3), July 2005. [5] S. Coraluppi, C. Carthel, D. Grimmett, and O. Gerard, Multistatic Sonar Tracking: Algorithmic Advances and Sea Trial Evaluation, to appear in U.S. Navy Journal of Underwater Acoustics. [6] N. Bomberger, B. Rhodes, M. Seibert, and A. Waxman, Associative Learning of Vessel Motion Patterns for Maritime Situation Awareness, in Proceedings of the 9th International Conference on Information Fusion, July 2006, Florence, Italy. [7] A. Smith, Use of Coastal AIS to Infer Vessel Behavior in the North Sea, in Proceedings of MAST 2006, September 2006, Nice, France. [8] M. Seibert et al, SeeCoast Port Surveillance, in Proceedings of SPIE Vol. 6204: Photonics for Harbor and Port Security II, April 2006, Orlando FL, USA. [9] S. Blackman and R. Popoli, Design and Analysis of Modern Tracking Systems, Artech House, 1999. [10] S. Coraluppi, Advances in Multisensor Tracker Modeling, in Proceedings of the 2006 SPIE Conference on Signal and Data Processing of Small Targets, April 2006, Orlando FL, USA. [11] S. Coraluppi and C. Carthel, Performance Limits of Real-Time Contact-Based Tracking, to appear in the Proceedings of OCEANS 2007, June 2007, Aberdeen, Scotland. [12] S. Coraluppi and C. Carthel, Advances in Active Sonar Tracking, in Proceedings of the 14th European Signal Processing Conference, September 2006, Florence, Italy.