Minimum Cost Multi-Way Data Association for Optimizing Multitarget

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,

VOL. 37,

NO. 3,

MARCH 2015

611

Minimum Cost Multi-Way Data Association for Optimizing Multitarget Tracking of Interacting Objects Chiwoo Park, Member, IEEE, Taylor J. Woehl, James E. Evans, and Nigel D. Browning Abstract—This paper presents a general formulation for a minimum cost data association problem which associates data features via one-to-one, m-to-one and one-to-n links with minimum total cost of the links. A motivating example is a problem of tracking multiple interacting nanoparticles imaged on video frames, where particles can aggregate into one particle or a particle can be split into multiple particles. Many existing multitarget tracking methods are capable of tracking non-interacting targets or tracking interacting targets of restricted degrees of interactions. The proposed formulation solves a multitarget tracking problem for general degrees of inter-object interactions. The formulation is in the form of a binary integer programming problem. We propose a polynomial time solution approach that can obtain a good relaxation solution of the binary integer programming, so the approach can be applied for multitarget tracking problems of a moderate size (for hundreds of targets over tens of time frames). The resulting solution is always integral and obtains a better duality gap than the simple linear relaxation solution of the corresponding problem. The proposed method was validated through applications to simulated multitarget tracking problems and a real multitarget tracking problem. Index Terms—Data association, binary integer programming, decomposition, lagrange dual relaxation

Ç 1

INTRODUCTION

A

multitarget tracking problem involves finding movement trajectories of multiple targets from their visual measurements in a video sequence. The problem is typically composed of two parts: a data association problem and a state estimation problem. The data association problem is to associate the visual measurements to one of multiple targets. Once the data association part is solved, estimating the trajectories of a target is reduced to a simple state estimation problem from the measurements associated with the target. In this paper, we study a new data association formulation and solution method for tracking a varying number of interacting targets that undergo birth, death, splitting and merging. We assume the visual measurements from a video sequence are noisy, so there might be false and missed measurements. Our motivating example is the problem of tracking interacting nanoparticles in a sequence of electron microscope images to understand the effect of their interactions on the growth mechanism. Previous publications in this area were able to manually





C. Park is with the Department of Industrial and Manufacturing Engineering, Florida State University, Tallahassee, FL 32310. E-mail: [email protected]. T. J. Woehl is with the Division of Material Science and Engineering, Ames National Laboratory, Ames, IA. E-mail: [email protected]. J. E. Evans is with the Department of Environmental and Molecular Sciences, Pacific Northwest National Laboratory, Richland, WA 9935. E-mail: [email protected]. N. D. Browning is with the Department of Fundamental and Computational Sciences, Pacific Northwest National Laboratory, Richland, WA 99352. E-mail: [email protected].

Manuscript received 24 Apr. 2013; revised 16 May 2014; accepted 28 July 2014. Date of publication 6 Aug. 2014; date of current version 13 Feb. 2015. Recommended for acceptance by F. Fleuret. For information on obtaining reprints of this article, please send e-mail to: [email protected], and reference the Digital Object Identifier below. Digital Object Identifier no. 10.1109/TPAMI.2014.2346202

identify various nanoparticle interactions in the acquired data [1], [2]. However an automatic and quantitative assessment of the data removes potential human bias that would lead to more accurate analysis and better statistics. Besides the motivating application, there are many other applications of the multitarget tracking of interacting targets, including radar target tracking in military [3], sonar tracking in the ocean [4], monitoring interaction of cells and pathogens [5], monitoring human hands’ interactions in retail activity [6], and general multiple tracking for multiple overlaying targets [7], [8], [9], [10]. Many conventional approaches for the multitarget tracking problem are based on the assumption that one target generates a single visual measurement for each time frame in a video sequence; i.e., one-to-one mapping in the targetmeasurement association. The approaches include joint probabilistic data association filters (JPDAF) [11] and multiple hypothesis tracking (MHT) [12]. However, when there are target splits and mergers, the conventional approaches do not work because the splitting of a target generates more than one measurement for that target, and vice versa for the merger of multiple targets, therefore one-to-many or manyto-one associations are possible. Hence, for the rest of this paper we focus our attention on the data association solution that can handle one-to-many or many-to-one mappings in the target-measurement association; for a comprehensive review on other popular multitarget tracking methods and data association methods refer to the cited review papers and books [13], [14]. Existing data association solutions can handle only oneto-two or two-to-one cases, which are largely grouped into two categories: a sampling-based approach and a global optimization approach. The sampling-based approach is a modified version of the MHT or the JPDAF, which samples

0162-8828 ß 2014 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

612

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,

a number of possible data associations with certain posterior weights and approximates the posterior probability of the data associations with the sample distribution. The posterior probability can be naturally incorporated into some filtering algorithms such as particle filter and Kalman filter for the final state estimation. The methods under this category differ in terms of how they populate data associations. Khan et al. [15], [16] used a Rao-Blackwellized MCMC sampler. This method does not handle birth and death events, and the likelihood function of a data association does not factor in the relationship between the merging events and the corresponding previous target locations; i.e., it would allow for merging of targets that were far apart. These problems were addressed by Storlie et al. [17] and its variant [18]. Yu and Medioni [19] proposed a computationally efficient MCMC sampler that sampled data associations by limiting the space of possible data associations based on the spatial and temporal proximity of data. Genovesio et al. [20] revised a soft-gating idea, which was originally proposed in the JPDAF, to populate potential split and merger events and maximize the joint probability of the events for the final solution of a data association problem. A similar idea was also proposed by Gennari and Hager [21]. The major limitation of the sampling-based approach is that it is not computationally efficient when the number of targets is more than ten, since the number of possible data associations to be sampled exponentially increases as the number of targets increases. It is not surprising that all of the aforementioned papers were applied for tracking less than 10 targets (in most cases, less than five targets); the combinatorial complexity of the sampling-based approach might be reduced by using the dynamic programming approach for some special cases [22]. In addition, the sampling approach sequentially solves the data association problem for each time frame of data, so the resulting solution is often locally optimal within a few neighboring time frames. Although a globally optimal solution is theoretically achievable with an infinite number of samples (by the theory of MCMC sampling), the solution is mostly locally optimal with a finite sample size. Therefore, the approach has certain limitation in handling long-term data associations, such as handling long-term target occlusions [23]. The optimization approach formulates the data association problem as a minimum cost network flow problem and pursues a global optimal solution or some sub-optimal solutions. The approach recasts a problem of assigning the measurements of each video time frame to a target into a problem of associating measurements from different time frames, where a group of the associated measurements are considered to be from the same target. The main advantage of the optimization approach over the sampling approach is that it puts all data associations for all multiple time frames of a video together as a single problem, so it generally provides a better solution for identifying data associations spanning over longer time frames, e.g. long-term occlusion. The minimum cost network flow problem is typically formulated as a binary linear integer programming problem and solved by either a linear programming (LP) relaxation [24], [25] or by some heuristic algorithms such as successive shortest path algorithms [26], [27]. These methods impose a constraint that one measurement is associated with only

VOL. 37,

NO. 3, MARCH 2015

one target. Some methods proposed to use some postprocessing steps for tracking one-to-two merges or two-to-one splits [28], [29]. More general methods of handling one-tomany or many-to-one associations are based on a binary quadratic programming formulation of a data association problem [30], [31]. The binary quadratic programming formulation can be solved by a general integer programming solver such as the branch and bound algorithm [30]. However, the branch and bound algorithm still has a combinatorial computation complexity, so using the algorithm is not appropriate for big data association problems. A linear relaxation reformulation of the binary quadratic programming was studied [31]. The binary quadratic programming formulations can only handle one-to-two or two-to-one data associations. In addition, they have a certain limitation in handling some cases of one-to-two or two-to-one associations; see our discussion in Section 2.1. Considering the computational limitation of the current sampling-based data association methods, we follow the optimization-based approach. Since the current optimization-based approach is rather limited to two-to-one or one-to-two associations, we propose a new optimization formulation for a general data association which can handle M-to-one or one-to-M (M  2) for multitarget tracking of interacting objects. We call this approach an M-way data association, and propose a new sub-optimal algorithm to solve the problem. The proposed algorithm obtains an integral solution, and the duality gap of the proposed algorithm (i.e., the gap from the obtained optimal value to the true optimal value) is less than or at least equal to that of the linear programming relaxation of the original problem. The proposed algorithm is based on a certain decomposition of the original problem into sub problem (SPs), so the computational complexity of the algorithm is smaller than solving the big original problem. This paper is organized as follows. In Section 2, we explain a new formulation for a two-way data association (i.e. M ¼ 2) with comparison to the existing formulation in literature. In Section 3, we extend the formulation for a general M-way data association and propose our sub-optimal solution approach for solving the formulation. Section 4 validates the proposed formulation and the solution method with simulated datasets and real datasets. Section 5 concludes the paper.

2

TWO-WAY DATA ASSOCIATION PROBLEM

We consider a multitarget tracking problem with a video sequence of multiple time frames. We suppose that there is an existing target detector to detect visual measurements of targets from each time frame of the video. The detector may generate false measurements which are not associated with any targets or may have erroneous detections for some measurements. In this section, we assume that two targets can merge, or one target can split into two targets. The data association part of the multitarget tracking is that the measurements are associated over multiple time frames via one-to-one, one-to-two, and two-to-one links, where a one-to-two link and a two-to-one link imply a split of a target and a merge of two targets, respectively. We call this a two-way data association problem. Once the

PARK ET AL.: MINIMUM COST MULTI-WAY DATA ASSOCIATION FOR OPTIMIZING MULTITARGET TRACKING OF INTERACTING...

613

Fig. 1. Cost function for exemplary two-way data associations.

association is complete, each set of the associated measurements becomes a target trajectory. Suppose that there are N measurements. The two-way data association problem is represented on a graph G ¼ ðV; EÞ, where V ¼ f1; . . . ; Ng is a set of indices for the N measurements. We let E be a set of an ordered twotuple of V (i.e. directed edges), which represents a set of all possible one-to-one associations among the elements in V . To consider the birth and the death of targets, we augment V to V 0 ¼ V [ fN þ 1; N þ 2g and augment E to E 0 ¼E [ Eb [ Ed , where Eb ¼ fðN þ 1; vÞ; v 2 V g implies measurement v is generated by a newly born target and Ed ¼ fðv; Nþ2Þ; v 2 V g implies measurement v is the measurement of a target that will disappear in the next time frame. We assume that a faulty measurement or a cluttered measurement stays in a video only for a very short time; i.e. every measurement v 2 V associated with only N þ 1 and N þ 2 (i.e. birth and immediate death) is regarded as a faulty measurement. In addition, we allow every measurement to be associated with other measurements from the next multiple frames, which naturally implies that an association over a certain gap (due to missing measurements) can be handled. A one-to-two association consists of two one-to-one associations starting at the same measurement v 2 V . Let OðvÞ  E be a set of the outgoing edges from v 2 V . The set of all one-to-two data associations starting at v 2 V is a collection of all two-combinations of OðvÞ. Similarly, the set of all two-to-one data associations is a collection of all twocombinations of IðvÞ, where IðvÞ is a set of the incoming edges to v 2 V . The solution of a two-way data association problem on ðV; EÞ is defined as a set of activated one-to-one, one-to-two, and two-to-one data associations. Let ze denote a binary indicator of activation for e 2 E; ze ¼ 1, which implies that e is activated. We also introduce birth/death ðdÞ binary indicators xðbÞ v and xv , which represent the activation of ðN þ 1; vÞ and ðv; N þ 2Þ, respectively. We use a product of two indicators ze1 ze2 to represent the activation of twoway data associations for e1; e2 2 IðvÞ or e1; e2 2 OðvÞ. A two-way data association of ðV 0 ; E 0 Þ is defined by the indicaðdÞ tors, Z ¼ fze g [ fxðbÞ v ; xv g, which satisfies In-degree constraint: 1  xðbÞ v þ

X

ze  2 for v 2 V

e2IðvÞ

Out-degree constraint: 1  xðdÞ v þ

X

ze  2 for v 2 V

e2OðvÞ

Binary constraint: ze 2 f0; 1g for e 2 E:

We want to find Z that minimizes a certain cost function. Let fðeÞ be the cost of an one-to-one association e 2 E and let fðe1; e2Þ be the cost of a two-way association related to e1 and e2 for e1; e2 2 IðvÞ or e1; e2 2 OðvÞ and v 2 V . Please note that fðe1; e2Þ 6¼ fðe1Þ þ fðe2Þ. The costs for the birth/ death of a trajectory are the constants f1 and f2 , based on the assumption that the birth and death rates are constant in time; one might consider f1 and f2 as functions of time when the birth and death rates vary over time. The total cost of the associations Z is the summation of the costs for all activated associations, which includes the following terms: C0. Cost for birth/death of a trajectory: for constants f1 and f2 , X  ðdÞ f1  xðbÞ : v þ f2  x v v2V

C1. Cost term for one-to-one association: X ce ze : e2E

C2. Cost term for one-to-two association: X X ce1;e2 ze1 ze2 : v2V e16¼e22OðvÞ

C3. Cost term for two-to-one association: X X ce1;e2 ze1 ze2 : v2V e16¼e22IðvÞ

C4. Cost term for the case that a 2-to-1 association and an 1to-2 association share a common edge e: X X de;e1;e2 ze ze1 ze2 e2E

e12OðsðeÞÞ; e22IðtðeÞÞ;e16¼e6¼e2;

where sðeÞ is the start node of an association e and tðeÞ is the end node of an association e; we used simpler notation, ce1;e2 ¼ fðe1; e2Þ  fðe1Þ  fðe2Þ, de;e1;e2 ¼ fðe; e1; e2Þ fðe; e1Þ  fðe; e2Þ þ fðeÞ, and ce ¼ fðeÞ. Please note that the C4 term is added to consider the case of a one-to-two association ðe1 2 OðsðeÞÞÞ and a two-to-one association ðe2 2 IðtðeÞÞÞ sharing a common one-to-one association ðe 2 V Þ. For example, please consider a cost function of a simple two-way data association in Fig. 1. In the figure, three one-to-one associations, e; e1 and e2, are activated. The three activated one-to-one associations compose a one-to-two association (related to e and e1) and a two-to-one association (related to e and e2). We denote the

614

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,

corresponding cost by fðe; e1; e2Þ. The cost may not be equal to fðe; e1Þ þ fðe; e2Þ for the case where the activation of the split (e and e1) affects the cost for the merge (e and e2), e.g., the merger’s appearance depends on whether the split is activated or not. If C4 term was not in the objective function, the resulting cost would be fðe1; e2Þ þ fðe2; e3Þ  fðeÞ. To cancel out fðe1; e2Þ þ fðe2; e3Þ  fðeÞ and add fðe; e1; e2Þ, we add C4. Please note that a one-to-two association and a two-to-one association cannot share more than one edge. This can be shown by contradiction. Suppose that a one-totwo association (starting at v1) and a two-to-one association (ending at v2) can share more than one edge. Let e and e0 denote two different shared edges. Since they are shared, both e and e0 should have starting node v1 and ending node v2, i.e e and e0 are identical. This is a contradiction. Finding Z that minimizes the total cost leads to a binary cubic programming problem. We can linearize the problem by replacing quadratic and cubic terms with linear terms. Let us introduce ze1;e2 :¼ ze1 ze2 , which is a binary variable because a product of two binary variables is also a binary variable. The new binary variable is constrained by the values of ze1 and ze2 such that ze1;e2 2 Cðze1 ; ze2 Þ,

VOL. 37,

NO. 3, MARCH 2015

ðdÞ Let z1 ¼ ðxðbÞ v ; xv ; ze Þ, z 21 ¼ ðze1;e2 ; e1; e2 2 IðvÞ; e1 6¼ e2; v 2 V Þ, z 22 ¼ ðze1;e2 ; e1; e2 2 OðvÞ; e1 6¼ e2; v 2 V Þ, and y22 ¼ ðye;e1;e2 ; e1 2 OðsðeÞÞ; e2 2 IðtðeÞÞ; e 2 E; e 6¼ e1; e 6¼ e2Þ be column vectors of the decision variables with corresponding cost coefficient vectors c1 , c21 , c22 and d22 . The two-way data association problem can be put in the following algebraic form,

Min c T1 z1 þ c T21 z 21 þ cT22 z22 þ dT22 y22 A1 z1  b1 A21 z1 þ B 21 z 21  b21

(2a) (2b) (2c)

A 22 z 1 þ C 21 z22  b 22 P 22 z21 þ Q22 z22 þ y 22  11 P 22 z 21  y 22  0

(2d) (2e0 ) (2e00 )

Q 22 z 22  y 22  0

(2e00 )

z1 2 Bp1 ; z 21 2 Bp21 ; z22 2 Bp22 ; y 22 2 Bq22 where constraints (2b) is the algebraic form of (1b) and (1a), constraints (2c), (2d), (2e’) and (2e”) are the algebraic forms of (1c), (1d), (1e’), and (1e”) respectively, and Bp is a set of all possible p-dimensional binary values.

Cðx; yÞ :¼ fz 2 f0; 1g:z  x; z  y; x þ y  1  zg: In the same way, we also introduce ye;e1;e2 ¼ ze ze1 ze2 ¼ ze ze1 ze ze2 ¼ ze;e1 ze;e2 with ye;e1;e2 2 Cðze;e1 ; ze;e2 Þ. The linearized version of the problem is X X ðdÞ Min ff1  xðbÞ ce ze v þ f2  xv g þ v2V

þ

þ

X

2 4

e2E

X

ce1;e2 ze1;e2 þ

v2V

e16¼e22IðvÞ

X

X

X

3 ce1;e2 ze1;e2 5

e16¼e22OðvÞ

de;e1;e2 ye;e1;e2

e2E e12OðsðeÞÞ;e22IðtðeÞÞ; e16¼e6¼e2

1  xðbÞ v þ

X

ze  2

for v 2 V

(1a)

ze  2

for v 2 V

(1b)

e2IðvÞ

1  xðdÞ v þ

X e2OðvÞ

2 Cðze1 ; ze2 Þ

for e1 6¼ e2 2 IðvÞ; v 2 V

(1c)

e1;e2

2 Cðze1 ; ze2 Þ

for e1 6¼ e2 2 OðvÞ; v 2 V

(1d)

ye;e1;e2 2 Cðze;e1 ; ze;e2 Þ for e1 2 OðsðeÞÞ; e2 2 IðtðeÞÞ; e 2 E; e 6¼ e1; e 6¼ e2

(1e)

ðdÞ xðbÞ v ; xv ; ze ; ze1;e2 ; ye;e1;e2

2 f0; 1g: (1f)  0, constraint (1e) can be simplified to

ye;e1;e2 2 fy 2 f0; 1g : ze;e1 þ ze;e2  1  yg;

ð1e0 Þ

because the upper bound of ye;e1;e2 in Cðze;e1 ; ze;e2 Þ, i.e., ye;e1;e2  ze;e1 and ye;e1;e2  ze;e2 , are unnecessary since the minimization of the total cost will bind ye;e1;e2 ’s value to its minimum. For the same reason, when de;e1;e2 < 0, constraint (1e) can be simplified to ye;e1;e2 2 fy 2 f0; 1g:y  ze;e1 ; y  ze;e2 g:

v1;v22V

þ

e1;e2

When de;e1;e2

2.1 Comparison to an Alternative Formulation in Literature Henriques et. al. proposed a different optimization-based formulation of a two-way data association problem [31]. In the formulation, zv1:v2 is introduced to represent a one-toone association from v1 2 V to v2 2 V with the association cost cv1:v2 . In addition to that, an artificial binary variable, zv:v1;v2 , is introduced to represent a one-to-two association from v to v1 and v2 with the association cost cv:v1;v2 , and another binary variable, zv1;v2:v , is introduced to represent a two-to-one association from v1 and v2 to v with the association cost cv1;v2:v . The constraint on zv1:v2 , zv:v1;v2 and zv1;v2:v is that each data feature v 2 V is involved in either one of oneto-one, one-to-two, or two-to-one association. This problem is formally expressed by a binary integer linear programming problem, X X Min cv1:v2 zv1:v2 þ cv1;v2:v zv1;v2:v

ð1e00 Þ

v;v1;v22V

X

cv:v1;v2 zv:v1;v2

v;v1;v22V

þ

X

X

zv:v2 þ

v22V

zv;v1:v2 ¼ 1

X

zv:v1;v2

v1;v22V

(3a)

for v 2 V

v1;v22V

X

zv1:v þ

v12V

X

zv1:v;v2

v1;v22V

þ

X

zv1;v2:v ¼ 1

for v 2 V

(3b)

v1;v22V

zv1:v2 2 f0; 1g; zv:v1;v2 2 f0; 1g; zv1;v2:v 2 f0; 1g; where (3a) ensures that each data feature v 2 V is a start node for either one of one-to-one, one-to-two or two-to-one association, and ð3bÞ ensures that each data feature v 2 V is an end node for either one of one-to-one, one-to-two, or two-to-one association. This formulation has an advantage in computation because the constraint coefficient matrix of the formulation is totally unimodular so we can simply

PARK ET AL.: MINIMUM COST MULTI-WAY DATA ASSOCIATION FOR OPTIMIZING MULTITARGET TRACKING OF INTERACTING...

solve its linear relaxation to obtain the global optimal solution. However, the formulation has some limitation in its modeling power of expressing complex multi-way data associations. The first limitation is that one data feature cannot be associated with both one-to-two and two-to-one associations. Examples for a data feature being involved with both of the associations can often be found in a multitarget tracking problem of nanoparticles. For example, please see Fig. 1. In the figure it is assumed that there are two particles detected from each time frame of a video sequence: particles xi and xj at time t and particles xk and xl at time t þ 1. The objective of this multitarget tracking is to associate the particles over two time frames for tracking trajectories of the particles. In the example, suppose that particle xi is split into two particles at time t þ 1, one of them a smaller particle xk , and the other a new particle xl jointly with xj . The optimal tracking should associate xi to xk and xl and associate xi and xj to xl , i.e. xi is involved in both a one-to-two association and a two-to-one association. In other words, the solution of the multitarget tracking problem should be zi:k;l ¼ 1 and zi;j:l ¼ 1. However, the solution violates (3a) and (3b), because the left hand sides of (3a) for a node i and (3b) for a node l are zi:k;l þ zi;j:l , which is equal to 2, while the solution violates none of constraints (2b), (2c), (2d), (2e’), and (2e’’) in our formulation. This example demonstrates that our formulation is more general than the Henriques’ formulation [31]. In addition, the Henriques’ formulation is only applicable for a two-way data association. Our formulation is extended to a general M-way data association; cf. Section 3.

3

EXTENSION TO M-WAY DATA ASSOCIATION PROBLEM

Consider a generalization of our two-way data association formulation to an M-way data association formulation. We first introduce a few new notations for simpler representation of the generalization. Let Cm ðEÞ denote a set of all possible m-combinations of edges in an arbitrary set E, and its element is denoted by feg :¼ fe1; . . . ; emg 2 Cm ðEÞ, which is a subset of size m out of E. We also introduce short notations Om ðvÞ :¼ Cm ðOðvÞÞ, Im ðvÞ :¼ Cm ðIðvÞÞ, O0m ðeÞ :¼ fe[ 0 fe0 g:fe0 g 2 Cm1 ðOðsðeÞÞneÞg, and Im ðeÞ :¼ fe [ fe0 g:fe0 g 2 Qm Cm1 ðIðtðeÞÞneÞg. Let zfeg ¼ i¼1 zei and let cfeg be the associated cost for zfeg , which is defined by cfeg ¼ fðfegÞ 

m 1 X

X

(4)

cfe1g :

j¼1 fe1g2Cj ðfegÞ

Intuitively, cfeg includes the cost terms for m-way association fðfegÞ and the negatives of the costs (cost cancellation) for j-way associations which are subsets of feg (j < m). The objective function of an M-way data association problem is the total cost for all data associations. The objective function consists of the following cost terms: C0.

Cost for birth/death: X  ðdÞ f1  xðbÞ : v þ f2  xv v2V

C1.

Cost terms for one-to-one associations:

P

e2E ce ze :

C2. C3. C4.

615

P terms for one-to-m associations: v2V feg2Om ðvÞ cfeg zfeg for 1 < m 3), faulty measurements, birth events and death events.

4.1 Simulation We first use different sets of simulated video frame data. When we simulate video frame data, we recorded the true associations among target detections, which serve as the ground truth solution of the data association problem. The ground truth is then compared with the data association obtained by each of the compared methods. In addition, the computation times were recorded. A system of N particles randomly moving and interacting in a bounded two dimensional space was simulated by a Monte Carlo simulation; please see Fig. 2 for examples. All particles have circular outlines of random radii; the radius of particle i (denoted by ri ) is sampled from log ri N ð0:5; 0:1Þ. At time t ¼ 1 of the simulation, all particles are equally spaced over R2 with the distances to their closest neighbors equal to g, and each particle moves independently by the Brownian motion with randomly changing centroid position ðxi;t ; yi;t Þ at time t with xi;tþ1 j xi;t

N ðxi;t ; s 2 Þ and yi;tþ1 j yi;t N ðyi;t ; s 2 Þ. We fixed s ¼ 0:3; please note that the movement speed of particles still varies over time with a fixed s. We indirectly change the degree of inter-particle interactions by changing g; as g decreases, particles locate closer together at the beginning of the random process, and thus the random Brownian motion of particles may cause some of the particles to merge to other particles or a merged particle may be split again into individual ones. We tried different values of g from 1.5 to 4.0 with step size 0.5. The smallest possible g is 1.5; otherwise almost all particles are touching from time 1, and for g larger than 4.0, few particles are interacting. To simulate faulty measurements, we randomly generated small spherical particles

VOL. 37,

NO. 3, MARCH 2015

(with its radius equal to one) at random time frames and locations, which immediately disappear one frame after their appearances. To simulate the births of new targets, we randomly generated new particles at random locations and random times; we determined the times between two successive births by sampling from an exponential distribution of mean 2. If a particle moves out of a predefined two-dimensional bounded box, we regard the particle as dead or disappeared. The number of time frames in a simulated video is fixed to 20, and the number of particles per frame varies depending on the number of faulty measurements and the numbers of birth and death events, with the average being 81. A problem of tracking particles and their interactions in the simulation requires solution of a data association problem of 1620 (¼81*20) measurements on average, with the number of simple associations among the measurements equal to 19*81*81.

4.1.1 Implementation Details The cost of associating two particle measurements is computed based on the Hausdorff distance between the interior point sets of the two measurements, each set of which is a set of point coordinates in R2 inside the outline of an object. Let x i denote the interior point set for particle measurement i. The cost of associating measurement i with measurement xi ; x j Þ, where j, i.e. ce , for e ¼ ði; jÞ, is equal to dH ðx dH ðX; Y Þ ¼ maxf supx2X inf y2Y jjx  yjj; supy2Y inf x2X jjx yjj g and jjx  yjj is the euclidean L2 norm in R2 . The cost of a multi-way data association feg is 0 fðfegÞ ¼ dH @

[

i2sðe0 Þ;e0 2feg

xi ;

[

1 xj A

j2tðe0 Þ;e0 2feg

which is the Hausdorff distance between a union of all starting nodes of the edges in feg and a union of all ending nodes of the edges in feg. The cost coefficient f1 and f2 for birth and death events is uniformly set to one, which is close to 3s. With the cost coefficients, we solved a data association problem by our M-way data association method with M ¼ 2 and M ¼ 3. We used MATLAB and the MATLAB interface to CPLEX for implementing our method. The implementation was run on a personal computer with Inteli7 CPU and 8 GB memory.

4.1.2 Discussion on Results For each g 2 f1:5; 2:0; 2:5; 3:0; 3:5; 4:0g, we generated 40 different simulated datasets as described in Section 4.1. For each dataset, we ran our method with M ¼ 2, our method with M ¼ 3, Henriques’ method [31], Jaqaman’s method [28] and MCMC Data Association [19]. We computed the false positive rates and false negative rates for each of overall, 1to-2 split, 2-to-1 merge, 1-to-3 split, 3-to-1 merge, 1-to-m split (m > 3), n-to-1 merge (n > 3), faulty measurements, birth events, and death events. We averaged the rates over the 40 simulation runs for each g. Table 1 summarizes the averages. Our method performed better when M ¼ 3 than when M ¼ 2 in terms of the overall FN rate. The performance gap becomes larger as g decreases. When g is small, more 1-to-m splits and n-to-1 merges occur, and our method with M ¼ 3 more effectively handled the splits and the merges. However,

PARK ET AL.: MINIMUM COST MULTI-WAY DATA ASSOCIATION FOR OPTIMIZING MULTITARGET TRACKING OF INTERACTING...

619

TABLE 1 Simulation Data—Data Association Errors of Our Method with M ¼ 2 and M ¼ 3, Henriques’ Method [31], Jaqaman’s Method [28], and MCMC Data Association [19]. g ¼ 1:5 Overall 1-to-2 2-to-1 1-to-3 3-to-1 1-to-m n-to-1 Faulty measurements Birth Death g ¼ 2:0

Our method (M ¼ 3)

Total

2090.9000 196.6500 235.3500 53.7750 66.2250 14.3000 27.6000 14.8750 9.6000 10.4000 Total

FN

FP

FN

FP

0.0935 0.3956 0.3701 0.4644 0.4915 1.0000 1.0000 0.5681 0.1380 0.3269

0.0628 0.2567 0.2736 0.4887 0.5877 0.0000 0.0000 0.1569 0.4626 0.6353

0.1165 0.3692 0.3537 1.0000 1.0000 1.0000 1.0000 0.3899 0.1016 0.2452

0.0519 0.3564 0.4036 0.0000 0.0000 0.0000 0.0000 0.1906 0.5556 0.7315

Our method (M ¼ 3) FN

Overall 1-to-2 2-to-1 1-to-3 3-to-1 1-to-m n-to-1 Faulty measurements Birth Death g ¼ 2:5 Overall 1-to-2 2-to-1 1-to-3 3-to-1 1-to-m n-to-1 Faulty measurements Birth Death g ¼ 3:0 Overall 1-to-2 2-to-1 1-to-3 3-to-1 1-to-m n-to-1 Faulty measurements Birth Death g ¼ 3:5 Overall 1-to-2 2-to-1 1-to-3 3-to-1 1-to-m n-to-1 Faulty measurements Birth Death g ¼ 4:0 Overall 1-to-2 2-to-1 1-to-3 3-to-1 1-to-m n-to-1 Faulty measurements Birth Death

2622.3500 152.1500 189.3500 17.2500 22.5750 1.8500 3.4750 15.4500 9.3000 9.5750 Total 2832.5500 89.8500 115.1000 5.4000 8.0250 0.4000 0.6000 15.3000 9.2500 10.7500 Total 2987.9000 49.5000 67.0500 1.6500 2.3250 0.0000 0.1000 14.8000 9.5750 9.3000 Total 3023.1000 25.1500 35.2000 0.2250 0.5250 0.0000 0.0000 15.0500 9.6250 10.1000 Total 3073.6500 12.4500 19.5500 0.0000 0.0000 0.0000 0.0000 15.2500 9.6000 9.4750

Our method (M ¼ 2)

0.0235 0.2152 0.1933 0.2522 0.2492 1.0000 1.0000 0.3285 0.0968 0.1775

Our method (M ¼ 2)

Henriques FN 0.1357 0.3860 0.3644 1.0000 1.0000 1.0000 1.0000 0.3782 0.0964 0.2356

FP 0.0493 0.3503 0.3965 0.0000 0.0000 0.0000 0.0000 0.3309 0.7209 0.8236

Jaqaman FN 0.1707 0.5230 0.4353 1.0000 1.0000 1.0000 1.0000 0.7664 0.4167 0.5601

Henriques

FP 0.0916 0.4766 0.4816 0.0000 0.0000 0.0000 0.0000 0.6559 0.6393 0.6914

Jaqaman

MCMC-DA FN 0.2849 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0773 0.0469 0.0553

FP 0.0268 1.0000 1.0000 1.0000 0.0000 0.0000 0.0000 0.6839 0.8742 0.9034

MCMC-DA

FP

FN

FP

FN

FP

FN

FP

FN

FP

0.0167 0.1165 0.1178 0.3175 0.3705 0.0000 0.0000 0.0326 0.2538 0.3879

0.0318 0.2106 0.1893 1.0000 1.0000 1.0000 1.0000 0.2298 0.0726 0.1332

0.0140 0.1751 0.1906 0.0000 0.0000 0.0000 0.0000 0.0442 0.3868 0.5475

0.0421 0.2244 0.2010 1.0000 1.0000 1.0000 1.0000 0.2233 0.0726 0.1305

0.0133 0.1699 0.1872 0.0000 0.0000 0.0000 0.0000 0.1257 0.6235 0.7170

0.0785 0.3076 0.2509 1.0000 1.0000 1.0000 1.0000 0.5712 0.2285 0.3786

0.0311 0.2804 0.2705 0.0000 0.0000 0.0000 0.0000 0.0804 0.2439 0.3568

0.1536 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0421 0.0188 0.0313

0.0112 1.0000 1.0000 1.0000 0.0000 0.0000 0.0000 0.6044 0.8475 0.8925

Our method (M ¼ 3)

Our method (M ¼ 2)

Henriques

Jaqaman

MCMC-DA

FN

FP

FN

FP

FN

FP

FN

FP

FN

FP

0.0067 0.1219 0.0969 0.1111 0.1028 1.0000 1.0000 0.1601 0.0243 0.0837

0.0051 0.0505 0.0618 0.2809 0.2558 0.0000 0.0000 0.0096 0.1502 0.1564

0.0095 0.1185 0.0956 1.0000 1.0000 1.0000 1.0000 0.0882 0.0135 0.0651

0.0039 0.0817 0.1106 0.0000 0.0000 0.0000 0.0000 0.0089 0.2224 0.2852

0.0127 0.1247 0.1012 1.0000 1.0000 1.0000 1.0000 0.0882 0.0135 0.0651

0.0038 0.0796 0.1097 0.0000 0.0000 0.0000 0.0000 0.0296 0.3976 0.4469

0.0254 0.1714 0.1295 1.0000 1.0000 1.0000 1.0000 0.2892 0.1108 0.2558

0.0127 0.1832 0.1971 0.0000 0.0000 0.0000 0.0000 0.0109 0.1320 0.1950

0.0883 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0310 0.0135 0.0209

0.0065 1.0000 1.0000 1.0000 0.0000 0.0000 0.0000 0.5407 0.7809 0.8364

Our method (M ¼ 3)

Our method (M ¼ 2)

Henriques

Jaqaman

MCMC-DA

FN

FP

FN

FP

FN

FP

FN

FP

FN

FP

0.0017 0.0556 0.0477 0.0000 0.0968 0.0000 1.0000 0.0777 0.0209 0.0403

0.0017 0.0209 0.0420 0.3125 0.1515 0.0000 0.0000 0.0055 0.0406 0.0746

0.0026 0.0545 0.0470 1.0000 1.0000 0.0000 1.0000 0.0507 0.0000 0.0376

0.0013 0.0410 0.0644 0.0000 0.0000 0.0000 0.0000 0.0071 0.0726 0.1322

0.0045 0.0616 0.0507 1.0000 1.0000 0.0000 1.0000 0.0507 0.0000 0.0376

0.0012 0.0383 0.0640 0.0000 0.0000 0.0000 0.0000 0.0277 0.2426 0.3017

0.0114 0.0838 0.0641 1.0000 1.0000 0.0000 1.0000 0.1622 0.0470 0.1532

0.0055 0.1337 0.1532 0.0000 0.0000 0.0000 0.0000 0.0045 0.0605 0.0992

0.0529 1.0000 1.0000 1.0000 1.0000 0.0000 1.0000 0.0304 0.0078 0.0296

0.0054 1.0000 1.0000 1.0000 0.0000 0.0000 0.0000 0.4693 0.6917 0.8041

Our method (M ¼ 3)

Our method (M ¼ 2)

Henriques

Jaqaman

MCMC-DA

FN

FP

FN

FP

FN

FP

FN

FP

FN

FP

0.0005 0.0219 0.0270 0.0000 0.0000 0.0000 0.0000 0.0216 0.0000 0.0124

0.0006 0.0180 0.0284 0.2500 0.2222 0.0000 0.0000 0.0000 0.0100 0.0290

0.0007 0.0219 0.0270 1.0000 1.0000 0.0000 0.0000 0.0150 0.0000 0.0124

0.0005 0.0219 0.0379 0.0000 0.0000 0.0000 0.0000 0.0017 0.0175 0.0453

0.0014 0.0258 0.0270 1.0000 1.0000 0.0000 0.0000 0.0150 0.0000 0.0124

0.0005 0.0220 0.0379 0.0000 0.0000 0.0000 0.0000 0.0017 0.0989 0.1361

0.0033 0.0457 0.0355 1.0000 1.0000 0.0000 0.0000 0.0282 0.0026 0.0668

0.0017 0.0419 0.1101 0.0000 0.0000 0.0000 0.0000 0.0000 0.0071 0.0179

0.0329 1.0000 1.0000 1.0000 1.0000 0.0000 0.0000 0.0465 0.0234 0.0371

0.0052 1.0000 1.0000 1.0000 0.0000 0.0000 0.0000 0.3902 0.6078 0.7194

Our method (M ¼ 3)

Our method (M ¼ 2)

Henriques

Jaqaman

MCMC-DA

FN

FP

FN

FP

FN

FP

FN

FP

FN

FP

0.0001 0.0161 0.0077 0.0000 0.0000 0.0000 0.0000 0.0066 0.0000 0.0132

0.0002 0.0041 0.0152 0.0000 1.0000 0.0000 0.0000 0.0000 0.0026 0.0027

0.0001 0.0161 0.0077 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0132

0.0001 0.0041 0.0152 0.0000 0.0000 0.0000 0.0000 0.0000 0.0026 0.0027

0.0005 0.0161 0.0077 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0132

0.0001 0.0041 0.0152 0.0000 0.0000 0.0000 0.0000 0.0033 0.0471 0.0579

0.0011 0.0241 0.0153 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0396

0.0004 0.0000 0.0587 0.0000 0.0000 0.0000 0.0000 0.0000 0.0025 0.0082

0.0222 1.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0328 0.0156 0.0290

0.0047 1.0000 1.0000 1.0000 1.0000 0.0000 0.0000 0.3608 0.5013 0.6530

Each of the false positive rates (FP) and false negative rates (FN) is the average error rate averaged over 40 simulation runs simulated with the same g.

620

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,

VOL. 37,

NO. 3, MARCH 2015

Fig. 3. Convergence of our Lagrange dual solution; all curves are rescaled to range from 0 to 1 in the vertical axis.

the FP rate was slightly higher when M ¼ 3, which is mainly due to its high FP rates for 1-to-3 and 3-to-1 cases. Our method outperformed the three directly relevant state-of-the-art methods, Henriques’ method [31], Jaqaman’s method [28], and MCMC Data Association [19] in terms of the overall FN and FP rates. Our method with M ¼ 2 was more accurate than the three state-of-the-art methods in terms of 1-to-2 and 2-to-1 associations, which is a major reason why our method outperformed the state-of-the-arts in the overall FP and FN rates. Our method can handle more general cases of 1-to-2 and 2-to-1 associations as we discussed in Section 2.1, so our method can be more accurate in 1-to-2 and 2-to-1 associations. In the meanwhile, the MCMC Data Association (MCMC-DA) handles split and merged measurements based on the Reversible Jump MCMC, but the acceptance rate of the MCMC step was low. Therefore it was not effective in handling merged and split measurements. The Jaqaman’s method had the FP rates comparable to our method, but significantly higher overall FN rates. The computation time for our method with M ¼ 2 for one simulation case was 51.6 seconds, which was slower than Henriques’ (38.6 seconds) and Jaqaman’s (25.1 seconds) but was faster than MCMC-DA (79.6 seconds). The computation time for our method with M ¼ 3 was 1,734 seconds. In summary, our method with M ¼ 2 has better accuracy than the state-of-the-art methods but has comparable computation times as the state-of-the-art methods. Our method with M ¼ 3 is slow but is able to handle higher degrees of data associations, which is effective in improving the overall accuracy.

4.1.3 Convergence of Our Lagrange Dual Solution Our formulation for multi-way data association is solved by an iterative algorithm of the two steps described in Section 3.2. In this section, we show the convergence behavior of the iterative steps by example. We simply chose one simulation case to plot the iteration versus the duality gap; see Fig. 3. The duality gap initially decreases monotonically fast but slows in the later stage. The convergence was reached faster when M ¼ 2 (350 iterations) than when M ¼ 3 (500 iterations).

As the duality gap decreases, the accuracy of the multiway data association improves. Our algorithm improved accuracy of the data association as follows. The (SP1) part of the algorithm solves the optimization,   (SP1) z 1 ¼ arg min cT1; z 1 ; A1 z 1  b1 ; z1  1 ; where z1 is a vector of binary variables indicating active oneto-one data associations; please note that every m-to-one or one-to-m association is a combination of m one-to-one associations. At iteration 1, all Lagrange multipliers are initialP P ized to zero, so c1; ¼ ðcc1  m ATm1 l m1  n ATn2 l n2 ÞT ¼ c1 . Since the cost coefficients c1 are all positive, each target is prone to be involved in only one-to-one associations for minimizing the total cost. However, after several iterations of the updates on l m1 ’s and l n2 ’s by solving (MP), some cost coefficients adjusted (cc1; ) become negative so some additional one-to-one associations are activated, which activates more multi-way associations. Therefore iteratively solving (SP1) and (SP2) with (MP) can be interpreted as an iterative process of updating l m1 ’s and l n2 ’s for identifying and activating cost-saving multi-way data associations. For example, see the FN rate curve in Fig. 3. The FN rate monotonically decreases over iterations, which implies that the number of correctly detected 2-to-1 associations increases.

4.2 Application to Real Electron Microscope Data In this section, we apply our proposed method to real video frame data for tracking nanoparticles and discuss the result. Solution phase silver nanoparticle growth was imaged by in situ transmission electron microscopy for 89 seconds with a frame rate of 1 frame per second (see [2] for details on the imaging technique). We took a subset of the video frames from 40 seconds to 80 seconds with 0.5 frame per second (total 20 time frames) as nanoparticles were not actively changing or interacting for the first 40 seconds. On average, 280 silver nanoparticles exist per time frame. Each individual nanoparticle grows with time, and some merge into larger aggregates or split into smaller particles. Understanding how the particles interact and grow is very important to understand the nanoparticle growth mechanism.

PARK ET AL.: MINIMUM COST MULTI-WAY DATA ASSOCIATION FOR OPTIMIZING MULTITARGET TRACKING OF INTERACTING...

Fig. 4. Sample nanoparticle trajectories.

In the video frames, nanoparticles are not moving fast but they are actively interacting and growing. We first applied a simple image thresholding method on the video frames to extract the outlines and the interiors of nanoparticles. We applied the proposed method with M ¼ 3, Henriques’ method [31], Jaqaman’s method [28] and MCMC Data Association [19] to the extracted nanoparticle outlines to associate the individual extractions over different time frames to track nanoparticles and their interactions. We randomly picked 18 nanoparticles and manually annotated their movements and interactions to create a ground truth; see Fig. 4 for some examples. We compared the results from the compared methods with the ground truth in terms of the false positive rates and false negative rates for each of overall, 1-to-2 split, 2-to-1 merge, 1-to-3 split, 3-to-1 merge, 1-to-m split (m > 3), n-to-1 merge (n > 3), faulty measurements, birth events and death events. Table 2 summarizes the comparison. Similar to the results of our simulation study, our method was more accurate than the state-of-the-art methods in terms of 1-to-2 splits and 2-to-1 merges, so it was better in the overall performance.

4.3 Application to People Tracking Data Although our development is motivated by a problem of tracking nanoparticles, it is also applicable for many other multitarget tracking problems. In this section, we

621

demonstrate the application of the proposed approach for a problem of pedestrian tracking with the PETS 2009 datasets, S2.L1 and S2.L2. Since our focus is on the data association, we used the existing target detection results (available at http://iris.usc.edu/people/yangbo/downloads.html for S2.L1 and http://www.gris.informatik.tu-darmstadt.de/ ~aandriye/data.html#detectionsfor S2.L2), and present how our method and the state-of-the-art methods perform the M-way data association with the same detection results. The existing target detections are in the form of bounding boxes for potential foreground regions occupied by people, which include faulty detections and miss some bounding boxes for some people in the video. In addition, when a group of people walk very close with high visual overlaps, the detection results have one bounding box for the group, so there are merged measurements. Each of the bounding boxes defines one data feature indexed by numbers in V ¼ f1; 2; . . . ; Ng. The cost of associating two bounding boxes i and j for people tracking was studied a lot in literature. We simply use one from the MCMC Data Association [19]. The cost of associating one-to-one association e ¼ ði; jÞ is defined as  ðmÞ   ðaÞ  fðeÞ ¼ logN dij ; s a  logN dij ; dt s m ; where N ðd; sÞ denotes the probability density function of a zero-mean normal distribution with standard deviation s, ðaÞ

dij is the difference in appearance (the symmetric Kullback-Leibler Distance (KL) between the histogram-based descriptors of foregrounds covered by boxes i and j), and ðmÞ

dij is the L2 distance between the centers of the two bounding boxes; s a and s m were estimated using a training dataset (a subset of ground truth data). The cost of one-to-n association in between bounding box i and other n bounding boxes is defined by one-to-one association cost between box i and an artificial bounding box created as an union of the n bounding boxes. The cost of m-to-one association and the cost for one m-to-one and one-to-n associations sharing one common edge are also defined in a similar way. We compared the results from each method with the ground truth in terms of false positive and false negative rates for PETS S2.L1 and PETS S2.L2. The PETS S2.L1 dataset includes 794 frames of images showing 19 different people’s

TABLE 2 Real Microscope Data—Data Association Errors of Our Method with M ¼ 3, Henriques’ Method [31], Jaqaman’s Method [28], and MCMC Data Association [19] Types of Data Association 1-to-1 1-to-2 2-to-1 1-to-3 3-to-1 1-to-m n-to-1 Faulty measurements Birth Death

Total 1208.0000 200.0000 228.0000 0.0000 1.0000 0.0000 0.0000 0.0000 3.0000 0.0000

Our method (M ¼ 3) FN

FP

0.0331 0.0200 0.0351 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.0379 0.1091 0.0984 0.0000 0.0000 0.0000 0.0000 0.0004 0.0000 1.0000

Henriques FN 0.0861 0.1000 0.1140 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000

FP 0.0612 0.1667 0.1368 0.0000 0.0000 0.0000 0.0000 0.0013 0.7500 1.0000

Jaqaman FN 0.4912 0.9600 0.8947 0.0000 1.0000 0.0000 0.0000 0.0000 0.3333 0.0000

FP 0.2614 0.8000 0.5862 0.0000 0.0000 0.0000 0.0000 0.0004 0.8333 1.0000

MCMC-DA FN 0.5066 1.0000 0.9912 0.0000 1.0000 0.0000 0.0000 0.0000 0.3333 0.0000

FP 0.2857 1.0000 0.9091 1.0000 1.0000 0.0000 0.0000 0.0134 0.9524 1.0000

622

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,

VOL. 37,

NO. 3, MARCH 2015

TABLE 3 PETS.S2.L1 Data—Data Association Errors of Our Method with M ¼ 3, Henriques’ Method [31], Jaqaman’s Method [28], and MCMC Data Association [19] Types of Data Association 1-to-1 1-to-2 2-to-1 1-to-3 3-to-1 1-to-m n-to-1 Faulty measurements Birth Death

Total 8940.0000 82.0000 80.0000 0.0000 0.0000 0.0000 0.0000 147.0000 15.0000 12.0000

Our method (M ¼ 3) FP

0.0114 0.0488 0.0750 0.0000 0.0000 0.0000 0.0000 0.2313 0.4000 0.3333

0.0250 0.5000 0.5132 0.0000 0.0000 0.0000 0.0000 0.0000 0.0323 0.0690

movement, which is relatively less crowded with three to seven people shown per time frame. Table 3 summarizes the complexity of the dataset and the performance of the compared methods. The PETS S2.L1 dataset does not have 1-to-m or n-to-1 associations. The Jaqaman’s method is slightly better than our method in the FP rates for two-way data associations. However, our method outperforms the state-of-the-art methods in terms of the accuracy in the overall association and it is better than the Henrique’s method and Jaqaman’s method in terms of FN rates of two way data association. The PETS.S2.L2 dataset is more crowded with about 40 people per time frame and it also includes more interactions and overlaps among people. Table 4 summarizes the complexity of the dataset and the performance of the compared methods. Our method outperformed the state-of-the-art methods in the overall accuracy, which is mainly due to its higher accuracy in multi-way data associations.

5

Henriques

FN

CONCLUSION

We proposed a binary integer programming formulation for a data association problem which pursues the minimum cost data associations among target measurements via one-to-one, one-to-m, and m-to-one associations. The formulation for the multi-way data associations among the measurements was shown useful to solve multitarget tracking problems for interacting objects with general

FN 0.0154 0.0488 0.1000 0.0000 0.0000 0.0000 0.0000 0.2313 0.4000 0.3333

FP 0.0243 0.4868 0.5200 0.0000 0.0000 0.0000 0.0000 0.0000 0.3448 0.3818

Jaqaman FN 0.0206 0.0976 0.0750 0.0000 0.0000 0.0000 0.0000 0.1701 0.4000 0.3333

FP 0.0118 0.3833 0.3833 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

MCMC-DA FN 0.1573 1.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0884 0.4000 0.2500

FP 0.0718 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000 0.5113 0.8394 0.8953

degrees of interactions. We showed that the new formulation is better than the existing formulations for data associations in terms of generality of expressing multi-way object interactions. To the best of our knowledge, the formulation is among the few trials that formulates a general multi-way data association problem. The new formulation is our first contribution. The binary integer programming problem is hard to solve in general. We proposed a procedure for obtaining a Lagrangian dual relaxation solution for the binary integer programming problem. The procedure guarantees that its solution is integer-valued, and the duality gap of the solution is smaller than that of the simple LP relaxation solution. In addition, the procedure converges within polynomial time. The proposed formulation and solution procedure were applied to different sets of simulation studies and one real electron microscopy image data set, which showed the superiority of our approach to three state-or-the-art methods in terms of data association accuracy. We also applied our approach to real video frame data for pedestrian datasets, which demonstrates the applicability of our method to more general multitarget tracking problem.

ACKNOWLEDGMENTS The authors thank Abhishek Shrivastava for useful discussions. We would like to acknowledge support for this

TABLE 4 PETS.S2.L2 Data—Data Association Errors of Our Method with M ¼ 3, Henriques’ Method [31], Jaqaman’s Method [28], and MCMC Data Association [19] Types of Data Association 1-to-1 1-to-2 2-to-1 1-to-3 3-to-1 1-to-m n-to-1 Faulty measurements Birth Death

Total 17324.0000 444.0000 464.0000 33.0000 27.0000 4.0000 0.0000 72.0000 10.0000 8.0000

Our method (M ¼ 3) FN

FP

0.0130 0.2072 0.2328 0.4455 0.4556 1.0000 0.0000 0.8750 0.3000 0.2500

0.0173 0.3226 0.2992 0.5059 0.5895 0.0000 0.0000 0.0000 0.6667 0.7027

Henriques FN 0.1369 0.2658 0.2716 1.0000 1.0000 1.0000 0.0000 0.9167 0.3000 0.2500

FP 0.1040 0.6479 0.6397 0.0000 0.0000 0.0000 0.0000 0.5709 0.9456 0.9512

Jaqaman FN 0.1474 0.2658 0.2672 1.0000 1.0000 1.0000 0.0000 0.7917 0.6000 0.5000

FP 0.0221 0.3849 0.3585 0.0000 0.0000 0.0000 0.0000 0.0000 0.7500 0.7895

MCMC-DA FN 0.0921 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000 0.1250 0.0000 0.0000

FP 0.0274 1.0000 1.0000 1.0000 1.0000 0.0000 0.0000 0.6034 0.9364 0.9261

PARK ET AL.: MINIMUM COST MULTI-WAY DATA ASSOCIATION FOR OPTIMIZING MULTITARGET TRACKING OF INTERACTING...

project. Park is supported by the FSU COFRS 032968, the Ralph E. Powe Junior Faculty Enhancement Award, and NSF-CMMI-1334012. Evans and Browning acknowledge NIH funding support from grant no. 5RC1GM091755. Browning also acknowledges DOE funding support from grant no. DE-FG02-03ER46057. Support for Woehl was provided by the UC Lab Fee Program and the UC Academic Senate. A portion of this work is part of the Chemical Imaging Initiative at Pacific Northwest National Laboratory (PNNL) under Contract DE-AC05-76RL01830. It was conducted under the Laboratory Directed Research and Development Program at PNNL.

REFERENCES [1]

[2]

[3]

[4] [5]

[6]

[7] [8] [9] [10]

[11] [12] [13] [14] [15] [16]

[17] [18]

J. Evans, K. Jungjohann, N. Browning, and I. Arslan, “Controlled growth of nanoparticles from solution with in situ liquid transmission electron microscopy,” Nano Lett., vol. 11, no. 7, pp. 2809– 2813, 2011. T. Woehl, J. Evans, I. Arslan, W. Ristenpart, and N. Browning, “Direct in situ determination of the mechanisms controlling nanoparticle nucleation and growth,” ACS Nano, vol. 6, no. 10, pp. 8599–8610, 2012. K. Chang and Y. Bar-Shalom, “Joint probabilistic data association for multitarget tracking with possibly unresolved measurements and maneuvers,” IEEE Trans. Autom. Control, vol. 29, no. 7, pp. 585–594, Jul. 1984. T. Fortmann, Y. Bar-Shalom, and M. Scheffe, “Sonar tracking of multiple targets using joint probabilistic data association,” IEEE J. Oceanic Eng., vol. 8, no. 3, pp. 173–184, Jul. 1983. A. Genovesio, B. Zhang, and J. Olivo-Marin, “Interacting multiple model based method to track moving fluorescent biological spots,” in Proc. IEEE Int. Symp. Biomed. Imaging: Nano Macro, 2004, pp. 1239–1242. H. Trinh, Q. Fan, P. Gabbur, and S. Pankanti, “Hand tracking by binary quadratic programming and its application to retail activity recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2012, pp. 1902–1909. B. Bose, X. Wang, and E. Grimson, “Multi-class object tracking algorithm that handles fragmentation and grouping,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2007, pp. 1–8. S.-W. Joo and R. Chellappa, “A multiple-hypothesis approach for multiobject visual tracking,” IEEE Trans. Image Process., vol. 16, no. 11, pp. 2849–2854, Nov. 2007. C. Huang, B. Wu, and R. Nevatia, “Robust object tracking by hierarchical association of detection responses,” in Proc. Eur. Conf. Comput. Vis., 2008, pp. 788–801. J. Xing, H. Ai, and S. Lao, “Multi-object tracking through occlusions by local tracklets filtering and global tracklets association with detection responses,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2009, pp. 1200–1207. C. Rasmussen and G. D. Hager, “Probabilistic data association methods for tracking complex visual objects,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 23, no. 6, pp. 560–576, Jun. 2001. D. B. Reid, “An algorithm for tracking multiple targets,” IEEE Trans. Autom. Control, vol. 24, no. 6, pp. 843–854, Dec. 1979. A. Yilmaz, O. Javed, and M. Shah, “Object tracking: A survey,” ACM Comput. Surv., vol. 38, no. 4, p. 13, 2006. I. J. Cox, “A review of statistical data association techniques for motion correspondence,” Int. J. Comput. Vis., vol. 10, no. 1, pp. 53–66, 1993. Z. Khan, T. Balch, and F. Dellaert, “Multitarget tracking with split and merged measurements,” in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., 2005, pp. 605–610. Z. Khan, T. Balch, and F. Dellaert, “MCMC data association and sparse factorization updating for real time multitarget tracking with merged and multiple measurements,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 28, no. 12, pp. 1960–1972, Dec. 2006. C. B. Storlie, T. C. Lee, J. Hannig, and D. Nychka, “Tracking of multiple merging and splitting targets: A statistical perspective,” Statistica Sinica, vol. 19, no. 1, pp. 1–52, 2009. A. Makris and C. Prieur, “Bayesian multiple hypothesis tracking of merging and splitting targets,” IEEE Trans. Geoscience Remote Sensing, vol. 52, no. 12, pp. 7684–7694, 2014.

623

[19] Q. Yu and G. Medioni, “Multiple-target tracking by spatiotemporal Monte Carlo Markov chain data association,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no. 12, pp. 2196–2210, Dec. 2009. [20] A. Genovesio and J.-C. Olivo-Marin, “Split and merge data association filter for dense multi-target tracking,” in Proc. 17th Int. Conf. Pattern Recognit., 2004, vol. 4, pp. 677–680. [21] G. Gennari and G. D. Hager, “Probabilistic data association methods in visual tracking of groups,” in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., 2004, pp. II-876–II-881. [22] P. Kumar, S. Ranganath, K. Sengupta, and H. Weimin, “Cooperative multitarget tracking with efficient split and merge handling,” IEEE Trans. Circuits Syst. Video Technol., vol. 16, no. 12, pp. 1477–1490, Dec. 2006. [23] A. A. Perera, C. Srinivas, A. Hoogs, G. Brooksby, and W. Hu, “Multi-object tracking through simultaneous long occlusions and split-merge conditions,” in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., 2006, vol. 1, pp. 666–673. [24] H. Jiang, S. Fels, and J. J. Little, “A linear programming approach for multiple object tracking,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2007, pp. 1–8. [25] L. Zhang, Y. Li, and R. Nevatia, “Global data association for multiobject tracking using network flows,” in Proc. IEEE Conf. Comput. Vis. and Pattern Recognit., 2008, pp. 1–8. [26] H. Pirsiavash, D. Ramanan, and C. C. Fowlkes, “Globally-optimal greedy algorithms for tracking a variable number of objects,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2011, pp. 1201– 1208. [27] J. Berclaz, F. Fleuret, E. Turetken, and P. Fua, “Multiple object tracking using k-shortest paths optimization,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 33, no. 9, pp. 1806–1819, Sep. 2011. [28] K. Jaqaman, D. Loerke, M. Mettlen, H. Kuwata, S. Grinstein, S. L. Schmid, and G. Danuser, “Robust single-particle tracking in livecell time-lapse sequences,” Nat. Methods, vol. 5, no. 8, pp. 695–702, 2008. [29] A. Serge, N. Bertaux, H. Rigneault, and D. Marguet, “Dynamic multiple-target tracing to probe spatiotemporal cartography of cell membranes,” Nat. Methods, vol. 5, no. 8, pp. 687–694, 2008. [30] W. Choi and S. Savarese, “A unified framework for multi-target tracking and collective activity recognition,” in Proc. Eur. Conf. Comput. Vis., 2012, pp. 215–230. [31] J. Henriques, R. Caseiro, and J. Batista, “Globally optimal solution to multi-object tracking with merged measurements,” in Proc. IEEE Int. Conf. Comput. Vis., 2011, pp. 2470–2477. [32] A. Schrijver, Combinatorial Optimization: Polyhedra and Efficiency, vol. 24. New York, NY, USA: Springer, 2003. [33] G. Sierksma, Linear and Integer Programming: Theory and Practice, vol. 245. Boca Raton, FL, USA: CRC, 2001. [34] P. Pardalos and S. Vavasis, “Quadratic programming with one negative eigenvalue is NP-hard,” J. Global Optim., vol. 1, no. 1, pp. 15–22, 1991. [35] G. Nemhauser and L. Wolsey, Integer and Combinatorial Optimization. New York, NY, USA: Wiley, 1988. [36] B. Polyak, Introduction to Optimization. New York, NY, USA: Optimization Software, Publications Division, 1987. 3 [37] K. Anstreicher, “Linear programming in Oðlnn n LÞ operations,” SIAM J. Optim., vol. 9, no. 4, pp. 803–812, 1999. Chiwoo Park (M’ 08, M’12) received the BS degree from Seoul National University and the PhD degree from Texas A&M University, all in industrial engineering. He is currently an assistant professor in the Department of Industrial and Manufacturing Engineering at Florida A&M and Florida State University. His research interests include data mining with its applications to nanoimaging and remote sensing. He received the best student paper award at the fourth annual IEEE Conference on Automation Science and Engineering and the Ralph E. Powe Junior Faculty Enhancement Award from the Oak Ridge Associated Universities. He is a member of the IEEE.

624

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,

Taylor J. Woehl received the BS in ceramic engineering at Missouri University of Science and Technology in 2005, and the PhD in chemical engineering at the University of California, Davis in 2013. He is currently an assistant scientist III at Ames Laboratory in the Division of Materials Science and Engineering. His research interests include utilizing in situ electron and optical microscopy to study the nucleation and growth mechanisms of solution-phase nanoparticles and colloidal crystals with applications in biomineralization and functional nanostructures. He received the 2014 Zuhair H. Munir Award for best doctoral dissertation in the College of Engineering at University of California Davis. James E. Evans received the PhD degree in biochemistry and molecular biology at the University of California, Davis in 2007. He is currently a scientist lll supporting the microscopy capability at the Pacific Northwest National Laboratory. He currently oversees the installation and operational testing of a liquid helium, cryo-transmission electron microscope, and a new dynamic transmission electron microscope. He received the University of California Davis C4E Business Development Fellowship in 2006 and the Microscopy and Microanalysis Presidential Award in 2007.

VOL. 37,

NO. 3, MARCH 2015

Nigel D. Browning received the undergraduate degree in physics from the University of Reading, United Kingdom, and the PhD in physics from the University of Cambridge, United Kingdom. He is currently a chief scientist for the Chemical Imaging Initiative and a Laboratory fellow at the Pacific Northwest National Laboratory. His research focuses on the development of new methods in electron microscopy for high spatial, temporal and spectroscopic resolution analysis of engineering and biological structures. He received the Burton Award from the Microscopy Society of America in 2002, the Coble Award from the American Ceramic Society in 2003, and was a corecipient of R&D 100 and Nano 50 Awards in 2008 and a Microscopy Today Innovation Award in 2010 for the development of the DTEM. " For more information on this or any other computing topic, please visit our Digital Library at www.computer.org/publications/dlib.