Multiple-camera Tracking of Rigid Objects - Semantic Scholar

INSTITUT NATIONAL DE RECHERCHE EN INFORMATIQUE ET EN AUTOMATIQUE

Multiple-camera Tracking of Rigid Objects Frédérick Martin — Radu Horaud

N° 4268 September 2001

ISSN 0249-6399

ISRN INRIA/RR--4268--FR+ENG

THÈME 3

apport de recherche

Multiple- amera Tra king of Rigid Obje ts

Frédéri k Martin , Radu Horaud

Thème 3  Intera tion homme-ma hine, images, données, onnaissan es Projet Movi Rapport de re her he n° 4268  September 2001  30 pages

Abstra t:

In this paper we des ribe a method for tra king rigid obje ts using one or several

ameras. The tra king pro ess onsists of aligning a 3-D model representation of an obje t with image ontours by measuring and minimizing the image error between predi ted model points and image ontours. The tra ker behaves like a visual servo loop where the internal and external amera parameters are updated at ea h new image a quisition. We study in detail the Ja obian matrix asso iated with this minimization pro ess in the presen e of both point-to-point and point-to- ontour mat hes. We establish the minimal number of mat hes that are needed as well as the singular ongurations leading to a rank-de ient Ja obian matrix. We nd a mathemati al link between the point-to-point and point-to- ontour ases. Based on this link we show that the latter has the same kind of singularities than the former. Moreover, we study multiple amera ongurations whi h optimize the robustness of the method in the presen e of single- amera singularities, bad, noisy, or missing data. Extensive experiments done with a omplex ship part and with up to three ameras validate the method. In parti ular we show that the tra ker may well be used as a amera alibration pro edure. Key-words:

obje t tra king, visual servoing, amera alibration

Work supported by the European Commission, Esprit Rea tive LTR proje t VIGOR  Visually guided robots using un alibrated ameras, grant number 26247.

Unité de recherche INRIA Rhône-Alpes 655, avenue de l’Europe, 38330 Montbonnot-St-Martin (France) Téléphone : +33 4 76 61 52 00 — Télécopie +33 4 76 61 52 52

Suivi d'objets rigides ave plusieurs améras Résumé :

Nous dé rivons une méthode de poursuite d'objets rigides utilisant une ou

plusieurs améras.

Le pro essus de poursuite onsiste en l'alignement d'un modèle 3-D

représentant l'objet ave des ontours image en mesurant et en minimisant l'erreur entre des points prédits à partir du modèle et des ontours image.

Le tra keur agit omme

une bou le d'asservissement visuel et les paramètres internes et externes des améras sont mis à jour pour haque nouvelle aquisition d'images.

Nous étudions en détail la matri e

ja obienne asso iée au pro essus de minimisation pour des appariement point-point et point ontour. Nous établissons le nombre minimal d'appariements qui sont né essaires de même que les ongurations singulières onduisant à une ja obienne qui n'est pas de rang plein. Nous trouvons un lien mathématique entre les as point-point et point- ontour. base de e lien nous montrons que les deux as ont le même type de singularités.

Sur la Nous

étudions des ongurations omportant plusieurs améras qui optimisent la robustesse de la méthode en présen e de singularités dues à une seule améra, bruit, données manquantes ou données erronnées. Des expérimentations ee tuées ave une piè e de bateau et ave trois

améras valident la méthode. En parti ulier, elle peut être utilisée omme une te hnique de

alibration. Mots- lés :

suivi d'objet, asservissement visuel, alibration de améra

Multiple- amera tra king

3

Contents 1

2

3

4

5

Introdu tion, ba kground, and approa h

4

1.1

Previous work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5

1.2

Paper organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6

Problem formulation

6

2.1

Camera model

7

2.2

The image Ja obian

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

7

2.3

Camera model errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

8

2.4

Tra king as a minimization problem

9

. . . . . . . . . . . . . . . . . . . . . . .

Point-to-point orresponden es

12

3.1

Fixed internal amera parameters . . . . . . . . . . . . . . . . . . . . . . . . .

13

3.2

Varying fo al length

13

3.3

Varying internal amera parameters

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

13

Point-to- ontour orresponden es

14

4.1

Fixed internal amera parameters . . . . . . . . . . . . . . . . . . . . . . . . .

14

4.2

Varying fo al length

16

4.3

Varying internal amera parameters

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Multiple amera tra king

5.1

6

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

17

17

The multiple amera Ja obian matrix

. . . . . . . . . . . . . . . . . . . . . .

Experiments

18

19

6.1

Finding point-to- ontour orresponden es

. . . . . . . . . . . . . . . . . . . .

19

6.2

Lo ating a stati obje t

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

19

6.3

Tra king the fo al length . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

22

6.4

Camera alibration during tra king . . . . . . . . . . . . . . . . . . . . . . . .

22

6.5

Tra king with three ameras . . . . . . . . . . . . . . . . . . . . . . . . . . . .

25

RR n° 4268

Martin & Horaud

4

7

1

Dis ussion

26

Introdu tion, ba kground, and approa h

The problem of lo alizing and tra king moving obje ts using one or several ameras has been an a tive resear h topi . Obje ts fall into several ategories, from rigid, deformable, arti ulated and rigid, to arti ulated and deformable. A ommon approa h onsists of using an obje t model and of estimating the position and orientation of this model su h that some image-based error is minimized. The image error des ribes the dis repan y between measured obje t features and predi ted model features. The ability to properly move the model su h that it optimally orresponds to the a tual obje t depends of a number of fa tors. Roughly speaking, the 3-D parameters asso iated with the obje t's behavior (motion parameters, shape deformation parameters, joint parameters for an arti ulated obje t, et .) are related to the image error by a Ja obian matrix. With only one amera there are inherent ambiguities and singularities. Indeed, one image

onguration may lead to a number of 3-D a tions and Ja obian singularities may lead to no a tion at all. Therefore, it may be advantageous to use several ameras instead of one. From a pra ti al point of view this raises the problem of dealing with multiple amera geometry. It is more tedious to alibrate a multiple- amera system and to update the alibration data than to alibrate a single amera.

Moreover, the vast majority of previous approa hes

onsider the multi- amera system as a stereo devi e requiring that ea h obje t feature is viewed in at least two images and that image-to-image feature mat hes are provided. In this paper we propose a multi- amera based method for tra king rigid obje ts. geometri model of the tra ked obje t is provided in advan e.

A

The method onsists of a

visual servoing approa h applied to the obje t model: At ea h iteration of the tra king pro ess the image error between model and obje t features allows to update the parameters asso iated with the position and orientation of the model. First, we investigate the ase of one amera. to-point tra king and point-to- ontour tra king.

We establish the link between pointWe analyse both the ases of xed and

varying amera internal parameters. We study the singularities of the Ja obian matrix and we reveal the ases where single- amera tra king annot be properly performed. Se ond, we investigate the ase of several ameras. We show how the single- amera ase

an be used to alibrate the multiple- amera layout. We establish the tra king formulation whi h allows a rigid amera layout with possibly varying internal amera parameters. This formulation onsists of running in parallel several single- amera tra kers and does not require any image-to-image orresponden e. We show how most of the single- amera singularities

an be avoided by using two or more ameras. Third, we des ribe an implementation using three ameras and a omplex rigid obje t. Ea h amera observes a dierent obje t part and therefore the system is very robust with

INRIA

Multiple- amera tra king

5

respe t to partial o

lusions simultaneously o

urring in several images and with respe t to total o

lusion of one or two ameras.

1.1

Previous work

The idea of using pose for tra king stems from the work of Lowe [14℄ who used line-segment mat hes and the Levenberg-Marquardt non-linear minimization method.

Tra king with

variable internal amera parameters has been introdu ed by Kinoshita and Degu hi [11℄ who perform visual servoing and amera alibration

simultaneously.

Espiau performed an

in-depths analysis of the onvergen e of visual servoing in the presen e of varying fo al length [7℄. Armstrong and Zisserman [1℄ proposed to predi t a model ontour in the image and to sear h around this predi tion for image points that are likely to lie on a mat hing ontour. They apply this te hnique to the ase of straight lines and use a robust method to t a line to the image points. Drummond and Cipolla [5, 3, 6℄ showed that it is possible to ast the rigid model tra king problem into a linear problem using the Lie algebra of the rigid motion - the kinemati s rew. They suggest a series of papers dealing with a omplex 3-D obje t (a ship part) and with arti ulated obje ts. The approa h of Drummond and Cipolla is interesting be ause it relies only weakly on point or line mat hes.

Instead their te hnique relies on

ontour-to-points mat hing. They show that their approa h an be used for alibrating a

amera. Other similar approa hes to tra king using an image predi tion and sear hing in a window around this predi tion an be found in [19℄ and [21℄. All these approa hes to tra king onsider only one amera and put emphasis on the data asso iation problem. Multi-view tra king has been barely investigated. There are several ways of using several ameras.

One way is to onsider the multi- amera setup as a 3-D

sensor, apture 3-D measurements, and perform the tra king dire tly in 3-D spa e, [20℄, [2℄. Another way is to use the epipolar onstraint into the tra king loop itself and Lamiroy et al. [13℄ ombine the tra king equations with the epipolar geometry onstraint. Finally, when the ameras are far apart, there are very few image-to-image orresponden es and single

amera tra kers may be performed in parallel. In this paper we derive an expli it algebrai expression for the Ja obian matrix asso iated with point-to- ontour orresponden es.

Although in the past robust results were

obtained with this type of data asso iation, none thoroughly studied the algebrai stru ture of the Ja obian, the minimal data sets ne essary to run a tra ker, and the possibly singular

ongurations whi h lead to tra king failures. Based on this analysis we are able to show that a multiple- amera approa h based on point-to- ontour orresponden es lead to a robust rigid-obje t tra ker that an deal with large o

lusions.

RR n° 4268

Martin & Horaud

6

1.2

Paper organization

The remaining of this paper is organized as follows.

Se tion 2 reviews the problem of

tra king a rigid obje t using a perspe tive amera model. The relantionship between the pose parameters (to be estimated by the tra ker) and errors asso iated with the internal amera parameters is thoroughly investigated. Rigid obje t tra king is treated as a minimization problem and expli it formulae are derived for two types of image-to-model mat hes: point-topoint and point-to- ontour assignments. Se tion 3 analyses the singularities asso iated with point-to-point assignments while se tion 4 provides a similar analysis for point-to- ontour assignments. In parti ular, singular ases are revealed espe ially when the internal amera parameters are allowed to vary. Multiple- amera tra king is studied in se tion 5 where it is shown that the singularities asso iated with one amera disappear when two or more

ameras are being used. Finally se tion 6 des ribes extensive experiments that illustrate all the variations of the tra king method.

2

Problem formulation

Without loss of generality we onsider a moving rigid obje t observed by one amera. Tra king onsists of a model being moved su h that its apparent position and orientation with respe t to the amera orresponds to the real position and orientation of the obje t. Let ⋆ model points proje ted onto the image and let s be the set of

s = (m1 , . . . , mk ) be a set of

orresponding image points. The relationship between the model velo ity and its apparent image velo ity is:

s˙ = JT where

T

is the kinemati s rew asso iated with the 3D model motion. The error between

the predi ted model position and the a tual obje t position may be measured from their image proje tions:

e = C (s − s⋆ ) Where C is ombination matrix allowing to onsider more measurements than the number of degrees of freedom (six in this ase  three for the rotational velo ity ve tor and three for the translational velo ity ve tor). A ommon hoi e whi h insures onvergen es is to set ⊤ C = J . The obje tive is to move the model su h that the image error de reases:

λJ⊤ (s − s⋆ ) = −J⊤ s˙ = −J⊤ JT whi h allows as solution:

T = −λ



J

⊤

−1

J

⊤

J

(s − s⋆ )

(1)

INRIA

Multiple- amera tra king

7

This is the basi visual servoing equation whi h is solved in order to maintain

s − s⋆

as small as possible. In order to apply this formulation to the tra king problem, one may approximate the velo ity s rew by:

x − x0 dx = dt t − t0

T =

Hen e, the equation above allows the in remental update of the pose parameters



x = x0 − λ(t − t0 ) 2.1

⊤

J

−1

J

⊤

J

x:

(s − s⋆ )

(2)

Camera model

The model-based obje t tra king just des ribed assumes a pinhole amera model, i.e., an obje t point proje ts onto an image point using the well known amera proje tion matrix:

˜ =K m where

˜ m



R

t



˜ M

is a homogeneous 3-ve tor des ribing the proje tive oordinates of an image

point, K is the matrix of internal amera parameters, R is a rotation matrix, ve tor, and

˜ M

u0

and

v0

f

(horizontal and verti al s ale fa tor, or fo al

(image oordinates of opti al enter):



f K =  0 0 2.2

t is a translation

is a homogeneous 4-ve tor des ribing the oordinates of a model point.

The internal amera parameters are length) and

(3)

0 f 0

 u0 v0  1

The image Ja obian

The 2-ve tor

˜. m des ribes the pixel oordinates asso iated with the proje tion m

By taking

the time derivatives we obtain the lassi al relationship:



      dm = Jm   dt      RR n° 4268

vx vy vz wx wy wz df dt du0 dt dv0 dt

             

(4)

Martin & Horaud

8

T ⊤ = (vx vy vz wx wy wz ) and with:    y xy x2 − z1 0 zx2 − 1 + 2 2 z z  z 2 =f −x 1 + yz2 − xy 0 − z1 zy2 z2 z

with the kinemati s rew

Jm

where

2.3

(x y z)

are the Eu lidean oordinates of point

M

x zf y zf

1 f

0

0

1 f



(5)



expressed in amera frame.

Camera model errors

The amera parameters may be unknown, partially known, or badly known. One important feature of the tra king algorithm des ribed below is that it an a

ommodate with rough estimates of these parameters whi h are updated during tra king. We analyse the ee t of badly known amera parameters onto the pose parameters. Let K be the exa t amera parameters and let R and K and with the point orresponden es

m ↔ M.

t

be the pose parameters asso iated with

ˆ be an estimation of the true amera K

Let

parameters. With these estimated parameters one an asso iate estimated pose parameters

ˆ R

and

ˆt.

The proje tion equation holds in both these ases:

h i  ˆ R ˆ ˆt M ˜ =K ˜ =K m

We have:

˜ =K m By writing

ˆ = K + dK K



−1

K

ˆR ˆ K

K

t



ˆ ˆt K



R

−1

˜ M

˜ M

and by rst-order Taylor expansion we obtain, [10℄

K

′ −1

K



df f

 =I− 0 0

df f

du0 f dv0 f

0

0

0



  = I + Kε

(6)

The pose parameters are ae ted by these internal parameter errors as follows:

ˆ = R ˆt = The estimation of

f , u0

and

v0

− Kε R t − Kε t R

is ae ted by the value of

internal parameters have equal importan e.

f.

For small fo al lengths the

For large fo al lengths, the a

ura y of

f

is

intrinsi ally more ru ial than the a

ura y of the opti al enter. If the fo al enter lies at approximatively the image enter, the value of the values of

u0

and

v0 .

f

is, on an average, 4-5 times greater than

Therefore, the ability to tra k in the presen e of variable fo al

length, estimate and orre t its value on line is an important feature.

INRIA

Multiple- amera tra king

2.4

9

Tra king as a minimization problem

We onsider now a more general tra king problem where the obje t is observed simultaneously by several ameras with possibly varying internal amera parameters. Let

q

denote the

state-ve tor asso iated with su h a setup: this ve tor en apsulates the pose parameters of the obje t with respe t to a global amera entered frame as well as the internal parameters asso iated with the amera models. Therefore, the model predi tions

s

are a fun tion of

q.

One must distinguish between two ases:

1. Tra king based on point-to-point orresponden es 2. Tra king based on point-to- ontour orresponden es

These two ases are illustrated on Figure 1. In the rst ase (point-to-point) the fun tion to be minimized may be written as the sum of the Eu lidean distan es from the predi ted ⋆ model point mi to its image mat h mi :

n

Q1 (q) =

n

1X 1X 2 kmi (q) − m⋆i k2 = e 2 i=1 2 i=1 i

Therefore, the general form of the minimization problem is, in this ase:

Q1 (q) =

1 R(q)⊤ R(q) 2

(7)

In pra ti e it is well known that it is di ult to obtain point-to-point orresponden es. A urrent approa h is to proje t a model ontour onto the image using the urrent pose and amera parameters, sear h in a dire tion normal to the predi ted model ontour, nd a

orresponding image point lying onto a ontour, and measure the distan e to this point [17℄, [4℄. For a proje ted model point

mi and a normal dire tion ni , this distan e is approximated

by, e.g., Figure 1:

⋆ di (q) = n⊤ i (q)(mi (q) − mi ) Noti e that this formula is exa t for straight-line ontours and hold only approximatively for urved ontours. The fun tion to be minimized may be written as:

n

Q2 (q) = Matrix N of size

RR n° 4268

n × 2n

1 1X 2 ⊤ d (N(q)R(q)) N(q)R(q) = 2 2 i=1 i

ontains the normals to the proje ted

(8)

Martin & Horaud

10

m * (virtual point match) image contour

d

n

e

m

projected model contour

Figure 1: A model ontour is predi ted in the image and a point lo ation

m

along this

ontour is onsidered. For point-to-point orresponden es, it is su ient to sear h for a ⋆ mat hing point m end to determine the distan e e between these points. For point-to ontour orresponden es the strategy onsists of raising a normal to the model ontour at lo ation

m,

nding the interse tion of the line supporting the normal ve tor with the

mat hing image ontour, and determining the distan e from

m

to the image ontour. The

algebrai expression of this true point-to- ontour distan e an be approximated by n · (m⋆ − m).

d =

INRIA

Multiple- amera tra king

11

n:

model ontours. Its rank is equal to



n1u  0  N =  .  .. 0

0 n2u

n1v 0 0

0

0 n2v 0

... ...

0 0

. . . nnu



0 0 nnv

The rst- (Ja obian) and se ond-order (Hessian) derivatives of

   

Q1

(9)

with respe t to

q

are:

dQ1 = J⊤ 1 (q)R(q) dq d2 Q1 d2 R = J⊤ 1 (q)J1 (q) + 2 dq dq 2 Where the Ja obian is dened by: J1

=

dR(q) dq

We derive now the Ja obian matrix for point-to- ontour orresponden es: J2

= =

d (N(q)R(q)) dq ! i=k X ∂ N(q) R(q) N(q)J1 + ∂qi i=1

Matrix N ontains the normals to the ontour.

Its derivatives may therefore be on-

sidered as se ond-order terms and therefore they an be negle ted in the expression of the Ja obian. The latter be omes:

J2

= N(q)J1 (q)

(10)

The rank of J2 is related to the rank of J1 by the formula:

r(J2 ) ≤ min(rank (J1 ), rank (N)) = min(rank (J1 ), n) be ause the rank of N is equal to

n.

(11)

This will be useful for analysing the onvergen e of

tra king. A ommon way to solve this problem is to onsider the Gauss-Newton approximation of d2 Q ⊤ dq 2 = J (q)J(q) and to perform a number of quadrati iterations. One su h

the Hessian,

quadrati iteration writes for J1 (where there is a similar expression for J2 ):

q⋄ = q −

RR n° 4268



⊤

J1

−1 ⊤ J1 (q)R(q) (q)J1 (q)

(12)

Martin & Horaud

12

In pra ti e and due to time onstraints, only one iteration is performed ea h time a new image is grabbed. Therefore, there are two situations:

ˆ

The obje t is stati with respe t to the amera in whi h ase the method behaves like a non-linear pose estimation te hnique or

ˆ

The obje t moves in whi h ase the method a ts like an obje t tra ker maintaining a pose estimation as lose as possible to a previously omputed solution.

Noti e that the Gauss-Newton iteration given by eq. (12) is stri tly equivalent to eq. (2). Both these equations require the Ja obian matrix to be of rank equal to the dimension of the state ve tor

q.

In the sequel we analyse ases for whi h the Ja obian matrix has singulari-

ties. An alternative way to avoid su h singularities is to onsider trust-region minimization methods for solving eq. (7) or eq. (8), [14, 15℄, [18℄.

3

Point-to-point orresponden es

The relationship between image measurements and in remental rigid motions allowing the model-to-obje t alignment is en apsulated in the Ja obian matri es J1 and J2 mentioned above. We distinguish the following ases:

1. Fixed internal amera parameters 2. Varying fo al length 3. Varying internal amera parameters (fo al length and opti al enter) In the ase of point-to-point orresponden es, the Ja obian is of dimensions 2n × k where n is the number of point orresponden es and k is the number of parameters to be re overed (the dimension of q ). For the general ase (varying internal parameters) and for one pointto-point mat h it has the algebrai stru ture given by eq. (5). In the ase of point-to- ontour orresponden es, the Ja obian is of dimension

n

n×k

where

is the number of point-to- ontour orresponden es. In order to study potential failures of the tra king method and, possibly, avoid them we

must analyse the rank of the Ja obian for the minimum number of points and ontours, and determine geometri ongurations for whi h the Ja obian is rank de ient.

INRIA

Multiple- amera tra king

3.1

13

Fixed internal amera parameters

There are six parameters to be estimated in this ase (three for rotation and three for translation) and hen e the least number of orresponden es is equal to 3.

This ase was

thoroughly studied within the onvergen e analysis of visual servoing methods. The Ja obian is rank de ient when (i) the image points are ollinear or when (ii) the three model points together with the opti al enter lie on a ylinder:



J1

. . .

   − z1i =  0  

xi zi2 yi zi2

0 − z1i

. . .

3.2

xi yi 2 zi

1+

yi2 zi2



 − 1+

x2i zi2

− xzi 2yi i



yi zi −xi zi

       

(13)

Varying fo al length

In this ase there are seven parameters to be estimated.

Therefore, a minimum of four

orresponden es are required. Due to the stru ture of the Ja obian matrix, a planar onguration with four (or more) points lying in a plane parallel to the image plane gives rise to a Ja obian of rank equal to 6. Indeed, for su h a onguration, the depth parameters

zi

are

the same for all the points and the third olumn is proportional to the last olumn in the Ja obian matrix:



J1

. . .

   − z1i =f  0  

0 − z1i

. . .

xi zi2 yi zi2

xi yi 2 zi

1+

yi2 zi2



 − 1+ − xzi2yi i

x2i zi2



yi zi −xi zi

xi zi f yi zi f

       

(14)

This means that it is not possible to simultenously tra k a planar obje t and estimated the fo al length when this obje t lies in a plane parallel (or almost parallel) to the image plane. This singularity extends to the ase where an arbitrary 3-D obje t is at a distan e from the

amera su h the weak-perspe tive approximation be omes valid. In fa t this onguration is the only known situation whi h does not allow the distinguish between variations in depth and variations in fo al length.

3.3

Varying internal amera parameters

In this ase there are 9 parameters to be estimated (provided that the amera's horizontalto-verti al s ale ratio is known) and hen e the Ja obian must be of rank at least equal to 9.

RR n° 4268

Martin & Horaud

14

Again, a fronto-parallel planar obje t gives rise to a rank-de ient Ja obian matrix for the same algebrai reasons as above.



J1

. . .

   − z1i =f  0  

0 − z1i

xi zi2 yi zi2

. . .

4

xi yi 2 zi

1+

yi2 zi2



 − 1+

− xzi2yi

x2i zi2



i

yi zi −xi zi

xi zi f yi zi f

1 f

0



  0   1  f  

(15)

Point-to- ontour orresponden es

The relationship between J2 and J1 established in the previous se tion  J2

= NJ1

 allows

analysis of the singularities asso iated with this type of orresponden es. Unlike the point-topoint ase, ea h point-to- ontour mat h ontributes with one row in the Ja obian matrix J2 . Therefore, twi e more point-to- ontour than point-to-point orresponden es are ne essary. At rst glan e this may appear as a disadvantage.

In pra ti e, it is easier to establish

point-to- ontour orresponden es and hen e a method based on su h orresponden es is likely to be more robust. It is often the ase that several points mat h a single ontour. It will be shown that in the ase of linear ontours, the minimum number of image lines required by the point-to- ontour tra ker is equal to the number of image points required by the point-to-point tra ker. In order to implement a tra ker based on point-to- ontour orresponden es, the rank of the Ja obian matrix J2 must be equal to 9 (position, orientation, and varying internal

amera parameters), 7 (position, orientation, and varying fo al length), or to 6 (position and orientation). Consequently a minumum of 9, 7, or 6 mat hes are ne essary. We analyse these situations in reverse order.

4.1

Fixed internal amera parameters

Without loss of generality we onsider a set of olinear model points, i.e., model points

A and B be two points belonging to a linear edge with amera A = (xA yA zA )⊤ and B = (xB yB zB )⊤ . A third point M has as oordinates M = A + λ(B − A) or: lying on linear edges. Let

oordinates

where

λ

xM yM

= xA + λ(xB − xA ) = yA + λ(yB − yA )

zM

= zA + λ(zB − zA )

is a free parameter, e.g., Figure 2.

INRIA

Multiple- amera tra king

15

A M model line

B

image line n projected model line

Figure 2: This gure shows a mat h between a model point (lying onto a straight-line and an image line, i.e., a point-to-line orresponden e.

RR n° 4268

Martin & Horaud

16

These three model points proje t onto the image produ ing an image line and let

(nu nv )⊤

be the normal to this line. The pixel oordinates of

oordinates of ve tors

A

and

p n= kpk

B

n

n=

an be omputed from the

[18℄:

with

p=



pu pv



=



yB zA − yA zB xA zB − xB zA



Matrix N, ontaining the image normals to the predi ted model ontour, has a simple expression for three ollinear points:



p 1  u 0 N = kpk 0

0 pu 0

pv 0 0

0 pv 0

0 0 pu

 0 0  pv

The Ja obian asso iated with this onguration  three olinear points mat hing an image ontour  writes, i.e., eqs. (9), (13), and (10):

J2



1 2 zA

p  = kpk  0 0

0 1 2 zB

0

2 2 pu xA yA + pv (zA + yA ) 2 2 pu xB yB + pv (zB + yB ) 2 2 pu xM yM + pv (zM + yM )

 0 −pu zA 0    −pu zB 1 −pu zM 2

zM

2 −pu (zA + x2A ) − pv xA yA 2 −pu (zB + x2B ) − pv xB yB 2 −pu (zM + x2M ) − pv xM yM

By substituting the oordinates of

oordinates of

A

and

B

−pv zA −pv zB −pv zM

M

and

p

pu xA + pv yA pu xB + pv yB pu xM + pv yM

(16)

 pu yA zA − pv xA zA pu yB zB − pv xB zB  pu yM zM − pv xM zM

with their expressions as fun tions of the

we obtain that the third row of the 3×6 matrix above is a linear

ombination of the rst and se ond rows. Therefore the rank of J2 is equal to 2. In other terms, points

Mi

whi h are ollinear with

A and B

of points the rank of the Ja obian asso iated with

are redundant. Regardless of the number

n ollinear points

mat hing an image line

is at most equal to 2.

4.2

Varying fo al length

The Ja obian is a 3×7 matrix in this ase. The rst six olumns are identi al to eq. (16). The last olumn is obtained by ombining the expression of matrix N with eq. (14) and is equal to:

zA f (pu xA zB f (pu xB zM f (pu xM

+ pv yA ) + pv yB ) + pv yM )

As before, by substitution, we obtain the third row as a linear ombination of the rst two. Therefore the rank of the 3×7 Ja obian matrix is equal to 2.

INRIA

Multiple- amera tra king

4.3

17

Varying internal amera parameters

The Ja obian is a 3×9 matrix in this ase.

The rst seven olumns are identi al to the

previous ase. The last two olumns are obtained by ombining the expression of matrix N with eq. (15) and yield the following 3×2 blo k: 2 zA pu f 2 zB pu f 2 zM pu f

2 zA pv f 2 zB pv f 2 zM pv f

Again, the rank of this matrix is equal to 2.

5

Multiple amera tra king

Single- amera tra king suers from three basi limitations:

ˆ

As explained above, there are ases where the Ja obian matrix is rank-de ient and hen e tra king annot be performed (planar obje ts lying in a fronto-parallel plane);

ˆ

Due to o

lusions, bad imaging onditions, and so forth, there may not be enough features available in one image, and

ˆ

While the obje t moves it may, totally or partially, disappear from the amera eld of view.

There are two options enabling one to introdu e multiple- amera tra king.

The rst

possibility is to make use of multiple- amera geometry, as des ribed in several books [8℄, [9℄, [16℄, and whi h implies that all ameras have a large eld of view in ommon. The se ond possibility is to exe ute single- amera tra kers in parallel while making expli it the obje t's rigidity. Indeed, if ea h one of the ameras looks at a dierent obje t part, the whole obje t is nevertheless rigid and geometri knowledge allows individual alibration of ea h one of the

ameras. On e this alibration step has been performed, all the ameras tra k the obje t in parallel estimating its motion in a ommon referen e frame. Su h a tra ker allows, one feature to be seen by one amera, another feature to be seen by another amera, et . Sin e the latter method does not use any form of stereo orresponden e, the ameras an be quite far apart thus insuring that a planar pie e of the obje t appearing in one's amera fronto-parallel plane is viewed slanted by the other ameras. Moreover, even if ea h amera sees planar features in its own fronto-parallel plane, the Ja obian matrix resulting from all

ameras is not rank de ient.

RR n° 4268

Martin & Horaud

18

5.1

The multiple amera Ja obian matrix

Without loss of generality, we onsider that ea h amera has been individually alibrated using the tra ker des ribed in the previous se tion in onjun tion with a stati obje t. This means that both internal and external (pose) parameters are available for ea h amera and that the position and orientation of ea h amera is known in a ommon referen e frame. Therefore the multiple amera tra ker is left with six parameters to be estimated: the position and orientation of the obje t with respe t to the ommon referen e frame. With the same notations as above, we generalize the point-to-point Ja obian matrix su h that the referen e frame is an arbitrary frame rather than the amera frame. For amera

k

the Ja obian writes:

k J1



. . .

   − z1ik =   0 

0 − z1ik

. . .

where Rk and

tk

xik 2 zik yik 2 zik

xik yik 2  zik 2  y 1 + zik 2 ik

 − 1+

x2ik 2 zik xik yik − z2 ik



yik zik −xik zik



   Rk   0  

[tk ]× Rk

(17)

are the rotation and translation mapping 3-D oordinates of model points

from a global ommon referen e frame to the amera frame. The notation skew symmetri matrix asso iated with a 3-ve tor, of a model point

Rk



i

in amera frame

xik , yik , and zik

[]×

stands for the

represent the oordinates

k.

The multiple amera Ja obian matrix writes:

J1



  =  

1

J1

. . . k J1 . . .

     

(18)

There is a similar expression for the point-to- ontour multiple- amera Ja obian matrix:

J2



  =  

1 1

N J1 . . .

k k

N J1 . . .

     

(19)

INRIA

Multiple- amera tra king

6

19

Experiments

The single and multiple amera tra king methods des ribed above were used to lo ate omplex ship parts for subsequent ship building operations su h as welding [12℄. Throughout the experiments we used an RGB frame grabber in onju tion with 3 bla kand-white ameras. Su h a setup guarantees image grabbing syn hronization. The ameras are thus interfa ed to a PC/Pentium3/733MHz platform running under Linux. The image size is 768×574 pixels. The tra ker runs at 25Hz with one amera, at 17Hz with two ameras and at 12Hz with three ameras. The obje t to be lo ated ontains both linear and urvilinear edges.

The urvilinear

edges are approximated by pie ewise polygonal lines. There are approximatively 300 model edges. Ea h edge is sampled su h that, on an average there are 10 points per model edge.

6.1

Finding point-to- ontour orresponden es

One of the most ru ial steps of any tra king method is to establish mat hes (or orresponden es). In our ase we must asso iate model points with image ontours. Su h asso iations are easier to nd than the lassi al point-to-point, line-to-line, or ontour-to- ontour mat hes. The method des ribed below is noise-, error-, and o

lusion-tolerant be ause it eliminates any model-to-image data asso iations that are likely to lead to an ambiguous solution. The image lo ations of the model edges and their asso iated model points are estimated by using a hidden-line elimination te hnique and by setting the view parameters to some initial values.

Image ontours are sear hed in a dire tion perpendi ular to the predi ted

model edge, on its both sides. In pra ti e, one-dimensional edge dete tion is performed for ea h model point along a line perpendi ular to its supporting model edge. This te hnique is illustrated on Figures 3 and 4. Figure 3 shows a model edge, points along this edge (bla k dots) and the image line that is eventually asso iated with ea h one of these points. Figure 4 shows an example. A number of model edges are predi ted in the image. The small white segments show su

essful asso iation of a model point with an image

ontour. Noti e that a large number of model points do not have image ontours asso iated with them either be ause of o

lusions or be ause an image ontour has not been found.

6.2

Lo ating a stati obje t

The rst experiment onsists of using one amera with xed internal parameters observing a stati obje t. A omplete obje t lo ation sequen e is shown on Figure 5. The initial model lo ation was set manually. After 18 iterations the pose parameters onverged to a stable solution orresponding to an average image error of 0.8 pixels.

RR n° 4268

Martin & Horaud

20

image line

model edge

bounded search

Figure 3: Point-to- ontour (here point-to-line) orresponden es are established by sear hing for image ontours around a predi ted model point lying onto a model edge. The sear h is in a dire tion perpendi ular to the predi ted model edge and is bounded.

INRIA

Multiple- amera tra king

Figure 4:

21

Points along model edges are asso iated with image ontours  these model-

point-to-image- ontour orresponden es are illustrated by short segments perpendi ular to the model edges. Noti e that a large number of model edges do not have image ontours asso iated with them. The ability to restri t the number of mat hes to the most reliable ones is key to su

essful onvergen e of the tra king pro ess.

RR n° 4268

Martin & Horaud

22

Ea h iteration omprises (i) the gathering of a new image, (ii) model predi tion in the image using the previously estimated pose parameters, (iii) hidden-line elimination, (iv) dete ting an image ontour mat h for ea h model point, and (v) one Gauss-Newton iteration based on valid point-to- ontour orresponden es. Sin e with one amera the algorithm runs and 25Hz, it takes 0.7 se onds to rea h onvergen e in this ase. On e onvergen e is rea hed, the system is ready for tra king.

6.3

Tra king the fo al length

The se ond experiment onsists of using one amera with varying fo al length observing a stati obje t. The fo al length varies ba k and forth from its minimal to its maximal values with a ratio of 1:3. The left olumn of Figure 6 shows the result of tra king in su h a ase. Both the pose and the fo al length are allowed to vary during the tra king pro ess using the adequate Ja obian matrix, i.e., se tions 3.2 and 4.2. The right olumn of Figure 6 shows the same experimental setup but the fo al length is set to a xed value. The tra ker tries to ompensate the zoom ee t with depth variations but, learly, the results are not very good. Table 1 summarizes a series of tests. The rst test (top) is designed to tra k the fo al length as just des ribed.

Small variations in depth are du to the fa t that the opti al

enter moves while the zoom varies. The se ond test (middle) maintains the fo al lenght to its initial value and therfore the depth ompensate for zoom variations. The third test (bottom) allows both internal and external parameters to vary. Although the opti al enter moves a lot, the fo al lentgth values estimated in this ase are very lose to those estimated in the rst test. This is lear experimental eviden e that the method des ribed in this paper is able to make a lear distin tion between depth variations and zoom variations. The only ase that

onfuses the tra ker is when the obje t is planar and lies in a fronto-parallel plane.

6.4

Camera alibration during tra king

Modern digital ameras have built-in auto-fo us. The ability to tra k an obje t with su h a amera ne essitates on-line estimation of internal amera parameters be ause a hange in fo us is asso iated with hanges in internal amera parameters. The third experiment onsists of estimating the internal amera parameters  tra king with one amera and 9 parameters while the obje t is allowed to move. Starting with a good initial guess and after only a few iterations, the tra ker onverges su h that the estimated

amera parameters mat h the real values. In order to assess the quality of the estimated amera parameters we ompared the results of tra king with the result obtained by using a alibration jig, Table 2. The rst row

INRIA

Multiple- amera tra king

23

Figure 5: This gure shows an obje t lo ation example from initialization to onvergen e. 18 iterations were ne essary in this ase. The gure shows (left to right and top to down) the initialization, as well as iterations 3, 6, 9, 12, and 18. Although the algorithm performs hidden-line elimination and onsiders only the model points predi ted visible, the display shows all (visible and hidden) model edges.

RR n° 4268

24

Martin & Horaud

Figure 6: The ee t of zooming in and zooming out an be taken into a

ount by the tra ker (left olumn). The right olumn shows the ee t of tra king with xed fo al length: the zoom ee t is ompensated by variations in depth, but this ompensation is far from being perfe t.

INRIA

Multiple- amera tra king

25

f (pixels)

u0

(pixels)

v0

(pixels)

Depth (mm)

Fo al length &

1615.2

288.4

386.8

2351.7

External

3226.4

288.4

386.8

2503.7

Parameters

4899.5

288.4

386.8

2553.4

External

1569.8

289.0

386.3

2282.1

Parameters

1569.8

289.0

386.3

1293.8

1569.8

289.0

386.3

840.2

All

1566.7

288.4

386.8

2476.8

Parameters

3190.2

335.5

623.7

2512.3

4956.2

164.4

105.2

2527.5

Table 1: This table shows a series of three tests where only the fo al length varies in pra ti e.

fu

u0

std

std

283.1

v0

std

o-line

1524.5

400.2

Exp. 1

1468.7

3.1

282.8

1.6

376.6

1.2

Exp. 2

1479.6

4.2

321.7

2.2

349.1

3.2

Exp. 3

1490.1

2.65

317.2

2.1

347.9

2.6

Exp. 4

1458.4

3.1

263.2

1.4

366.1

1.6

Exp. 5

1433.3

3.24

254.1

3.91

380.5

1.94

Exp. 6

1554.3

0.83

286.0

1.50

338.9

0.76

Table 2: The internal amera parameters obtained with lassi al o-line alibration (rst row) are ompared with those obtained during the tra king pro ess.

displays parameter values determined o-line using a lassi al amera alibration te hnique in onjun tion with a alibrated obje t (the a

ura y of the 3-D measurements are of about 0.01mm). The subsequent rows orrespond to experiments performed with the tra ker. Ea h experiment onsists of 100 measurements. Ea h experiment orrespond to a dierent setup  a dierent relative position between obje t and amera. The dis repan ies between the two sets of parameters appear from the fa t that the ship part is a welded obje t whi h does not perfe tly orrespond to its CAD model. The a

ura y of the ship part real size with respe t to its theoreti al CAD representation is of the order of a few milimeters.

6.5

Tra king with three ameras

The last experiment that we des ribe uses 3 stati ameras observing a moving obje t. With three ameras the tra king system runs at 12Hz thus allowing obje ts to freely move in the

RR n° 4268

Martin & Horaud

26

viewing volume.

Noti e that the amera setup is arbitrary, no ommon eld of view in

between any amer pair is ne essary. One amera has a panorami eld of view while the other two ameras see parts of the obje t. The amera system is alibrated using the previously des ribed experiments for ea h individual amera. During this experiment the internal amera parameters asso iated with all three ameras are xed as well as the relative positions and orientations of the ameras. Therefore the Ja obian used by the tra ker is given by eq. 19. From the Ja obian singularity analysis arried out above it is lear that su h a 3 amera tra ker has no singularity asso iated with it. The minimal data asso iation required by the tra ker is omposed of 6 model points mat hing 3 image ontours, with two points mat hing one ontour. In pra ti e, the tra ker has to deal with partial o

lusions in all images, global o

lusion in one or two images, de ien y of one or two ameras, et . The advantage of the 3 amera system is that the minimal mat hing requirement just des ribed may be distributed. The tra ker is able perform well with as little mat hes as two olinear model points visible in ea h one of the 3 images. The next three gures show the result of tra king with three ameras. Figure 7 shows a typi al tra king sequen e where the obje t freely moves onto the ground.

Noti e the

ex ellent overlapping between model and obje t through the whole sequen e in spite of the fa t that the obje t travels away from the eld of view of two ameras. Figure 8 shows partial o

lusion in two images and total o

lusion in one image while Figure 9 shows partial o

lusion in all three images.

7

Dis ussion

In this paper we des ribed a method for tra king rigid obje ts using one or several ameras. The tra king pro ess onsists of aligning a 3-D model representation of an obje t with image

ontours by measuring and minimizing the image error between predi ted model points and image ontours. The tra ker behaves like a visual servo loop where the internal and external

amera parameters are updated at ea h new image a quisition. We studied in detail the Ja obian matrix asso iated with this minimization pro ess in the presen e of both point-to-point and point-to- ontour mat hes. We established the minimal number of mat hes that are needed as well as the singular ongurations leading to a rankde ient Ja obian matrix. We established a mathemati al link between the point-to-point and point-to- ontour ases. Based on this link we showed that the latter has the same kind of singularities than the former. Moreover, we studied multiple amera ongurations whi h optimize the robustness of the method in the presen e of single- amera singularities, bad, noisy, or missing data. Extensive experiments done with a omplex ship part and with up to three ameras validated the method. In parti ular we show that the tra ker may well be used as a amera

alibration pro edure.

INRIA

Multiple- amera tra king

27

Figure 7: A tra king sequen e. The obje t rotates and translates onto the oor. Noti e that the three ameras have very dierent points of view.

RR n° 4268

Martin & Horaud

28

Figure 8: Total o

lusion in one image and partial o

lusion in two images.

Figure 9: Partial o

lusion in three images.

One limitation of the method is that it only deals with edges. In omplex natural and industrial environments an edge-based method has failures. Therefore we plan to in orporate texture information with our model representation.

When a model edge is predi ted in

the image, nearby model texture is predi ted as well and hen e it is possible to measure the dis repan y with image texture and tune the tra ked parameters by optimizing the alignment between model and image textures. The fusion of several ues su h as edge and texture information, ombined with the in reasing omputing power available today, is likely to improve the robustness and realiability of visual tra kers for many roboti s appli ations.

Referen es [1℄ M. Armstrong and A. Zisserman. Robust obje t tra king. In

on Computer Vision,

Pro . Asian Conferen e

volume I, pages 5861, 1995.

[2℄ Q. Delamarre and O. Faugeras. 3D arti ulated models and multi-view tra king with silhouettes.

In

Pro eedings 7th IEEE International Conferen e on Computer Vision,

volume 2, pages 716721, Corfou, Gree e, September 1999.

INRIA

Multiple- amera tra king

29

[3℄ T. Drummond and R. Cipolla. servoing. In

September,

1999.

[4℄ T. Drummond and R. Cipolla. servoing.

Real-time tra king of omplex stru tures for visual

Pro . IEEE Seventh Int. Conf. on Computer Vision, Corfu, Gree e, 20-25

In

Real-time tra king of omplex stru tures for visual

Vision Algorithms, Theory and Pra ti e,

pages 6983. Springer Verlag,

LNCS, 1999. [5℄ T. Drummond and R. Cipolla. Real-time tra king of omplex stru tures with on-line

amera alibration. In

Pro eedings of British Ma hine Vision Conferen e,

volume 2,

pages 574583, Nottingam, UK, September 1999. [6℄ T. Drummond and R. Cipolla. in multiple views.

In

Real-time tra king of multiple arti ulated stru tures

Pro eedings of 6th European Conferen e on Computer Vision,

volume 2, pages 2036, Dublin, Ireland, June-July 2000. [7℄ B. Espiau. Ee t of amera alibration errors on visual servoing in roboti s. In

ings of the Third International Symposium on Experimental Roboti s,

Pro eed-

pages 187193,

Kyoto, Japan, O tober 1993. [8℄ O. D. Faugeras.

Three Dimensional Computer Vision: A Geometri Viewpoint.

MIT

Press, Boston, 1993. [9℄ R. Hartley and A. Zisserman.

Multiple View Geometry in Computer Vision.

Cambridge

University Press, Cambridge, UK, 2000. [10℄ R. Horaud, G. Csurka, and D. Demirdjian.

Stereo alibration from rigid motions.

IEEE Transa tions on Pattern Analysis and Ma hine Intelligen e,

22(12):14461452,

De ember 2000. [11℄ K. Kinoshita and K. Degu hi.

Simultaneous determination of amera pose and in-

trinsi parameters by visual servoing. In

Conferen e on Pattern Re ognition,

Pro eedings of the 12th IAPR International

volume 1, pages 285289, Jerusalem, September

1994. [12℄ B. Lamiroy, T. Drummond, R. Horaud, and O. Knudsen-Ne kelman. Visually guided robots for ship building. In 1st International Conferen e on Computer Appli ations and Information Te hnology in the Maritime Industries (COMPIT'2000), Potsdam/Berlin, Germany, Mar h 30  April 2, 2000. [13℄ B. Lamiroy, B. Espiau, N. Andre, and R. Horaud. Controlling robots with two ameras: How to do it properly.

Automation,

In

Pro . of IEEE International Conferen e on Roboti s and

pages 21002105, San Fran is o, CA, April 2000.

[14℄ D. Lowe. Fitting parameterized three-dimensional models to images.

on Pattern Analysis and Ma hine Intelligen e,

RR n° 4268

IEEE Transa tions

13(5):441450, May 1991.

Martin & Horaud

30

[15℄ D. Lowe. Robust model-based motion tra king through the integration of sear h and estimation.

International Journal of Computer Vision,

8(2):113122, 1992.

The Geometry of Multiple Images. MIT Press, Boston,

[16℄ Q.-T. Luong and O. D. Faugeras. 2001.

[17℄ E. Mar hand, P. Bouthemy, F. Chaumette, and V. Moreau. Robust real-time visual tra king using a 2D-3D model-based approa h. In

Conferen e on Computer Vision,

Pro eedings 7th IEEE International

volume 1, pages 262268, Corfou, Gree e, September

1999. [18℄ T. Q. Phong, R. Horaud, A. Yassine, and D. T. Pham. Obje t pose from 2-D to 3-D point and line orresponden es.

International Journal of Computer Vision,

15(3):225

243, July 1995. [19℄ C. Rasmussen and G. D. Hager. Probabilisti data asso iation methods for tra king

omplex visual obje ts.

gen e,

IEEE Transa tions on Pattern Analysis and Ma hine Intelli-

23(6):560576, June 2001.

[20℄ M. Tonko and H-H. Nagel.

Model-based stereo-tra king of non-polyhedral obje ts

for automati disassembly experiments.

International Journal of Computer Vision,

37(1):99118, June 2000. [21℄ M. Vin ze, M. Ayromlou, W. Ponweisser, and M. Zilli h. Edge-proje ted integration of image and model ues for robust model-based obje t tra king.

of Roboti s Resear h,

International Journal

20(7):533552, July 2001.

INRIA

Unité de recherche INRIA Rhône-Alpes 655, avenue de l’Europe - 38330 Montbonnot-St-Martin (France) Unité de recherche INRIA Lorraine : LORIA, Technopôle de Nancy-Brabois - Campus scientifique 615, rue du Jardin Botanique - BP 101 - 54602 Villers-lès-Nancy Cedex (France) Unité de recherche INRIA Rennes : IRISA, Campus universitaire de Beaulieu - 35042 Rennes Cedex (France) Unité de recherche INRIA Rocquencourt : Domaine de Voluceau - Rocquencourt - BP 105 - 78153 Le Chesnay Cedex (France) Unité de recherche INRIA Sophia Antipolis : 2004, route des Lucioles - BP 93 - 06902 Sophia Antipolis Cedex (France)

Éditeur INRIA - Domaine de Voluceau - Rocquencourt, BP 105 - 78153 Le Chesnay Cedex (France) http://www.inria.fr

ISSN 0249-6399