Automated Registration of Surveillance Data for Multi ... - CiteSeerX

Automated Registration of Surveillance Data for Multi-Camera Fusion P. Remagnino and G.A. Jones Digital Imaging Research Centre Kingston University, Penrhyn Road, Kingston upon Thames, Surrey KT5 8DG, UK.

fp.remagnino,[email protected] The fusion of tracking and classi cation information in multi-camera surveillance environments will result in greater robustness, accuracy and temporal extent of interpretation of activity within the monitored scene. Crucial to such fusion is the recovery of the camera calibration which allows such information to be expressed in a common coordinate system. Rather than relying on the traditional time-consuming, labour-intensive and expert-dependent calibration procedures to recover the camera calibration, extensible plug-and play surveillance components should employ simple learning calibration procedures by merely watching objects entering, passing through and leaving the monitored scene. In this work we present such a two stage calibration procedure. In the rst stage, a linear model of the projected height of objects in the scene is used in conjunction with world knowledge about the average person height to recover the image-plane to localground-plane transformation of each camera. In the second stage, a Hough transform technique is used to recover the transformations between these local ground planes. Abstract {

Auto-Calibration, Ground Plane Constraint, Camera Calibration, Visual Surveillance. Keywords:

1

Introduction

Accurately detecting and tracking moving objects within monitored scenes is crucial to a range of highlevel behavioural vision tasks[6, 8, 10, 16, 15]. Tracking accuracy can be greater enhanced by combining information from several cameras with overlapping views. Not only will this provide more observations but will ensure the object remains tracked for longer intervals and provide some protection from occlusion. However to successfuly fuse these multiple sources of information, it is necessary to project these observations into the same coordinate system. Most data fusion meth-

ISIF © 2002

1190

ods depend on a relatively complex calibration process based on variants of the Tsai calibration technique[14]. Real world positions of highly visible point events are recorded and manually matched to the equivalent image events in each camera. The method has the advantage of being accurate (where a suÆcient number of well dispersed correspondences have been manually established) as well as enabling data from the dierent camera sources to be expressed within the same coordinate system. Auto-calibration techniques seek to establish the camera-plane to ground-plane or image-plane to image-plane transformations without the need for these manual time-consuming, labour-intensive and skill-dependent calibration procedures. Ideally, the system should learn the calibration by watching the events unfold within the monitored scene and, where appropriate, combine learnt knowledge with any available world knowledge. In previous work, Black and Ellis[2] and Stein[13] use sets of moving object observations from two views. Since the correspondences between views is unknown, both employ a Least Median of Squares technique to recover the correct image-plane to image-plane homography from a large canidate population of all possible correspodence pairings. In Stein[13], the resultant initial transformation is re ned using an image greylevel alignment technique. Where more than two cameras are involved no single world coordinate system is available. Both this work and that proposed in this paper relies on the recovery of the correspondence between point observations. Point features oer little discriminatory power to prune the large number of false correspondences. The work of Jaynes[7] attempts to match the trajectories themselves. During an initial manual stage, the relative orientation of the ground plane to the image plane is computed by enabling an operator to recover the vanishing points of orthogonal bundles of parallel 3D line

structures on the ground plane. Trajectories positions are reprojected onto an arbitrary plane parallel to the ground plane. Trajectories of a minimum length are then matched using a non-linear least squares technique to recover the rotation, translation and scale transformation between the projection planes of any pair of cameras. Based on the assumption that 3D motion has on average a constant value across the image plane, Boghossian[3] is able to use an optic ow algorithm to recover the relative depth of the ground plane i.e. de ne an arbitrary reprojection plane.

the use of simple but highly discriminatory models of the appearance of scene objects which indirectly use the depth of the object to model its projected width and height. Since, the spatial extent of object are now a function of image position, any image tracker will be more robust when presented with the distorted observations which arise from fragmentation or occlusion processes. Our tracking model represents an object simply as a projected 2D rectangles whose height and width are determined by the object's 3D position in the scene and hence 2D position in the image[11].

Organisation of Paper

In section 2, we demonstrate that the projected image height of a person can be modelled to reasonably accuracy linear model as a linear function of vertical image position. The parameters of the model depend on the height and lookdown angle of each camera. This model - learnt from observations of detected moving objects - may be combined with world knowledge (i.e. the average height of a person and the height of the camera above the ground plane) to recover the image to ground-plane homography. Having calibrated each camera to its local ground plane, section 3 demonstrates how these ground planes may be registered. Again a learning procedure is pursued in which the projected trajectory positions in corresponding frames and their instantaeous velocity estimates are combined to create estimates of the rotation and translation. A clustering algorithm is used to locate the most likely transform between each pair of camera ground planes. 2

Auto-Calibration

of

the

Ground Plane

The objective of this work is to derive a very simple yet highly eective method of learning the planar homography of each camera to its camera speci c ground plane (see sections 2.3 and 2.4). Rather than relying on a time-consuming, labour-intensive and skilldependent calibration procedure to recover the full image to ground-plane homography[4], the system relies on a simple learning procedure to recover the relationship between image position and the projected width, height and motion of an object by observing examples of the typical object types within a surveillance scene. The learning calibration procedure utilises the simple but reasonably accurate assumption that in typical surveillance installations, the projected 2D image height of an object varies linearly with its vertical position in the image - from zero at the horizon to a

2.1

Ground Plane Projection

To establish the camera to ground plane homography, it is necessary to establish the position of origin of the ground plane coordinate system (GPCS). In this auto-calibration scenario, a local ground plane coordinate system must be speci ed for each camera. A second correspondence stage is then employed to fuse this set of camera-speci c ground planes together. The camera-speci c GPCS is de ned as follows: The Y -axis Y^ of the GPCS is de ned as the projection of the optical axis along the ground plane. The Z -axis Z^ is de ned as the ground plane normal, and ^ is de ned as the vector within the ground the X -axis X plane normal to the camera optical axis. The position of the focal point of the camera in the GPCS is directly `above' the GPCS origin i.e. at the point (0; 0; L).

The image plane is situated at distance f (focal length of the optical system for the camera) perpendicular to the optical axis z^. In this con guration a point P on the image plane has coordinates x0 = (x; y; f )T . The pixel coordinate system i; j (representing the row and column position respectively from the top left of an image) is related to the image plane coordinate system by x = x (j j0 ) and y = y (i0 i) where i0 ; j0 is the optical centre of the image and x and y are Camera reference frame

yˆ

zˆ

xˆ

uˆ

Image plane

f

vˆ optical axis

zˆ

yˆ

xˆ Ground Plane

maximum at the bottom-most row of the image. This height model is derived from the optical geometry of a typical visual surveillance installation illustrated in Figure 1: Camera, World and Image Plane co-ordinate gure 3(a). In addition, such an assumption enables systems. 1191

the horizontal and vertical inter-pixel widths. Thus

2.2

Projected Object Height

The camera is required to have a look-down angle  (1) with no signi cant roll. If one assumes that the height of a moving object is known (for instance a person f f where x = x =f and y = y =f are the horizonwalking in the eld of view of a camera, see Figure tal and vertical pixel dimensions divided by the focal 2), then the point of intersection X 0 can be shifted length. An optical ray L containing the focal point of the along the Z^ direction of the known00 height H . Using to camera passing through the point P on the image plane , we can write the 00coordinate0 X corresponding ^ . The new 0 the new point as X =  Rx + t + H Z can be represented in vectorial form as x = x . Let 00 Q be the point of intersection of the optical ray L and image point x corresponding to the projection of the the ground plane . In order to calculate the position top of the personT can be computed from the inverse of the point Q on the ground plane  in the ground transformation R (X t). This transformation yields plane coordinate system, one must convert the direcx00 = x0 + H RT Z^ (8) tion of the optical ray L given the transformation (R; t) between the camera (image plane) and world (ground where  is the structure parameter for the top of the plane) coordinate systems person. Substituting  and tz = L yields 0 X = Rx + t = Rx + t (2)   L 1 H RT Z^ 0 00 x (9) x = ^
Rx0 Writing the ground plane equation as n
X = 0,  Z where the ground plane normal n  Z^ , then the To measure the projected vertical height of an obposition X 0 the point of intersection Q between the ject, we simply de ne a plane  containing the optical optical ray and the ground plane is obtained by centre and the image plane raster line containing the X 0
Z^ = (Rx0 + t)
Z^ = 0 (3) new point x00 . The normal n of this plane is de ned by the cross-product between the projection line x00 yielding and the rasterline direction vector x^ as follows t = ^ z 0 (4)   Z
Rx L 1 T ^ 0 n =  H R Z  x^ Z^
Rx0 x  x^ (10) The scale factor  is usually referred to as the structure parameter, since it identi es the distance between the camera and the observed object. Substituting the The rasterline containing the point x00 can be thought of as lying at a distance h above the projection of the value of  into the expression of X 0 we obtain bottom of the person. Therefore the point vertically z 0 above (in terms of the image coordinate system) x0 can X 0 = Z^
tRx Rx + t (5) 0 be expressed as x = x0 + hy^ and belongs to the plane The GPCS is de ned with a zero pan angle. If we as- . Substituting x0 + hy^ into the equation of plane  sume that there is no signi cant roll angle, then the generates n
x0 + hn
y^ = 0 (11) ground plane coordinates are related simply to the look-down angle  as follows f (j j0 ) L (6) X = f x y (i i0 ) sin  cos  L(fy (i i0 ) cos  sin ) Y = (7) fy (i i0) sin  cos  Thus to compute the real world ground plane position of an image point, the intrinsic and extrinsic camera parameters i0 ; j0 , fx ; fy and  must be estimated. In our approach the optical centre i0; j0 is computed by an optical ow algorithm which robustly ts a global zoom motion model to a three frame sequence undergoing a Figure 2: Computing Projected Height: Image height small zoom motion. The rest of the parameters may be is the vertical image distance between the projection of recovered in the following training procedure based on the foot of an object and the raster line containing the watching events unfolding within the monitored scene. projection of the top of an object.

x0 =

fx (j

j0 ); fy (i0


T

i); 1 f

Camera reference frame

zˆ

yˆ

n

xˆ

uˆ

Image plane

vˆ

h

H

H

Ground Plane

1192

yielding

n
x0 h=  n
y^

(12)

After removing the  factor from numerator and demoninator by cancellation, further simpli cation can be derived by expanding the numerator of the equation 12 using equation 10 as follows L 0 0 n
x0 = H (RT Z^ x^ )
x0 ^ Z
Rx0 (x x^ )
x (13)

where the latter term is zero since (x0  x^ )
x0 = 0. The denominator can be written as follows

parameters f = 10mm, H = 1:76m, L = 6m, zero roll angle and look-down angle  = 76Æ (i.e. 14Æ down from horizontal). Note for the given range of image positions corresponding to an angular eld of view of  35Æ , the plot is essentially linear. The intercept with the vertical position axis (or h = 0 axis) de nes the horizon where objects become in nitely small. We are currently investigating the validity of this linear assumption. For example, we know for steep lookdown angles that the projected height of objects very close to the camera actually peak and subsequently decrease in size. This will result in a limit on the angle of view for a given lookdown angle.

L 0 ^Z
Rx0 (x  x^ )
y^ 2.3 Learning the Height Model For the zero roll optical con guration, the projected = H z^
RT Z^ ^ L 0 z^
x0 Z
Rx image height of an object may be captured as a linmodel varying against vertical image position only. = H Z^
Rz^ ^ Lf 0 (14) ear In pixel coordinates, this linear relationship may be Z
Rx expressed as Combining the equations 13 and 14, the image plane  = (i ih ) (17) height h can be written as a function of the world where  is the object height in pixels, and  = 0 at height H as follows the horizon i = ih . This model may be extracted from H (Z^
Rx0 )(RT Z^  x^ )
x0 scene automatically by accumulating a histogram h= (15) the H [in ; n ] from a large number n of object observaH (Z^
Rx0 )(Z^
Rz^ ) Lf tions of the monitored scene. A motion detection proWhere there is a zero roll angle, equation 15 simpli- cess is employed to extract connected components of es to the following expression which depends only on moving pixels[12]. The bounding box (imin; imax; j min the vertical image plane position. and j max ) of each extracted blob generates a height
hn = imax imin and position in = imax n n n . Since the calH (f 2 y2) sin  cos  + yf (cos2  sin2 ) ibration method employs knowledge about the height h= yH sin  cos  (H cos2  L)f of the average person, the accumulation process uses (16) weights to signi cantly increase the in uence of peopleFor typical camera installations, h can be shown to like rather than car-like objects. This weight, based on eectively vary linearly with vertical image position the expected bounding-box shape of a moving person, relative to the position of horizon. Figure 3(a) plots is de ned as the projected height against image position for typical   max i imin (18) wn = max nmax nmin 1; 0 j j n
y^ = H (RT Z^  x^ )
y^

n

Person Projected Height (mm)

2

n

Currently the operator drags an appropriate line segment along the ridge structure - see gures 4(b) and (d). We intend to automate the process with a robusti ed line tting procedure supervised by the operator.

1.5

1

2.4 0.5

0 -0.003

-0.002

-0.001

0

0.001

0.002

0.003

Vertical Image Position (mm)

Figure 3: Projected Height versus Vertical Position 1193

Ground Plane Calibration

Since the vertical image height of an object is independent of the horizontal image position of the projected object, the following derivation may assume without loss of generality that the object is located on the vertical axis i.e. x = 0 (or alternatively j = j0 ). In its trajectory down the image, two key positions of a projected object may be de ned at i = ih at the horizon, and i = i0 at the optical centre of the image. At

3

Hough Transform approach to Calculating the Inter-camera Transformation

(a) Camera 1

(b) Histogram

(c) Camera 2

(d) Histogram

In Section 2.1, the positions and velocity of objects tracked in each eld of view were backprojected onto a local reference frame set on the ground plane. The transformation between cameras is unknown but it can be easily calculated if the correspondences between object positions are known between views[5]. In our autocalibration scenario, we cannot assume that the correspondences are known. Further, while we assume the availability of object positions with associated velocity vectors, no temporal association is assumed. The Hough Transform approach[1] has been adopted to recover the inter-camera transformation by taking advantage of the fact that the ground plane coordinate systems of temporally synchronised observations of the same 3D object are related by a simple rotation and translation T transformation.

X 0 = R( )X + T V 0 = R( )V (23) where X ; V and X 0 ; V 0 are positional and velocity ob-

Figure 4: Learning the Projected Height Model: Fig- servations measured in the local GPCS of two cameras ures (c) and (d) show the (inverted) histograms for the C and C 0 respectively. Note that the velocity estimates are computed from the partial derivatives of equation 7 viewpoints illustrated in Figures (a) and (b). the former, the look-down angle  may directly related to the horizon parameter ih extracted from the accumulated training data acquired in the learning stage described in section 2.3 i.e. cot  = fy (ih i0 ) (19) For the latter case, consider an object of height H standing on the ground plane point given by the projection of the optical axis. As captured by equation 16, the vertical height at this point h(i = i0) may be related to the look-down angle as follows h H cos  sin  = (20) f L H cos2  The height h may also generated from the learnt linear projected height model of equation 17 i.e. h(i0 ) = x (i0 ih ). Combining this with equations 19 and 20, the following expressions for the camera parameters  and fy may be derived sin2  = H L1 H

cot  fy = (i i ) 0

h

(21)

with respect to image coordinates, and the 2D tracker image position (i; j ) and visual velocity (u; v) estimates i.e.

VX =

@X @Y @Y @X u+ v ; VY = u + v @i @j @i @j

(24)

In every frame interval, each camera outputs a set of measurements about all objects located in its eld of view. As object correspondences are unknown, every pair of observations from each of the cameras must be used to generate a candidate estimate of the transformation. Given a pair observations X 0t;i ; V 0t;i and X t;j ; V t;j at time t from camera C 0 and C respectively, transformation estimates may be de ned as cos

t;i;j

T t;i;j

0 = V^ t;i
V^ t;j = X 0t;i R(

t;i;j

)X t;j

(25)

where V^ is the unit vector in the direction of V . If t and 0t are the set of observations in frame t for cameras C and C 0 respectively, then the set of all observations

f

;i;j

; T ;i;j ; 8i 2 0 ; 8j 2  ; 8

 tg

(26)

(22) should ideally exhibit a distinct cluster of estimates around the true solution ; T within an noise oor of 1194

uncorrelated false estimates generated by incorrectly corresponded observation pairs and noise observations. To detect this cluster, the space could be tesselated into bins and a Hough transform technique applied to locate the maxima that correspond to the optimal transformation parameters. However, the range of translation values required is diÆcult to predict a priori. Therefore to avoid the storage problems such a voting strategy introduces, a robust clustering approach is adopted. The expectation-maximisation mixture of Gaussian technique was implemented and adapted to iteratively perform the cluster analysis on the incoming stream of transform estimates. The clusterer continually reports the most likely transformations between cameras.

Test Installations Scene 1 Scene 2 Scene 3 Correct Height 9.1m 13.9m 6.7m Tsai Height 9.9m 15.4m 5.7m Æ Correct Angle 16:0 24:3Æ 13:5Æ Æ Æ Tsai Angle 16:7 24:5 7:7Æ Æ Æ Our Approach 15:5 23:3 11:7Æ Table 1: Look-Down Angle Results For clarity the lookdown angle has been rede ned as =2  and indicates the angle of intersection between ground plane and optical axis.

around the optical axis - typically less than 4Æ . The project image height method proposed in this work accurately located the lookdown angle although care had 4 Results to be taken to correctly t the linear model to the proThree test installations have been setup involving jected height ridge of the histogram. three dierent types of camera placed at dierent heights jointly overlooking a typical carpark and en- Multi-Camera Calibration trance way scene. The carpark has been surveyed to Using the computed angles tabled in table 4, the generated real-world ground plane positions in a com- tracked object trajectories recovered from our motion mon coordinate system. These points have been se- detection and tracking software[9] can be projected lected to ensure that each camera has ten well dis- onto the local ground plane of each camera. Three tributed point in the image plane. The convex hull of thousand roughly contemporaneous trajectory obserthese points contains most of the carpark and over fty vations are shown in Figures 5 and 6. These obserpercent of the visual plane. Image sequences of several thousand images have been acquired.1 Image to Ground Plane Calibration

As described in section 2.3, the projected height model for each camera can be recovered by accumulating in a height versus vertical image position space and tting a straight line to the resultant histogram. Results for Cameras 1 and 2 are shown in gure 4. In conjunction with the measured height of the cameras above the ground plane, the parameters of these models can be used to derive the extrinsic and some of the extrinsic camera parameters - see equation 22. To compare the accuracy of the proposed method, the ground truth data2 and the results from the traditional Tsai[14] technique and our approach are tabulated in table 4. The Tsai calibration results were not particularly accurate at estimating the camera height and look-down angle. Nonetheless, they do estimate the ground plane projections of image points with a resonable degree of accuracy. We also employed the Tsai results to con rm that the camera had no signi cant roll i.e. rotation 1 The

PETS2001 datasets (visualsurveillance.org) are prob-

lematic as they contain so few tightly distributed calibration points.

2 The

real lookdown angle was computed from the ground

plane position of the pro jection of each camera's centre of expansion.

1195

Figure 5: Projected Trajectories from Camera 1 vations are used to build the rotation and translation Hough space described by equations 25. The populated space and estimates associated with the dominant cluster shown in Figure 7. Note that this peak is located in an extensive noise oor generated as a result of the lack of correspondence information. Nonetheless the robust characteristics of our clustering procedure ensure that the inter-camera registration is achieved in athe absence of trajectory or positional correspondence and without demanding a high degree of tracking coherence.

25

20

15

Figure 6: Projected Trajectories from Camera 2:

These three thousand position observations are roughly contemporaneous to the camera one trajectories.

The position of the Hough space peak is used to rotate and translate the data from the two camera ground planes into a single coordinate system (that of the second camera). The overlapped trajectories are displayed in Figure 8. While not a perfect alignment, the accuracy appears suÆcient to establish correspondence of any new objects that enter the scene. Any lack of alignment arises from a number of sources: (i) any existing roll angle on either camera; (ii) inaccuracies in estimation of the lookdown angle and intrinsic parameters of either camera; and (iii) view-dependent positional bias in trajectories. 40

Translation Y Component

30

20

10

0

−10

−20

0

10

20 30 40 Translation X Component

50

10 −15

−10

−5

0

5

10

15

Figure 8: Overlaying Trajectories in a common Ground Plane 5

Conclusions

A stream of work carried out at the Digital Imaging Research Centre focuses on the development of learning techniques for use in plug-and-play visual surveillance multi-camera systems. Many camera calibration techniques exist, however most of them require the assistance of an expert to tune a set of parameters. The strategy of DIRC is to develop a suite of algorithms that could be installed by non-technical personnel, and as much as possible based on self-adjusting techniques that learn how to adapt to the camera set-up, the environmental changes and possibly to weather conditions. This paper proposes part of the work developed within DIRC. The paper presented a novel camera calibration approach, based on two separate stages. In the rst stage a linear model of the projected height of objects in the scene is used in conjunction with world knowledge about the average person height to recover the image-plane to local-ground-plane transformation of each camera. In the second stage, a clustering technique (expectation-maximisation) is then used to recover the transformation between these local ground planes. A comparison between the proposed technique and the standard approach of Tsai was carried out. Results for both techniques, evaluated with ground truth measures, show that the accuracy of the proposed approach is similar to Tsais approach.

60

Figure 7: Locating the Maximal Cluster in Transform Space

1196

References

[1] Dana H. Ballard and Christopher M. Brown. "Computer Vision". Prentice-Hall, Inc., New Jersey, 1982.

[2] J. Black and T. Ellis. \Multi-Camera Image [14] Roger Y. Tsai. \A versatile Camera Calibration Tracking". In Second IEEE International WorkTechnique for High-Accuracy 3D Machine Vision shop on Performance Evaluation of Tracking and Metrology Using O-the-Shelf TV Cameras and Surveillance, Hawaii, December 2001. Lenses". IEEE Journal of Robotics and Automation, RA-3(4):323{344, August 1987. [3] B.A. Boghossian. \Motion-based Image Processing Algorithms applied to Crowd Monitoring Sys- [15] C.R. Wren, A. Azarbayejani, T. Darrell, and A. P. tems". PhD thesis, 2000. Pentland. "P nder: Real-time Tracking of the Human Body". IEEE Transactions on Pattern [4] D. Forsyth and J. Ponce. "Computer Vision: A Analysis and Machine Intelligence, 19(7):780{785, Modern Approach". Prentice-Hall, 2001. July 1997. [5] R.M. Haralick and H. Joo. "2D-3D Pose Estima- [16] T. Zhao, R. Nevatia, and F. Lv. \Segmentation tion". Proceedings of the International Conference and Tracking of Multiple Humans in Complex Siton Pattern Recognition, pages 385{391, November uations". In IEEE Computer Society Conference 14-17 1988. on Computer Vision and Pattern Recognition, volume 2, pages 194{201, Hawaii, December 2001. [6] I.Haritaoglu, D.Harwood, and L.S.Davis. \W4: Real-time Surveillance of people and their Activities". IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(7):809{830, July 1997. [7] Christopher Jaynes. \Multi-View Calibration from Planar Motion for Video Surveillance". In Second IEEE International Workshop on Visual Surveillance, pages 59{66, Fort Collins, Colorado,

June 26 1999. [8] A.J. Lipton, H. Fujiyoshi, and R.S. Patil. "Moving target classi cation and tracking from real-time video". In Proceedings IEEE Image Understanding Workshop, pages 129{136, 1998. [9] J. Orwell, P. Remagnino, and G.A. Jones. \From Connected Components to Object Sequences". In First IEEE International Workshop on Performance Evaluation of Tracking and Surveillance,

pages 72{79, 2000. [10] C.S. Regazzoni and A. Teschioni. \Real Time Tracking of non rigid bodies for Surveillance applications". In ISATA Conference, Firenze, 1997. [11] J. Renno, J. Orwell, and G.A. Jones. \Towards Plug-and-Play Visual Surveillance: Learning Tracking Models". In IEEE International Conference on Image Processing, page Accepted for Publication, Rochester, New York, September 22-25 2002. [12] C. Stauer and W.E.L. Grimson. \Learning Patterns of Activity using Real-Time Tracking". IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(8):747{757, August 2000.

[13] G.P. Stein. \Tracking From Multiple Viewpoints: Self-Calibration Of Space And Time". In DARPA Image Understanding Workshop, pages 1037{1042, Monterey, CA, November 1998. 1197