Camera Handoff with Adaptive Resource Management for Multi

IEEE Fifth International Conference on Advanced Video and Signal Based Surveillance

Camera Handoff with Adaptive Resource Management for Multi-Camera Multi-Target Surveillance 1

Chung-Hao Chen1, Yi Yao2, David Page1, Besma Abidi1, Andreas Koschan1, and Mongi Abidi1 Imaging, Robotics, and Intelligent Systems Laboratory, Department of Electrical Engineering and Computer Science, The University of Tennessee Knoxville, TN 37996 USA 2 GE Global Research Center, Niskayuna, NY 12309 USA based, geometry-based, and hybrid approaches. In feature-based approach [1, 2], color or other distinguishing features of the tracked objects are matched, generating correspondences among observing cameras. In geometrybased approach [3, 4], consistent labeling can be established by projecting the trace of the tracked object back into reference coordinates and then establishing equivalence between objects projected onto the same location. Within the geometry-based approach, various reference coordinates are employed, such as the world coordinates, usually the ground plane [3, 4], and the selected camera coordinates [5, 6]. The hybrid approach [7] is a combination of geometry- and feature-based methods. Although proven efficient, most multi-object tracking algorithms [8, 9] find it difficult to maintain a constant frame rate given limited resources. Herewith, resources include (1) CPU capacity for executing object tracking, crowd segmentation, and behavior understanding in an automated manner and (2) network bandwidth for exchanging camera handoff information. The computational complexity of most existing tracking systems is of the order from NpO(n) to NpO(n3) [10, 17], where Np is the number of tracked objects and n represents the number of steps to execute the algorithm. Given limited resources, there inherently exists an upper bound on the number of objects that can be tracked simultaneously without deteriorating the system’s frame rate. Figure 1 illustrates one example of a decreased frame rate when the system discussed in [10] is used to track an increased number of objects with limited CPU capacity. To be applicable to real-time surveillance, it is crucial for a tracking system to maintain a reasonable frame rate at all times. A lower frame rate may result in the following problems. (1) The system’s ability to automatically detect a threatening event degrades, causing possible observation leak. This dangerous loophole impedes the practical application of real-time multicamera multi-object tracking systems [11]. (2) The decreased frame rate also affects the performance of consistent labeling and consequently that of camera handoff. Consistent labeling requires accumulated

Abstract Camera handoff is a crucial step to generate a continuously tracked and consistently labeled trajectory of the object of interest in multi-camera surveillance systems. Most existing camera handoff algorithms concentrate on data association, namely consistent labeling, where images of the same object are matched across different cameras. However, most real-time object tracking systems see a decrease in the system’s frame rate as the number of tracked objects increases. To address this issue, we propose to incorporate an adaptive resource management mechanism into camera handoff. In so doing, cameras’ resources can be dynamically allocated to multiple objects according to their priorities and hence the required minimum frame rate can be maintained. Experimental results illustrate that the proposed camera handoff algorithm is capable of maintaining a constant frame rate and of achieving a substantially improved handoff success rate by approximately 20% in comparison with the algorithm presented by Khan and Shah.

1. Introduction With the increase in the scale and complexity of a surveillance system, it becomes increasingly difficult for a single camera to accomplish object tracking and monitoring with the required resolution and continuity. Camera networks emerge and find extensive applications. The employment of multiple cameras not only improves coverage but also brings in more flexibility. However, the use of multiple cameras also induces problems such as camera handoff. Camera handoff is a decision process of transferring a mobile object from one camera to another. Most existing camera handoff algorithms focus on developing efficient consistent labeling schemes. Consistent labeling, as an important step in camera handoff, solves the identity problem among multiple observing cameras. In literature, consistent labeling methods could be grouped into three categories: feature-

978-0-7695-3341-4/08 $25.00 © 2008 IEEE DOI 10.1109/AVSS.2008.13

79

10

80

9

60

8 7

40

6

20

Frame rate (fps)

CPU utility (%)

100

5 0 0

5

10 Number of objects

15

(a)

Figure 1. Illustration of system overload for a multiple object tracking system [10]. The solid curve illustrates that the CPU utility increases and saturates as the number of objects increases. The dashed curve shows that the frame rate decreases as the number of objects increases after the CPU utility reaches 100%.

information of the object of interest over a period of time [12, 13, 14]. The reduced frame rate leads to a decreased number of available frames and thus a decreased amount of accumulated information. As a result, the accuracy of consistent labeling deteriorates. Figure 2 illustrates the scenario where an overloaded surveillance system fails to discern a suspicious event. The example system has a frame rate of 4fps when performing multiple object tracking, as shown in Figure 2(a). The surveillance system suffers from observation leak and fails to detect the threatening event. When performing single object tracking, the system’s frame rate is 10fps as shown in Figure 2(b). The surveillance system successfully detects a threatening event that one pedestrian picks up an object and tries to hide it in his handbag. From the above illustration, the study of camera overload, in addition to consistent labeling, is another important criterion to be incorporated into camera handoff. In this paper, we model a multi-object tracking system as a Markov chain and derive the probability of camera overload according to objects’ dynamics and priorities. Based on the probability of camera overload, the camera’s limited resources are adaptively assigned to objects with different priorities. In so doing, we are able to dynamically select the objects with higher priority to track and avoid latent camera overload that may lead to a decreased frame rate. Equipped with adaptive resource management, our camera handoff algorithm is capable of not only minimizing the number of dropped objects during the handoff process but also maintaining a constant frame rate to avoid possible observation leaks. The resulting surveillance system has an improved competency in situational awareness and threat assessment. In summary, the contribution of this paper is a camera handoff algorithm with adaptive resource management that automatically and dynamically allocates resources to

(b) Figure 2. Illustration of observation leak for a surveillance system to track a different number of targets in the environment. (a) Frames sampled at a frame rate of 4fps when performing multiple object tracking. The surveillance system fails to detect the threatening event. (b) Frames sampled at a frame rate of 10fps when performing single object tracking. The surveillance system successfully detects a threatening event that one pedestrian picks up an object and tries to hide it in his handbag. An observation leak occurs because of the reduced frame rate.

80

the (j’)th camera. A positive handoff response for the ith

objects with different priority ranks. The proposed handoff algorithm can achieve a significantly improved handoff success rate and maintain a constant frame rate. The remainder of this paper is organized as follows. Section 2 illustrates the overall pipeline of our proposed camera handoff algorithm. Section 3 presents the adaptive resource management scheme. Experimental results are demonstrated in Section 4 and Section 5 concludes the paper.

object is granted if

The flow chart of the proposed camera handoff algorithm is shown in Figure 3, where operations are carried out at the handoff request and handoff response ends. Let the jth camera be the handoff request end and the ith object be the one that needs a transfer. To maintain persistent and continuous object tracking, a handoff request is triggered before the object of interest is untraceable or unidentifiable. Afterwards, the jth camera keeps tracking the ith object and waits for confirmation responses from adjacent cameras while the object is still visible. At the handoff response end, the (j’)th camera examines its current load. Let N th , j ', r denote the maximum number

of objects with a priority rank r that have been tracked by

Handoff request end

r

∑n Yes

Handoff Request

Target Visible

j ', k

< N th, j ',r

k =1

No

Yes

n j ',k < N th , j ',r .

Handoff response end

Target Traceable

Handoff Response

k =1

be adaptively adjusted according to the system’s current computational load. Given limited capacity, more resources should be allocated to objects with higher priorities at the cost of dropping out objects with lower priorities in order to maintain the required frame rate. Such a system provides a higher threat awareness level compared with systems where objects have the same priority ranks. Sometimes additional requirements on the overload probabilities of objects with different priority ranks are given. To meet these requirements, we also need an online learning process to automatically adjust the distribution of the capacities according to the estimated system load. Therefore, for a more efficient distribution of limited resources, an adaptive resource management algorithm is proposed and implemented at the response end. With the

of objects with a priority rank smaller than or equal to r that can be tracked simultaneously and n j ', r the number

No

r

Back to the handoff request end, if no positive handoff response is received before the jth camera loses track of the ith object, a handoff failure is issued. Otherwise, consistent labeling is carried out between the handoff request end and all available candidate cameras. The candidate camera that produces the most suitable observation is selected and the ith object is transferred to the chosen camera. To achieve a higher acceptance rate or equivalently a higher handoff success rate, the thresholds N th , j ', r should

2. Camera handoff

Yes

∑

Handoff Response

No

Handoff Reject

Consistent Labeling

Yes

Consistent Labeling Camera Selection

Resource Management

No

Handoff Failure

Update N th , j ', r

Handoff Success

Figure 3. Flow chart of the proposed camera handoff algorithm. Operations are carried out at the handoff request and handoff response ends. The adaptive resource management scheme is implemented at the handoff response end.

81

adaptively adjusted resource allocation, less number of objects would be dropped and hence a higher handoff success rate can be achieved. Furthermore, with the capability of actively selecting the objects with higher priorities to track, a constant frame rate can be maintained at the cost of dropping out objects with lower priorities if necessary.

P ( n) =

(

where N th,r −1

∑

µ

N pr k =r

⎡1 ⎢ ⎣⎢ n!

r −1

N pr λl N − N ) th,k th,k −1 l =k μ

∏ (∑ k =1

λk n − Nth,r −1 ⎤ ⎫ ) ⎥⎬ μ ⎦⎭

−1

.

(3)

∑ P ( n) .

(4)

The overload probability is one important criterion to evaluate the performance of a multi-camera system fulfilling multiple object tracking. It determines the number of objects that may be dropped due to limited resources. Therefore, in practice, it is desirable to distribute the resources dynamically according to the system’s current load and the object’s priority rank. From the above derivations, we find out that Nth,r determines the overload probabilities. Given the overload probabilities for objects at different priority ranks, we could adjust these thresholds to achieve the requirements. If the realtime estimated overload probability for the object with a priority rank r PˆO ,r exceeds the overload probability, we need to decrease the thresholds Nth,k with 1≤kFth, Nth,k is increased by one, where Fth is a predefined threshold. If Fk Pth ,r , the thresholds Nth,r-1 and Nth,r should be adjusted. Ideally, we want to increase Nth,r and decrease Nth,r-1. However, varying Nth,r-1 and Nth,r also affects the overload probability of objects from other priority ranks. In addition, the estimated overload probability PˆO ,r may

o, r

∑

O⎛⎜ nk ⎞⎟ . The adjustment of thresholds Nth,k has a ⎝ k =1 ⎠ computational complexity of O(Npr). As a result, the proposed resource adjustment is able to dynamically relocate the available resources with marginally increased computational cost in comparison with the complexity of multiple object tracking and consistent labeling.

fluctuate over time, which in turn induces unnecessary adjustment of the thresholds. Therefore, to smooth the decisions over a period of time and incorporate the requirements from objects of other ranks, a flag is set up for the thresholds at each priority rank, Fr. If PˆO , r > Pth , r , decrease Fr-1 by r suggesting that a decrease in Nth,r-1 is requested and increase Fr by r suggesting that an increase in Nth,r is preferred. Since it is cumulative, Fr takes the previous decision into consideration as well. If multiple handoff requests are received, the same procedure repeats for each object and the decisions from multiple objects are combined in Fk with k=1, …, Npr. The contribution in Fk from each object is associated with its

∑ No

Reject object

r

n k =1 k

< N th ,r Yes

Accept object

3.3. Example system To further study the effect of adjusting Nth,r, we consider a system with Npr=2 for example. Such a system has two types of objects with high and low priorities. Let λ H and

λ L be the arrival rate of objects with high and low priorities. The probability of n tracked objects is given by: P (0) λH + λL n ⎧ ( ) ⎪⎪ μ n! P (n) = ⎨ P (0) λH + λL N th λH n − N th ⎪ ( ) ( ) μ μ ⎩⎪ n!

FkFthÆ Nth,k=Nth,k+1,Fk=0 k=1,…, Npr

nr = nr+1

0 ≤ n ≤ N th

,

N th < n ≤ N max

(7)

with ⎛ N th 1 λ + λ L n ( H ) + P ( 0) = ⎜ ⎜ n! u n = 0 ⎝

∑

Fr-1 = Fr-1-r Fr = Fr+r

1 λH + λL N th λH n − N th ⎞⎟ ( ) ( ) ⎟ n! u u n = N th +1 ⎠ N max

∑

No

PˆO , r ≤Pth,r

nall,r = nall,r+1

N pr

Yes

.

(8)

−1

The overload probabilities for the object of high and low priorities are PO, H = P( N max ) and PO , L =

Update PˆO , r

∑

N max n = N th

P ( n) .

These two probabilities are monotonously increasing and decreasing functions of the threshold Nth as shown in Figure 6.

Figure 5. Flow chart of the proposed adaptive resource management scheme.

83

resource management in a real-life scenario. We used existing algorithms for multi-object tracking and consistent labeling. Background differencing and homography-based approaches are implemented for object tracking and consistent labeling, respectively. In order to examine the effectiveness of our proposed camera handoff algorithm, the algorithm discussed in [12] is implemented and serves as the comparison reference. The reference algorithm triggers a handoff request whenever the object of interest is close to the edges of the camera’s FOV disregarding the system’s load. The handoff success rate, the ratio between the number of successful handoffs and the total number of requested handoffs, is used to describe the system’s overall performance. To obtain a statistically valid estimation of the handoff success rate, simulations are carried out to enable a large amount of tests under various conditions. Several points of interest are generated randomly to form a pedestrian trace. The handoff success rate is obtained from simulation results of 300 randomly generated traces. We also tested the system’s performance using various ratios between the arrival rates of the objects with low and high priority ranks. The ratio λL λH is varied from 0.8 to 1.2. The expected probability of camera overload for objects with low and high priorities is Pth,L = 0.2 and Pth,H = 0.2. Figure 7 compares the performance of our adaptive resource management method and the reference algorithm [12] with various λL λH in term of the handoff success rate. The notation Adaptive-0.8 suggests a system using our proposed resource management method with λ L λ H = 0.8 and the notation static-0.8 means the

0

10 Overload probability

PO,L PO,H

Pth,L=0.2 -1

10

Adjusted value: Nth=5 Initial value: Nth=2

-2

10

1

2

3

4

5

6

Nth

Figure 6.

Illustration of the overload probabilities PO,H and

PO,L as functions of Nth.

λH μ

Suppose that we have

=2, λH μ

λL μ

=2,

= 1 , Nmax=6. λL μ

= 1 , and Nmax=6 and

that the overload probability of objects with low and high priorities should be below 0.2: Pth , L = Pth , H = 0.2 . The

Nth can be initialized by

∑ ∑

r

λ k =1 k N pr λ k =1 k

N max = 2 . The

corresponding PO,H and PO,L are 0.015 and 0.710, respectively. The PO,L is much higher than Pth, L . Our resource management algorithm is able to increase Nth by one at one time so as to decrease PO,L. At equilibrium, we arrive at Nth=5 resulting in PO,H=0.035 and PO,L=0.142. The overload probability of both types of objects is below 0.2. Figure 6 also depicts the adjustment process.

reference system with λL λH =0.8. The system equipped with our adaptive resource management can keep a steady frame rate of 8fps while the frame rate of the system based on the reference algorithm varies between 1fps and 8fps. In addition, in Figures 7(a) and (b), regardless of λL λH , the handoff success rate of our adaptive approach is higher than that of the static approach, indicated by a considerable improvement of 20%. The observed inferior handoff success rate of the reference method results from its fluctuating frame rate. When the frame rate is low, less information is acquired for the execution of consistent labeling, hence deteriorating the accuracy of identity matching and the overall handoff success rate. Figure 7(c) compares the CPU utility of our adaptive method with the reference static method. As expected, our adaptive method requires less CPU utility. Furthermore, the CPU utility of the reference method increases and saturates rapidly as the number of tracked objects increases. Figure 8 illustrates sampled frames at fn and fn+10 from a real-time tracking sequence with three static perspective cameras. Figure 8(a) depicts the cameras’ positions and the objects’ positions at frame fn+10. To visualize our experimental results, in Figure 8(b) bounding boxes with

4. Experimental results In the following experiment, eight static cameras with a resolution of 640×480 pixels are employed to monitor an indoor facility with dimensions 25m×10m×3m. The camera placement is optimized using a modified Erdem and Sclaroff’s method [16]. Two priority levels are assigned to the objects, Npr=2. The maximum number of objects that can be tracked simultaneously is three for all The discrepancy between the cameras, Nmax = 3. maximum number of tracked objects in Figure 1 and in our experiment is that the surveillance system in our experiment includes behavioral understanding in addition to multiple object tracking algorithm. The behavioral understanding part is necessary for assigning different priorities to tracked objects. As a result, the surveillance system illustrated in our experiment can only sustain at most three tracked objects without deteriorating the system’s frame rate. In other words, to generate Figure 1 the system only includes multi-object tracking. Thus, it can monitor 10 objects without deteriorating the frame rate. This observation also exemplifies the importance of

84

Object 3

Handoff success rate (%)

85

Object 1 Camera 3

80

Object 2

75

Object 4 Camera 1

70 65

Adaptive-1.2 Static-1.2 Apative-1.0 Static-1.0 Adaptive-0.8 Static-0.8

60 55 50

5

10 15 Time (minutes) (a)

Camera 2

(a) Camera 1

20 1

2

1

3

4

Handoff success rate (%)

75

2

3

70 65

Camera 2

60

4

55

Adaptive-1.2 Static-1.2 Apative-1.0 Static-1.0 Adaptive-0.8 Static-0.8

50 45 40

4

5

10 15 Time Time (minutes) (munites) (b)

20

Camera 3

100

4

2

4

CPU utility (%)

90 80

fn+10 (b) Figure 8. Illustration of our proposed adaptive resource management in camera handoff. (a) The placement and field of views of camera 1, 2, and 3, and (b) sampled frames at fn and fn+10 in a real-time tracking sequence. fn

70 Adaptive-1.2 Static-1.2 Adaptive-1.0 Static-1.0 Adaptive-0.8 Static-0.8

60 50 40 30

different colors represent objects that undergo different handoff stages. The tracked objects are marked by a green bounding box. The objects in the handoff request end and handoff response end are marked by a blue and yellow bounding box, respectively. To illustrate the effectiveness of the adaptive resource management, we focus on object 4. At frame fn, object 4 is tracked by camera 2. As object 4 leaves the FOV of camera 2, a handoff request is issued. At frame fn+10, object 4 can be observed by camera 1 and 3. Since camera

5

10 15 20 Time (minutes) (c) Figure 7. Performance comparison between our adaptive and the static resource management methods for various λ L λ H : (a) handoff success rate for objects with high priority, (b) handoff success rate for objects with low priority, and (c) CPU utility.

85

1 has reached its maximum computational load, camera 3 takes over object 4. We can see that our adaptively resource management is able to effectively guide camera handoff without deteriorating the system’s frame rate. This can reduce the probability of missing critical events and improves the system’s level of threat awareness. The maintained frame rate also stabilizes the performance of consistent labeling and leads to a higher handoff success rate.

coordinate frame”, IEEE Trans. on Pattern Recognition and Machine Intelligence, vol. 22, no. 8, pp. 758-767, 2000. [6] S. Calderara, A. Prati, R. Vezzani, and R. Cucchiara, “Consistent labeling for multi-camera object tracking”, Int’l Conf. on Image Analysis and Processing, Toronto, Canada, Sept. 2005, pp. 1206-1214. [7] J. Kang, I. Cohen, and G. Medioni, “Continuous tracking within and across camera streams”, IEEE Int’l Conf. on Computer Vision and Pattern Recognition, Madison, WI, Jun. 2003, pp. 267-272. [8] C. Beleznai, B. Fruhstuck, and H. Bischof, “Multiple object tracking using local PCA”, Int’l Conf. on Pattern Recognition, Hong Kong, Jun. 2006, pp. 79-82. [9] X. Luo and S. M. Bhandarkar, “Multiple object tracking using elastic matching”, IEEE Int’l Conf. on Advanced Video and Signal Based Surveillance, Como, Italy, Sept. 2005, pp. 123-128. [10] Y. Yao, B. Abidi, and M. Abidi, “Fusion of omnidirectional and PTZ cameras for accurate cooperative tracking”, IEEE Int’l Conf. on Advanced Video and Signal based Surveillance, Sydney, Australia, Nov. 2006, pp. 3925-3930. [11] M. Shah, “Understanding human behavior from motion imagery”, Machine Vision and Applications, vol. 14, pp. 210-214, Sept. 2003. [12] S. Khan and M. Shah, “Consistent labeling of tracked objects in multiple cameras with overlapping fields of view”, IEEE Trans. on Pattern Recognition and Machine Intelligence, vol. 25, no. 10, pp. 1355-1361, 2003. [13] F. Fluret, J. Berclaz, R. Lengagne, and P. Fua, “Multicamera people tracking with a probabilistic occupancy map”, IEEE Trans. on Pattern Recognition and Machine Intelligence, vol. 30, no. 2, pp. 267-273, 2008. [14] S. Guler, J. M. Griffith and I. A. Pushee, “Tracking and handoff between multiple perspective camera views”, IEEE Applied Imagery Pattern Recognition Workshop, Washington DC, Oct. 2003, pp. 275-281. [15] L. Huang, S. Kumar, and C.–C. Jay Kuo, “Adaptive resource allocation for multimedia QoS management in wireless networks”, IEEE Trans. on Vehicular Technology, vol. 53, no. 2, pp. 547-558, Mar. 2004. [16] U. M. Erdem and S. Sclaroff, “Automated camera layout to satisfy task-specific and floor plan-specific coverage requirements”, Computer Vision and Image Understanding, vol. 103, no. 3, pp. 156-169, 2006. [17] X. Luo and S. M. Bhandarkar, “Multiple object tracking using elastic matching,” IEEE Conf. on Advanced Video and Signal Based Surveillance, Como, Italy, Sept. 2005, pp. 123-128.

5. Conclusions In this paper, we modeled multi-object tracking as a Markov chain and derived the probability of camera overload for objects with various priorities. Based on the probability of camera overload, we developed an adaptive resource management algorithm according to system’s current load in order to adaptively allocate the limited computational resources among multiple objects with different privileges. Experimental results illustrated that our handoff algorithm equipped with the adaptive resource management scheme outperforms the method described in [12] by achieving a higher camera success rate and a more stable frame rate. This improves the reliability and level of situational awareness of the surveillance system for continuously tracking multiple objects across multiple cameras.

6. Acknowledgements This work was supported in part by the University Research Program in Robotics under grant DOE-DEFG52-2004NA25589. The authors would like to also thank Dr. Andrea Cavallaro, Dr. Ashok Veeraraghavan, and Dr. Carlo Regazzoni for providing valuable feedback for this paper.

References [1] M. Balcells, D. DeMenthon, and D. Doermann, “An appearance-based approach for consistent labeling of humans and objects in video, pattern and application”, Pattern Analysis & Application, vol. 7, no. 4, pp. 373385, Dec. 2004. [2] D. G. Lowe, “Distinctive image features from scaleinvariant keypoints”, Int’l Journal of Computer Vision, vol. 60, no. 2, pp. 91-110, 2004. [3] P. Kelly, A. Katkere, D. Kuramura, S. Moezzi, S. Chatterjee, and R. Jain, “An architecture for multiple perspective interactive video”, ACM Int’l Conf. on Multimedia, San Francisco, CA, May 1995, pp. 201-212. [4] J. Black and T. Ellis, “Multiple camera image tracking”, Image and Vision Computing, vol. 24, no. 11, pp. 12561267, Nov. 2006. [5] L. Lee, R. Romano, and G. Stein, “Monitoring activities from multiple video streams: establishing a common

86