Development of an Angular Characterisation System for ...

Development of an Angular Characterisation System for Cooperative UAV / UGV Applications Paul Thompson & Salah Sukkarieh ARC Centre of Excellence for Autonomous Systems, Australian Centre for Field Robotics. The Rose St. Building, J04, University of Sydney, NSW, Australia 2006 {p.thompson,salah}@acfr.usyd.edu.au Summary. This paper develops a theory for the decentralised estimation of angular profiles for point characterisation. Angular profiles are an example of spatially distributed characterisation which may be further processed for classification and identification. This paper also describes the development of a vision system for integration into a decentralised data fusion system for tracking and characterisation. Visual results are presented showing the field setup and sensor observations.

Key words: characterisation, decentralised, vision, angular profile, information theoretic properties

1 Introduction This project extends our previous work on multi-air vehicle decentralised data fusion (DDF), which used point feature detection for object tracking, localisation and mapping [1]. This paper discusses our developments in the incorporation of point characterisation for detecting and identifying low signature, partially obscured or hidden objects. Multiangular image processing is common in geoscience applications for land classification [2]. A wide variety of application specific domains implement angular characterisation, but generally not in a recursive estimation framework, let alone in a method amenable to probabilistic decentralised data fusion. In SLAM, tracking and structure from motion, point features are not generally considered to possess interesting properties as a function of viewing angle. Multiangular characterisation in a decentralised, real time object tracking environment is a new approach. The rest of this paper is laid out as follows: Section 2 introduces our approach, explaining the purpose of characterisation and angular profiles in particular.

2

Paul Thompson & Salah Sukkarieh

Section 3 describes the setup of our vision system and field tests. Section 4 shows visual results from the field tests. Section 5 develops a technique for the decentralised estimation of angular profiles, with simulated demonstrations. Section 6 outlines directions for future development.

2 Approach Our aim is to incorporate identity and class of objects into our existing sensing and decentralised data fusion system [1, 3] and to include this in the development of an information theoretic control layer. In general, we are proposing to perform feature characterisation from sensor data prior to performing identification upon the characterisation results. These techniques incorporate spatially distributed observations and build spatially distributed representations of objects to provide a rich basis for classification. Angular dependence of properties is one particular dimension we are considering for characterisation. This is based on the challenges of feature identification and classification from a single frame or scan, especially given the possibility of hidden or obscured objects. Characterising angular properties requires observing from a wide range of angles. This behaviour will be useful for characterisation in the form of angular profiles described here and also for other characterisations such as estimating 3D structure [4] and bearing only localisation. In this paper, recent results are shown from a flight trial involving circling around features of interest.

3 Experimental Setup The vision system is mounted as a nose module on our UAV, as shown in figure 1(a). The colour camera is a Sony XCD-X710CR. This was chosen for its high resolution (1024 × 768) and high frame rate (up to 30 fps at full size). The colour camera interfaces via a standard firewire device for commanding and image transfers. The trigger input on the camera is driven from a parallel port output signal driven from a custom Linux kernel module for periodic events and timestamping. Both cameras are mounted in fixed positions, along the lateral axis of the UAV. This geometry is consistent with the need for viewing ground features from many angles and consistent with the flight parameters of the UAV. The UAV flies in arc trajectories around the ground features with the bank angle, forward velocity, relative altitude and turn radius matched to ensure that the sensor footprint covers the ground feature and that flight parameters are kept within safe limits. Details of the UAV control architecture are given in [5].

Angular Characterisation

#

*

+ + x

3

IR Camera Colour Camera

x #

*

PC104 stack Logging Drive

(a) Sensors setup

(b) UAV in flight

(c) Setup for ground based observations

Fig. 1. Sensors, flight vehicle and ground vehicle setup

Ground vehicle observations were obtained using the same vision system, as shown in figure 1(c). Flight tests were carried out in a rural environment consisting of natural and pre-existing man-made features. Some typical features can be seen in figure 2. These include isolated trees and vegetation patches, live animals (cattle), long straight fences with junctions and corners, straight and curved roads (including the airstrip), a continuous curved river, isolated dams and water tanks, sheds and other buildings.

Ground Vehicle

White Squares

Trees (live, dead)

Ground Vehicle tracks

(a) Trees, white squares (b) Horizon, and ground vehicle ture

Infrastruc- (c) river and vegetation

Fig. 2. Environmental setup

Various artificial features were introduced into the environment; white squares, patterned cylinders and a vehicle. The white squares are 1×1 metre plastic squares which are highly visible from the air and ground. These were surveyed in position using DGPS to act as fiducial markers for the confirmation of tracking and image registration results (of their own position and of nearby natural features). White squares are the most basic object for tracking purposes. The patterned cylinders are 2.5 metres high by 1.5 metres diameter cylinders with a printed pattern wrapped around the circumference. The vehicle acts as a realistic test case for tracking and classification, (see figure 3(b)).

4

Paul Thompson & Salah Sukkarieh

4 Flight Imagery Results are shown here for the images obtained from the UAV mounted vision system. Images will be subjected to feature extraction and used for construction of angular profiles of the various natural and artificial features. Figure 3 shows the ground vehicle as viewed from multiple positions during flight. Images in figure 2 were also obtained from the UAV vision sensor. 6

x 10

Marulan, 1 metre per pixel vehicle position observation points flight tracks

6.1679

6

x 10

Marulan, 1 metre per pixel

6.1678

6.1679 6.1677

6.1678

6.1677

6.1677

6.1676

X - Northing MGA (m)

X - Northing MGA (m)

6.1678

6.1678

2.297 2.2975 2.298 2.2985 2.299 2.2995 2.3 5 Y - Easting MGA (m) x 10

6.1677

6.1677

6.1677

6.1676 2.297 2.2975 2.298 2.2985 2.299 2.2995 2.3 5 Y - Easting MGA (m) x 10

(a) A sample of observation positions. Dots indicate the UAV vehicle positions for each observation shown below.

(b) A selection of views of the vehicle. Each row is obtained from a single pass of the UAV past the site.

Fig. 3. Images from the flight data

5 Estimation of angular profiles This section describes an approach to the estimation of angular profiles. Angular profiles will be important to the development of identification and classification algorithms (for example, [2]) and will play a role in directing information theoretic control algorithms into circling features of interest. Angular profiles can be considered as a container for observables which vary in angle. It is a subject of future development to incorporate the extraction of such observables from images, however the angular profiling technique is demonstrated here with preliminary observations. Possible observables include geometry, colour, texture and outputs from single-view image processing techniques. Angular profiles can be considered as a useful intermediate between image fusion and three dimensional object estimation on the one hand, and the fusion of single aspect classification results on the other [6]. Angular profiles are cast in an information filter (inverse covariance) form [7]. Recent developments from the SLAM community in large state space management in information form will be beneficial in the implementation [8]. State Description. An angular profile of an object is a function  x(θ). The function x(θ) is discretised into a finite set of angles θ1 · · · θN covering all

Angular Characterisation

5

angles 0 6 θi < 2π. Each xi = x(θi ) is a continuous scalar, xi ∈ R. The state  T space for estimation consists of the state vector X = x1 · · · xN . An angular profile covering two dimensions of angle is similarly a function x(θ, φ) discretised into a finite set covering 0 6 θi < 2π and −π 6 φi < π . Problem Description. The challenge in the theory behind estimation of angular profiles lies in describing the correlations between angular states such that the profile behaves well during estimation. In particular, it is desired to apply a consistency model, to reflect the fact that there can not be sudden jumps in the profile across small changes in angle. This consistency model plays the role of a process model in ordinary scalar estimation, in that it serves to smooth out the noise presented by observations. Consistency models are required in order to smooth out observation noise for observations from multiple angles. By analogy with conventional scalar estimation, process models smooth out observation noise for observations from multiple points in time. Consistency models are required so that correlations can be propagated around the profile such that observations at one angle affect the estimates at all other angles. By analogy with conventional scalar estimation, process models describe correlations such that observations affect future estimates. This means that consistency models are as fundamentally important to the estimation as process models are to conventional estimation and that consistency models play the same role in the estimation as conventional process models. 5.1 A Differential Observation Technique for Angular Consistency Models This section describes a technique for constructing the correlations between the angular profile states. This section is described with reference to a onedimensional angular profile. The extension to two dimensions is described in section 5.3 A consistency model for the angular profile is assumed, taking the form given in equation 1 and re-arranged in terms of all states as in equation 2. The consistency model in examples and demonstrations here is a constant value model, F = I. This consistency model is an observation of the difference between successive states. The true difference is w, which is modelled as zero mean Gaussian noise of covariance Rc . Rc , Zc and Hc refer to the parameters of the consistency model update.



0 · · · −F I · · · 0



x0 · · · xk−1

xk = Fxk−1 + w (1) (xk − Fxk−1 ) = w T xk · · · xN =w (2)

6

Paul Thompson & Salah Sukkarieh

Equation 2 is of the form given in equation 3 Zc = Hc X + v Zc = [0] v=w

(3)

  Hc = 0 · · · −F I · · · 0   E [w] = 0 E wwT = Rc

The consistency model is applied to each pair of successive states, resulting in a concatenated Hc matrix, shown in equation 4 for an example with five states.     −F I 0   0 −F I        −F I 0 (4) H = Zc =  c      0 −F I  I −F 0 The final row in equation 4 is a periodic boundary condition which observes the difference between the first (x1 ) and last (xN ) state elements1 . The angular profile must be periodic by definition, i.e. x (0) = x (2π). This consistency model observation can be applied as an observation update. Here we assume that an Information filter (inverse-covariance) representation of the state variables is being used. The Fisher Information matrix, Y, is initialised to zero to represent infinite uncertainty. The consistency model observation becomes an addition to Y, as shown in equation 5 Y = 0 + Hc T Rc −1 Hc

(5)

The resulting Y information matrix is shown by example in equation 6, with Rc −1 = I and F = I.   2 −1 −1   −1 2 −1     −1 2 −1 (6) Y=   −1 2 −1  −1 −1 2 Note that the information matrix is sparse. The number of non-zeros is given by 3N n where N is the number of angles in the discretisation and n is the size of the state at a given angle. Note that at this point det(Y) = 0, as there have been no genuine observations. The rank of Y is N − 1. There is zero information along the sum of all states eigenvector but all others are nonzero.

1

Note that x1 6= xN , since θ1 6= θN (they are separated by one interval)

Angular Characterisation

7

Observations. The most basic observation simply observes the value at one angle θi , updating xi . In this case the
observation H matrix selects one particular angle state eg: H = 1 0 · · · 0 . Note that there is a data association problem involved in choosing which angle state to associate an observation with. Using the Information filter observation update equation, observations of x at particular angles are made by information addition: Y+ = Y− + HT R−1 H

(7)

The form of the information update results in additions to the diagonal of Y, retaining the sparse layout regardless of the number of observations. This technique inherits the scalability and decentralisation properties of the information filter. Multiple observations are fused recursively without increase in the size of the representation. Furthermore, the information matrix is always highly sparse, in a banded pattern. By arriving at an information form description of the angular profile, the technique is readily applicable to incorporation into our existing decentralised data fusion architecture [1]. 5.2 Angular Profile simulations The following simulations demonstrate the theory. A true angular profile was created from an arbitrary smooth function. Observations were taken from the true profile with zero mean Gaussian noise consistent with the observation model. To plot the angular profile, the information matrix and vector are converted to covariance and estimate. The estimated angular profile is then given by x. Error bounds at each angle are given by the diagonals of P. Figure 4(a) shows the angular profile after two observations. There is no direction imposed on the propagation of uncertainty around the profile; it is symmetrical in both directions. Uncertainty grows as one moves away from angles where an observation was made. This is reflected in the structure of the covariance matrix, Figure 4(f). Note that correlations exist broadly across regions without observations, yet the information matrix remains sparse and banded. Note that the individual observations become simple additions onto the diagonal with almost no processing required for successive observations (especially not a matrix inversion). The profile construction problem is essentially deferred until the information matrix is inverted. 5.3 A Two Dimensional Angular Consistency Model Section 5.1 described the development of the consistency model for a circular (one dimensional) angular profile. This section extends the structure of the consistency model to cover two dimensions, forming a surface. Equations 1 to 3 refer to the application of a consistency model on pairs of elements according to their adjacency on the profile, including a periodic boundary condition. This technique applies to pairs of states in no special order, therefore it is possible to apply the consistency model to states which are adjacent in an undirected graph of arbitrary structure. The undirected

8

Paul Thompson & Salah Sukkarieh +ve 3sig bound estimate -ve 3sig bound truth observations

2

90

4

6

8

10

12

14

2

180

16 2

0

4

6

8

10

12

14

16

(b) Y matrix after 2 observations 2

4

6

8

10

12

14

270

16 2

(a) Profile after 2 observations (simulated)

+ve 3sig bound estimate -ve 3sig bound truth observations

4

6

8

10

12

14

16

(c) P matrix after 2 observations 2

90

4

6

8

10

12

14

2 16 2

180

0

4

6

8

10

12

14

16

(e) Y matrix after 32 observations 2

4

6

8

10

12

14

270

(d) Profile after 32 observations (simulated)

16 2

4

6

8

10

12

14

16

(f) P matrix after 32 observations

Fig. 4. Simulated Angular profiles.

graph structure for the one dimensional profile is a simple ring. The undirected graph structure for the two dimensional profile is a spherical mesh. The spherical mesh is generated from DistMesh [9]. The mesh is uniform overall but not perfectly regular. The positions of the nodes and the mesh connectivity is constant throughout the estimation process. In future revisions the irregular mesh will be replaced by a regular mesh with identical edge lengths and number of edges per node in order to ensure symmetry and uniformity.

Angular Characterisation

9

The consistency model is described by an Hc matrix equivalent to equation 4. This matrix is formed by iterating through each of the graph edges of the mesh structure, where each edge yields one row of Hc . For each edge, where the edge is between nodes i and j: Hcrow,i = I

Hcrow,j = − I

(8)

The consistency model observation becomes an addition to Y, as in equation 5. 5.4 Two Dimensional Angular Profiles Demonstration Figure 5 shows a demonstration of angular profiles from our flight vehicle and ground vehicle. The patterned cylinder object was characterised according to the metric area of the object as viewed from the (air or ground) borne image sensor. This is a preliminary observable for demonstration of the estimation structure. Figure 5(d) shows the separate contributions from the air and ground vehicle, which are separate due simply to their differing angles of elevation. The fusion of information from air and ground is simplified by the use of angular profiles because they allow explicit differences in value at viewing angles. Hence it is not required that features be absolutely identical from air and ground. Figures 5(a) and 5(b) are shown at the same orientation. The peaks in profile information correspond to the groups of observation points where multiple observations have been fused. Regions without observations take on an estimate obtained through the network of consistency models, causing those regions to have non-zero information. 5.5 Information Theoretic Properties of Two Dimensional Angular Profiles One application of angular profiles is in causing information theoretic control schemes [10] to explore multiple viewing angles of point features (in addition to spatial exploration over multiple features). This section describes the properties of the determinants of the information matrices of angular profiles. The entropic information i of an n-dimensional Gaussian variable with Fisher information, Y and the mutual information I between two alternate information matrices Ya and Yb are given by: h i n a| (9) i = 12 log [(2πe) |Y|] I = 12 log |Y |Yb | Angular profiles determinants have the following properties: • After application of the consistency model but before observations, |Y| = 0. This means that Y retains the properties of a non-informative prior after application of the consistency model. • A single observation causes the determinant, |Yb |, to be non-zero.

10

Paul Thompson & Salah Sukkarieh 5

Cyclinder and View positions

x 10 2.297

Cylinder Ground Obs Air Obs

Easting MGA (m)

2.2975

−1500−1000 −500

0

500 1000 1500 2000 −1000

2.298

−500

2.2985

0

2.299

500 2.2995

1000

2.3

1500

2.3005 6.1675

6.1676 6.1677 6.1678 Northing MGA (m)

6.1679 6

2000

x 10

(a) Cylinder Object location and air and ground viewing positions

(b) Radius represents the profile information (inverse covariance). The orientation matches that of 5(a) Air vehicle contributed information

-500 0 500 1000 1500 −4

−2

0

-1500 -1000 -500

0

500

1000 1500 2000 -1000

2

-500 −2

0

−1 0 1 2 3 4

(c) Radius represents the profile estimate (Projected area of the object, m2 )

Ground vehicle contributed information

(d) Radius represents the profile information (inverse covariance). Information peaks correspond to air and ground observations

Fig. 5. Angular Profiles Demonstration

The mutual information properties of angular profiles are distinct from those of three dimensional bearing only point localisation [10]. Given a single observation, the next observation to maximise information gain should be 180 degrees around the profile. A sequence of adjacent observations optimised for information gain explores all angles. 5.6 Other Applications and extensions The method of interpreting prediction models as differential observations used here to develop a technique for developing the consistency models for angular profiles can be applied to other problems in the estimation of spatially distributed states. In particular, it could be applied to the estimation of the trajectory of near-linear features such as fences, roads and rivers presented by our field site.

Angular Characterisation

11

Interpreting prediction models as differential observations can also be applied to temporal estimation. It is a subject of future investigation to compare this to other treatments of delayed and asequent data handling [11] and to other smoothing formulations of estimation. [12] Interpreting spatial consistency models and temporal prediction models as differential observations (in space a time respectively) allows one to describe consistency in space and time simultaneously. This provides a method for simultaneously estimating spatially distributed random fields and providing temporal smoothing (spatio-temporal estimation). This can be compared to the spatial Kalman filtering described in [13] and [14]. It will be necessary to describe temporal process models for the angular profiles, primarily to introduce uncertainty over time. There are difficulties involved with handling observations of the angular profile from uncertain angles. As described, the technique treats the angular states as fixed on a set of angles around the object and so observations must be subject to data association to choose the angle to update.

6 Conclusion and Future Work In this paper we introduced our project and approach, described the vision system and environment. We introduced a theory for the estimation of angular profiles with demonstrations from simulation and field data. In future developments we will be incorporating the image processing algorithms and observation models necessary to observe angular profiles as described here. We will be revising the decentralised data fusion system to allow greater flexibility in the choice of states associated with each feature in order to support the communication and fusion of angular profiles. Feedback from the angular profile states and localisation states will need to be used simultaneously for information theoretic decentralised control. As discussed in section 5.5, an angular profile of a single feature has well behaved properties in entropy and mutual information, causing decentralised control algorithms to explore not only different positions in space but different angles of view. However, implementing angular profiling alongside localisation presents many challenges. The technique of angular profiling is limited by the choice of the profiled observable. For general vision based applications it may be preferable to focus on methods for estimating the three dimensional geometric structure and colour or itensity of regions, rather than relying upon low dimensional remote observables. However, the ability of angular profiles to provide an entropic measure of angular information coverage is a relevant and beneficial feature.

Acknowledgments This project is supported by the ARC Centre of Excellence programme, funded by the Australian Research Council (ARC) and the New South Wales State Government. This project is supported by BAE Systems, Bristol, UK.

12

Paul Thompson & Salah Sukkarieh

References 1. Salah Sukkarieh, Eric Nettleton, Jong-Hyuk Kim, Matthew Ridley, Ali Goktogan, and Hugh Durrant-Whyte. The ANSER project: Data fusion across multiple uninhabited air vehicles. The International Journal of Robotics Research, 22(7-8):505–539, 2003. 2. Nadine Gobron, Bernard Pinty, Michel M Verstraete, Jean-Luc Widlowski, and David J. Diner. Uniqueness of multiangular measurements. IEEE Transactions on Geoscience and Remote Sensing, 40(7):1574 – 1592, 2002. 3. Eric Nettleton. Decentralised Architectures for Tracking and Navigation with Multiple Flight Vehicles. PhD thesis, The University of Sydney, 2003. 4. Frank Dellaert, Steven M. Seitz, Charles E. Thorpe, and Sebastian Thrun. Structure from motion without correspondence. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2:557 – 564, 2000. 5. D.T. Cole, S. Sukkarieh, A.H. Goktogan, H. Stone, and R. Hardwick-Jones. The development of a real-time modular architecture for the control of uav teams. In The 5th International Conference on Field and Service Robotics, July 2005. 6. Ed Waltz. Handbook of Multisensor Data Fusion. The Principles and Practice of Image and Spatial Data Fusion. CRC Press, 2001. 7. Peter S. Maybeck. Stochastic models, estimation, and control, volume 1 of Mathematics in Science and Engineering. 1979. 8. Sebastian Thrun, Yufeng Liu, Daphne Koller, Andrew Y. Ng, Zoubin Ghahramani, and Hugh Durrant-Whyte. Simultaneous localization and mapping with sparse extended information filters. The International Journal of Robotics Research, 23(7-8):693–716, 2004. 9. Pen-Olof Persson and Gilbert Strang. A simple mesh generator in matlab. SIAM Review, 46(2):329 – 345, 2004. 10. Ben Grocholsky. Information-Theoretic Control of Multiple Sensor Platforms. PhD thesis, Australian Centre for Field Robotics Department of Aerospace, Mechatronic and Mechanical Engineering The University of Sydney, 2002. 11. Eric W. Nettleton and Hugh F. Durrant-Whyte. Delayed and asequent data in decentralised sensing networks. Proceedings of SPIE - The International Society for Optical Engineering, 4571:1 – 9, 2001. Decentralised sensing networks. 12. Robert F. Stengel. Optimal Control and Estimation. Dover, 1994. 13. K.V. Mardia, C. Goodall, E.J. Redfern, and F.J. Alonso. The kriged kalman filter. Test (Trabajos de Estadstica), 7(2):217–285, December 1998. 14. Noel Cressie and Christopher K. Wikle. Space time kalman filter. Encyclopedia of Environmetrics, 2002.