An implementation and performance evaluation of a ...

An implementation and performance evaluation of a space variant OT-MACH filter for a security detection application using FLIR sensor Akber Gardezi, Ahmed Alkandri, Philip Birch, Tabassum Qureshi, Rupert Young, Chris Chatwin School of Engineering and Design University of Sussex Falmer, Brighton BN1 9QT

Abstract—A space variant Optimal Trade-off Maximum Average Correlation Height (OT-MACH) filter is designed specifically for images acquired from a forward looking infrared (FLIR) sensor, using the maximum of the power spectral density (PSD) of the input image instead of the white noise covariance factor. The kernel can be locally modified depending upon its position in the input frame, which enables adaptation of the filter dependant on background heat signature variances and also enables the normalization of the filter energy levels. The detection capabilities of the filter were evaluated using different data sets of real images and 3D models for a suspected threat in order to define a thresholding parameter. The parameter was based on peak to correlation energy (PCE) and peak to side lobe ratio (PSR) of the correlation output which led to the definition of a criterion for predicting true and false detections. The hardware implementation of the system has been discussed in terms of FPGA versus DSP chipsets and a performance benchmark has been created using millions of multiply-accumulate operations per second (MMAC) and the cost. In this paper we propose an implementation and performance evaluation of a security detection application which uses a space variant OT-MACH filter with different data sets. Also a performance benchmark has been created for the hardware implementation of the proposed system on popular FPGA and DSP chipsets. Keywords: MACH filter; correlation filter; space variant filter; pattern recognition; thermal images

I.

INTRODUCTION

The detection and recognition of targets in FLIR images has always been a challenging problem due to the varying heat signature of the object and background clutter. The FLIR imagery used in this paper has been acquired from a moving platform and contains multiple objects at different orientations and background variations. The movement of the sensor and the object induce coupled motions into the FLIR images which make the detection and tracking of the target object extremely complicated due to varying criteria. Considering all the factors and taking into account the variable nature of FLIR images it was decided to use the Optimal Trade-off Maximum Average Correlation Height (MACH) filter [2] due to its tunable nature.

An infrared sensor can detect infrared radiation and by converting the temperature differences between the object and the surroundings creates a temperature scale. The temperature scale is converted into a grayscale map and an image is obtained. The images obtained from such a sensor enables the detection of an object through smoke in a burning building, adverse weather conditions or in the absence of any reflected light. The OT-MACH filter can be used to maximize the height of the correlation peak in the presence of distortions of the training object and provide resistance to background clutter. One of the major characteristics of the OT-MACH filter is that it can be tuned to maximize the height and sharpness of the correlation peak by using trade-offs between distortion tolerance, peak sharpness and the ability to suppress clutter noise [4]. It is commonly known as the Optimal Trade-off Maximum Average Correlation Height Filter (OT-MACH) [13]. However, the peak height of the MACH filter is unconstrained making it more difficult to use a predefined threshold function [3]. The transfer function for the OT-MACH filter (in the frequency domain) is given as: h=

m ∗x αC + βDx + γS x

(1)

Where α, β and γ are non-negative OT parameters, m x is the average of the training image vector , and C is the diagonal power spectral density matrix of additive input noise. Normally a white noise covariance matrix is used as additive noise but in this case, due to varying heat signatures, the maximum power spectral density of the input image is added as noise. Dx is a diagonal matrix with the average power spectral density of the training images. In this paper emphasis has been made on the spatial domain implementation and performance evaluation of the OT-MACH filter on Field Programmable Gate Array (FPGA) chips and Intel Core 2 Duo processors for target recognition in

FLIR images. The spatial domain implementation offers the advantage that it does not have shift invariance imposed on it as the kernel can be modified depending upon its position within the input image. This allows normalization of the kernel and allows inclusion of a space domain non-linearity to improve performance [10]. This is discussed in more detail in section IV. II.

SPACE VARIANT OT-MACH

When working with FLIR images there a number of well known challenges involved such as: significantly high levels of variability of thermal signatures; aspect ratio; location; no prior knowledge; weather conditions and also time of day. Other known issues comprise: inconsistencies in the signature of the targets; similarities between the signatures of different targets; limited training data and the non repetitive nature of the target’s heat signature. The FLIR images are obtained by measuring the radiation in the infrared spectrum being emitted or reflected by the objects. Because of this the images acquired from an infrared sensor can have extremely low signal to noise ratio (SNR) which limits the information used in detecting an object. Also in FLIR images the object edges and corners are smoothed out and this causes problems in discriminating different objects. In the case of moving objects the generation of kinetic energy causes them to appear brighter than the background [14]. Considering the above mentioned factors, the OT-MACH filter is implemented in the spatial domain using a 256x256 pixel moving kernel to detect an object in a cluttered background plane of size, typically, between 512x512 and 1024x1024 pixels. To implement the kernel, the filter is designed in the frequency domain using the following adaptation of Equation 1: h=

m x∗

α C + β D x + γS x

(2)

The 512x512 spectrum is inverse Fourier transformed to yield a space domain image from Equation 2. This is truncated to the target object size which is, typically, less than 256x256 pixels. In addition, care must be taken to store the image as a bi-polar array (rather than an intensity image). The square root operation in the denominator of Equation 2 is required because the filtering operation must also be applied to the input image separately so this is pre-processed with the OT-MACH transfer function prior to its spatial domain correlation with the OT-MACH kernel function. The use of a 256x256 kernel enables the filter to be applied on smaller areas of the image and hence minimize the impact of varying heat signatures. III.

PERFORMANCE RATIOS

In order to define the criteria of detection it is necessary to calculate some basic measures to represent the quality of the output correlation plane. The most basic measure of the

correlation plane is the correlation output peak intensity (COPI) which is defined as [9]: COPI= max{| C( x,y ) |² }

(2)

Where C(x, y) is the output correlation plane’s amplitude at position (x, y) To provide the best detection capability and performance it is necessary for a filter to yield a sharp correlation peak as well as a high COPI value, while keeping side lobes to a minimum. The filter’s ability to do this can be measured using the peak-to-correlation energy (PCE) measure [9]. The basis of the PCE measure is that the correlation peak intensity should be as high as possible while the overall correlation energy in the plane should be as low as possible. A high PCE value therefore implies that the filter performed well. The PCE ratio is generally defined as: PCE =

COPI Energy

(3)

Also another parameter in defining the performance of a correlation filter is Peak to Side lobe ratio (PSR) which forms an important criterion especially in the case when multiple objects are present in the input scene. In most cases considering just the COPI values would not be enough to analyze the performance of the filter. The PSR is calculated by subtracting the mean of the correlation plane from the COPI and than dividing it from the standard deviation of the correlation plane as shown in Equation 8 below: PSR=(COPI-MEAN)/

(4)

Where ԢߪԢ denotes the standard deviation of the correlation plane. The PSR of the target object should be greater than for any background clutter peaks. It can be used in the case when the difference between the two COPI values of target and reference objects is negligible. From these equations the performance criteria for detection of a target can be quantified, giving an indication of how well the filter is performing. Based on these defined criteria the target recognition capabilities of the OT-MACH filter are analyzed. IV.

TARGET RECOGNITION

In this paper a space variant design for the OT-MACH is proposed and the target recognition capabilities have been evaluated in different scenarios relating to perimeter breaches for security applications. The FLIR imagery has been provided by the courtesy of the KUWAIT Ministry of Defense (MOD) which covers a harsh desert terrain for testing purposes. A 3D Computer Aided Design (CAD) model of a Nissan Patrol (NP) oriented

from 0 to 360 degrees with a 10 degree shifft between every sample has been used generate training imaages as shown in “Fig. 1”.

Figure 3. Correlation output plane p for NP at 0 degree

Figure 1. 3D Model of Training Imaages

The training images have been tested witth real life FLIR imagery of a Nissan Patrol entering a secuured perimeter in harsh weather conditions to evaluate the detecction capabilities of the filter. The designed OT-MACH is invvariant to limited in-plane and full out-of-plane rotations and can be tuned to define the detection criteria. In order to detect the object in a cluttered environment the training set as shown in “Fig. 1” is preprocesssed using the OTMACH filter and then correlated using the spaatial correlator to locate the object in a background scene. T The preprocessed image obtained from the training images is shhown in “Fig. 2” given below.

The correlation plane shown in “Fig. “ 3” has a sharp peak at the center which shows that the co orrelation filter successfully matched the reference object with th he target object in this case the NP image. The correlation plane p in “Fig. 3” has the performance ratios as shown in Tab ble I given below. TABLE I. CORRELATION PLANE RATIOS FO OR NP ORIENTED AT ZERO DEGREE COPI

PSR

PCE







0.0795

98.12

0.1603

0.0001 0

0.2

0.001

n TABLE I show that the The performance ratios shown designed correlation filter has COPI and PSR values for successful recognition of the targett object. Now the designed filter must be tested with different orientations and with cluttered backgrounds to evaluate th he performance. n angle of 180 degrees with In “Fig. 4” a NP is oriented at an a telemetric data feed being seent by the FLIR sensor superimposed on the images. The performance p of the filter is evaluated by judging its capability to t distinguish the NP in the presence of this and background clu utter.

Figure 2. Preprocessed training images from 00-360 degrees

In order to test the filter cross correlationn was performed using the 3D model of the NP oriented at zeero degrees. The correlation peak achieved is shown in “Fig. 3””.

Figure 4. NP at 180 degree with w background clutter

Correlating the NP template with the images shown in “Fig. 2” results in a sharp peak in the correlation output plane shown in “Fig. 5” below.

the constant improvements in the processing speeds of these devices and built-in look up tables an efficient spatial domain convolution processor could be built. V.

IMPLEMENTATION AND PERFORMANCE EVALUATION

In order to design a system say 10 – 20 years ago the designer had to bear a very high cost due to the fact that rapid testing and re-design was constantly required. In the computing industry Application specific integrated circuits (ASIC) are used to form non-reconfigurable but fast systems being manufactured on silicon wafers of a specific die. The cost of designing ASICs is increasing every year. In addition to the non-recurring engineering (NRE) and mask costs, development costs are increasing due to ASIC design complexity. Issues such as power, signal integrity, clock tree synthesis, and manufacturing defects can add significant risk and time-to-market delays. To overcome the risk of recalls, high NRE costs, and to reduce time-to-market delays, FPGAs offer a viable and competitive option to ASIC development.

Figure 5. Correlation output plane for NP at 180 degree

It can be seen in “Fig. 5” that there is a sharp peak at the location of the target object but there are also visible side lobes. The performance ratios for the correlation plane in “Fig. 5” are given in Table II below: TABLE II. CORRELATION PLANE RATIOS FOR NP ORIENTED AT 180 DEGREE COPI

PSR

PCE







2.78*10^-5

9.29

0.0017

0.0001

0.2

0.001

All of the above tests were conducted with the OT-MACH parameters kept constant to evaluate the changes in the correlation output plane and calculate the performance ratios accordingly. From “Fig. 5” and TABLE II it is shown that there is good detection in the correlation output plane with a sharp peak on the location of the object. The only drawback of the spatial domain implementation is the significant computational requirement (effectively a 256x256 pixel size convolution kernel applied to a full resolution input image). A possible means of overcoming the computational requirements would be to employ the volume holographic based correlator system recently proposed by Birch et al [1]. Another alternative implementation which is better suited for the currently proposed application would be a FPGA based hardware correlator as proposed by Gardezi et al [11]. The FPGA based correlator uses configurable logic to update the parametric values of the OT-MACH filter depending on the PCE values of the correlation plane. With

FPGA applications have led to higher density devices, intellectual property (IP) integration, and high-speed I/O interconnect technology. All of these elements have allowed FPGAs to play a central role in the defense and security industry. In digital design an n-bit lookup table can be implemented with a multiplexer whose select lines are the inputs of the LUT and whose inputs are constants. An n-bit LUT can encode any n-input Boolean logic function by modelling such functions as truth tables. This is an efficient way of encoding Boolean logic functions, and 4-bit LUTs are in fact the key component of modern FPGAs. In order to implement the OT-MACH in a FPGA sequences of steps have to be followed. The design tools used were ISE “PROJECT NAVIGATOR” by “XILINX” and for simulating “MODEL SIM” by “MENTOR”. The reason behind choosing ISE PROJECT NAVIGATOR is due to the fact it has an extensive choice of tools that range from synthesis to place and route. Also it has options for manual place and route. The project navigator has specific libraries for the SPARTAN XC2S 200E. It also has “IMPACT” which enables JTAG flashing of the boot Read Only Memory (ROM). “IMPACT” runs a boundary scan and checks for connected devices and shows a virtual view of the device. The simulation tool “MODEL SIM” is one of the most widely used tools in the industry because of its relatively flexible system requirements as opposed to CADENCE tools. A block diagram for the design of the OT-MACH for FPGA implementation is shown in “Fig. 6” given below:

values for the non negative paraameters to improve target recognition. The system has been tested by running r tests on an INTEL CORE 2 DUO processor and a SPA ARTAN XC2S200E FPGA. The performance evaluation criterria were defined by taking into account the processing time and resources required during post place and route simulation on a FPGA.

M.A.C.H Filter

Input Image

Spatial al Correlattor

E 2 DUO the algorithm was In the case of the INTEL CORE simulated in MATLAB using the profiling p option to calculate the computation time required fo or the space variant OTMACH. The results are shown in “F Fig. 7” given below.

M.A.C.H Filter

Kernel Image

Store FPGA LUT

GOTO Next Iteration

YES

If PCE>Threshold

α, β , γ

Calculate PCEValue

P a ra m e te rs p Ma ry mo Me

α, β , γ P a ra m e te rs

N O

Change

YES

If Count>(256(RefImageKernel))

NT (HEADING 5) ACKNOWLEDGMEN

N O

Re-Tune M.A.C.H Filter

Reset Count=0

Count=Count+1;

GOTO Next Iteration

ReStart

Figure 6. OT-MACH FPGA Implementation B Block Diagram

In “Fig. 6” the hardware implementation oof a space variant OT-MACH is shown. When the kernel and tthe input images are acquired they are first preprocessed througgh an OT-MACH filter and then passed on to the spatial correllator. The spatial correlator comprises a multiply accumullate unit which normalizes the kernel and performs cross coorrelation on the two images.

Figure 7. OT-MACH INT TEL implementation

From “Fig. 7” it can be seen thaat the multiply accumulate function at line number “41” is taking 5.568 seconds to complete. This shows the computtational complexity of the space variant OT-MACH. When the same algorithm wass used on a SPARTAN-II XC2S200E FPGA the timing sum mmary generated by “ISE PROJECT NAVIGATOR” is given in TABLE III. TABLE III. SPARTAN XC2S200E E TIMING SUMMARY Min.period

13.219ns

The optimum OT-MACH parameters arre stored in the lookup table. The system keeps on findingg new optimized

Min. input arrival time before clock 8.840ns

Max. output fter time aft clock k:

Maximum combinational path delay

14.126ns

11.87ns

The total design frequency in this case iss 75.69 MHz and by comparing the clock speeds it can be seenn that the FPGA implementation is considerably faster. Also ‘PROJECT NAVIGATOR” gives thee option of design optimization as well. The design can be optiimized according to speed or area used on the FPGA. This is sshown in “Fig. 8” given below.

VI.

CONCL LUSION

In this paper FLIR images have been used for the recognition of targets which dem monstrates high levels of variability of their thermal signatures within a scene. The OTMACH filter was initially designed d in the frequency domain and then implemented in the spacee domain as a space variant convolution kernel applied to a pre-filtered input image Sample results have been preseented which indicate an orientation multiplexed spatial dom main filter is able to detect, locate and recognize a target objecct within a cluttered scene. The performance of the filter was evaluated using the PCE , PSR and COPI measures to determine the detection capabilities of the filter for targets from fr cluttered backgrounds. The computationally intensive nature n of the space domain implementation may be tackled by the use of efficient digital hardware implementations using FP PGAs [11]. A performance evaluation has been performeed by comparing the computational results from an INTEL CORE 2 DUO processor and a SPARTAN XC3S200E board.

Figure 8. OT-MACH FPGA design optiimization DGMENT ACKNOWLED

The chip layout for the FPGA is shown iin “Fig. 9” given below.

We would like to thank the KUWAITI M.O.D for providing the image data used in this project.

NCES REFEREN [1]

Figure 9. OT-MACH FPGA chip layout

It can be seen in “Fig. 9” that the multiplierr and adders have taken a small area on the chip. The smaller tthe area required the quicker the data bus speed. The FPGA haas been proposed as an ideal choice for implementing the propossed system. The current best choice would be to uuse a VIRTEX-5 FPGA which has a clock speed of 550 MHz and can compute up to 528 Giga Multiply Accumulate functioons. Also it has a DSP48E slice giving it the additional advanttage of having a DSP processor which has dedicated 25x18 mulltipliers.

P.M. Birch, A.A.Gardezi, B.K.Mitra , R.Young , C.Chatwin, , “An optical space domain volume holograaphic correlator” , Proceedings of SPIE Vol. 7340, 734004-1, (2009) mar, S. Song , S. R. F. Sims, J.F. [2] A., Mahalanobis, B. V. K Vijaya Kum Epperson, “Unconstrained correlation filters,” f Appl. Opt. 33, 3751–3759 [3] Z. Bahri, B. V. K. Kumar, “Gen neralized synthetic discriminant functions”, Journal of Optical Society of o America, Vol. 5, No. 4, 562-571 (1988) [4] Ph. Refregier, “Optimal trade-off filters for noise robustness, sharpness of the correlation peak and Horner effficiency”, Optics Letters, Vol. 16, No. 11, 829-831 (1991) D Casasent, , “Minimum average [5] A. Mahalanobis , B. V. K. Kumar, D. correlation energy filters”, Applied Opttics, Vol. 26, No. 17, 3633-3640 [6] K. Mersereau, G. Morris, “Scale, rotation, and shift invariant image N 14, 2338-2342 (1986) recognition”, Applied Optics Vol. 25, No. [7] D. Casasent, D. Psaltis, “Position, rottation, and scale invariant optical correlation”, Applied Optics Vol. 15, No. N 7, 1795-1799 (1976) [8] P. Bone, R.Young , C.Chatwin , “Position-, rotation-,scale- and orientation-invariant multiple object reecognition from cluttered scenes”, Optical Engineering, Vol. 45, No.7, 07 77203-1 to 8, (2006) [9] B. V. K. Kumar , Hassebrook, L., “Performance measures for correlation filters,” Appl. Opt. 29, 2997–3006 (199 90) [10] L.S. Jamal-Aldin, R.Young , C.Chatwiin, “Application of nonlinearity to wavelet-transformed images to improv ve correlation filter performance”, Appl. Opt. Vol. 36, No. 35, pp. 9212-9224 ,(1997) B , R.Young. , C.Chatwin, [11] A.A.Gardezi , I. Kypraios, P.M. Birch “Implementation of the Maximum Aveerage Correlation Height (MACH) filter in the spatial domain for object o recognition from clutter backgrounds” , Proc. SPIE Vol. 7696, 76961S 7 (May. 12, 2010) [12] K. Brunndstorm, , B.N. Schenkman., B. Jacobson, “Object detection in cluttered infrared images”, Opt.Eng. Vol. 42, No. 2, pp. 388-399, (2003)

[13] A. Mahalanobis, B. V. K. Kumar, “Optimality of the maximum average correlation height filter for detection of targets in noise”, Optical Engineering, Vol. 36, Issue 10, 2642-2648, 1997 [14] A. Strehl, J.K. Aggarwal, , “Detecting moving objects in airborne forward looking infra-red sequences”, Journal of Machine Vision Applications, Vol 11, pp.267-276, (2000)