Distorted IR target detection and tracking using composite filters

Distorted IR target detection and tracking using composite filters Khan M. Iftekharuddin and Jahangheer S. Shaik Intelligent Systems and Image Processing (ISIP) Lab Department of Electrical and Computer Engineering The University Of Memphis Memphis, TN 38152 Abstract The automatic target recognition (ATR), often time, is limited by the presence of background clutter and distortions such as scale, translation and rotation (both in-plane and out-of-plane) in both single and multi object cases. Such distortion invariant ATR and image understanding have been the subject of intense research in machine vision. In a previous work, we have demonstrated the usefulness of an amplitude-coupled minimum-average correlation energy (AC-MACE) filter in in-plane rotated SAR image ATR. The AC-MACE filter outperforms the regular MACE filter in rotation-related cases. Motion tracking is also an important task in computer vision, especially, when objects are subjected to certain viewing transformation. There are many problems in which very small objects undergoing motion must be detected and then tracked. For example, one of the most difficult goals of ATR is to spot incoming objects at long range, wherein the motion appears small and the signal to noise ratio (SNR) is poor. The system must be able to track such targets long enough to identify whether the object is a friend or foe. In this work, we are interested in locating both long-range and short-range moving objects in IR images wherein the object may vary from a few pixels in size to a large number of pixels in a sequence of IR images. The targets are submerged in background noise and clutter. Additionally, the tracking problem also involves out-of-plane rotation of the target. Thus, we investigate both MACE and AC-MACE filter for rotation and size invariant target detection and tracking using realistic IR images.

Introduction The application of optical correlation for automatic target recognition (ATR) has always been an area of intense research. The possibility of achieving high speed and massive parallelism and the apparent success of several different types of matched filters offer good opportunities for suitable implementation of optical correlators. A classical application of correlators is mainly for the recognition of objects in complex images. The most important performance is the capability to identify an object, or classes of objects, even if it is magnified, shifted, rotated and surrounded by noise. Correlation is one of the most useful tools for pattern recognition, comparison, or search. Correlation is easily implemented optically, for example, with Vanderlaugt 4-f coherent configuration [1]. The conventional correlation is shift-invariant operation Thus, shifting of the input pattern provides a shifted correlation output plane. An example is when one wants to obtain a correlation peak only when a specific object appears at a certain location. However, it has a disadvantage that it does not provide invariant properties such as rotation, scale or projection. Since the development of Vanderlaugt classical filter [1], several correlation filters for pattern recognition have been proposed to improve recognition capability. Most of these filters are based mainly on the modifications of the amplitude or phase of the original matched filter, and in this sense, perform as a nonlinear Wiener filter. In coherent data processing, we need spatial filters to modify the phase angles and amplitudes of Fourier components of two-dimensional images. More often only way to make filter is to plot its amplitude transmission on paper using plotter guided by computer [2]. The spatial filter thus produced is the synthesized Fourier transform. Phase only filters (POF’s) [3] and inverse filters (IF’s) [4] can be considered as nonlinear transformations of matched filter. Usually these filters are used in the frequency domain in Vanderlugt architecture, although they can be employed in the object space. To obtain the filters in object space, the inverse Fourier transform of the filter defined in the frequency domain has to be calculated. These filters in object space are real functions that take positive and negative values in different zones. The POF is a method for improving the correlation based on the fact

that the phase of a Fourier transform contains most of the significant information of the input. The main advantage is that it has much sharper correlation peaks than the matched filter does. IF similar to POF offers delta function detection peak in the correlation plane. However, IF has severe limitations associated with mathematical poles and small optical efficiency. Another variant of the matched filters, the amplitude-modulated phase only filter (AMPOF) [5] is of particular interest because of its sharp correlation peak and improved noise-related performances [6-7]. It offers a built in noise tolerance which is essential for low-powered optical correlators. However AMPOF performs poorly in distortionrelated cases. Further, these matched filters are not suitable for Multi-class ATR applications since they do not offer enough flexibility for adequate training using different classes of distorted images. However, when limited in-plane rotation invariance of minimum average correlation energy filter (MACE) is coupled with AMPOF’s built in noise tolerance and sharp autocorrelation features, one may actually obtain much better ATR performance such as minimum noise, sharper autocorrelation, minimum correlation energy and improved in-plane rotation invariance [810]. We investigate both MACE and amplitude-coupled MACE filters in terms of detection and tracking of scaled, rotated (out-of-plane) IR image sequences [11,12].

Background Review The phase of the signal is a vital parameter in recovering the signal [13]. However, the amplitude also is of considerable importance [14-16] since it provides an added dimension in the signal retrieval process. The AMPOF, which retains information pertaining to both phase and amplitude, has already been shown to outperform other types of matched filters in discriminating in-class and out-of-class targets. Further, different parameters involved in the AMPOF function formulation can be suitable tuned to provide better noise tolerance. The generalized AMPOF function is [5],

H ampof (u, v) = {D /[ R(u, v) + A]} exp[− jϕ R (u, v)],

(1) where R (u , v ) is the amplitude of the Fourier spectrum R (u , v ) of the reference function r (u , v ) , ϕ R (u , v ) is the phase factor of R (u , v ) and D and A are either constants or functions of u and v . Constraint

D ≤ R(u, v) min + A ensures that the gain of AMPOF-based correlator is less than or equal to unity for all A is function of u and v , R(u, v) ≥ A(u, v) and D is 1, the AMPOF is nothing but an inverse filter. Hence reasonably small A(u , v ) value in AMPOF may offer sharp autocorrelation

frequency components. Further when

feature of an IF while at same time it can be used to avoid the indeterminate condition otherwise associated with an IF. However like any other matched filter, a particular shortcoming of AMPOF is that it does not incorporate multiple image information in its formulation. Hence any rotation or distortion related application necessitates the use of multiple filers, each corresponding to the particular distorted or rotated images. As noted earlier, a particular frequency domain filter, namely MACE may include possible distortion information in its formulation. A MACE filter is a linear combination of the preprocessed distorted images [9]. It minimizes the average correlation plane energy while producing a pre-specified correlation to a location of the training images. The basic MACE filter is given as [17],

H mace = D −1 X ( X + D −1 X ) −1 c, where,

D=

Di , X =

th

X i and c =

(2)

ci , H mace is the Fourier transform of the filter function, X i is the

i training set image (Total number of in-plane rotated images is N).

Di is the diagonal matrix composed of the

Fourier transform-square of the ith correlation peak at the filter output plane. Note that bold-faced parameters represent respective Fourier transformed versions and the subscripts –1 and +1 denote inverse and transpose operations, respectively. The MACE filter minimizes the total correlation plane energy subject to the constraint

ET = H + Mace DH mace

H mace X i = ci for i number of training images. The vector c is usually set to 1 for in-

class target recognition and 0 for out-of-class case. Bank of amplitude coupled MACE filters may represent different

distortion-related cases. Alternatively, for the case of IR application, each class of IR target images may be used to generate a composite filter, which can be then trained and tested. A broad aspect of this type of an ATR system is to operate in a wide area search mode, maintain a very low false alarm rate density in case of different distortions such as rotation and provide a high probability of correct classification. The ultimate purpose of the system is to provide a reliable baseline ATR process based on a pattern matching classifier approach. A similar effort for the SAR target identification using matched spatial filters has also been reported [18].

Algorithm development Figure 1 shows the algorithm for AC-MACE [5]. The values for the vector c are derived from AMPOF autocorrelation using unrotated reference images only for AC-MACE. The autocorrelation value is replicated for each element of vector c that corresponds to the in-class target class. The A and D parameters in AMPOF function are both set to 1, for simplicity. Since AMPOF offers a very sharp and enhanced correlation value, the choice of this autocorrelation peak for vector c is expected to improve the target detection feature of the composite filter. Accordingly, this particular selection of c is also expected to offer better performance even in the presence of rotation related distortion cases. The algorithm for amplitude-coupled MACE filter is summarized by the flow chart shown in Fig.1. The rest of the elements of vector c are set to 0 for the out-of-class targets. An alternative choice to implementing an amplitude-coupled MACE filter is to predefine c as 1 and 0 respectively for in-class and out-ofclass targets. Unrotated/undistorted reference image (r (x, y))

N-training set images X1(x,y)…..XN(x,y) Obtain

H ampof (u , v) = {D / [ R(u, v) + A]}exp(− jϕ R (u, v))

Discretize XN(u, v) and define it as X

H ampof : Amplitude-modulated phase only filter.

Obtain discrete diagonal matrix D:

D= X

Discretize H ampof and call it h

2

Obtain

c , c = X +h

Hamp-MACE(Amplitude coupled MACE)= D

−1

X ( XD −1 X ) −1 c

Fig.1 Algorithm for Amplitude coupled MACE filter [5]

The bold characters in Algorithm of Fig.1 represent discretized vectors in frequency domain. We start with N number of distorted training set images and obtain matrix X as described in Sec.2. Next, we obtain the diagonal matrix X as described in background review. Next we obtain diagonal matrix D as using matrix X. In order to obtain the useful features of an AMPOF, the AMPOF is further coupled with the MACE function. To accomplish this, each of magnitude and phase components of the MACE filter is accessed and arranged according to Eq. (1). This offers an overall AMPOF nature to the composite MACE filter. Finally generalized MACE filter as given by Eq. (2).

For rotation invariance algorithm, the output correlation peak intensity is considered as a measure of ATR performance. As pointed by Farn and Goodman [19], the maximization of the correlation peak metric is a reasonable measure for the performance of any composite filter. Furthermore, we test the suitability of this algorithm in terms of probability of correct classification ( Pc ). More

Image

Intensity normalization followed by Rank Order Filtering

importantly this Pc is obtained at an increased signal-tonoise ratio (SNR). Note that the Pc is defined but he number of occurrences (in scale of 0 through 1) of maximum test correlation corresponding to the respective test-pattern filter Pre-processing algorithm is shown in Fig.2. The image is first intensity normalized to remove the intensity bias and to move the data points closer. Then the image is rank order filtered to enhance the image features. The typical rank order filter used is Maximum filter. The image is divided into blocks and maximum filter is applied on each independent block. Edges of the image are then found out using logarithm of Gaussian operator. Then morphological operations like binary closure, spur pixel removal and isolated pixel removal is performed to remove the unwanted noise pixels in the image. The image is then dilated to form compact entity and then number of object pixels is counted to determine the approximate area of the object. The box size is determined from the area of the object.

Edge extraction

Morphological Processing

Dilation

Box size determination Fig. 2. Pre-processing algorithm

Pre-processed reference Frame

Pre-processed test Frame Area determination

Correlation

Maximum correlated Point

Algorithm in Fig 2

Area

Tracking image frame

Tracked frame

Fig 3. Algorithm for tracking single object in a frame

Figure 3. shows the tracking algorithm. The reference image in the database and the test images are first preprocessed ( intensity normalized and rank order filtered). Then the correlation matrix is found between the test image I ( − x,− y ) and reference image j ( x, y ) using the formula Fcorr (α , β ) = I ( − x,− y ) * j ( x, y ) , where ‘*’ represents the convolution operation. The analysis of the coefficient that created the highest correlation point in the matrix gives us the idea of exact location of the object in the image frame. The algorithm in Figure 2 is used to determine the box size that is used to track the image.

Image Database Database consists of IR images [20] that exhibit scale, shift and rotation (out-of-plane). The images from this sequence are further subdivided into seven sequences as shown in Fig.4.

Sequence # 1

Sequence # 2

Sequence # 3

Sequence # 4

Sequence # 5

Sequence # 6

Sequence # 7

Fig.4 Example images from difference sequences in the database The images shown are further resized to 32 x 32 size for the sake of memory constraint.

Training and Results As discussed earlier in algorithm development section, we explore two baselining properties, namely, rotation and scale invariant detection and the corresponding tracking of the amplitude-coupled MACE filter. The results for both MACE and amplitude coupled MACE filters are discussed below.

(a)

(b)

(c) Fig.5. Correlation peaks AMPOF-MACE filter for different training and test sequences

(a)

Figure 5 shows the correlation peaks obtained using Amplitude modulate phase only filter. The filters are first trained using different sequence of un-normalized, un-noise flattened images independently and tested using all the sequences. Figures 5 (a), (b), (c) show the results obtained using AMPOF-MACE filters trained using images of sequence#1, sequence#3, sequence#7 respectively. The filter is first trained using 3 images at regular intervals of 30 from a sequence and tested using all the sequences. From the figure 5 (a) it is clear that the highest correlation is produced for the test images from the sequence#1. Similar analysis is applicable for Figs. 5 (b) and (c) respectively.

(b)

(c) Fig. 6. Correlation peaks MACE filter for different training and test sequences

Figures 6 (a), (b), (c) show the results obtained using MACE filter only. Similar to Fig. 5, the MACE filter is first trained using 3 images at interval of 30 frames from sequences #1, #3 and #6 respectively and tested using all the sequences. From the Figs. 5 and 6, it is clear that high correlation peaks are obtained using AMPOF-MACE. In order to show better performance, the images are normalized and noise flattened next.

Fig. 7. Raw images from sequence #1, #2 #3

Fig. 8. Normalized, noise flattened images

Figures 7 and 8 show raw images and noise-flattened images from different sequences respectively. The experiment is repeated with both AMPOF- MACE and MACE only. The filters are first trained using three images at regular intervals of 30 from 3 sequences and tested using sequences 1 to 3 respectively. Figures 9 (a) and (b) show the performance of AMPOF-MACE and MACE only for training using images of sequence 1.

(a)

(b)

(c) Fig.9. Correlation peak vs. range (a) AMPOF- MACE ; (b) MACE and (c) normalized correlation plane and peaks for training sequence#1

From the Figs. 9 (a) and (b) it is clear that high correlation peaks are obtained for sequence 3 images and AMPOFMACE shows better correlation results. Figure 9 (c) shows normalized amplitude response and normalized correlation plane of MACE and AMPOF-MACE respectively.

(b)

(a)

(c)

Fig.10. Correlation peak vs. range (a) AMPOF- MACE ; (b) MACE and (c) normalized correlation plane and peaks for training sequence#2

Figures 10(a), (b) show the results using AMPOF-MACE and MACE algorithm. From the Figs., it is clear that AMPOF-MACE and MACE produced better correlation peaks but performance using AMPOF-MACE is better. Figure10 (c) shows the normalized amplitude response and normalized correlation plane of AMPOF-MACE and MACE respectively.

(a)

(b)

(c) Fig.11. Correlation peak vs. range (a) AMPOF- MACE ; (b) MACE and (c) normalized correlation plane and peaks for training sequence#3

Figures 11 (a) and (b) shows the correlation results for training sequence 1using AMPOF-MACE and MACE respectively. From the Figs., it is evident that AMPOF-MACE offers better correlation results. Figure 11(c) shows the normalized amplitude response and normalized correlation plane of MACE and AMPOF-MACE respectively.

Fig.12. Tracking results obtained using algorithm of Fig. 3

As discussed above, the image frames are first intensity normalized and rank order filtered. Then the images are resized to 32 x 32 size to overcome the memory constraints. The algorithm of Fig.3 is used to track the object of interest as shown in Fig. 12. Note that the tracking results in Fig. 12 are obtained using a MACE filter. The box size is determined using the algorithm of Fig. 2.

Conclusion This work investigates the usefulness of composite optical filters in image tracking in extensive background and clutter noise. The advantage of choosing the c vector for AMPOF correlation is that it imitates more realistic approach of implementing the amplitude-coupled filter using spatial light modulators (SLM). Since most of the electro-optics devices, including SLMs, necessitate control over phase and amplitude; the choice of an AMPOFrelated c vector imposes specific phase and amplitude constraints on type of SLMs required for optical implementation of the filter. This type of constraint is quite similar to other SLM-dependent constraints [21-23] that have been exploited before. As expected, the amplitude-coupled MACE algorithm offers a much better correlation when compared to a MACE algorithm. Our proposed tracking algorithm has been successfully exploited to track the object of interest even when the image is resized to 32 x 32 because of memory constraints.

Acknowledgement The authors wish to acknowledge the partial support of this work through a grant form the Army Research Office (Grant# 43004-CI).

References 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15.

A. B. VanderLugt, “Signal detection by complex filtering,” IEEE Trans. Inf. Theory, IT-10, 139-145 (1964). W. H. Lee, “Sampled Fourier transforms generated by computer,” Appl. Opt. P 639, (1970). J. L. Horner and P. D. Gianino, “Phase-only matched filtering, “Appl. OPT. 23, 812-816 (1984). G. G. Mu, X. M. Wang, and Z. Q. Wang, “Amplitude-compensated matched filtering,” Appl. Opt. 27, 34613463 (1988). K. M. Iftekharuddin, F. Ahmed and M. A. Karim, "Amplitude-coupled MACE for automatic target recognition applications", Optical Engineering, Vol. 35, pp. 1009-1014, (1996). A. A. S. Awwal, M. A. Karim, and S. R. Jahan, “Improved correlation discrimination using an amplitudemodulated phase-only filter, “Appl. Opt. 29, 233-236 (1990). K. M. Iftekharuddin, T. Tuthill, M. A. Karim and K. Jemili, "Wavelet pre-processed amplitude modulated phaseonly filter for recognition applications", Optical Engineering, Vol. 35, pp. 2088-2093, (1996). K. M. Iftekharuddin, M. A. Karim, P. W. Eloe, and A. A. S. Awwal, “Discretized amplitude-modulated phaseonly filter,” J. of Mathl. Comput. Modeling, Vol. 24, pp. 103-112, (1996). A. Mahalanobis, B. V. K. V. Kumar and D. Casasent, “Minimum average correlation energy filters, “Appl. Opt. 26, 3633-3640 (1990). G. Ravichandran and D. Casasent, “Generalized in-plane rotation-invariant minimum average correlation energy filter, “Opt. Eng. 30, 1601-1607 (1991). D. Casasent and D. Psaltis, “ Position rotation and scale invariant optical correlator,” Appl. Opt.15 1795-1799 (1976). Y. Hsu and H. Arsenault, “Pattern discrimination by multiple circular harmonic components, “Appl. Opt. 23, 823-840 (1984). B. V. K. V. Kumar, “Tutorial survey of composite filter designs for optical correlators,” Appl. Opt. 31, 47334801 (1992). A. V. Oppenheim and J. S. Lim, “The importance of phase in signals, “Proc. IEEE 69, 529-541 (1981). M. A. Kaura and W. T. Rhodes, “Optical correlators performance using a phase-with-constrained-magnitude complex spatial filter,” Appl. Opt. 25, 2587-2593 (1990).

16. L. P. Yaroslavsky, “Is the phase-only filter and its modifications optimal in terms of the discrimination capability in pattern recognition,” Appl. Opt. 31, 1677-1679 (1992). 17. A. Mahalanobis and D. Casasent, “Performance evaluation of minimum average correlation energy filters,” Appl. Opt. 30, 561-572 (1991). 18. L. M. Novak, G. J. Owirka, and C. M. Netishen, “Radar target identification using spatial matched filters,” Pattern Recog. 27, 607-617 (1994). 19. M.W. Farn and J.W. Goodman, “Optimal maximum correlation filter for arbitratily constrained devices,” Appl. Opt. 28, 3362-3366 (1989). 20. IR Image dataset, AMCOM 1993. 21. A. Khan and P. K. Rajan, “Design of SLM-constrained MACE filters using simulated annealing optimization,” Proc. SPIE 1959, 284-292 (1993). 22. F. T. S. Yu and S. Jutamulia, “Applications of optical signal processing,” in Optical signal processing, Computing, and Neural Networks, Chap. 3, pp. 69-116, Wiley, New York (1992). 23. L. Bigue and P. Ambs, “Implementation of optimal trade-off correlation filters with and optimized resolution technique,” J. Opt. A: Appl. Opt, pp.280-282, (1999).