12th International Conference on Information Fusion Seattle, WA, USA, July 6-9, 2009
Modeling ATR processes to predict their performance by using invariance, robustness and self-refusal approach Boris Kovalerchuk Department of Computer Science Central Washington University Ellensburg, WA, USA.
[email protected] fundamental methodological advantage over the experimental testing of algorithms against particular images/data. Test cases can cover only a very small fraction of all real data variability. As a result, a conclusion about the success or failure of the algorithm based on such data can be unreliable or even accidental. Evaluations in many domains such a drug design and computer-aided medical diagnostics and many others show that the real datasets (even large ones) are small relative to the size of the attribute space. This paper is organized as follows. Section 2 introduces the main concepts and provides motivation. Section 3 presents methodology and outlines the algorithm. Section 4 describes specifics of combining invariance, robustness and self-refusal capabilities. Section 5 presents the global performance forecast in the network of ATR algorithms. Section 6 provides an example and Section 7 summarizes this paper and outlines the future research.
Abstract - A tremendous variety of types of targets, sensors, and environments is a great challenge for modern ATR systems, applications, and technologies. This makes extensive large-scale experimentation to evaluate performance of Automated Target Recognition (ATR) systems nearly impossible and motivates the proposed approach and algorithms to overcome this problem by using predictive modeling. This paper presents a methodology of ATR performance prediction called Predictive Modeling of Invariance and Robustness (PMIR). The key idea is to use parameters of ATR algorithms for predicting ATR performance. The levels of robustness and invariance of parameters are used here as predictive indicators of ATR performance along with selfrefusal capabilities of the algorithms. Keywords: ATR, predictive modeling, performance, invariance, robustness, data fusion.
1
Introduction
2
Evaluation of algorithm performance is critical in multiple data fusion domains [11,12] Extensive large-scale experimentation to evaluate the performance of such systems is nearly impossible in general and for ATR specifically [5, 14-22]. This motivates the proposed approach and algorithms to overcome the problem by using predictive modeling. To build these models we use a hybrid approach that combines theoretical and experimental analysis of parameters of ATR algorithms. The key idea is to exploit parameters of ATR algorithms that are indicative for predicting performance of these algorithms instead of direct running of the algorithm on millions of possible datasets. The direct run is not practical and fundamentally incomplete. We will never be able to run algorithms on all possible datasets. It requires exponential time. The scientific challenge of this approach is in identifying critical parameters of ATR algorithms for predictive modeling of their performance. Specifically we focus on the levels of invariance and robustness of parameters as predictive indicators of the ATR performance. The predictive approach has a 978-0-9824438-0-4 ©2009 ISIF
Concepts and Motivation
2.1. Invariance Consider an ATR algorithm A that recognizes object s as belonging to class c1 when s is viewed from position p, A(p,s)=c1, and A recognizes the same s as belonging to class c2 when s is viewed form position q, A(q,s)=c2. This algorithm A is not invariant to the change of the viewing geometry, A(p,s) ≠ A(q,s). (1) Let M be the traditional machine learning and pattern recognition performance measure M based on estimates of the probability of true positive recognition of objects based on a set of test cases. This measure will produce two different accuracies M(A,p) and M(A,q) for the same algorithm A under these two different conditions for the same scene. M(A,p) can indicate high performance of A, but M(A,q) can indicate low performance of A. These measures are contradictory and each of them is insufficient for ATR applications with moving targets and sensors, where viewing geometry is changed. They can be
1139
used only for static applications. Thus, we need an adequate performance measure. Consider an accuracy measure TP+TN M(A, x, V) = , (2) TP+TN+ FP+FN where x is a view position, V is a 3-D scene, TP is the number of true positive recognition cases in the scene V, TN is the number of true negative cases, and FP and FN are, respectively, the number of false positive and false negative cases. Let T be a required accuracy threshold, then the “perfect” performance of algorithm A for scene V will take place if for all viewing positions x from a set of positions X,
to minimize experiments/ tests needed for prediction and evaluation of ATR system performance. As we discussed above in situations such with ratio 0:100 or 100:0 we need just one field experiment to predict a good performance or failure. Similarly, if we know that the situation is 50:50 we do not need even a single field experiment. We know that we can not trust the result. It is likely to be random.
2.2. Robustness Robustness means invariance to small local data relocation with possible small change values or significant change of values, but for relatively small number of pixels (or other data units). In general, if parameters of the algorithm are not invariant and robust relative to changes in real situations, then it is only a matter of time when ATR algorithm with fail when these changes emerge. Consider the same ATR algorithm A that recognizes object s as belonging to class c1 on scene V that is given with noise pn, A(pn,s)=c1. However this algorithm A recognizes the same s as belonging to class c2 when V is given with a slightly different noise qn, A(qn,s)=c2. This algorithm A is robust to the slight change of noise,
M(A,x,V) ≥T (3) To evaluate performance of another algorithm A that is less perfect, we compute measure W(A,x,V) for that algorith, |{ x ∈ X: M(A,x,V) ≥ T}|| (4) ||X|| Thus, W counts the relative number of successful cases. Consider a situation when X consists of n viewing positions and for a half of them M(A,x,V) ≥ T, then W(A,X,V)=0.5. Similarly, consider all pairs (p,q) of viewing positions from X such that for a half of these pairs, algorithm A produces different results similar to (1), W(A,X,V) =
(5) A(pn,s) ≠ A(qn,s). It is assumed here that the difference between noise pn and noise qn is below some threshold T, (6) ||pn-qn||
