Dynamic Real-Time Sensor Performance Evaluation John J. Sudano Lockheed Martin Moorestown, NJ, 08057, USA
[email protected]
Abstract: A weakness of many information fusion systems is
exponential
the assumption of a static sensor error model in the fusion process. Monitoring sensor real-time measurements gives the system some information on sensor performance, environmental effects, and adversary jamming capability. This article outlines the impact of sensor probability measurement errors on system decisions.
e
Keywords: Probability of detection, sensor probability
distribution
− n ( Log (1 / PD ))
since
it
can
be
written
as
.
For the total number of attempted measurements, N Total , there will be an expected
NTotal PD actual measurements. These
measurements can be grouped into clusters. A
characteristic
integer
cluster
density
function
distributions, cluster distribution
δ (nCluster, PD , N Total ) can be defined representing the number
1 Introduction: In a normal sensor operation, a data stream
of clusters
δ
of information is presented to the system with a certain probability of detection, PD . The Probability of detection, PD ,
an
expected
is the ratio of detected measurements to the total number of attempted measurements that could have been detected. Assuming that the measurements are independent, i.e. not correlated, then the continuous measurements can be arranged into clusters or groupings of continuous measurements. (Anything between zero measurements is a cluster.) An integer distribution can be calculated by collecting the number of the same size clusters as a function of the cluster size. If the calculated integer distribution deviates from the theoretical expected one then it can be inferred that (geometrical, environmental propagation, adversarial jamming…) changes have occurred. The signature of the integer distribution deviation dictates the type of change that has occurred.
The constant A is proportional to
having the same cluster size, average
δ (nCluster , PD , NTotal ) = A * ( PD )
n
nCluster . There are
number clusters of size
of nCluster .
N Total .
The total number of clusters is: ∞
∑ δ (n, P n =1
D
∞
, N Total ) = ∑ A * ( PD ) n = n =1
PD A 1 − PD
For the total number of attempted measurements, will be an expected
N TotalPD
(1)
N Total , there
actual measurements.
∞
NTotal PD = ∑ n * δ (n Cluster , PD , NTotal ) n =1 ∞
The probability of a sensor not detecting a measurement is 1 − PD ; i.e., a cluster of size 0.
= ∑ n * A * ( PD ) n = n =1
(2)
PD A
(1 − PD )2
Solving for A: The probability of a sensor detecting a single measurement is PD ; i.e., a cluster of size 1. The probability of a sensor detecting two measurements, one 2 after the other, is (PD ) ; i.e. a cluster of size 2. The probability of a sensor detecting three measurements, one 3 after the other, is (PD ) ; i.e. a cluster of size 3. The probability of a sensor detecting n continuous n measurements (cluster of size n) is (PD ) . This is a discrete
ISIF © 2002
A = NTotal(1 − PD )2
(3)
Substituting the value of A, the expected average number of clusters of size n for a total number of attempted measurements N Total can be written as: 2 δ (n Cluster, PD , NTotal ) = NTotal (1 − PD ) (PD ) n (4)
The number of clusters is:
361
∑ δ (n
Cluster
∞
∑N
Total
n =0
, PD , N Total ) =
(1 − PD )2 ( PD ) n
(5)
= N Total (1 − PD )
40 35 30
The cluster size average is:
25 ∞
< n Cluster > =
(n ) N Total (1 − PD )2 ( PD ) n ∑ n= 0 ∞
∑= 0 N Total (1 − PD )2 ( PD ) n
20
=
PD 1 − PD
15
n
4000
6000
8000
10000
(6) The cluster size variance is: 2
2
Total
2 < n Cluster > =
A Plot of cluster size
∑ (n ) N (1− P ) ( P ∞
n= 0
n D)
D
=
∞
∑ N Total (1 − PD )2 ( PD ) n
PD (1 + PD ) (1 − PD ) 2
nˆ as a function of PD = {0.0 to 0.9} for
values of Nˆ Total {1000 , 10000 , 100000 , 1000000 }
n=0
80
(7) 2 < n Cluster > − < n Cluster > 2 =
60
PD (1 − PD ) 2
40
(8) 20
Hence, the average cluster size standard deviation is: PD /(1 − PD )
(9)
Solving for the largest cluster size nˆ that has the characteristic integer cluster density function, δ nCluster , PD , NTotal , equal to 1 for any PD with
(
)
0.2
0.4
0.6
0.8
A table of cluster size nˆ for the values of PD = {0,.1,.2,.3,.4,.5,.6,.7,.8,.9} and values of
Nˆ Total{1000 , 10000 , 100000 , 1000000 } is shown below.
corresponding Nˆ Total :
1 = Nˆ Total (1 − PD ) ( PD ) 2
PD nˆ Nˆ t = 1000 nˆ Nˆ t =10000 nˆ Nˆ t =100000 nˆ Nˆ t = 1000000
nˆ
i 0. 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 k 0.9
(10)
Taking the log of both sides gives:
0 = Log[ Nˆ Total] + 2 Log[1 − PD ] + nˆLog[ PD ] (11) Solving for
nˆ yields:
nˆ = ( Log[ Nˆ Total] + 2 Log[1 − PD ]) /( − Log[ PD ])
0 2.90849 4.01474 5.14497 6.42384 7.96578 9.93524 12.616 16.5314 21.8543
0 3.90849 5.44541 7.05746 8.93678 11.2877 14.4428 19.0717 26.8503 43.7087
0 4.90849 6.87609 8.96995 11.4497 14.6096 18.9504 25.5274 37.1691 65.563
0 y 5.90849 8.30677 10.8824 13.9627 17.9316 23.458 31.9831 47.488 87.4174 {
2 The Chi-Square Test
(12) The chi-square right tail test is used to test if the observed sample distribution of clusters differs from the expected distribution of clusters. A Plot of cluster size
nˆ as a function of
Nˆ Total {1000 to 10000 }
χ2 = ∑
for values of PD = {.9,.8,.7 ,.6}
362
(Observed − Expected )2 Expected
(13)
Example:
2.1 Calculating Chi-Square
The frequency of cluster sizes from a sample of 1000 attempted sensor measurements is:
The number of rows in the calculation is dictated by the value of δ Expected ≥ 1 . The last row is calculated from the infinite sum
δ OBSERVED = [ 55,37 ,36,37 ,16,16,10,6,4 ,11,4 ,5 ,2,2 ,0,0 ,1] (14)
The observed probability of detection is calculated as: 16 PD = ∑ i * δ OBSERVED [i ] / 1000 = .758 i =0
(15)
of δ Expected < 1 as:
∑δ =
OBSERVED
= N Total(1 − PD )
242 = 1000 (1 − PD ) PD = .758
i 0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 k Sum
(16)
2 δ (nCluster, PD , N Total) = N Total(1 − PD ) (PD ) n (17)
Substituting the values for this example gives
δ (nCLUSTER − SIZE ) = 1000 (1 − 0 .758 ) (0. 758 ) n (18) 2
A table is calculated describing the cluster size, expected frequency of cluster size, and the observed frequency of cluster size.
i 0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 k 16
δ EXPECTED δ OBSERVED 58.564 44.3915 33.6488 25.5058 19.3334 14.6547 11.1083 8.42006 6.38241 4.83786 3.6671 2.77966 2.10698 1.59709 1.2106 0.917633 0.695565
55 y 37 36 37 16 16 10 6 4 11 4 5 2 2 0 0 1 {
n =15
Cluster δ Obs Size
The expected cluster distribution is calculated with the following equation.
Cluster Size
∑ 1000 (1 − 0.758 )2 (0.758) n = 3.79187 (20)
It can also be calculated from the sum of all clusters
∑δ
∞
55 37 36 37 16 16 10 6 4 11 4 5 2 2 0 1
χi2
δ Expected
58.564 0.216893 y 44.3915 1.23074 33.6488 0.164294 25.5058 5.17991 19.3334 0.574724 14.6547 0.1235 11.1083 0.11057 8.42006 0.695564 6.38241 0.889297 4.83786 7.84891 3.6671 0.0302207 2.77966 1.77356 2.10698 0.00543219 1.59709 0.101643 1.2106 1.2106 3.7919 0.736278 { (21)
Chi-Square Sum is calculated to be
2 χCalculated = 20.892
Degrees of freedom = 16 - 2 = 14
(22)
For this example, the probability of exceeding a predetermined critical value of 0.01 implies a threshold test value of:
χ.201 =29.141 Since
(23)
2 χCalculated < χ .201 the inferred conclusion is that the
sensor is within the predetermined requirements and it is working well.
3 Fluctuations in the Number of a Specific Cluster Size
(19)
Some insight can be gained in the analytic description of the fluctuations in the number of a specific cluster size; i.e., the fluctuations in δ nCluster , PD , N Total .
(
363
)
Since the Poisson distribution [1] describes the normalized probability of a cluster measurement occurring in an interval of time, the fluctuation in δ is described by non-normalized Poisson distribution. This is derived in the following process.
( 0 ≤ i ≤80 ;δ ( 0 ≤ n
π
Cluster
≤10 , PD = .758 , NTotal =1000
))
80 60
The discrete Poisson distribution is:
40 20
P (i; µ) =
e− µ µi i!
0
(24) 40 20
The non-normalized distribution is computed by matching the mean of the Poisson distribution to the average frequency of cluster sizes µ = δ .
π (i; µ ) =
i+ 1− µ
2 4 6
e−µ µ µ µ Γ (µ + 1)
8
π
Γ (µ + 1) i!
(26)
i = µ , the above equation gives the appropriate result of π ( µ; µ) = µ . Therefore, the number distribution
π (i;δ (nCluster, PD , NTotal)) = δ (nCluster, PD , NTotal)
10
(25)
Note that for
=
0
−1
e−µ µ i π ( i; µ ) = i! µ
0
i + 1 − δ (nCluster,PD , NTotal)
Γ(µ +1)
The discrete distribution for a sensor system with a probability of detection PD = .758 and attempted total sensor measurements of N Total
= 1000 , as a function of number of
clusters of size n Cluster from 0 to 10 and varying the possible number i of the cluster size from 0 to 80. Note the unique distributions of the eleven cluster sizes.
( 0 ≤ i ≤80 ;δ ( 0 ≤ n
π
Cluster
≤10 , PD = .758 , NTotal =1000
))
i! (27)
with δ (nCluster , PD , NTotal ) = N Total (1 − PD ) ( PD ) n 2
(28)
Recapping, a sensor system with a probability of detection PD and attempted total sensor measurements of N Total , has an
40
n Cluster equal to
0
average number of clusters of size
20
δ (n Cluster , PD , NTotal ) . The variation on this mean is described by
(
)
the π ( i ;δ nCluster, PD , NTotal )
(
80
0 60
2.5 40
5
distribution. Therefore the
)
20
7.5
distribution π ( i ;δ nCluster, PD , NTotal ) gives the expected
10 0
number of cluster size n Cluster for any possible number i of
4 Conclusion
cluster size.
π
The following plot is the “continuous ” distribution for a sensor system with a probability of detection PD = .758 and attempted total sensor measurements of N Total
= 1000 , as a
function of number of clusters of size n Cluster from 0 to 10 and varying the possible number i of the cluster size from 0 to 80. This plot illustrates the exponential form of the function.
A weakness of many information fusion systems is the assumption of a static sensor error model in the fusion process. This article introduced a methodology to reverse engineer the dynamic real-time sensor performance evaluation. Monitoring sensor real-time measurements gives the system some information on sensor performance, environmental effects, and adversary jamming capability. In a normal sensor operation, a data stream of information is presented to the system with a certain probability of
364
detection, PD . The continuous measurements can be arranged into clusters. The calculated cluster distribution deviation from the theoretical expected distribution is used to infer that (geometrical, environmental propagation, adversarial jamming…) changes have occurred. The signature in the change of the integer distribution dictates the type of change that has occurred.
π
A discrete distribution has been introduced that describes the real-time system cluster distribution.
References [1] P. H. Bevington, “Data Reduction and Error Analysis for the Physical Sciences,” McGraw-Hill 1969
365
