Context-Based Information Quality for Sequential

Context-Based Information Quality for Sequential Decision Making Galina ROGOVA∗ , Melita HADZAGIC† , Marie-Odette ST-HILAIRE‡ , Mihai C FLOREA§ and Pierre VALIN¶ ∗ Encompass

Consulting, Honeoye Falls, NY, USA Email: [email protected] † Universit´e de Montr´eal, Montr´eal, QC, Canada Email: [email protected] ‡ OODA Technologies Inc., Montr´eal, QC, Canada Email: [email protected] § Thales Canada, Qu´ebec, QC, Canada Email: [email protected] ¶ Defence R&D Canada Valcartier, QC, Canada Email: [email protected]

Abstract—The context-dependent nature of information quality has been well recognized in the literature. Depending on the context, the overall information quality may relate to a single quality attribute, a combination of several, or all the attributes. Selection of quality attributes, methods for their combination as well as quality control strategies require consideration of decision makers’ objectives and functions together with context parameters such as characteristics of the environment and current situations. This paper considers such context-dependent information quality attributes, namely, credibility, reliability, and timeliness, and presents a method of incorporating their combination into sequential decision making for pattern recognition. The context is represented by the time-dependent distance between an observed target and a sensor, and a situation-based time-dependent threshold on credibility. A case study involving an evidential learning system for sequential decision making designed for a quantitative evaluation of the described method, is also presented.

Keywords: context, information quality, sequential decision making, information fusion. I. I NTRODUCTION The problem of building information fusion systems for decision support is complicated by the fact that data and information obtained from observations and reports to be processed are of variable quality and may be unreliable, of low fidelity, insufficient resolution, contradictory, and/or redundant. The success of decision making in such complex fusion driven environment depends on the success of being aware of and compensating for the insufficient information quality. The strategies for quality control can include [1]: • Eliminating information of low quality from consideration. • Explicitly incorporating information quality into models and processing. • Utilizing process refinement by sensor management. • Delaying transmission of information to decision makers until it has matured as a result of additional observations and/or computations by improving their associated quality.

•

Combination of all or several strategies mentioned above.

The implementation of quality control measures requires methods of representing and measuring quality values as well as criteria for defining when the quality of information is not sufficient. One such criterion is the threshold satisfaction when the quality is compared with a certain threshold [1]. This threshold is highly context specific and depends on decision makers and their attitude toward risk. Information Quality (IQ) represents ”information about information”, or meta-information. As such, it is best characterized and measured through its attributes which may include credibility, accuracy, timeliness, relevance, etc. [2]. Certain attributes can be considered as ”the higher level quality” [1]. Higher level quality measures how well the quality of information is assessed. The need for considering the higher level quality stems from the fact that the processes for assessing the values of the IQ attributes have their limitations. These limitations come from the imperfect domain knowledge, the difficulty of finding an adequate quality model and its parameters, the lack or insufficient amount of additional information that may be required for quality assessment, and the subjectivity of quality evaluation performed by experts and end-users. IQ attributes have to be considered in relations to specific user’s objectives, goals, and functions in a specific context. For example, reliability of an optical sensor can be high during a sunny day and low at night. Due to the fact that all users have different data and information requirements, the set of attributes and the satisfactory level of quality vary with the user’s perspective, the type of the models and the algorithms used, and processes comprising the system as well specific context parameters. Depending on the context, the overall information quality may relate to a single quality attribute, a combination of several or all the attributes. Contextual parameters include characteristics of the environment such as the weather and the time of the day (so-called ”context for”) as well as the characteristics of a current situation such as its

statistical information or relations between situational items under consideration (so-called ”context of”) [3]. Selection of a particular quality control method depends on the type of quality attributes under consideration. For example, irrelevant information is usually eliminated from consideration while reliability is often explicitly incorporated into models and processes. This paper introduces a model of incorporating a combination of context-based quality attributes, namely, reliability, credibility and timeliness into sequential decision making for pattern recognition. This model combines two quality control strategies: incorporation of information quality onto fusion model and processes and delaying decision making until the context dependent quality of information is satisfactory. The presented model is based on the simplification of the approach introduced in [4], [5], in which an anytime sequential decision making was incorporated into an adaptive fusion model for decision making, but advances it by consideration of timedependent reliability. The paper is organized as follows. Section II discusses quality control in sequential decision making, Section III describes the anytime sequential decision making model incorporating context-based information quality, Section IV describes a case study considered to demonstrate the feasibility of the model described in Section III. Section V presents conclusions. II. Q UALITY C ONTROL IN S EQUENTIAL D ECISION M AKING A. Anytime Decision Making Incorporation information quality into sequential decision making represents one of the quality control methods in which decision making is delayed until the information quality satisfies context-dependent needs of the decision maker (socalled anytime decision making). This requires balance of timeliness and other characteristics of information quality e.g. credibility, reliability, accessibility, relevance, etc. The idea here is that the decisions can be improved over time since additional data may be able to improve the quality of decisions and reduce false alarm [4], [5]. At the same time, waiting may result in unacceptable decision latency and unwanted even catastrophic consequences. Therefore, the cost of waiting for additional information has to be justified by obtaining results of better quality. Thus, the timeliness of the decision is defined by the context-dependent balance between the waiting time and improved information quality. The criterion used to stop observations and make a decision can be based on the utility theory or on a situation-based time dependent threshold on quality. The selection of quality attributes to be combined with timeliness and a method of measuring this quality depend on a situation and decision makers’ objectives, goals and function. In pattern recognition, the quality attributes to be considered are credibility of possible hypotheses about the identity of the object to be recognized and reliability of the credibility. In this case, reliability is considered as a second order of uncertainty indicating how well credibility has been estimated. The notion

of context-dependent reliability will be discussed in the next subsection. B. Context-Dependent Reliability Decision making in pattern recognition is usually based on beliefs assigned to hypotheses about a possible pattern class, which are represented by a specific uncertainty model (e.g. probability, possibility, fuzzy, Dempster-Shafer, etc.). Estimation of beliefs into hypotheses can be hampered by noisy, imprecise, erroneous or ill-suited to the problem data, ambiguous observations, incomplete and poorly defined prior knowledge as well as an inadequate recognition model used for belief estimation [5]. Thus, it is important to consider the quality of the beliefs characterized by reliability, which serves here as the second level quality and can be represented by reliability coefficients α ∈ [0, 1], [6]. Utilization of reliability coefficients allows for quality control by explicitly incorporating information quality into uncertainty models (e.g . α is close to zero if estimation of belief is unreliable and closer to 1 if it is more reliable). Reliability depends not only on a selected belief model but also on the context such as characteristics of the environment and the specific situation under consideration. In dynamic situations, reliability has also to be considered in the context of the past source accuracy [7]. Reliability as a function of context has been reported in many publications (see e.g. [8], [9]). In [8], the context is modeled by a set of parameters influencing reliability of each sensor and expert knowledge is used to represent validity of the sensor domain as a fuzzy membership function of the context. The reliability coefficients are modeled as the probability of fuzzy events. Two other methods of defining sensor reliability based on contextual information are introduced in [10]. Both methods utilize subjective contextual information modeled by the theory of fuzzy events and are used in connection with the probability theory. In one (Local Contextual Combination method), reliability of each source is defined by associating each sensor and each hypothesis to the context and computed as a Bayesian probability mass on the frame of discernment defined by validity domain of each source. The obtained reliability values are used in a discount rule introduced in [9]. In the second one (Global Contextual Processing method), inspired by the method suggested in [11], reliability of one or of a combination of several sensors is computed as the probability of conjunction of the fuzzy subsets corresponding to each source for each contextual variable. This probability of validity is subsequently used for the construction of reliability of one or several associations of sources and hypothesis. The methods utilizing contextual information either in the form of a priori distributions or represented by validity domain for modeling reliability are very successful, however, these information may not always be available, [8]. The context-based reliability, addresed in this paper, is modeled as a function of situation (as a time-dependent distance relationship between an observed target and a sensor) and the environment characteristics.

III. T HE M ODEL The anytime sequential decision making target recognition model presented in this Section represents a simplification of the model described in [4], [5], in which a combination of neural classifiers trained with reinforcement learning was used. In this paper, we consider a competitive learning single evidential neural classifier, which advances the model presented in [4], [5] by considering the time-dependent reliability. Let consider the following frame of discernment, Θ = {θ1 , . . . , θK }, where hypothesis θk , 1 ≤ k ≤ K, is that an observation belongs to a class k, K being total number of hypotheses. Upon arrival, each new observation is evaluated by a classifier, an output of which is used for computing pignistic probabilities defined over the frame of discernment Θ, where the observed data belongs to a subset A ⊂ Θ. The pignistic probabilities are used to decide on one of two actions: decide now (and if so which hypothesis), or defer decision and request another observation. Information flow in the model is presented in Figure 1.

the number of observations in episode j, where Nj = (Tj − Tj−1 )/∆t, and ∆t is the period between two subsequent observations. Consider episode j such that tnj ∈ [Tj−1 + ∆t, Tj ], where tnj = Tj−1 + nj ∆t, and 1 ≤ nj ≤ Nj ; nj corresponds to an observation made at Tj−1 +∆t and nj = Nj to an observation made at time Tj . Let Td = Tj−1 + nd ∆t be the deadline by which a decision must be made such that Tj ≤ Td , where nd ≥ Nj is the maximum number of observations permitted before the decision maker reaches the deadline. The hypothesis θk will be selected in the j-th episode at time Tj−1 < tnj ≤ Td if max Bet Pk ≥ Th(t)

(1)

k

where the probability threshold for decision making as a function of time, Th(t), is calculated as Th(t) =

exp(βnj tnj ) 1 + exp(βnj tnj )

(2)

and nj ) − ln 2 (3) nd β is a context specific parameter that defines the shape of the threshold Th(t), see Figure 2. The thresholds is set to achieve a quasi-optimal performance while guaranteeing that the decision will be made by a fixed context specific deadline. The value of the threshold depends on how certain the decision maker needs to be in the context under consideration. If in the j-th episode inequality (1) is not satisfied, the decision is made at the end of the episode j based on the value of maxk Bet Pk . βnj tnj = (β + ln 2)(1 −

Confidence Threshold

Pignistic Probability

1.2 1 1 3 5 7 9

0.8 0.6 0.4 0.2 0 1

2

3

4

5

6

7

8

9

10

tn

Figure 2. Confidence threshold versus the number of observations for different values of the threshold shape parameter β (from [4], [5]).

Figure 1.

Information flow model.

Following [4], [5], consider a situation where episode j as a subsequence of observations between two decision makerenvironment interactions at times Tj−1 and Tj . Let Nj be

In this paper, reliability is a known function of context characteristics such as the distance relationship between a sensor and a target (situational context parameter) and the environmental parameters (the weather and the time of the day). It is assumed that context characteristics as well as decision makers attitude twards risk are constant. We also assume that if a target is moving towards the sensor then

the observations obtained at a later time are considered more reliable and vice versa. Hence, reliability can be represented as an increasing function of time α(t) = α(dt ), if the distance, dt , between the target and the sensor decreases and a decreasing function otherwise. Reliability as a function of environmental parameters can be obtained from the sensor model.

distance between a feature vector of a pattern in question m, ¯ k corresponding to class k. It P¯m , and the weight vector W serves as a measure of confidence in support of the hypothesis that pattern m belongs to class k, and as such, can be used to represent confidence in each hypothesis θk . Specifically,

IV. C ASE S TUDY

where b is a bias. The evidential classifier comprises an input layer, which contains the number of nodes corresponding to the dimension of the feature vector, the first hidden layer constructed from K radial basis transfer functions, HL1; the auxiliary layers designed to perform the Dempster rule for combining discounted basic probability assignments HL2-HL4; and the output represents pignistic probability corresponding to each hypothesis. During training, the weights were updated according to the delta rule [13]:

A. The Classifier Altough any classifier can be used for this processing, we consider here an evidential neural network classifier similar to ones described in [4], [5]. In [4], [5] reliability was not considered, while in [12], unknown reliability, independent of context and learned, was addressed. Let P¯ tnj be a pattern feature vector obtained at time tnj , ¯ Rk the class representative (or reference) vector, α(tnj ) the ¯ k ) a proximity context-dependent reliability, and Φ(P¯ tnj , R ¯ measure between a pattern representation P tnj and a class ¯ k . The proximity measure Φ is formed representative vector R so as to represent the confidence into each hypothesis θk . ¯ k ) can serve as a weight of support to the hypothesis Φ(P¯ tnj , R tn θk , and yields a simple support function mk j (θk ), i.e., tn ¯k ) mk j (θk ) = α(tnj )Φ(P¯ tnj , R tn

¯k ) mk j (Θ) = 1 − α(tnj )Φ(P¯ tnj , R tn mk j (A)

¯ kinew = W ¯ kiold + δki lrate(W ¯ kold − P¯m ) W where δki =



1, k=j −1, otherwise

and lrate is a given learning rate, k is the class pattern (friend/foe), and j = arg maxk BetPk .

(5)

B. Data

A combination of all simple support functions obtained with the normalized Dempster rule of combination leads to the basic L tnj mk for hypotheses probability assignment mtnj (A) = A ⊂ Θ. Reliability α(tnj ) is modeled as a linear function of the distance between a sensor and a target, dtnj , at the time of observation nj , tnj , i.e. (dtnj − dmin ) (dmax + dmin )

(8)

(4)

= 0, ∀A 6= θk ⊂ Θ

α(tnj ) = αmin + (αmax − αmin )

¯ k ) = Φ exp(−b||P¯m − W ¯ k ||), Φ = Φ(P¯m , R

(6)

It is assumed that the minimum and maximum values of reliability dmin and dmax , correspond to the environmental characteristics under consideration. Decisions about the class of observation in question is based on the highest pignistic probability X mtnj (A) (7) Bet P tnj (θk ) = |A|

In the experiments we used the data set composed of 2545 forward looking infrared (FLIR) ship images from the United States Naval Airfare Center, China Lake, California, provided by Dr. Jack Sklansky of the University of California at Irvine. This dataset was also used for ship recognition in [5], [14]. The images were digitized into 256x64 array of 8-bit pixels. Each ship image belongs to one of the eight classes listed in Table I. Class Name Destroyer Container Civilian Freighter Auxiliary Oil Replenishment Landing Assault Tanker Frigate Cruiser Destroyer with Guided Missile

Class No. 1 2 3 4 5 6 7 8

Group Foe Friend Friend Friend Foe Foe Foe Foe

No. of Images 340 455 186 490 348 279 239 208

Table I S HIP CLASSES .

θk ⊆A

In order to obtain pignistic probabilities the class reference ¯ k has to be defined and a form of Φ has to vector R ¯ k reflects the be determined. The class reference vector R knowledge about a particular class accumulated during the training the classifier. In general, the class reference vector computation procedure and the form of Φ depend on the selected type of learning paradigm. In this paper, we consider the competitive supervised learning based on the Radial Basis Neurons [13]. The radial basis transfer function computes the

The images were organized into two groups: friend and foe, which corresponds to the following frame of discernment Θ = {θ1 , θ2 }, where θ1 corresponds to the hypothesis that ”an observation is friend” while θ2 corresponds to the hypothesis that ”an observation is foe”. Figure 3 presents typical silhouettes for the eight classes. The features comprise six moments given in [15] (M2 − M7 ). These moments are invariant under translation, rotation and scale. Since these moments deliver information primarily of the global shape of

Figure 4.

Figure 3.

Typical silhouettes for eight ship classes.

the object representing poorly the details of the object, they are augmented by four additional features extracted by fitting an auto regressive model to one-dimensional sequence of the projected image along the horizontal axis [14]. The quality of each image varies a lot depending on the context represented by the distance, d, of the ship to the camera. Figure 4 shows the example of the image variability for Destroyer with Guided Missile. C. Experiments and Results The performance of the system was evaluated by a crossvalidation, in which the group reserved for testing has contained randomly selected 20% of the total available dataset, while the rest of the data has been used for training. The test set has been built in episodes of ten patterns from the same object class, i.e. Nj = 10. The object class has first been selected with uniform probabilities over the eight classes. Then, the samples from this class are sorted in the decreasing order of distance between the ship and the sensor. The minimum and the maximum values for the context-based reliability were set αmin = 0.5 and αmax = 0.9, respectively. In the experiments we have compared the recognition accuracy achieved with the reliability represented by a linear function of distance between the object and sensor (contextbased reliability) with the recognition accuracy achieved with a constant context independent reliability. We have considered two constant reliability values: α = 0.7, corresponding to the average context-based reliability and α = 1, corresponding

Example the data of Destroyer with Guided Missile.

to the case in which reliability has not been taken into account (the sensor has been assumed to be completely reliable independently of the distance between the sensor and object). All results were obtained with the threshold parameter β = 3, 200 iterations, and averaged over 10 Monte Carlo runs. Table II presents the recognition accuracy corresponding to a different reliability. Contextbased reliability

Recognition accuracy

0.75

Average contextbased reliabilty (α = 0.7) 0.68

Totally reliable sensor (α = 1) 0.59

Table II R ECOGNITION ACCURACY OBTAINED WITH DIFFERENT RELIABILITY.

The obtained results can be interpreted by using a relative measure, the misclassification reduction (MR), calculated for comparing the misclassification in the situations with and without context-based reliability: %without context − %with context . % without context As seen from the Table II, the consideration of the contextbased reliability allows for a reduced misclassification (MR) by 21% as compared to the situation with the average reliability, and by 39% as compared to the situation in which context-based reliability was not taken into account. MR =

V. C ONCLUSIONS This paper introduces a decision support model for incorporating a combination of context-based reliability, credibility

and timeliness into sequential decision making for pattern recognition. The model combines two quality control strategies: incorporation of information quality into models and processes, and delaying decision making until the quality of information improves as the result of additional observations and satisfies context-dependent requirements of the decision makers. The context is represented by a time-dependent distance between an observed object and a sensor, and a situationbased time-dependent threshold on credibility. The case study described in the paper has demonstrated the feasibility of the presented model involving an evidential learning system for sequential decision making. ACKNOWLEDGMENTS This research was supported in part by the DRDCValcartier, Canada under Contract No. W7707-093839 /001/QCV. R EFERENCES ´ Boss´e, “Information quality in information fusion,” [1] G. Rogova and E. in Proceedings of the 13-th Intl. Conference on Information Fusion, pp. 1–8, 2010. [2] M. Bovee, R. P. Srivastava, and B. Mak, “A conceptual framework and belief-function approach to assessing overall information quality,” Journal of Intelligent Systems, vol. 18, pp. 51–74, 2003. [3] A. Steinberg and G. Rogova, “Situation and context in data fusion and natural language understanding,” in Proc. of the 11-th Intl. Conference on Information Fusion, FUSION 2008, Cologne, 2008.

[4] G. Rogova, P. Scott, and C. Lollett, “Utility-based sequential decision making in evidential cooperative multi-agent systems,” in Proc. of the 6th Intl. Conference on Information Fusion, FUSION 2003, 2003. [5] G. Rogova, P. Scott, and C. Lollett, “Distributed reinforcement learning for sequential decision making,” in Proc. of the 5th Intl. Conference on Information Fusion, FUSION 2002, 2002. [6] G. Rogova and V. Nimier, “Reliability in information fusion: literature survey,” in Proceedings of the 7-th Intl. Conference on Information Fusion, pp. 1158–1165, 2004. [7] R. Pon and A. Cardenas, “Data quality inference,” in Proc. of the Int’l Workshop on Information Quality in Information Systems, IQIS 2005, pp. 105–111, 2005. [8] V. Nimier, “Supervised multisensor tracking algorithm by context analysis,” in Proceedings of the Intl. Conference on Information Fusion, pp. 149–156, 1998. [9] A. Appriou, “Situation assessment based on spatially ambiguous multisensor measurements,” Intl. Journal of Intelligent Systems, vol. 16, no. 10, pp. 1135–1166, 2001. [10] S. Fabre, A. Appriou, and X. Briottet, “Presentation and description of two classification methods using data fusion based on sensor management,” ELSEVIER Jour. on Information Fusion, vol. 2, pp. 47–71, 2001. [11] F. Delmotte and P. Borne, “Context-dependent trust in data fusion within the possibility theory,” in Proceedings of the IEEE Intl. Conference on Systems, Man and Cybernetics, pp. 78–88, 1998. [12] G. Rogova and J. Kasturi, “Neural network for distributed decision making,” in Proceedings of the 4-th Intl. Conference on Information Fusion, pp. 1–8, 2001. [13] C. M. Bishop, Neural Networks for Pattern Recognition. Oxford University Press, NY, 1994. [14] Y. Park and J. Sklansky, “Automated design of linear tree classifiers,” Pattern Recognition, vol. 23, pp. 1393–1412, 1990. [15] M. Hu, “Visual pattern recognition by moment invariantss,” IRE Trans. on Information Theory, vol. 8, pp. 179–187, 1962.