Knowledge Engineering Tools for Probability Elicitation L. R. Hope
A. E. Nicholson
K. B. Korb
June 4, 2002 Abstract Unwieldy probability entry interfaces are the norm for Bayesian Network knowledge engineers. This requires domain experts to provide unreasonably accurate probability estimates. To solve these problems, we have developed two applications: CPTable improves direct probability entry, and provides node customisation and sliding scale binary elicitation; Verbal Elicitor allows entry of probability values in ordinary English. The domain expert selects a verbal cue such as “unlikely” or “almost certain.” The probabilities are then set manually or optimised to minimise probabilistic incoherency.
Keywords: Probability Elicitation, Bayesian Networks, Knowledge Engineering.
1
Introduction
Knowledge engineering of Bayesian Networks (BNs) is a daunting task. The engineer tackles expert opinions of causal relations which are vague and, occasionally, just wrong. These unclear and contradictory judgements, extracted from one or many domain experts, must be woven into a coherent whole. Then, the long-suffering engineer turns to the task of probability elicitation. How to access the probabilistic experience of a domain expert? They are rarely willing to assess probabilities directly. Rarer still is coherency; their probabilities usually don’t add up. CPTable (CPT) and Verbal Elicitor (VE) were designed to help. With CPT you can use ratios and sliding scales and automatically fill tables. VE is friendly: the user need not know anything about probability or BNs—these details are hidden.
1.1
Knowledge Engineering
The stages of knowledge engineering can be defined in a loose order. The first stages are variable identification and defining the causal structure. Neither are simple tasks. After this the variable states are set, and probabilities elicited. Two problems with variable identification are leaving out necessary variables and including the unnecessary. Unnecessary variables complicate the structure and will confuse later steps. Hidden (or latent) variables are unknown variables that should be in the model but aren’t. Their detection is a significant research problem[1]. Structural information can be determined by one domain expert or several, or learnt by a machine learning program (e.g., [2, 1]). The task is to determine when a variable directly influences another. Next, each variable’s values must be identified, exclusively and exhaustively. If a variable is continuous, it must be discretised. This is not as easy as it sounds—a tradeoff exists between level of detail and complexity. A higher number of states exponentially increases the complexity of conditional probability tables (CPTs). The final stage is often the most difficult and time consuming: probability elicitation. When data is available, it is not hard to find the BN parameters automatically. Unfortunately, domain experts are often the only source of probabilities. Even experts comfortable with making these estimates frequently provide incoherent probabilities. Powerful and complex BN development packages (such as Netica[3], Hugin Expert[4] and Analytica[5]) exist. These are general packages: they are used to create variables and structure, enter probabilities, and perform Bayesian updating and inference. Probability entry is where the programs fall short, the interfaces 1
Figure 1: The main CPT window.
unwieldy and unforgiving. They hardly scratch the surface of the more general task of probability elicitation. We created two simple, tight applications: one to deal with probability entry efficiently, the other equipped for probability elicitation. CPTable (CPT) delivers direct, rapid probability entry. It includes fill functions, node customisation, and a sliding scale for binary elicitation. It can be used by someone a little experienced in probability estimation. CPT interfaces directly with the Netica Application[3] (version 2.06 or above) via a COM interface on the Microsoft Windows platform. COM is Microsoft’s application to application interface. The source code and documentation is available via anonymous CVS with the following commands: export CVSROOT=:pserver:
[email protected]:/usr/local/cvsroot cvs login cvs checkout CPTableEdit The password for anonymous is empty. CPT is described further in Section 2. Verbal Elicitor (VE) removes the expert from direct probability entry, and thus is aimed at those less experienced in the probability elicitation task. The user responds to scenarios with verbal cues. The probability of these verbal cues may be set manually or optimised to achieve minimal incoherency. It is written in Java, and uses Netica’s NeticaJ java interface. Because NeticaJ is only available for Microsoft Windows, VE is also only available for Windows. The source for VE can be accessed through anonymous CVS: export CVSROOT=:pserver:
[email protected]:/usr/local/cvsroot cvs login cvs checkout verbalBN VE is described in Section 3. CPT and VE interact with Netica in completely different ways; CPT uses the Netica Application, a GUI for interacting with Bayesian and Decision Networks. CPT loads the Netica Application if it is not already running. VE uses a separate product, the Netica Java API. Netica Java is a set of Java classes for Bayesian Network manipulation.
2 2.1 CPT
CPTable Overview assists and speeds direct probability entry. It links closely with the Netica Application version 2.06[3].
2
Figure 1 shows a screen-shot of CPT. The upper left side of the screen lists the nodes in the network; each node can be selected by name. Also on the left is the network and system oriented command buttons, while the buttons at the bottom manipulate the CPT. The main section of the window displays the CPT. The current node’s states are at the side, with parent states along the top; so probability vectors are vertical. CPT also includes node customisation and binary elicitation. Node customisation includes renaming a node and adding, deleting or renaming its states. Binary elicitation uses a description of a node’s parent configuration and a slider to enter probabilities, in place of direct numerical entry.
2.2
Features
2.2.1 CPT Manipulation The main strength of CPT is its interface for probability entry. The CPT can be traversed using directional arrows, and the manipulation buttons can be accessed by typing -char, where char is the underlined character on the button (see below and Figure 1). The CPT manipulation functions are Normalise, Maximum Entropy Fill, Zero Fill and Clear, all of which work on a column-by-column basis. Apply Changes affects the entire CPT. Normalise This allows the use of ratios in place of probabilities. For example, a binomial variable may be twice as likely to be in its first state. To reflect this, f0.6666666, 0.33333333g could be entered. However, it is easier to enter f2, 1g, then select Normalise. A column must be full to use this feature. Press -N while accessing the CPT to Normalise. Maximum Entropy Fill This fills all blank cells in a column uniformly with the probability remaining after subtracting existing probabilities from 1. As the probabilities are uniform, the entropy of the distribution is maximised. Because probabilities must sum to 1, this function will not work when using ratios which sum to greater than 1. Maximum entropy fill allows the probability of one or some states to be entered, spreading the remainder uniformly. For example, a variable has states A, B and C . Set the probability for B to 0.95, then fill. A and C each show 0.025. For binary variables, just enter the probability p for one of the states, then use maximum entropy fill to automatically set 1 p in the other. Press -F while accessing the CPT to use maximum entropy fill. Zero Fill This fills empty cells in a column with zero. It is especially useful when a variable with a great many states has its probability distributed over only a few states. Instead of entering the zero probabilities, use zero fill. Press -Z while editing the CPT to access this feature. Clear
This clears the column of the CPT and moves the cursor to the top. Press -L to clear.
Apply Changes This commits the CPT updates to the BN in memory. Otherwise, if another node is selected, the display refreshed, or the node is customised (as in Section 2.3), the changes will be lost. Changes can be applied by pressing -A while editing the CPT.
2.3
Node Customisation
Node customisation (shown in Figure 2) is used to change the names and titles of nodes, and add, delete or rename states. All changes to the states are committed to the node immediately, and altering the number of states clears the CPTs of the node and all its children. This last behaviour, possibly not optimal, is the same as the Netica program. The top of the window has text boxes to edit the node’s name and title. These changes are confirmed when the window is closed with the OK button. On the left is a box to select states, next to that are buttons to add, delete and rename states. The buttons are detailed here. 3
Figure 2: Node customisation.
Add State This adds a state with the name and title specified, inserted before the current selected state. Because of this, adding a state to the end of the list is impossible. The last state can be renamed instead. Delete State This deletes the currently selected state. A node must have one or more states, so this will not work when the node only has one state. Rename State This replaces the state’s name and title fields with those specified. Unlike adding or deleting states, the CPT is not reset.
2.4
Binary Elicitor
The binary elicitor (shown in Figure 3) uses a meter (or sliding scale) to set the chance of a binary variable taking a value. This assists accuracy by providing visual, verbal and numerical cues[6]. It presents a scenario description comprising the node’s parent configuration and state value. The parent configuration is changed with Next and Previous buttons. The user estimates the probability of a scenario by adjusting the sliding scale.
2.5
Suggested Improvements
The first improvement is a complete rewrite of the program. It is currently a Microsoft Visual Basic program using Netica Application’s COM interface. This creates two problems. First, the program is only available on Windows, and the COM interface for Netica is fiddly, buggy and poorly documented. Second, complex programs are difficult in Visual Basic and the level of complexity for CPT is at its limit. Some possible rewrite languages could be: Sun Java and the NeticaJ API, or C/C++ and the Qt or GTK toolkits. The node configuration window (Section 2.3) performs changes immediately, and there is no undo feature. Changes should not be performed until confirmed by the user. Also, altering the number of states clears the CPTs of the node and its children. Something more intelligent might be done. The binary elicitor’s layout is bad, and should be redesigned similar to VE. Regardless, the binary elicitor is superfluous, and probably should be removed from the program, because CPT was designed for quick probability entry into CPTs. The binary elicitor was a prelude to the elicitation techniques in VE, and it doesn’t match the simple flavour of the rest of CPT.
4
Figure 3: The binary elicitor.
3 3.1
Verbal Elicitor Overview
VE eases probability elicitation. It presents scenario descriptions and accepts answers like ‘certain’, ‘unlikely’, ‘almost impossible’ and ‘very likely’. Actual probability values are not shown. After this, the BN may be exported to a Netica .dnet file with user-defined probabilities, or an optimised set (see Section 3.7). Similar to CPT, VE is not for building BN structure, but for parametrizing existing structure. Instead, VE imports Netica .dnet files for BN structure, with any probabilistic information being ignored.
3.2
VE Definitions
Technical terms used in the description of VE are defined below. Verbal cue likely.’
This is a phrase representing a chance or probability. For example, ‘improbable’ and ‘not very
Verbal net (or network) This has the same structure as a BN, but rather than having tables of probabilities, the tables contain verbal cues. Verbal map This is a collection of verbal cues and corresponding probabilities. A verbal map should be customised to suit the individual. The default map is:
5
Figure 4: The VE program. Different variables are accessed via the tabs at the top of the window.
Cue word Certain Probable Expected Fifty-fifty Uncertain Improbable Impossible
Probability 1.00 0.85 0.70 0.50 0.30 0.15 0.00
This scale was suggested by Van der Gaag et al. in [6] and, having been translated from Dutch, contains some inappropriate cues. Notably, the term ‘uncertain’ is ambiguous, and could be replaced with ‘unlikely.’ Coherency This is the degree to which elicited probabilities obey the probability axioms. That is: 1.
p
0.
2. For any probability vector ( p1 , : : : ,pn ),
P
n p i=1 i
= 1.
Enforcing probabilities to be non-negative is easy, axiom two is more difficult. A possible measure of coherency is the absolute difference of the sum of probabilities in Axiom 2 from unity. This assumes that axiom 1 is satisfied.
3.3
Features
The main window (see Figure 4) features menus, variable selection tabs, a column of verbal cues, a paragraph describing a scenario and controls to change the scenario. The variable selection tabs are at the top of the window. When a variable is selected, the window displays it. Even when not displayed, the elicitation of a variable is remembered. If a variable has been fully elicited, its name will be followed by an asterisk (*). The verbal cues are to the right. They are used to designate the chance of a given scenario. When a cue is selected, the next scenario will be presented. If the network has been imported from a .dnet file, VE will load the default verbal map (See verbal map, Section 3.2).
6
There are two navigation controls: one for the current variable and another for its parent configuration. The variable’s state is changed by previous and next buttons. As there is usually a greater number of parents, a scrollbar is employed to navigate their combinations, allowing for rapid navigation. Most of the window displays the scenario of the current variable and its parents. The job of the domain expert is to estimate the chance of a scenario occurring using the verbal cues. The scenario paragraph is detailed in Section 3.5.
3.4
Menus
Two menus are available in VE, Verbal Net and Verbal Map. The Verbal Net menu contains save and load functions for both VE’s and Netica’s formats. The Verbal Map menu allows the user to save, load, edit and optimise verbal maps. Edit and optimise are described in Sections 3.6 and 3.7 respectively. These are the Verbal Net options: Load This loads a network previously saved by VE. Save This saves a network, all elicitation and the verbal map to disk. Import Netica Net This creates a verbal net using the structure of the imported network. The default verbal map is also loaded. Export as Netica Net VE is able to export the network to a fully specified Netica BN by normalising the probabilities of the verbal cues. However, the exported network may not be coherent. Here are the Verbal Map options: Load This loads a new verbal map or cues file and deletes all previous elicitation from the network. This is because the new map is incompatible with the old elicitation. Save This saves the verbal map to disk. Optimise Optimise minimises incoherency of elicitation by adjusting the probability of each cue. Details on how optimisation works are in Section 3.7. Edit This allows viewing and changing of the verbal map. If cues are deleted, any elicitation using that cue is cleared (see Section 3.6 for more detail).
3.5
Eliciting Probabilities
As shown in Figure 4, the majority of the window is dedicated to an auto-generated scenario designed to elicit an entry for the displayed variable. VE presents scenarios of the following form: Consider that is and that the chance that is ?
Parent2> is
. What is
