1
Introduction
2
Anthony Jameson Barbara Großmann-Hutter Leonie March Ralf Rummer
These slides and the full paper are available from: http://w5.cs.uni-sb.de/~ready/
Creating an Empirical Basis for Adaptation Decisions Department of Computer Science Department of Psychology
University of Saarbrücken, Germany http://w5.cs.uni-sb.de/~ready/ (slides, etc.)
Overview General issue IUIs often adapt their behavior • to material being presented • to properties of the situation • to properties of the user How can we help them make sound adaptation decisions?
Overview 1. Rule-based vs. decision-theoretic adaptation 2. Method for empirically based decision-theoretic adaptation 3. What to do when this method is infeasible?
2
3
Introduction
3
Contents
4
Introduction
1
Overview Contents
2 3
Basic Concepts Good Adaptation? Rule-Based Adaptation Decision-Theoretic Adaptation
Experiment Everyday Example Experimental Setup Stepwise vs. Bundled Instructions Variables in Experiment Main Results
The Decision Mechanism Learned Bayesian Network Influence Diagram Properties of the Learned Decision Mechanism
Fallbacks Fallback 1: Modify the Model by Hand Fallback 2: Collect Real Usage Data Fallback 3: Do Analysis Without Data Interactors Revisited
5 5 6 7
8 8 9 10 11 12
14 14 17 18
19 19 21 22 23
Conclusions
24
Conclusions
24
5
Basic Concepts
6
Basic Concepts In the original talk, this slide is accompanied by a playback of the answering machine recording.
Good Adaptation?
5
Good day. You’ve reached my mobile communication center. I don’t wanna waste your time, so I’m gonna make this really quick. To leave a voice message, wait for the tag. To page me, press 5. You can also leave a voice message after you page me. Or, email me, at
[email protected]
Well now, mate, that wasn’t so bad, was it?
Rule-Based Adaptation Rules for choosing interactors for an interface
If the type of data is boolean and the type of form is a control panel then use a check box
If the type of data is boolean and the type of form is a questionnaire then use radio buttons (Cf. the decision trees of Eisenstein & Puerta, IUI2000)
6
7
Basic Concepts
7
Decision-Theoretic Adaptation
RADIO BUTTONS
INTERACTOR
QUESTIONNAIRE
CONTROL PANEL
CONTROL PANEL
QUESTIONNAIRE
SPEED
VALIDITY
CHECK BOX
VALIDITY
SPEED
TYPE OF FORM
8
RADIO BUTTONS
CHECK BOX
UTILITY
When is decision-theoretic adaptation useful? • There are multiple criterion variables • There are quantitative tradeoffs • The (exact) nature of the relationships is an empirical question
Experiment Everyday Example
Possible output of spoken help system: Choose "PostScript Level 2 only" Set "Fit to Page" to "off". Set "Print to File" to "on". Set "Use Printer Halftone Screens" to "on". Set "Download Asian Fonts" to "off".
8
9
Experiment
9
Experimental Setup
Stepwise vs. Bundled Instructions
Stepwise: S: Set X to 3 U: ... OK S: Set M to 1 U: ... OK S: Set V to 4 U: ... Done
Bundled: S: Set X to 3, set M to 1, set V to 4 U: ... ... ... Done
10
10
11
Experiment
12
Variables in Experiment
11
Independent variables: Number of Steps • 2, 3 or 4 steps
Distraction? • No secondary task
Presentation Mode • Stepwise vs.
vs. "monitor the flashing lights"
bundled
Dependent variables (selection): Error • All instructed buttons
Execution Time • Total time to execute an
pressed (and no others)?
instruction sequence (including "OK"s, etc.)
Four-ste Main Results (1)
12
7000
Three-step sequences:
1000 0
No
Yes
Distraction?
10
0
No
Yes
Distraction?
Execution time (msec)
2000
20
40
5000 30 4000
Errors (%)
3000
Execution time (msec)
Two-step sequences: Errors (%)
Execution time (msec)
6000
3000 2000
20
10 1000 0
6000 5000 4000 3000 2000 1000
No
Yes
Distraction?
0
No
Yes
Distraction?
0
No
Y
Distracti
13
Experiment
14
Main Results (2)
13
Four-step sequences: 50 7000
Three-step sequences:
1000 0
No
Yes
Distraction?
10
0
No
Yes
Distraction?
30 4000 3000 2000
20
10 1000 0
40
6000 5000
Errors (%)
5000
Errors (%)
2000
20
Execution time (msec)
3000
Errors (%)
4000 3000
30
20
2000 10 1000
No
Yes
Distraction?
0
No
Yes
Distraction?
0
No
Yes
Distraction?
0
No
Yes
Distraction?
The Decision Mechanism Learned Bayesian Network (1) Number of Steps Four 0 Three 100 Two 0
Execution Time s9 0.65 s8 0.65 s7 1.96 s6 1.31 s5 8.50 s4 14.4 s3 39.2 s2 30.7 s1 2.61 3.1 ± 1.3
Presentation Mode Bundled 100 Stepwise 0
Distraction? Yes 100 No 0
Error Yes 18.3 No 81.7 0.18 ± 0.39
Bayesian network learned on the basis of the experimental data, showing a prediction for a specific combination of values of the independent variables
14
In the original talk, demonstrations of the example network and influence diagram are given instead of this and the following three slides. The tool depicted is Netica, which is available from http://www.norsys.com.
Execution time (msec)
Two-step sequences:
Execution time (msec)
40
6000
15
15
The Decision Mechanism
16
Learned Bayesian Network (2) Number of Steps Four 0 Three 100 Two 0
Presentation Mode Bundled 100 Stepwise 0
Execution Time s9 0.65 s8 0.65 s7 1.57 s6 0.92 s5 3.79 s4 7.71 s3 31.8 s2 48.8 s1 4.18 2.7 ± 1.2
Distraction? Yes 40.0 No 60.0
Yes No
Error 10.0 90.0 0.1 ± 0.3
The same network as in the previous slide, showing a prediction made under uncertainty about the independent variable "Distraction?"
Learned Bayesian Network (3) Number of Steps Four 0 Three 100 Two 0 Execution Time s9 0 s8 0 s7 0 s6 0 s5 0 s4 100 s3 0 s2 0 s1 0 4
Presentation Mode Bundled 100 Stepwise 0
16
Distraction? Yes 92.3 No 7.72
Yes No
Error 100 0 1
The same network as in the previous two slides, showing the interpretation of an observation of U’s performance
17
The Decision Mechanism
17
Influence Diagram Number of Steps Four 0 Three 100 Two 0
s9 s8 s7 s6 s5 s4 s3 s2 s1
Execution Time 0.65 0.65 1.96 1.31 8.50 14.4 39.2 30.7 2.61 3.1 ± 1.3
Presentation Mode Bundled -6.0679 Stepwise 0
Weight of Error w28 0 w25 0 w22 0 w19 0 w16 100 w13 0 w10 0 w7 0 w4 0 w1 0 16
18
Distraction? Yes 100 No 0
Yes No
Error 18.3 81.7 0.18 ± 0.39
Utility
An influence diagram defined as an extension to the BN of the previous slides
Properties of the Learned Decision Mechanism18 Learned Bayes net • Embodies experimental results • Also allows probabilistic prediction and interpretation Influence diagram • Generates a decision for each situation • Computes general policies Summary of Internal Policy Table Steps
Distraction = "No"
Distraction = "Yes"
Four Three Two
Stepwise iff w > 9 Stepwise iff w > 21 Always bundled
Stepwise iff w > 3 Stepwise iff w > 6 Stepwise iff w > 9
19
Fallbacks
20
Fallbacks 19
Fallback 1: Modify the Model by Hand (1)
Motivation • Real application situation is different from data-collection situation Procedure • Replace learned relationships by theoretically based formulas Prospects − New aspects may be highly speculative + Decision-theoretic tools permit sensitivity analyses
Fallback 1: Modify the Model by Hand (2) Three-step sequences:
Three-step sequences:
60
9000
60
9000
8000
8000 50
50
40
6000 5000 4000
30
20
3000 2000
40
6000
Errors (%)
Execution time (msec)
7000
Errors (%)
Execution time (msec)
7000
5000 4000
30
??
20
3000 2000
10
10
1000 0
20
1000
No
Yes
Distraction?
0
No
Yes
Distraction?
Original situation
0
No
Yes
Distraction?
0
No
Yes
Distraction?
Sliders instead of buttons
21
21
Fallbacks
22
Fallback 2: Collect Real Usage Data
Motivation • Experiment is not feasible or could not be realistic Procedure • Learn influence diagrams while system is in real use Prospects − Massively missing data may make useful learning impossible
Fallback 3: Do Analysis Without Data Motivation • [Same problems as above] • Unwillingness to deal with decision-theoretic tools Procedure 1. Draw influence diagrams on paper 2. Graph hypotheses about causal relationships 3. What’s the reasoning behind adaptation?
Prospects + You can check the assumptions on which your adaptation policy is based + You may decide to change or reject the policy
22
23
Fallbacks
24
Interactors Revisited
23
RADIO BUTTONS
INTERACTOR
QUESTIONNAIRE
CONTROL PANEL
CONTROL PANEL
QUESTIONNAIRE
SPEED
VALIDITY
CHECK BOX
VALIDITY
SPEED
TYPE OF FORM
RADIO BUTTONS
CHECK BOX
UTILITY
Check boxes or radio buttons for Boolean data?
Conclusions Conclusions Content [Empirical results and theoretical analysis concerning ways of presenting instructions]
Methodology 1. Rule-based adaptation is often too simple ⇒ Consider decision-theoretic adaptation 2. An optimal adaptation mechanism can in principle be learned fully automatically from empirical data And it has useful additional functions 3. Theory-based tweaking is often necessary It can be more or less reliable Decision-theoretic tools can help explore possible strategies 4. Even purely conceptual decision-theoretic analysis can be useful
24
