Creating Novel User Interfaces Albrecht Schmidt Pervasive Computing and User Interface Engineering
University of Duisburg-Essen http://www.pervasive.wiwi.uni-due.de/
[email protected]
How many computer did you use today? Machines/devices with a processor Machines/devices with electronics included Machines/devices which are powered Machines/devices that process/use/acquire information
User Interface Engineering • • • •
User centered design process? Focusing on user needs? Intuitive user interfaces? Straight forward to do?
Novel Uses of SMS (1) designed for user‘s needs?
http://www.textually.org
Novel Uses of SMS (2) designed for user‘s needs?
http://www.textually.org
User Interface Engineering • • • •
User centered design process? No tt he Focusing on user needs? wh ole Intuitive user interfaces? sto Straight forward to do? ry !
• Developing new technologies with a focus on human use • Evaluating these developments with people • Allowing technology to drive innovation • Situating new technologies in the context of use
technology creates user needs?
Sketch Pad 1963
Mouse 1964
…look back at the example of electricity
and extrapolate to ubiquitous information access…
User Needs & Technology Drive • Looks at user needs on a more general level (e.g. Maslow’s hierarchy of needs)
• Successful designs have addressed – – – – – – –
Survival Safety Food Love Communication Recognition / admiration …
• Allow technology to drive actual applications
Definition User Interface Engineering User Interface Engineering is a structured approach for designing and implementing useful and usable interactive systems. By following the user interface engineering process the interactive qualities of a system are ensured.
User Interface Engineering • create user interfaces in a structured way • ensure system properties by design – – – – – –
utility efficiency usability pleasurably durability openness
• research challenges – – – –
develop models to allow prediction and validation systematic exploration of modalities and interaction techniques toolkits and development support designing specific interfaces that allow user creativity
Models KLM for Phones and Beyond
Task Switch your mobile phone to silent mode Get your phone
3.89 sec.
Remove key lock
0.99 sec.
Enter PIN
1.72 sec.
Wait for confirmation
2.70 sec.
Enter silent mode
0.99 sec.
Enter key lock
0.99 sec. -----11.28 seconds
Task Switch mobile phone to silent mode Get your phone
Initial Preparation
I
3.89
Remove key lock
Key [long]
K
0.99
Enter PIN
4*Key, Hotkey
4*K, K
1.72
Wait for confirmation
System Response
R
2.70
Enter silent mode
Key [long]
K
0.99
Enter key lock
Key [long]
K
0.99
Calculate Interaction Time Texecute =
∑n
op∈OP
op
⋅ op
OP
set of available operators
nop
#occurrences of the operator op
I + 3*K[long] + 4·K[hotkey] + K[hotkey] + R = 11.28 sec.
Basics to Build On • GOMS (Card, Moran Newell, [7,8], 1983) • Cognitive modelling (ACT-R) • Keystroke-Level Model (KLM)
Interactions with Mobile Phones Input capabilities: • Keypad, hotkeys (numbers, text, control) • Microphone • Camera • Sensors (like accelerometers) • Bluetooth etc. • Readers (tags)
Interactions with Mobile Phones Input capabilities: • Keypad, hotkeys (numbers, text, control) • Microphone • Camera • Sensors (like accelerometers) Used as: • Bluetooth etc. • Remote control, data storage, personal key, organiser, payment, • Readers (tags) •
Enabled additional interactions: • Speech • Gestures • Location sensitive • Marker detection (RFID, NFC, barcodes, 2D visual codes) • Pointing
browser “physical mobile interactions” (Rukzio et al. [32])
Peculiarities: • Size • More likely to be distracted • Need to switch attention between phone and real world
Goal
Offer Keystroke-Level Model parameters for advanced mobile phone interactions
Method • Experimental • • • • • •
Large set of studies Software on the phone Video analysis Eye-tracker Different but similar phones Total number of actions measured: 2134
Original & New KLM Operators
• • • •
Micro / Macro attention shift, S Gesture, G Finger Movement, F Distraction, X
• Selected Parameters
unchanged
not applicable
678 678
• Keystroke / button press, K • Homing, H • Pointing, P
678
• Drawing, D
678
• Mental Act, M • System Response, R
adopted
newly introduced
New KLM Operators: S • Macro Attention Shift, S[Macro]
Switch attention between phone and real world
• Micro Attention Shift, S[Micro]
Switch attention between phone parts
New KLM Operators: S • Macro Attention Shift, S[Macro]
Switch attention between phone and real world
• Micro Attention Shift, S[Micro]
Switch attention between phone parts
keypad
hot keys
display
New KLM Operators: S • Micro Attention Shift, S[Micro]
Switch attention between phone parts
New KLM Operators: S • Macro Attention Shift, S[Macro]
Switch attention between phone and real world
…
…
…
Read instruction on phone
M
1.35
Attention shift to poster
S[macro]
0.36
Movement to tag
P
1.00
Marker interaction
A[marker]
1.23
Process marker
R[marker]
2.58
Attention shift to phone
S[macro]
0.36
Validate
M
1.35
…
…
…
New KLM Operators: S • Macro Attention Shift, S[Macro] Macro Shift: 9 participants, 22-46 years, 2 female 121 detected shifts Frame by frame analysis of video tapes No significant difference in the two directions (t=0.57, p>0.36)
Switch attention between phone and real world
0.36 sec. (quartiles: 0.24, 0.44)
0,90
Macro Attention Shift
0,80 0,70 0,60 0,50 0,40 0,30 0,20 0,10 0,00 1
6
11 16 21 26 31 36 41 46 51 56
61 66 71 76 81 86 91 96 101 106 111 116 12
New KLM Operators: S • Micro Attention Shift, S[Micro]
Switch attention between phone parts
display↔hotkeys: 0.14 sec. display↔keypad: 0.12 sec. keypad↔hotkeys: 0.04 sec. 400
10 participants, 24-34 years, 6 female
Micro Attention Shift 350 300 250
Eye-tracking to detect more than 1500 shifts 3 pre-set tasks
200 150 100 50 0 1
59
117 175 233 291 349 407 465 523 581 639 697 755 813 871 929 987 1045 1103 1161 1219 1277 1335 1393 1451 1509
New KLM Operators: G • Gesture, G = 0.80 sec.
Movements with the phone treated as input
10 participants, 23-33 years, 5 female Samsung SGH-E760, built-in accelerometer, standard applications, games Simple gestures only
New KLM Operators: F • Finger Movement, F = 0.23 sec. (hotkey region only: F = 0.16) Analyzed 323 key presses Used S[Micro] to calculate Holds for semi- and fully experienced people Silfverberg et al. [36]: 0.27, Mori et al. [26]: 0.19
New KLM Operators: X • Distraction, X = 6% - 21% 10 participants, 24-33 years, 3 female Short message in 3 settings (quiet room, standing outside, walking)
Time to type one character (in sec.)
Relative slow-down (significant: t=2.23, p
