ISRE: Immersive Scenario-based Requirements ... - Semantic Scholar

ISRE: Immersive Scenario-based Requirements Engineering with Virtual Prototypes Alistair Sutcliffe Brian Gault Centre for HCI Design, Dept of Computation University of Manchester Institute of Science and Technology (UMIST) P.O. Box 88, Manchester M60 1QD, UK tel +44 (0)161 200-3315; fax +44 (0)161 200-3324 [email protected]

Neil Maiden Centre for HCI Design, School of Informatics City University, London EC1V 0HB

1

Abstract Virtual prototyping is a useful approach for refining requirements for complex designs; however, use of virtual reality technology can cause usability problems which can be interpreted as “false positive” requirements errors. The ISRE method guides the analysis of problems encountered during the testing of virtual prototypes and helps assign causes to either genuine requirements defects or to usability issues with VR technology. The method consists of techniques for walkthrough testing, testing with users, causal analysis of observed problems, and design of scenario-based analysis sessions. The method is described and its use illustrated with a case study of validating requirements for an aircraft maintenance application.

Keywords: RE methods, virtual prototypes, problem taxonomy, causal analysis.

1. Introduction One of the more important applications of virtual reality (VR) technology has been for requirements analysis using virtual prototypes [1]. The appeal of using VR is saving the cost of physical mock-ups and prototypes, especially for complex systems. This approach to RE blurs the distinction between requirements specification and design by creating a prototype which can be tested to refine the design, hence virtual prototyping follows in the genre of RE methods that incorporate a prototyping approach (e.g. SCRAM [2, 3]). Unfortunately the realism of VR is constrained by the realism of the 3D graphical images, and more critically by absence of haptic (sense of touch) feedback. Consequently the difficulties experienced with virtual prototypes may be caused by usability problems rather than by any design defects per se [4].

This problem has been partially avoided by augmented reality, in which critical components of the system are physically prototyped so the users can grip and manipulate physical controls while the VR provides visual rendering of the prototype and its environment. An example of such hybrid approaches is a naval gun simulation where the controls were physical but most of the system and the surrounding environment was virtual [5]. However, in many cases building physical mock-ups is too time consuming or expensive, so much testing has to be carried out on virtual prototypes. Although virtual prototyping has become established

2

practice, there is little understanding of the requirements analysis process with VE technology and the confounding problem of usability issues. In our previous research we produced usability evaluation methods for VR that diagnose problems arising from the user interface [6], and a design method for virtual environments (VEs) [4]. These methods, combined with other evaluation approaches in the VE literature (e.g. [7, 8]), provide the baseline for developing a VE requirements analysis which can also take user interface problems into account.

This paper describes a method for VR-RE, Immersive Scenario-based Requirements Engineering (ISRE), which provides guidance for requirements analysis via scenarios. Scenario-based requirements validation is a powerful approach which has appeared in several methods, e.g. ScenIC [9]; SCRAM [2, 3, 10], and more formally in obstacles analysis in KAOS [11]. Virtual prototypes have the potential to enhance scenario-based testing since aspects of the design (i.e. manifestation of the requirements specification) can be changed rapidly. In this respect VR-RE develops the theme of requirements validation by animation in which informal, concrete views of a specification are animated for user inspection, while being coupled to a more formal requirements specification that is hidden from view [12]. In VR this can be effected by coupling the virtual image to a CAD (Computer Aided Design) system so changes to the prototype automatically update manufacturing specifications. Furthermore, VR can change aspects of the system environment such as visibility and weather conditions, so environmental scenarios, which are critical influences on system failure [13], can be incorporated into testing schedules. However, scenario variations are potentially infinite, so this raises the problem of selecting an appropriate set of scenarios that gives acceptable test coverage for a cost. The ISRE method addresses this problem by proposing a scenario test schedule for virtual prototyping.

The rest of the paper is structured into four sections. First we propose a taxonomy of requirements problems which also includes user interface defects. This is necessary because virtual prototyping is considering not only missing or inadequate requirements but observable design defects which may be attributable to either the virtual prototype or operation of the VE itself. The next sections describe the virtual prototype requirements analysis method, followed by a case study applying the method to an engineering application. The paper concludes with future work and a discussion of the potential contributions and limitations of RE by virtual prototyping.

3

2. Requirements Problems and Usability Errors The aim of the method is to separate usability errors from requirements problems. Usability evaluation methods do not recognise requirements problems explicitly, so such methods from an HCI (Human Computer Interaction) perspective have to be extended to diagnose requirements. First we review the potential contribution of usability evaluation methods. 2.1 Usability Evaluation Methods

Usability evaluation methods focus on discovering potential operational difficulties which users may, or do, encounter when operating a user interface, either as a prototype, product or a specification. Methods can be grouped into analytic approaches which involve users interacting with real user interfaces, of which Cooperative Evaluation [14] is a well know exemplar; and model-based analytic techniques which operate on specifications, such as the Goals, Operators, Methods, Selection Rules (GOMS) [15], or Cognitive Walkthroughs [16]. The former provide processes for capturing and analysing observed difficulties experienced by users, whereas the latter provide metrics and techniques for estimating operational times and potential errors. A third class of methods are checklist and heuristic approaches, frequently referred to as expert evaluation in the sense that it helps, but is not essential, to have an HCI expert carry out the evaluation. Checklist methods use a series of questions, derived from HCI design guidelines, to audit the user interface (e.g. [17, 18]); while heuristic evaluation uses a set of principles or high level-design desiderata to perform a similar audit and explain the reason for potential usability problems[19]. The range of evaluation methods and the tools support available for them are ably reviewed by [20].

A summary of commonly used HCI evaluation methods is presented in Table 1. GOMS [15] is a model based method, used by experts, which analysis user interface specifications or design to derive performance times and error predictions. It doesn’t provide explicit guidance for diagnosing the causes of usability problems, although the metrics will pin point where errors are more probable in a dialogue sequence. In the User Action Notation framework [23], errors are interpreted with a model of interaction consisting of the user action, the system state, and system response. User problems may be observed or the framework can be applied to walkthrough specifications by expert HCI evaluators. The UAN approach was augmented by the User Problem Taxonomy[24] in which error types were associated with different stages

4

of Norman’s model of action [25], so the model could be “walked through” to guide the evaluator to potential causes of errors. UAN and the problem taxonomy focus on diagnosis of usability problems rather than error metrics. Cognitive walkthroughs [16] can be used in expert mode to inspect a user interface specification or as an instrument to interpret observed user problems. In the initial method diagnostic guidance was limited to simple error categorization, although the method has been extended to provide improved causal analysis of usability problems [26,38]. All the model based methods require considerable training and are generally used by HCI experts. Cooperative evaluation [14] advises on conducting evaluation sessions with users and observing critical incidents and breakdowns as manifestations of users’ problems. It contains little diagnostic guidance. Usability evaluation methods which involve users have tended to leave causal analysis of the observed problems to the evaluators’ expertise in interpreting principles or design rules [21]. This approach is shared with Heuristic evaluation [19] in which experts inspect an application noting problems and interpreting them in light of design desiderata such as consistency, minimal and aesthetic design, prevent errors, etc. However, several taxonomic approaches have been proposed that attempt to classify usability errors and provide diagnostic guidance towards their cause. Taxonomic analysis coupled to Norman’s model has also been developed for GUIs and VR user interfaces [4], while a simpler classification of the causes of usability problems [26] was developed to partition errors into user interface problems, poor task fit of system functionality, and missing requirements [27]. The usability problem extraction approach [28] focused on the context wherein errors were observed to guide the evaluator towards probable error causes. HCI therefore provides a rich source of evaluation methods which can be adapted to requirements validation, especially when prototypes (virtual or real) exist. The problem is to factor out errors that require changes to the user interface from more deep seated problems arising from requirements analysis failures. The ISRE method builds on this research and applies it specifically to virtual prototyping environments. 2.2 Overview of the Approach

There are three potential sources of user problems which are not attributable to the virtual prototype itself; instead, they have causes in the limitations of VR technology, as follows:

Operation of the user’s presence. The user may be represented in the virtual world by a simple cursor or more commonly by a hand or a whole body avatar. The presence may be controlled by a variety of devices ranging from 3D mouse, space ball, joystick to pinch

5

gloves. Many VEs implement controls that allow users to fly through VEs to reach and select distant objects by ray-casting. The user’s presence and controls can cause many problems since they provide a less than perfect rendering of the user’s natural action. Complex manipulations may be particularly prone to presence control problems.

Lack of haptic feedback. True virtual prototypes have no haptic feedback (sense of touch) so the user’s presence can pass through representations of solid objects. Problems caused by absence of haptic feedback can be frequent with complex manipulations and physical tasks. To mitigate the lack of haptic feedback many applications use visual feedback with collision detection algorithms to prompt users when objects are selectable or have been selected. These problems can be avoided by designing augmented reality prototypes in which interactive surfaces are modelled as physical mock-ups, but in many VEs this is too expensive, so haptic errors need to be factored out.

Inadequate graphics. VEs may not do justice to the presentation of the prototype, since most applications are not rendered in photorealistic detail. Although some evidence suggests that people can perform tasks naturally without detailed visual representations [29], graphical detail will be important for information displays, and tasks when the virtual prototype or the system environment is visually complex. In some cases audio input may also be important.

The ISRE method aims to diagnose and eliminate usability problems in VE technology so they are not reported as false positive requirements defects.

3. The ISRE Method for Requirements Analysis The method is structured in three steps, as illustrated in figure 1. The first step defines the users’ tasks and scenario testing schedule, then conducts an expert audit. This is followed by an expert walkthrough and interaction analysis with users, while the third step, data analysis, pinpoints problems attributable to usability of the VE or design of the virtual prototype. The overall method follows an iterative cycle of testing and design improvement, in the tradition of user-centred design [30].

6

Configuring the VE for analysis is not covered in depth in this paper, since we have described guidelines elsewhere [4]. However, for completeness, configuration choices include the type of VR technology: CAVE, immersive workbench, immersive Head Mounted Display or nonimmersive desktop VR. Desktop VR is cheaper but gives less realism than the other options. The requirements engineer might be immersed to see the virtual world from a user’s viewpoint in a walkthrough. Alternatively, the requirements engineer might observe the environment externally to investigate the behaviour of avatars that may either be controlled by immersed users or may be scripted. Controls for changing the external viewpoint (i.e. from behind, above, from the side, etc.) can help investigation. 3.1 Technology Audit

The first step is to audit the virtual prototype for naturalness. The more faithful the representation and interaction in the VE are to the real world, the more accurate requirements capture is likely to be. User interface design features intended to improve usability may result in a less than fully natural representation. The following checklist is used when exploring the VE to assess how user controls and representation of the environment may interfere with the prime task of discovering requirements defects. The features are scored as either present or absent and assessed for how easy to use or how effective they are, using standard Likert (1-7) scales. The more UI design features that are present and poorly designed (i.e. score 1-2 on ease of use), the more interference will be encountered from usability defects during requirements analysis.

Interactive techniques: (i)

User movement is commonly enhanced by flying through the VE using hand gestures, or metaphors such as magic carpets. These may not interfere with requirements if they simply help the user arrive at the location in the VE where the scenario operations are carried out.

(ii)

Selection of distant objects can be facilitated by ray-casting techniques [8] that either cause the user’s presence to travel to the selected objects or, vice versa, the object to move to the user. This enhancement is clearly unnatural in physical manipulation tasks.

(iii) Selection of proximal objects may be helped by collision detection algorithms and “snap-to” actions when the selectable object jumps to the user’s presence if it

7

approaches within close proximity. Snap-to will interfere with requirements by making manipulation tasks easier. (iv)

Selectable objects change colour when approached, then another colour may be used to provide feedback that the user has selected the object. This technique may be a necessary compromise with naturalness to enable effective interaction in the absence of haptic feedback.

(v)

GUI (graphical user interface) controls such as pop-up menus, hover-text boxes and floating palettes all deviate from a natural environment, although they may be justified if they help the user navigate or select functions in the VE.

Techniques (iv) and possibly (i) are frequently necessary for VE operation, so may be counted as acceptable, but the other techniques are unnatural extensions.

Feedback and representation features: (i)

Areas of the VE (e.g. background) may not be rendered in detail. This compromise with realistic graphical detail is not as serious as it may seem, since natural interaction is observed even with low fidelity graphical worlds [14]. However, if visual inspection of the VE is critical for the task then this compromise may be more serious. Critical controls and interactive objects should be displayed in more detail.

(ii)

Lack of haptic feedback is a common limitation of many VEs. In physical tasks with complex manipulations this will be a serious compromise, hence augmented reality should be considered.

(iii) Provision of high quality audio can make the VE seem to be more realistic for the user. (iv)

Implementation of gravity and physical laws governing objects helps to enhance naturalness. Allowing objects to remain suspended in space when they are released gives an obvious signal that the VE is an artificial world.

(v)

Allowing users to move through solids compromises realism; however, when there is no haptic feedback, this can be difficult to prevent. Programming movement constraints into the VE, e.g. ensuring users can only move on a 2D surface, or enforcing precise placement of objects, can confuse users because they have to learn the constraint rule in the absence of normal haptic sense. In many cases constraint rules cause more usability errors which obscure requirements problems.

Representation features (iii) and (iv) are desirable advantages; while the others are undesirable, they are common limitations of technology. A naturalness audit has to take the task, domain and available technology into account. The outcome will be a list of UI design

8

features which can either be sources of usability problems or may cause “false positive” results for requirements analysis.

Assessing presence: Naturalness can also be audited by assessing presence, or the sense of “being there” in a VE. Presence is important because the more engaged the user is and the less aware of the illusion of virtual reality, the closer the approximation between interaction in the virtual and real worlds. Presence is usually measured by questionnaire checklists, that capture users’ ratings of their experience in using a VE [31, 32]. The most comprehensive presence inventory from Witmer and Singer [33] consists of 20 questions (see appendix). As well as providing a global measure of naturalness, users’ ratings of individual questions also provide a useful cross check for usability errors and potential interaction problems with the virtual prototype. For example, several questions relate to control and interaction: able to control the system (No. 1), natural interactions (3), natural movement mechanism (4), examine objects (13), easy to move or manipulate objects (14); while other questions elicit the user’s perception of interaction quality: sense of movement compelling (11), proficient at moving (18). Low scores on the presence questionnaire indicate that the users’ experience with the virtual prototype may not map to their experience with the real world product in the future. 3.2 Requirements Analysis Walkthrough

This technique involves the analyst walking through the VE using him or herself as a surrogate user, and asking questions at set points in the operational scenario. The requirements engineer follows a scenario script to the best of his/her ability, role-playing the user’s task as closely as possible to real life. Unexpected difficulties and problems in system operation are noted, especially critical incidents when difficulties were encountered in deciding what to do next or understanding the system’s state. The point in the scenario when the incident occurred is noted, as well as any design features implicated in the problem. The walkthrough produces a list of design faults and recommendations for improvements. Walkthroughs are quick and cheap to do, but may suffer from the analyst’s lack of domain knowledge. The process is adapted from the VE evaluation walkthrough proposed by Sutcliffe and Kaur [6]. This extended Norman’s model of interaction [34] for virtual environments by proposing three interaction cycles. The most important cycle for requirements analysis is task-based activity, in which the user is trying to achieve a goal. Subordinate to this cycle is

9

navigation/exploration when users are finding their way around the virtual environment. The third cycle describes system initiative when the user’s role is to interpret actions taken by the system and respond to them. Further details of the interaction theory and its validation are reported in Kaur [35].

The task cycle starts with goal formation (see figure 2, step 1) in which the user forms an intention to achieve a task goal or sub-goal. Ideally the VE should contain an object or tool (i.e. a manifestation of functional requirements) that allows the user to carry out the task. The user’s first problem is whether the immediate environment contains the necessary objects for interaction. If the objects are not visible then the user has to search the environment for them (step 2). This may lead into the navigation sub-cycle and exploration until the necessary objects are found or the user gives up. Assuming that a familiar part of the virtual environment is found and contains the necessary objects, then the user has to locate the objects (step 3), approach and orient the representation of self in the correct position for interaction (step 4). With large and simple objects these steps will not be necessary, as indicated by the dashed line that by-passes them.

Specifying action (step 5) – deciding what action to carry out – requires an object or tool in the VE to provide hints/prompts to suggest the appropriate action. This is often referred to as an “affordance” in the HCI literature [25], a familiar example being a door handle which suggests pushing, pulling or depressing down depending on its shape. Specification of action is frequently determined in response to artefacts in the environment, or “situated” action [36]. Specifying action is followed by manipulating objects and executing actions (step 6). Actions should be followed by feedback from the environment. In continuous actions, manipulation and feedback are interleaved as the user may adjust action in light of feedback. First the user must receive and recognise the feedback (step 7) and then interpret the nature of the change (step 8). The artefact should indicate completion of the operation; however, feedback may also be necessary to support interaction. Once an action is complete the user may continue with specification of the next action (step 9), which normally leads to approach and orientation, unless the necessary objects are not visible, in which case the sequence forks to step 2. Alternatively, if completion of the action has reached the end of a procedure then the user proceeds to the next sub-goal and the cycle resumes at step 1.

Two other cycles describe different interaction contexts; however, these are less important for the ISRE method and have not been included. Briefly, these are system initiative, which

10

reverses the task-action cycle by starting with the stimulus which needs to be recognised and interpreted; and the exploration sub-cycle. In serendipitous exploration when the user has no specific goal in mind, interaction is composed of a series of scan environment/decide direction – navigate – interpret change loops. Further details of the models and usability requirements can be found in [4].

3.2.1 Walkthrough Process

First the appropriate scenario has to be selected. Then the requirements analyst follows a scenario of interaction with the VE, asking appropriate questions for each model step, e.g. can I carry out the task goal? Can I locate appropriate objects/parts of the VE? etc. The interaction support features in figure 2 help to refine the questions. To help this process a taxonomy of usability and requirements problems is provided based on previous studies in requirements engineering [22, 37] and usability evaluation [27, 38]. Requirements defects may be omissions when no facility or component exists in the virtual prototype; alternatively, the design might be inadequate in some way because it does not support the user’s goals. The following lists classify requirements and usability problems with indications about how they might be discovered during the walkthrough. The problems can be caused either by poor design of the virtual prototype or defective usability. The task-action model (see figure 2) provides a means of interpreting the context of interaction to prompt evaluation questions for each stage of interaction, as recommended in cognitive walkthroughs for standard user interfaces [16].

Requirement defects (goal or assess feedback stages): •

Task

fit

(missing

functionality):

the

prototype

does

not

contain

a

function/tool/interactive object to enable the user to achieve his or her goal, indicating missing requirements. Missing requirements are usually reported directly but may be indicated by the user searching unsuccessfully for a tool or object. •

Missing information: the information content necessary for the user’s task, decision or learning objectives is not provided. Missing information is more likely to be reported directly in decision support or training applications.

•

Task fit (inadequate information or functionality): users can partially achieve their goals but they find it difficult to do so because the information provided is incomplete, ambiguous or inappropriate, or the function does not exactly match their expectations. This is manifest when users have difficulty in operating the prototype.

11

The design causes can be diverse, ranging from poor affordances for action, inflexible and unnatural means of operation, to inadequate automated functions.

Interaction problems (following cycle stages from locate to assess feedback): •

Search problems (poor location): the VE does not guide the user to find the appropriate information, objects or commands even though they exist. Possible causes are poor representation of the VE, misleading metaphors and ambiguous spatial representation.

•

Cue/prompt/metaphor (predictivity): the interface does not help the user guess how to operate the prototype or tool. This problem is related to poor task fit, ambiguous metaphors or poor affordances for interaction.

•

Disorientation: the user is lost and cannot find the required items; possible causes are poor navigation controls, and misleading navigation cues in the VE.

•

Manipulation/operation: movement of the user’s presence or other manipulations are difficult because the target is too small, or hard to operate, or carrying out the action exceeds the normal physical coordination abilities of the user.

•

Performance problems (non-functional requirements): the system can be operated effectively but it doesn’t respond rapidly enough or cannot process the necessary volume of input.

•

Missing feedback: no message or effect is visible or audible. Changes to the virtual tool/object are not signalled or cannot be recognised.

Information delivery problems (recognise, assess feedback): •

Recognition failure: the information is present but the user fails to see or hear it. This can be apparent when signals and warnings are missed, but it may only be discovered in de-briefing interviews. Information presentation defects are the cause.

•

Comprehension failure: the user did perceive the information but could not make sense of it. Poor integration with existing knowledge is the usual cause but comprehension failure may also be a consequence of inappropriate presentation. User-related causes need to be explored, such as poor motivation or lack of domain knowledge. Objects in the VE do not change in a manner expected by the user so the effect cannot be understood.

Other problems may emerge at several stages:

12

•

Hidden effects: modes or parameter settings are not apparent or have been forgotten by the user, leading to unexpected effects. This often results from missing feedback.

•

User error: the system functions correctly but is used sub-optimally for the task. This may be a training problem but task fit problems should also be considered.

Errors are identified by design feature and rated for severity. Expert walkthroughs are quick and cheap to carry out because they don’t involve user testing. Studies on walkthroughs suggest 4-6 experts should be sufficient to trap most problems [39]; however, experts don’t see the world from the users’ viewpoint so they are prone to miss requirements related to task support, unless they are also domain experts.

4. Interaction Analysis

In this approach, users carry out tasks in the virtual environment. The requirements analyst observes the users and asks them to think aloud. Problems are detected by observing difficulties in operating the virtual prototype, by failure to achieve operational performance targets, or by misunderstanding and difficulties reported by users during interaction and in debriefing interviews. Analysis is diagnostic in nature, i.e. when the user experiences some difficulty, the analyst has to trace the problem back to either a fault in the design or a problem in the operating procedure, or possibly a false assumption about what is humanly possible. After the evaluation session, de-briefing interviews follow up on observed problems.

Interaction analysis with users may be more expensive in analysis time, but it pays off in better requirements capture and validation. Having gone to the expense of creating a virtual prototype it is important to capitalise on the investment. Virtual prototypes can be used in slightly different versions to gain feedback on different aspects of system operation, e.g. •

Physical and spatial requirements: checks that people and components can fit into the spaces specified, that movement of objects is physically possible and that people can move in constrained spaces, reach switches, etc. Close coupling between spatial specifications in CAD models and VEs is advisable to ensure spatial requirements are reflected in design models.

•

Human operations are feasible and effective: elicits requirements for an effective design including operational controls. Scenarios are walked through to check that the user can find and understand the necessary information and carry out task actions.

13

Limitations of VE come into play here if operations involve haptic feedback, although augmented reality can avoid these problems. •

Operation will be effective under a variety of environmental conditions: the VR design is run under a variety of simulation conditions, e.g. degrees of lighting, visibility, noise, etc. The scenario simulation environment that surrounds the basic design can be changed for different test runs.

•

Operation will be effective with different user roles: immersed actors play different user roles (e.g. trained, novice), or automated agents are scripted to respond with different degrees of skill.

4.1 Interaction Analysis Process

The first step is to select users and a representative set of tasks. Tasks are selected to cover normal operations and exceptional conditions (see normal and alternative pathways in use cases: [40]). Choosing tasks involves a trade-off between increasing test coverage and the cost of running analysis sessions. Ideally test subjects should complete a variety of tasks but in practice the number of evaluation tasks depends on the system’s complexity and availability of test subjects. A range of users/stakeholders should be chosen, who are likely to be domain experts and operators of the equipment in the virtual prototype. If possible the sample of users should include individuals with different levels of training and backgrounds.

Evaluation starts with scenarios involving normal tasks with skilled users and good environmental conditions. The idea is to stress-test the prototype by making operational conditions progressively worse until the limits of safe and reliable operation are established. The framework for scenario-based testing is summarised in figure 3.

First the basic ergonomics of the application are tested. This involves simple scenarios of physical tasks such as reaching for controls, and manipulating objects to make sure designed components can be reached and manipulated within acceptable bounds of human ability. This testing can be carried out using an ergonomically engineered mannequin instrumented to detect normal human physical abilities and anthropometric properties, e.g. fit in constrained spaces, comfort in sitting positions, etc.

The next level uses scenarios that describe the operational tasks that are acted out in the VE either by the user controlling a “presence” in the VE (e.g. virtual hand, arm, whole body), or

14

by scripting an avatar to carry out the actions and observing the result. Scripting is more time consuming; however, the process of writing scripts can reveal requirements problems since a precise specification has to be created. Once normal tasks have been completed, exception or error condition tasks are tested. At this stage the analyst is looking for problems in completing user actions, and how well the system provides information to help the users.

A second layer of tests may be run by systematically varying the ability of the human operators (i.e. selecting a range of skilled and non-skilled users) to test the potential impact of human mistakes on system requirements. The third layer runs the prototype with the same operational scenarios but adds variations of environmental conditions to stress-test the design. Choice of environmental parameters depends on the application domain; however, more typical ones will be changing the weather conditions to worst case, e.g. night time, poor visibility, stormy conditions. Visual and audio parameters can be changed easily within VEs, whereas motion-related environmental effects require simulator technology. Setting the system environment to worst case then repeating the test with a set of better conditions gives a quick analysis about the likely impact of the environment on system operation.

The user is asked to complete a task during each session. The time taken to complete the task and performance measures, i.e. how successfully the task was completed, are recorded. The requirements analyst observes the users and asks them to think aloud. When the users are puzzled, hesitate or can’t proceed with the task, a problem is recorded. Problems are detected in three ways: •

observing difficulties in operating equipment

•

failure to achieve operational performance targets

•

misunderstandings and difficulties reported by users either during the session or in de-briefing interviews.

Analysis of observed errors is diagnostic in nature, i.e. when the user is observed to experience some difficulty, the analyst has to trace the problem back to either a fault in the design or a problem in the operating procedure, or possibly a false assumption about what is humanly possible. Requirements are captured in the form of design critiques which update the requirements specification and subsequently become design improvements. The cycle of scenario-based testing is repeated until an effective and usable solution is produced.

4.2 Requirements Problem Analysis

15

Unfortunately, problems observed in a VE may have many causes, several of which have no requirements implications for the prototype. The analysis procedure, therefore, has to tease apart problems caused by poor usability from genuine requirements defects. Problems may have one of three high-level causes: •

User training problems. Users do not understand or follow the scenario script; these problems are caused by poor user domain knowledge or training, or misunderstanding of the scenario script.

•

Usability problems. Users experience difficulty with VE features that do not directly represent the design solution, such as the navigation controls. For example, problems in controlling movement, navigation, or conceptual disorientation, limited realism or haptic feedback, mean that manipulations appear to be more difficult than they would be in reality.

•

Design problems. Users have difficulty with an aspect of the VE that directly represents the prototype design. These problems point to requirements issues and design refinements.

To tease apart the many potential causes of observed problems a set of decision trees are provided. These are integrated with the task-action model shown in figure 2, so they can be used either in the expert walkthrough to diagnose probable problems at each step or to follow up an observed user problem. The decision trees eliminate usability problems by a series of questions, leaving residual errors to be attributed either as requirements failings or lack of user knowledge. The latter will only occur when domain novices are used as test subjects. At the beginning of a task, if the user can’t proceed or is puzzled then either missing functionality (requirement not implemented), hidden functionality error (can’t find function/affordance), task fit error (missing functionality) or a user error (poor task/domain knowledge) is indicated.

The first question in the decision tree at this stage (see figure 4 ) traces problem causes to a lack of user knowledge or vague intentions. This is unlikely if domain experts are the test subjects, so the next question focuses on the scenario which may be misleading. If the scenario was clear, then there is a requirements defect or a usability problem. If the user was navigating or trying to interact with the VE controls then a usability problem is indicated; otherwise, a requirements defect has been found. The final question tests whether an appropriate command, control or feature of the virtual prototype can be found, leading to

16

several potential defects. If the desired feature is known to have been implemented then analysis proceeds to the next decision tree, illustrated in figure 5.

The location analysis tree traces problem causes in finding tools, affordances and commands. Initial questions probe whether the user knows where to look, since problems may be caused by a lack of familiarity with the VE. The next question focuses attention on the design, which should provide hints about where the control/interactive feature is, and points to design requirements to help user navigation. The final question raises the possibility that even if the design does provide location guidance the user may not have explored it adequately. If the user has found a target object/tool/VE feature but hesitates or can’t proceed, a cue/affordance/metaphor error is indicated. The action/manipulation tree (figure 6) traces causes of these errors either to a lack of integration with feedback over complex actions, or poor devices that don’t support precise motor control. Questions first eliminate haptic feedback problems, then one branch of the tree explores design to improve feedback by substituting another modality. Questions on the other branch explore the difficulty of the physical operation and how it might be improved. In these cases the problem may be an excessively difficult manipulation or may reside in usability defects in the operation of the user’s presence.

Problems at the action stage may also involve feedback (see figure 7). Questions in this decision tree probe whether feedback is recognisable, comprehensible and makes sense in context. Choice of an inappropriate modality may also be responsible (e.g. physical action expected when speech commands are implied by the VE) or a virtual tool may have obscure controls. After a successful action, if the user can’t proceed or is puzzled then a goal formation problem is indicated, due either to missing/inadequate functionality or task compatibility error. If the user has completed the action but is puzzled by an unexpected effect then feedback may be either inadequate or absent. The tree traces causes back to problems in user knowledge and fit with their task; however, feedback problems often uncover hidden causes in modes or mismatches between the user’s and system models. Mode errors are caused by hidden settings in the prototype or the VE interface that either lead to unpredictable effects or users trying operations which the system does not allow.

The output from the diagnostic analysis is a list of observed problems with causal attribution, either to the virtual prototype or to usability defects with the testing environment. Problems

17

are ranked for their severity of impact, i.e. for the prototype requirements: essential, desirable, additional extra; and for the usability problems: must solve, solve if possible, can tolerate.

5. Case Study The application was an aircraft maintenance training VE which contained an aircraft fuselage with an assembly which had to be removed (see figure 8). The application displayed an aircraft fuselage with two brackets attached by bolts: three holding the right-hand bracket, four holding the left-hand bracket. The view of the brackets was partially occluded by part of the fuselage. The task was to remove the brackets from the fuselage assembly, first removing the bolts holding the right-hand bracket and then the left-hand bracket. The subject had a choice of using a screwdriver or their virtual hand to perform the tasks. To select the screwdriver interaction mode, the user needed to pinch thumb and little finger together; then releasing the pinch selected the screwdriver tool. The screwdriver and virtual hand functioned as one unit until the user pinched thumb and little finger again and then released them, which deselected the screwdriver. The user could then use the virtual hand to remove the bracket. The screwdriver would only interact with the bolts while the virtual hand would only interact with the brackets. When the virtual hand and screwdriver approached the bolt it changed colour to signal selection was possible. The user pinched thumb and index finger to select the bolt, which snapped to and moved with the hand-tool as one unit so long as the user did not release the pinch. Releasing the pinch released the bolt. The user’s task was to undo the bolts with a screwdriver tool then remove the bracket manually. The requirements focus was on the bolt “screwdriver” tool which once engaged would automatically undo the bolt or refasten it according to the user’s command. The bolt screwdriver was also intended to be magnetic so loose bolts would be attracted to the tool. These functions were imitated in the VE by a “snapto” function whereby selected bolts automatically jumped into position with the tool. The VE was implemented in a fully immersive CAVE environment in which the user wore shutter glasses to give the 3D stereo image and interacted via a pinch glove. The pinch glove was represented as a virtual hand in the VE. Closing the thumb and index finger selected an active object (e.g. bolt, bracket) if it was in close proximity. Interaction between the user’s virtual hand and interactive objects (the screwdriver, bolts and the bracket) was governed by a collision detection algorithm. 5.1 Technology Audit of the VE

18

The VE had no particular navigation features since only a relatively small virtual world was represented, and no enhancement was present for selecting distant objects. No navigation or distance selection techniques were implemented; however, collision detection was used with colour changes to signal selected objects (the screwdriver, bolts and bracket) since no haptic feedback was available. Furthermore, the bolts “snapped-to” when the screwdriver was placed close by and this represented a serious compromise with reality. On the feedback side the haptic and audio modalities were not used, no gravity was implemented so bolts could float in thin air, much of the VE was rendered in minimal detail and the user’s presence could pass through solids such as the aircraft fuselage. These representation features were a further compromise with reality. No GUI features intruded into the VE, so the design received a good score on interactive controls. Much of the VE background was not rendered in detail. The presence audit gave an overall score of 4.8 on a 7-point scale. The lower range ratings that gave cause for concern were poor awareness of events (3.2), match to real world (4), natural action (4.4) and movement controls (4.5). The upper range ratings concerned the user’s ability to adapt to and operate their presence in the CAVE, although low scores on quality of interaction and control of movement indicated that control of the user’s presence may be less than natural. 5.2 Walkthrough Analysis

Space precludes description of all the walkthrough analysis so the task of selecting the screwdriver tool has been chosen to illustrate the process. Following the task-action cycle the evaluation questions produced the following results: 1. Form goal. The user’s goal is to pick up the screwdriver, which suggests a natural mapping between the virtual and real-world hand; no immediate problem is apparent. 2. Locate active environment. This step presents no problems since the screwdriver tool is visible. 3. Approach and orient. Again no immediate problems are apparent, but there are latent action precision difficulties which emerge in stage (5). 4. Specify action. The natural assumption would be to just pick up the tool with the virtual hand; however, absence of haptic feedback means that usability concerns

19

intrude at this stage. The thumb-little finger pinch selection is not intuitive and has to be learned by the user. This is an overhead which has nothing to do with the real-world task and constitutes a task fit usability problem. 5. Manipulate object. Superficially this may seem to be easy, assuming the user remembers the pinch operation; however, successful interaction depends on positioning the virtual hand in close proximity to the screwdriver tool to trigger the collision detection algorithm, so there could be a missing/inadequate feedback problem. 6. Recognise feedback. This depends on precise positioning of the virtual hand so the screwdriver tool changes colour to red, indicating that it is selectable. Since the bounding box around the screwdriver is large, selection is reasonably easy, although the perception of selecting the tool when the hand is clearly not gripping it makes the application feel less natural. 7. Assess feedback. Once the user completes the pinch, the screwdriver tool snaps to the virtual hand and the two objects are manoeuvred as a single composite object. This departs from reality since no continuous grip has to be exercised, whereas in the real-world task learning the degree of grip force to exert may be important. This sample of the walkthrough illustrates how problems and compromises with the realworld task can be discovered. The complete walkthrough indicated several task fit requirements problems where the system’s operation did not fit the users’ expectations of operating the tool, such as tool selection, selecting the bolt with the tool (the bolt snap-to was unrealistically helpful), undoing the bolt (no time delay was apparent) and removing the bolt (no pull out of socket movement was necessary).

5.3 Interaction Analysis

Time did not allow testing scenario variations in this study, although variations in lighting were planned for future studies. No environmental variations could be analysed during the available time. In this evaluation ergonomic testing was merged with task operation since spatial requirements did not appear to constitute a problem.

The application and task were simple; however, observation of users’ difficulties revealed several problems with the prototype and VE usability. Eighteen undergraduate students acted

20

as users. They worked in pairs; one subject acted as the observer and recorded problems encountered by the other subject who carried out the experimental task of undoing the bolts and removing the bracket. The user/observer roles were then reversed. The sessions were video recorded. Problems were analysed by integrating the observers’ records of problems made during the session, with observation of users’ problems from the video and audio recordings of users’ experience.

The observed problems are shown in table 1. The users were frequently unsure about how to proceed. They had received a short training period and written task instructions which were also read aloud to them. They often asked questions about what to do next; this points to the top pathway in the goal/function decision tree (see figure 4) and a lack of task knowledge.

The major problems encountered by all of the subjects were: 1. Problems with screwdriver and virtual hand when selecting and deselecting objects (bolts and brackets) during object manipulation, relating to both colour feedback and to use of the pinch glove. 2. Selecting the screwdriver tool, caused by mode confusion and manoeuvring difficulties to trigger the bounding box algorithm and pinch glove operation. Nearly all users experienced this problem, which disrupted normal interaction. 3. Perception problems when the subject could not see the bracket assembly because it was occluded by the fuselage. Two interpretations were possible: either a manoeuvring problem in seeing and finding the bracket assembly; or a task/domain knowledge error by the user who was unaware of the bracket’s location. 4. The subject indicated that he/she could not reach the desired object because of “perceptual distortion”; this was related to problems in manoeuvring the virtual hand or screwdriver to cause the detection algorithm to be triggered. 5. Part deselection errors happened when users forgot to release the pinch to drop the bolt. This error was infrequent, and not a design defect.

The naturalness audit and the observed usability errors both indicated that the environment for testing the virtual prototype interfered with realistic testing to a considerable extent. Diagnosis of the causes of the top five most frequent errors, using the decision trees, discovered the following:

21

Tool/interaction mode: problems encountered in switching between the screwdriver tool and use of the virtual hand. Mode errors are usually manifest when the user tries to carry out an impossible or inappropriate action, or the effect of an action has surprising results. The need to switch between the virtual hand and screwdriver mode by selecting the tool which then became an extension of the users’ presence caused considerable confusion. Users encountered difficulties in the select-pinch with little finger and then forgot how to deselect the screwdriver because they became confused between the two selections: index finger for the bolt, and little finger for the screwdriver. This problem can be traced along the middle branch of the action decision tree (see figure 6) when the user tries an impossible action (deselect tool with the index finger pinch). Probing why the difficulty persisted (simple action difficulty) in de-briefing interviews discovered the above cause of a pinch-finger mode confusion.

Part selection: these problems occurred when the bolts unexpectedly snapped to the screwdriver. The users did not expect this effect and all commented that it was unnatural, so this was analysed as a feedback error comprehensible but inappropriate to the user’s task (feedback decision tree, (see figure 7), poor information requirements, change not related to task). Although the users became familiar with the snap-to function and it helped them complete the task, it was an unnatural effect which detracted from realistic requirements testing. There were a small number of selection problems with the bracket caused by positioning the virtual hand to trigger the collision detection algorithm. These were interpreted as a combination of lack of haptic feedback in the action decision tree (figure 6), simple action difficulty, design affordances to help users, and poor visual feedback; coupled with perceptual 3D depth problems in positioning the virtual hand, (figure 5: poor location of controls and layout).

Parts occluded: these problems occurred when the aircraft fuselage obscured the bracket and bolts. These errors could be interpreted in three ways using the location decision tree in figure 5. First was poor VE layout, which was too sparse and failed to provide hints for navigation; however, this was unlikely since once users had learned the location, they didn’t repeat the error. The second explanation, lack of user domain knowledge, is unlikely for the same reason. A third explanation is poor location of controls and layout points in the virtual prototype, which made access to the parts difficult, or poor controls for changing the users’ perspective and viewpoint. De-briefing interviews indicated the latter cause.

22

Reaching parts/distortion: difficulty reaching bolts and the bracket was a perceptual error (see figure 5: poor layout/controls) when trying to approach and orient the self representation as a precursor for action. The graphical representation of 3D depth was not always accurate since distortion occurs in the corners of the CAVE. This, in combination with limited depth cues from other objects in the environment, made visio-motor coordination difficult.

Part deselection: this infrequent problem occurred when the subject forgot to release the bolt, confusing the pinching operations for deselecting the screwdriver tool (thumb and little finger) and deselecting the bolt from the screwdriver tool (thumb and index finger). Although this was a simple action difficulty (figure 6) which could be attributed to user training, the design could be criticised for an unnatural action that the user had to remember.

The analysis exposed more usability than requirements problems; however, some potential requirements problems with the virtual prototype emerged when the interaction problems were discussed with the users in de-briefing interviews. The decision tree diagrams provided a shared resource to discuss interaction problems, possible solutions, and how these related to requirements from the real world product: •

ability to access the brackets and bolts in restricted spaces may need to be considered

•

automatic undo of the bolts may cause difficulties, unless there was clear feedback when the tool was accurately located on the bolt

•

a control for removing the bolt once undone was required.

The analysis suggested that the VE design may have obscured several requirements problems in the design of the screwdriver and the bolt-removing task. The snap-to function in particular made testing the automatic undo feature invalid since the VE made it too easy and locating the screwdriver on to the bolt was not tested. The automatic undo feature was not explicitly represented and dropping the bolt by releasing the pinch did not map to a requirement for a release bolt control on the real-world tool. Similar problems were encountered on the assembly task in other tests. Furthermore, if non-functional requirements were considered, such as ease of operation and task completion times, the usability errors caused by the VE design would have invalidated any results.

The ISRE method produced insight into the requirement defects in the virtual prototype, while identifying usability problems which interfered with accurate requirements capture. The overall conclusion was that more extensive use of augmented reality was advisable, while use

23

of VE magic effects for helping user selection hindered requirements analysis of the realworld prototype.

6. Discussion The ISRE method has contributed the first procedure for requirements analysis with virtual prototypes. It has also demonstrated some of the important limitations of using VE technology for requirements engineering. While some industrial virtual prototyping applications avoid some of the confounding usability errors by augmented reality implementations [5, 41], use of VEs without haptic feedback for prototyping is commonplace. The dilemma of enabling effective user interaction while not compromising naturalness of the virtual prototype has no easy solution; however, our work can point to some guidelines such as avoiding magic effects, implementing gravity effects, etc. In our future work we will augment our existing design and configuration guidelines for virtual prototyping environments [4] to improve RE effectiveness.

Virtual prototyping fits within RE by extending the concept of design refinement and validation to discover requirements defects. Prototype-led design refinement has spawned a range of technologies ranging from paper prototypes [42] to concept demonstrators and full prototypes [22]. Virtual prototyping extends this genre for problem domains that have an important visual-spatial component, such as requirements for equipment controls, buildings, and collaborative work groups. The ISRE scenario-based approach extends our previous work on multimedia prototypes in the SCRAM method [2, 3, 10], in which users’ problems were observed and diagnosed. The SCRAM method also used probe questions and dialogues to focus on key points in scenario sequences. This concept could be extended to the ISRE method to test specific design features, possibly identified by the expert walkthrough analysis acting as an initial dry run. Another possibility is to use collaborative VE prototypes or obstacle analysis [11, 43]; for example, acting out user roles in a CVE (collaborative virtual environment) might help identify different classes of obstacles such as equipment operation, lack of information, inability to communicate, and information access in groupworking tasks that involve distributed cognition [44]. One merit of virtual prototyping is that it can be closely coupled to design by integration with CAD systems [45]. In RE this is akin to coupling a prototype to a formal requirements specification; observing problems in the former can therefore be diagnosed by reference to the latter. This points to further integration of

24

virtual prototypes with requirements validation via informal animation, as exemplified by the Albert system [12]. Indeed, if the virtual prototype is maintained as a surrogate for the implemented system, scenarios could be acted out in the VE to monitor performance and requirements [46].

Another contribution of ISRE has been to suggest a systematic approach to scenario-based requirements analysis. While a large number of scenario types have been proposed (see [47, 48]), little guidance has been proposed for employing scenarios systematically, apart from ScenIC [9]. Some directions can be gleaned from normal and abnormal use cases [29] and their extension in abuse cases [49]. We have also suggested layered scenario-based testing that covers different event permutations in a variety of environments [13]. However, the event permutations proved to be an overload in practice, so we regard the ISRE approach of key scenarios, error conditions with worst case environment testing to be an acceptable compromise between ad hoc and exhaustive testing. For systematic testing a large number of scenario variations and automated tools are necessary [22, 50], so we advocate a combination of interactive testing using virtual prototypes with automated tools to validate requirements more formally [51].

While the ISRE method was specifically designed for virtual prototypes, it could be adapted for problems diagnosis, and hence requirements validation of real prototypes. For instance it could be incorporated with the SCRAM method [2, 3, 10] to provide decision tree analysis techniques. This would require modest reworking of the decision trees to remove VR-specific problems such as the lack of haptic feedback and manoeuvring the user presence, while adding new diagnostic probes to question provision of information for task support and decision making. In the longer term the ISRE method forms part of a programme of method and tool development for scenario-based RE to provide requirements engineers with the means to trap and diagnose problems with prototypes and requirements specifications, either with walkthrough style methods [2, 10], or tools to reason about potential requirements problems [52, 53].

In conclusion, we have demonstrated a method that gives the necessary process guidance for VR-RE and have shown that the method is effective in diagnosing the source of errors. However, further tests with large-scale industrial applications are necessary to assess the effectiveness

of

the

approach;

and

the

economic

payback

of

requirements

discovered/validated compared to the cost of constructing virtual prototypes has yet to be

25

established. Lightweight prototypes, e.g. storyboards, and 2D mock-ups [54, 2, 3] might uncover many problems for less expense, although VEs should pay off when domains are spatial in nature; and operational effectiveness needs to be tested with many different scenarios. Clearly considerable further research is necessary to establish the effectiveness of virtual prototyping as a cost-effective requirements engineering technique.

Acknowledgements This research was partially funded by EPSRC grant GR M68749 Immersive Scenario-based Requirements Engineering. The authors wish to thanks Terrence Fernando and Luis Marcelino of the Centre for Virtual Environments, University of Salford for their help in testing the aircraft maintenance application.

References 1. R. J. Stone, Virtual Reality for Interactive Training: An Industrial Practitioners Viewpoint, International Journal of Human-Computer Studies, vol. 55, pp. 699-711, 2001. 2. A. G. Sutcliffe, Requirements Rationales: Integrating Approaches to Requirements Analysis, DIS 95 Conference Proceedings, 1995, pp. 33-42, New York: ACM Press. 3. A. G. Sutcliffe and M. Ryan, Experience with SCRAM: A SCenario Requirements Analysis Method, IEEE International Symposium on Requirements Engineering: RE '98, 1998, pp. 164-171, Los Alamitos, CA: IEEE Computer Society Press. 4. A. G. Sutcliffe, Multimedia and Virtual Reality: Designing Multisensory User Interfaces, Mahwah NJ: Lawrence Erlbaum Associates, 2003. 5. R. J. Stone, Applications of Virtual Environments: An Overview, in Handbook of Virtual Environments: Design, Implementation and Applications, K. Stanney, Ed. Mahwah NJ: Lawrence Erlbaum Associates, 2002, pp. 827-856. 6. A. G. Sutcliffe and K. D. Kaur, Evaluating the Usability of Virtual Reality User Interfaces, Behaviour and Information Technology, vol. 19, pp. 415-426, 2000. 7. J. L. Gabbard, D. Hix and J. E. Swan, User-Centered Design and Evaluation of Virtual Environments, IEEE Computer Graphics and Applications, vol. 19, pp. 51-59, 1999. 8. D. A. Bowman, D. Koller and L. F. Hodges, Travel in Immersive Virtual Environments:

26

An Evaluation of Viewpoint Motion Control Techniques, IEEE 1997 Virtual Reality Annual International Symposium, 1997, pp. 45-52, Los Alamitos: IEEE Computer Society Press. 9. C. Potts, ScenIC: A Strategy for Inquiry-Driven Requirements Determination, 4th IEEE International Symposium on Requirements Engineering, 1999, pp. 58-65, Los Alamitos CA: IEEE Computer Society Press. 10. A. G. Sutcliffe, Scenario-Based Requirements Analysis, Requirements Engineering, vol. 3, pp. 48-65, 1998. 11. A. Van Lamsweerde, Requirements Engineering in the Year 00: A Research Perspective, 22nd International Conference on Software Engineering, 2000, pp. 5-19, New York: ACM Press. 12. P. Dubois, E. Dubois and J. Zeippen, On the Use of a Formal Representation, ISRE '97: 3rd IEEE International Symposium on Requirements Engineering, 1997, pp. 128-137, Los Alamitos CA: IEEE Computer Society Press. 13. A. G. Sutcliffe, N. A. M. Maiden, S. Minocha and D. Manuel, Supporting Scenario-Based Requirements Engineering, IEEE Transactions on Software Engineering, vol. 24, pp. 1072-1088, 1998. 14. A. Monk, P. Wright, J. Haber and L. Davenport, Improving Your Human-Computer Interface: A Practical Technique, London: Prentice Hall, 1993. 15. B.E. John and R.E. Kieras, The GOMS Family of User Interface Analysis Techniques: Comparison and Contrast, ACM Transactions on Computer-Human Interaction, vol. 3, 320-351, 1995. 16. C. Wharton, J. Reiman, C. Lewis and P. Polson, The Cognitive Walkthrough Method: A Practitioners Guide, in Usability Inspection Methods, J. Nielsen and R.L. Mack, Eds. New York: Wiley, 1994, 105-140. 17. S. Ravden and G. Johnson, Evaluating Usability of Human-Computer Interfaces, New York: Ellis Horwood, 1989. 18. ISO, ISO 9241: Ergonomic Requirements for Office Systems with Visual Display Terminals (VDTs): International Standards Organisation, 1997. 19. J. Nielsen, Usability Engineering: Academic Press, 1993. 20. M. Ivory and M. Hearst, The State of the Art in Automated Usability Evaluation of User Interfaces, ACM Computing Surveys, vol. 33, 173-197, 2001. 21. B. Shneiderman, Designing the User Interface, Strategies for effective Human computer Interaction., Reading MA: Addison Wesley, 2002. 22. A.G. Sutcliffe, User-Centred Requirements Engineering, London: Springer-Verlag, 2002.

27

23. D. Hix and H.R. Hartson, Developing User Interfaces: Ensuring Usability Through Product and Process, New York: Wiley, 1993. 24. T. S. Andre , H. R. Hartson , S. M. Belz , F. A. McCreary. (2001). The user action framework: a reliable foundation for usability engineering support tools, International Journal of Human-Computer Studies, v.54 n.1, p.107-136. 25. D.A. Norman, The Psychology of Everyday Things, New York: Basic Books, 1988. 26. A.G. Sutcliffe and M.V. Springett, From Users' Problems to Design Errors: Linking Evaluation to Improving Design Practice, HCI '92, 1992, 117-134, Cambridge: Cambridge University Press. 27. A.G. Sutcliffe, A. Economou and P. Markis, Tracing Requirements Errors to Problems in the Requirements Engineering Process, Requirements Engineering, vol. 4, 134-151, 1999. 28. G. Cockton and D. Lavery, A Framework for Usability Problem Extraction, INTERACT 99, 1999, 347-355, : IOS Press. 29. J. L. Gabbard and D. Hix, A Taxonomy of Usability Characteristics in Virtual Environments: Deliverable to Office of Naval Research, Grant No. N00014-96-1-0385, Blacksburg VA: Dept of Computer Science, Virginia Polytechnic Institute, 1997. 30. ISO, ISO 13407: Human-Centred Design Processes for Interactive Systems: International Standards Organisation, 1995. 31. M. Slater, Measuring Presence: A Response to the Witmer and Singer Questionnaire, Presence: Teleoperators and Virtual Environments, vol. 8, pp. 560-572, 1999. 32. M. Slater, M. Usoh and A. Steed, The Depth of Presence in Virtual Environments, Presence, vol. 3, pp. 130-144, 1994. 33. B. G. Witmer and M. J. Singer, Measuring Presence in Virtual Environments: A Presence Questionnaire, Presence, vol. 7, pp. 225-240, 1999. 34. D. A. Norman, Cognitive Engineering, in User-Centred System Design: New Perspectives on Human-Computer Interaction, D. A. Norman and S. W. Draper, Eds. Hillsdale NJ: Lawrence Erlbaum Associates, 1986. 35. K. Kaur, Designing Virtual Environments for Usability: Doctoral Thesis, London: Centre for HCI Design, School of Informatics, City University, 1998. 36. L. A. Suchman, Plans and Situated Actions: The Problem of Human-Machine Communication: Cambridge University Press, 1987. 37. J.R. Galliers, A.G. Sutcliffe and S. Minocha, Models and a method for safety critical user interface design. ACM Transactions on Computer Human Interaction special issue on Interface issues and designs for safety critical systems, Vol 6(4), pp 341-369, 1999 38. A. G. Sutcliffe, M. Ryan, A. Doubleday and M. V. Springett, Model Mismatch Analysis:

28

Towards a Deeper Explanation of Users' Usability Problems, Behaviour and Information Technology, vol. 19, pp. 43-55, 2000. 39. J. Nielsen and R. Molich, Heuristic Evaluation of User Interfaces, SIGCHI Bulletin, pp. 249-156, 1990. 40. A. Cockburn, Writing Effective Use Cases, Boston MA: Addison-Wesley, 2001. 41. W. E. Mackay, Augmented Reality: Dangerous Liaisons or the Best of Both Worlds?, ACM DARE 2000, Designing Augmented Reality Environments, 2000, pp. 171-172, New York: ACM Press. 42. M. J. Muller, Layered Participatory Analysis: New Developments in the CARD Technique, Conference on Human Factors in Computing Systems, 2001, pp. 91-97, New York: ACM Press. 43. C. Potts, Invented Requirements and Imagined Customers: Requirements for Off-theShelf Software. Briefing for Working Group 2, Requirements Engineering '95, 1995, pp. 128-130, Los Alamitos CA: IEEE Computer Society Press. 44. E. Hutchins, Cognition in the Wild, Boston MA: MIT Press, 1995. 45. T. Fernando, N. Murray and K. Tan, Software Architecture for a Constraint Based Virtual Environment, VRST 99, 1999, pp. 147-154, New York: ACM Press. 46. S. Fickas and M. S. Feather, Requirements Monitoring in Dynamic Environments, 1995 IEEE International Symposium on Requirements Engineering (RE '95), 1995, pp. 140147, Los Alamitos CA: IEEE Computer Society Press. 47. J. M. Carroll (Ed.), Scenario-Based Design: Envisioning Work and Technology in System Development, New York: Wiley, 1995. 48. C. Rolland, N. Prakash and A. Benjamin, A Multi-Model View of Process Modelling, Requirements Engineering, vol. 4, pp. 169-181, 1999. 49. I. Alexander, Initial Industrial Experience of Misuse Cases in Trade-Off Analysis, IEEE Joint International Conference on Requirements Engineering, 2002, pp. 61-70, Los Alamitos CA: IEEE Computer Society Press. 50. A.G. Sutcliffe , J.E. Shin and A. Gregoriades A., Tool Support for Scenario-based Functional Allocation, In Proceedings 21st European Conference on Human Decision Making and Control, Glasgow, Scotland, pp81-88, Ed. Johnson C.W., GIST report 20021, University of Glasgow., 2002 51. C. L. Heitmeyer, J. Kirby and B. Labaw, Tools for Formal Specification, Verification, and Validation of Requirements, 12th Annual Conference on Computer Assurance (COMPASS '97), 1997, 52. A.G. Sutcliffe, J. Galliers and S. Minocha, Human Errors and System Requirements, 4th

29

IEEE International Symposium on Requirements Engineering, 1999, 23-30, Los Alamitos CA: IEEE Computer Society Press. 53. A.G. Sutcliffe and A. Gregoriades, Validating Functional System Requirements with Scenarios, IEEE Joint International Conference on Requirements Engineering, 2002, 181-188, Los Alamitos CA: IEEE Computer Society Press. 54. M. J. Muller, J. H. Hanswanter and T. Dayton, Participatory Practice in the Software Lifecycle, in Handbook of Human Computer Interaction, M. G. Helander, T. K. Landauer and P. V. Prabhu, Eds. Amsterdam: Elsevier, 1997.

Appendix: Presence Questionnaire Subjects were asked to rate each VE against each question on a scale of 1-7, where 1 = negative rating and 7 = positive rating.

1. How well were you able to control the system? 2. How responsive was the environment to actions that you initiated (or performed)? 3. How natural did your interactions with the environment seem? 4. How natural was the mechanism which controlled movement through the environment? 5. How aware were you of events occurring in the real world around you? 6. How aware were you of your display and control devices? 7. How compelling was your sense of objects moving through space? 8. How far did your experiences in the virtual environment seem consistent with your real world experiences? 9. Were you able to anticipate what would happen next in response to the actions that you performed? 10. How completely were you able to actively survey or search the environment using vision? 11. How compelling was your sense of moving around inside the virtual environment? 12. How closely were you able to examine objects? 13. How well could you examine objects from multiple viewpoints? 14. How well could you move or manipulate objects in the virtual environment?

30

15. How involved were you in the virtual environment experience? 16. How much delay did you experience between your actions and expected outcomes? 17. How quickly did you adjust to the virtual environment experience? 18. How proficient in moving and interacting with the virtual environment did you feel at the end of the experience? 19. Were you involved in the experimental task to the extent that you lost track of time? 20. How effective was the sense of perspective (depth of field)?

31

Tables

Table 1. User problems analysed from video and audio recording.

Problems Part selection

Tool/interaction mode

Part occluded

Reaching parts distortion

Part deselection

N users with error (max. 18)

14

16

16

8

2

Total frequency

22

20

17

9

3

% total

30

28

24

13

4

32

Figures Figure 1.

Process model and resources in the ISRE method. guidelines

Walkthrough model + checklist

Technolog y audit

Configure VE for analysis Select & brief users

scenario types Chang escenari o

Expert walkthrough Interaction analysis

Virtual prototype Re-desig n prototype

problems Analyse problems

procedures

explanations causes

question checklists decision trees

Specify requirements Not covered in ISRE method

requirements

33

Figure 2.

VR interaction model for the task-action cycle (after [34]), showing the appropriate design features associated with each stage. Tool, object, control exist to match user’s need

Task Goals

objects not visible 2 Locate active environment

1 Form goal

appropriate environment located

NAV

task goal Change and feedback messages comprehensible

To Explore Cycle

intention formed

end of procedure 8 Assess feedbac k

3 Locate objects Tool, object visible Navigation cues satisfactory action

NAV objects for action

SEL perceived change

9 Specify next action VSL

7 Recognise feedback

Tool, object, control exist to match user’s need

action selected 4 Approach & orient action selected

objects and environmental changes

Change in state visible or audible

Tool, object features clearly visible

PRES

self positioned for action 5 Specify action

effects 6 Manipulate object SEL

action details

SEL

Affordances and intuitive controls, tools

Control surfaces and operations easy to use

User interaction support (functional requirements): navigation facilities, fly through, magic effects, maps user presence controls easy to use, sufficient for task SEL techniques for selecting objects, snap-to VSL visual feedback for object selection/deselection

NAV

PRES

34

prototype usability requirements

Figure 3.

Scenario framework for testing virtual prototypes.

weather visibility temperature, etc.

permutations

different roles training level fatigue, stress

Operator roles

operational personnel

Operational tasks

operational procedures emergency procedures

Basic ergonomics

performance criteria?

Environmental

mistakes? errors ?

necessary and sufficient scenarios?

Virtual prototype

spatial requirements functional requirements interaction requirements

35

Figure 4.

Diagnostic decision tree for task action mode: goal/function problems. Was the scenario clear?

Do users know the task?

Y

no goal, task knowledge inadequate N User can’t decide how to proceed

N

Y

N task knowledge present

Follow up questions

Probable cause

user unsure, lacks task knowledge

probe domain and task knowledge for script step; training requirements

mismatch between task and scenario, poor task knowledge

explore reasons for inadequate scenario and extent of task knowledge

mismatch between task and scenario

explore reasons for inadequate scenario

Requirements or UI design Y problem

Y

control/object not appropriate for task, or missing

explore reasons why control missing or was not useful; missing requirement

N

control/object difficult to locate/absent, missing requirement

see Location problems tree

Can the appropriate object/control be found?

36

Figure 5.

Diagnostic decision tree for location problems. Does the VE provide cues/ structure to locate control?

Does the user know where to look?

Y poor domain knowledge

N

Probable cause

Follow up questions

insufficient domain knowledge to interpret results

explore additional training for domain knowledge

poor design/ location of controls

elicit user’s mental model of domain; compare with design

N

User puzzled searching/ reaching for control/object N

poor location or layout of objects/controls

Y

N

VE too sparse poor structure

Y

Perceptual /motor problems

Difficulty reaching objects ?

adequate domain knowledge Y

design and domain knowledge adequate

N

Y Has the user explored the VE?

37

explore reasons why design doesn’t match user’s model ask if perceptual problem or movement of user presence

user lacks confidence/ motivation, unsure of VE

ask if more VE familiarisation will help, probe why hints not followed

control/object not locatable, missing cue, missing requirement

elicit user’s expectation of control/object location and appearance; compare with design

Figure 6.

Diagnostic decision tree for task action mode: action/manipulation problems and interaction requirements. Will adding other feedback help?

Y Is lack of haptic feedback a problem? VE not N requirements problem User trying Impossible Y action ? User has difficulty manipulating object/ operating control

Y

redesign VE with visual/ audio feedback

provide scenarios for alternative feedback; elicit suggestions

consider adding haptic device or augmented reality

discuss scenarios for augmented reality; explain haptic devices

mode error

redesign with natural interaction

Is action too precise? Y

N

N design/ requirements problem Y

simple action difficulty

complex action difficulty

Is the action too complex?

N N

Y Will training help?

38

relax constraints; provide better support for precise movement

How to design affordances to help users

user motivation training problem

probe why user difficulty persists

simplify action/ change requirement

explore how action can be decomposed or partially automated

provide practice for action

explore better affordances & controls for motor skill, coordination, etc. interaction requirements

Figure 7.

Diagnostic decision tree for task action mode: feedback problems. Does feedback make sense to user’s task?

Can user interpret feedback?

Y

Can user see/hear change?

Y feedback present

Y N

User puzzled after operating control/ manipulation

N N feedback inadequate

Y Is feedback present but remote?

feedback comprehensible but inappropriate

N

feedback not understood

training/motivation problem; poor task knowledge

explore user’s mental model of task and motivation

mode errors, poor informat -ion requs, inaccurate info

investigate task/domain model of VE for metaphor/feedback error

Y

poor information requs change not related to task

N

mode error training problem, task/domain knowledge

Does user have missing domain knowledge feedback; to critique? UI requirement add feedback

give user hint where feedback can be found/ change design

investigate user’s mental model of VE/domain

39

elicit user suggestions for improvement Identify mode problem & redesign

Figure 8.

CAVE virtual environment showing the maintenance task application (a) with the screwdriver tool removing a bolt from the bracket, and (b) showing the virtual hand removing the bracket. The outline box is the proximity detection algorithm indicating the object is selectable.

(a)

(b)

40