Continuous Interaction with Computers: Issues and Requirements. G.Faconti, M.Massink. Istituto CNUCE, National Research Council, Via V.Alfieri 1, 56010 ...
Continuous Interaction with Computers: Issues and Requirements G.Faconti, M.Massink Istituto CNUCE, National Research Council, Via V.Alfieri 1, 56010 Ghezzano (PI), Italy This paper introduces continuous interaction as a requirement of interactive systems to support native human behavior. Subsequently it addresses a framework of modeling notations and tools to drive design decisions during the development of interactive systems. 1. INTRODUCTION Interactive systems in the modern world are becoming both increasingly pervasive, and rich in the variety of tasks supported. In fact, one of the next major developments in computing will be the widespread embedding of computation, information storage, sensing and communication capabilities within everyday objects and appliances [1]. Interacting with such systems will involve multiple media and the coupling of distributed physical and digital artifacts, supporting a continuous flow of information. The concept of Continuous Interaction [2,3] was brought about by advances in technologies that will have a profound effect on the way we interact with computers and information, and ultimately the way we work and live. Continuous interaction differs from discrete interaction in the sense that the it takes place over a relatively longer period of time in which there is an ongoing relevant exchange of information between the user and the system at a relatively high rate, such as in vision based or haptic interfaces, that can not be modeled appropriately as a series of discrete events. This shift towards more continuous interaction between user and system is a direct consequence of two aspects of a new generation of interactive systems: ubiquity and invisibility. Since interaction devices will be spread in the surrounding environment, the distinction between real world objects and digital artifacts will become immaterial. Users will behave naturally as human beings without adopting a simplified behavior that characterizes a state of technological awareness. Consequently, there’s a need for the systems to adapt to users, to be aware of their operating context, and to be able to take autonomous decisions to some extent. 2. EVOLUTION OF APPLICATIONS Today miniaturization of hardware and increase in computing power makes it possible to put on one’s pocket a relevant computing capability. Most every-day consumer goods incorporate microprocessors and support interaction: computer hardware becomes invisible and ubiquitous. In this context, sensing and effecting devices are developed that show an autonomous behavior and are able to cooperate to achieve a given goal (e.g. tele-passing facility in the highways). On the other end, virtual and augmented reality environments are being developed where common user tasks are supported by electronic devices (e.g. networks of domestic appliances on a home LAN). Although ubiquitous technologies were initially conceived as part of the infrastructure of work, these are now being extended to merge digital and physical worlds and the distinction between them becomes blurred. This process of
merging is occurring at even the finest level, with the advent of smart matter. It incorporates small-scale sensors and effectors into physical materials, in such a way that the general behavior of the material may be actively altered and controlled by distributed techniques [21]. Those new environments are a great challenge for the future of Human Computer Interaction studies. More senses (vision, earring, touch) and more means of expression (gestures, facial expression, eye movement, speech) are involved in interaction from the human side with respect to traditional WIMP based applications. 2. DESIGN COMPLEXITY As has become clear in the introduction, continuous interaction is closely related to the development of smart and virtual environments where the computing system is preferably invisible. The trend is that the system is adapting to the complexity of user’s behavior rather than the user adapting himself to a simple system’s interface. The consequence is that the system needs to be able to deal with a larger part of the interaction. It must deal with the multiple and simultaneous way in which users and groups of users may interact with and via the system. This requires the coordination of many parallel, loosely coupled activities and processes, sensed by even so many different devices under tight real-time constraints in order to guarantee responsiveness of the system and a sense of reality to the user. The responsiveness needs to satisfy the constraints that make human perception and cognitive processes possible. All these aspects together lead to many design decisions that have to be taken into account during the development process. In order to address the behavioral issues as early in the design process, formal modeling techniques for distributed real-time and stochastic systems supported by powerful analysis tools could be considered. However, to address the numerous issues systematically, also a more general framework that can guide the modeling approach is needed. 3. A REFERENCE FRAMEWORK FOR CONTINUOUS INTERACTION Structuring system design in hierarchical layers is a common approach to reduce design complexity in the software engineering practice. Reduction of the complexity can be obtained by defining for each layer what services are provided to the next layer. This way a designer can concentrate on the problems that are relevant to a particular layer while assuming that the lower layers are providing more basic, lower level means of interaction. He also can abstract from the higher level layer problems because he knows what are the means of interaction that the higher level requires. In the reference model presented in [3] and illustrated in fig. 1, we tentatively propose five different levels. A physical level where physical interaction takes place, followed by three levels of information processing: the perceptual, propositional and conceptual or task level. These are closely following the layers in well-known frameworks such as the Skill-Rule-Knowledge-model [4] and are not too far from the levels in Norman's Interaction model [5]. Finally we introduce a group level that explicitly deals with interaction problems related to coordination of tasks in the context of cooperative work. 5. Modelling notations and tools
The Reference Model on its own does not provide any specific technique or notation for the description of the behavioral aspects of interaction at the different layers of abstraction. It just provides a framework to guide the way in which a complex problem such as the design of continuous interfaces could be split into sub-problems at different levels of abstraction. It is well-known that in the software development process the requirements analysis and design phase are very important for the successful development of software. This holds even more so for the development of user interfaces. Formal modeling is increasingly recognized as a promising way in which behavioral and performance aspects of different designs can be assessed at an early stage of software development [6]. Given the particularities of continuous human computer interfaces, such as critical real-time dependency, distribution and parallelism, we discuss a number of formal modeling approaches that could provide appropriate concepts to deal with this form of interaction and the control issues this poses for the system.
Figure 1: Reference Model for Continuous Interaction Manual Control Theory An approach that is used to analyze human and system behavior when operating in a tightly coupled loop is manual control [7]. The theory has been refined to a very high degree and there is now a very large base of predictive [8] and explanatory [9] theory validated by a wealth of experimental evidence. The general approach followed in manual control theory is to express the dynamics of combined human and controlled element behavior as a set of linear differential equations in the time domain. In order to obtain a solution this set of equations is transformed into a set of linear algebraic equations in the complex frequency domain by wellknown mathematical techniques. In several models human related aspects of information processing are explicitly included in the model such as delays for the visual process, motor nerve latency and neuro-motor dynamics. Control theory can be linked to Fitts' Law [10] by viewing the pointing movement towards the target as a feedback control loop based on visual input and the limb as the controlled element. Hybrid Modelling Hybrid models allow to express both discrete and continuous aspects within the same formalism. Hybrid Automata [11] are seen as an extension of the purely control theoretic approach in manual control theory. It combines the description of continuous behavior as sets of algebraic equations with that of discrete behavior. In the hybrid automata approach behavior is described as sets of automata where each automaton consists of nodes and
transitions between the nodes. The nodes are labeled by sets of equations that describe the continuous behavior when control is at that node. The transitions are labeled by guards on variables that define under which conditions one continuous behavior is replaced by another. The formalism is supported by software tools that provide reachability analysis of the state space and check properties of the system model formulated as temporal logic formulas. Several approaches to the application of hybrid modeling of continuous interaction have been examined. In [12] a case study is presented where the interaction between a pilot and an hydraulic subsystem of an aircraft is analyzed. In [13] a Hybrid Petri-Net model is discussed presenting two-handed navigation through a 3D architectural space in a VR-application. A hybrid automata approach to modeling a gesture language for 3D CAD applications is discussed in [14]. Stochastic approaches In continuous interaction, the human variability in performance of activities is an issue. Stochastic modeling can provide qualitative information about the shape of the probability distribution of real-time related issues concerning the combined human and system performance. Estimates can be made of the effect of the combination of mostly log-normal distributions generated when the reaction time of humans is involved [15] and the system performance that is often characterized by variants of exponential distributions [16]. Stochastic modeling, simulation and analysis using stochastic automata is still a relatively new field. As such the expressivity of the specification languages based on the technology, the theories concerning analysis of specifications, and the incorporation of these into automated support are still at an early stage of development. We believe such techniques have an exciting potential for modeling performance in interactive systems, taking into account the abilities and limitations of both user and system. Such models allow us to generate a richer set of answers to design questions, which enables meaningful comparison of the results of our analysis to human factors data. Two preliminary case studies of the use of stochastic automata for the modeling of polling behavior and that of a pointing task can be found in [17,18]. The Unified Modelling Language The Unified Modelling Language (UML) is a de facto industrial standard. The UML is semi-formal in the sense that its syntax and static semantics are defined formally, but its dynamic semantics are described only informally as are the relations that may exist between different diagrammatic notations. Several formal semantics have been proposed for UML Statechart Diagrams, that are meant for the modeling of behavioral aspects of systems. Currently, the UML provides only limited support for the modeling of continuous, real-time and stochastic behavior of systems. Nevertheless, it provides a rich set of notations that are widely adopted in industry and as such, they form an interesting starting point for further extensions of the notations for the modeling of complex interactive systems and intelligent environments. Preliminary work in this direction can be found in [17] where a stochastic model of a user checking the progress of a system is described using a stochastic extension of a subset of UML statechart diagrams [19] In [20] UML interaction diagrams are used for a high level specification of interaction in a VR application. It is shown how the interaction diagrams may be used to reflect the different levels of abstraction in the reference model in fig. 1.
6. CONCLUSIONS Although continuous interaction is not a completely new phenomenon in user interfaces, it has become a much more prominent and critical issue in many future computing systems such as virtual reality applications and intelligent environments. To guarantee responsiveness of a system and to provide the user with a pleasant sense of (virtual) reality the designer is confronted with many new critical design decisions. The information needed to make these decisions require insight in the human cognitive aspects of processing continuous and often simultaneous information combined with the distributed and parallel operation of the system in which many non-homogeneous input and output devices are involved. This, in turn, requires modeling techniques in which these aspects can be expressed and analyzed before a complete system is implemented. It is well known that complex, distributed real-time systems like intelligent environments, are notoriously hard to debug after their implementation, and therefore need a design that is setup carefully and systematically supported by appropriate analysis tools. REFERENCES 1. D.A.Norman, The Invisible Computer, MIT Press, 1998. 2. The TACIT Research Network (http://kazan.cnuce.cnr.it/TACIT) 3. G.Faconti and M.Massink, Continuity in Human Computer Interaction, Workshop Report, CHI 2000, The Hague (http://www.acm.org/sigchi/bulletin/2000.4) 4. J.Rasmussen, What can be learned from human error reports?, In K.Duncan et al., eds, Changes in Working Life. John Wiley & Sons, 1980. 5. D.A.Norman, The Design of Everyday Things, Basic Books, New York, 1988. 6. I.Sommerville, Software Engineering, Fourth Edition, Addison-Wesley, 1992. 7. G.Salvendy (Ed.), Handbook of Human Factors and Ergonomics, Wiley, 1997. 8. D.McRuer, Human dynamics in man-machine systems, Automatica, 16(3):237-253, 1980. 9. R.A.Hess, A model-based theory for analyzing human control behaviour, Advances in Man-Machine Systems Research, 2:129-175, 1985. 10. P.M.Fitts, The Information Capacity of the Human Motor system in Controlling the Amplitude of Movement, Journal of Experimental Psychology, 47, pp. 381-391, 1954. 11. T.Henzinger, The Theory of Hybrid Automata, In: Proceedings of 11th Annual IEEE Symposium on Logic in Computer Science, pp. 278—292, 1996 12. G.Doherty, M.Massink, and G.Faconti, Using hybrid automata to support human factors analysis in a critical system, Journal of Formal Methods in System Design. Kluwer Academic Publishers}, 2000. To appear. 13. M.Massink, D.Duke, and S.Smith, Towards hybrid interface specification for virtual environments, In D.Duke and A.Puerta, editors, Design, Specification and Verification of Interactive Systems '99, pages 30--51. Springer Computer Science, 1999. 14. G.Doherty, G.Faconti, and M.Massink, Formal verification in the design of gestural interaction, To appear in Electronic Notes on Theoretical Computer Science, Elsevier. 15. A.D.Swain and H.E.Guttmann, Handbook of human reliability analysis with emphasis on nuclear power plant applications - Technical Report NRC FIN A 1188 NUREG/CR-1278 SAND80-0200, US Nuclear Regulatory Commission; Washington, D.C., 1983.
16. R.Jain, The art of computer systems performance analysis : techniques for experimental design, measurement, simulation, and modeling, Wiley, 1991. 17. G.Doherty, M.Massink, and G.Faconti, Stochastic modeling of interactive systems, In Int. Workshop on Towards A UML Profile For Interactive Systems, York, 2000 (http://math.uma.pt/tupis00/programme.html) 18. G.Doherty, M.Massink, and G.Faconti, Reasoning about interactive systems with stochastic models, TACIT Report, Under submission. 19. S.Gnesi, D.Latella, and M.Massink, A stochastic extension of a behavioural subset of UML statechart diagrams, In Fifth IEEE International High-Assurance Systems Engineering Symposium, pages 55--64. IEEE Computer Society Press, 2000. 20. L.Sastry, G.J.Doherty, M.D.Wilson, D.R.S.Boyd, Continuity of Interaction in VR Facilitated Groupwork, Submitted. 21. L.A.Watts, J.Coutaz, D.Thevenin, E.Dubois, M.Massink, G.Doherty, Environmental interactive systems: Principles of systematic digital-physical fusion, TACIT Report – TR0013, 2000.
