Toward a Predictive Framework of User Experience ...

Toward a Predictive Framework of User Experience: The Interplay of Choice & Emotions during Interaction Jan Kallenbach Helsinki University of Technology Abstract Research into user experience is rich in descriptive theoretical approaches. From Forlizzi and Ford’s building blocks of experience (2000) over Hassenzahl’s product characters (2003) to McCarthy & Wright’s framework of experience (2004) all introduce a variety of variables that influence users’ experiences with an interactive artifact. Common to all accounts is that at least two aspects are considered to be essential: interaction and emotions. In fact many authors content that userproduct interaction is a requirement for emotional responses to emerge and experiences to unfold (e.g. Beauregard & Corriveau, 2007; Hassenzahl, 2003). But what actually makes up a “good” interaction and what a “bad”? In other words, what factors cause people to evaluate interaction with a product to be pleasurable or annoying? This paper makes an attempt to provide answers to these questions by developing a predictive framework of users’ interaction experiences. On the basis of various empirical results from human-computer interaction, psychology, and cognitive science research my contribution is three-fold. First, I show that people’s judgments and decisions represent key variables of their interaction experiences. In general, people’s choices during interaction serve as a beneficial unit of analysis for in-depth investigations of their experiences. Second, I provide an initial predictive model that illustrates how emotions and judgment and decision making mutually influence each other during interaction and thus govern the dynamics of user experiences. Third, I introduce a framework of subjective interaction quality that integrates people’ choices and emotions during interaction. It compares their expectations on and perceptions of progress and effort and allows predictions of people’s overall experiences with a product. Finally, the paper concludes with a brief analysis of how future research could generate empirical results in order to test the validity of the framework.

Keywords User Experience, Interaction, Emotions, Judgment and Decision Making, Progress, Effort

Introduction Interaction has been the central element of investigation for as long as Human–Computer Interaction and related research fields exist. Numerous theoretical models have been developed mostly describing its generic stages or phases occurring in a variety of, if not all, situations in which a human interacts with some kind of interactive artifact for some purpose. The model of 7-stages of interaction from Don Norman (1988) is perhaps the most well-known. A human actor with a goal in mind (1) forms an intention to act on the world or environment surrounding her/him (2). This intention must then be translated into some sequence of concrete actions (3). In stage (4) the actor executes the action sequence against the world. Her or his perception of the state of the world marks stage (5). The actor then interprets the perceptions according the previously formed intention (6) and finally evaluates the result against the initially formed goal (7). Stage 2 to 4 are commonly labeled “gulf of execution” whereas stages 5 to 7 are commonly labeled “gulf of evaluation”. With regard to user experience interaction is commonly considered to be its requirement as the absence of interaction with an interactive artifact will reveal only little or even nothing about its nature, purpose, or functionality to the potential user (e.g. Beauregard & Corriveau, 2007; Hassenzahl, 2003). Interaction, however, acts also as a driver of user experiences as at least some continuous interaction with a product is necessary for user experiences to unfold. Given the centrality of emotions in user experiences an important question arises: how is interaction related to emotions and vice versa? Clearly, Norman’s model of interaction does not involve the role of emotions; in fact, it rather represents a rational perspective on interaction involving generic stages that occur every time a person uses an interactive artifact. Thus it does not allow explaining either user experience or its dynamics over the time of interaction. In the following sections I will outline a way how interaction and emotions may be integrated on the basis of people’s judgments and decisions during interaction that ultimately cause and shape user experiences.

Judgments, Decisions, and Experience Emotions in Judgment and Decision Making The role of emotions in human judgment and decision making has increased significantly since the discovery that emotions act as guides in situations where the bounded rationality of a decision maker fails to deliver fully comprehensive judgments about possible options and their outcome. Based on experiments involving repeated choices of similar nature Damasio (1994) found that emotions and feelings aid rationality in filtering out and selecting beneficial choice options (rewards)from disadvantageous ones (punishments). Based on rewards and punishments the brain learns, over time, to associate and generate a “gut feeling” with the outcomes of presented options before a conscious cost/benefit analysis (judgment) occurs. In other words, choice options become “marked” with a body or somatic response that acts as an automatic alarm signal for bad options. This discovery led Damasio to pose the somatic marker hypothesis. It states that somatic markers may not be sufficient for normal human decision making but they “probably increase the accuracy and efficiency of the decision process” (p. 173). In the light of this development Slovic, Finucane, Peters, and MacGregor (2002) proposed the Affect Heuristic, a rule of thumb in which people rely on their emotions and feelings in order to make judgments and decisions. A number of studies illustrate people’s use of the affect heuristic. For instance, Hsee (1996a,b) asked people to evaluate two used music dictionaries which were differed in their number of entries and their defects. Dictionary A had 10,000 entries and had no defects thus was like new whereas dictionary B had 20,000 entries but had minor defects on its cover. When a group of participants was asked to state how much they would be willing to pay for each of them dictionary B received the higher amount of money than dictionary A presumably because of the higher number of entries. On the contrary, if participants in two other groups were asked to evaluate each of the dictionaries alone without the comparison, dictionary A was given the higher amount of money. To explain this preference reversal Hsee argues that without a direct comparison, the number of entries is hard to evaluate, because the evaluator does not have a precise notion of how good or how bad 10,000 (or 20,000) entries is. However, the defects attribute of the dictionary is evaluable more easily in this situation because of the rapid and automatic affective responses it causes in the evaluator. Thus in a single evaluation condition evaluators rely on their feelings on the basis of attributes that are easily evaluable in terms of affect. Mellers et al. (1999, 2001) studied emotion-based choice in several experiments in which they distinguished anticipated from experienced emotions influencing judgment and decision making. They found that people’s rated anticipated pleasure towards different uncertain choices can be divided into three effects. Outcome effects explain that people generally anticipate feeling more pleasure the higher an anticipated outcome will be (in amounts of money, for example). Comparison effects address peoples anticipated pleasure by means of disappointment and regret. Comparisons of the imagined outcome with other outcomes of the chosen alternative are called disappointment or elation. Comparisons of the imagined outcome with an outcome of the un-chosen alternative are called regret or rejoicing. Finally, surprise effects take into account that people’s anticipated pleasure depends on the likelihood that an uncertain outcome would occur. If positive outcomes, such as wins, are very uncertain but occurred then the anticipated pleasure is high. Likewise, if negative outcomes, such as losses, are very uncertain to happen but occurred then the anticipated pleasure is very low.

Choice and Experience Given the strong relationship between judgment, decisions, and emotions, the influence of choice on experience has received increasing attention in the last years. Hsee (2006), for example, applies results from psychology and behavioral decision theory to develop accounts that describe why people often fail to predict future experiences accurately and thus make decisions that do not maximize their happiness. Several systematic biases account for these failures. The impact bias causes people to overestimate the impact of affective events, for example, the joy or disappointment of getting the feedback of a job application. The projection bias addresses the situations in which people that predict (or anticipate) are often in a different visceral state than people who experience. This is the case for example, when one anticipates in the morning that playing a computer game in the evening after work will be enjoyable and relaxing. However, in the evening playing the video game is not experienced as relaxing anymore as one realizes that it requires a lot of attention and concentration which may not be available anymore after a long working day. In the distinction bias people often make decisions based on the joint evaluation of choice alternatives, i.e. when choosing which mobile phone to buy, but can evaluate the chosen option later only alone and without any comparison. The memory bias stresses the fact that people often base their judgments and decisions on past experiences. These, however, may strongly bias the decisions as past experiences are often evaluated according to the peak-end rule (Fredrickson, 2000). That is, the (affective) peak of an

experience and its end serve as evaluation anchors and thus may cover up the remaining emotions during the experience. Finally, the belief bias emphasizes the importance that the pure existence of choice is not always beneficial to obtain happiness and pleasure. For instance, a decision maker that has won a competition and may chose between a strategy and a car racing computer game as a price may experience the dilemma of choice as s/he might like both. In case the choice would not be present the decision maker would probably be happier as no alternative would exist to be evaluated.

Emotions during Interaction In the light of the importance of judgments and decisions and choice in experience a question arises now: if choice and emotions are connected to form experiences and interaction represents a requirement and a driver of experiences, how then is choice and interaction related? So far interaction has often been considered a sequence of actions that all have certain properties, e.g. Norman’s seven stages of interaction (Norman, 1988), or Monk’s (1998) cyclic interaction theory. The problem with these models is that they consider interaction at an obvious and observable level of actions and behavior. Given their inherent nature they certainly offer recurring and generic processes that, undoubtedly, take place every time a person interacts with the world. However, they cannot account for the dynamic properties of interaction, e.g. its speed, its progress, or why a person first executes action B and then action A in order to achieve a goal. Consequently, they cannot describe the dynamics of user experience. What is needed is therefore a model that takes into account people’s choices during interaction and their resulting emotions. In order to integrate judgment and decision making and interaction I propose a change in perspective. If emotional responses are caused by anticipations and their comparison with actual experiences in choice situations then it may be beneficial for experience research to view interaction not only as a sequence of actions but foremost as a sequence of judgments and decisions. In other words, choices, represented by judgments and decisions, serve now as the primary unit of analysis in the research of interaction experiences. From this perspective, actions, as described in Norman’s model of interaction, would then be the consequential behavioral response occurring as real-world results of judgments and decisions. However, what judgments and decisions do exist in interaction? Superficially, one could argue that choice is omnipresent during interaction: we chose what kind of software product we want to use for what purpose, in what order we want to carry out tasks and activities, and so on. This is certainly true but in my opinion again an obvious observation. What is more important is to look for choice classes that re-occur during interaction; in other words, choices that are building blocks of interaction and user experiences. Based on Norman’s model of interaction I propose that (at least) the following three classes of choices occur frequently during interaction: First, before an actor engages in any physical action s/he usually chooses a goal s/he wants to achieve as part of the overall activity s/he is engaged in. This choice is the most important one as it may determine fundamentally the course of actions within a given activity. Second, an actor then selects the appropriate user interface element whose function is expected to achieve the previously chosen goal sufficiently. This “right” element is chosen based on a judgment that integrates the semantics of the label or icon of the element with the actor’s goal and her or his prototypical cognitive representation of a function that would help achieving it on the basis of its anticipated outcome (see e.g. Hommel & Elsner, 2009; Storbeck, Robinson, & McCourt, 2006). The third choice is finally the appropriate trigger that allows activating the element and executing its function, e.g. a physical interaction device such as a mouse or a keyboard, or a finger in case of touch-based interaction. The combination of a single instance of all three choices constitutes a single action. Given that interaction involves usually more than one action an actor therefore makes repeatedly these choices throughout the life-time of interaction. Consequently I argue that in combination with the relationships between choices and emotion they account for and determine most interaction experiences.

The Role of Effort Interaction does not take place without a cost. In other words, whenever an actor interacts with the world s/he can do so only by exerting some form of cost or effort. Effort maybe purely mental most of the time but may also be physical, i.e. when moving the mouse. The influence of effort on experiences should not be underestimated. Numerous recent studies have prominently shown that even small additional costs change user behavior profoundly. Bhavnani and John (2000), for example, have shown that even skilled users of software do not use it efficiently. In fact, people maintain least-effort strategies, even though these incorporate a higher number of actions and take longer to

execute than (theoretically) optimal strategies. Payne, Howes, and Reader (2001) argue that people actively and adaptively distribute cognition. That is they often interleave planning and action and “off-load” certain cognitive efforts onto the interface. For example, Payne (1991) showed that people often cannot remember the labels of frequently used menu commands, even in software that was familiar to them. Instead they rely on their recognition memory to judge if a label semantically fits to the meaning of the action they have in mind. If it fits approximately then they (decide to) click it. Gray, Sims, Fu, and Schoelles (2006) have shown that people’s micro-behavior during interaction is constrained by their cost/benefit tradeoffs due to the availability and nature of interactive elements of the environment.

Interaction Experience and Quality An Initial Predictive Model Interaction Experience My initial predictive model of interaction experience is based on two aforementioned accounts. First, Mellers et al’s (1999, 2001) decision-affect theory with its distinction of anticipated and actual pleasure serves as the basis. It allows predicting emotional responses from users that make repeated choices of goals, user interface elements, and appropriate triggers during interaction in order to carry out a task. Second, to reflect the important role of effort during interaction I expand decision-affect theory by integrating also anticipated and experienced effort given the aforementioned results from Bhavnani and John (2000) or Gray et al. (2006), for example. However, in the context of judgment and decision making and interaction the term pleasure deserves a more specific definition. The question is what makes an interaction pleasurable? Taking the assumption that humans usually behave in a goal-directed manner when carrying out tasks I propose to define pleasure more concretely as the degree or magnitude of progress towards the final outcome or end goal of a given task that is experienced by making a (single) choice. This elaboration results in the initial predictive model of interaction experience illustrated Table 1. For the progress component the mechanism of emotional response generation works as follows: A comparison of the anticipated magnitude of progress with the actual one results in the emotional response of elation if the actual magnitude is equal or greater than the anticipated; otherwise the emotional response will be disappointment (according to Mellers et al, 1999; 2001). For the effort component the mechanism is similar: A comparison of the anticipated magnitude of effort with the actual one results in the emotional response of stress if the actual magnitude is equal or greater than the anticipated; otherwise the emotional response will be relief. Table 1: Initial predictive model of interaction experience

Anticipated Magnitude

Actual (Experienced) Magnitude

Emotional Response based on Comparison

Progress (Benefit, Pleasure) (see Decision-Affect Theory, Mellers et al. (1999, 2001))

Elation (positive)

Effort (Cost, Pain) (see e.g. Bhavnani and John (2000), or Gray et al. (2006))

Stress (negative)

Disappointment (negative)

Relief (positive)

A Framework of Subjective Interaction Quality Subjective interaction quality builds upon the initial predictive model of interaction experience and represents a user’s overall evaluation of her or his interaction (and/or user) experience for activity.

In a first step of integration the framework considers both components of progress and effort as necessary. In fact, given that people constantly perform cost/benefit tradeoffs during interaction, progress and effort form a ratio: progress represents the numerator whereas effort represents the denominator. In general, a ratio value of around 1 indicates that progress and effort are about equal. Values significantly smaller than 1 indicate that effort is greater that progress, and values significantly greater than 1 indicate that progress much higher than the raised effort. Furthermore, for a single choice during interaction a user then evaluates in fact two cost benefit ratios (see Figure 1). The first one is the ratio between the anticipated progress and effort whereas the second one is the ratio between the progress and effort actually experienced by the user. In a second step the framework then integrates the three choice classes by combining the single choice ratios into a layered structure constituting a single action (Figure 2). Subjective interaction quality is then a user’s evaluated average of the resulting emotions from the three choices, ultimately summed over all actions over the life-time of interaction during a task. Thus the framework allows describing and predicting the quality of interaction with an interactive artifact.

Figure 1: Comparisons of anticipated and actual progress vs. effort within a single choice during interaction

Figure 2: Three core choices (goal, UI element, & trigger) constituting a single action during interaction

Conclusion This paper described the importance of judgment and decision making in user experience research. It illustrated how judgments, decisions, and emotions interrelate and how choices influence experiences. With respect to interaction the paper elaborated three common choices repeatedly occurring during interaction and highlighted the importance of effort. An initial predictive model of interaction experience was then introduced that, combined with the three choices, form a framework of subjective interaction quality. Users constantly evaluate their anticipated progress and effort and compare it to the actual experienced one. Throughout the life-time of interaction in a task this framework allows predicting the overall quality of interaction with an interactive artifact. Future research may generate empirical results to test this framework by manipulating the elements involved in the three choices. For example, effort may be increased by simply adding another click, such as in Gray et al. (2006), or delaying the response of a user interface. Likewise research may evaluate how different label semantics in relation to a task description hinder or support positive interaction experiences. All these manipulations should generate different emotional responses and thus interaction experiences that can be captured with common emotion measurement methods, such as electrodermal activity, or facial electromyography as well as questionnaires. In summary, the presented account here makes an attempt to deliver a predictive model of user experience that allows prediction and thus hypothesis generation. It complements existing descriptive models and frameworks of user experience that identify important variables and their possible connections but lack the description of mechanisms that

References Beauregard, R. & Corriveau, P. (2007). User Experience Quality: A Conceptual Framework. In: Duffy, V.G. (ed.) HCII 2007, LNCS, vol. 4561, pp. 325-332. Springer, Heidelberg, Germany. Bhavnani, S.K. & John, B.E. (2000). The Strategic Use of Complex Computer Systems. Human Computer Interaction 15(2/3), 107-137. Damasio, A. (1994). Descartes’ Error. New York, USA: Penguin Books. Forlizzi, J. & Ford, S. (2000). The Building Blocks of Experience: An Early Framework for Interaction Designers. In: Proceedings of the DIS 2000 seminar. New York: Communications of the ACM. Fredrickson, B.L. (2000). Extracting meaning from past affective experiences: the importance of peaks, ends, and specific emotions. Cognition & Emotion 14(4), 577–606. Gray, W.D., Sims, C.R., Fu, W., & Schoelles, M.J. (2006). The Soft Constraints Hypothesis: A Rational Analysis Approach to Resource Allocation for Interactive Behavior. Psychological Review 113(3), 461-482. Hassenzahl, M. (2003). The Thing and I: Understanding the Relationship between User and Product. In: M.A. Blythe, K. Overbeeke, A.F. Monk, & P.C. Wright (Eds.), Funology – From Usability to Enjoyment (pp. 31-42). Dordrecht, the Netherlands: Kluwer Academic Publishers. Hommel, B. & Elsner, B. (2009). Acquisition, Representation, and Control of Action. In: Morsella, E. Bargh, J.A., & Gollwitzer, P.M. (Eds.). New York: Oxford University Press, pp. 371-398. Hsee, C.K. & Hastie, R. (2006). Decision and experience: why don’t we choose what makes us happy? Trends in Cognitive Sciences 10(1), 31-37. Hsee, C.K., 1996a. Elastic justification: How unjustifiable factors influence judgments. Organizational Behavior and Human Decision Processes 66, 122– 129. Hsee, C.K., 1996b. The evaluability hypothesis: An explanation for preference reversals between joint and separate evaluations of alternatives. Organizational Behavior and Human Decision Processes 67, 242–257. McCarthy, J. & Wright, P. (2004). Technology as Experience. Cambridge, MA: MIT Press. Mellers, B., Schwartz, A., & Ritov, I. (1999). Emotion-Based Choice. Journal of Experimental Psychology: General 128(3), 332-345. Mellers, B.A. & McGraw, A.P. (2001). Anticipated Emotions as Guides to Choice. Current Directions in Psychological Science 10(6), 210-214. Monk, A. (1998). Cyclic interaction: a unitary approach to intention, action and the environment. Cognition, 68(2), 95-110. Norman, D. A. (1988). The Psychology of Everyday Things: Basic Books. Payne, S.J., Howes, A., & Reader, W.R. (2001). Adaptively distributing cognition: a decision-making perspective on human-computer interaction. Behaviour & Information Technology 20(5), 339-346. Payne, S. J. (1991). Display-based action at the user interface. International Journal of Man-Machine Studies 35, 275-289. Slovic, P., Finucane, M. L., Peters, E., & MacGregor, D.G. (2002). The Affect Heuristic. In: Gilovich, T. Griffin, D. & Kahneman, D. (Eds.), Heuristics and biases: The psychology of intuitive thought. New York: Cambridge University Press, pp. 397–420. Storbeck, J., Robinson, M.D., & McCourt, M.E. (2006). Semantic Processing Precedes Affect Retrieval: The Neurological Case for Cognitive Primacy in Visual Processing. Review of General Psychology 10(1), 41-55.