Axe UX: Exploring Long-Term User Experience with ...

Axe UX: Exploring Long-Term User Experience with iScale and AttrakDiff Tanja Walsh

Jari Varsaluoma

Sari Kujala

Tampere University of Technology Korkeakoulunkatu 6, P.O. Box 589, 33101 Tampere, Finland

Tampere University of Technology Korkeakoulunkatu 6, P.O. Box 589, 33101 Tampere, Finland

Aalto University P.O. box 31000 FI-00076 Aalto, Finland

[email protected]

[email protected]

Piia Nurkka

Helen Petrie

Tampere University of Technology Korkeakoulunkatu 6, P.O. Box 589, 33101 Tampere, Finland

University of York Heslington, York, YO10 5DD, United Kingdom

[email protected]

[email protected]

Positive user experience (UX) is an important goal in product design. Positive long-term UX is believed to improve customer loyalty, therefore being vital for continuous commercial success. Most UX research investigates momentary or short-term UX although the relationship between the user and the product evolves over time. There is a need to develop methods for measuring long-term UX and evaluate their feasibility in different product contexts. In this explorative study, 18 customers reported their experiences during the first three months of use of a noninteractive design tool, an axe. UX was evaluated with retrospective iScale tool and monthly repeated AttrakDiff questionnaire. iScale demonstrated the long-term trend of the attractiveness of the product well and provided information about the causes of the change in the experience. The AttrakDiff questionnaire was a good indicator of attractiveness during a longer period of time and is also well suited to remote studies.

It is also suggested that long-term UX and relationship with a product evolve over time from early learning and enthusiasm to its becoming a part of daily life is more important than single momentary experiences [13]. It is long-term UX that makes people continue to use a product and recommend it for others, not individual experiences at a certain point in time [13].

H.5.2 [Information Interfaces and Presentation (e.g., HCI)]: User Interfaces evaluation/methodology, theory and methods, ergonomics.

Kujala et al. [13] showed that the trend of long-term UX, whether it improves or deteriorates over time, is also meaningful to users. In their study using the UX Curve method, the improving trend of the perceived attractiveness of mobile phones was related to user satisfaction and willingness to recommend the phone to friends. Thus, the optimal user-product relationship may not just stay the same, but actually improves over time as a consequence of learning and other factors.

General Terms Measurement, Design, Human Factors.

Keywords Long-term user experience, AttrakDiff, iScale.

1. INTRODUCTION Users seem to have relationships with products that resemble interpersonal relationships [4, 7]. These relationships develop

In this paper, we discuss a study of long-term UX of a noninteractive pragmatic product, namely an axe. This study had two distinct but complementary goals. First, we wanted to understand how the UX and relationship with this particular product evolved over time. Second, from a methodological point of view, we investigate the strengths and weaknesses of two different methods in a long-term UX evaluation of this pragmatic, non-interactive tool. Furthermore, we discuss studies. The methods we used were retrospective iScale tool and repeated measurements with the AttrakDiff questionnaire.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. 07 2014, Tampere, 978-1-4503-3006-

[email protected]

Several studies have shown that the nature of different UX qualities change over time [5, 10, 11]. For example, Karapanos et al. [11] followed six participants for one month after the purchase of an Apple iPhone and found that, while the importance of its novelty and social meanings faded away quickly, over time different sources of hedonic quality emerged, such as the participation of the product in daily rituals and cherished activities.

Categories and Subject Descriptors

-

Chris Power University of York Heslington, York, YO10 5DD, United Kingdom

over time as experiences accumulate and a good relationship is related to factors such as trust, improving learning [4], and brand loyalty [7]. User experience (UX) has an essential role in building a good user-product relationship. The positive experiences of a product create user satisfaction and, conversely, negative ones can lead to product abandonment [16].

ABSTRACT

AcademicMindTrek '14, November 04 Copyright 2014 ACM http://dx.doi.org/10.1145/2676467.2676480

[email protected]

Finland

34

iScale is an online survey tool, developed by Karapanos et al. [12]. With iScale, one can gather data from longer periods of time e.g. after several months of use. iScale is designed to minimize retrospective bias by assisting users to recall their experiences

and presents goodness as being affected primarily by pragmatic aspects and beauty by identification. Tractinsky and Zmiri [22] studied satisfying and pleasant experiences and found perceptions of usability to be better predictors of a satisfying rather than a

of the most used methods in measuring both pragmatic and hedonic dimensions of UX. It is a questionnaire of how attractive the product is in terms of usability and appearance [2, 8, 9]. Since one measurement of UX with AttrakDiff does not tell how the UX develops during a longer period of time, repetitive measurements are required during the usage of a product. We were interested to study how these two methods can complete each other as iScale provides mainly qualitative and AttrakDiff quantitative data related to UX.

aesthetics were better predictors of a pleasant experience than a satisfying one. But how stable are these relations over time? In an exploratory study to identify differences in the way users form overall judgments about products, Karapanos et al. [11] found that while pragmatic qualities were the primary predictor of the goodness of the product in early interactions, in prolonged experiences identification became the primary predictor of goodness. Kujala et al. [14] also state that the importance of hedonic product qualities increases in the longer term. Thus product qualities that make initial experiences positive and satisfying do not necessarily have the same effects after prolonged use.

Our study investigated the UX of a new product series of a highquality design brand axe during a 3-month period of use in Finland with 18 users. We chose the axe as it represents iconic experiential design and is more than just a functional product. The Finnish manufacturer of the axe is famous for its high quality tools and further the axe represents a pleasurable lifestyle product that creates very positive experiences especially in Finland. We believe this makes it an excellent locus for developing methods for evaluating long-term UX that can then be applied to a variety of other interactive experiences. Finally, if a method is designed for capturing the experiences of people with technology, then they must work across a variety of computer based and non-computer based technologies. This study serves to demonstrate how these methods can be used in broader product design.

2.2 Measuring Long-term User Experience Long-term methods such as diary studies may provide rich insights into long-term UX in a natural context, but such longterm studies are very rare because of their expense and laborious nature. The participants have to sustain motivation in reporting for the full length of the study. In addition, people have a huge number of varied experiences related to products and services and it is not known which ones are the critical experiences that affect the overall evaluation and value of the product or service. Retrospective methods such as the UX Curve [13, 14], iScale [12], and CORPUS [24] are more practical, but they may be vulnerable to memory biases as users report their experiences several months after they occurred.

2. BACKGROUND The recent shift of emphasis from usability to UX has made it a central focus of product design and evaluation [23]. Although interest in UX in industry and academia is high, there is still lack of systematic research on how to evaluate and measure UX [24]. Many methods exist for performing traditional usability evaluations, but UX evaluation clearly differs from usability evaluation [21]. Vermeeren et al. [23] point out that while it is relevant to evaluate short-term UX, given the dynamic changes in user goals and needs related to contextual factors, it is also important to know how (and why) UX evolves over time.

However, Norman [17] and Karapanos et al. [12] argue that the memories of experiences are as important as the actual experiences, as the memories of experiences will be reported to others and guide the future behavior of the individual. Indeed, the results of Kujala et al. [13] suggest that memories are related to user satisfaction and recommendation to friends. Thus, the memories of the trend of long-term UX may provide important feedback for product design [14]. From the design point of view, it is essential to understand how UX evolves over time and what the underlying reasons for changing UX are.

2.1 Experience and Temporality

There are several potential factors in addition to UX that affect how the image of a product is created, such as the brand and expectations [18]. For design purposes, we need to identify the critical product properties that affect UX and, finally, the userproduct relationship. In summary, information about the trend of long-term UX has the potential to describe the user-product relationship and measure user satisfaction more reliably than a single measurement of momentary UX. However, in order to support designing for UX, the relationship of memories to real experiences needs to be understood.

Several frameworks have been developed to describe how UX is formed, adapted and communicated. For example, Forlizzi and Battarbee [6] state that an experience develops from unconsciousness to a cognitive state and finally becomes a memorable experience that can be communicated to others. Our experiences are not stored in our memory as we initially experience them, but experiences are lifted and downgraded by social mechanisms [3]. Hassenzahl [9] presents UX as a combination of pragmatic and hedonic qualities and this definition is referred to in several other UX papers [e.g. 20, 22]. Pragmatic

iScale and UX Curve: Karapanos et al. [12] developed iScale, an online survey tool designed to minimize retrospective bias by assisting users in recalling their experiences with a product by

of behavioral goals (e.g. usefulness and ease-of-use) whereas 9]. Hedonic qualities are further divided into stimulation, which stimulate and facilitate personal growth, and identification, which

of two versions of iScale and free recall without any form of sketching. Their study showed that sketching increases the amount and the richness of the information recalled. However, the study did not include qualitative or interpersonal analysis of the results or analysis of trend information.

through objects one owns [9]. But what is the relation of these qualities when forming an overall impression of a product? Hassenzahl [9] suggests beauty and goodness as two distinct overall evaluative judgments of products

35

Facebook. As the use of an axe can be seasonal and the peak season for purchase is spring-summertime, we had to time the recruitment for May.

Kujala et al. [13, 14] developed the UX Curve method for assisting users in retrospectively describing their long-term UX over several months. Participants are asked to recall the moment when they began using a product or service and to draw a curve describing how their relationship changed from the first time they used the product or service until today and to explain the reasons for the experience improving or deteriorating. The method supports users in reporting pleasure-inducing experiences that are otherwise difficult to articulate [14]. Using the method showed that the improving trend of attractiveness is related to customer satisfaction and customer loyalty as measured by word of mouth [13].

Out of 102 respondents, 24 were selected for the study according to a set of screening questions. We required participants to have actually bought the axe and planning to use it more than once a month. We also required them to have written down three expectations they had for the axe. All 102 participants who had answered the screening questionnaire joined a raffle where they had a chance to win a gardening toolset. Participants selected for the study received a gardening toolset at the end of the study. Three out of 24 of the selected participants dropped out of the study right at the very beginning. Another three participants did not fill in all the online AttrakDiff questionnaires and therefore they were excluded from the analysis. Motivating the users during a long-term study is important, as the drop-out rates can be high, especially in online surveys, where the response rate is, on average, approximately 11% lower than in other survey modes [15]. We phoned the chosen participants in the beginning to ensure their commitment to our study. We also sent reminder text messages to them when it was time to fill in the AttrakDiff questionnaires. In the end, 18 participants finished the study.

Both iScale and the UX Curve methods were designed to measure long-term trends in UX and to understand the quality of design supporting the user-product relationship or causing it to deteriorate. In principle, both methods can be used to measure any UX dimension, but the dimensions used in the studies cited were different. iScale focused on usefulness, ease of use, and innovativeness [12], whereas UX Curve focused in addition on a general open dimension and attractiveness [13]. The UX Curve and, in particular, its attractiveness dimension focus more on the pleasure-oriented aspects and the method also provides a framework for analyzing the curve trends [13]. iScale was designed as an online survey technique, whereas the UX Curve is paper-based.

As the product in the study was an everyday design tool, the consumer base for it was wide. The age range in the study was from 24 to 73 years old (mean 45, SD = 13). Nearly all were male (16 male, 2 female). The occupations of the respondents varied greatly. There were retired people, an architect, ICT professionals, a carpenter, a maintenance clerk, and civil engineers. All of the respondents had used an axe before and the majority (16/18 participants) said they had earlier used the same brand of axe that was in the study. In the screening questionnaire, we asked what kind of expectations the users had of the axe. The most frequently mentioned expectations were that the axe would be durable, effective, and of high quality.

AttrakDiff is a questionnaire measuring how attractive a product is in terms of usability and appearance [2, 8, 9]. AttrakDiff consists of 32 7-point bipolar items that represent opposites (e.g. good - bad). The 32 items measure the following UX dimensions: 1) pragmatic quality; 2) hedonic quality identity; 3) hedonic quality stimulation, and 4) attractiveness [2, 8, 9]. The word of mouth (WoM) comments and recommendations that users pass on to others about a product or service are related to company success [20]. This is why a question about the willingness to recommend a company is one of the most used questions for measuring customer satisfaction and predicting loyalty and growth [20]. For example, the spread of mobile services is affected by how users talk about the services to their

Process: The study was divided into four parts: 1) user recruitment and screening; 2) 1st AttrakDiff questionnaire after the first month of use; 3) 2nd AttrakDiff questionnaire after the second month of use, and 4) 3rd AttrakDiff questionnaire, iScale, and interviews after the third month of use. The results of the interviews are not in the scope of this paper.

complemented by explicit recommendation of the service, may lead to its adoption by new users [1]. In our study, WoM was used as a comparison to other UX measures.

Repetitive measurements with the AttrakDiff questionnaire were planned for every month. Since the axe is a product that is not used daily, but perhaps weekly, the period of one month between the measurements was set. The participants were asked to fill in the first two AttrakDiff questionnaires remotely from their homes either on the internet or on paper. 16 out of 18 users filled the AttrakDiff questionnaire remotely in Webropol and two users filled it in on paper because they did not have an internet connection at home.

3. METHOD Product: Our study investigated the UX of a new axe product series in Finland during a 3-month period of use. The axe is a common tool in Finland and it is used on many occasions, such as for chopping wood to make it suitable for heating, for renovation and construction work, for gardening and forestry, and in summer cottages where lots of Finns like to spend their summer holidays. The design of the axe used in the study is by a highly appreciated, iconic brand. In addition, axe culture is very popular in Finland. For instance, there are videos on YouTube of wood being skillfully chopped with over 3 million viewers (https://www.youtube.com/watch?v=2vThcK-idm0).

After the third month of use, a face-to-face session was arranged with each participant. The session included the final Attrakdiff questionnaire, an interview, and the iScale curve drawing task. The Attrakdiff questionnaire was always filled in first. Then the interview and iScale followed and their order was counterbalanced.

Participants: The majority of the participants (22 out of 24) were recruited directly from shops selling the particular axe in order to reach users who had just bought the product and not used it yet. Stands with paper questionnaires including screening questions were set next to the products in four different hardware shops. In addition, two of the users were recruited through advertising on

AttrakDiff: We measured the UX after one, two, and three months of use with the AttrakDiff questionnaire version 2.0 [2]. We added four product-specific word pairs in addition to the original AttrakDiff items in order to acquire more detailed feedback

36

related to the axe context. The added product specific items were: Wears out fast-Durable, Dangerous-Safe, Unreliable-Reliable, -Fits well in the hand. In addition to the AttrakDiff items, the questionnaire included a question about the

Open coding was used to identify themes in the data, resulting in attractiveness had increased, stayed neutral, or decreased. In the end, the two resulting sets of categories were combined and discussed to produce the final categorization. There were eight instances the comment and curve shape did not match, for example a positive comment on a decreasing part of the curve. In these cases, we prioritized the content of the narrative when compared with the curve shape and categorized the comment accordingly.

friend was going to buy this product, how likely is it that you would recommend this axe to your friend on a scale of 0(0 = Not at all likely, 10 = Very likely) (adapted from Reichheld [19]). iScale: iScale was modified to correspond more closely to the UX Curve, because the UX Curve, and in particular its Attractiveness dimension, focus more on the pleasure-oriented aspects of UX and the method also provides a framework for analyzing the curve trends [13]. As the Attractiveness Curve trend used by Kujala et al. [13] was related to user satisfaction and WoM, it was selected as the only curve type to be used. We also wanted to keep the length of the study reasonably short, as several methods were used in the same session.

4. RESULTS AttrakDiff: Figure 1 shows the mean values of the four AttrakDiff dimensions and the four additional product-specific word pairs for each of the three measurements. A bigger number on the y-axis (on a scale of 1-7) depicts better experience with the product. The experience in all areas improved slightly during the three months of use. The experience in all dimensions of AttrakDiff went down slightly after the second month of use except in Pragmatic quality. Interestingly, after three months of use, all dimensions improved above the two previous measurements.

To simplify the curve-sketching task for the participants, we did not ask for exact dates for experience narratives as in [12], but merely added vertical lines to the background to represent different months. The top of the y-

A Friedman Test was conducted for each AttrakDiff dimension measures to verify if the differences between any two months were statistically significant. This was found only in the Attractiveness dimension and post-hoc analysis with Wilcoxon Signed-Rank Tests was conducted. Bonferroni corrected significance levels for the pairwise comparisons indicate that there was a statistically significant increase in attractiveness between the first and third months (Z = 2.422, p