How to Make Agile UX Work More Efficient: Management and Sales Perspectives Kati Kuusinen Tampere University of Technology Tampere, Finland
[email protected]
Kaisa Väänänen-Vainio-Mattila Tampere University of Technology Tampere, Finland
[email protected]
to challenges in cooperation between those functions. To overcome such challenges, some companies are starting to adopt Agile1 practices in contexts other than R&D [25].
ABSTRACT
Many companies have been conducting their software development activities using Agile methods for years; however, in many cases the management and sales processes of these companies are still run in a more traditional way. This can cause clashes between development and management; and especially in terms of user experience specialists, who are often operating in between these organizations.
User experience (UX) design [13] has traditionally been conducted prior to starting the development. Therefore, it is still often run outside Agile deve lopment, as a separate stream of work. Pre-development activities, including user experience design, are in many cases divided into work that is conducted within Agile development, and work that is conducted before starting Agile practices.
This paper describes the present state of user experience work in development and sales processes in a l arge, globally-operating IT service company. The data were gathered via an online survey (with 67 respondents) and 17 theme interviews. The study indicates that although the user experience approach is well established inside the organization, the impact of th is approach is l imited by mismatching sales and manageme nt practices and processes, which are, in many cases, working in more traditional ways. I n specific, the business model in sales was seen to hinder user experience work.
Previous research addresses numerous management perspectives on UX work from within companies, e.g. [16, 20]. However, cooperation between Agile business management and UX design—especially between sales and design—are less thoroughly studied areas. The goal of the study presented in this paper is to reveal the issues that most hinder Agile UX work in the company, and current good p ractices, in order to improve the overall situation. By Agile UX we refer to work that systematically results in desired user experience of the outcome and is conducted according to Agile principles. This paper outlines the experiences of UX practitioners working within an Agile sales process, and identifies the consequences of strategic sales management for later UX work. The findings indicate that not only needs the UX team be skilled, but in addition the management decisions and power relations have a great influence on the impact of the UX work.
Author Keywords
Agile user experience, user experience (UX) management, Agile development ACM Classification Keywords
H.5.2 [User interfaces]: Evaluation/methodology. D.2.2 [Design tools and techniques]: User interfaces. K.6.3 [Software Management]: Software process. General Terms
This study examines the state-of-the-practice of Agile UX work in a l arge, globally-operating IT service company, where development is internationally distributed and the development style is usually selected by the customer. This paper also describes the UX practices that were observed to work well both within sales and development activities.
Human Factors; Design; Measurement; Management. INTRODUCTION
Agile methods are increasingly being used for research and development (R&D). However, other areas, such as management and sales, are often run in mor e traditional way. Different processes and management styles may lead
The data for analysis consists of an international web survey with 49 questions and with 67 respondents. The survey was followed by 17 theme interviews.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NordiCHI '12, October 14-17, 2012 Copenhagen, Denmark Copyright © 2012 ACM 978-1-4503-1482-4/12/10... $15.00"
The rest of the paper is structured as follows. Initially some related work on A gile UX is presented, especially research 1
139
Agile Manifesto 2001. http://agilemanifesto.org
produce images. They also found that developers had stronger power to make decisions in the projects that resulted in unsuccessful releases compared to successful ones: they suggest that close cooperation between designers and managers is therefore a success factor in software development. In addition, they identify a strong association between common goals and perceived release success.
that examines the management perspective. Following this, the company in which the study was conducted is outlined. Then, a description of how the actual study was implemented is provided, and the main findings are presented. Finally, the s tudy is discussed as a whole, and some conclusions are offered. RELATED RESEARCH
In the related research, usability and user experience have been addressed from various management perspectives, such as business metrics, UX management, and power relationships. Some brief insights into these areas are presented in this section.
Power of Decision
Boivie et al. [3], state that management support is essential for effective usability work. Lund [16] and Rohn [20] emphasize the importance of a high-level usability chief. Lund [16] claims that a UX team working as a service organization is a major threat to influential UX work. Boivie et al. [3] identify problems in prioritization of the HCI profession, where usability and user involvement were found to have low priority in projects.
Business Value and Metrics
Innes [12] states that managers often focus on short-term cost and r isk reduction, while ignoring the costs that inefficient design processes and the building of inadequate products incur. He argues that as technology maturity (in general) is a lready sufficient, such that it is difficult for a company to stand out from the competitors only by being a technology leader. Companies that fail to produce good UX are losing market share, wh ile their competitors are successfully applying UX methods to create superior products with lower costs, as they have more efficient design processes [12].
In summary, these studies find that although UX professionals are skillful in their use of methods, they struggle to find the positions in companies that will ensure the impact of the UX work in their organizations. UX Management
Managers often ask for return on investment (ROI) analyses for any UX activities. However, calculating the ROI for UX activities is not a straightforward task, and it ignores aspects other than directly financial contributions: such as customer satisfaction and influence on corporate reputation [24]. Rajanen et al. [19] state that the costs of UX work are evident immediately; whereas benefits only appear later on during the software lifecycle. Thus, systematic UX work should not be considered as a cost but as an investment in better processes and practices.
User experience management has been addressed in various panels and special interest groups of the CHI conference. For example, Jain et al. [14] list ke y issues that need to b e tackled to achieve a successful role for UX within organizations. These issues include engagement, accountability, communication and developing standard operation mechanisms. Similarly, UX management issues have also been covered in tw o separate special issues of Interactions magazine [18, 21]. These magazine issues include several articles on how user experience work should link to the various functions in an organization (e.g. Rosenberg [21]), and th ey identify that the ownership of UX work is often unclear and needs to be negotiated [18].
Vredenburg et al. [29] identify that measures of effectiveness of human-centered design (HCD) [13] are lacking and rarely applied. They also identify a major discrepancy between the commonly cited measures (such as customer satisfaction, ease of use and increased sales), and the measures that are actually applied.
Rohn [20] notes that managers and other stakeholders rarely understand how UX can support their goals, and she suggests that it is the UX professional’s responsibility to translate UX activities into stakeholder goals. She also considers that UX managers should continue to b uild strategies to raise the awareness of UX inside the company. Rohn [20] also considers that the funding model has an impact on UX: she argues that the extent to which a project utilizes UX resource should not be connected to the funding model, and that use of the UX resources should be straightforward and unhampered by bureaucracy.
Cooperation between Business and Design
There are numerous studies about cooperation between development and UX design [e.g. 5, 7, 10]. However, cooperation between management and UX , or betwe en sales and UX, are less studied areas in software industry [4]. Botzenhardt et al. [4] conducted a h ybrid study, where they first interviewed 13 product managers and UX persons and later applied an online survey with 176 participants from software industry. In the survey, they compared organizational setup, cooperation, communication and decision-making between successful and unsuccessful software projects. They claim that when development teams have the power of decision, UX team’s role is just to
Sales Management
Van Solingen et al. [25] describe a successful process of implementing the Scrum [23] method in the sales process of a large company. As a result of this method, sales became a more predictable practice, sales teams were able to eng age in ongoing improvement and the sales process became more transparent. Solingen et al. [25] state that their study was
140
References were studied to determine significant areas and common problems in Agile UX work to generate appropriate questions. Questions and concerns were collected from the mentioned studies and iterated into a survey by the two researchers.
the first to examine Agile sales. Further studies about UX work within Agile sales activities are unavailable. Bias et al. [2005] state that companies are increasingly utilizing UX specialists during sales. Vaughan [26] who is working in a UX organization at O racle reported that Oracle is aligning UX and sales. At Oracle, they believe that sales organization should understand how to position UX with customers. Additionally, they see that aligning UX and sales can increase the strategic relevance of UX for both the business and customer.
Survey Participants
Participants of the online survey were selected from R&D and organizations working close to it, such as management, sales and marketing. Due to the large size of the company, and the fa ct that large p ortion of the work was conducted with methods other than Agile, the participant population was selected from certain lists (including lists of Agile development organizations). Systematic sampling with an equal-probability method was us ed, with a s ampling interval of 10. The starting point was chosen at random.
THE COMPANY
The information technology service company studied in the present research has around 20 000 employees worldwide. As the company is providing tailored software for customer companies, it has to adapt to customers’ various ways of working. Thus, the development style varies in each case. The company has both public and private sector customers and the delivery varies from small applications to organization-wide business software. In addition, the company has its own product development that also has divergent approaches depending on pr oduct, organization and team. The company has common guidelines for ways of working on a general level, and each in-house project can select the most suitable methods amongst those. Methods are also further developed in certain projects.
The survey was open for answers for four weeks in March and April 2011. It had 67 respondents: 58.2% from Finland, 26.9% from other countries (the majority from Sweden and Czech Republic), and 18.0% did not provide a country. The designation and job content were asked in separate openended questions, and the respondents were classified into five task groups based on their answer. Of the respondents, 44.7% worked mainly as developers or architects, 16.4% as business or sales managers, 11.9% as product owners or scrum masters, 10.4% were UX practitioners, 6.0% were Agile or Lean coaches, and 10% did not provide an answer.
The company has a centralized, physically co-located UX team and has individual UX specialists inside different organization units. Typically, the distributed staffs are UX designers. The centralized team consists of a team leader, UX designers, usability specialists and UX consultants. There are also other individuals, with various backgrounds, who are responsible of UX issues inside the organization or development team. In this paper, we refer to all these UX related roles as ‘UX practitioners’. The majority of the UX work is conducted in Finland and Sweden, whereas development is also conducted in other countries.
Survey Analysis Method
Open-ended questions of the survey were analyzed with a qualitative, theory-bound content analysis method, using an emergent coding approach as described by Lazar et al. [15, pp. 285–301]. The questions that are discussed in this paper were analyzed by two researchers. Before the actual analysis, undesirable responses, such as “I do not know” or “I have no opinion,” were removed from the open-ended question data. In the beginning of the analysis, the two researchers separately read the material and p re-classified the data per question. After that the researchers discussed the categories and derived actual classes. The dat a were then coded i nto those classes individually (by two subjective coders) to quantify them.
RESEARCH METHOD
The goal of the study presented in th is paper was to understand the present state of UX work in Agile software development projects, in order to be able to improve it. The applied research approach was a case study, as instructed by Runeson et al. [ 22]. The study was conducted by a researcher from a university research organization. The data for this case study were gathered using a m ixed methods approach utilizing first an online survey, and continued with interviews and analyzing internal documentation (such as process descriptions and work instructions).
Interviews
Interviews were conducted in October and November 2011. Interview questions covered the participants’ work responsibilities and how they contribute to UX. In addition, current best practices and areas of improvement concerning UX activities were addressed. As t he survey results suggested that hindering issues occurred mainly on the process and m anagement levels, the ma jority of the interviewees that were chosen were business or UX representatives (Table 1). The in terviewees were selected from various organizations in Finland and Sweden, to cover both public and private sector processes. The selection criteria included that the person was working in an area that utilized Agile working practices.
Survey
The survey consisted of 49 questions, 19 c losed-ended and 30 open-ended, on processes and tools, collaboration and communication and concepts and knowledge in the company. Two researchers generated the questions, based on a v ariety of studies [3, 6, 8, 9, 11, 17, 27, 28, 30].
141
possible further analysis. Each software note was then marked as excluded or labeled with a green, orange, and yellow code.
Altogether, 14 employees and 3 custo mer representatives were interviewed, either face-to-face or via Microsoft Live Meeting. Of these, J17 was from Sweden and the others were from various locations in Finland. Code J1 J2 J3 J4 J5 J6 J7 J8 J9
Role Software architect Director Senior UX designer Head of services Delivery manager Senior UX designer Analyst Senior consultant Project manager
Code J10 J11 J12 J13 J14 J15 J16 J17
RESULTS
Role Service designer Head of innovation Head of sales Customer Customer Customer UX architect UX designer
The following begins by considering survey results separately to clarify the reasons for directing the focus of the interviews into management issues. After that, the main findings are introduced, which are organized into three levels of management. Although listed separately, the issues on top management are often reflected on the s hop floor (and vice versa). Therefore, improving the situation requires consideration of every level of the organization. The results concentrate on interview findings, as th ese address the most management issues.
Table 1. Roles of the interviewees
The roles of the interviewed customers included a manager (who has a general responsibility over development projects in a customer organization), head of service offering (who ensures that service development meets their customers’ expectations) and first vice president (who is responsible for different distribution channels and service offerings in the Nordic countries). All the customer representatives were from financial sector and acted as product owners in projects where the studied supplier company delivered software services.
Survey Results
Although the survey was primarily focused on development activities, respondents clearly highlighted that the main problems in Agile UX were related to t he absence of UX activities in the development process and management issues. We asked the participants to n ame three issues that currently hinder Agile UX work the most, and process and timing issues were the most prevalent among their responses: 26.8% of respondents (N=41) named the lack o f UX on process level, 19.5% mentioned timing or budget issues and 12.2% mentioned that customers were not committing sufficiently, or they have deficiencies in understanding the importance of UX issues. Other commonly mentioned issues included poor communication between stakeholders (19.5%), lack of understanding the end users (22.0%) and missing the big picture (14.6%).
Analysis Method for the Interview Data
The interviews were recorded and analyzed from written transcripts using the affinity wall method as ins tructed by Beyer and Holtzblatt [1, pp. 153–163]. The method is datadriven and utilizes theory-independent coding. Written transcripts were edited into 1067 short notes, each including one piece of information with a code iden tifying the interviewee and consecutive numbering for identifying the note. Altogether, six researchers (2 subjective and 4 objective coders) attended in coding, 2–4 at a time. Each note was individually categorized: either included into an existing category, or p laced in a new category (which was created if no suitable category was available). If the note was considered irrelevant to the study, it was excluded from the analysis. Such notes primarily contained data that purely discussed the interviewee’s job content (e.g. “Currently, I am working as product owner”), or described a process with no connection to UX work (e.g. “We can have five week sprints”).
Figure 1. How would you improve cooperation in UX issues? Open-ended question, N=34.
When approximately half of the data was on the wall (or excluded), new categories were labeled with pink post-it notes to make t he data easier to understand. Later, the data were revisited to form adequate-sized low-level categories, with yellow labels. These categories were then grouped under upper (middle-level) categories that were labeled with orange post-it notes. Those middle-level categories were numbered and converted into digital form, after which the top-level categories were created and gi ven green labels. Category codes were written in to a Microsoft Excel file with the original notes, to simplify reporting and
The most desired improvements (Figure 1) included integrating UX into development processes and increasing awareness of UX. Increasing stakeholders’ awareness included: a UX toolbox for developers, a virtual UX team where UX persons could help each other and an easy and effortless channel for projects to get instant help in UX issues. Increasing the awarenes s of managers was also a suggested approach.
142
People who make decisions about UX work should have understanding of both the possibilities UX provides and the consequences of the decisions concerning UX. As individuals who are not UX specialists were usually making decisions concerning the scope and timing of UX work, they should, at least, consult UX staffs at an early stage. An even better practice would be to hav e high-level UX managers with sufficient authority to make decisions.
Business Management UX as an Important Business Asset
The majority (9) of interviewees believed that delivering software with good user experience is highly important for business, and that both customers and end users are increasingly demanding improvement to user experience (J3, J6, J7, J8, J10, J12, J13, J14, J15). A customer commented on the prevailing situation: “There are new players on our business field who have services with better UX. Little by little, UX is becoming a must in our field (banking) because users’ expectations are growing” (J15).
There was a centralized UX team operating at one site. This team had a team l eader and role s that seemed functional. However, UX staff on other sites was less pleased with the situation. They felt the ownership of UX was unclear and that there was no clear UX management. They wanted peer support from other UX practitioners, and to be able to coordinate and delegate UX work more ea sily. UX practitioners were usually operating at the project level and distributed UX staff in particular felt that they were alone with UX issues.
All the interviewed customers and a s ales person reported that good UX i mproves software quality and increases economic value added; in contrast, poor UX drives customers away and leads to a bad reputation (J4, J10, J12, J17). Both c ustomers and suppliers reported they have recently started to pay more attention to UX: “Previously it did not need to be pleasant to use (our services) because using internet services from home was an option to walking to a se rvice counter in rain and snow” (J15). “Nowadays bad UX still might not be a reason to change service provider but it strongly guides what is bought from the internet channel; is it used only for the necessary or do we get some additional sales” (J13).
Understanding User Experience
The awareness of the customer had a strong effect on the extent to which UX was considered in the delivery process (J9, J10). Customers had high ly divergent understandings of user needs and user experience issues. A customer might have even withheld information about users for business reasons (J1, J9). In the survey, the awareness of stakeholders was identified as one of the main concerns. A mana ger stated that one of hindering issues in Agile UX was a “lack of seeing UX and usability work as an own profession—not to be done by developers or managers.” Interviewees saw the term UX as trendy (J4, J6, J10): When you say “usability” no one is interested. But when you say “UX,” managers’ eyes glitter. They have noticed it is very important issue in mobile applications. In desktop applications people can live with bad usability, but no one will use a poor mobile application. Especially on touch screen, it is very easy to fail with user experience. Even the top management understands we really need to design it well (J6).
Nobody is Responsible for User Experience
In general, no individual seemed to be responsible for UX issues in a wider perspective. The UX staffs were more action-driven or consultancy-based than decision-makers. It was also unclear whether UX issues were a customer’s or a supplier’s concern. Interviewed supplier representatives stated that customers are responsible for the user experience of the outcome (see J1, J5, J6, J9): “No, users are not involved because we are not selling the software to users but customers and it is their concern” (J6); conversely, customers stated that the supplier is responsible of delivering software that fulfills user needs. Customers naturally made decisions on what they were buying, and in many cases the supplier respected customer wishes in a loy al manner. The c ustomer needs to be f ully aware of the needs of their users: otherwise, good UX could be compromised when the supplier listened to the customer literally. Some of the i nterviewed supplier representatives felt it was difficult to disagr ee with a s trong customer: customers who did not have had expertise in UX nevertheless presented demands concerning user flow and user interface (UI) details.
The Proportion of Agile Projects is Rising
As the company is producing tailored software to customer orders, the customer can naturally select the process model that is used on the project. According to interviewees, customers were increasingly selecting Agile models, especially Scrum. Therefore, the company was pressured to make its sales and ma nagement activities more agile. Customers found it difficult to buy software with Agile sales approach. Also, the Agile concept creation was considered difficult in many cases (J1, J2, J3, J9, J10).
Managing User Experience Work
However, there were also experiences in the company of successful Agile sales and concept creation. One such example is an approach the interviewed sales person and customer had started a couple of years previously. They
Recently, UX has been gaining more attention, at both strategic and project levels in the company studied. However, the interviewees reported that UX was not sufficiently understood by customers and management.
143
development. In add ition, the d evelopment time is lon g. Naturally there is a significant employee turnover, as a large number of people are involved over a period of years. Also, customer contact persons may change. In general, the work is conducted under sequential steps although the project might include some Agile methods especially during the development phase.
were in ver y close cooperation, with an integrated Agile sales and development process. They started idea creation in the very beginning, before making any contracts. According to the customer, the goal was “to find areas that would bring millions of savings or add-on sales.” They were utilizing the approach on small mobile cases, and were convinced that they wanted to continue and further develop the new successful approach. A customer stated that: The best thing was that we achieved exactly what we wanted, in schedule. We achieved incredible results. No one else believed it was possible because we have a long history of heavy processes with long projects (J13).
Typically, requirements are defined into detail before starting development. Due to th e turnover of people involved in the project (and as required by the law in the early phases) heavy documentation is usually preferred. However, public sector customers are also increasingly requesting Agile methods. The extensive design upfront brings problems in utilizing Agile methods later during the development phase. Interviewees reported unclear Agile roles and confusing responsibilities. In some cases, employees had double roles: they might have been project managers and Scrum masters at the same time, or members of several teams with numerous daily Scrums.
Sales Management
When sales is separated from development, the required experts might not be consulted during the sales process; a salesperson might not have adequate understanding of the acquired software. Interviewees reported that i t is a good practice to involve UX person in sales: It is indeed a go od thing that I am involved from the beginning. It is good for the sales; we can get graphical elements in the sales material. UI, we concept apps, lots of ideation: it cou ld be like this, sell this. It is g ood for both the sales and UX (J6).
Customer Requirements
As the general responsibility of UX between the supplier and customer was unclear, UX issues might not have been sufficiently considered. Employees were able to make an individual choice as to wh ether they wanted to think about UX issues in their work or not. Setting UX goals in the beginning of a p roject was very rare; typically, a customer might simply state that the service needed to be usable.
In the Agile sales process, a salesperson and customer representatives work in close c ollaboration, and certain UX practitioners were involved already during sales. In s ome cases, the UX specialist was the first contact for the customer. A p roject manager outlined that in their project, the “UX person acts as a connector between customer and supplier” (J9). Also, te chnical specialists were involved to help to understand the possibilities of the approach.
Usually, a customer sent an offe r request with a list of functionalities without non-functional requirements, such as usability or UX issues (J1, J2, J4): “A customer might have mentioned vaguely in the offer request that the software should be usable, but that’s it” (J4). The software was considered ready when all the functionalities on the list ar e implemented. The supplier and customer might have had divergent opinions about the adequacy of usability or quality of user experience: “Sometimes we have to explain usability defects to the customer, ‘it does not matter if you need to click ten times to get there,’ it is awkward” (J7).
The customer might fix certain issues already during sales process: In many cases we get an offer request from customer containing some requirements and description of UI. But the description of UI is done by some person without awareness in UX , and still it enforces us into certain frame (J2). If a UX specialist was involved during sales, these issues could be better negotiated.
Current Business Model Restricts UX
Project Management
The UX work had a separate price for the customer: that is, when an architect or d eveloper made UI design, it was included in the price, but if the customer wanted UX work, this would cost more. Arguably, if the user needs were not understood initially, a UX specialist designing the UI would not necessarily produce substantial improvements.
As the company is large and has various types of customers, its projects are very diverse. In general, the projects can be divided into customer and in-house projects. Customer projects are further divided into public and private sector projects. Project types range from small mobile applications to large business systems, such as enterprise resource planning (ERP) systems.
Due to the separate pricing model, some of the interviewees felt that making software with good UX would increase costs and therefore they were not alwa ys investing in U X work (J3, J6, J7, J8, J9, J10, J11). Some of the interviewees also believed that making usable software takes more time than making it less usable (J7, J12), and that involving users into development would slow down the process (J6, J14). For these reasons, some projects tried to improve UX afterwards (J9, J12):
Public Sector Projects
Typically, public sector projects are long an d traditionally managed. They are strictly regulated by law and thus contain a lengthy and competitive bidding phase in the beginning, which is followed by a phase where requirements are defined further: thus, it mig ht take two to three years from the first specification to starting
144
If a customer does not want to pay for UX work, we try to collect feedback later and thus remodel the service… But it is a bit difficult if we have made the service and then we tell them that we have an even smarter guy who could improve the service. The customer says ok, but we won’t pay more, you should have made it well at once (J9). Interviewees wanted to change the sales process to bet ter promote UX work (J3, J4, J6, J9 and J17). The present sales process was viewed as an obstacle: When I am (UX person) involved in the sales process, the customer might fully buy the idea ‘it should be done just like this.’ But then our sales person tries to shoot it down for fear of making it too expensive. The business part, how the price is calculated, should be changed so that UX person and sales person would not be against each other in the sales situation (J3). As the customers’ awareness in UX varied markedly, some of the customers demanded UX work, while others would even try to avoid it: “Some customers say we just take the cd and install the system. Users are the last persons we will listen to, they just ruin everything” (J9). In that case, the supplier ‘secretly’ utilized the data they had collected earlier on similar users and systems.
Seven interviewees indicated that in their projects, the UX specialist assists only after UI is already designed and the main ideas and structure of the software already exist (J3, J4, J5, J6, J8, J10, J11). Some respondents (J2, J3, J5, J8) stated that the UI (or at least the first versions of it) is designed by someone other than a UX specialist. Three interviewees (J1, J5, J7) stated that they involve a UX specialist only after experiencing major problems which the project team that they were unable to solve by themselves: When we get some really d ifficult UI th ing, then we ask help from a UX person (J7) We redesigned UI twice because it did not please users… We made the first version of UI by ourselves and tried to fix it. The customer was still unhappy and we had a UX person to redesign it once more and then it was reviewed by users and now it is re-implemented (J5). All Together from Early On
Early collaboration between different stakeholders is important for obtaining a clear overall picture, and in order to be able to notice different needs and limitations sufficiently early. The Agile approach emphasizes collaboration between developers and customer. Inside the supplier company business, UX and technical experts should work together from early on to be able to create an optimal combination of business and user needs that can b e fulfilled by a reasonable technical solution.
The Earlier, the More Influential
The outcome and performance were considered to be better when UX was addressed early in software development (J2, J3, J5, J7). The interviewees felt that influences of not considering UX in the early project phases recurred in later work; hence, effectively considering UX in an it erative process is more affordable than fixing bad outcome (J2, J5, J7, J8). Interviewees also stated that, normally, a UX specialist’s workload decreases as the project proceeds (J6, J7, J10, J17), and this was supported by survey results.
Interviewees named people who should work together at an early stage. Those included: an architect, UX specialist, customer and sales or business representative. When use of existing systems or solutions behind the service is planned, a systems expert should be involved also. Obtaining UX Contribution
In some cases, UX was ignored, either through the early phases or thr oughout the entire project. Some interviewees stated that they had kept talking about UX, but were told: “it is not that important, we can make it later.” One responded stated that: “It is a shame; you cannot add quality on software later.” Another interviewee stated they had discussed their project with a UX specialist, who had said: “Let’s do this well at once. If the UX is bad, we are not allowed to fix it later.” In the l atter instance, they decided to work unofficially to obtain better quality, without telling anyone. On the other hand, one UX specialist indicated that he has been telling to everyone that: “you can get UX from me.” He does not need to advertise himself anymore; people are asking for his contribution early. He is now responsible of coordinating offshore developers’ work and communicating with customers.
Figure 2. Cooperation with UX persons should happen early and constantly, and it should involve different stakeholders. Open-ended question, N=39.
Many projects indicated the desire for a UX specialist’s contribution, but that they had not known there was such a resource available. Also, the majority of UX specialists were extremely busy with their current work and projects needed to be able to predict their needs for UX work many weeks prior if they did not want to wait for UX. The UX
Both the interviewees and survey participants (Figure 2) felt that the contribution of UX specialists should be utilized more and earlier in projects (J2, J3, J4, J5 J8, J11). The current development process was considered to encourage project teams to involve UX specialists too late in projects.
145
the UX de sign for the next sprint, reviews the previous sprint implementation and monitors the current sprint development. They also have a pre-demo session, one-anda-half days before the actual demo, where U X issues are reviewed. If there are any problems, the team still has oneand-a-half days before the actual demo session to fix them.
specialists were seen as a service that could be ordered for projects. The UX professionals adapted the working model (e.g. Scrum) of the current project, but their allocation and work planning were strongly waterfall based. One respondent clarified the practice: “The (development) team decided (when to invite a U X person). We checked our budget if we can afford one and decided ‘now we need to invite someone.’ Then it depended on the company resourcing mechanisms; does anyone have time for us” (J11).
Use of early prototypes was a highly recommended practice among almost all the interviewees. Prototypes were commonly used for communicating the idea to cus tomers, developers and other stakeholders. Detailed UI pictures and wireframes were also used. However, prototype was seen superior for describing user flow and interaction, as it was considered to be the best way to ensure that others understand the idea (J3, J6, J10, J12, J15, J16). In addition, prototypes were used for user testing between implementation sprints (J5, J10).
We are too Busy to Do Good UX
In some projects, UX issues were not fixed because of tight timetable. In one case, the project management ordered cessation of all usability work, because it would take too much time. The UX specialists felt that their work was not properly coordinated when they had to work simultaneously in several projects. When they tried to follow the schedule for each project separately, they felt that this c aused stress and decreased efficiency.
The UX specialist and the main architect should work together in the early phases, to ensure that the design is both implementable and fulfills user needs (J3, J6, J10, J16). Also, close cooperation between the UX specialist and the product owner was considered to be a good practice. Early involvement of all the stakeholders is essential for understanding and communicating the big picture. Cooperation should be continuous throughout the project, although the most important UX work is performed at the early phases (J2, J3, J4, J5, J7, J8, J11, J17). A UX specialist should also be involved during development with a smaller role, to answer developers’ questions and to ensure that UX will be implemented as designed (J3, J5, J16). Developers should consider UX issues in their work, and they can have an impact on the UX of the outcome (J1, J3, J10, J11, J16, J17).
UX Development
Interviewees felt it is d ifficult to p lan and sch edule UX work using Agile. Four interviewees believed UX wo rk does not fit into Agile methods. Two interviewees believed that Agile methods do not fit into the concept creation phase. Two interviewees reported they had not succeeded to perform UX work using Agile. One of those, a UX person, explained the reasons: It is hard to get UX work in an agile way because schedules for UX work do not come from Scrum but from a steering group…The biggest problem (in being involved in Scrum) is that they have not included me in outlining Agile UX work (J3). That is, the projects define when and how UX work should be performed; however, actual UX schedules are imposed from a steering group. On the other hand, some interviewees described that in t heir projects, UX work is conducted successfully using Agile method. They felt that Agile UX work needs to be early-oriented: it should be conducted slightly ahead implementation (J1, J2, J5, J15).
Reusable components and guidelines were seen as useful for decreasing the workload of both developers and UX designers (J2, J3, J6, J9, J10, J17). The more developers were able to do b y themselves, the mo re UX specialists were able to concentrate on the impactful early work. Also, usability studies were, in best cases, planned in a way such that the results could be used also in oth er similar cases. Reusable user testing models were also utilized effectively.
Good Practices in Development
Challenging Issues in Development
In general, conducting UX work in agile manner was considered challenging (J1, J2, J3, J9, J10, J11). In many cases, UX specialists were located within product management or otherwise separated from development. The physical separation of UX practitioners and developers was seen as a major obstacle for Agile UX work (J2, J3, J6, J7, J10, J15, J17). UX pr actitioners switched to each project’s working modes whenever they were invited to one. Usually the team or product owner decided when UX work was needed, which may have led into too late UX work. Sometimes the need for the work w as noticed only when the requirement was immediate and waiting to be implemented (J1, J11). As the UX sp ecialist joined the project from outside, it usually took some time to
Some interviewees indicated that they involve a UX specialist in the early phases of development (J3, J9, J10, J14, J17). They perform user studies before defining user stories. One UX specialist stated that they first interview users, then a UX specialist draws initial sketches of the UX design and the deve lopment team (together with product owner) creates a product backlog based on those sketches. Meanwhile, the UX specialist defines the UX design. The team creates sprint backlog and informs the UX designer as to what they are going to implement on next sprint. The UX designer works one s print ahead, finishing the design for the next sprint. Together with the team, they review the design. As the team implements, the UX designer refines
146
understand the project requirements. Usually, UX specialists were busy with other projects and UX work on a specific project was delayed; if th e project was no t able to wait for UX work, they had to cope by themselves. In many cases, the UX specialist was included in a project only if the team or product owner considered that separate U X work was needed, the project had money for it and there was a UX specialist available to do the work (J1, J11).
The sales process should be changed to promote UX work. The current business model highlights the costs UX work causes. Therefore, it is t empting for the customer to ‘save’ in project costs by cutting down the UX portion; the consequences are not visible to th e customer but the price is. The impact of UX work should be shown to the customer more clearly, whereas the costs could be included within other development costs.
Some interviewees complained about unclear roles and responsibilities in projects. It was esp ecially challenging in longer public sector projects, where staffing changed, the early phases were co nducted with wat erfall approach and development was done using Agile methods.
Rajanen et al. [19] state that in software development, costs of UX work are seen immediately whereas the benefits are intangible. This was o bserved in our inte rviews. For example, UI design costs were seen additional when UX specialists were d esigning the UI , and w ere included in development costs when someone else made the de sign work. The company had also made the UX costs clearly visible to customers.
There was no clarity as to when, and on which issues, a UX specialist should be contacted, or who can make the contact. Amongst both the interviewees and survey respondents, there were people who did not know there was a UX team available. There has been clear effort to communicate UX issues inside the company, but as the company is large, it is apparently difficult to reach everyone.
Projects needed to pay for UX when they utilize the resource and this had a substantial impact on the willingness to cooperate with UX specialists. This finding is similar to that of Rohn [20]. In addition, UX work was excluded from process models, which caused separated and late UX work in many cases.
DISCUSSION
After the study presented in this paper, several workshops and online presentations have been conducted inside the company to raise awareness and create new practices. On the corporate level, Agile coaches will be t rained to spread UX as they are conducting the Agile adoption within the company. In addition, on the project level, two pilot projects are ongoing where certain practices (such as close cooperation between UX designer and product owner, including UX designer in decision-making, and regular user feedback cycles) are tested. The effects of those practices are being measured within follow-up research.
As the case company is large, it has various approaches within management, sales and d evelopment. At th e best, UX was seen as an important business asset and was included from sales to deployment. However, in many cases the role of UX specialist was u nclear and it was excluded from process descriptions. Interested individuals were working towards UX, whenever they had a change to do it. In the case company, the prevailing conception about UX seemed to be that performing effective UX increases project costs. In general, more emphasis was placed on cost considerations than on satisfying the customer. Projects were fulfilling customer’s business needs: but if t he customer did not provide separate UX goals, the supplier usually did not take the responsibility to deliver software with good UX. The softwa re was c onsidered ready when the customer’s list of functional requirements was implemented. In instances when both the customer and the supplier fail to cla rify user needs, the resulting software may be unsatisfactory: although the so ftware fulfills all the requirements set in th e beginning, the customer could be unhappy after taking it in service and finding major problems in terms of usability.
SUMMARY AND CONCLUSIONS
This paper presented the results of a case study about UX work in Agile software development from various management perspectives. Respondents working on both operational and strategic levels considered UX to be an important business asset. However, in certain organizations, processes and management style limited the impact of UX work. In particular, the business model in sales was seen to be hindering UX work. As the costs of UX work were separated from other project costs, it was easy for a customer to exclude UX work to cut costs. In some cases this led into customer disappointment, when negative user experiences were encountered only after the software had been taken in use in the customer organization.
No related rese arch about the role o f UX specialists in Agile sales could be identified in the existing literature. This paper therefore contributes some insights into the topic.
The company had an established UX team, however, in general, UX work was not inc luded in development processes. Decisions as to when and how UX work was needed were not being made by UX specialists, and this led to the consideration of UX issues occurring too late in the process, and to inefficient utilization of the UX resource.
Certain restrictions were found to be set already during the sales process, and therefore it is beneficial to include a UX specialist in sales activities. Increasing customers’ awareness on UX d uring the sales process may also increase the UX demand.
147
14. Jain, J., Courage, C., Innes, J., Churchill, E., Lund, A., and Rosenberg, D. Managing Global UX Teams. In Proc. of CHI EA (2011).
ACKNOWLEDGMENTS
First of all, our special thanks go to the company contact person for the dedicated commitment to the study process. We want to thank Tommi Mikkonen for his valuable comments and support, Santtu Pakarinen for being the main assistant during the research, and Jani Heikkin en, MarieElise Kontro, Jarmo Palviainen, and Thomas Olsson for assisting in the analysis with the affinity wall. We are also grateful to the Finnish Funding Agency for Technology and Innovation (TEKES) for the financial support.
15. Lazar, J., Feng, H.F., and Hochheiser, H. Research Methods in Human-Computer Interaction. John Wiley and Sons (2010). 16. Lund, A.M. Creating a user-centered development culture. Interactions 17, 3 (2010), 34–38. 17. Marcus, A., Ashley, J., Knapheide, C., Lund, A., Rosenberg, D. and Vredenburg, K. A survey of userexperience development at enterprise software companies. LNCS 5619 (2009), 601 –610. 18. Norman, D. A. Whose profession is this? Everybody’s, nobody’s. Interactions, May+June 2005, 51.
REFERENCES
1. Beyer, H., and Holtzblatt, K. Contextual Design: Defining Customer-Centered Systems. Morgan Kaufmann (1998). 2. Bias, R.G.; Mayhew, D.J. Cost-Justifying Usability: An Update for the Internet Age. Morgan Kaufmann, 2005. 3. Boivie, I., Gulliksen, J., and Göransson, B. The lonesome cowboy: A study of the usability designer role in systems development. IWC 18, 4, (2006), 601–634.
19. Rajanen M. and Iivari N. Usability Cost-Benefit Analysis: How Usability Became a Curse Word? In proc. of INTERACT 2007. LNCS: 4663 (2007). 20. Rohn, J. A. How to organizationally embed UX in your company. Interactions 14, 3 (May 2007), 25–28.
4. Botzenhardt, A., Meth, H., and Maedche, A. CrossFunctional Integration of Product Management and Product Design in Application Software Development: Exploration of Success Factors. Proc. ICIS (2011).
21. Rosenberg, D. Introducing the 360° view of UX management. Interactions 14, 3 (May 2007), 22–24. 22. Runeson, P. and Höst, M. Guidelines for conducting and reporting case study research in software engineering. Empir. Software Engineering 14 (2009), 131–164
5. Budwig, M.; Jeong, S.; Kelkar, K. When User Experience Met Agile: A case study. In Proc. of CHI EA 2009. ACM (2009), 3075–3084.
23. Sutherland, J. and Schwaber, K. The Scrum papers: Nuts, bolts, and origins of an Agile framework (2011). Available .
6. Bygstad, B., Ghinea, G., and Brevik, E. Software development methods and usability: Perspectives from a survey in the software industry in Norway. Interacting with Computers 20, 3 (2008), 375–385.
24. Turner, C.W. A strategic approach to metrics for user experience designers. JUS, 6, 2 (2011).
7. Chamberlain, S., Sharp, H., and Maiden, N. Towards a framework for integrating Agile development and usercentred design. LNCS 4044 (2006), 143–153.
25. Van Solingen, R.; Sutherland, J.; de Waard, D. Scrum in sales: How t o improve account management and sales processes. In proc. AGILE conference (2011), 284-288
8. Chow, T., and Cao, D. B. A survey study of critical success factors in Agile software projects. Journal of Systems and Software, 81-I, 6, (2008), 961–971.
26. Vaughan M. W. Tough Sell: Selling User Experience. Journal of Usability Studies (JUS), 6, 2 (2011). 27. Venturi, G. and Troost, J. Survey on user centred design integration in the industry. In proc. of the third NordiCHI. (2004).
9. Earthy, J. Usability maturity model: Processes (1999).
28. Venturi, G., Troost, J. and Jokela, T. People, organizations, and processes: An inquiry into the adoption of user-centered design in industry. In International Journal of Human Computer Interaction 21, 2 (2006), 219–238.
10. Federoff, M. and Courage, C. Successful user experience in an Agile enterprise environment. LNCS 5617 (2009), 233–242. 11. Hussain, Z., Slany, W. and Holzinger, A. Current state of Agile user-centered design: A survey. LNCS 5889, 416–427. 12. Innes, J. Why Enterprises Can’t Innovate: Helping Companies Learn Design Thinking. In HCII 2011, LNCS: 6769, 442–448. Springer Berlin / Heidelberg
29. Vredenburg, K., Mao, J., Smith, P. W., and C arey, T. A survey of user-centered design practice. In Proc. CHI 2002. ACM (2002), 471–478.
30.Zhou, R., Huang, S., Qin, X. and Huang, J. A survey of
13. Ergonomics of human-system interaction. Part 210: Human-centered design for interactive systems. ISO 9241–210 (2010).
user-centered design practice in China. In IEEE international conference on systems, man and cybernetics, SMC (2008), 1885–1889.
148
