Abstract Title - Semantic Scholar

Third International Seville Seminar on Future-Oriented Technology Analysis: Impacts and implications for policy and decision-making – SEVILLE 16-17 OCTOBER 2008

USER INVOLVEMENT IN LIVING LAB RESEARCH: EXPERIENCES FROM AN INTERDISCIPLINARY STUDY ON FUTURE MOBILE APPLICATIONS Katrien De Moor1, Katrien Berte1, Lieven De Marez1, Wout Joseph2, Tom Deryckere2, Luc Martens2 (1)

MICT-IBBT, Dept. Of Communication Sciences, Ghent University; [email protected], [email protected], [email protected] (2) WiCa-IBBT, Dept. Of Information Technology, Ghent University; [email protected], [email protected], [email protected]

Summary Open and user-driven innovation strategies seem to have influenced the shift beyond the ‘technology push’ perspective, which has dominated for a few decades and which minimizes the importance of social and user-related dimensions in the context of ICT development, adoption and diffusion. Although ‘the consumer’ has always been important to a certain degree, companies are now even more forced to put users (as key stakeholders) and customerunderstanding at the core of their innovation strategies in order to sustain a strong market position and to pursue product development that doesn’t neglect the user. As a result, the traditional user research arsenal has also been extended with alternative analytical methods and tools from various disciplines (such as design, foresight, anthropology, ...). These methods tend to focus on people’s behavior, future needs, emotions, ... (Sleeswijk Visser, Stappers et al., 2005, p. 123) in view of the collection of those insights that are rather implicit and tacit. Building on the above mentioned shift, this paper reflects on the implications of the new innovation context for scholars and practitioners. In this respect, two important challenges are identified: the need for a continuous interaction with the user and the need for mechanisms to integrate and transform the knowledge that is gathered in the increasingly multidisciplinary development process. Complementary to the theoretical framework, we critically reflect on our own experiences from an interdisciplinary research project in a living lab setting. This paper shares findings from the ROMAS-project (Research on Mobile Applications and Services), which took place in the living lab setting of i-City Hasselt and which is co-funded by IBBT 1 . For the purpose of this paper, we focus on ‘user involvement’ in three particular phases of the NPDphases, i.e. opportunity identification (1), concept evaluation (2) and test market (3). The first and second phase comprise a rather general market exploration (N=312) of both new and existing mobile applications based on a wide scan of ‘wild ideas’ from users. During the phase of test market, a new methodology for linking the user experience of mobile applications to QoS (in a living lab setting), was developed.

1

The Interdisciplinary Institute for BroadBand Technology was founded by the Flemish Government in 2004 to stimulate ICT innovation. More on www.ibbt.be

THEME: FTA IN RESEARCH AND INNOVATION -1-


1 Context and introduction The industry developing information and communication technologies (ICTs) has been challenged in various ways over the last decades. Due to the far reaching convergence in the domains of communication, consumer electronics, computing and content on the one hand (Yovanof & Hazapis, 2008, p. 570) and the hyper competition and increased market liberalization on the other hand, companies that aim to take in or sustain a leading market position in the ICT industry, have increasingly been forced into rapid technological advancements and innovation. Although largely aimed at the differentiation from other market players, this situation has influenced new product development to a large extent. The shortening of product life cycles has necessitated accelerated product development and even the skipping of important research stadia. As a result, there has been an explosion of non-disruptive innovations, which are not always clearly distinct from other products at the market (e.g. at the level of features) (De Marez, 2006; Yovanof & Hazapis, 2008). Furthermore, traditional product development strategies are said to have crucial shortcomings since they are no longer able to guarantee the successful adoption and diffusion of new ICTs. Although innovation traditionally is considered as a rather linear, research- and price-driven process, this focus seems to have shifted over the years (Rosted, 2005, p. 11), influenced by the altered role of one of the most important stakeholders in the innovation process, namely the user/consumer. Confronted with almost unlimited choices and a variety of alternatives, his demands seem to have become more ‘sophisticated’ and distinct. Consumers increasingly tend to seek out those products, services and experiences that fit their personal and situational needs. Consequently, a clear insight in users’ needs and experiences has become indispensable (De Marez, 2006; Veryzer & Borja de Mozota, 2005). Indeed, although ‘the consumer’ has always been important to a large extent, the rationale of interacting with and involving the user has drastically changed. The new innovation context has urged companies to put user needs and customer-understanding at the core of their innovation strategies, in a more systematic and structured way. This allows companies to build or sustain a strong market position and to pursue product development, which meets customers’ demands better than before (Magnus et al., p. 6). Many authors have explored this shift from traditional ‘technology push’ to more pull and user-driven approaches. Whereas the former are characterized by technology-centered strategies and limited user involvement, the latter acknowledge the crucial role of the user and emphasise the need of far reaching interaction with users (Rickards, 2003, p. 1095, Trott, 2003, p. 836). Given the many ‘do it yourself’ possibilities in the Web2.0 sphere and beyond, many authors have explored the shift from passive consumption to active and dynamic (co-)production or ‘produsage’ (Bruns, 2005; Toffler and Tromp, 1981). Increasingly, users seem to be able to initiate innovation from a ‘pull’-perspective. Undoubtedly, some of the current approaches have been influenced by advocates of user-driven innovation such as Von Hippel (1986, 2005). In this context we can also refer to policy actions to support user-driven innovation such as the rise of living labs 2 , which can be regarded as ‘user-driven innovation environments’. Many other policy

2

In 2006, The European Network of Living Labs (ENoLL) was launched under the Finnish EU presidency.



initiatives are embedded in this new innovation context, e.g. the allocation of overwhelming budgets both at national and supranational levels in order to support open and demand-driven innovation. Despite these initiatives however, the context in which new product development takes place, remains challenging (Veryzer and Borja De Mozota, 2005, p. 133). Although crossdisciplinary research is considered to be crucial in this respect, it still remains difficult to create a meaningful synergy between users and technology in the field of ICT development. This paper therefore aims to discuss the integration challenges that continue to present themselves in a user-centered innovation context. It is organised as follows: section two expands on a number of theoretical contributions dedicated to the relation between technology and societal factors, and previous research in the field of user-driven innovation. Further, this paper explores the implications for traditional innovation and development processes. Given these implications, section three identifies two important challenges for scholars and practitioners, from a user-driven innovation perspective: the need for adequate user involvement and a continuous interaction with the user (which requires methodological renewal) and the need for mechanisms to integrate the knowledge that was gathered in the multidisciplinary process. As a complement to the literature, we draw on our own experiences in the ROMAS 3 project - funded by the ‘Interdisciplinary Institute for BroadBand Technology’ (IBBT) 4 - to illustrate how both challenges can be tackled. To conclude, the last section summarizes our findings.

2 Theoretical perspectives The relationship between technology and society has already been studied from various perspectives. The idea of ‘technological determinism’, which considers technology as the prime mover of transformations, and which propagates the industry ‘push’ perspective, has dominated the theoretical debate for a few decades. It largely ascribes changes in society to technological advancements, which are thus assumed to have important social consequences (Haddon et al., 2005). This theory of ‘technological determinism’ fits in the ‘diffusion of innovations’ framework (Rogers, 1995), dedicated to the introduction and adoption of new technologies in society. Technology adoption is assumed to follow a predictable path and is considered to be influenced by change agents (e.g. private firms, influential individuals, …). In the theory of diffusionism, the first group of people who adopt the new technology (innovators and early adopters) are seen as katalysts for the successive adoption waves. The final aim is to reach the rest of the market up to the point at which the adoption rate has become so high that the innovation can be considered successful (this is referred to as the ‘critical mass’) (Rogers, 1995). A common critic on the diffusion theory has to do with this notion of technological determinism and the assumed linearity of the innovation and adoption process. This is also referred to as the ‘pro-innovation’ bias of the diffusionism theory (Rogers, 1995). Despite the criticism however, in practice, the Sshaped diffusion curve still tends to be pushed forward from a technology-push perspective (Haddon and Paul, 2001, p. 202).

3

ROMAS stands for Research on Mobile Applications and Services (cfr. Infra).

4

IBBT was founded in 2004 by the Flemish Government in order to stimulate innovation in the field of information & communication technology. Flanders is the Northern part of Belgium, home to the Dutch speaking community.



Nevertheless, from the 1960s on, this industry-push perspective was challenged by more human-centered paradigms that largely reject this technological determinism. Indeed, in practice it also became clear that the diffusionism idea did not always apply (anymore) for certain sectors, such as the ICT sector. Not surprisingly, more and more authors pointed to the deviation of adoption curves from the traditional Rogers’ curve. In this respect it is relevant to refer to the ‘Social Shaping of Technology’ (SST) framework, which focuses on the consumption and daily use of technology and which stresses the role and power of human actors and societal forces (Lievrouw, 2006). This social constructivist vision does not intend to ignore the importance of technological advancement. Rather, it aims to procure that technology development becomes more user- and human centered (Williams and Edge, 1996). Or put differently: ‘SST advocates intervention in the development of technologies before they are ‘pushed’ into society’ (Lievrouw, 2006, p. 252). Closely related to the social shaping-perspective is the ‘Social Construction of Technology’ approach (Bijker & Law, 1992), in which the concept of ‘interpretative flexibility’ is used to refer to the choices of and differences among individuals and social groups when it comes to giving meaning to and understanding the role and purpose of technology development (Haddon et al., 2005, Lievrouw, 2006). Within the SCOT perspective, it is assumed that the negotiation between ‘relevant social groups’ influences the construction and emergence of new technologies (Bijker & Law, 1992; Haddon et al., 2005). Although both abovementioned approaches tend to emphasize the interaction between technological and societal forces, they have been criticized for their rather linear social determinism. In this respect, a number of theories have attempted to focus on the shaping of technology in a less linear way: the actor-network theory (Latour, 1993) states that both technologies and people are part of ‘sociotechnical networks’, which influence the shaping, forms and uses of (new) technologies. Rather thus than seeking for an explanation in either a technology ‘push’ or market ‘pull’ vision, such approaches try to focus on technology development from a ‘mutual shaping’ perspective (Lievrouw, 2006). This ‘mutual shaping’ or ‘interactionism’ perspective provides us with a theoretical basis for uniting both the technologycentered and user- or human-centered vision: it explains the success of the adoption and diffusion of technology by the continuous interaction between technological and societal forces (Rickards, 2003, p. 1095, Trott, 2003, p. 836; Boczkowski, 2004, p. 255). As was already mentioned above, this traditional tension between technology-centered and user-centered approaches still entails a number of challenges for research in the communications industry. In this respect, we would like to go back to the changed innovation context and more specifically the notion of ‘user-led’ or ‘user-driven’ innovation. We have seen that user-driven innovation is more and more considered to be a necessity for successful innovation development. Some of its essential characteristics are the ‘strategic focus on consumer pull’ and ‘more direct involvement of the user’ (Hansson, 2006, p. 12). In current definitions, ‘user-driven innovation’ refers to the process of collecting a particular type of information on the user: it deals with those insights that are difficult to grasp (Rosted, 2006, p. 22). In order to uncover users’ needs both at an acknowledged, observable level as well as at a latent and rather tacit level, the involvement of competencies and insights from different disciplines is required.



Several approaches have been pushed forward in order to collect this type of knowlegde. Hansson (2006, p. 13) distinguishes two types of user-driven innovation methods: ‘Voice of the Customer’ methods and lead-user methods. Undoubtedly, Eric Von Hippel’s work on ‘lead users’ can be regarded as pioneer work in this respect. Von Hippel (1986, p. 791) states that a specific group of users, namely ‘lead users’ can serve as a kind of ‘need-forecasting laboratory for marketing research’. Furthermore, the traditional user research arsenal (including methods such as focus groups, surveys, etc.) has been extended with alternative analytical methods and tools 5 from various disciplines 6 in order to support user-driven innovation. Whereas the so-called ‘traditional’ methods usually focus on what people say and think, methods from other disciplines are now used to ‘dig deeper’ in what people do or want (e.g. ethnographic research, observations, user toolkits, …), feel or dream (e.g. generative methods) (Sleeswijk Visser, Stappers et al., 2005, p . 123). The rise of living labs, i.e. city-based innovation areas, which provide full-scale test bed possibilities for inventing, prototyping, interactive testing and marketing of (new) mobile technology applications (Experientia, 2006), should also be mentioned in this context. Contrary to other test beds, they provide a more ‘natural’ testing environment and strongly encourage continuous and meaningful interaction between developers/suppliers and users. Although one could question whether these ‘natural settings’ provided by living labs, truly reflect the way endusers use and interact with technology in their daily lives, they are usually regarded as environments that provide possibilities for advanced and strong user involvement. When now turning to the differences between some of the abovementioned ‘new’ and ‘traditional’ practices at a more generic level, the latter are characterized by a ‘technology driven focus, limited multidisciplinary cooperation and development prior to user validation’ (Vredenburg, Isensee et al., 2002, p. 2). User-oriented approaches however, tend to be userdriven and increasingly multidisciplinary. Furthermore, they focus on current and future users/experiences instead of only looking at current users/uses (Vredenburg, Isensee et al., 2002, p. 2). Although the importance of such a systematic multidisciplinary approach is emphasized in the literature, the current initiatives to broaden NPD processes also entail new complexities: ‘disparate disciplines, each with their own emphasis and sense of ‘product direction’, have overlapping and sometimes conflicting areas of concern with respect to a product being developed’ (Veryzer & Borja de Mozota, 2005, p. 133). This important ‘knowledge integration’- challenge is largely ascribed to the abovementioned ‘clash’ between technologydriven and user-centered approaches. Given their relevance for research and development in the user-driven innovation context, two important challenges are discussed in the following section.

3 Integration challenges In this section, we aim to focus on two particular challenges regarding the shift towards more user-driven approaches, which was discussed more comprehensively above. Although not new, these challenges have shown to be still very relevant today.

5

Such as personas, archetypes, contextual inquiry, sensitizing packages, proxy technology assessment, ...

6

Such as design, sociology, foresight, FTA, anthropology,...



The first challenge has to do with the need for continuous and adequate involvement of the user. Several scholars have focused on the fact that there are still rather few companies that effectively involve the customer or user in the innovation process (Alam, 2002; Kristensson et al., 2004). Kristensson et al. (2004, pp. 4-5) attribute this discrepancy between theory and practice mainly to the lack of empirical ‘evidence’ on the benefits of user involvement and useroriented strategies compared to traditional R&D. Although research has indicated that if NPD projects fail, it usually went wrong from the beginning (Khurana & Rosenthal, 1998), user involvement is too often limited to one single stage (e.g. usability testing) only to the final stages of the process (e.g. evaluating) (Haddon et al., 2005, p. 10; Mulder and Steen, 2005). However, the benefits of involving users closely and continuously have already been investigated: it has been shown that users in close interaction with a particular company, can come up with ideas for future products that are perceived as ‘unique’ and ‘valuable’ (Kristensson, et al., 2004, p. 4). User involvement is therefore assumed to influence the successful adoption and diffusion (Limonard and de Koning, 2005, p. 176). User-driven innovation should thus go beyond merely asking users for feedback after the piloting phase or launch. Instead, users should be involved from the early, more generative phases until the post-launch evaluation phase. Furthermore, user-driven innovation deals with those user insights (needs, expectations, …) that users cannot always easily articulate. This has forced development teams to explore new and interdisciplinary methodological tools. In the empirical part of this paper, it is suggested how this challenge was countered. The second challenge refers to the problem of integrating the knowledge that is being gathered by multidisciplinary teams, and coming from either user- or technology-centered methodologies. This transfer of knowledge in an interdisciplinary team often remains difficult. The phase of user research is often considered by engineers as ‘merely an intermediary stage, a task that had to be carried out in order to start the ‘real’ work’ Limonard and de Koning (2005, p. 174). Nevertheless, it is crucial that the generated ‘user insights’ find their way into the development process so that the technical partners can take them into account. In this respect, Veryzer and Borja De Mozota (2005, p. 141) refer to the need of ‘tools for better integrating the disparate disciplines and the unique perspectives each offers’. Indeed, the adequate translation and transformation of user insights and requirements in more technical requirements (and vice versa), is considered to be an important challenge: knowledge on both explicit and implicit users’ needs, expectations, wants… needs to be passed on to the development team in such a way that it can be successfully incorporated in the development process. The notion of ‘translators’ is used in this context (Veryzer and Borja De Mozota, 2005, p. 140). In this respect it is relevant to mention the prevailing gap between Quality of Experience and Quality of Service, two important concepts in the field of ICT development. The former deals with the all dimensions and factors influencing users’ experience with technology (e.g. usage context, personal context, expectations…), the latter represents the technical parameters and performance metrics such as latency, jitter, packet loss… that influence users’ experiences. The sharing of insights on users’ experiences and expectations (in a particular context or for a particular application) with engineers still appears to be a missing link. To this end, a new interdisciplinary methodology for correlating user experience to QoS parameters in a living lab environment is introduced further in this paper.

4 General methodology



4.1 Project description and research setting The empirical findings described here illustrate how it was attempted to overcome the abovementioned challenges. These findings are part of the interdisciplinary ‘Research on Mobile Applications and Services (ROMAS)’-project 7 , in which we aimed to conduct a user-oriented assessment of (future) wireless city applications in a large-scale living lab environment. These technological applications were put to the test of social value, market relevance, legal conditions, usability requirements as well as quality of experience performance. This research took place in the living lab setting of i-City Hasselt 8 , a wireless city environment that offers unique possibilities for this extended user research. Using technologies such as WIFI, Bluetooth, GPRS, Edge, Mesh, UMTS, HSDP, Wimax etc., i-City offers several wireless applications for PDA’s, portable computers, smart phones etc. to a large panel of test users and is therefore the largest living lab in the world for testing mobile applications in real life situations 9 . The value of innovation potential research is usually highly dependent on the consumer’s comprehension of the new concept. In the case of ROMAS, all test users voluntarily joined the iCity research panel and can therefore be considered as committed to and familiar with the ‘mobile city’ concept. At the moment of the research, 450 consumers 10 were members of the iCity user panel. Although non-representative for a larger Flemish or Belgian population 11 , the explorative nature of the project and open access to the panel, justified the choice for this research setting.

5 Results This section zooms in on three distinct moments of user involvement within the ROMAS-project in order to illustrate how the challenge of adequately involving the users throughout the whole process, was tackled. More specifically, we aim to show how a living lab setting can be complemented with other research methods in order to gain the necessary user insights.

5.1 Phase 1: identification of opportunities The first phase, the identification of mobile opportunities, started with a wide scan of possibly interesting mobile applications for a wireless city environment. The purpose of this scan was to generate input in order to identify current and future mobile applications which could make significant differences to consumer’s everyday lives and generate revenues for technology 7 8

This project was co-funded by IBBT (cfr. Supra). Hasselt is the capital of the Belgian province, Limburg.

9

Several work groups cover a large number of topics like health care, mobility, tourism, culture and heritage, logistics, education, e-government, food and retail. The i-City mobile platform is supported by the Flemish government and several industry partners such as Microsoft, Telenet, Siemens, Concentra, Fujitsu-Siemens Computers and Research Campus Hasselt (i-City, n.d.). 10

In February 2008, the i-City panel consisted of 750 members.

11

it is ‘more than average’ interested in mobile technologies, dominantly male etc ….



providers. One of the major challenges in this phase was not only to successfully involve the users in this early stage of the process, but also to overcome the limited capacity of the user to imagine future technological opportunities. Firstly, extensive desk research was conducted in order to list existing mobile applications and new concepts developed by the mobile industry. This inventory was used as background information for the familiarization of the researchers with the possibilities of mobile technologies. Secondly, in order to generate some new (wild) ideas for future mobile city services, users were involved In two focus groups. The first group consisted of members of the i-City panel, all familiar with advanced mobile applications and their use in a city environment. The second group consisted of regular consumers, only familiar with the basic applications of the traditional cell phones. Unfortunately, an often recurring issue in user research is the lack of imagination to break loose from the existing reference framework and truly reflect on future needs and applications. Users often keep referring to familiar technologies like MMS, making phone calls etc. and find it difficult to empathize with other users’ lifestyles e.g. a 25 year old reflects only on his daily activities and has difficulties to imagine oneself in the life of an elderly person. In order to eliminate these shortcomings, the framework of time spending was used in the focus groups. Research on time spending was conducted and eight large categories of time spending during the day were identified. The categories were ‘social participation’, ‘household activities’, ‘study’, ‘work’, ‘transportation’, ‘leisure’, ‘health’ and ‘sleeping & resting/relaxing’. As a complement the ‘traditional’ social science method of focus groups, three archetypes of users were developed to help the users empathize with other lifestyles. Working with archetypes is a alternative way of conducting user research, inspired by design practices (cfr. supra). An example of one of the archetypes is Patricia (see table 1). She is 40 years old, manager in a major international firm, mother of two children and therefore always trying to balance work and family life. For each archetype, we listed a series of daily activities within the time spending framework. Table 1: Archetype Patricia and some of her daily activities



Participants in the focus groups were asked to describe their daily activities for different times of the day using questions like ‘what do you normally do on a work day between 7 and 9 AM?’ with simple answers like ‘Take the kids to school, have breakfast, drive to work, take the bus to the university etc.’ The participants were asked to think about how mobile technologies could facilitate these activities. The archetypes were used to reflect on activities of people with other lifestyles. During the brainstorm, participants envisioned to be in the year 2010 and therefore, were not restricted by the current legislation and technological limitations. 47 ‘wild ideas’ were generated in these sessions, all original and very usable for the next phases in the research project. When combining the wild user ideas with the results from the desk research, a list of 80 mobile applications was created. The list was preliminary divided into 8 categories based on the time spending research. Although the full list of 80 applications (table 2) is too long to be discussed in detail in this paper, it still serves as input for the composition of attractive and successful application clusters (phase 2).

Table 2: The final list of 80 (future) mobile applications



Given the results that were yielded in this first moment of user involvement, the logical step consisted of a feedback moment with the supply-side, i.e. the content and service providers that can also be considered as ‘professional’ users of the mobile applications and that can therefore also provide valuable input for the technological developments). Potential service providers were contacted and probed for their interest in the use of mobile application to support their existing products and services. Finally, the new ideas were mapped on the daily activities of the three archetypes with an indication of the origin and status of the mobile application (table 3). Applications described by a red color are applications which are user-generated applications which the industry is not currently developing. Green applications were suggested by the focus groups but are already being implemented by the industry. White applications are existing applications which are being developed or are already commercialized by the mobile industry but were not suggested by the users in our focus groups (push driven). Table 3: Integration of research results for archetype Patricia

THEME: FTA IN RESEARCH AND INNOVATION - 10 -


5.2 Phase 2: concept evaluation Next, all applications considered in the first phase were transformed into workable paper concepts and presented to a large audience in order to evaluate the adoption potential of these applications and identify interesting market segments. For this ‘concept evaluation’ phase, we conducted a large survey within the i-City panel (n=420). The advantage of a concept evaluation within this panel is their familiarity with mobile city concepts and experience with actual working applications. Firstly, we tried to cluster the 80 applications and/or ideas in an attempt to summarize the long list of applications/ideas in some clearly distinguishable ‘application clusters’. The criterion for this application clustering are the correlations and similarities in ‘interest patterns’ for certain subsets of applications. The difference in interest for each of these ‘application clusters’ can be considered as a first exploration of the potential of different kinds of applications. Secondly, the application clusters were ranked in order to identify the most promising application(s) (clusters). Factor analysis on the interests of the 312 respondents in the 80 ‘mobile city application(s)’ by means of Principal Component Analysis learned that these interests can be summarized in 21 factors, still explaining 67,5 % of the total variance (R²= 0,67482). In order to use these factors for further analysis, we first needed to investigate the reliability of each factor using Cronbach’s Alpha. Thirteen clusters were discovered using this methodology (see table 4). Since each of the clusters represents a set of applications for which there appeared to be strong correlations in (dis)interest among the 312 respondents, they can also be considered as a ‘potential added value domain’ for mobile city applications for a certain part of the market. 16 applications could not be clustered and were analysed separately. Clusters and single applications were ranked based on the interest level of the respondents. The overall average interest ranking for all the clustered learned that the most important innovation of these mobile THEME: FTA IN RESEARCH AND INNOVATION - 11 -


applications are not the most high tech innovations but applications enabling to save time and ensuring and improving participants’ quality of life 12 . Among the applications for which there is hardly any interest, we notice some remarkable findings. Despite the popularity of the ‘virtual social contacts’ on the web, the cluster of ‘mobile social contacts and friends’ is certainly not appealing to the majority of the population (2.94/5). Despite the high rankings of ‘news’ in ‘most wanted content rankings’, the mobile news cluster had only an average interest of 3.11/5. Also for ‘sports on mobile’ (2.74/5) there does not seem to be great enthusiasm either. A possible explanation for this may be found in the vague description of the application ideas.

Table 4: The ranking of application clusters and separate applications based on interest level of the respondents

12

For a detailed discussion of the clusters and results, please see Berte, De Moor et al. (2008).



Further and deeper analysis is certainly necessary to make definite conclusions about the appeal of each of these applications. In addition, these correlations between interests in the different kinds of applications were also used to cluster participants.

5.3 Phase 3: test market For the third moment of user involvement, we take a look at the ‘test market’ phase. One application, i.e. mobile news, was selected from the list by the project industrial partners as being the most interesting application for the development in a business environment. This was a rather surprising choice, since mobile news was generally assessed as not appealing to the users 13 . In this respect, the results from the preceding user research, were disregarded. This choice illustrates that sometimes decisions are made at the expense of the user-centered rationale. Particularly in this phase, other factors 14 were considered to be of greater importance

13

Among the highest ranked were applications such as ‘Indication of parking space and availability’, ‘public transport schedules’ etc. 14

i.e. high involvement of the inhabitants of Hasselt in the local community, presence of existing local news initiatives and of a community of city reporters and one of the industrial partners being a local content provider (Concentra) with a proper local news TV channel (TV Limburg) expressing the need for a ‘cross-media’ approach



for the decision of the project’s industrial partners, than the knowledge that was yielded by user research. During the same phase, the Quality of Experience 15 of the Wapedia 16 application was investigated in a controlled setting. In the field of ICT innovation research, we cannot ignore the growing importance of QoE: it is assumed that a good ‘user experience’ is crucial for adoption and loyalty purposes (Crisler et al., 2004, p. 61; Jain, 2004, p. 96-97) and that experiences are a ‘new source of value’ (Pine and Gilmore, 1999, p. 2). Corrie et al. (2003, p. 2) emphasize the importance of users’ expectations and experiences: ‘QoE is how the user feels about how an application or service was delivered, relative to their expectations and requirements’. Whereas the concept of Quality of Service 17 (referring to technical and performance parameters) received a lot of attention in the past, it seems that QoE has now taken over this role, driven by the shift from ‘push’ to ‘pull’. De Marez and De Moor (2007) looked into QoE at a conceptual level and identified five main dimensions and over 70 sub-dimensions 18 : Given the various factors influencing users’ experiences with technology, the adequate measurement and translation of these expectations and experiences, remains challenging. Although crucial for enhancing users’ QoE, sharing findings from user research with engineers and developers appears to be a missing link. To this end, a 5-step interdisciplinary methodology for linking QoE to QoS parameters in living lab environments was developed 19 . This approach draws on ‘hard’ technical parameters as well as on more subjective (social, contextual etc) dimensions and their ‘translation’. It was tested in a small-scale study (N=10) 20 for the Wapedia application. We do not aim to elaborate on results of this case-study here. Rather we intend to show how the above mentioned challenges were tackled in this research 21 . We now briefly turn to the five phases. 1. Pre-usage user research i.e., to detect “most relevant QoE dimensions” and users’ expectations. This phase included a semi-qualitative group session per two participants, who were asked to reflect on factors influencing their experience with and expectations for mobile phone use. Methods used here included free listing, questionnaire, brainstorm, QoEdimensions list, prioritizing exercise and conjoint analysis. This technique is used to determine which product features or attributes are considered to be most important, when a set of attributes or features are offered. It allows us to analyze the preferences of the respondents. In our study, a set of 6 QoE-dimensions was offered, resulting in a total of 15

15

Also referred to as: QoE, user experience, likeability, ...

16

This application is available on the i-City test devices. It is a Wikipedia application for mobile access.

17

Also referred to as QoS

18

‘Quality of Effectiveness’ refers to the accuracy and technological performance, at four levels (application/service, server, network, device/handset), ‘Quality of Efficiency’ deals with the question: does the application, device… work well enough for the user? ‘Usability’ refers to ease of use and of accomplishing tasks. ‘Expectations’ refers to the subjective character of the ‘experience’ concept. The degree up to which the expectations are met, will then determine the Quality of Efficiency. Since experience does not happen in ‘a vacuum’, it is also necessary to consider experience in its broader ‘Context’ (De Moor and De Marez, 2007a; 2007b). 19

This methodology is the result of a collaboration between two IBBT-research groups: WiCa (Dept. of Information Technology, Ghent University) and MICT (Dept. of Communication Sciences). 20 21

Ten test users were involved in this study. These were all recruited by a specialized office. For a detailed overview of the results, see De Moor, Deryckere et al, 2008.



combinations. Based on the mean scores for the six dimensions, an overall ranking was created: 1) availability of network (connection at any time, anywhere), 2) user-friendliness, 3) interface, 4) battery lifetime + security, 5) response time. 2. Pre-usage translation workshops to find optimal match between ‘user-indicated QoE dimensions’ and ‘measurable QoS parameters’. This phase intends to translate insights from the user research into workable requirements. Here e.g., a photo-download application was developed to simulate different download times (ranging from 0 to 5 second-scenarios). This application was then shown to the test users, who were asked to indicate those scenarios that were acceptable for them (in view of a good experience), in a mobile context (cfr. Table 5). Table 5: simulation download time (N=10).

3. Monitoring during usage of QoS parameters. During this phase, the respondents tested the selected application. Different usage scenarios, consisting of a number of task to perform with the PDA, had to be completed. These tests were performed in an indoor environment at four different locations with a different signal strength at each location. The four locations were at different distances from the access point, corresponding with different measurement signal strengths (dBm). By using different scenarios, the influence of repeated tests was minimized. The test users were not aware of the fact that the signal strength was manipulated 22 . Figure 6: illustrations of usage phase

4. Post-usage questions on device (e.g. PDA). Immediately after the completion of each scenario, the test users were asked to fill in a short experience-questionnaire of 6 questions, displayed on the PDA. The monitored signal strength and responses were saved on the PDA 22

For a detailed overview of the technical aspects and QoE probes, see Deryckere, T., Joseph, W. et al (2008).



and automatically transmitted to the server. This resulted in 60 samples per location (total of 240 samples). Figure 7: example of question on PDA

5. Post-usage comparison of expectations versus experience (based on information gathered In step 3 and further user research) in order to identify and explain differences/matches between both. During this phase, a similar methodological approach as in phase 1 was taken. In this case-study, only signal strength was related to perceived experience to show that there might be a relation between QoE and QoS. For example, we selected user 10 (male, 33 years old) to explain the results for an individual user. Figure 8 shows the rating of the answers given by user 10 for different questions (Q1, Q2, Q5, and Q6) as function of the median signal strength at the different indoor locations 23 . User 10 shows great satisfaction up to -79 dBm, with ratings of 5 for expectations, reuse and general experience. At -79 dBm a slight reduction in speed is noticed by this user due to the much lower signal strength: more time is needed to load e.g., pictures on the PDA which causes the application to be slower. The ratings for speed and general experience drop significantly at -83 dBm. Expectations and reuse remain relatively high for user 10, and despite the bad experience at -83 dBm user 10 would still reuse this application.

Figure 8: Ratings for user 10 on questions Q1, Q2, Q5 and Q6

23

Location 1 corresponds with a median P of -43 dBm, location 4 with -83 dBm for this user.



With this methodology, we aimed to illustrate how QoE might be measured by an interdisciplinary team and how insights from user research might be adequately translated in technical requirements. Future research will include the testing of this methodology with a high number of users and several usage contexts.

6 Conclusion In this paper, we focused on the shift from traditional ‘technology push’ to more user-oriented and user-led approaches in the communications industry. Drawing on a number of theoretical frameworks that have studied the relationship between technology and users/society more profoundly, we reflected on the implications of this new innovation context on traditional development processes. It was mentioned that this predominant focus on the user leaded to an extension of the traditional user research arsenal with research methods from other fields. However, it was also argued that it remains difficult to create a meaningful synergy between users and technology, despite this increasing multi-disciplinarity. Given these implications with regard to user-driven innovation and the traditional tension between user- and technology-centered strategies, two crucial challenges were identified: the need for a continuous interaction with the user and the need for mechanisms to integrate the knowledge that is gathered in the increasingly multidisciplinary development process. Empirical findings from an interdisciplinary research project (ROMAS), was presented in this paper in order to illustrate how both challenges can be tackled. However, it was also shown that in some cases there is still a discrepancy between theory and practice. Although in theory many projects intend to draw on a user- and pull-driven perspective, it was shown in practice, this is not always the case: the ‘mantra’ stating that ‘innovation should start with the user and end with the user’ is not always pursued. The rise of living labs certainly is a major step, but there is a margin for improvement. The establishment of more pull-driven



living labs might provide a more accurate insight in the performance and potential of new devices and services in the daily lives of the users. These user panels could be recruited based on the devices and services they adopted and domesticated themselves. The applications to be tested would therefore only be an additional layer on top of the ‘domesticated’ networks and devices. This could be a starting point for research with more accurate insight into the natural habitat of the users.

7 References Alam, I. (2002). An Exploratory Investigation of User Involvement in New Service Development. Journal of the Academy of Marketing Sciences, 30(3), 250-261. Berte, K., De Moor, K., De Marez, L., Joseph, W., Deryckere, T. and Martens, L. (2008). Userdriven innovations? Reassessing the value of bottom-up approaches within an interdisciplinary mobile media context. Proceedings of the European Communications Policy Research Conference (EuroCPR), 31 March- 1 April 2008, Seville. Bijker, W.E. and Law, J. (eds). (1992). Shaping Technology/Building Society: Studies in Sociotechnical Change. Cambridge, MA: MIT press. Boczowski, P. J. (2004). The Mutual Shaping of Technology and Society in Videotex Newspapers: Beyond the Diffusion and Social Shaping Perspectives. The Information Society, 20(4), 255-267. Bruns, A. (2005). Gatewatching. Collaborative Online News Production. New York: Peter Lang. Crisler, K., Turner, T., Aftelak, A., & al. (2004). Considering the User in the Wireless World. IEEE Communications Magazine, 42(9), 56-62. De Marez, L. (2006). Diffusie van ICT-innovaties: accurater gebruikersinzicht voor betere introductiestrategieën. Unpublished Doctoral dissertation, Universiteit Gent: Vakgroep Communicatiewetenschappen. De Marez, L., De Moor, K. (2007a). The challenge of user- and QoE-centric research and product development in today's ICT environment. Observatorio (OBS*) online journal, (1)3: 1-22. De Moor, K., Deryckere, T., Joseph, W. et al. (2008). (ROMAS, D1.1bis) Measuring QoE: correlating user experience to network QoS in a mobile media context. Unpublished report, Ghent: IBBT. Deryckere, T., Joseph, W., et al. (2008). A software tool to relate technical performance to user experience in a mobile context. Proceedings of the 3rd IEEE Workshop on Advanced Experimental Activities on Wireless Networks & Systems, 2008. Experientia (2006). Living Labs Europe: user-driven innovation environments in the information society. Submission posted on 13 September 2006. Haddon, L., Mante, E., Sapio, B., Kommonen, K.-H., Fortunati, L., & Kant, A. (Eds.). (2005). Everyday Innovators: Researching the role of users in shaping ICT's. Dordrecht: Springer. Haddon, L., Paul, G. (2001). Design in the ICT industry: the role of users. In Coombs, R., Green, K., Richards, A., Walsh, V. (Eds.), Technology and the Market: Demand, Users and Innovation, Cheltenham: Edward Elgar Publishing. Hansson, E. (2006). Understanding User-Driven Innovation. Norden: Unpublished report. Available at www.norden.org/pub/velfaerd/naering/sk/TN2006522.pdf Jain, R. (2004). Quality of Experience. IEEE MultiMedia, 11(1), 95-96.



Khurana, A., & Rosenthal, S. R. (1998). Toward Holistic 'Front End' in New Product Development. Journal of Product Innovation Management, 15(1), 57-74. Kristensson, P., Gustafsson, A., & Archer, T. (2004). Harnessing the Creative Potential among Users. Journal of Product Innovation Management, 21, 4-14. Latour, B. (1993). We have never been modern. New York: Harvester-Wheatsheaf. Lievrouw, L. (2006). New media design and development: diffusions of innovations v social shaping of technology. In: L. Lievrouw and S. Livingstone (eds), The handbook of New Media, London: Sage, 246-265 Limonard, S., & de Koning, N. (2005). Dealing with Dilemmas in Pre-competitive ICT Development Projects: The Construction of 'The Social' in Designing New Technologies. In L. Haddon, E. Mante, B. Sapio, K.-H. Kommonen, L. Fortunati & A. Kant (Eds.), Everyday Innovators: Researching the Role of Users in Shaping ICT's (pp. 155-167). Magnus J, Ryste, N. et al. (2007). User-driven Innovation: When the user makes the difference. Report from the Nordic Innovation Centre. Mulder, I., & Steen, M. (2005). Mixed emotions, mixed methods. Conceptualising experiences of we-centric context-aware adaptive mobile services: 'User experience design for pervasive computing'. Paper presented at the Pervasive Conference, May 11, 2005. Pine, J. B., & Gilmore, J. H. (1999). The Experience Economy: Work is Theatre and Every Business a Stage. Boston: Harvard. Rickards, T. (2003). The Future of Innovation Research. In L. V. Shavinina (Ed.), The International Handbook on Innovation (pp. 1094-1100). Oxford: Pergamon, Elsevier. Rogers, E. M. (1995). Diffusion of innovations (4th ed.). New York: The Free Press. Rosted, J. (2006). User-Driven Innovation: an introduction. Presentation at the Northern Dimension Learning Forum on User-Driven Innovation. Sleeswijk Visser, F., van der Lugt, R., Stappers, P.J. (2007). Sharing user experiences in the product innovation process: Participatory design needs participatory communication. Journal of Creativity and Innovation Management, 16(1), 2007, 35-45. Toffler, A. and Tromp, T. (1981), The Third Wave. (Utrecht: Veen). Trott, P. (2003). Innovation and Market Research. In L. V. Shavinina (Ed.), The International Handbook on Innovation (pp. 835-844). (Oxford: Pergamon, Elsevier). Veryzer, R.W., Borja de Mozota, B. (2005). The impact of User-Oriented Design on New Product Development: An Examination of Fundamental Relationships. The Journal of Product Innovation Management, 22, 128-143. Von Hippel, E. (1986). Lead users: A Source of Novel Product Concepts. Management Science, 32(7), 791-805. Von Hippel, E. (2005). Democratizing Innovation. Cambridge: MIT Press. Vredenburg, K., Isensee, S. and Righi, C. (2002). User-Centered Design: An Integrated Approach. Upper Saddle River, New York: Prentice Hall. Williams, R., Edge, D. (1996). The Social Shaping of Technology. Research Policy, 25, 865-99. Yovanof, G.S., Hazapis, G.N. (2008). Disruptive Technologies, Services, or Business Models? Wireless Personal Communications, 45(4), 569-583.

8 Acknowledgements 8B

This work was supported by the IBBT – ROMAS (Research On Mobile Applications & Services) project, co-funded by the Interdisciplinary institute for BroadBand Technology (IBBT), and a consortium of companies: I-City, Microsoft and Concentra. Wout Joseph is a post-doctoral fellow of the FWO-V (Research Foundation Flanders).
