Triangulation strategies in user requirements ... - Semantic Scholar

BEHAVIOUR & INFORMATION TECHNOLOGY,

1998, VOL. 17,

NO.

2, 103 ± 112

Triangulation strategies in user requirements investigations: a case study on the development of an IT-mediated service MATTI A. KAULIO and I.C. MARIANNE KARLSSON Department of Consumer Technology, School of Technology Management and Economics, Chalmers University of Technology, S-412 96 GoÈteborg, Sweden; email [matti,mak]@ mot.chalmers.se

Abstract. EŒective investigations of user requirements for products and services are most often discussed in terms of which method to use. Nevertheless, when user investigations are planned and carried out in a system design context, several related aspects must be considered. For example: Which users should be included? Where should the study be carried out? How should data be collected? and What kind of information is relevant to what stage of the design process? In this paper, the concept of triangulation is introduced as a `conceptual tool’ for managing the multidimensionality of user requirements investigations. To illustrate how triangulation works, a case study of the user requirements investigation process for a home-shopping service system is presented. Three types of triangulations: location, knowledge, and methodological triangulation are identi® ed and de® ned, and are proven appropriate in the characterization of strategies of user requirements investigations. Moreover, the concept of triangulation as a means for increasing the reliability of a user requirement investigation, as well as some underlying factors in determining a triangulation strategy are discussed.

1. Introduction Several researchers, as well as practitioners, have argued the importance of a user-centred approach to software development in order to improve the quality, eŒectiveness, and user acceptance of the product (e.g. Gould and Lewis 1985, Norman and Draper 1986). However, while extensive research has been concerned with user-centred design processes in the context of working life (e.g. Ehn 1988, Bù dker 1991, Damodaran 1996, Heinbokel et al. 1996), less research has addressed the special pre-conditions in the development of products and services for public applications. Public, information technology-mediated systems and services (IT-mediated services) are `commonware’ today. Electronic banking, electronic shopping, and a 0144-929X/98 $12.00 Ó

number of telematic services for acquiring information, etc., are available to a large number of households. `Public’ , however, implies that something concerns people as a whole, is open to all, is maintained for the use of the community (Dillons 1990), i.e. the general public. This means that the users of public systems and services are a heterogeneous group, consisting of individuals of diŒerent socio-economic characteristics, possibly divergent attitudes towards new technology, with no or limited opportunities for formal training, and with infrequent, as well as frequent, use behaviour. Given these prerequisites; What forms can a usercentred design take in the development of public, ITmediated services? How do we assure that the user’ s requirements are taken into consideration when the user could be `anybody’? Earlier research on the development of consumer products most often emphasizes the question of which method to use for the collection and analysis of data from consumers. Some studies have been carried out, assessing the potential of diŒerent methods for generating a more complete set of problems and requirements (Gri n and Hauser 1993), but with a main focus on quantitative (`the number of’ ) rather than qualitative (`the character of’ ) aspects. Other approaches stress the importance of identifying the `right’ customer or user categories to contribute with design information. The `Lead User Method’ is one such example (Herstatt and von Hippel 1992). The choice of setting for the collection of information is a third issue. Several marketing methods are based on a de-contextualized approach (e.g. customer clinics), while more recent arguments stress visits to the customer and the importance of direct contact as the more critical dimension, not least in system development (e.g. Ealey and Soderberg 1990, Wixon et al. 1990). 1998 Taylor & Francis Ltd.

104

M.A. Kaulio and I.C.M. Karlsson

The above depicts the management of an e cient requirements investigation process as a matter of choosing the optimal strategy in relation to one dimension only. In this paper, however, the concept of triangulation is introduced and a multi-dimensional approach for achieving the best possible coverage of the total user requirement space is argued. The paper describes a project concerned with formulating user1 requirements for a home shopping system. The case study is used as an illustration of how triangulation may work in practice, and as a basis for discussion and methodological conclusions. The user focus applied in the study can be regarded as `user-oriented’ , i.e. `a cognitive-emphatic approach which pertains to a positive attitude to users and user input in the design process’ , rather than user involvement, i.e., `the organizational device of involving users in the development team’ (Heinbokel et al. 1996).

2. The concept of triangulation So far, the concept of triangulation has only been applied to data collection procedures in scienti® c research settings. The process of identifying user requirements, however, has several similarities with an explorative research process. In academic research, the observations of diŒerent phenomena culminates in the development of a theory or a hypothesis. In system design, observations of a phenomenon (i.e. information on user problems, wishes, and experiences) should converge into a new and/or improved product. The distinction between the characters of the respective outcomes of the process means that the criteria used to assess the quality of the investigations will diŒer. While, for example, the quality of the results from academic research is evaluated according to criteria such as validity, reliability, generalizability, and explanatory power, the quality of the results from user requirements investigations in a system design context relates to issues such as the relevance of the gathered information to the design decisions to be made, and, in the long term perspective, to customer satisfaction and market success. In academic research, the concept of triangulation is broadly de® ned as `the combination of methodologies in the study of the same phenomenon’ (Denzin 1974, in Patton 1987). The triangulation metaphor originates from navigation and military strategy and denotes the use of multiple reference points in order to locate the exact position of an object. Triangulation has been adapted to qualitative research (Patton 1987) and to case study research (Yin 1994) where the most common form is methodological triangulation, i.e. `the use of

multiple methods to study a single problem or program such as, interviews, observations, questionnaires and documents’ (Denzin 1974, in Patton 1987). By utilizing diŒerent data collection techniques, a greater accuracy and a more con® dent interpretation of a phenomenon, than would be possible with one viewpoint only, is possible. In addition to methodological triangulation, three additional types are suggested: data triangulation (i.e. the use of a variety of data sources in a study), investigator triangulation (i.e. the use of several diŒerent investigators in a study) and theory triangulation (i.e. the use of diŒerent perspectives to interpret a single set of data) (Denzin 1974, in Patton 1987). In applying the concept of triangulation to the product development context, three types of triangulation have been identi® ed (® gure 1). In accordance with the categorization above, the selection of data collection methods will be described in terms of methodological triangulation. The choice of diŒerent user categories (or segments) is designated as knowledge triangulation.2 A new category is also introduced: the choice of setting for the study, which is described as location triangulation.

3. The case study: identifying user requirements for a home shopping system Several of the existing public IT-mediated services concern systems for electronic shopping or `home shopping’ . A number of Swedish development projects

Figure 1. The Triangulation Model. During the development process, several studies have to be undertaken concerning decisions to be made on choices of methods, customer categories and settings.

Triangulation strategies have been initiated by municipal authorities in order to secure the distribution of groceries to, for instance, elderly persons, as well as develop a new kind of service for the general public. To form a basis for service development, a project aimed at the formulation of user requirements for home shopping systems (HSS) for goods and services was initiated. 3 The term system, here, refers to the course of events ranging from ordering groceries with the assistance of diŒerent types of media to the respective delivery of goods (® gure 2). User requirements were gathered in a two-step process in order to complete the speci® cation of requirements; (1) an initial user requirements study (Karlsson and Kaulio 1992, 1993) in which general requirements on an HSS were formulated and (2) a prototype evaluation study (Karlsson 1995) in which two diŒerent computer interfaces for an HSS were evaluated and the speci® cation for the user interface further elaborated.

3.1. The user requirement study In the user requirements study, the primary sources of information were households with access to diŒerent types of HSS (i.e. telephone based, facsimile based and computer based). These households were chosen to represent three diŒerent consumer categories with potentially diŒerent requirements for the system; (1) elderly households, (2) households with lack of time, and (3) households in sparsely populated areas. In the initial ® eld survey, individual users’ present problems and complaints, as well as their wishes and requirements for a satisfactory home shopping service were mapped and described on the basis of personal in-depth interviews with 50 individuals. In addition to interviews, observations of a series of use situations (i.e. planning purchase, ordering goods, receiving deliveries etc.) were carried out. Phone-based interviews with 20 households

Figure 2. A representation of a Home Shopping System (from Karlsson 1994).

105

utilizing HSS systems with only a limited range of products and an investigation into 48 non-system users (i.e. the general shopper) and their problems in relation to shopping provided additional information. Furthermore, a complementary picture of possible problems and requirements was obtained through interviews with 35 experts from relevant academic disciplines (e.g. consumer behaviour and cognitive science), as well as from `practice’ (e.g. store owners and information management companies). The results from the study showed that the problems brought forward by the users were almost identical for all of the diŒerent categories and were also independent of the diŒerent features of the HSS with the exception of diŒerent priorities attributed to certain aspects. For example, time conscious households gave high priority to exact delivery times while rural and elderly households were more ¯ exible. Judged by the interviews, the prime factors contributing to the individual’s assessment of the HSS were (1) the perceived quality of the goods delivered, (2) the perceived quality of the `shop’ (e.g. a limited or a more complete assortment) and (3) the perceived quality of the `delivery’ (e.g. goods delivered on time, employees judged as `service minded’ ). Even though both observation studies and heuristic evaluations of the interface design according to usability principles (e.g. Shneiderman 1987, Nielsen 1993) revealed several shortcomings, these were rarely or never mentioned by the users themselves. Examples are the wording of diŒerent videotext commands, commands built as sequences of key strokes (e.g. `* 9 #’ to exit the system) and lack of complete and understandable instructions in the menus. If media aspects were mentioned, they were related to service access (e.g. the line being busy) or reliability (e.g. lack of trust) rather than to interface design issues as such. The analysis concluded that the main problems were primarily associated with (1) the organization of the total system and (2) the interface communication between customer /user and shop /supplier. The results of the study lead to the development of a requirements’ speci® cation describing the general goals of the total service system. By attributing each of the mentioned problems to one or several functions within the home shopping system, requirements were allocated to diŒerent functions within the service organization: such as, the `gathering function’ , `delivery’ , etc. (® gure 3). From a service production perspective, the user took part in the production of the service by specifying the `system input’ ( i.e. the order). It became obvious that the `system output’ (i.e. the delivery and the groceries being delivered) was highly dependent upon the order contents and the level of speci® cation. Lack of, or incomplete, information resulted in mistakes and inter-

106

M.A. Kaulio and I.C.M. Karlsson

Figure 3. The requirements tree. The diŒerent levels of requirements elicited in the two studies.

pretation errors. Consequently it was concluded that the ordering medium must be designed to support the user by providing a comprehensive set of information while at the same time be easy to use. Based upon these ® ndings and upon general usability criteria (e.g. Shneiderman 1987, Nielsen 1993), the requirements for the dialogue between user and home shopping system interface were formulated (see ® gure 3) and a new computer interface was developed.

3.2. The prototype evaluation study The aim of the prototype evaluation study was to compare a typical version of the old (videotex) systems used, with the new prototype. The evaluation was carried out in a usability laboratory. The study was comprised of 24 subjects from three diŒerent categories: experienced computer users (n = 8), computer novices (n = 8) and, a speci® c target group, elderly ( > 70 years old) computer novices (n = 8). The choice of categories was based upon the assumption that experienced computer users pose diŒerent requirements on the graphic design of an interface than computer novices (e.g. Allwood 1986). The test followed general usability testing procedures (cp., e.g. Shneiderman 1987, Whiteside et al. 1988,

Nielsen 1993) and was run according to a within-subject design. All subjects tested both systems but on diŒerent occasions (approximately one week apart), the order randomized in order to avoid systematic errors. The actual evaluation task was divided into an unspeci® ed task (consisting of ordering items of the subject’ s own choice), and a speci® ed task (consisting of ordering items from a list). The second task served as a basis for the objective comparison of the systems while the purpose of the ® rst task was to simulate a more natural shopping situation. The groceries were delivered to the subjects on arriving to the second trial to make the trials more realistic and to probe comments. The main objective of the study was to determine if an interface had been created which was both easy-to-use and useful in a home shopping system. These questions had to be addressed by applying usability guidelines and performing both objective, quantitative measures (e.g. the number of errors made, (shopping) task completion time), and subjective assessments (e.g. perceived ease-ofuse). The assessments clearly indicated that the initial problem analysis and the usability guidelines had helped in reducing several problems related to interaction. Total task completion time, for instance, was reduced by between 20 ± 40% , error correction time, as well as those instances when the subjects asked for help/assistance were also reduced. Nevertheless, a number of issues that

Triangulation strategies concerned pre-purchase planning and other, speci® c shopping related, aspects could not be evaluated by formal quantitative measures. In this instance, observations and post-test interviews provided an important, complementary picture. The interviews brought forth suggestions to improve the selection process of the system by making choices between diŒerent items easier, for example by adding critical information regarding food content. In addition, suggestions for improving screen layout and the choice of icons, etc. were proposed. As a consequence of these results, the speci® cation of user requirements could be elaborated from a sub-system level to one of a more operational form (i.e. requirements for the solutions of actual features of the user interface and the structure of the dialogue) (® gure 3). The choice of input device (e.g. mouse or keyboard) and help messages were the most critical usability aspects for the novice user while other aspects (such as system shortcuts) played a more important role for the experienced user. However, the anticipated con¯ ict between requirements was not con® rmed. Actually, the interface which was initially designed for the elderly computer novice, the main target group, resulted in few negative consequences for the expert users. Therefore, by simply changing or adding a few features, a more ¯ exible and versatile system could be achieved. Hence, the prototype evaluation study could be described as summative with the main purpose being to determine whether or not design goals had been met. It can also be described as formative in character since remaining problems and design errors were identi® ed and analysed as a basis for further system improvement.

4. Analysis In the following section, the aim is to link ® ndings from literature to those of the described case study. In order to make the analysis clearer, the considerations are presented as a process of sequential decisions instead of in a simultaneous process, its original format.

4.1. Location triangulation

4.1.1. The importance of context: In planning user requirements investigations, a major question is whether or not investigations should be set in the ® eld or in a decontextual setting, such as, in a usability laboratory. In accordance with a positivistic view, a decomposed research approach, in which all variables can be standardized and controlled, may be argued. Never-

107

theless, removing people from their accustomed environment, their usual context and by removing the product from its intended usage environment, will all have an aŒect on the validity of the information retrieved. At least in system design, an awareness has grown of the importance of studying practice and gathering information from customers /users in `context’ (e.g. Whiteside et al. 1988, Wixon et al. 1990, Bannon and Bù dker 1991, von Hippel 1994, Karlsson 1996). The overall project goal was to contribute to the development of a service that provided the user with the bene® ts and values anticipated and required. As use qualities emerge in context and as a consequence of the use environment, in-context aspects must be regarded as a necessary pre-requisite for developing a commercially successful service. The case study used a combination of an initial ® eld approach, followed by a laboratory investigation of speci® c interface features. The ® eld study typically diagnosed the existing state, while the laboratory test evaluated the change of a systemcomponent, excluding all aspects but those related to user ± interface interaction. Changing context also meant that the character of the information retrieved changed. In the ® eld study, the users’ dominating relation to the system was obviously that of the customer utilizing a home shopping service. Consequently, the goals of the shopping activity was focused. As the home shopping systems that were investigated had been in use for some time, media design defaults had been compensated for by the individual learning ways of how to deal with problems. In summary, the information elicited from the ® eld study amounted to an understanding of what factors contributed to the service system quality. Through an analysis, the role of the media and the media interface within that system was understood. In the laboratory test, however, the role of the user was that of a user interacting with an interface. This meant that sub-system activities were monitored and usability problems were highlighted. 4.1.2. EŒects on the list of requirements Ð strategic to operational: The role of the media within a service system is one of mediator to that service. The requirements on the media are determined by that role, in relation to the overall goals of the service. Hence, these overall goals must be understood before design criteria for the media interface can be determined. The triangulation of locations indicated that information with a diŒerent focus was provided and could be triggered in the diŒerent settings (® gure 4). A decontextual study alone, such as the laboratory study, would not have provided enough information in order to diagnose the total situation as it represented a focused search for speci® c aspects. While speci® c knowledge of

108

M.A. Kaulio and I.C.M. Karlsson

pre-requisites for unhampered interaction could be obtained, information on the usefulness of the interface in a functioning system service could not.

4.2. Knowledge triangulation In a broad perspective, knowledge triangulation includes the triangulation of the user’ s knowledge (i.e. the knowledge and experience inherent in the user, often as tacit knowledge), as well as knowledge about users and product use (i.e. expert knowledge on consumer behaviour, man-machine interaction etc.). 4.2.1. Sample strategies Ð randomization or strati® cation: The target group for public systems and services is the general public, i.e. a large number of individuals with diŒerent experiences, attitudes, and motivation. As users’ knowledge is to be brought into the design process, the selection of the users and user groups is critical. The overall strategy in a user oriented development process is to study the intended users (e.g. Nielsen 1993) or, if a larger population, individuals representative of the user population in order to `know the user’ . Conventionally a randomized sample is recommended, however, an alternative principle is `theoretical’ sampling. The latter approach originated in case study research (Eisenhardt 1989). By identifying those sub-groups of users which, on a logical basis, pose `extreme’ requirements on a product or service, the total set of requirements, valid for a larger population can be determined. This approach has here been labelled the use of `critical users’ . In the case study, which included both a ® eld study and a laboratory study, three diŒerent types of user groups were investigated. By combining the requirements’ pro® les for each of the sub-groups, it was possible to construct a larger requirement picture (® gure 5). In the

Figure 4. Location triangulation. DiŒerent locations induce the search for diŒerent aspects.

® eld study, for instance, it was concluded that the major diŒerences between the service user groups (i.e. between the customer groups) were the priority given to diŒerent (sub)functions and requirements. For example, were time conscious households speci® c about precision in delivery on time and agreement between order and delivered groceries, while rural households did not consider these aspects as critical. On the one hand, most elderly and rural households were totally dependent upon the service while other households most often could compensate a delivery default by going shopping themselves. 4.2.2. EŒects of the list of requirements Ð validation: Knowledge source triangulation resulted in the crosschecking of ® ndings and formed the basis for a validation. A kind of `sensitivity’ analysis could be carried out. Since a major part of the problems and requirements were actually the same, data from the one user group validated data from the other. Hence, the theoretical sampling strategy, based on the above described concept of `critical users’ , seemed to be a possible strategy for keeping a user centred approach while developing public systems.

4.3. Methodological triangulation

The third factor associated with eliciting user requirements is the choice of method for collecting information. Two types of methodological triangulation can be found in literature: `between-method’ triangulation and `withinmethod’ triangulation (Jick 1979). Between-method triangulation refers to the process of combining diŒerent

Figure 5. Triangulation of `critical’ user groups from the ® eld study, illustrated by a radar chart. By combining diŒerent requirements pro® les, an assessment of both the kernel (i.e. the intersection) and the total requirement space (i.e. an extrapolaration of the total set of requirements) can be made.

Triangulation strategies methods in order to test external validity. Withinmethod triangulation, on the other hand, refers to the process of cross-checking for internal validity, for example in a survey in which multiple scales are used in order to measure the same construct. Furthermore, for `between-method’ triangulation, two other strategies are possible: sequential or simultaneous combinations of methods (Granstrand 1995).4. 4.3.1. Sequential combinations Ð divergence and convergence: An iterative development process, in which interactions with customers /users occur at repeated instances, is often argued as opposed to a linear process in which all data is collected in the initial phase and the product is evaluated in the ® nal phase (e.g. Gould and Lewis 1985) in order to capture the total requirement space (Karlsson 1996). However, diŒerent questions are posed in relation to diŒerent stages of the product development cycle as initial strategic issues develop into operational ones. In turn, diagnostic questions (questions about ends) develop into decision-making questions (questions about means to the end) (Kaulio 1995). Consequently the character of the methods used also changes. In the case study, the sequential combination of methods were structured in three distinct phases (see ® gure 6). The phases can be described as the speci® cation of requirements, the veri® cation of requirements, and, ® nally, prototype evaluation for the validation and elaboration of requirements. The user requirements study could be said to be explorative and diagnostic in nature. The aim was not to test a `hypothesis’ but to describe the key design issues of the multidimensional interaction pattern between user,

Figure 6. The structure of the simultaneous and sequential utilization of diŒerent methods for data collection. The boxes represent the depth and width of the scope of each investigation.

109

system, and task, thereby obtaining a foundation for system design. What key aspects of the service system aŒect user satisfaction? What are the border lines of the system which have to be addressed? In-depth, face-toface, interviews combined with contextual observations were used as the primary approach. In order to gain a broader understanding, as well as to verify the initial results, personal interviews with non-users and phonebased interviews with other user groups were conducted while at the same time questions posed to the users changed character and became less explorative (e.g. Are the results valid also for other user groups?). In the prototype evaluation, the issues addressed had converged into media interface related aspects (e.g. Is the order of the screens acceptable? Is the design of screens consistent? Is terminology consistent? etc.) and the approach had changed to one of a more traditional usability study. 4.3.2. Simultaneous combinations Ð validation and con® rmation: Throughout the study, the simultaneous use of several methods made it possible to cross-check, e.g. statements from the users with observations of the same users. Especially in the prototype evaluation study, several data collection methods were used in a simultaneous approach in order to further validate and provide complementary information about the same design object. Consequently, any improvement of the new design was cross-checked both by subjective data and objective data. A necessary diagnostic approach was maintained through the combination of observations and interviews, in order to secure that remaining defaults could be penetrated enough to guide decisions on changes in the continued design process. In addition, a within-method triangulation was applied through the use of several types of measurements (e.g. simple rating scales and Likert scales ) in order to address one and the same quality feature, such as, `ease-of-use’ . 4.3.3. EŒects on the list of requirements Ð strategic and operational: As a consequence of the triangulation strategy used, the requirements picture emerged and evolved in terms of its scope, its `width’ and `depth’ . Initially, qualitative data (based on interviews and observations) was collected in order to frame user problems and wishes. These initial stages of the requirements investigation focused both on more strategic requirements (i.e. requirements related to the functionality or to the overall features of the service system) and on operational requirements (i.e. requirements related to the actual design of the diŒerent service components, as well as aspects of the interface design). Here, a description of the users’ problems was obtained and a tentative list of requirements was compiled. In the latter phase (i.e. the prototype evaluation study)

110

M.A. Kaulio and I.C.M. Karlsson

operational requirements on a `second level’ were addressed, this time focusing on particular features of the technical interface for this speci® c IT-mediated service. One question raised, for example, was how should the more global requirement for `ease of use’ be transformed into speci® c operational requirements describing the sequencing of screens, the labelling of commands, and acceptable response times etc.? A multi-method strategy is by no means a unique strategy for this study. The major part of the existing literature on usability engineering and related issues argue for the use of, at least, more than one method. However, as the requirements’ picture must be secured on several levels and against a heterogeneous population, the relevance of methodological triangulation could be further stressed.

5. Discussion and conclusions When the goal of a study, or a series of studies, is to identify user problems or user requirements, diversity, in terms of locations, (human) knowledge sources and methods, is positive. The broader the cast of the net, the more likely a broader array of user requirements will be the catch. This diversity may make triangulation seem industrious. The idea of triangulation, however, is not to apply the approach en masse but to apply it adaptively, on the basis of the prerequisites given by the circumstances of the development project at hand. In this paper, the concept of triangulation is introduced to the area of system design. The paper is meant to raise and broaden the question of user requirements investigations from a discussion of how to conduct one speci® c method to a question of how to ® nd synergies in combinations of locations, knowledge and methods. On the basis of literature and a case study the following conclusions are drawn. First, the case study shows that the concept of triangulation in general, and the three types of triangulation, in particular, are appropriate in the characterization of diŒerent strategies of user requirements investigations in a system design context. A fundamental starting point in a triangulation approach is a neutral view of methods and approaches. All methods, whether qualitative or quantitative, or whether laboratory or ® eld approaches, are considered possible. The challenge is to ® nd a mix suitable to the problem at hand. The idea of triangulation is not new, however. Several studies have earlier emphasized the bene® ts and importance of methodological triangulation (e.g. Nielsen 1993; Noyes et al. 1996). Results from this paper stress the importance of also considering the other two types of triangulation; location and knowledge triangulation.

Second, the results indicate that the underlying factors in determining a triangulation strategy are: the heterogenity of the user population, the size of the user population and the stage of the system development life cycle. (1) The heterogenity of the user population is believed to be one factor in determining the degree and the dimensions of triangulation. An anticipated homogenous set of user requirements, as, for example, for the up-date of an administrative system would lead to a lower need for triangulation in terms of methods and knowledge. In contrast, the development of a system with an anticipated heterogeneous set of user requirements, as, for example, the development of a new public IT-mediated service, would require a greater variety in the triangulation strategy applied in order to identify possible convergences and divergences of the total user requirements space. (2) In a similar way, requirements elicitation for a public system will require a strati® ed sampling strategy as the user population will be large. Such a sampling process has no meaning if the target is a small, well organized population where principally all users can be reached. (3) Along the development life cycle, the sequencing of ® eld ± laboratory ± ® eld studies seems appropriate. As uncertainty regarding user requirements is likely to be high in the early development stages, diagnostic information is central and, therefore, an explorative and contextual approach essential. Consequently, diversity, regarding methods and knowledge is favoured, as well as ® eld studies. On the basis of such a diagnosis, key areas of remaining uncertainty may emerge which require further elaboration. Nevertheless, as uncertainty successively is reduced as the system is de® ned and decisions concerning solutions are made, the need for triangulation changes. At these intermediate stages, i.e. when a satisfying solution is to be decided upon, laboratory studies that focus on particular parts of the dialogue and interface are necessary. Finally, in the latter stages of the development life cycle, the complete product must be tested and veri® ed in context. Third, triangulation could be argued to be a means of increasing the reliability of user requirements investigations. Reliability in a statistical meaning focuses on making statements on the degree of certainty of the proposed conclusions. Typical ways of increasing reliability are standardizing the method or measurement, standardizing the conditions in which the investigation is carried out or by increasing the number of respondents (e.g. Hellevik 1990). In contrast, triangulation deals with the concept of reliability in terms of the convergence of results from diŒerent locations, diŒerent knowledge sources and from diŒerent methods. Instead of repeating one measurement an increasing number of times, the object is investigated several times over using diŒerent

Triangulation strategies methods, from diŒerent perspectives or in diŒerent locations. The primary aim is not to reduce variance but instead to investigate where data diverge, i.e. to identify the boundaries, areas of co-occurance and areas of con¯ ict for speci® c sets of requirements. Data can and will diverge. In practice, however, there are few guidelines for the systematic ordering of eclectic data into converging categories. Consequently, the initial investigations should focus on the identi® cation of key areas of success and on areas of uncertainties. In later investigations, the key areas are validated and the areas of uncertainties could be analysed more in depth. In the case study, issues of uncertainty in the initial requirement studies concerned the design of the computer interface and were more deeply investigated in later prototype evaluations. This strategy could be applied in order to handle also divergent data, i.e. to use those areas where divergence occurs as areas for further investigations or follow up studies in later stages of the process. The present study is an illustrative and explorative single case study on the development of an IT-mediated public service. The study reveals that future research on triangulation strategies and triangulation design principles is necessary. The aim is that the analysis of the case study will inspire practitioners in the invention of new triangulation strategies, and motivate researchers to conduct more in-depth investigations into the ® eld. From the practitioner’ s point of view, it seems important to further examine the process of triangulation and which triangulation strategies are eŒective in diŒerent design situations. A future list of issues to triangulate is, by no means, limited to the ones mentioned above. For example, prototypes of the diŒerent degrees of concreteness is one example of a component which can be used in order to vary the approach to user requirements investigations.

Acknowledgements The authors would like to thank the Nordic Committee for Consumer AŒairs and the National Swedish Board for Consumer Policies for ® nancial support, and the two anonymous reviewers for useful comments.

NOTES 1

In this paper the term `user’ is applied even though the service providers most often talk about `users’ as `customers’ since their relation includes an economic transaction.

111

2

The term `knowledge’ is used instead of data triangulation for two reasons. The information gained from users includes more than data; for example, diŒerent perspectives (cp. theory triangulation). In addition, the term data refers to users as `objects’ , not as living `subjects’ which attitudes, preferences and emotions. The distinction is important as it, implicitly, communicates the nature of the source. 3 The user requirements study was carried out during 1992 ± 1994. The computer based systems were all videotext based. Nevertheless, the ® ndings are relevant for systems using other communication media, such as for example the Internet. 4 In this analysis, between-method triangulation is emphasized, as this approach proved to be more important in developing the requirement picture.

References ALLWOOD , C.-M. 1986, Novices on the computer: a review of the literature. International Journal of Man-Machine Studies, 25, 633 ± 658. BANNON , L. and Bé DKER , S. 1991, Beyond the Interface: Encountering Artifacts in Use. In Caroll J.M. (ed.) Designing Interaction. Cambridge series on Human-Computer Interaction. (Cambridge: Cambridge University Press) 227 ± 253. Bé DKER, S. 1991, Through the Interface. A Human Activity Approach to User Interface Design (Hillsdale, NJ: Lawrence Erlbaum Associates). DAMODARAN , L. 1996, User involvement in the systems design processÐ A practical guide for users. Behaviour & Information Technology, 15, 363 ± 377. DENZIN , N.K. 1978, The Logic of Naturalistic Inquiry. In Denzin, N.K. (ed.) Sociological Methods: A Sourcebook (New York: McGraw-Hill). DILLONS Modern English Dictionary, 1990. EALEY, L. and SODERBERG , L.G. 1990, How Honda cures `design amnesis’. The McKinley Quarterly, Spring. 3 ± 14. EHN, P. 1988, Work-Oriented Design of Computer Artifacts (Stockholm, Sweden: Arbetslivscentrum). EISENHARDT , K.M. 1989, Building theories from case study research. Academy of Management Review, 14, 532 ± 550. GOULD J. D. and LEWIS, C. H. 1985, Designing for usability. Key principles and what designers think. Communications of the ACM, 28, 300 ± 311. GRANSTRAND , O. 1995, Methodology of Combined CaseSurvey Studies. School of Technology Management and Economics. Working paper. Dept. of Industrial Organisation and Economics. Chalmers University of Technology. GoÈteborg. GRIFFIN, A. and HAUSER J.R. 1993, The voice of the customer. Marketing Science, 12, 1 ± 27. HEINBOKEL, T., SONNETAG , S., FRESE, M., STOLTE , W. and BODBECK , F. 1996 Don’t underestimate the problem of user centeredness in software development projectsÐ There are many! Behaviour & Information Technology, 15, 226 ± 236. HELLEVIK, O. 1990, Research Methods in Sociology and Political Science. Natur och Kultur. SoÈdra Sandby (In Swedish).

112

M.A. Kaulio and I.C.M. Karlsson

HERSTATT , C. and VON HIPPEL, E. 1992, From experience: Developing new product concepts via the lead user method: A case study in a `low-tech’ ® eld. Journal of Product Innovation Management. 9, 213 ± 221. JICK, T. 1979, Mixing qualitative and quantitative methods: triangulation in action. Administrative Science Quarterly, 24, 602 ± 611. KARLSSON , M.A. 1994, Order your Groceries by Fax. A Feasibility Study. TemaNord 1994:657 (Copenhagen: Nordic Council of Ministers) (In Swedish). KARLSSON , M.A. 1995, Evaluation of Two Interfaces for a Home Shopping System in a Laboratory Setting. Report from the Dept of Consumer Technology, Chalmers University of Technology, GoÈteborg, (In Swedish). KARLSSON , M.A. 1966, User Requirements Elicitation. A Framework for the Study of the Relation between User and Artefact. Dissertation from the Dept of Consumer Technology, Chalmers University of Technology, GoÈteborg. KARLSSON , M.A. and KAULIO, M. 1992, User Requirements for Home Shopping Systems (Copenhagen: Nordic Council of Ministers) (In Swedish). KARLSSON , M.A. and KAULIO, M. 1993, Do computers solve the problem? Telephone vs Telefax vs Computers in home shopping systems in Scandinavia. In M.J. Smith and G. Salvendy (eds), Advances in Human Factors/Ergonomics 19 A. Human-Computer Interaction: Applications and Case Studies (Amsterdam: Elsevier Publishers), pp. 427 ± 432. KAULIO, M. 1995, Requirements Engineering. An Analysis of Design Principles, Approaches and Methods for User Involvement in Product Design. Thesis for the Licentiate Degree. From the Dept of Consumer Technology, Chalmers University of Technology, GoÈteborg.

NIELSEN, J. 1993, Usability Engineering (San Diego: Academic Press). NORMAN , D. and DRAPER, S. (eds) (1986), User Centered System Design. New Perspectives on Human-Computer Interaction (Hillsdale, NJ: Lawrence Earlbaum Associates, Inc.). NOYES , J.M. STARR , A.F. and FRANKISH , C.R. 1996, User involvement in the early stages of development of an aircraft warning system. Behaviour & Information Technology, 15, 67 ± 75. PATTON , M. Q. 1987, How to Use Qualitative Methods in Evaluation (Newbury Park: Sage Publications Ltd.). SHNEIDERMAN , B. 1987, Designing the User Interface: Strategies for EŒective Human-Computer Interaction (Reading, MA: Addison-Wesley Publishing Co.). VON HIPPEL, E. 1994, `Sticky information’ and the locus of problem solving: Implications for innovations. Management Science, 40, 429 ± 439. WHITESIDE, J., B ENNET, J. and HOLZBLATT, K. 1988, Usability engineering: Our experience and evolution. In M. Helander (ed.) Handbook of Human-Computer Interaction (Amsterdam: Elsevier Sciences Publishers) 791 ± 817. WIXON D., HOLTZBLATT K. and KNOX S. 1990, Contextual design: An emergent view of system design. In J.C. Chew and J. Whiteside (eds), Proceedings of the CHI’90 Conference on Human Factors in Computing Systems (New York, N.Y: Association for Computing Machinery), pp. 329 ± 336. YIN, R. 1994, Case Study Research: Design and Methods. Second Edn. (Newbury Park: Sage Publications Inc.).