present a methodology that incorporates incremental, iterative business .... of the kinds of benefits directly attributable to different types of information technology.
The I.S.S.U.E Methodology for Quantifying Benefits from Information Systems
George M. GIAGLIS Department of Information Systems and Computing Brunel University, United Kingdom
Nikolaos A. MYLONOPOULOS The Business School Loughborough University, United Kingdom
Georgios I. DOUKIDIS Department of Informatics Athens University of Economics and Business, Greece
Abstract
The assessment of Information Systems (IS) benefits is an important practical problem in IS investment appraisal. In this paper, after briefly reviewing the nature of IS benefits, we argue that an incremental measurement approach can help an organisation obtain quantitative estimates of expected IS impacts on business performance. Such an approach should start from quantifiable benefits directly attributable to the information system and gradually considering more intangible and indirect effects. We suggest that Business Process Simulation can be an effective technique in applying this approach and we present a methodology that incorporates incremental, iterative business process modelling and simulation into five practical steps. To illustrate this approach, we present a case of measuring the potential improvements in inventory management introduced by Electronic Data Interchange and discuss issues of feasibility and directions for further research and development.
1
1. Introduction
In his highly critical work, Paul Strassman (1990) points out that if one read what experts have been saying about IT investments they would become severely discouraged. Although recent studies have produced hard evidence of the contribution of Information Technology to economic performance (Hitt and Brynjolfsson 1996, Reardon et al 1996, Powell and Dent-Micallef 1997), the problem of preimplementation (ex ante) evaluation of investments in IT keeps puzzling researchers and practitioners alike.
In the present paper we focus on the assessment of benefits in the context of IT investment appraisal at the level of firms or business units. While a definite solution to this problem has not appeared yet, the problem itself and its causes have been widely discussed. The most commonly cited causes of the difficulty to appraise IT investments include among others (Strassman 1990, Parker et al 1988, Farbey et al 1993, Banker et al 1993): a)
The intangible nature of the benefits.
b) The benefits of IT are realised in the long run. c)
Strategic and competitive advantages are inherently difficult to quantify.
d) The benefits of IT are indirect and therefore indistinguishable from several confounding factors. e)
The theories and techniques available are inappropriate for understanding and capturing the value of information systems.
In the absence of a definite theory of IT investment appraisal (Powell 1992, Hitt and Brynjolfsson 1994), a multiplicity of methods and techniques for deciding on the desirability and priority of different IT projects have been proposed (for example, Parker et al 1988, Strassman 1990, Veryard 1991, Grindley 1991, Gregory and Jackson 1992, Wiseman 1992, Cornford et al 1994, Hogbin and Thomas 1994, McBride and Fidler 1994, Sauer 1994). Farbey et al (1992, 1993) expose the implicit features and assumptions of a number of evaluation methods. They provide a classification of IT projects and suggest sets of methods with different attributes that better suit different classes of projects. Despite the availability of various different approaches, Ballantine et al (1994) found that most companies use
2
simple accounting techniques, notably cost-benefit analysis, in order to decide whether to proceed with a certain IT investment or not. Simple, general purpose and widely understood measures of value are required by top management, auditors and stockholders. These criteria are satisfied by standardised accounting and financial methods such as cost-benefit analysis and return-on-investment. Such indices convey significant advantages since they make alternative investments directly comparable and are readily understood by any interested party.
One of the main problems when trying to apply any of these financial techniques for assessing a particular IS investment, seems to be the difficulty of identifying and measuring the expected benefits of a proposed information system (e.g. Ballantine et al 1994). Estimates for the expected benefits of a given information system on business performance are far from easy to obtain, especially in the case of complex, sophisticated IT applications with intangible, indirect, or strategic impacts on business performance. Such so-called “soft” benefits (Brown, 1994) are usually realised in the form of improved customer service, better management control, organisational change, facilitation of new management strategies, competitive advantage, and so on. These factors are very difficult to capture and analyse, let alone objectively measure and quantify before the investment is made. Management intuition, although necessary for project initiation and commitment to implementation, cannot be considered as an adequate investment evaluation criterion. After all, a proposal for IS development must be able to make a convincing case to management in terms of expected costs and revenues associated with it (see, for example, Ormerod 1996). This in return, calls for quantitative estimates of the expected benefit of the proposed investment.
This paper focuses on the problem of measuring the benefits of information systems and suggests a set of principles and a technique which can generate quantitative estimates of the expected benefits of information systems. Briefly, we argue that Business Process Simulation (BPS), in the form of detailed models of the intended usage of a given IS, can provide the needed laboratory-like setting in which objective, quantitative estimates on expected benefits can be obtained. In what follows, we briefly review the nature of benefits from information systems, their characteristics and categories. Next the principle of incremental modelling and measurement is introduced and substantiated. Business process simulation is proposed as a suitable technique in the section that follows. Next, the I.S.S.U.E
3
methodology is presented, which consolidates the principle and the technique into a coherent five-step process for applying them to practice. The application of the methodology is then demonstrated with a case of actual evaluation of the benefits from EDI. The final section concludes the paper with a discussion of broader considerations pertaining to the relevance and applicability of I.S.S.U.E.
2. The Nature of IS Benefits
Organisations may arrive at the need for an information system through a number of alternative avenues. It may be a response to an identified problem, the realisation of an established IS strategy, a result of a Business Process Redesign (BPR) exercise, or even imitation of a competitor’s tactics. In any case, a proposal for a particular system is initiated with specific business goals in mind, and with a set of primary objectives being sought by the introduction of the system. These objectives constitute the expected benefits that have to be comparatively evaluated against costs, in order to justify the proposal. The costs associated with developing a particular information system are relatively easier to measure, at least the direct ones, usually during the feasibility study. Indirect costs arising from implementation setbacks or from organisational resistance to change are virtually impossible to assess a priori. However, in comparative terms, it is significantly more difficult to obtain hard evidence of the expected benefits as it is of the costs.
Brown (1994, p. 187) distinguishes between hard and soft benefits. Hard benefits are a direct result of the introduction of the information system and are easily measured. According to Brown, soft benefits include at least intangible, indirect and strategic. Figure 1 builds upon this classification to surface the importance of both the extent to which benefits are directly attributable to the introduction of the information system and the extent to which they can be readily quantified. The horizontal axis distinguishes between quantifiable and non-quantifiable benefits, while the vertical axis distinguishes between those benefits that are realised solely as a result of the introduction of the information system, from those that depend, to a greater or lesser extent to other organisational factors as well.
4
Attributable to the IS
Weakly Indirect
Strategic
Hard
Intangible
Strongly Quantifiable
Non-Quantifiable Measurable
Figure 1. Different types of IS benefits
Hard benefits are usually related to cost reduction, such as the reduction in data-entry staff made possible by the introduction of an electronic ordering system or to revenue generation, such as the increased throughput as a result of a new production control system. Such measurable benefits are relatively easy to incorporate in traditional investment appraisal techniques.
The problem of measurement discussed above is mainly related to the remaining three categories of socalled “soft” IS benefits. Intangible benefits can be attributed to particular applications but they cannot be easily expressed in quantitative terms. Benefits of this type arise, for example, with the introduction of a Decision Support System. Such systems are primarily expected to improve the quality of decision making as well as the job structure of their users. Firstly, it is difficult to define ‘quality of decision making’ and ‘job structure’. Secondly, even if this is achieved, it may still be difficult to assign a quantitative (and, preferably, monetary) measure of improvement in advance.
Indirect benefits are potentially easy to measure but cannot be wholly attributable to the proposed investment and can only be realised as a result of further investments, enabled by the new system. For example, the implementation of a Local Area Network (LAN) across an organisation provides the infrastructure on which valuable shared applications can later be implemented. Although this is a potential benefit made possible by the LAN, it cannot be realised unless these shared applications are also successfully introduced. Such complementary investments may be in IT or in any other
5
organisational resource, such as a change in business processes enabled by the introduction of IT. Although in the broader sense of the term, many indirect benefits may not be directly measurable, in this quadrant we emphasise those indirect benefits that can be quantified. For ease of reference, we place all indirect benefits that are not easily quantifiable at the ‘strategic benefits’ quadrant.
Strategic benefits refer to positive impacts that are realised in the long run and usually come as a result of the synergistic interaction among a number of contributing factors. They are the outcome of, for example, a new business strategy or a better market positioning of the organisation, which can only be partially attributed to a given IS. Such benefits are notoriously difficult to quantify in advance due to their very nature and to the risk associated with their realisation.
At this point, it must be noted that rarely does one information system yield one type of benefits alone. Any given information system can be expected to deliver a range of various types of benefits. Moreover, different kinds of systems can produce different combinations of types of benefits, with more sophisticated systems usually moving towards soft and difficult to measure benefits. Table 1 illustrates this concept in terms of the project ladder given by Farbey et al (1993). While any type of benefit can generally be sought and realised by an information system, Table 1 emphasises those types of benefits that are typically associated with each project type.
Clearly, there are well known examples of information systems, which started out as relatively mundane automation projects but eventually proved to be of great significance, yielding sustained strategic benefits. It should be noted, however, that strategic benefits to such automation projects are not an immediate consequence of the implementation of the respective information system but the indirect result of suitable organisational transformations and environmental conditions. Thus, Table 1 should not be seen as dictating the necessary focus of the benefits evaluation exercise. It is rather an indicative classification of the kinds of benefits directly attributable to different types of information technology investments.
6
Rung
Project Types
Typical Benefit Types Hard
8 7
Intangible
Indirect
✔
Business Transformations ✔
✔
✔
✔
✔
✔
✔
Strategic Systems
6 5 4 3 2 1
Strategic
Inter-Organisational Systems Infrastructure
✔
MIS and DSS ✔
Direct Value Added
✔
✔
Automation
✔
Mandatory Changes
Table 1. Typical Benefits of Different IS Projects
Given the nature of IS benefits as discussed so far, two questions can now be asked: a)
Where should our efforts to measure IS benefits focus on?
b) How can the evaluation process be supported in practice by means of quantifiable measures of IS benefits?
In section 3 the first question is discussed and a reference framework for IS benefits measurement is proposed. In section 4 the potential of dynamic business process modelling techniques to support the evaluation process is investigated.
3. The Principle: Incremental Benefits Measurement
Many approaches reported in the literature adopt a wide focus attempting to approximate the introduction of an IS with aggregate measures of firm performance, such as profitability (Brynjolfsson 1993, Smith and Keen 1993, Lubbe 1994). This applies equally to ex ante investment appraisal as well as to ex post performance assessment (Banker et al 1993).
Several researchers however argue that research should be more narrowly focused on the immediate environment of the system in order to limit the number of confounding factors co-determining the
7
outcome measures. For example, Wheeler (1994) examines the current benefits and future opportunities of Executive Information Systems and supports this argument by concluding that best practice should focus on setting targets and using measurable diagnostics. Kelley (1994) measures the productivity gains from IT in manufacturing processes. She argues that such effects should be measured at the microlevel of the production process itself and not at firm level.
We follow the same argument and expand it further by suggesting that the benefits measurement exercise should start with those benefits which are realised as a direct outcome of the system under examination and are readily quantifiable, namely hard benefits (Figure 1). Once these are studied, understood and measured, indirect and intangible benefits can gradually be brought into perspective. Strategic benefits can then follow as the ultimate step of this incremental approach. In other words, the benefits at higher levels of aggregation and complexity can be studied and measured more easily and accurately after a well understood model of the direct and quantifiable benefits is established. Knowledge gained at each step of this incremental process enables both the incorporation of more indirect effects, as well as the partial quantification of intangible benefits.
For example, it is possible to study in detail the possible strategic benefits of a given information system on market structure and competition, only after establishing a sound understanding of the operational benefits in the business processes it supports. Figure 2 illustrates this argument. The arrows present the proposed route to IS benefits measurement: climbing the ladder from quantifiable benefits attributable to the information system, to more indirect, intangible and/or strategic ones.
8
Attributable to the IS
Weakly Indirect
Strategic
Hard
Intangible
Strongly Quantifiable
Non-Quantifiable Measurable
Figure 2. Incremental measurement of IS benefits: Starting from hard benefits and gradually incorporating intangible and/or indirect before studying strategic benefits.
The principle for benefits measurement presented here is intended to form part of the overall process of IS investment evaluation and appraisal. In the process of planning and managing their applications portfolio, companies contemplate a number of alternative investment proposals, some of which are approved and prioritised and some others are rejected. The approach advocated here focuses on measuring the intended benefits of a single IS investment. It is possible to extend this principle and to conceive an enterprise-wide modelling approach (Petrie 1992) wherein the benefits of alternative IS investments could be studied and compared directly. However, this is an ambitious research direction, out of the scope of the present paper.
4. The Technique: Business Process Modelling and Simulation
In practical terms, the aforementioned incremental approach to benefits measurement requires the support of appropriate tools that will comprise the following characteristics (Giaglis et al 1997): a)
Provide objective, quantitative estimates of expected benefits at required levels of aggregation. This derives from the stated aim of this approach.
9
b) Facilitate a modular approach to measurement, allowing the incorporation of additional IS benefits to existing models, as the evaluation exercise progresses. This is the essence of the principle articulated in the previous section. c)
Generate a laboratory-like experimentation facility on which benefits from proposed IS specifications can be isolated from other confounding factors and thus be subjected to independent experimentation and measurement. Indirect benefits can thus be disaggregated and studied in terms of their constituent parts. Models of intangible benefits can also be tried and tested. This is the nature of the technique that we propose in order to meet the above aim and principle.
These requirements can be effectively implemented by computer models of the information system and its surrounding environment (i.e. the business processes it supports or affects).
The term Business Process Modelling (BPM) has been used (Scholz-Reiter and Stickel 1996) to cover all activities relating to the transformation of knowledge about business systems into models that describe the processes performed by organisations. BPM can be facilitated by a number of computer supported-tools, each with its own strengths and weaknesses: a)
Static modelling tools (for example, flowcharting environments) provide the easiest-to-use mechanism for depicting business processes by providing representational formalisms which facilitate common understanding, consensus, and decision making. However, they do not possess any analysis capabilities and they cannot be used to provide quantitative data on business performance.
b) Dynamic modelling tools (for example, Business Process Simulation) incorporate the element of time in an attempt to capture the inherently complex, dynamic, and stochastic nature of business processes. They provide a powerful mechanism for conducting controlled experiments by systematically varying specific parameters and rerunning the model (Paul and Doukidis 1987, 1992). Moreover, they can provide quantitative data on the performance of the systems and processes being modelled. c)
Other tools (for example, CASE tools and workflow management environments) can also be used to a certain degree to model business processes and prospective information systems, but they are best
10
suited to the implementation phase of an information system (CASE) or to provide operational support to the related business processes (workflow).
Simulation modelling has been widely used within the context of various business problems. However, simulation models have traditionally been utilised to represent specific aspects of a business environment (e.g. finance or production) and they tend to be quite focused (see for example, Seila and Banks 1990, Hlupic 1994, Cohran et al 1990, Pruett and Vasudev 1990).
In the context of IS benefits measurement, Business Process Modelling and Business Process Simulation (BPS) can be a potentially effective mechanism for experimenting with alternative information system investments, assessing the benefits introduced by particular system and process configurations, and helping management reach informed decisions on the desirability of undertaking the investments.
MacArthur et al (1994) have identified simulation as a suitable technique to determine opportunities for changes in business processes in BPR projects. The same basic idea is followed by Giaglis and Paul (1996) who have developed a methodology for the application of discrete-event simulation in BPR projects. BPS has also been successfully applied to the case of Inter-Organisational Information Systems (IOS) redesign (Giaglis et al 1996) and specific requirements for modelling IOS, based on an EDI case study, have been derived.
In the context of IT investment appraisal, Wolstenholme et al (1993) have successfully applied System Dynamics. The technique is relatively easy to apply, accommodates subjective judgement and change, and leads to improved system specifications. However, it is not rigorous enough to support low level analysis and assessment. Moreover, detailed modelling is difficult or impossible given the existing formalism and software tools (Wolstenholme et al, 1993: p. 195).
Nissen (1994) has also presented a small scale simulation of business processes for selecting among alternative IT investments. This approach emphasises modelling of and experimentation with alternative
11
organisational processes for purposes of redesign and re-engineering. Prospective information systems are then designed to suit the chosen process schema.
Our approach follows the same general direction, providing a methodological framework for the application of Business Process Simulation (BPS) in IS benefits measurement exercises. According to the expressed intended goals and the original design of the system under investigation, the simulation would model the business processes expected to be affected and any relevant changes that the system is expected to introduce. As previously explained, modelling would begin with quantifiable areas directly affected while indirect or intangible effects would be added later, if needed.
Once implemented, the simulation would then be used as a tool in scenario planning sessions, during which decision makers would experiment with alternative future scenarios and obtain measurements of the various expected benefits of the proposed information system.
Such a simulation effects a ‘virtual’ (Nissen 1994) implementation of the proposed system. By measuring the performance of the relevant business operations with and without the information system, one can collect the necessary quantitative information needed to conduct further investment appraisal using established financial or other methods. Moreover, the simulation modelling process itself and the subsequent experimentation with alternative system and business configurations, constitute additional learning processes which can support a feedback mechanism adjusting the whole decision making process.
Simulation modelling is an effective technique for implementing the principle set out above, as it supports incremental and modular development of models (Law and Kelton 1991). Furthermore, simulation supports model reuse, modification and extension. Once built, simulation models can be reused across functions and organisations with limited adaptations. The advantages of the proposed incremental business process modelling and simulation approach can be summarised as follows: a)
Cost-effectiveness. Measurement need only continue until benefits exceed the estimated costs to an acceptable degree. Therefore, difficult to capture and analyse benefits may not actually need to be assessed, thus saving time and effort.
12
b) Learning. The incremental and analytic nature of the process generates substantial awareness and knowledge of the systems and the organisation among participants at every step. This knowledge can facilitate the transformation of vague statements about aggregate and/or intangible benefits into concrete, measurable system diagnostics. Thus, it enables the incorporation of subsequent, more complex, benefits and, at the same time, feeds back to the specifications of the system and the redesign of the respective business processes. Finally, simulation encourages a culture of continuous measurement of business performance (MacArthur et al 1994), thus enabling consistent identification of improvement opportunities. c)
Modularity. The proposed approach allows the modular development of business process models so that modelling effort is not wasted from one incremental step to another. Moreover, it is likely that models developed for one benefits measurement exercise may be reusable in other projects (for example, workflow management).
5. Business Process Simulation in Practice: The I.S.S.U.E Methodology
Although incorporating BPS in IS evaluation projects presents significant advantages discussed above, the sheer complexity of most real-world business process models and the specialist knowledge required for valid model construction, experimentation, and analysis of results, can jeopardise the success of the modelling exercise. In what follows we propose a methodological framework, consisting of certain steps and guidelines supporting the management of the measurement process (Figure 3).
I.S.S.U.E System objectives and characteristics
Initiation
Estimation
Simulation
Utilisation
Substantiation
13
Measured benefits for appraisal
Figure 3. The I.S.S.U.E methodology
Initiation The first step of the process is concerned with the identification of the business goals the proposed system is going to serve. On the basis of these goals, the expected benefits from the system should be identified and classified according to their dependence on the system and their quantifiability (Figure 1). Moreover, the business processes that are going to be affected by the information system should also be identified and demarcated. Finally, analysts and modellers should be able to pinpoint the performance variables that will have to be monitored and measured in order to assess the contribution of the system.
Simulation The next step involves the actual construction of the model. The processes identified in the previous phase are modelled in a simulation environment and the resulting model is used to measure system performance. The model represents the current mode of operation of the organisation (AS-IS model), i.e. without incorporating the changes the proposed investment is expected to deliver. The development of this model is essential for two main reasons: (a) to be used as a basis for validity checking (see next step); and (b) to provide a basis for comparison with any future change planned or introduced. The variables used for measuring system performance are those highlighted in the previous step.
Substantiation The substantiation stage (i.e. validation) is basically concerned with providing the analysts and the decision makers enough confidence that the model is an adequate and unbiased representation of reality and can therefore be safely used as a basis for decision making. Depending on the complexity of the model, validating the simulation model could involve the deployment of specifically designed techniques1.
)RU D PRUH FRPSOHWH GLVFXVVLRQ RQ YDOLGDWLRQ DQG YHULILFDWLRQ WHFKQLTXHV IRU VLPXODWLRQ PRGHOV VHH %DOFL
14
Utilisation When the model has been checked, it can then be used for experimentation with and analysis of the proposed IT investment. This involves the development of a second model representing the business processes under the planned information system-supported structure. This new model (TO-BE model) could be just an updated version of the AS-IS model in cases where the new information system is mainly targeted to supporting existing business structures. Where the new system is expected to be followed by transformations in the whole business environment, building the TO-BE model might require considerable effort.
Estimation Finally, after quantitative estimates of business performance have been obtained for both models (AS-IS and TO-BE), simulation experts and decision makers can proceed analysing the simulation output in order to decide on the extent of improvements introduced by the new system. The complexity of this analysis depends on model size and on the number of variables which will influence the final decision. Law and Kelton (1991) present a thorough introduction to simulation experimental design and model output analysis.
It must be noted that the I.S.S.U.E methodology is not a sequential process. It should rather be viewed as an iterative, spiral framework whereby the user may need to repeat previous steps until they are able to prove (or otherwise) the viability of the proposed investments. The incremental approach to benefits measurement dictates that, for example, if the users are not satisfied with performance improvement based on hard benefits alone, they may wish to go back and initiate a new I.S.S.U.E cycle in order to include (and measure) more complex benefits. Each step of the iteration builds upon the previous models and analyses and incorporates additional effects.
Given that almost every IS investment evaluation exercise is not performed in isolation, but is generally part of a broader project (e.g. BPR, IS strategy implementation etc.), many of the data and insights needed for I.S.S.U.E will have already been generated in the context of the overall project. The case study presented in the next section confirms this argument.
15
6. I.S.S.U.E in Practice: The Case of EDI Benefits Measurement
The following case was part of a wider project the goal of which was to evaluate the impact of Electronic Data Interchange (EDI) in three specific industry sectors in Greece, namely, supermarkets, pharmaceuticals and textiles and to propose appropriate strategies for EDI adoption. Within this project, the main aim of the work presented here was to provide quantitative measures of the potential operational benefits of EDI.
Initiation The adoption of EDI has been associated with a number of potential benefits including increased dataentry accuracy, improved trade efficiency, stock level reductions, customer and supplier lock-ins, competitive advantages, and so on (Reekers 1994, Srinivasan et al 1994). While the project participants (companies at various levels of the value chains in the three industries involved) were naturally interested in reaping all the benefits they could, the initial motivation for the introduction of EDI was the inventory reduction allegedly achieved through EDI adoption. The main problem was that, in the particular business context, improvements in inventory efficiency were not self-evident and there was a need for the identification of ways to transform these theoretical benefits into practical results that could be communicated to the companies involved.
The companies were willing to adopt the system if substantial reductions in inventory levels were achieved. Otherwise, they would consider alternative adoption strategies, including delayed implementation and/or alternative types of benefits. It was therefore considered a priority that the reduction of stock levels was the main variable to be measured by the evaluation exercise. Given this principal aim, we developed a model simulating companies’ behaviour at different levels of the value chain in the industries involved. The processes that we isolated for modelling were demand forecasting, inventory management, production planning, and materials requirements planning.
16
Therefore, the key performance indicator that was being monitored was the level of inventories. Of course, the simulation model also handled a number of additional variables, including actual consumer demand, demand forecasts, inventory plans, materials requirements plans and orders. However, inventory levels were decided to be the Key Performance Indicator (KPI), in the sense that it was the state of this variable that would primarily determine the acceptability of changes imposed on the model.
Simulation The simulation system modelled the behaviour of a three-level value chain consisting of: a)
Material suppliers producing raw materials and selling them to
b) manufacturing companies buying the materials and subsequently using them to produce final products. c)
Retailing companies buying final products and subsequently selling them to the consumers.
This structure is a model of the value chain for the clothing/textile sector. It was slightly modified to represent the other two industries2. The system incorporated a number of trading rules between the companies as well as a set of operational research models for the business processes involved. An example of the structure of the textile industry, together with the operational and trading models used, is illustrated in Figure 4.
2
The interested reader is referred to Mylonopoulos et al (1995) for further technical details of the project and the models used. 17
Materials Suppliers Inventory Management (Materials)
Production Planning
Manufacturers Inventory Management (Materials)
Materials Requirements Planning
Production Planning
Inventory Management (Products)
Retailers Inventory Management (Products)
Demand Forecasting
Figure 4. The Textile/Clothing Industry Value Chain and the Models Used
All companies in each stage of the value chain were assumed to be homogeneous in the way they plan their production, ordering and inventory control. For example, the estimation of the optimal level of inventory for all companies was based on the Economic Order Quantity (EOQ) model for a fixed reorder cycle inventory with backordering. Although the processes of each simulated firm were homogenous, each firm’s characteristics (such as market share and number of trading partners) were separately calibrated using firm-level data from industry statistics. Modelling different processes and scenarios for each simulated firm would, perhaps, give a more realistic picture of the sector. However, it would also add excessive complexity to the model without significantly improving the validity of results. Moreover, since the results were averaged over the sample of simulated firms, the effect of heterogeneous structures would ultimately be ignored in the analysis of simulation results.
Substantiation After the AS-IS (without EDI) simulation was developed, the model was run to generate estimates for inventory levels under the existing scenario. Before the results could be used for decision making, the model was tested to establish that it provided an adequate and satisfactory representation of the real world (i.e. the industries modelled). Model substantiation was divided into two parts: verification and validation. Verification is substantiating that the conceptual model is correctly transformed into an
18
executable program, while validation is substantiating that the model behaves with satisfactory accuracy and is consistent with the study objectives (Carson 1986, Sargent 1994).
To verify the AS-IS model, a number of generic and simulation-specific techniques were used (Law and Kelton 1991): gradual model development/testing, structured program walk-through, logical testing under simplified assumptions, and manual simulation. Validation was done against real-world industry data (for the three industries modelled) and the model was tested for conformity with seasonal and other trends observed in the industries concerned. After an iterative process of testing and modification, the model was accepted by the project participants as an adequate representation of the real-world system modelled.
Utilisation The most difficult part of the whole project proved to be the development of the TO-BE models and the subsequent experimentation. The difficulty was identifying how EDI would actually influence inventory reduction and incorporating this into the AS-IS simulation models.
The introduction of an ordering and invoicing EDI system does not by itself cause any immediate modifications to the inventory-related business processes we wanted to assess. However, by introducing a more broad and effective communication channel between trading partners (Milgrom and Roberts 1988, Bakos 1987, Srinivasan et al 1994), EDI can provide the necessary infrastructure for the introduction of formal planning methods that rely on advanced communication and co-ordination (Riggins and Mukhopadhyay 1994).
In this context, EDI improves communication by enabling more frequent estimation of realised and expected demand by retailers. If a fast and reliable form of communication, such as EDI, is introduced, retailing companies can re-estimate the expected demand more regularly and signal appropriate adaptations along the value chain. In this way, more efficient production and inventory planning is achieved. The proposed idea was implemented in the TO-BE model (which was actually an updated version of the AS-IS model rather than a new implementation).
19
Estimation We ran the two models that represented the industry operations with and without EDI for a period of four (simulated) years. The results were indicative of the potential of EDI as an enabling technology for inventory reduction. All companies achieved significant reductions in their material and product inventories. The reductions were more notable in the product inventories of retailing and manufacturing companies (average reduction percentages are shown in Table 2). Moreover, it was observed that the deviation of inventory levels was significantly reduced in all cases. This is in line with the theoretical expectation that a more sophisticated method of inventory management would result in lower and more stable inventories.
Category
Reduction
Retailers’ product inventory
25-40%
Manufacturers’ product inventory
20-30%
Manufacturers’ materials inventory
5-15%
Suppliers’ materials inventory
10-15%
Table 2. Typical Levels of Inventory Reduction
Based on the results, we were able to make recommendations to project participants for business process change initiatives to be initiated together with the system introduction if the expected benefits (inventory reduction) were to be realised in practice. This was indeed an interesting side-effect of using simulation for benefit measurement as we gained valuable qualitative insight into the operation of the system within the broader inter-organisational context.
The results obtained from the simulation models regarding expected inventory reductions were translated into monetary figures by the firms concerned in the project. It was concluded that the impact on inventories alone would justify the adoption of the EDI system as the benefits substantially exceeded expected costs. Therefore, it was not deemed necessary to initiate another I.S.S.U.E circle to explore other potential benefits of the system and include them into the simulation models.
20
7. Discussion and Conclusions
In this paper we focused on the issue of benefits measurement as one of the most important practical problems of IS/IT evaluation. Based on extant classifications of IS benefits, we proposed an incremental and iterative approach to measurement and outlined the characteristics of a technique (BPS) and a methodology (I.S.S.U.E) to support this approach. The case study presented has demonstrated the feasibility of the approach in a practical setting.
IS benefits measurement
Our approach is not intended to be a generic investment appraisal method. Rather, it provides assistance in obtaining the necessary quantitative and qualitative data needed for a typical investment appraisal method (such as Cost-Benefits Analysis).
The principles and techniques set out in this paper can be extended to take account of cost measurement. This can also be incorporated in BPS models, thus providing a more integrated approach to measuring the effects of information systems. In this context, further research could concentrate on enriching and enhancing the ‘tool sets’ available for IS evaluation “workbenches” (Serafeimidis and Smithson 1994).
Incremental Measurement
The incremental, iterative approach to benefits measurement that we advocate promotes learning, feedback and modular model development in a cost-effective process with clear exit criteria.
It can be argued that the unanticipated effects of an information system may prove to be more important than the intended goals, so that they must also be sought and measured. Side effects can be attributed to pure chance, to improper analysis and design, to poor implementation or inadequate management of change. Such sources of uncertainty generate risks for IT investment decisions that can rarely be
21
completely avoided. Risk analysis and management thus becomes another important part of IS evaluation (Griffiths and Willcocks 1994), especially in higher levels of the project ladder. Risk analysis and cost measurement are however out of the scope of this paper. Nonetheless, the technique proposed involves considerable analysis and experimentation, additional to the system’s analysis and design, which can identify other areas of the business which affect or are affected by the success or failure of the system.
I.S.S.U.E in practice: Feasibility and practical usefulness
The EDI simulation exercise dealt with a limited set of IS impacts (i.e. inventory levels and changes in a limited number of related business processes). The potential benefits were measured by comparing the performance of the business system (the three industries) with and without the information system (EDI). This foundation of the basic workings of the industries can subsequently be used as the basis of models of more aggregate effects to study a broader range of impacts.
The greater the demand for modelling accuracy and the more extensive and strategic the benefits sought, the greater the reliance on detailed and complex modelling. This increases the requirement for expert human resources and consequently renders such exercises more costly. The question, therefore, is whether it is worth undertaking such simulation projects or proceed with the original IT investment bearing the risk of uncertain benefits. There are technical and organisational solutions that can reduce modelling and measurement costs. Object oriented techniques allow the development of reusable and re-configurable simulation components that can be re-deployed in similar simulations. Moreover, specialist suppliers of ‘benefits measurement by simulation’ software and services (for example, through the development of special-purpose simulation software environments) may be able to achieve significant scale economies in building and applying this technique.
Further research efforts can be aimed at assessing the functionality and usability of the proposed approach (incremental measurement, BPS models) in comparison to other IS evaluation and benefits management approaches. Furthermore, it would be interesting to apply the approach to more complex IS evaluation projects in order to assess the increasing complexity and cost introduced when the
22
I.S.S.U.E circle has to be repeatedly followed before a decision on the desirability of the project is made. Finally, “softer” issues related to IS investments, such as organisational change management or risk management warrant further attention with a view to incorporating them into I.S.S.U.E.
Acknowledgements
The work described in this paper was partially funded by a EC/TEDIS II project (main contractor: Greek EDI Awareness Centre-EDIGRAC) and a Telematique project (main contractor: Intrasoft SA). We also wish to thank the three anonymous referees for their valuable comments and suggestions.
References
BAKOS Y J (1987) Interorganisational Information Systems: strategic implications for competition and co-operation. PhD Thesis, Massachusetts Institute of Technology. BALCI O (1994) Validation, Verification, and Testing Techniques Throughout the Life Cycle of a Simulation Study. Annals of Operations Research 53, 121-173. BALLANTINE J A, GALLIERS R D and STRAY S J (1994) Information System/Technology investment decisions - the use of capital investment appraisal techniques in organisations. In Proceedings of the First European Conference on IT Investment Evaluation, Henley, England, September, 148166. BANKER R D, KAUFFMAN R J and MAHMOOD M A (1993) Strategic Information Technology management: perspectives on organisational growth and competitive advantage. Idea Group Publishing. BROWN A (1994) Appraising intangible benefits from Information Technology investment. In Proceedings of the First European Conference on IT Investment Evaluation, Henley, England, September, 187-199.
23
BRYNJOLFSSON E (1993) The productivity paradox of Information Technology. Communications of the ACM 36(12), December, 67-77. CARSON JS (1986) Convincing Users of Model’s Validity is Challenging Aspect of Modeller’s Job, Industrial Engineering, 18, June, 74-85. COHRAN M, RICHARDSON J, NIXON C (1990) Economic and financial simulation for small business: a discussion of the small business economic, risk, and tax simulator. Simulation 54(4), 177-188. CORNFORD T, DOUKIDIS G I and FORSTER D (1994) Experience with a structure, process and outcome framework for evaluating an Information System. Omega 22(5), 491-504. DAVENPORT T H (1993) Process Innovation: Reengineering Work Through Information Technology. Harvard Business School Press, Boston, Massachusetts. FARBEY B, LAND F and TARGETT D (1992) Evaluating investments in IT. Journal of Information Technology 7, 109-122. FARBEY B, LAND F and TARGETT D (1993) How to assess your IT investment: A study of methods and practice. Butterworth-Heinmann, Oxford. GIAGLIS G M and PAUL R J (1996) It’s Time to Engineer Re-engineering: Investigating the Potential of Simulation Modelling in Business Process Redesign. In Scholz-Reiter B. and Stickel E. (Eds.), Business Process Modelling, Springer-Verlag, Berlin, 313-332. GIAGLIS GM, PAUL RJ and DOUKIDIS GI (1997), Simulation for Intra- and Inter-Organisational Business Process Modelling, Informatica, 21(4), 613-620. GREGORY A J and JACKSON M C (1992) Evaluation methodologies: a system for use. Journal of the Operational Research Society 43(1), 19-28. GRIFFITHS C and WILLCOCKS L (1994) Are major Information Technology projects worth the risk?. In Proceedings of the First European Conference on IT Investment Evaluation, Henley, England, September, 256-269. GRINDLEY K (1991) Measuring benefits. In Managing IT at board level, Price Waterhouse, 179-195. HITT L and BRYNJOLFSSON E (1994) The three faces of IT value: theory and evidence. In Proceedings of the 15th International Conference on Information Systems, Vancouver, Canada, 263-277. HITT L and BRYNJOLFSSON E (1996) Productivity, business profitability and consumer surplus: Three different measures of information technology value. MIS Quarterly 20(2), 121-142.
24
HLUPIC V (1994) A comparative analysis of six manufacturing simulators. In Proceedings of the 4th International Conference on Information Systems Development, Bled, Slovenia, 693-696. HOGBIN G and THOMAS D V (1994) Investing in Information Technology: Managing the decision making process. McGraw Hill, London. KELLEY M R (1994) Productivity and Information Technology: the elusive connection. Management Science 40(11), 1406-1425. LAW A M and KELTON D W (1991) Simulation Modelling and Analysis, 2nd ed. McGraw-Hill, New York, NY. LUBBE S (1994) The effect of investing resources in Information Technology on the performance of an organisation. In Proceedings of the Second European Conference on Information Systems, Netherlands, May, 569-589. MACARTHUR P J, CROSSLIN R L and WARREN J R (1994) A strategy for evaluating alternative information system designs for business process reengineering. International Journal of Information Management 14, 237-251. MCBRIDE N and FIDLER C (1994) An interpretative approach to justification of investment in executive information systems. In Proceedings of the First European Conference on IT Investment Evaluation, Henley, England, September, 16-26. MILGROM P and ROBERTS J (1988) Communication and inventory as substitutes in organising production. Scandinavian Journal of Economics 90(3), 275-289. MYLONOPOULOS N A, DOUKIDIS G I and GIAGLIS G M (1995) Assessing the expected benefits of Electronic Data Interchange through simulation modelling techniques. In Proceedings of the Third European Conference on Information Systems, Athens, Greece, June, 931-943. NISSEN M E (1994) Valuing IT through virtual process measurement. In Proceedings of the 15th International Conference on Information Systems, Vancouver, Canada, 309-323. ORMEROD R J (1996) Information Systems Strategy Development at Sainsbury’s Supermarkets Using “Soft” OR. Interfaces 26(1), 102-130. PARKER M M, BENSON R J and TRAINOR H E (1988) Information Economics: linking business performance to Information Technology. Prentice-Hall International. PAUL R and DOUKIDIS G I (1987) Artificial Intelligence aids in discrete-event digital simulation modelling. IEE proceedings 134(4), 278-286.
25
PAUL R and DOUKIDIS G I (1992) Artificial Intelligence and Expert Systems in Simulation Modelling. In Artificial Intelligence in Operational Research (DOUKIDIS G I and PAUL R, Eds.), Macmillan, 229-238. PETRIE, C.J., Jr. (Ed.) (1992) Enterprise Integration Modeling, MIT Press, Cambridge, MA. POWELL P (1992) Information Technology evaluation: is it different?. Journal of the Operational Research Society 43(1), 29-42. POWELL TC and DENT-MICALLEF A (1997) Information technology as competitive advantage: The role of human, business and technology resources. Strategic Management Journal 18(5), 375-405. PRUETT J and VASUDEV V (1990) MOSES: manufacturing organisation simulation and evaluation system. Simulation 54(1), 37-45. REARDON J, HASTY R and COE B (1996) The effect of information technology on productivity in Retailing. Journal of Retailing 72(4), 445-461. REEKERS N (1994) Electronic Data Interchange use in German and US organisations. International Journal of Information Management 14, 344-356. RIGGINS F J and MUKHOPADHYAY T (1994) Interdependent benefits from interorganisational systems: opportunities for business partner reengineering. Journal of Management Information Systems 11(2), 37-57. SARGENT RG (1994) Verification and Validation of Simulation Models, In The Proceedings of the 1994 Winter Simulation Conference, Lake Buena Vista, Florida, 77-87. SAUER C (1994) A model of the Information Systems evaluation process: a synthesis of politicorational and interpretivist views. In Proceedings of the Second European Conference on Information Systems, Netherlands, May, 805-822. SCHOLZ-REITER B and STICKEL E (Eds.) (1996) Business Process Modelling. Springer-Verlag, Berlin. SEILA A and BANKS J (1990) Spreadsheet risk analysis using simulation. Simulation 55(3), 163-170. SERAFEIMIDIS V and SMITHSON S (1994) Evaluation of IS/IT investments: understanding and support. In Proceedings of the First European Conference on IT Investment Evaluation, Henley, England, September, 135-147. SMITH H A and KEEN J D (1993) How does Information Technology affect business value? A reassessment and research propositions. Canadian Journal of Administrative Sciences 10(3), 229240. 26
SRINIVASAN K, KEKRE S and MUKHOPADHYAY T (1994) Impact of Electronic Data Interchange on JIT shipments. Management Science 40(10), 1291-1304. STRASSMAN P A (1990) The business value of computers. The Information Economics Press, New Canaan, Connecticut. VERYARD R (1991) The economics of Information Systems and software. Butterworth-Heinmann. WHEELER F P (1994) Executive support: current benefits and future opportunities. In Proceedings of the First European Conference on IT Investment Evaluation, Henley, England, September, 71-78. WILLCOCKS L (1992) IT evaluation: managing the catch-22. European Management Journal, 10(2), 220-229. WISEMAN D (1992) Information economics: a practical approach to valuing Information Systems. Journal of Information Technology 7, 169-176. WOLSTENHOLME E, HENDERSON S and GAVINE A (1993) The evaluation of management information systems: A dynamic and holistic approach. Wiley.
27
