second part of method consists of establishing the business value based on the evaluation of the ... The existing assessment methods e.g. Software Metrics (Fenton and ..... Originally switch action scheduling was thought to be beyond the.
Paper #153
Business Value Evaluation of IT Systems: Developing a Functional Reference Model Magnus Gammelgård, Per Närman, Mathias Ekstedt, Lars Nordström Dept. of Industrial Information and Control Systems School of Electrical Engineering KTH - Royal Institute of Technology Osquldas vag 12, SE-100 44 Stockholm, Sweden Address {magnus | pern | mek101 | larsn}@ics.kth.se
Abstract In this paper an approach to develop and refine a functional reference model for IT-systems is presented. Such reference models form a fundament on which to evaluate the business value of IT systems. The approach involves scrutinizing the functional reference model with respect to correctness, completeness, granularity and measurability. In the particular case of this study, the functional reference model represents systems functionality needed to support asset management processes in industries such as electric energy production and distribution. It is based on the IEC 61968 standard and its Interface Reference Model, IRM, which has been refined in the process described in the paper. The refinement included a comprehensive field study, interviewing experts in the field of asset management at a large European energy company, as well as vendors of asset management systems. A general method for evaluating IT investment scenarios is outlined in the paper and the refined functional reference model is an important component in this method. Creation of this IT evaluation method is an ongoing research project, where the functional reference model forms a central part. The purpose of the IT investment evaluation method is to give decision makers a tool to evaluate potential investment scenarios with respect to the value the investment would generate to the business. The first part of the method consists of evaluating the technical quality of the system scenarios. Technical quality is divided into functional and non-functional attributes, where the functional reference model is used for the functional assessment. The second part of method consists of establishing the business value based on the evaluation of the technical qualities. This paper thus focuses on the first part, and in particular the development of the functional reference model which is a central piece in the IT investment evaluation method.
Background to Research This paper presents results from an ongoing research project focusing on the evaluation of business value of IT investment scenarios. The project is part of a comprehensive research program, the Enterprise Architecture Research Program EARP, at KTH - the Royal Institute of Technology in Sweden. EARP exploits the discipline of Enterprise Architecture as an approach for managing an organization’s total information system portfolio. The overall goal of the research program is to provide top IT management with architecture-based tools and methods for planning and decision making with regard to enterprise-wide information systems.
1
Purpose of the research Today we experience a lack of support for estimating the business value of different potential future IT system scenarios. The existing assessment methods e.g. Software Metrics (Fenton and Pfleeger 1998) (Zuse 1998) are often technical. Their focus is on either functional or nonfunctional aspects rather than considering the complete spectrum of support from IT systems in terms of process efficiency, organizational flexibility, and product quality, etc. Other methods e.g. Return on Management (Berghout and Renkema 2001), Information Economics (Parker et. al. 1988), McFarlan Matrix (McFarlan 1984) consider the business related aspects of investments but fall short of concretely relate the systems to the implications. Thus there is a gap in many methods between business and IT stakeholders. The purpose of this research project is to develop a method that indicatively assesses IT investments in terms of their business value. Another neglected aspect of assessments methods is the credibility of the produced results. All results are not of the same quality and cannot be relied on to the same extent. The proposed method takes this into account and estimates the credibility of the estimated business value score.
Scope of the paper One of the cornerstones for achieving value to the business is that the IT system features the appropriate functionality. But in order to be able to assess the business value it is important that the set of IT functions is expressed in an implementation independent way. This is so, both to avoid biasing the value towards a specific implementation, but also to ensure that the set of IT functions is complete and covers the whole spectrum of the business that shall be supported. This paper focuses on the functional aspects of the total IT investment evaluation method. More precisely the paper describes how to develop a relevant functional reference model for a specific business area.
Industrial setting The presented method for developing a reference model has been applied to asset management processes in electricity production and distribution. The work has been conducted in cooperation with a large European energy company, primarily involved in generation, transmission, and distribution of heat and electricity to customers in Northern Europe. The business is dependent on complex and capital-intensive assets such as turbines, boilers, transmission, and distribution lines. Today, the company uses an extensive amount of different information systems within the different business units to support their asset management processes. An important question for the top IT management is hence to ensure that the most efficient and effective system support is employed in the company.
Outline The next chapter serves as motivation and presents the overall IT investment evaluation method. The following chapter, constitutes the main contribution of this paper, and explains the process used for developing the functional reference model and the results from that work.
Overview of the IT investment evaluation method As stated, the purpose of the overall investment method is to produce a first indicative assessment of the value of IT investment scenarios for a particular business area. A scenario
2
could consist of changes to, or implementation of, a single IT system or a combination of IT systems intended to support a certain part of the business, e.g. sales, production, or asset management. The results of the method are intended to be used to support strategic information systems planning. Before a scenario actually is realized a more thorough process of development or acquisition needs to be set up that produces requirements specification, design documents etc. Due to inherent difficulties in directly assessing business value of technically oriented IT investment scenarios, the IT Investment Evaluation Method is divided into two parts (see Figure 1): Assessment of the technical qualities of the IT investment scenarios, and Estimation of the business value of the IT investment scenarios.
Part 1 Assessing the Technical Qualities When attempting to assess the overall technical quality of an IT system scenario, the first problems encountered is what is to be assessed, and how. An established way of categorizing IT system qualities, found within software and systems engineering, is to separate the functional from the non-functional attributes (Bass 1999) (Maier 2000). Part II: Establishing the business value
Functional Quality
Business Organisation Input
Strategy formulation & Planning
... Business Organisation structure Business Organisation Output
Business Value
Part I: Assessment of the technical qualities Investment Investment Scenario X Scenario Y
Short cut analysis Asset faliure history Work task planning Work order status tracking …
Non-Functional Quality Security Interoperbility Data Quality Reliability …
2 3 1 1 …
2 2 2 1 …
1 2 4 2 …
1 3 3 3 …
Paper scope
Figure 1. Overview of the Framework for the IT Investment Evaluation Method. The dashed box illustrates the scope of the paper. Functional Quality. The functionality relates to the kind of business that the IT system should support e.g.: control and supervision of an industrial process, customer relations, human resources, etc. The larger the conformity between the functionality of the IT system and the activities of the business, the higher the score would be with respect to functional quality of the system. However, this can be difficult to implement; exactly what are the various functions that the particular business considers relevant to have IT support for? To answer this question a functional reference model is needed. Prefabricated vendor independent reference models are oftentimes found within standardization organizations for different industries, such as power, pulp and paper, petroleum, and the well known U.S. Federal Enterprise Architecture (EOPUS 2005) for the U.S. government.
3
A method for adapting such a standardized functional reference model into a relevant reference model for a specific organization is the focus of this paper. The particular business area considered here is asset management in the electric power industry. Non-Functional Qualities. A perfectly functionally business-aligned IT system does however not necessarily imply a high quality system. If the system is failing to be reliable, secure, or user-friendly, the system might not be of much value to the business anyway. Therefore quality attributes is the second cornerstone of the technical assessments in the IT investment evaluation method. Altogether the method incorporates eight condensed quality attributes that need to be assessed; availability and reliability, data quality, information security, interoperability, modifiability, performance, safety, usability and user productivity (Gammelgard et. al. 2006) (Lindstrom et. al. 2005). Even though the non-functional qualities as such are constant over different business domains, a difficulty lies in the fact that they all are large and complex topics in themselves. In order for the method to produce relevant results, all eight nonfunctional qualities have to be condensed into unambiguous definitions that reflect the state of the art in each discipline. One problem with this undertaking is that the final definitions may end up being quite vast; using e.g. four of the most renowned information security standards as a base for a definition of that quality, we end up with more than 1300 different sub qualities (Johansson 2005). Fortunately however, only a subset of these sub-qualities needs to be assessed for serving the purpose of this method. Assessing the qualities for the system scenarios. When a functional reference model has been developed and the measures of the non-functional qualities established, the technical quality of each system scenario can be measured. This is done by determining what parts of the functional reference model each scenario fulfils as well as how secure, user-friendly, etc. it is, i.e. assessing the non-functional qualities. The measurements are later used when deriving the business value of each scenario.
Part 2 Establishing the Business Value Having a high-quality assessment of the technical characteristics of an IT system scenario is however not sufficient from a business perspective. It is for instance very likely that the qualities measured are of different value to the business. Some IT system functions might be of outmost importance whereas others are less vital. The same is true for the non-functional qualities. What is important and what is not depends on the particular case. Or in other words; what aspects of the IT system scenario provide value to the business? This is the relevant question to ask when IT investment scenarios should be compared. Business Value. In order to be able to assess the business value, it first needs to be defined in more detailed. From a large literature survey covering more than 80 references (Gammelgård et. al. 2006), 23 different types or categories of business value achieved from IT have been identified. These include categories related to the various parts of the business, such as: the input to the organization, i.e. supplies and materials, outputs, i.e. improvements in product or service quality, the structure of the organization, e.g. improved flexibility and communications, the resources used, e.g. better information available, the actors of the organization, e.g. better decision-making, and benefits related to improved organization culture. All the 23 categories represent generalized types of business value. The purpose is to express the benefits of IT in terms understood by business managers. In order to achieve a business relevant overall assessment the benefits are then prioritized according to what is most important in the specific business.
4
Business value
Linking technical quality to business value. The last step before actually calculating the business values is to determine the relations between the technical properties of the systems scenarios and the categories of business value. That is, for the specific situation at hand, to determine what the actual business values are of having certain functional and non-functional qualities. In the method, the functions found in the functional reference model are linked to the 23 dimensions of business value, see Figure 1. Hence, the relation between each function in the reference model and each dimension of business value is derived, i.e. it is assessed how much each function contributes to each dimension of business value. The strengths of the relations, i.e. the size of the contributions, are assessed by experts on the business, not IT specialists. The influences of the non-functional qualities are also considered, in terms of that they influence the strengths of the relations between functions and categories of business values. The Final Result with Credibility. Put together, the final result can now be 100 calculated as business value for each IT investment scenario. For decision support, however, the calculated score is in itself insufficient. There is a big difference if the final results are based on guesses about the system qualities and business value correlations made by people that just 0 happened to be available, or if thorough Scenario 1 Scenario 2 Scenario 3 Scenario X interviews were made with experts on each Figure 2. Final result of the IT specific matter. Moreover, a complete investment evaluation method. assessment requires an extensive amount of different sub-qualities to be assessed. Since experts are not always available and since time and resources are limited, the method needs to compensate for the credibility of the final result. The final result of the method is thus not as single business value score, but also the value of the corresponding confidence level and confidence interval, as illustrated in Figure 2.
Refining a Functional Reference Model This second part of the paper describes the initial steps of the proposed investment evaluation method. That is to find and refine a functional reference model. This reference model is, as stated, used both for assessing functional qualities of the investment scenarios as well as for deriving the links to business value. Hence it is a central component in the overall method and needs to be established before the other steps outlined above can be taken.
Requirements on the refined model In order to be useful and relevant for the purpose of assessing the business value of IT systems a functional reference model must be: Complete. If the model spans the subject matter it is complete. A complete model describes what it is intended to describe, and there are no vital parts missing in the description.
5
Correct. Correctness means that the model accurately describes the functions in the model. Accurate in this context means that the definitions given in the model are roughly lexical definitions, i.e. they correspond to the common usages of the terms. Accuracy in the model will be assured by using multiple sources that are deemed reliable. The sources should be compared and cross-referenced, and descriptions of functions on the lowest level will be based on sources published in scientific journals and/or preliminary interviews with in the business. Measurable. It must be possible to actually go out and find information about the functions in the IT-systems. Appropriately granule. It must be easy to determine if the function exists and that that it is self contained and limited.. For instance the functions cannot be to extensive containing many sub-functions. It is for example not relevant to include e.g. a “plan all asset management” function since it is very different complex consisting of many subfunctions. In this paper, these requirements have been applied on a functional reference model for asset management within energy distribution and generation. The process used to refine the model is presented below. However, the requirements as such are general and can be applied in order to refine any functional reference model. Also the general process steps, i.e. to find and then refine a reference model using the requirements are generic. Of course, if no standard model is available, the refinement process needs to incorporate the development of the functional model e.g. by aggregating a number of smaller models or developing from scratch.
Finding a model to refine As stated in the introduction there exist a number of general functional reference models for a number of industries and types of businesses. By basing the reference model on a standard reference model not only time is saved. The credibility of the reference model is also strengthened.
Figure 3. The structure of the Interface Reference Model. The functions are examples from (IEC 2003). Fortunately in this study, a suitable standard reference model was available. The IEC 61968 set of standards (IEC 2003) have been developed to facilitate integration of distribution 6
management systems in the electric power industry. This set of standards cover several business areas within a utility and related IT-systems. The standards are all based on an Interface Reference Model, IRM. The IRM specifies a list of business functions, which exist in electrical distribution utilities. These business functions are broken down into subcomponents called subbusiness functions. These sub-business functions are in turn supported by IT-system functionality grouped into abstract components. These abstract components are a first approximation to be used as measurement points for functional fulfillment. Using the IRM from IEC 61968 as a starting point has several benefits. First, it is reasonably vendor independent since several different major vendors have participated in the development of the standard. Second, the standard deals with the area of Distribution Management Systems, which contains, but is not limited to, applications for asset management activities. This property together with the fact that the IRM was developed fairly recently (2003) means that the outer boundaries of the relevant area of inquiry will be present in the model. In this study the IRM was extended with functions from SAP (SAP 2005), Oracle (Oracle 2005), and Powel (www.powel.com) since systems from these vendors were in use in the organization. Additionally, company specific business documentation, i.e. Business Architecture Documentation was also included.
Refining the reference model. When a standard model is found, or a base-model has been developed, the refinement process can start. A standard model such as the IRM is a good foundation for the reference model, but would not be possible to use directly. It needs to be established that all the requirements are met. The refinements were performed in five phases. A large effort was made in order to find the person expected to provide the most credible result for each question. Mostly, the refinements were made by interviewing experts in the business. In addition, vendors were consulted, especially for the purpose of covering the completeness criterion since the business might run the risk of being unaware of any new innovative features of the systems. Initial scoping. A first meeting was held to determine the scope of what should be considered as complete under the completeness requirement. This was done at a high abstraction level and the IEC business function level was used for scoping. Overall completeness and correctness. People from all levels of the business were asked to give their opinions regarding the completeness and the correctness of the model. Nine respondents were used in total. It should be noted that the respondents were selected primarily based on their knowledge and understanding of the business. They also came from different hierarchical levels of the business hence targeting different levels of the model, primarily the two highest. The purpose of this phase was to identify any major problems while the next two phases constituted more detailed investigations of the lowest level of the model. In addition documents of e.g. processes and business function specifications were reviewed. Also three respondents from systems suppliers were consulted regarding the completeness and correctness since they possessed both business and system knowledge and experiences. Completeness and correctness on a detailed level. This was the most extensive refinement round in which all the lowest level functions of the model (abstract component) were closely reviewed. Sixteen respondents were asked about the completeness and the correctness of the model. The respondents possessed detailed knowledge about the business needs and in
7
aggregation covered the whole functional area of the model. In addition all suggested measurements were reviewed with respect to correctness. Verifying granularity. A number of respondents were asked about the granularity of the functions at all levels in the model. These persons were primarily used for verifying the indications given by the respondents when completeness and correctness were investigated. Testing measurability. A small test was made with respect to the measurability. System documents from one particular IT-system were reviewed with respect to a smaller set of the measurements points developed in the model.
Examples of refinements In total there were over 25 different changes made to the IRM in order to create the refined functional reference model. Given the limited space only examples are presented here. The majority of changes were related to the completeness and correctness requirements. It should be noted that this also meant excluding parts of the IRM in order to adapt the model to the situation at hand. Clarification of definitions and finding correct definitions of parts of the model was an important part of the work. Also, verifying that the same things were not treated in multiple parts of the model proved to be a part of the process. Concerning granularity, it became apparent that the overall division of the IRM worked well. Using the abstract component level as measurement level also proved suitable for measuring, providing results with a reasonable effort. Table 1 shows three examples of refinements that were made. The changes were derived from both documents and interviews, where the respondents often suggest various improvements to the model. These suggestions were then taken into consideration in the analysis and found suitable with relation the requirements, included in the model, with any necessary modifications. Table 1. Examples of refinements to the model. Change To include the business sub-function “Switch Scheduling” in the model
Change type Completeness
To change the content of the business subfunction Premises.
Granularity/ Correctness
To only include Services and not materials in the “procurement”abstract component.
Correctness
Comments Originally switch action scheduling was thought to be beyond the scope of any asset management application. But it was described in the business process documentation. Also respondents indicated that many maintenance activities of both hydropower plants and distribution networks require either water or electricity to be switched off. It was the original intention to just include the premises owned by the utility in this sub-function, but it should also include all real estate matters connected to customers and land owners affected. This was reasonable for the distribution case since the network affects land owners and customers external to the company. The procurement functions in the company mostly deals procures services. Also, materials purchased are handled in the materials management sub-function
The resulting refined functional reference model. Figure 4 shows the results of the refinement process. Compared to Figure 3, an abstract component has been added and also a number of measurement points. These are used to define the abstract component and to measure the actual functionality of the system scenarios. Hence they are relevant for the measurability of the functional reference model. 8
What is not shown in the figure, but exemplified in the table above are refinements on higher levels, primarily related to correctness and completeness. On the business function level three out of the original five business functions in the IRM where included in the refined reference model and on business sub-function level 13 out of 34 original business sub-functions where used. Some major parts that were excluded from the IRM was e.g. Network operations, since the operations of the electricity grid and the hydropower plants are not considered a part of asset management. The meter reading business sub-function was considered not to be asset management, but rather related to operations, for the distribution company. For the production it is of course not even applicable. On the abstract component level even more could be excluded. The final refined reference model for the asset management process within the electrical distribution has 82 abstract components the corresponding model for the production unit has 57 abstract components. Field Recording & Design
Business Function level
Business Sub-Function Level
Maintenance & Inspection (MAI)
Abstract Component Level
Work Scheduling (SCHD)
Construction & Design (CON)
Field Design
Importing work order feedback from field crews (results, comments)
Field Recording & Design (FRD)
Field Inspection Results
Work Dispatch (DSP)
Crew Time Entry
Added function on the abstract component level
Actual Materials
Actual Equipment
Function to display an electronic checklist of the agreed delivery
Examples of added measurment points to abstract components
Function to display data from asset repository in the field, either by being online or by storing it.
Function to connect the "delivered"-status of the material to payment of the delivery
Figure 4. The refined reference model, business sub-function level. In the picture examples are shown of changes made in the refinement process.
Summary and Further Work The functional reference model is one important part in the overall method for assessing the business value of IT investment scenarios. Using the process for developing and refining a functional reference model described above, starting with the Interface Reference Model, IRM from the IEC 61968 standard, a functional reference model for asset management within the electric power industry has been developed. In the process, the IRM was refined stepwise with respect to correctness, completeness, granularity and measurability. By applying these four criteria the resulting reference model forms a sound basis for functional fulfillment measures of the investment scenarios as well as a basis for relating functionality to business value. Furthermore, since the refinement process heavily relies on the verification from the business
9
point of view, rather than basing the functions on existing applications the functional reference model is intended to reflect the businesses’ needs. Based on the reference model, measurements of the functional fulfillment of different investment alternatives will be made in the next step of the IT investment evaluation method. In addition, the business value of the functions can be established. When this is done the final step of the method can be taken when deriving the aggregated business value of an investment alternative by propagating the measurements via the links to business value.
References Bass, L., Paul Clements, Rick Kazman, Software Architecture in Practice, Addison Wesley Longman, 1999 Berghout E. & Renkema T-J., “Methodologies for IT-investment Evaluation: A Review and Assessment”, Information Technology Evaluation Methods & Management, van Grembergen V., Eds., Idea Group Publishing, 2001 Executive Office of the President of the United States (EOPUS), FY07 Budget Formulation FEA Consolidated Reference Model Document, May 2005, http://www.whitehouse.gov/omb/egov/documents/CRM.PDF Fenton, N.E. & Pfleeger, S.L., Software Metrics – A Rigorous and Practical Approach, Chapman & Hall, 1998 Gammelgård M., Lindström Å, Simonsson S,. ”A reference model for IT management responsibilities”, submitted to the 14th European Conference on Information Systems, (Gothenburg, Sweden, June 12-14, 2006) IEC, International Standard IEC 61968 – Application integration at electrical utilities – System interfaces for distribution management – Part 1: Interface Architecture and General Requirements, Technical committee 57, work group 14, 2003. Johansson E. et al. Assessment of Enterprise Information Security – An Architectural Theory Diagram Definition, Proceedings of the 3rd Annual Conference on Systems Engineering Research (CSER), March 23-25, 2005, New York, NY, USA, March 23-25, 2005 Lindström Å, Gammelgård M., Simonsson M, Jonsson N., “A method to assess the enterprisewide IT resources for performance and investment justifications”, Proceedings of the 2005 Conference on Systems Engineering Research, New York, NY, USA, March 23-25, 2005 Maier, M. and E. Rechtin, The Art of Systems Architecting, 2nd ed., CRC Press, 2000.McFarlan, F.W., “Information technology changes the way you compete”, Harvard Business Review, May-June, 1984. Oracle, “Oracle Enterprise Asset Management 11i”, Oracle White Paper, 2005 Parker M.M., Benson, R.J., Trainor, H.E., Information Economics: Linking Business Performance to Information Technology, Prentice-Hall 1988. SAP AG, SAP Business Solution Maps, 2005-09-01, http://www.sap.com/solutions/ businessmaps/038F6B910E694218A03916647EF49E3B/index.epx, 2005 Zuse H. A Framework of Software Measurement, Walter de Gruyter, 1998.
Biographies All authors are researchers at the department of Industrial Information and Control Systems and are all active in the area of enterprise architecture for support of information systems planning in industrial domains such as power system operation and maintenance. 10
