Performance Indicators in Software Project Monitoring

SISY 2012 • 2012 IEEE 10th Jubilee International Symposium on Intelligent Systems and Informatics • September 20-22, 2012, Subotica, Serbia

Performance Indicators in Software Project Monitoring: Balanced Scorecard Approach Lj. Kazi, B. Radulovic and Z. Kazi University of Novi Sad, Technical faculty “Mihajlo Pupin”, Zrenjanin, Serbia [email protected], [email protected], [email protected] Abstract — Balanced scorecard is one of most used methodologies for performance measurement in enterprises and non-profit organizations. It presents a general framework that has been applied more precisely by many scientists and practitioners. It has been adapted to particular performance measurement systems. These monitoring systems are based on collecting data and analysis of these data. This paper presents overview of implementing balanced scorecard in IT project management. Framework for software project success monitoring, based on PRINCE2 methodology and balanced scorecard is proposed. Key performance indicators as measures to be performed upon data collected during software project implementation are defined. This way, analytics to be performed upon data according to specified key performance indicators, needed for decision support is enabled.

I. INTRODUCTION Software development is very much organized as a strategic management process, especially with new approaches, such as agile methodology [1]. The project starts with general idea of the final product functionality defined in requirements definition phase. During the process of implementation, requirements are divided to steps and assigned to teams. In implementation phase, details on functionality are refined. Final product is delivered first at prototype and later as a final version module, according to end-user specifications. In project monitoring, success of is measured by process and product characteristics. There is similarity of strategic management process to software project management process. Top managers start with general idea on the strategy and split it to tasks to be done. For each of them, success is to be measured. If software project success is defined as a strategic goal (since target quality characteristics of final product usually are not precise in advance, but only generally specified), this implies using strategic management methods and tools. Strategic management methods are used for specification of key performance indicators, while tools are used for monitoring measurements results upon these indicators and estimation of quality regarding these measurements. “Performance measurement is evaluating how well organizations are managed and the value they deliver for customers and other stakeholders" [2]. The significance of performance measurement is described in well-known sentences: "if you can't measure it, you can neither manage it, nor improve it" and “what gets measured gets done" [3]. Performance measures should be [4]: meaningful, unambiguous and widely understood; owned and managed by the teams within the organization; based on high level of data integrity; such that data collection is

978-1-4673-4750-1/12/$31.00 ©2012 IEEE

embedded within the normal procedures; able to drive improvement; linked to critical goals and key drivers of the organization. Four key steps in a performance measurement framework [4] are: the strategic objectives of the organization are converted into desired standards of performance, metrics are developed to compare the desired performance with the actual achieved standards, gaps are identified, improvement actions initiated. Business performance management (BPM) can be described as a series of business processes supported by technology designed to optimize both the development and the execution of business strategy [5]. BPM standards group defined a BPM framework by several functional steps: an organization define strategic goals, create plan for implementing them, measure and analyze effectiveness of strategy implementation, determine the gap between actual and targeted performance and take corrective actions to improve performance [6]. Balanced scorecard, data warehouse and data mining are basic elements of business intelligence systems [7] that enable technology support to business decisions. Balanced scorecard is a well-established methodology for performance measurement of enterprises and is used as a basis for strategic management [8]. It enables preparing foundation for later implementation of data analytics upon performance indicators data. Balanced scorecard as a general framework has been adapted by many practitioners to specific implementation areas. In this paper, balanced scorecard review of research in the field of development of balanced scorecard frameworks in IT project management and particularly software project management is presented. The extension to the previously proposed business process model and balanced scorecard framework for software project management success monitoring, based on project management PRINCE 2 methodology is described. Data model elements are given within the proposed framework. This set of elements enables design of a database that could be used for collecting data during software project implementation. This way, the basis is prepared for future analytics upon structured data in a database. II.

THEORETICAL BACKGROUND

A. Balanced scorecard's key performance indicators "One of the best approaches to identifying the appropriate performance metrics (i.e. key performance indicators - KPI) is through the use of a methodology known as the Balanced Scorecard (BSC). The Balanced Scorecard approach provides executives with a comprehensive framework that translates a corporation’s strategic objectives into a coherent set of performance

– 19 –

Lj. Kazi et al. • Performance Indicators in Software Project Monitoring: Balanced Scorecard Approach

measures. It provides a framework that not only provides performance measurements, but also helps planners identify what should be done and measured. It enables executives to truly execute their strategies." [9] "The balanced scorecard approach is consistent with the concepts of cross-functional integration, customersupplier partnerships, continuous improvement and team accountability." [10] Kaplan and Norton expanded the existing view of performance metrics, which were then primarily financial, into four perspectives: (1) financial, (2) internal business, (3) customer, and (4) innovation and learning (i.e. learning and growth). " [5] The balanced scorecard suggests that the organization is viewed from four perspectives that effect each other in the process of implementing vision and strategy. "Generally speaking, improving performance in the objectives found in the learning & growth perspective enables the organization to improve its internal process perspective activities, which in turn enables the organization to create desirable results in the customer and financial perspectives." [9] These four perspectives give a framework for developing metrics, collecting data and analyzing it relative to each of these perspectives [9]. "Learning and growth" perspective includes metrics related to employee continual learning, training, corporate cultural attitudes toward individual and corporate selfimprovement, ease of communication among workers as well as knowledge management [9]. This perspective "involves goals and measurements related to continuous improvement in the company's ability to innovate, improve and learn." [10] "Internal business process" perspective metrics allow managers to know how well their business is running, and whether its products and services conform to customer requirements." [9] This perspective "includes the business processes that have the greatest impact on customer satisfaction, such as those that affect cycle time, quality, employee skills and productivity. These measurements should be decomposed to the local levels to provide linkages to upper level measurements on the scorecard to insure that lower level employees have clear targets for actions and decisions that contribute to the company's overall mission." [10] "The customer" perspective emphasizes focus on customer and customer satisfaction. In developing metrics for satisfaction, customers should be analyzed in terms of customers' types (groups) and the kinds of processes for which a product or service is provided to those customer groups. [9] Metrics include categories such as: lead time (i.e., time from order receipt to delivery) and quality (e.g., defect levels) of products and services [10]. The "financial" perspective metrics could include timely and accurate funding data, risk assessment and cost-benefit data, [9] cash flow, sales growth, market share, operating income and return on equity, operating expenses, asset turnover [10].

cycle of is defined through several steps [12]: project initiation, project planning, project implementation, project monitoring and control, ending project. According to PMBOK, all activities of project management are divided in several areas of knowledge and practice, as presented at figure 1.

Figure 1. Basic knowledge areas of project management process according to PMBOK [11]

While PMBOK gives general descriptive guide to project management, specific more precise methodologies give more prescriptive approach [13]. One of these methodologies is PRINCE2 (Projects in Controlled Environments, Version 2). This methodology has initially been developed in 1975, while in 1989 has been accepted and developed by Office of Government Commerce of Great Britain). Since 1996 version 2 was developed and this version is available in public sector. Latest version of PRINCE2 methodology is launched in 2009. [14] Basic elements of PRINCE2 methodology are presented at figure 2.

B. PRINCE 2 methodology in project management Project Management Institute (PMI), established as institution in 1969, in aim to develop standard project management methodologies guide, created in 1987 PMBOK (Project Management Body of Knowledge). Project management is defined as direction and coordination of human and material resources in aim to implement project within planned time constraints, with planned quality and planned resources [11]. Project's life

– 20 –

Figure 2. Basic elements of PRINCE2 methodology [13]

SISY 2012 • 2012 IEEE 10th Jubilee International Symposium on Intelligent Systems and Informatics • September 20-22, 2012, Subotica, Serbia

Comparison of PMBOK and PRINCE2 methodologies is presented at table I, according to [13]: TABLE I.

COMPARISON OF PMBOK AND PRINCE2 [13]

PMBOK Comprehensive Largely descriptive, prescriptive on a high level Core and facilitating processes; need to be scaled to needs of project Customer requirements driven Sponsor and stakeholders US/international standard

PRINCE2 Focuses on key risk areas only, does not claim to be complete Highly prescriptive, especially on process structure, but adaptable to any size project All processes should be considered; also need to be scaled Business case driven Clear project ownership and direction by senior management UK standard

III. RELATED WORK Since 1992, when BSC as a methodology has been introduced, many scientists and practitioners adapt it to specific application areas. BSC application is based on collecting data and analytics software that perform processing of these data in aim to compute performance metrics values. Operative implementation of BSC depends on particular semantic field, but general structure of working roles and activities and appropriate software tools, that support these activities, are present. Dobrović et al. [15] describe operative implementation of BSC in e-government application by description of activities, grouped by levels and working roles. In this particular case of BSC implementation, government determine development policy and approve vision and strategic goals set by management. Management activities are: defining vision, defining strategic goals, identifying SWOT elements, SWOT elements ranking, defining strategies, identifying activities for strategy implementation, defining activity goals, defining relationships among goals, identifying metrics, determining relationships among measures, reviewing strategies, managing by BSC model. BSC model administrator does BSC model updating, metrics feeding, data source creating, preparing BSC model validation, model validating, while database administrator does database administration of existing data sources, adding new data resources and adaptation of data sources to extending performance indicators and functionality. Virtanen [16] defines BSC implementation phases as: BSC Model synthesis, technical implementation, organizational implementation, technical integration and operation with activities:update, analyze, report, refine. To accomplish complex tasks of data feeding and processing for analysis and visualization, many commercial software solutions are recently developed. Generally they can be categorized as stand-alone applications and integrated solutions. One of the solutions is QPR software system [16] that enables integration of BSC software with data warehouses, databases and spreadsheets, such as MS Excel, for feeding BSC metrics. Additionally, in [16] is described that "common tools used to help structure the strategy work are: Strategy Mapping, PEST (Political, Economical, Societal, Technological) analysis, SWOT (Strengths-Weaknesses-OpportunitiesThreats) analysis, Porter value chain analysis, Porter Five

forces of competition analysis, BCG Matrix analysis". Finally, it is suggested that introducing BSC to an organization need to be guided as a "management system development project, rather than an IT project" [16], and that this process of including of BSC to everyday management practice need to follow these phases: model synthesis, technical implementation, organizational implementation, technical integration, operation (Update, Analyze, Report, and Refine). BSC as a framework has been implemented at IT project management area. Alleman [17] discusses that balanced scorecard and project management should be unified (especially for IT project focused organizations) in aim to link strategy and implementation. Brock et al. [18] propose model of balanced approach to IT project management. Asosheh et al. [19] propose integration of balanced scorecard and data envelopment analysis (DEA) for information technology project evaluation in the process of selecting among project proposals. Control Objectives for Information and related Technology (CobiT) framework [20] includes linking business goals to IT goals, providing metrics (based on BSC) and maturity models to measure their achievement and ensure that the enterprise’s IT supports the business objectives. Particularly, BSC has been developed for software projects management application. Specific methodologies, such as agile approach, resulted in agile balanced scorecard [21], which emphasizes teamwork, velocity, reliability and performance as key metrics categories. Software Engineering Institute describes the synergistic application of the balanced scorecard and goal-driven measurement methodologies (goal - question - (indicator) -measurement) to develop measures and associated indicators [22]. Each of balanced scorecard models suggests specific set of measures related to goals to be achieved, and defines data needed for those measures to be collected and processed. Firstly introduced balanced scorecard model was developed for performance measurement in production enterprises, widening criteria from financial to other relevant scopes and perspectives. This paper aim is to integrate balanced scorecard approach with PRINCE 2 methodology that is to be applied particularly in software project management in implementation phase, i.e. in success monitoring. Authors propose a set of KPI as a basis for software project success monitoring. This way data is specified that is needed for this process by defining measures and data stores as resources for collecting data during project implementation and for processing measures needed for success monitoring. IV.

RESULTS

A. Business Process Model for Monitoring Project Success In this section it is presented the business process model of project implementation and success monitoring. General project management activities and data stores according to PRINCE 2 project management methodology and document templates [23] adapted to BSC and software project management is presented at extended business process model (Figure 3). This business process model presents an extension of the previously introduced

– 21 –

Lj. Kazi et al. • Performance Indicators in Software Project Monitoring: Balanced Scorecard Approach

model [24, 25] with some activities, data flow and data stores added, such as Resource usage log data store. This model includes activities and data stores regarding "learning and growth" perspective of BSC, such as issue log and lessons log, which enable knowledge sharing during project and preparing a knowledge base for other projects. This model also include evaluation of a team member (relating to results, issues and soft skills) that

enable ranking of team members as a basis for future team formation, i.e. choosing team members according to achievements and results at previous projects. This way "learning and growth" perspective is also present, since it enables continual data acquisition and decision support regarding team member performances during project implementation.

Figure 3. Business process model for software project success monitoring (extended from [24] and [25])

– 22 –

SISY 2012 • 2012 IEEE 10th Jubilee International Symposium on Intelligent Systems and Informatics • September 20-22, 2012, Subotica, Serbia

•

B. KPI for software project success monitoring based on BSC and PRINCE 2 Based on the previously presented business process model (Figure 3), software project success monitoring framework is introduced by set of key performance indicators (Table II and Table III). Measures use data resources that are actually data stores from business process model. Two balanced scorecard perspectives as scopes for measurements are presented at Table II:

TABLE II.

Scope

Process (relate to BSC's "Internal process" perspective) • Product (relate to BSC's "Customer" perspective). Table III presents other two BSC's perspectives by appropriate KPI’s: • Financial perspective (Finance) • Learning and Growth perspective ("Learning", "Improvement").

KPI FOR SOFTWARE PROJECT MONITORING FRAMEWORK BASED ON BSC &PRINCE 2 - PROCESS AND PRODUCT SCOPE ([20], [21])

Perspective Quality of team members

Project proposal

Tasks Process

Issues

Communicatio n

Partial results Product Integrated product

Goal Choose quality team members Team member behavior positively assessed Project proposal according to requirements Achievable results in project proposal Setting achievable tasks

Measures

Data resource for measures

Team member rank (position at list)

Team members rank list

Assessment points at peer questionnaire

Questionnaire results

Number of change requests according to project proposal

Project proposal Change register

Percent of software functions implemented

Task results

Number of issues per task Number of tasks implemented per day/week/month Number of tasks finished at specified time Number of tasks finished at specified quality Number of functionalities implemented per day Number of implemented use cases Number of software functions implemented with required quality

Task results, Issue register Task results Task results Task results Daily log Task results

Not excessive workload to team member

Number of tasks per team member

Task assignments

Minimum issues

Number of issues Number of issues per task Number of issues per team member Number of unsolved issues

Issues archive

Tasks within quality

finished time and

Minimum communication Maximum clarification of tasks Maximum team cohesion - mutual help Partial results according to project proposal Partial results according to user requirements and expectations Has all modules integrated Has all functionality specified Has minimum bugs

Task results

Number of messages Number of clarifying messages regarding tasks

Communication archive

Number of issues solved by messages from team members Number of change requests for partial results

Tasks evaluation

Number of change requests for partial results

Change register

Percent of modules integrated

Configuration register

Percent of software functions realized comparing to required number of software functions Number of test cases executed Number of bugs resolved

Project proposal Task results

– 23 –

Testing register

Lj. Kazi et al. • Performance Indicators in Software Project Monitoring: Balanced Scorecard Approach

TABLE III.

KPI FOR SOFTWARE PROJECT MONITORING FRAMEWORK BASED ON BSC &PRINCE 2 - "FINANCIAL" AND "LEARNING AND GROWTH" PERSPECTIVE

Scope

Perspective

People

Finance Equipment Material Communication costs Issues Learning

Soft skills Lessons

Improvement (Growth)

Team members

Goal Minimum engagement of workforce Minimum workload per task Minimum traveling costs Maximum productivity Minimum usage of equipment Minimum usage of material Minimum communication Minimum repeating issues Best soft skills of team members Maximum lessons learned Maximizing team member rank position Maximum number of successfully completed projects tasks Maximum number of successfully completed projects

Data resource for measures

Measures Number of people engaged

Task assignments

Number of working hours engaged per task

Daily log

Team member residency address (for traveling costs)

Team member rank list

Team member age (for working speed)

Team member rank list

Number of computer working hours and type of computers, type of other computing devices used

Resource usage Log

Number and type of material resources used (paper, DVD/CD) per task Number of messages, number of phone calls, duration of phone calls Number of repeating issues (comparing to previous tasks/people or other projects)

Resource usage Log Communication archive Issues archive

Soft skills points achieved at questionnaire

Soft skills record

Number of lessons at lessons archive

Lessons Log

Team member rank position - comparing changes/improvements for each member during time

Team member rank list

Number of successfully completed tasks of team members

Team member rank list

Number of successfully completed projects of team members

Team member rank list

V. CONCLUSION Balanced scorecard is widely accepted as performance measurement system as well as strategic management system. Many scientists and practitioners have applied BSC concept as performance measurement system in particular working environment. In this paper, review of some results in the field of applying balanced scorecard in IT sector, especially in IT project management and particularly in software project management are presented. Based on BSC, many software tools were developed in aim to implement measurement by defined key performance indicators. In this paper, integration of PRINCE 2 methodology for general project management with balanced scorecard methodology for performance management is proposed. This integration is particularly targeted to software project success monitoring. In aim to define success criteria, set of KPI's is defined. Sources of data to these KPI's are defined from data stores of business process model based on PRINCE 2 methodology for project management. Special attention has been given to KPI's related to learning and growth perspective of BSC. This perspective

is especially important and targeted to team members regarding their position in teams. Team member ranking process has been introduced, based on the previously defined particular KPI's. Learning and growth perspective is possible to evaluate in longer time period at series of projects that team members implement. Single project is not sufficient to enable long term based data collection about team members' progress. The proposed approach of integrating PRINCE 2 with BSC for software project success monitoring could be useful in large software producing companies. REFERENCES [1] [2] [3]

[4]

– 24 –

Agile Manifesto, http://agilemanifesto.org/ F. Bocci, "Defining performance measurement – a comment", Perspectives on Performance Journal, Vol. 3, Issue 1/2, Sept. 2004 R. S. Kaplan and D.P. Norton, "The balanced scorecard Measures that drive performance", Harvard Business Review (January-February 1992.): 71-79. Performance measurement management framework, http://www.businessballs.com/dtiresources/performance_measure ment_management.pdf

SISY 2012 • 2012 IEEE 10th Jubilee International Symposium on Intelligent Systems and Informatics • September 20-22, 2012, Subotica, Serbia

[5] [6] [7] [8] [9]

[10] [11] [12] [13] [14] [15]

[16] [17] [18]

M. Krolick and T.Ariyachandra, "Business performance management: one truth", Information system management journal, Winter 2006 Business performance management standards group, http://www.bpmstandardsgroup.org/ B. Ciric, "Business intelligence", Data Status, Belgrade, 2006. B. Pesalj, "Enterprise performance measurement", University of Belgrade, School of Economics, 2006. Balanced Scorecard Institute, "About Balanced Scorecard", http://www.balancedscorecard.org/BSCResources/AbouttheBalan cedScorecard/tabid/55/Default.aspx M. James, "Summary of Kaplan's Balanced Scorecard", http://maaw.info/ArticleSummaries/ArtSumKaplanNorton92.htm Project management institute, "A guide to project management body of knowledge", 2005. A. Yeong, "The marriage proposal of PRINCE2 and PMBOK", 2007. J. M. Siegelaub, "How PRINCE2 can complement PMBOK and your PMP", PMI/Westchester chapter, 2004. P. Jovanovic: Project management [Upravljanje projektom], Faculty of organizational sciences, University of Belgrade, 2006. Z. Dobrovic, M. Tomicic M and N. Vrcek, "Towards an effective e-government: implementation of a balanced scorecard in the public sector", Intellectual Economics Journal, 2008, No. 1(3), p. 7-17 T. Virtanen, "Guidelines for implementing Balanced Scorecard", QPR software white paper, 2009. G. B. Alleman, "Using Balanced Scorecard to Build a Project Focused IT Organization", Balanced Scorecard Conference proceedings, San Francisco, Oct 28-30, 2003. S. Brock S, D. Hendricks, S. Linnell S and D. Smith, "A balanced approach to IT project management", SAICSIT '03 Proceedings of the 2003 annual research conference of the South African institute of computer scientists and information technologists on

[19]

[20] [21]

[22]

[23]

[24]

[25]

– 25 –

Enablement through technology Pages 2 - 10, Publisher South African Institute for Computer Scientists and Information Technologists , Republic of South Africa, 2003. A. Asosheh, S. Nalchigar and M. Jamporazmey, "Information technology project evaluation: An integrated data envelopment analysis and balanced scorecard approach", Expert Systems with Applications: An International Journal Vol. 37 Issue 8, August, 2010, pages 5931-5938, Pergamon Press, Inc. Tarrytown, NY, USA ISSN: 0957-4174 CobiT 4.1., Excerpt Executive Summary Framework, IT governance Institute, 2007. Agile Balanced Scorecard – Measuring the effectiveness of an Agile Software Development Team, http://agile101.net/2009/07/18/agile-balanced-scorecardmeasuring-the-effectiveness-of-an-agile-software-developmentteam/ W. Goethert and M. Fisher, "Deriving Enterprise-Based Measures Using the Balanced Scorecard and Goal-Driven Measurement Techniques", Technical Note, Software Engineering Institute, Carnegie Mellon University, October 2003 Department for Business innovation and skills: PRINCE 2 templates, http://www.berr.gov.uk/aboutus/corporate/projectcentre/pmtemplates/page12526.html Lj. Kazi and B. Radulovic, "Information system based on Balanced Scorecard for student teamwork software project management", International symposium MIPRO 2011, Opatija, Croatia, 2011. Lj. Kazi, D. Radosav, M. Nikolic and N. Chotaliya: Balanced Scorecard Framework In Software Project Monitoring, Journal of Engineering Management and competitiveness, Vol. 1, No. 1-2, 2011, Technical faculty "Mihajlo Pupin" Zrenjanin, Online ISSN: 2217-8147