“Public Value of IT” Frameworks

“Public Value of IT” Frameworks June 2007

Overview Government agencies are leveraging targeted IT frameworks to radically improve the standard of governance and enhance the quality of interaction with citizens. However, the need to justify the costs incurred in such activities has been a point of contention among governments, citizens, and other stakeholders.

The modern world is dominated by communication frameworks hinged on increasingly pliable information technology tools. As a direct result of the growing reach and relevance of IT, government agencies around the world are leveraging targeted IT frameworks to radically improve the standard of governance and enhance the quality of interaction with citizens. However, the need to justify the costs incurred in such activities has been a point of contention among governments, citizens, and other stakeholders. Governments have experimented with various quantitative methods to measure benefits and costs and assess whether IT applications deliver value commensurate with the investments made on them. Some of the parameters considered for this assessment include payback period, net present value (NPV), internal rate of return (IRR), and economic value added (EVA). However, the idiosyncrasies attached with the way governments operate have made these methods ineffective for a comprehensive assessment of their IT initiatives. There exists a fundamental difference in the raison d’etre of governments and private organizations. Government bodies perform multiple roles for stakeholders with diverse interests at the same time. They are authorities in areas such as justice and tax collection, protectors of public welfare and public safety, and providers of utility, education, and health care services. This establishes a unique relationship between government organizations and their stakeholders. The pervasive reach of government policies and their multi-dimensional effects ensure that any (IT) initiatives taken up by the government impacts all other sectors, in a way that no other system can. Thus, it can be concluded that conventional methods used for measuring value can not be applied to government initiatives, since they tend to engage almost exclusively with financial parameters and do not account for other forms of returns. In addition to measuring financial value, an assessment of social benefits is imperative for a comprehensive assessment of any initiative undertaken by governments, and the IT sphere is no exception. To deal with this special need, several governments have formulated frameworks to measure the public value of IT investments. Some examples of these frameworks are the “Demand and Value Assessment” in Australia, the “Federal EA Performance Reference Model” in the United States, “WiBe” in Germany, and “MAREVA” in France.

Gartner Definition of Public Value In July 2003, Gartner, Inc. published a research document titled "Traditional ROI Measures Will Fail in Government”, which highlighted the need to appraise IT investments by exploring "public value." Gartner defines Public value of IT (PVIT) as “Measures that demonstrate how ITrelated changes and investments contribute over time to improved constituent service level, operational efficiency and political return."

Gartner created the concept of PVIT in another research paper titled "Value for Money is not enough in Public Sector IT Projects." The paper defines public value of IT (PVIT) as “Measures that demonstrate how IT-related changes and investments contribute over time to improved constituent service level, operational efficiency and political return." The definition foregrounds the following three dimensions of PVIT: Constituent service level – Constituent service level measures the impact on time and cost of users or beneficiaries of government services, in terms of accessibility, quality, and convenience of services. Operational efficiency – Operational efficiency measures the internal impact of an initiative – both on individual departments and across government organizations. Political return – Political return measures the impact of investments on political goals and consensus, the overall impact on society in terms of wider reach of information, alleviation of digital and cultural divides, etc., and impact on the economy (reduced unemployment, growth in exports, growth of small and medium-sized businesses, etc.).

1

Criteria for Evaluation The Gartner study brings out certain key features that should be present in any framework designed for similar purposes. These include: PVIT calls for inclusion of risk factors in the evaluation process. This helps in conducting risk assessment as a part of the overall assessment process and not as an independent exercise.

ƒ PVIT dimensions – A good framework covers all the dimensions of PVIT. These may not be restricted to the dimensions defined by Gartner and may only be conceptually similar to them. ƒ Value separation – Value separation calls for a simplified classification of basic value criteria. It requires independent measurement of each dimension of public value while ensuring that no measure belongs to more than one dimension. ƒ Balance – A good measure does not attach preferential weightage to one particular dimension and treats each dimension equally. Thus, when the users choose measures for evaluation, they should evenly cover each dimension. ƒ Multiple vs. single value score – This relates to the “balance” criterion. The framework should help users in prioritizing different measures from the same PVIT dimension and helping them avoid the prioritization of different PVIT dimensions. PVIT should result in at least one aggregated measure per dimension instead of a single number or value. ƒ Inclusion of risks – This criterion calls for inclusion of risk factors in the evaluation process. This helps in conducting risk assessment as a part of the overall assessment process and not as an independent exercise. ƒ Mandatory use – A good framework includes pre-defined budget and capital planning processes. This helps in easier adoption and better application of the framework.

Various PVIT Frameworks across the World France: MAREVA MAREVA or the “methode d'analyse et de remontee de la valeur” (the method for value analysis and increase) was developed by BearingPoint for the Agency for the Development of Electronic Administration (ADAE). It was originally formulated to evaluate around 140 projects launched under France’s e-Government program. The methodology ensures accurate evaluation of financial gains of e-Government services for the government and the public sector as well as gains and benefits for their users. According to this framework, the definition of value includes the following: ƒ Profitability for the state – It measures various costs such as development and operation costs, financial benefits in terms of productivity gains, increase in revenues, etc. MAREVA framework assesses different types of risks related to a project. These include risks associated with budgeting and scheduling, legal risks, technical risks, and deployment risks

ƒ Internal aspects – These include qualitative factors affecting governments such as improvement in service delivery, employee satisfaction, and effects of decentralization. It also measures the quantifiable impact on the public sector. ƒ External aspects – These include assessment of the benefits and drawbacks of government services to citizens or users. The assessment is both qualitative (service quality, social impact, etc.) and quantitative in nature (monetary savings and reduction in time). ƒ Necessity – This criterion measures the criticality of a project in comparison to other projects undertaken or to be undertaken by ADAE. The MAREVA framework assesses different types of risks related to a project, such as risks associated with budgeting and scheduling, legal risks, technical risks, and deployment risks (related to staffing, political support, continuity of service, etc.). It defines public value as a function of several variables that cannot be aggregated as a single value. However, the framework fails to clearly demarcate the different dimensions of PVIT. Although, various elements of value are specifically identified, they are not sufficiently differentiated. According to Gartner, the criterion of necessity, which forms a part of 2

the value assessment model, is not entirely accurate. In an ideal scenario, only suitable projects are required to be assessed with the help of the framework. Also, the framework does not perform a risk-to-benefit analysis.

European Union: e-Government Economics Project

e-GEP project comprised an in-depth analysis of publicsector-specific value assessment methods (VAMs).

In order to measure the success of e-Government initiatives in Europe, the European Commission (EC) launched the e-Government Economics Project (eGEP) in 2005. The study included the formulation of an assessment framework, analysis of expenditure and funding related to e-Government, and creation of an economic model to assess macroeconomic effects of e-Government. The project was initiated by an Italian consulting company, RSO, in collaboration with the Luiss University of Rome. The study, completed in February 2006, comprised an in-depth analysis of public-sector-specific value assessment methods (VAMs). Along with the objective to establish a relationship between investments in e-Government and its impact on the economy, the other purpose of eGEP was to identify indicators for benchmarking other EU member countries against the objectives of i2010 initiative. The primary dimensions of the eGEP framework include efficiency, effectiveness of services, and democracy (transparency and participation). Sample eGEP Measurement Framework

eGEP Framework

Efficiency

Cashable financial gains

Democracy

Openness

Effectiveness

Reduced administrative burden

Empowered employees

Better organizational and

Transparency and

Increased user value and

accountability

satisfaction

Participation

IT architecture

The eGEP framework measures quantitative and qualitative components for each dimension of PVIT.

Financial and organizational value

More inclusive public services

Political value

Constituent value

Source: Worldwide Examples of Public Value of IT Frameworks

The eGEP framework measures quantitative and qualitative components for each dimension of PVIT. It measures these components collectively and focuses on relative improvements expressed in percentage terms. As mentioned, these measures are observable and objective (leading to tangible value creation), unobservable and subjective (leading to intangible value creation), and externally measurable (qualitative and measurable against standard, auditable criteria). Demand assessment helps in identifying the needs of the end user and the way in which they would be addressed through the project.

Australia: Demand and Value Assessment In 2003, the Government of Australia undertook the “e-Government Benefit Study” to assess the benefits of its e-Government initiatives. The study highlighted the need for a comprehensive methodology for the assessment of e-Government initiatives, since the benefits of many such initiatives were intangible. This created the requirement for a framework that could capture tangible and intangible benefits as well as the probable risks associated with each project. In 2004, the Australian Government Information Management Office (AGIMO) developed a demand assessment method (DAM) and value assessment method 3

(VAM) to assist agencies in developing transparent and auditable assessments of online government initiatives.

Value assessment measures the costs and benefits, social and governance implications, and intangible benefits.

Demand and value are the two significant dimensions of the framework. The demand assessment helps in identifying the needs of the end user and the way in which they would be addressed through the project. The value assessment measures the costs and benefits, social and governance implications, and intangible benefits. It identifies benefits and risks on the same scorecard. The method also facilitates retrospective measurement of past IT programs. Sample AGIMO Profile

10 8 6 4

Target Score The EAF assesses eGovernance projects on attributes such as service orientation, technology, sustainability, cost effectiveness, and feasibility of replication.

Risks to achievement of benefits

Risks to program delivery

Strategic alignment

Governance value

User financial value

Agency Value

0

Social value

2

Actual score

Source: Demand and Value Assessment Methodology – AGIMO According to Gartner, this model is completely aligned with the concept of PVIT. The framework is simple to understand and adaptable to a wide variety of assessments. It acts as a tool for monitoring projects over a period of time. The method is based on uncomplicated rules and lends a balanced view to risks and benefits. It has attracted attention across the world and is being adopted by other countries such as Ireland. India: E-Governance Assessment Framework The E-Governance Assessment Framework (EAF Version 2.0) was designed in 2004 to assess e-Governance projects on attributes such as service orientation, technology, sustainability, cost effectiveness, and feasibility of replication. The methodology consists of assessments for G2G, G2B, and G2C services. The attributes are further divided into several parameters that are independently measured to arrive at scores for each attribute.

Conclusion The effectiveness of a PVIT framework depends on 1) its ability to integrate and align itself with other essential processes such as budgeting and planning, 2) its applicability across various agencies, and 3) its utilization across the entire life cycle of a process or project. The value derived from these methods should be used to make accurate judgments pertaining to projects and to rationalize and prioritize investments. References

ƒ Worldwide Examples of Public-Value-of-IT Frameworks – Gartner ƒ Beyond ROI in e-Government: introducing the Public Value of IT – IDABC (IDA eGovernment Newsletter - December 2003) ƒ Australian Government Information Management Office – Demand and Value Assessment

4

Evalueserve Disclaimer The information contained herein has been obtained from sources believed to be reliable. Evalueserve disclaims all warranties as to the accuracy, completeness or adequacy of such information. Evalueserve shall have no liability for errors, omissions or inadequacies in the information contained herein or for interpretations thereof.

5