Constructing performance measurement in the public sector - CiteSeerX

Critical Perspectives on Accounting 21 (2010) 266–282

Contents lists available at ScienceDirect

Critical Perspectives on Accounting journal homepage: www.elsevier.com/locate/cpa

Constructing performance measurement in the public sector Michela Arnaboldi ∗ , Giovanni Azzone 1 Politecnico di Milano – Dipartimento di Ingegneria Gestionale, Piazza L. Da Vinci n. 32, 10133 Milano, Italy

a r t i c l e

i n f o

Article history: Received 13 August 2008 Received in revised form 15 November 2009 Accepted 12 January 2010 Keywords: Performance measurement Innovation Public sector Actor-Network Theory Latour

a b s t r a c t This article investigates the process of translating Performance Measurement Systems (PMSs) into operational use, in the context of public sector reforms. Drawing on Actor-Network Theory, the article examines the Good Practice project, an 11-year PMS implementation in Italian universities. Analysing this experience, the article shows how controversies play a central role in the process of translation, as essential risks for revitalising the PMS. Furthermore, it underscores the heterogeneity of actors’ stratagems, giving voice to louder as well as weaker participants, who all proved crucial to the vitality of the PMS, by raising unexploited and unresolved issues within the network. © 2010 Elsevier Ltd. All rights reserved.

1. Introduction Performance measurement has been a matter of interest to governments for some time (Beyle and Parratt, 1938; Rosa, 1921), but it was with the advent of New Public Management reforms (Hood, 1991, 1995; Lapsley, 2008) that Performance Measurement Systems (PMSs) became firmly established as a central tool for transforming old bureaucratic administrations into efficient and effective organisations (Dent, 1991). It has been claimed that PMSs and other managerial techniques offer a ‘solution’ (Pollitt and Bouckaert, 2004) to the problem of containing public expenditure and improving the services delivered to users, with particular emphasis on the management of results and the ability to measure public organisations on the basis of an input–output (outcome) model (Johnsen, 2005). Despite the large body of research on NPM (e.g. Bouckaert, 1990; Chow et al., 1998; De Bruijn, 2002; Modell, 2001; Pallot, 2003; Watkins and Arrington, 2007; Yang and Holzer, 2006), there are gaps in our understanding of the processes by which PMSs are translated into operational use (Modell, 2009). This article seeks both to analyse these processes and more generally to extend our knowledge in the sphere of public sector reforms. The focus of the work is an examination of the Good Practice project, an 11-year experience of developing PMSs in Italian universities. Drawing on Actor-Network Theory (ANT), and specifically on Latour’s work (Latour, 1987, 1996, 2005), the objective of this paper is to investigate how universities translated the PMS over the 11-year period, during which the stratagems and roles of actors evolved and grew more heterogeneous, resulting in a combination of more powerful and weaker voices within the network. The empirical research was carried out with a two-level approach. Firstly, data were gathered through participant observation over the 11 years of the project. Secondly, as differences between the universities’ approaches began to emerge, we decided to follow (Latour, 1987) certain actors that highlighted markedly different approaches within the network. Adopting

∗ Corresponding author. Tel.: +39 02 23997328; fax: +39 02 23997321. E-mail addresses: [email protected] (M. Arnaboldi), [email protected] (G. Azzone). 1 Tel.: +39 02 23994071; fax: +39 02 23994067. 1045-2354/$ – see front matter © 2010 Elsevier Ltd. All rights reserved. doi:10.1016/j.cpa.2010.01.016

M. Arnaboldi, G. Azzone / Critical Perspectives on Accounting 21 (2010) 266–282

267

a Latourian perspective, the analysis was unfolded with reference to four stages identified as pivotal in previous ANT studies of accounting in the public sector (Chua, 1995; Gendron et al., 2007; Preston et al., 1992): setting the goal and enrolling actors; circulating translation and dealing with controversies; acting the quasi-object; and insiders going out. These stages should not be viewed as a rigid sequence of steps, but rather as focal points for exploring actors’ stratagems and tactics over time. The results reported here show that the persistency of the rhetorical process of translation pivots around controversies (Latour, 1987). Controversies were found to be essential risks within the PMS implementation: they prompted universities to problematize and engage with the PMS in the attempt to solve them, but they also undermined the possibility of achieving a socio-technical compromise. With this ongoing process of dealing with controversies, the actors’ approaches evolved over time, but also grew more heterogeneous, shaped by the role that each actor had assigned to the PMS (Modell, 2009; Townley et al., 2003; Watkins and Arrington, 2007). Without inferring any precise rule of behaviour, this heterogeneity of actions was captured by identifying three distinctive categories of actors: strategic allies, technical warriors, and searchers. The purpose of these categories was not to provide a rigid classification, but only to represent the diversity of approaches within a spectrum actions, and of actors with louder or softer voices. All these voices, irrespective of their strength, proved essential for creating the glue which held the network together and sustained interest in the PMS. To develop our arguments, the remainder of this paper is organised as follows. First, we provide an analysis of the difficulties of implementing PMSs in the public sector. The conceptual perspective is then illustrated (Section 3), followed by a description of the research approach (Section 4). An overview of the Italian reform context introduces the presentation of results. Finally, we discuss the findings and draw some conclusions. 2. Performance measurement in the public sector: implementation difficulties Performance Measurement Systems have been seen as a key element of NPM reforms (Hood, 1991, 1995), depicted as objective, rational tools that could be used for various purposes: steering and controlling the organisation, establishing rules of conduct, disseminating a culture of accountability (Broadbent and Laughlin, 1997; Dent, 1991; Pallot, 2003). Despite the envisioned benefits of PMSs, their implementation trajectories are far from linear and straightforward: various difficulties may arise along the way that lead to different outcomes or even to abandonment of the system (Lawton et al., 2000; Popper and Wilson, 2003; Smith, 1995). In light of this, many studies have explored the functional difficulties of PMSs in the public sector, identifying four main issues that affect their implementation: the diverse nature of public sector services, the wide range of users, the difficulties in defining targets, and the lack of competencies. The diverse nature of public sector services was one of the major issues that emerged during initial applications of the PMS, in the wake of the NPM reforms. The criticisms mainly concerned the need to tailor techniques derived from business for application to the public sector. This need for adaptation extends beyond the simple public–private dichotomy, and various authors have noted the difficulties encountered when systems are applied ‘universally’ by governments, to different types of institutions (see for example Beryl, 2000; Boland and Fowler, 2000; Bouckaert, 1990, 1993; Bouckaert and Balk, 1991; De Bruijn, 2002; Holzer, 1991; Smith, 1993). One suggested way to tackle the problem of diversity is through implementation of multidimensional PMSs, such as the ‘Balanced Scorecard’ (Kaplan and Norton, 1992) or the ‘Tableau de Bord’ (Lauzel and Cibert, 1959). However, in the public sector these often result in a proliferation of indicators, which are then not acted upon by managers (Chow et al., 1998; Modell, 2001). The second problematic issue is the existence of a wide range of users. Service users have been at the centre of NPM since its inception, and this has prompted public managers and policy makers to shift their attention from procedural compliance to output delivery. Emphasis on users of PMSs has recently been further increased, with many organisations adopting approaches that directly involve citizens in performance measurement design (Epstein et al., 2006; Holzer and Kloby, 2005; Modell, 2001; Yang and Holzer, 2006). Their involvement is considered beneficial for the design of the system and for building a relationship of trust between public institutions and their users. The third issue is the difficulty of defining targets for performance, which has often been portrayed as a problem specific to public organisations. This view assumes that private firms adhere to profit and value imperatives which enable them to more easily set target figures, whereas this is more difficult in the public sector (Bohte and Meier, 2000; Popper and Wilson, 2003; Van Thiel and Leeuw, 2002). Though differences exist, the advent of non-financial measures and the ‘Balanced Scorecard’ have posed a similar problem for private organisations; in both cases, a possible solution is offered by Relative Performance Evaluation (RPE), which is more commonly known as performance benchmarking (Bogan and English, 1994; Elnathan et al., 1996; Siverbo and Johansson, 2006). However, a number of problems have arisen in its implementation, such as the difficulty of comparing institutions with different organisational structures (CIPFA, 2008). A final difficulty is the lack of competencies. Public managers are traditionally accustomed to dealing with financial measures, but are less familiar with non-financial indicators and concepts such as output and outcome. This can create problems during the early phases of adoption, when the design of the system is delegated to internal managers. Furthermore, lack of competencies is also a major hindrance during development of the system, and can lead to its abandonment or to unintended consequences (Lawton et al., 2000; Popper and Wilson, 2003; Smith, 1995; Wang and Gianakis, 1999). Besides these studies which focused on the functional aspects, more critical research (Bracci, 2009; Broadbent and Laughlin, 2003; Cooper and Hopper, 2007; Jones and Mellett, 2007; Robinson, 2003; Watkins and Arrington, 2007) has investigated the PMS implementation, challenging the objectified vision of system. Among the critical approaches, post-

268


structural theories are of particular interest for their ability to capture interconnections between the macro, micro and personal level, underscoring the constitutive nature of accounting (Cooper and Hopper, 2007). According to this perspective, measures are not seen as merely providing an objective representation of reality. Instead, “[accounting numbers] produce a vision of what good government entails and they constitute public sector actors in particular ways” (Cooper and Hopper, 2007, p. 228). Some authors (Chua, 1995; Gendron et al., 2007; Preston et al., 1992) expand on this vision, using ActorNetwork Theory (ANT). These works adopt a Latourian approach, treating accounting innovation not as a technical matter but as a process of translation (Latour, 1987, 2005), in which an initial idea of the PMS is diverted as its proponents encounter other actors during its implementation. These ANT studies of public sector accounting have greatly enhanced our knowledge of the translation of PMSs in public settings, but further exploration is needed (Modell, 2009) to understand longitudinally the heterogeneity of actions associated with diverse organisations and with evolving reforming contexts (Townley et al., 2003; Watkins and Arrington, 2007).

3. Theoretical frameworks: action and social relations This paper aims to enhance our knowledge of the processes by which PMSs are translated into operational use in NPM settings. The analysis is carried out drawing on Actor-Network Theory (ANT), and specifically on the work of Latour (1987, 1996, 2005), whose ideas have been eliciting increasing interest in the accounting literature (for example Chua, 1995; Doolin and Lowe, 2002; Gendron et al., 2007; Harris, 2005). Latour theorises that innovation is a collective process of translation, in which an initial idea is progressively changed as scientists interact with other actors for its implementation. Latour (1987, 2005) emphasises the rhetorical and relational nature of this process, in which innovation becomes a search for allies in an attempt to raise or divert interest and convince other people of the validity of the proposed innovation. The concept of translation is part of Latour’s view of science (Latour, 2005); from his perspective, technological and scientific facts are not the products of a pre-existing social order, but emerge from a lengthy, difficult process of association in which an initial idea of the technology is ‘detoured’ (Latour, 1987) as scientists try to gain its acceptance. In doing this, Latour (1996) moves away from what he calls ‘classical sociology’, which he depicts as follows: “Classical sociology knows more than the ‘actors’; it sees right through them to the social structure or the destiny of which they are the patients. It can judge their behaviour because it has a fixed reference frame with respect to which the patients behave in a pathological fashion. It has its ether. There are norms, and thus there are deviations with respect to the norms; there are norms; there are reasons, and thus there is irrationality; there is logic, thus there is illogicality; there is common sense, and thus perverted senses; there are norms, and thus there are abnormality and anomie” (Latour, 1996, p. 199). Latour (1996) goes on to argue that, to study science, one must instead be a ‘relativist’ (or relationist) sociologist, abandoning reference frames and metalanguage to follow the human actors, as well as the technological objects, while scientific facts are being created (Latour, 1996, p. 100). The importance attributed to technical, natural objects, as well as to human actors, is a central feature of ANT. According to ANT, the distinction between technological and social innovation is blurred, and human and non-human actors are equally important; it is only through their interactions and the construction of a socio-technical network that innovations can propagate. This equality of human and non-human actors is a controversial point of Latour’s philosophy (McLean and Hassard, 2004), which has both its opponents (Bloor, 1999; Collins and Yearley, 1992) and supporters (Lowe, 2001, 2004). The strongest objections have been raised by Collins and Yearley (1992), who criticise the ‘radical’ symmetrical treatment and question the ability of ANT researchers to capture it: “The analysts remain in control the whole time, which makes their imposition of symmetry on the world seem something of a conceit. Would not complete symmetry require an account from the point of view of the scallops? Would it be sensible to think of scallops enrolling the scallop researchers so as to give themselves a better home and to protect their species from the ravages of the fishermen? (Collins and Yearley, 1992, p. 313). However the symmetry between humans and non-humans should not be interpreted as giving them equal status; it is a framework that provides a different perspective for analysing events, focusing attention on the interactions between technologies and social actors, and the manner in which they reciprocally alter each other in the process of trying to turn an initial idea into an innovation. Innovations are socio-technical compromises, attained and kept alive by the process of stimulating interest and detouring controversies when they arise (Latour, 1987, 2005). To understand how this happens, it is necessary to follow actors in their individual rhetoric, what Latour calls the study of “how people are made to believe and behave and taught people how to persuade others” (Latour, 1987, p. 30). Latour (1987) highlights how scientists may enact their rhetoric, using stratagems to persuade people, attempting to elicit their interest and put forward an initial idea. Drawing specifically on this work (Latour, 1987), as well as on previous accounting contributions (Chua, 1995; Gendron et al., 2007; Preston et al., 1992), we read the translation of PMSs as a four-stage process during which different stratagems are used to accomplish the innovation. The first stage is setting the goal and enrolling actors. This should not be thought of as a rational and clear-cut statement of objectives, but more broadly as succession of earlier translations, carried out to arrive at an initial proposal of the innovation


269

(in this case the PMS). Once they have this initial idea, scientists, government and experts enrol actors in the innovation process (Gendron et al., 2007; Mennicken, 2008); this implies involving all relevant people and convincing them of the idea, but also finding other experts and objects that can make the idea a reality (Latour, 1987, 1996). Once enrolled, actors have to be convinced, and their interest in the innovation needs to be maintained. This leads us to the second stage of circulating translations and dealing with controversies. As pointed out above, innovation, in Latour’s perspective (1987, 2005), is a process of translation in which the initial idea is continually reshaped as it interacts with the actors involved. When the number of actors involved increases, translations need to be circulated to maintain the actors’ interest and their alignment with the innovation as it evolves (Gendron et al., 2007). However, the circulation of new translations generally triggers further controversies which challenge the network and the accomplishment of the innovation. Latour (1987) suggests a number of Machiavellian modes for dealing with these controversies, based on the interactions between humans and technological objects. Previous studies in accounting and auditing (Chua, 1995; Gendron et al., 2007; Mennicken, 2008; Preston et al., 1992) have highlighted the importance of this stage, evidencing the central role of fact-building (Latour, 1987) in gaining consensus for the innovation. The third stage is acting the quasi-object. The term quasi-object was introduced by Serres (1982) and then used by Latour (1987, 1999) to denote the indeterminate nature of innovation, which changes continually in the hands of different actors. This fluidity needs to be interpreted from the standpoint of Actor-Network Theory, which regards innovation as an indivisible whole consisting of technical devices, humans, and the relationships between them. The stage of acting the quasi-object thus embodies this fluidity, following the application of technology by humans, who shape and change what ultimately becomes the ‘innovation’ (Chua, 1995). This interpretation is especially helpful for managerial innovations, which tend to be by nature ill-defined (Busco et al., 2007) and easily modified by social actors (Modell, 2009). The fourth stage is that of insiders going out, and involves transferring the innovation and best practices (Gendron et al., 2007) from the network in which the innovation was developed to the outside world, and so to other networks. This element emerges as particularly relevant in accounting studies informed by ANT (Chua, 1995; Gendron et al., 2007; Mennicken, 2008; Preston et al., 1992), given the portability of numbers and measures. Inscriptions from the system enable actors to propagate the innovation, and make distant comparisons feasible (Chua, 1995). Though listed last, this should not be thought of as the concluding step of a linear sequence: the boundaries are not closed and people are not isolated, ideas can also flow out of the network at any stage of the project, either voluntarily or not. The dynamic nature of the social relationships surrounding innovation development sometimes makes it difficult to mark precisely what lies ‘inside’ and what lies ‘outside’ the network. These four stages of translation are used as a reference framework for analysing the manner in which the PMS is translated into operational use. In so doing we follow the actors (Latour, 1987, 2005), highlighting how their behaviour evolves and is differentiated based on the different roles that they assign to the PMS: legitimacy device, instrumentality to reform, instrumentality to organisational goals, or bonding system (Modell, 2009; Townley et al., 2003; Watkins and Arrington, 2007). Although it is not within our scope here to set out precise rules of behaviour, the spectrum of actions enacted by universities can be captured by three categories of actors, which were found to display distinctive approaches to the translation of the PMS: strategic allies, technical warriors and searchers. 4. The research approach The empirical focus of this paper is the Good Practice project, an 11-year initiative which saw an expanding group of universities translate PMSs into operational use. We, the authors, were brought into the project from its inception in 1998 to provide advice on performance measurement, which gave us privileged access for following the PMS translations.2 In terms of methodology, the empirical research was conducted on two levels. The first level involved gathering data, mainly informally, through participant observation, as we engaged with universities during project meetings and discussions, in conversations external to the project, or in training courses. Table 1 shows a general overview of the participating universities, and their involvement in the seven cycles of the project.3 On a second level, as divergences between the universities’ approaches began to emerge, in 2003, we decided to follow some selected actors (Latour, 1987) which captured the range of heterogeneity within the network. Observations on this level were complemented with formal interviews and site visits, during which we were given access to archival data and documents not usually made available to the public. This documentation provided useful evidence of how the PMS was actually used at the organisational level; specifically we had access to: internal simulations of data and performance; further applications of the PMS at the organisational level; internal circulars demonstrating the use of Good Practice measures in top managers’ incentives, in informing reorganisations and reengineering processes; and in official presentations made by governing boards drawing on the Good Practice results. However, the principal source of evidence for this part of the research were interviews conducted with key actors. These were chosen to represent the heterogeneity of the network, focusing on

2 As also pointed out later in this section, we wish to highlight the lack of independence inevitably entailed by our role as consultants. This paper accordingly does not purport to give a purely objective analysis of facts. Nevertheless, we did seek to counter this subjectivity through use of multiple evidence sources and peer confrontations with other academic researchers, observing the Good Practice phenomenon. 3 Over the 11 years there have been 7 cycles of the Good Practice project: (1) Best Practice (later renamed Good Practice); (2) Good Practice 2; (3) Good Practice 3 – South; (4) Good Practice 2003; (5) Good Practice 2005; (6) Good Practice 2007; (7) Good Practice 2009, which is still ongoing.

270


Table 1 The participating universities. N. students (2005/2006) 1 2 3 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37

North North North North North North South North South North South North South South North North North South South North North North North South North South South South South South South South South North North North

95,771 38,343 15,413 21,369 59,560 24,193 31,511 8754 62,493 17,456 28,775 19,824 60,436 63,350 16,695 57,707 21,902 10,608 132,575 8754 27,640 6461 57,649 11401 19,377 9123 35,574 96,400 9698 10,250 39,165 12,139 18,974 13,319 3252 6658 1,172,569

BP (later GP)

GP2

X X X X

X X X X X X X X X

X X X

GP3 (or GP South)

GP 2003

GP 2005

GP 2007

GP 2009

N. projects

X X X X X X X X X X

X X X X X X X

X X X X X X X X X X X X

X

X X X X X X X

X X X X X X X X X X X X X X X X X X

6 6 6 6 6 6 6 5 5 4 4 4 4 4 4 3 3 3 3 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1

X X X

X X

X X X

X X X X

X X X X X X

X X X X X X X X

X X X

X X

X

X X X X X X X

10

14

7

16

19

21

X X 23

distinctive categories of approaches that had begun to emerge (which we termed strategic allies, technical warriors and searchers) and on the project initiators: central government, and more specifically the members of the central national committee for evaluating the university system (CNVSU — Comitato Nazionale di Valutazione del Sistema Universitario). Ten face-to-face interviews were carried out between 2004 and 2008. This information was complemented with follow-up phone interviews (10 overall) to validate the theoretical insights and, where needed, integrate the collected data. Table 2 gives details of the informants for this second stage of research. Finally, it should be noted that, as participating observers of the phenomenon under study, we were in a privileged position for capturing the facts and fact-building, but were also ourselves involved in the process of negotiation and translation of the PMS. Hence, this article is not an objective, rational, description of events, but rather a narration mediated by our own experiences and changing viewpoint during the course of the project (Chua, 1995).

Table 2 Details of the informants for the second stage of the research. Informant

Organisation

Role in the project

Face-to-face interviews

Phone interviews

CNVSU member 1 CNVSU member 2 Administrative director Human resource manager Head of finance Finance director Officer Administrative director Administrative director Officer

Central government Central government University 1 University 1 University 1 University 7 University 7 University 10 University 20 University 20

Initiator External observer Strategic allies Strategic allies Strategic allies Technical warrior Technical warrior Technical warrior Searcher Searcher

1 1 1 1 1 1 1 1 1 1

1 1 1 1 1 1 1 1 1 1


271

5. The process of reform and the Good Practice project To understand the context of the Good Practice project, we require knowledge of the reform process of the Italian university system. This can be divided into two periods: the New Public Management (NPM) initial phase and the NPM interlocutory phase, in which the central government impetus has weakened. The NPM initial phase began in the mid-nineties with the introduction of a new financing system that marked a real break with the past. A 1993 law (n. 537/93) set out new rules for managing the flow of resources from central government, which contributed to budgetary devolution. This law changed the role of the Ministry of Universities, which had previously been in charge of determining the overall resources allocated, the budget for each university and, within each budget, the allocations to specific subject areas and lines of expenditure (teaching and non-teaching staff, research financing). The 1993 reform removed this mechanism, introducing in its stead a single line budget that is now managed by universities at the local level. This first reform diminished the role of central government as a decision maker, but was followed, in 1996, by an expansion of the government’s role as an evaluator. A 1996 law (n. 662/96) introduced a new evaluation system for universities that operated on two levels: locally, each university had to set up an internal committee to oversee the transparency of resource management, research activities and teaching; and nationally, the government created a central national committee to evaluate the university system (CNVSU). Another step towards autonomy was taken with Law n. 210/1998, through which the government decentralised the selection and hiring of academic staff, which had previously been accomplished through national competitions. This devolution was enthusiastically welcomed by academics (Moscati, 2001), who saw it as an opportunity to increase their autonomy. However the administrative environment of the universities did not respond so immediately to the legislative reforms. This reluctance to implement change reflected the situation of the university administrative infrastructure in those years, which embodied all the characteristics of the old public sector model criticised by public management innovators (for example Osborne and Gaebler, 1992): the institutions were heavy and bureaucratic, staff generally came from a law background and had no managerial skills; there were no control systems in place. The only accounting system available was for providing financial reporting, based on cash principles, and mainly used for gaining formal authorisation and approval of expenditure. The idea of a change project targeted at universities emerged during this NPM initial phase in 1998, on the initiative and with the financial support of the CNVSU. In light of the reforms, the CNVSU wanted to conduct a benchmarking project with two main goals: first, to introduce PMSs and an accountability culture into university support services; second, to test the possibility of establishing an extensive national evaluation system. The initial experiment was enthusiastically welcomed by the participants, and the CNVSU financed two further projects to achieve wider consensus and diffusion. During the early 2000s, while the second and third project cycles were in progress, the university reform process moved into its second phase, the NPM interlocutory phase which is still underway today. After the major reforms of the nineties, central governments and evaluations bodies appeared to have lost the ability to steer and control universities (Rebora and Turri, 2008). Though some universities positively seized the newly granted autonomy, others exploited it unwisely (for example by hugely increasing the number of staff), causing long-term national expenditure to escalate further, and creating marked performance differences between universities (Rebora and Turri, 2008). To curb these rises in expenditure, the government took some measures that limited the freedom of universities, such as cutting specific lines of expenses (e.g. consultancies) and freezing the hiring of staff. However these responses were fragmented and inconsistent with the autonomy policy (Capano, 2008; Rebora and Turri, 2008). On the evaluation side, even though some assessment exercises were carried out (Minelli et al., 2008), the results only marginally affected resource allocation and decision making. This interlocutory phase is still ongoing, and there are increasing worries about the government’s ability to lead strategically and coherently (Capano, 2008). It was during this phase, and precisely in 2002, that the CNVSU decided to stop financing the Good Practice project, because they considered that the first task of introducing an accountability culture had been achieved, and were reluctant to pursue the idea of a national evaluation. However, despite the CNVSU’s decision, the universities involved decided instead to continue the project, which was from then on self-financed by the universities. Overall there have been a total of seven cycles, with the last one still ongoing in 2009. During the 11 years spanned by the Good Practice projects, the scope of the PMS evolved, extending both the range of operational processes tackled and the performance indicators measured (costs, objective quality and user satisfaction). Table 2 presents the evolution of the seven projects. 6. Translating ideas and shaping the network The examination of the translation of the PMS in the Good Practice project is divided into two sections; the first analyses construction of the PMS from 1998 to 2002, during which the government financed and supported the project. The second section focuses on the ensuing phase, when the universities decided to self-finance the benchmarking. 6.1. The Good Practice initial phase The idea of testing a PMS for university support services initially came from the CNVSU. In the wake of the NPM reforms, the Committee had already started to evaluate teaching and research. However support services were at that time neglected,

272


as noted by one of the Committee members: “We wanted to start evaluating support services, because they were becoming a significant item in the overall budget of the universities, but actually we started from zero in this area; we did not have a model, as in research and teaching. Furthermore, the administrative personnel were not used to measurement and management accounting: a system imposed by the central government would have been merely a formal exercise” (CNVSU member). The CNVSU’s intention was underpinned by an economic rationale and instrumentality to NPM reforms: the government wanted, on the one hand, to disseminate PMSs and a culture of accountability, and on the other hand to define a method for evaluating the performance of university administrative services. Though it is difficult to mark any clear starting point of the translation process (Latour, 1987), here we can consider it to have coincided with the enrolment of the researchers. From this first association, there emerged the initial statement of purpose of the project: to develop a PMS for support activities in universities and for comparing their performances. The first task that had to be tackled was eliciting the interest of university managers and, in view of their dormant state, this first enrolment was highly selective: it involved a search for institutions willing to commit to the idea, actively participate in the design, apply the system and then discuss their performance for comparison purposes. The CNVSU wanted to assemble a group that was receptive to the experiment, but also sufficiently authoritative to command external legitimacy within the Italian higher education system. The initial project was submitted, in 1998, to six administrative directors of universities in the south, centre and north of the country. The directors enthusiastically welcomed the initiative, even though its benefits and the activities entailed were not fully understood, as the following comment reveals: “When we met in Rome to discuss the initial ideas about Best Practice; we did not clearly understand what we were going to do as none of us had a performance measurement system; but we were curious and we trusted both the researchers and the committee. In some ways, we started the project without having a clear idea of where we were going to go” (Administrative Director at University 1). This group was enlarged after the first public presentation of the project, with four more universities included in the group, after they had made an official request to the CNVSU. The adherence of the universities to the committee’s idea in reality concealed different motives. Seven directors highlighted that their goals were to acquire knowledge about performance measurement and to understand how their performance rated in comparison with other institutions. Other universities instead participated in the study chiefly to gain legitimacy in the eyes of the CNVSU and the system, as evinced by this comment: “The government is starting this project with a selected group of universities and they are going to measure the costs, therefore it is better to be in this group, you never know what they will do afterwards” (Administrative Director at University 25). Once the initial group of university directors was enrolled, a first model of the PMS had to be defined and proposed. At this stage, the CNVSU and our research team entered into a negotiation: we wanted to test a system that was complete and interesting, while the CNVSU wanted a simple instrument that could be applied in less complex areas. The CNVSU’s reason for emphasising simplicity was the lack of competencies on PMSs, which it was feared might lead to abandonment of the system or to unintended consequences (Lawton et al., 2000; Popper and Wilson, 2003; Smith, 1995). The scope of the first PMS was accordingly limited only to costs, and restricted to three areas (Table 3). However the team persuaded the CNVSU to select three diverse areas to better reflect the range of users served by universities: research support services, accounting, and human resources. This choice then raised a further worry for the CNVSU: “These three services are organised very differently in the universities, how can we compare them reliably? The benchmarking should define target values for support service efficiency. Comparability between universities is crucial, otherwise they can say that the performance of University X is not comparable with the performance of University Y” (Member of the CNVSU). To tackle these controversies, the model was grounded in the existing literature, which led to the selection of a system based on activities rather than on organisational units. Such an activity-based model overcomes the problem evidenced in Relative Performance Evaluation (Bogan and English, 1994; Elnathan et al., 1996; Siverbo and Johansson, 2006). After the plenary presentation, the project entered its most active phase, circulating the first translation. During the first project cycle, only few changes were made to the initial proposal, and the suggestions were mainly focused on the activities. One officer suggested that the processes should be further subdivided in accordance with the official ‘Procedural Manual’ of his university, proposing a list of nearly 100 activities for each area. When a suggestion was not feasible, as in this case, it was necessary to detour the proponent’s original idea (Latour, 1987) using fact-building. This was accomplished, for example, through simulations which highlighted the difficulty of both collecting data and interpreting the results for 100 activities. These fact-building exercises helped to overcome the initial reluctance of officers and managers to embrace the ‘technology’, and they began to autonomously interact with the excel files, running their own simulations and proposing new changes to the protocol. These changes in turn meant that circulating translations was essential for maintaining the association between the managers and the PMS, as well as for enhancing their competencies.

Table 3 The evolution of the PMS—measurement indicators and areas. BP (later GP)

GP2

GP3 (or GP South)

GP 2003

GP 2005

GP 2007

GP 2009

Accounting

Costs

Costs Objective quality



Costs User satisfaction


Costs Objective quality Users satisfaction Costs Objective quality

Costs Objective quality User satisfaction Costs Objective quality

Costs Objective quality User satisfaction Costs Objective quality







Costs Objective quality Customer satisfaction Costs Objective quality Users satisfaction Costs Objective quality User satisfaction Costs Objective quality


Costs User satisfaction Objective quality

Costs Objective quality User satisfaction


Costs User satisfaction Objective quality

Student support services

Human resources

Costs

Procurement and logistics

Research support services

Information services

Costs


Costs

Costs Objective quality User satisfaction

Library services

Experimental areas Decision support systems Strategic planning Governance

Costs User satisfaction Objective quality Process comparison Process comparison


Area

Process comparison

273

274


Fig. 1. A visual inscription from the first project.

Although defining the protocol proved easier than expected, controversies began to increase when managers had to act the ‘quasi-object’. All the directors and managers realised that they needed the support of officers to collect data, and some reluctance emerged, at the organisational level, when administrative staff were asked to provide working percentages for the activity-based model: “Without having an idea of the project and the purpose for which the data would be used, we were asked to furnish percentages for a set of 15 activities. I thought they wanted to control my unit and how much we work, so I inserted 200% of time for each person” (Middle managers at University 18). This trivial device highlighted two problems that had previously been underestimated. First, the commitment of just the top managers was not enough, because the PMS affected the whole organisation; in particular, middle managers were key to permeating accountability across the organisation. Second, it emerged that the logic of the PMS was not clear: assigning a 200% percentage would theoretically have doubled the cost. Other controversies emerged in deciding the percentage for each activity, as the following phone conversation reveals:

Officer at University 7: “It is impossible to fill out this form [Activity percentage sheet]; how can I remember if my staff worked 12% or 11% on one activity or the other.” Researcher: “Actually the impact of a 1% difference on the total cost of the activity is marginal, do not worry about this.” Officer at University 7: “I do worry, though. I’m used to being precise, not to providing rough numbers!”

Here again, fact-building (Latour, 1987) proved precious, since it allowed the effects on overall costs to be simulated, showing how a change in percentage would have impacted on it. A final point of controversy emerged when collecting the activity drivers. An initial inconsistency of data collection was immediately detected from the variances of the unit costs, and the drivers were subsequently rendered consistent by verifying the collection modes one by one. Despite these efforts, the PMS was truly ‘discovered’ only when the benchmarking data were presented for the first time. During the final plenary presentation, we attempted to simplify the data and to use straightforward inscriptions to show the relative rankings of the universities (see Fig. 1) The simplicity of the graph helped stimulate interest, and the comparison with other universities became a moment of truth for several administrative directors, as the following comment shows: “Seeing, for the first time, how much it costs to carry out a competition, to deliver a certificate, to support our researchers in preparing a proposal, is incredible; what is more, we have a picture of our relative ranking in comparison with other universities. Now I need to understand our costs, their costs, and to try to figure out where we can improve our performance” (Administrative Director at University 1). The inscriptions that emerged from the first PMS succeeded in stimulating the interest of managers, not just in using the data, but also in continuing to test the system and enlarging its scope:


275

“I have the feeling that we have higher costs in this area (human resources), but I think the quality of our service is higher compared to other universities; I think we need to go further with extending the exercise, otherwise the picture of performance will be incomplete.” (Technical Referee at 5). With this awareness, managers also discovered the portability of numbers (Chua, 1995; Latour, 1987), grasping the possibility of going outside the network and using the measures and benchmarking data to justify organisational interventions. University 1 was a pace-setter in this process, acting soon after completion of the first project cycle: “After the first Good Practice Project, we had for the first time a clear idea of how much a student costs in terms of support services, and the comparative quality of our human resource services. I could support my perceptions with numbers, and for the first time I could demonstrate the need for reengineering interventions in some areas” (Administrative Director at University 1). The first project cycle gave the participants a novel perspective on their situation: support services were broken down, analysed and then reproduced in the PMS. Managers saw their tasks in a different light: from being a set of procedures often at the margin of the university world, services became activities that consumed costs and produced outputs. This shift, and the interactions with others, also altered their perception of their identity and importance within the higher education system. An awareness of this change, and the interest of managers, prompted the CNVSU to finance a second project, which was started in 2001 (Good Practice 2). The network grew in size (from 10 to 14), accepting the requests to participate of four other important universities. While this second project was underway, seven other universities asked to participate in the study. These institutions were smaller and based in the south of Italy; they were curious and willing to experiment, but were also afraid of being compared with universities already ‘legitimated’ in (and by) the project space. In the end, they suggested undertaking a separate experiment to gain a preliminary awareness of their performance. In 2002, these seven institutions formed a separate group and a third project, called Good Practice South, was set up. During the course of the three projects, direct interactions between the universities themselves increased; managers and officers belonging to specific areas created associations (Latour, 2005) with their counterparts in other universities. Furthermore, while the actors initially tended not to interact with the technology until they were obliged to do so by the research timescale, in the second and third projects the participants started to analyse and disassemble the PMS model and the calculation sheets (excel files) in order to understand the results in advance. The participants thus began to appropriate the technology, which had originally been regarded only as a tool in the hands of designers. Universities became much more active in giving suggestions, which in turn caused more controversies to arise when different universities gave contrasting suggestions. After discussion and simulations, all decisions and revisions of the system were then circulated within the network, communicating when a choice was a mediation of competing interests. Some controversies arose when comparisons of the universities’ performance were presented in the plenary sessions. At this stage, the controversies were raised by top managers, who are generally less involved in the operational phases, and dealt with by re-opening the ‘black-box’ (the evaluated measures) to show where different levels of performance came from. This heightened attention to the translation was instrumental for augmenting use of inscriptions outside the project context. It is here that the instrumentality of the PMS to organisational goals started to emerge. The numbers were discovered to be portable (Chua, 1995; Latour, 1996), and the data started to increasingly be used for official reports, external accountability, official speeches, and even to touch the academic sphere, generally considered to have little awareness of support services. Some universities used their comparative performance data to support and propose changes to their services. This led to actions that were sometimes circumscribe, such as the extension of opening hours for front line services, while in other cases involved extensive reengineering. The project outcomes in these cases were only a springboard that helped managers to amplify their voices (Latour, 1987): “The strength of the numbers we presented to the academic board comes from the Good Practice network: the reputation of the researchers, the comparison with other universities, and the external validation” (Technical referee at University 5). To summarise, during this initial phase, the four stages of PMS translation were a constitutive process (Chua, 1995) for the Good Practice network, in which interactions and experimentations provided a means for transferring and creating knowledge and for shaping the technology (Gendron et al., 2007; Mennicken, 2008). During this phase, university managers with no background in PMSs began to gain awareness of the system, its potentiality and the portability of the numbers. Although the translation had initially been dominated by the government as part of its broader reforming plan, the universities began to progressively gain a louder voice. This happened through the vehicle of controversies, which intensified and grew more visible during acting of the object. By interacting with the technology, the actors started to gain awareness of the PMS and of the difficulties inherent in data collection and comparability, but also of the possibility of interpreting their activities in terms of costs and quality. During this phase the approaches of the universities were fairly homogenous, however in the last project cycle some differences started to emerge, particularly when going out with the results. Some universities began to increasingly exploit the results of the Good Practice project, experimenting with the use of numbers to pursue their organisational goals; other universities were instead passive, sometimes because they were still uncertain about the PMS, and sometimes because their participation was merely a device for gaining legitimacy in the eyes of the government.

276


6.2. The Good Practice autonomy phase Rome, 2002, meeting between the CNVSU member, the research leader and three university directors who had been very active in previous Good Practice projects. CNVSU member: “I think we have achieved the goal of triggering dissemination of an accountability culture; continued financing in this direction would not be justifiable, and I’m very pessimistic about the situation here [central government]: the possibility of using benchmarking for central evaluation appears to be unfeasible; I’m sorry we cannot finance the project anymore”. Director of University 1: “I understand your [the CNVSU’s] position, but we cannot stop the project. We need continuity; the Good Practice project is an arena in which we can compare ourselves with others to see where we stand, how others are doing, and where we can improve our performance; why don’t we propose that the universities self-finance the study?” Director of University 5: “I totally agree, we cannot stop the comparisons, we need to continue measuring ourselves against others, otherwise we will become self-referential in our evaluations.” Director of University 3: “I agree, too, and I think the payment of a fee will in some way guarantee that all the organisations involved are committed to the project; now we know what we can obtain from the PMS”. This marked the start of the second phase of the Good Practice project, in which both the roles of the participants and the role of the PMS for them changed. In the initial phase the CNVSU and the financing were the pivotal elements for the universities’ association; the withdrawal of these elements challenged the Good Practice project and the PMS. Though sceptical of this, the project team made a direct proposal to the university directors. The enrolment in this second phase was not selective, and the project proposal was sent to all the Italian state universities. Surprisingly, 16 universities accepted to take part in the project, but the network was considerably reconfigured (see Table 1): some universities decided to no longer participate, now that they had to directly finance the project themselves, but on the other hand other new universities asked to be enrolled in the network, which was now perceived to be freer (without government control) and more committed. Whereas during the initial phase all the universities played a similar role in the project, after the CNVSU’s withdrawal, something changed: some universities saw this as a chance to partially appropriate the committee’s role and gain a prominent voice. The first self-financed project in 2003 was followed by three other cycles in 2005, 2007 and 2009, during which the network increased in size and the approach of the universities continued to evolve. The range of stratagems observed within the network was highly diverse and dynamic. Without inferring any set patterns of behaviour, we identified three categories of actors that captured the diversity of approaches on the spectrum: strategic allies, technical warriors, searchers. Following selected actors, we analysed how these three types acted within the network and their relative importance in the four stages of translation (Latour, 1987). Strategic allies refers to those top managers who progressively became spokespersons for the innovation (Latour, 1987): they wanted the project to survive and so promoted its results and benefits to other universities, and even to other networks. The Director of University 1, a major northern university, best exemplifies this category of actors, who understood the role that accounting could play not just within their own organisations, but in bonding the system: “In my university, but also more generally in our system, a culture of accountability is not so widespread; budget allocations and interventions are often the result of political negotiations in which costs and impact on performance are not evaluated. The benchmarking data provide us with a powerful instrument to show that we can measure performance in a manner that is meaningful to a large set of people, and which can provide a guide for future actions” (Administrative Director at University 1). The strategic allies saw accounting as instrumental to their organisational actions, but were also able to perceive the importance of the social relationships surrounding the technology, and the constitutive nature of the PMS: “Participating in the implementation process was crucial for my managers: they discovered how to implement measures consistently, but they also discovered that their activities have a cost.” (Administrative Director at University 1). Capturing this dual role of accounting, the strategic allies placed great emphasis on the initial stage of the PMS translation: setting the goal and enrolling actors. This focus was reflected in the network through their explicit requests to change the scope of the PMS, to include new organisational areas. In response to a growing number of informal requests of this type, in Good Practice 2007 we decided to formally collect suggestions from the top managers for possible future translations of the PMS. In the last project cycle, the Director of University 1 proposed a challenging aspect to be measured: governance capability. Despite the risk that no viable results could be achieved on this, governance was ranked as the first priority by all the universities. This, according to the Director, reflected a response to the void left by the government in the interrupted process of reform: “The government is now [2008] missing; they implement undifferentiated, sometimes controversial, actions to contain expenditure. The Ministry of Universities does not have a political plan and it does not take into account the


277

merits and performance of universities. If we do not ourselves find a way to maintain cohesion and compare ourselves on strategic matters, our system will implode” (Administrative Director at University 1). Furthermore, having understood the importance of relational aspects of the PMS, strategic allies were also particularly active in enrolling actors, both within or outside their own organisations. They wanted their staff to participate in the meetings, creating associations (Latour, 2005) with both the PMS and other actors, and preparing reports to facilitate the circulation of numbers internally: “Being directly involved in the meetings meant I could get in touch with my colleagues in Padua [northern Italy city] and create a virtual link with them; we are facing similar problems in Human Resources, and a comparison is always useful” (Human Resource managers at University 1). The contribution of strategic allies however weakened once the PMS entered the stage of operational design and technicalities, in which protocols and measures are revised. Here, it was the technical warriors that dominated. The actors included in this category regarded the technical aspects as the essence of the PMS. The Finance Director of University 7 was one of the most prominent actors in this category. Despite having a background in law, she was fascinated by the numbers and their power in supporting actions. After becoming involved in the project, she took part in eight training courses over a decade to enhance her knowledge of the subject. Her commitment proved instrumental to the use of data within her university, the effectiveness of which was, in her view, dependent on the technical features of the PMS: “This [the project space] is a safe area in which to discuss our performance and the benchmarking data, but when I use the data internally to convince my Rector and our deans that their costs are too high, or that the procurement services need improvements, I have to be sure that the methodology is perfect and that the data are comparable.” (Finance Director of University 7). Her vision was shared by several participants, and the technical warriors were particularly active during the circulation of translations for defining the protocol. Their strong focus on technical issues helped on the one hand to refine the system, but on the other hand it also increased the number of controversies in each phase of developing the PMS. Some of this resistance can be traced to the difficulties of implementing a PMS, for example the complexity of the services provided (Bouckaert and Balk, 1991; Holzer, 1991; Holzer and Kloby, 2005; Yang and Holzer, 2006). This was especially apparent in the last project, where one university proposed apportioning the costs of research support services among nearly 50 activities, using complex allocation criteria: “We cannot treat the Research Support Service in the same way as all the other services that we compare. This service should be split into a greater number of activities; this is my proposal [showing an excel table with 50 rows]. In the second column I suggest some more refined drivers, subdivided into different types of measures” (Research manager at university 18). Having become aware of the feasibility of the calculations, the technical warriors tended to propose increasingly detailed measures, sometimes losing sight of the cost of collecting data. Unexpectedly, our role here became to moderate the increasing love of the technology (Latour, 1996), by pointing out the limitations of the PMS. This was done by simulating the effects on the numbers and the impact on data collection; the availability of data from the previous project cycles proved precious here, since it allowed us to show real data before undertaking new data collections. With the increased association (Latour, 1987) with the technology, the number of controversies increased, and this affected the manner in which translations were circulated; more one-to-one interactions were used, with general communications sent out only to summarise the progressive revisions of the PMS. This deeper interaction with the technical warriors was also of benefit to us, the researchers; whereas in 1998 the services provided by universities were more standardised and stable (e.g. payroll payment, matriculation), they have since developed a wider range of activities, which change from year to year (e.g. spin off selection and support, psychological counselling, personnel climate assessment). Technical warriors and their controversies stimulated the research team to always maintain a link with the universities’ operational processes, as the following comment shows: “I know you are experts on the PMS, but research support is a complex service, and in the past few years its complexity has increased. Now we deal with patents, hundreds of different types of projects, licences and so on. The effectiveness and efficiency measures need to take this into account” (Officer at University 7) Although the operational translation of the PMS was dominated by technical warriors, when translations were circulated, controversies were also raised by searchers. The category of searchers comprises the universities and managers that participated in the project motivated by a ‘search for answers’: for example to improve their performance, to disseminate a culture of accountability within their university, or to gain legitimacy within the Good Practice network. All the universities that initially adhered to the project started out as searchers, before then changing their approach, or even continuing to be searchers. The Director of University 10 provides an interesting example of an enduring searcher: despite being involved since 2001, he continued to be only partially satisfied with the outcomes of the PMS, as the following comment shows: “We have not achieved the same results as the other universities; I struggle to motivate people and they raise a lot of oppositions when I show the results of the benchmarking” (Administrative Director at University 10).

278


Searchers were distinguished by their greater dependence on the research group. The number of questions, raised particularly when acting the object, was enormous. In the Good Practice 2007 project, University 10, alone, sent nearly 200 emails, and made more than 100 phone calls. While technical warriors and strategic allies were more independent at this stage, searchers still need clarification or reassurance. Nonetheless, their questions, too, proved extremely useful for enhancing the technical elements of the PMS, by sometimes revealing ambiguities that existed in the protocol definitions. Their contribution was also essential for continually verifying the managerial appropriateness of the PMS, since the searchers reported to the project space all challenges encountered locally in their organisations. They raised controversies at both the strategic and technical level, which were then discussed in the project space and translated into further improvements in the PMS: “Moving my organisation towards a measurement culture is so difficult, and sometimes it is really frustrating to see the passive reaction of the academic boards to the benchmarking data, or to see that my proposals are always opposed. When I discuss these problems here [project meetings] I can have an interaction that I cannot have within my university” (Administrative Director at 10). Searchers bring problems into the networks and try to seek ‘solutions’, by also observing and imitating, in a Machiavellian way (Chua, 1995), the stratagems of the universities, which they consider most successful: “We have observed what the other universities have done, so now we are trying to obtain higher commitment by involving more managers and officers in the project meetings. I’m also trying to motivate the technical referee to be our voice into and out of the network” (Administrative Director at University 10). Sometimes the searchers perceived that they did not have enough space in the discussions and in the network, and so decided to leave and withdraw from the project, as happened recently with University 35: “The GP2007 was our first project; we were enthusiastic at the beginning, but were not able to use the results effectively internally, and found it very difficult to raise issues during the meetings. For these reasons, and due to financial constraints, we decided not to participate in GP2009” (Financial Director at 35). But sometimes they came back, as in the case of University 20: “After GP2005 we had internal problems and did not have the time and the resources to follow the project, but the link with the leading universities is essential for us; given the absence of the government, the Good Practice meetings are way to see where universities are going” (Controller at 20). In the final translation stage of insiders going out, all three types of actors stepped up their actions over time, particularly in the second project phase, bringing the Good Practice results more frequently outside the project space. However they did so with different approaches. Strategic allies used the results to drive managers and governing boards to reflect upon key issues in higher education; in pursuing this goal they used the project results in public speeches within their universities, in national meetings and even in international events. Technical warriors were instead more focused on their organisational context, attempting to promote an accountability culture and ‘rational’ decision making. In so doing, the technical warriors discovered that it was not just the numbers that were portable, but also the method itself and the calculation sheets; using the Good Practice methodology and calculation tools, in 2005 University 7 tested an internal comparison between 57 departments: “I wanted to demonstrate that our medicine departments have very high costs. We used the efficiency measures defined in the Good Practice projects and the calculation sheets to conduct an internal benchmarking. The medical departments did in fact prove to be expensive, but they justified this because they are different; we need to compare them within the Good Practice project” (Finance Director of University 7). This internal comparison at University 7 prompted an enlargement of the scope of benchmarking in Good Practice 2007, to also compare the administrative activities of faculties and departments. While both strategic allies and technical warriors had become autonomous in going out, searchers’ reliance on the research group was greater also at this stage. We were asked to explain calculations, perform ‘ad hoc’ analyses and give presentations at their universities to legitimate the results of the benchmarking. To summarise, the autonomy phase of the Good Practice project revealed the heterogeneity of actors’ approaches during the process of translation. The government phase represented the ‘infancy’ of the Good Practice network, during which universities incrementally acquired awareness of the PMS, but with a role that was confined to certain stages of the translation. After the government’s withdrawal, the universities started to appropriate the whole of the translation process, reconfiguring the distribution of power within the network. In this reconfiguration, the roles and tactics of the universities became heterogeneous, resulting in a co-existence of louder and softer voices: strategic allies, technical warriors and searchers. Strategic allies, who perceived both the organisational instrumentality of the PMS as well as its power for bonding the system, dominated the first stage (setting the goals and enrolling actors), suggesting changes to the scope of the PMS and trying to enrol further actors in the network. Technical warriors instead had a louder voice during the circulation of translations, attempting with each cycle to improve the technical characteristics of the PMS. They regarded functional proprieties as crucial for leveraging the Good Practice results to pursue their own organisational goals, which was the main role technical warriors attributed to the PMS. Searchers instead had a softer voice within the network; they had not been able to fully


279

exploit use of the PMS within their organisations, and adhered to the network searching for ‘solutions’, sometimes mimicking the stratagems of other universities they considered to be more successful. Despite being less visible, they proved to play a crucial role in the process of translation, by signalling difficulties in exploiting the system outside the Good Practice arena, and preventing it from becoming self-referential. All three actor categories progressively increased the number and range of controversies which they raised during the translations, indicating greater interest but also posing a greater threat to the possibility of achieving a socio-technical compromise (Latour, 2005). 7. Discussion In this paper we have recounted the translation of a Performance Measurement System (PMS) across an expanding network of Italian universities. Adopting a Latourian perspective (Latour, 1987, 1996, 2005) and following up on previous contributions (e.g. Chua, 1995; Gendron et al., 2007; Preston et al., 1992; Townley et al., 2003; Watkins and Arrington, 2007), the longitudinal and expanding experience of the Good Practice project enabled us to observe the actors involved (Latour, 1987) examining specifically how university actors’ approaches in the network evolved and became differentiated over the 11 year of translation. With respect of this goal, there are two areas in which our findings contribute to an understanding of the translation of PMSs, and more generally of accounting, in the public sector: firstly, the significance of controversies during the translation, which were found to be not necessarily ‘bad’, but on the contrary essential for maintaining interest in the PMS and unfolding major issues in public sector measurement; secondly, the emergence of diverse categories of actors with louder or softer voices within the network, that enacted different tactics in endeavouring to reach their goals. A first area of results is related to the centrality of controversies. From a strictly functional view, the PMS works when it has solved the difficulties in public sector measurement. In this light, anyone entering the plenary meeting of the last Good Practice project in 2009 would certainly be convinced, from the discussion, that the 11-year experience had been a failure. When a new project is started there are always many controversies about comparability, measures, and activities. Participants indeed found some solutions in the Good Practice project, but the PMS never became a black-box, a closed ‘product’ that could be taken for granted (Latour, 1987), instead, the PMS remained a quasi-object (Serres, 1982), that was continually translated as new actors, areas and technical devices entered the network and as new controversies emerged (Latour, 1996). Controversies were found to be a central element for the process of translation. They constituted essential risks: essential because they reflected the enduring interest in measuring and managing public services; and risks because failure to achieve a compromise would bring the translation to an end. To ensure sustainability but also continuity, this study highlights the importance of defining a negotiable space within which some issues can continue to be discussed, while agreement is reached on others so that they become a ‘doctrine’ of the innovation (Latour, 1996). Specifically, the Good Practice network established three doctrinal features which addressed fundamental controversies of public services measurement. These ‘solutions’ were negotiated in the first project cycle, when the voices of CNVSU and the researchers were predominant. The first doctrinal feature responded to the need to define a target for support services (Bohte and Meier, 2000; Popper and Wilson, 2003; Van Thiel and Leeuw, 2002) and it is: target values are comparative values. This feature raised the problem of comparability (Bogan and English, 1994; Elnathan et al., 1996; Siverbo and Johansson, 2006), which was dealt with through an activity-based approach for comparing performance. Universities were compared based on a common set of activities, specified and detailed in an ad hoc protocol. The definition of this protocol proved to be an intensive phase, and the universities’ attention to it increased during the 11 years, improving the precision of the data collected. A second major problem was the need to take into account a wide variety of services and users (Epstein et al., 2006; Holzer and Kloby, 2005; Yang and Holzer, 2006). To confront this, in 1998 the CNVSU and the researchers settled on a second doctrinal feature: the PMS is a modular system based on activities. The ambition of defining a single measure for the university as a whole was never pursued; instead, the system was structured to be multidimensional and flexible in term of its areas (procurement; student support services) and performance metrics (costs, objective quality; subjective quality). This made it possible to add new services, as the need arose, and to tailor the quality accounting to different types of users: students, teaching staff, administrative personnel, top managers, academic board. The third doctrinal feature was: the implementation is interactive and incremental. This rule helped overcome the lack of competencies (Lawton et al., 2000; Popper and Wilson, 2003; Smith, 1995) of university managers, which was a major problem in 1998 when the project was first activated. University support services were a typical old-style bureaucracy, made up of administrators and managers who mainly came from law backgrounds, and had no expertise in management and performance measurement. Interactivity of the implementation made it possible, at each translation, to create an association between the managers, the researchers, and the technology. This association incrementally transferred awareness about the functioning of the PMS, its potentialities and problems, and its possible uses. Meanwhile, making the implementation incremental avoided introducing too many areas or performance measures all at once. Focusing only on costs and on a few areas in the first project cycle gave managers a chance to internalise the numbers and appropriate the potentiality of the PMS. Although the above ‘solution’ were never called into question, other problems instead continued to emerge during the 11 years of the project, and became the negotiable issues. These problems were related to the model of the PMS; although the activity framework was never questioned, as the managers began to interact more with the technology, they increasingly put

280


forward requests to change the model, generating controversies. Sometime these controversies arose from the conflicting views of different universities, which sought to detour the PMS to serve their particular needs or goals; in other cases, the controversies had to do with technical difficulties with calculations or data collection. The second area of results is related to the divergence of universities’ approaches during the process of translation. Without seeking to define any general rules of behaviour in translating the PMS, we decided to follow certain selected actors, thereby giving voice to more powerful as well as to weaker actors. In particular, we focused on three categories of actors, which we termed strategic allies, technical warriors and searchers. These actors exhibited markedly different stratagems and manners of associating in the network, in their individual attempts to exploit the role (Modell, 2009; Townley et al., 2003; Watkins and Arrington, 2007) that each attributed to the PMS. All the actors initially belonged to the searcher category, the softer voice in the network. Searchers adhered to the network because the PMS attracted their interest and appeared to offer a means for attaining their personal, organisational or visibility goals. In seeking solutions, searchers gave priority to association with humans, while viewing the technology as distant and not perfectly controllable. They relied on the researchers as their centre of calculation (Latour, 1987), and as a vehicle for putting forward organisational actions within their own universities. They also sought to forge ties with other key managers in the network, to find inspiration in their erratic search. In establishing these connections, searchers brought new (or old) controversies into the network, helping to sustain lively interest in the PMS, when acting the quasi-object. However searchers were also the actors most likely to abandon the network, especially when they viewed the PMS purely as a device for gaining legitimacy; if their problems became marginalised within the network, they readily diverted resources from Good Practice to other projects. That said, some searchers have remained in the network since 1998, attributing to the PMS the role of a bonding system, and treating the network as a privileged arena for interchanges in a public setting that is perceived as fragmented and contradictory. For technical warriors, instead, the main association was with the technology. They considered the technical features to be the central elements of the PMS and fundamental for pursing their organisational goals. They sought to increase precision and measurability by specifying protocols, they added indicators to achieve better completeness, and added new areas of comparison to extend the organisational penetration. When dissatisfied with the results, they tried to modify the technology. Technical warriors were particularly active during the circulation of translation, and it was sometimes challenging to mediate the controversies between conflicting technical warriors. However, they too proved to be a precious category within the network, since they immediately signalled inconsistencies in the measurements, and functional problems. Finally there were the strategic allies, who more holistically captured the role of the PMS. In addition to using the Good Practice results within their own universities, they saw the PMS as a bonding system. This role became very clear during the second phase of the project, after the government’s withdrawal. In terms of their approach to the network, strategic allies took a holistic view of the PMS as a socio-technical innovation, in which both human and non-human associations are important. They were active in all the stages of PMS translation, but they predominated in setting goals and enrolling actors, a stage considered essential, in their view, for stimulating university managers to address issues critical for responding systemically to the evolving socio-economic situation. Whereas technical warriors saw the PMS more as a diagnostic system, strategic allies also viewed the PMS as an interactive tool. This interactivity was leveraged not only within their organisations, but also within the Good Practice network, recognising the importance of comparing views on critical issues and giving voice to all universities, irrespective of whether they had good experiences or ‘bad practice’ (informant’s expression) to share in the arena. This comment from a strategic ally illustrates the relational approach of this category of actors: “The Good Practice project is a lever for change, but first and foremost a means for gaining knowledge about our strengths and weaknesses, and our margins for improvement. I’ve often talked about the effectiveness of the project for building a friendly comparison around the numbers culture, whilst respecting other experiences and realities. Good Practice is an accountability exercise: it teaches how to account for resources in processes and decisions. When I speak with other people outside the university system, they are surprised that we are able to account for the costs of student careers or the quality of services. Their surprise derives from the preconceived idea that the public sector in Italy operates bureaucratically, using money received from the government, without giving any consideration to the value of costs and quality. The Good Practice project enables me to support my decisions with data, to motivate top managers, to transfer accountability instruments to them, and provides a point of reference for administrative directors. I know that not all my colleagues have reaped the same ‘fruits’ from Good Practice, but not all fields are equally fertile. Still, it is important to continue sowing and to also give space to these situations; I know perfectly well what it means to not be heard within your organisation” (Administrative Director at University 1). 8. Conclusion The process of translating PMSs in public sector organisations is a field of research that has yet to be fully explored (Modell, 2009). Although previous contributions have analysed accounting and PMSs in the context of NPM reforms (see for example Bracci, 2009; Chua, 1995; Robinson, 2003; Townley et al., 2003; Watkins and Arrington, 2007), there was a need for further work (Modell, 2009) to capture the variety of dynamics involved in translating them into operational use. To help fill this gap, this article longitudinally examines the Good Practice project, promoted in 1998 by the government during the initial NPM phase (Moscati, 2001; Rebora and Turri, 2008), with the aim of implementing PMSs to install a new


281

accountability culture in Italian universities. Drawing on ANT, we sought to shed light on how universities translated the PMS over an 11-year period, highlighting two pivotal elements in the process of translation: the role controversies, and the emergence—and importance—of a diversity of actors’ approaches. From a functional perspective, an implementation is considered successful when all controversies have been solved; this study instead found that controversies represent ‘essential risks’ for the PMS: essential because they maintain interest and discussion within the network; and risks because they always challenge the possibility of achieving a socio-technical compromise. This critical reflection about the Good Practice project provides potentially useful insights for managers and policy makers undertaking PMS implementations. It suggests that controversies should not be seen as something that is necessarily ‘bad’, and therefore to be prevented or immediately resolved, but rather as useful signals indicative of the dynamicity, needs and divergences of actors in the network. Indeed, following actors’ controversies through the four stages of translations demands extensive time and effort, for which a mediator can be important not so much to impose a pre-defined view of the PMS, but to capture what is and is not negotiable. Together with the challenging role of controversies, the translation of PMSs into operational use is marked by the heterogeneity of actors involved, and the extent to which they have voice in the process. The Good Practice project made it possible not only to highlight the variety of actors and stratagems, but also to track their evolution over 11 years. To represent this broad spectrum of actors and actions, we chose to follow three categories of actors that exhibited particularly distinctive approaches: strategic allies, technical warriors and searchers. This focus enabled us to analyse in detail the relationships between actors and the PMS, and the ensuing repercussions on the process of translation. With their more powerful or weaker voices, they all contributed to the enduring experience of the Good Practice project. Strategic allies took a holistic relational approach, and looked upon both the technology and the human actors as important; though they were present in all stages, they played a predominant role in setting the goals and enrolling actors, bringing into the network controversies critical for bonding the university system. Technical warriors were instead engaged mainly with the technology, and saw the functional features of the PMS as the focal point. They played an essential role in translating the protocol, and in verifying the technical appropriateness of the PMS and its utility at the organisational level. Searchers instead represented a weaker voice within the network, that might at first glance appear not to be central to the PMS translation. However this study suggests that even uncertain, problematic actors are crucial to the translation process because they continually question the role of the PMS and its effectiveness. With their unresolved problems, searchers brought into the Good Practice network organisational and technical controversies which challenged the other actors. These results suggest some avenues for future research, to further capture the heterogeneity in the process of translations. First, they point to the utility of paying attention not just to the louder voices, champions or opponents, but also to the more silent actors or organisations. The silent and unheard help to capture the complexity of institutions and the polyphony (Arnaboldi and Lapsley, 2010) which underlies the translation of accounting techniques. Finally, our results, and specifically the centrality of controversies, suggest further analysing the roles and responsibilities of mediators. Immersed at the crossroads between actors and technologies, mediators guide the definition of negotiable and doctrinal issues of the PMS, and indirectly shape the configuration of the network, selecting and amplifying the actors’ voices, but also pushing them towards specific technological ‘solutions’. Acknowledgements The authors kindly thank David Cooper, Irvine Lapsley, and the three anonymous reviewers for their helpful comments on earlier versions of this paper. References Arnaboldi M, Lapsley I. Asset Management in Cities: Polyphony in Action? Accounting, Auditing & Accountability Journal 2010;23(3). Beryl AR. The government performance and results act and the tradition of federal management reform: square pegs in round holes? Journal of Public Administration Research and Theory 2000;10(1):111–35. Beyle HC, Parratt SD. Public attitudes and government efficiency. Annals of the American Academy of Political and Social Science 1938;199:26–32. Bloor D. Anti-Latour. Studies in History and Philosophy of Science 1999;30(1):81–112. Bogan CE, English MJ. Benchmarking for best practices: winning through innovative adaptation. New York: McGraw Hill; 1994. Bohte J, Meier KJ. Goal displacement: assessing the motivation for organizational cheating. Public Administration Review 2000;60(2):173–82. Boland T, Fowler A. A systems perspective of performance management in public sector organisations. International Journal of Public Sector Management 2000;13(5):417–46. Bouckaert G. The history of the productivity movement. Public Productivity & Management Review 1990;14(1):53–89. Bouckaert G. Measurement and meaningful management. Public Productivity & Management Review 1993;17(1):31–43. Bouckaert G, Balk W. Public productivity measurement: diseases and cures. Public Productivity & Management Review 1991;15(2):229–35. Bracci E. Autonomy, responsibility and accountability in the Italian school system. Critical Perspectives on Accounting 2009;20(3):293–312. Broadbent J, Laughlin R. Evaluating the ‘new public management’ reforms in the UK: a constitutional possibility? Public Administration 1997;75(3):487–507. Broadbent J, Laughlin R. Control and legitimation in government accountability processes: the private finance initiative in the UK. Critical Perspectives on Accounting 2003;14(1-2):23–48. Busco C, Quattrone P, Riccaboni A. Management accounting: issues in interpreting its nature and change. Management Accounting Research 2007;18(2):125–49. Capano G. Looking for serendipity: the problematical reform of government within Italy’s universities. Higher Education 2008;55(4):481–504. Chow CW, Ganulin D, Haddad K, Williamson J. The balanced scorecard: a potent tool for energizing and focusing healthcare organization management. Journal of Health Care Management 1998;43(3):263–80.

282


Chua WF. Experts, networks and inscriptions in the fabrication of accounting images: a story of the representation of three public hospitals. Accounting, Organizations and Society 1995;20(2-3):111–45. CIPFA. CIPFA benchmarking club; 2008, http://www.ipfbenchmarking.net. Collins H, Yearley S. Epistemological chicken. In: Pickering A, editor. Science, practice and culture. Chicago: University of Chicago Press; 1992. p. 301–26. Cooper D, Hopper T. Critical theorizing in management accounting research. In: Chapman C, Hopwood A, Shields M, editors. Handbook of management accounting research. Oxford: Elsevier; 2007. De Bruijn H. Performance measurement in the public sector: strategies to cope with the risk of performance measurement. The International Journal of Public Sector Management 2002;15(7):578–94. Dent JF. Accounting and organizational cultures: a field study of the emergence of a new organizational reality. Accounting, Organization and Society 1991;16(8):705–32. Doolin WR, Lowe A. To reveal is to critique: actor–network theory and critical information systems research. Journal of Information Technology 2002;17(2):69–78. Elnathan D, Lin TW, Young SM. Benchmarking and management accounting: a framework for research. Journal of Management Accounting Research 1996;8:37–54. Epstein P, Wray L, Harding C. Citizens as partners in performance management. Public Management 2006(November):18–22. Gendron Y, Cooper DJ, Townley B. The construction of auditing expertise in measuring government performance. Accounting, Organizations and Society 2007;32(1/2):101–29. Harris J. The ordering of things: organization in Bruno Latour. Sociological Review 2005;53(s1):163–77. Holzer M. Building capacity for productivity improvement. Public productivity & Management Review 1991;15(2):113–22. Holzer M, Kloby K. Public performance measurement: an assessment of the state-of-the-art and models for citizen participation. International Journal of Productivity and Performance Management 2005;54(7):517–32. Hood C. A public management for all seasons? Public Administration 1991;69(1):3–19. Hood C. The ‘New Public Management’ in the 1980s: variations on a theme. Accounting, Organizations and Society 1995;20(2/3):93–109. Johnsen Å. What does 25 years of experience tell us about the state of performance measurement in public policy and management? Public Money and Management 2005;25(1):9–17. Jones MJ, Mellett HJ. Determinants of changes in accounting practices: accounting and the UK Health Service. Critical Perspectives on Accounting 2007;18(1):91–121. Kaplan R, Norton DP. The balanced scorecard—measures that drive performance. Harvard Business Review 1992;70(1):71–9. Lapsley I. The NPM agenda: back to the future. Financial Accountability & Management 2008;24(1):77–96. Latour B. Science in action: how to follow scientists and engineers through society. Cambridge, MA: Harvard University Press; 1987. Latour B. Aramis: or the love of technology. Brighton: Harvester Wheatsheaf; 1996. Latour B. Pandora’s hope: essays on the reality of science studies. Cambridge, MA and London: Harvard University Press; 1999. Latour B. Reassembling the social: an introduction to actor-network-theory. Oxford: Clarendon; 2005. Lauzel P, Cibert A. Des Ratios au Tableau de Bord. Paris: Entreprise moderne d’edition; 1959, reprint in 1962. Lawton A, McKevitt D, Millar M. Coping with ambiguity: reconciling external legitimacy and organizational implementation in performance measurement. Public Money and Management 2000;20(3):13–20. Lowe A. After ANT: an illustrative discussion of the implications for qualitative accounting case research. Accounting, Auditing and Accountability Journal 2001;14(4):327–51. Lowe A. The spacing and timing of [a chic] critique. Critical Perspectives on Accounting 2004;15(5–6):279–91. McLean C, Hassard J. Symmetrical absence/symmetrical absurdity: critical notes on the production of actor-network accounts. Journal of Management Studies 2004;41(3):492–519. Mennicken A. Connecting worlds: the translation of international auditing standards into post-soviet audit practice. Accounting, Organizations and Society 2008;33(4/5):384–414. Minelli E, Rebora G, Turri M. The structure and significance of the Italian research assessment exercise (VTR). In: Mazza C, Quattrone P, Riccaboni A, editors. European universities in transition: issues, models and cases. Cheltenham: Edward Elgar; 2008. p. 221–36. Modell S. Performance measurement and institutional processes: a study of managerial responses to public sector reform. Management Accounting Research 2001;12(4):437–64. Modell S. Institutional research on performance measurement and management in the public sector accounting literature: a review and assessment. Financial Accountability & Management 2009;25(3):277–366. Moscati R. Italian university professors in transition. Higher Education 2001;41(1):103–29. Osborne D, Gaebler T. Reinventing government: how the entrepreneurial spirit is transforming the public sector. Reading, MA: Addison-Wesley Publishing Company; 1992. Pallot J. A wider accountability? The audit office and New Zealand’s bureaucratic revolution. Critical Perspectives on Accounting 2003;14(1/2):133–55. Pollitt C, Bouckaert G. Public management reform: a comparative analysis. Oxford: University Press; 2004. Popper C, Wilson D. The use and usefulness of performance measures in the public sector. Oxford Review of Economic Policy 2003;19(2):250–67. Preston AM, Cooper DJ, Coombs R. Fabricating budgets: a study of the production of management budgeting in the National Health Service. Accounting, Organizations and Society 1992;17(6):561–93. Rebora G, Turri M. La governance del sistema universitario in italia: 1989–2008. Liuc Papers Serie Economia Aziendale 2008;32(221):1–26. Robinson P. Government accountability and performance measurement. Critical Perspectives on Accounting 2003;14(1/2):171–86. Rosa EB. Expenditures and revenues of the federal government. Annals of the American Academy of Political and Social Science 1921;95(1518):1–32. Serres M. Le Parasite. Paris: Grasset; 1980 [Schehr LR (trans). The parasite. Baltimore, MD: John Hopkins University Press; 1982]. Siverbo S, Johansson T. Relative performance evaluation in Swedish local government. Financial Accountability & Management 2006;22(3):271–90. Smith P. Outcome-related performance indicators and organizational control in the public sector. British Journal of Management 1993;4(3):135–51. Smith P. On the unintended consequences of publishing performance data in the public sector. International Journal of Public Administration 1995;18(2/3):277–310. Townley B, Cooper DJ, Oakes L. Performance measures and the rationalization of organizations. Organization Studies 2003;24(7):1045–71. Van Thiel S, Leeuw F. The performance paradox in the public sector. Public Performance & Management Review 2002;25(3):267–81. Wang X, Gianakis GA. Public officials’ attitudes toward subjective performance measures. Public Productivity & Management Review 1999;22(4.):537–53. Watkins AL, Arrington CE. Accounting, new public management and American politics: theoretical insights into the national performance review. Critical Perspectives on Accounting 2007;18(1):33–58. Yang K, Holzer M. The performance—trust link: implications for performance measurement. Public Administration Review 2006;66(1):114–26.

Constructing performance measurement in the public sector - CiteSeerX

Constructing performance measurement in the public sector - CiteSeerX

Suggest Documents

Performance Measurement in the Flemish Public Sector: A ... - CiteSeerX

Performance Measurement in the Flemish Public Sector: A ... - CiteSeerX

The Use of Performance Measurement Systems in the Public Sector ...

The Roles of performance measurement in the English public sector

Public Sector Performance Measurement and Stakeholder Support

Benchmarking and performance measurement in public ... - CiteSeerX

Performance Appraisal in the Public Sector in Niger - CiteSeerX

Public Sector Performance

Personalities and Public Sector Performance: Evidence ... - CiteSeerX

Personalities and Public Sector Performance: Evidence ... - CiteSeerX

Public Sector ManageMent, truSt, PerforMance, and ... - CiteSeerX

Performance Audit in the Public Sector

ENTREPRENEURSHIP IN PUBLIC SECTOR ... - CiteSeerX

BUDGET RETRENCHMENT IN THE PUBLIC SECTOR - CiteSeerX

Exploring Entrepreneurship in the Public Sector - CiteSeerX

PUBLIC SECTOR PERFORMANCE CONTRACTING IN ... - OECD.org

investment appraisal in the public sector - CiteSeerX

Public Sector Performance Auditing in Fijiâ¦â¦

Improving public sector performance by strenghtening the ...

Design Performance Measurement in the Construction Sector: A Pilot ...

Design Performance Measurement in the Construction Sector: A Pilot ...

Agenda Item 6 - Public Sector Measurement - IPSASB

1 The Role of Performance Measurement in the Public Administration ...

development of performance audit in public sector - The Institute of ...

Constructing performance measurement in the public sector - CiteSeerX