Total Quality Management, Critical Systems

11te konferanse for nettverk for organisasjonsforskning i Norge – NEON 2014

The effect of EFQM self-assessments on action research performance: Lessons on how to improve the efficiency and effectiveness of research on New Public Management Petter Øgland Department of Informatics, University of Oslo, Norway Abstract. New Public Management (NPM) is based on the idea that public sector organisations can be made more effective by adopting management perspectives and methods more similar to those used in the private sector, but the economic and managerial rationality of NPM often conflicts with practitioner rationality, causing challenges in the implementation of NPM. Action research could be relevant for studying NPM implementation by searching for solutions that are seen as satisfying from both for the managerial and the practitioner perspective, but action research is a demanding type of research with high risk of failure. On the other hand, if one believes that NPM will improve organisational performance, then perhaps it will also improve action research performance. The aim of this study is to investigate whether the efficiency and effectiveness of action research can be improved by using the NPM logic of the EFQM self-assessment method. The question is theoretically investigated through the perspective of double-loop learning and empirically tested for eleven months in the context of using canonical action research (CAR) for implementing NPM in a Norwegian public sector organisation. Despite challenges in establishing and sustaining the CAR process, the EFQM approach proves helpful. Contributions to theory and practice is summarised in how the effect of EFQM self-assessments contributes in improving single-loop learning, double-loop learning and deutero-learning in action research. Keywords: New public management, EFQM self-assessments, total quality management, double-loop learning, canonical action research.

1. Introduction The aim of new public management (NPM) is to improve public sector cost-efficiency by changing how public sector organisations operate (Hood, 1991). Implementing NPM is not easy. Professionals working within organisations where NPM is being implemented sometimes complain that the NPM view on improvement brings along a type of economic and managerial logic that is conflicting with practitioner logic. Perspectives like NPM tend to introduce the need for procedures, documentation and measurements in a manner that stifles the creativity, flexibility and general attitudes that are necessary for making the professional perform at an optimum level. In support of the practitioner view, Fukuyama (2013) argues that NPM-like programmes have focused too much on measuring results and too little on measuring competence and autonomy. Similar views have also been argued by organisational and management scholars earlier, e.g. Mintzberg (1996). One way of developing knowledge about how to implement NPM in a manner that is not only acceptable by the practitioner community but also supports the logic of the practitioners and helps develop competence and autonomy is the method of action research. Writing specifically about the issue of using action research for cultivating local knowledge (practitioner knowledge) in the context of the economic and managerial knowledge developed through the use of means like business process reengineering (BPR) and total quality management (TQM), McNiff (2000) makes account of action research success in the context of organisational learning. She also explains some of the challenges of dealing with power

11te konferanse for nettverk for organisasjonsforskning i Norge – NEON 2014 and control when the researcher becomes part of organisational politics. As pointed out by Simonsen (2009), action research is a demanding type of research with high risk of failure. However, as the purpose of introducing NPM was to make the organisation more effective and efficient from an economic and managerial perspective, what would be the effect of applying a NPM-like logic for designing and managing the action research project? While many action researchers would be reluctant to try such an effort as the practitioner logic of the action researcher crashes in similar way with the economic and managerial logic of NPM-like efforts like TQM, it would nevertheless be interesting to study the effects of TQM on action research in a more controllable environment provided by doing self-assessments. Action research in the form articulated by Kurt Lewin in the 1940s was a type of engineering science used for researching strategies for social change (Gold, 1999, p. 295). Today there are many forms of action research. Canonical action research (CAR) is one of several action research variants that have been developed within the information systems (IS) community for the purpose of researching organisational change in the context of IS (Davison et al, 2004). As IS plays an important role in implementing NPM infrastructure, CAR could be an interesting type of action research to study. A further interesting aspect of CAR is that it is argued to be robust through a set of five principles that have been developed as a consequence of analysing numerous cases of IS action research failure. The principles CAR are related to (1) researcher-client agreement, (2) cyclical process model, (3) theory, (4) change through action, and (5) learning through reflection. Even though the principles may provide a sound way of making sure that the action research produces reliable and valid results, it may not always be an easy task to make sure that the criteria associated with the CAR principles are being met. According to Clark (1972), action research involves interplay between problem owners, practitioners (action researchers) and research audiences. When Øgland (2014) looks at the problem of developing CAR researcher-client agreements in politically challenging environments, he makes use of game models describing the relationship between clients (problem owners), researchers (practitioners) and the scholarly community (research audiences). Although understanding the researcher-client agreement game is important for succeeding with CAR, the researcherclient agreement is only one of the five principles. Getting a good researcher-client agreement without aligning with the other principles is not expected to result in sustainable and effective CAR. Studying the effect of TQM on action research in real-life practice requires having an action research process to investigate. Within the director-general’s IT staff at the Norwegian Tax Administration (NTAX) there has been a challenge on how to improve the practice of IT Governance since a reorganisation in 2008. It has been recognised both from within NTAX and by external consultants hired by NTAX (DNV, 2005; Gartner, 2010) that there is a vital need for improving ITG, but at the same time it has also been recognized both from within and without that the NTAX culture may be the organisations greatest enemy in the sense that it prevents conventional step by step methods of implementing ITG. In November 2013 there were initial discussions about starting an action research process for looking into the situation. As part of the feasibility study, the EFQM assessment model (Oakland, 1999) was used for diagnosing the organisational readiness for trying to improve ITG through the use of CAR. The paper is structured into six sections. After having motivated the research and presented the main hypothesis in this introductory section, the next section is a select review of

11te konferanse for nettverk for organisasjonsforskning i Norge – NEON 2014 literature on action research and the EFQM assessment model seen from the perspective of double loop learning. This is followed by a methodology section where it is explained how the theoretical models are to be empirically investigated through the use of a real-world experiment. Analysis of the outcome of the experiment is presented in the fourth section. This is followed by a fifth section that discusses the analysis of results in the context of related work. The final section summarises the study by identifying contributions to theory and practice plus directions for further research.

2. Literature review The purpose of the literature is to develop an action research strategy where the EFQM selfassessment model has been integrated with the action research methodology. The action research strategy will be explained within the context of what Argyris and Schön (1978) refer to as double-loop learning.

2.1 How to implement action research as double-loop learning There are studies on using action research to improve organisational EFQM scores (e.g. Prybutok & Ramasesh, 2005). There are also action research studies where the EFQM model has been integrated in the research design. For example, Kontostavlaki et al (2010) refer to experience in using EFQM as a foundation for action research in the context of assessing and evaluating environmental programmes at school with the aim of increasing the environmental awareness and sensitivity among the students. Nevertheless, no literature has been found on how to use the EFQM model for improving action research. This does not mean that the action research community sees no need for quality improvement. Boog et al (2008) have written specifically about the need for quality improvement of action research, but their focus has been on the ethics and moral standards that are used as governing variables in action research rather than the standards, methods and models of TQM. On the other hand, according to Lilford et al (2003) there is little difference between action research for organisational change and TQM. Both action research and TQM are cyclical activities involving examination of existing processes, change, monitoring the apparent effects of the change and further change. Both emphasise active participation of stakeholders. The examples used to illustrate action research would serve equally well as examples of TQM and vice versa, they claim. When considering the structure and nature of the EFQM model, the argument made by Lilford et al can be made more explicit. Action research is often structured as a cyclic process. The so-called RADAR logic of EFQM is a four-step cyclical ordering of activities involving specifying results (R), planning and developing approaches (A), deploying approaches (D), assessing and refining (A&R). In action research there is examination of existing processes and monitoring of effects on change after having designed interventions according to some theory of change. Examination of existing processes in the EFQM context is done by comparing the organisation with the five enabler criteria of the EFQM model. The effects of change are monitored through the lens of the four results criteria of the model. The theory of change is embedded in the model by how the results criteria and enabler criteria are linked at a sub-criterion level represented by a 32 by 32 logical matrix. If one compares this structure with a structured version of action research, like canonical action research (CAR), there is a strong match between action research and TQM, just as Lilford et al suggest. As the purpose of the EFQM model is to help organisations reflect both on what they are doing and how well they are doing it, double-loop learning (Argyris and Schön, 1978) can be

11te konferanse for nettverk for organisasjonsforskning i Norge – NEON 2014 a useful perspective when considering how to integrate TQM and action research in the context of organisational change. The theory of double-loop learning was developed from the perspective that people and organisations often have a tendency for changing without questioning whether they are focusing on the right issues to change. Although learning and improvement without questioning governing variables may result in increased efficiency, the increased efficiency may be related to things that do not matter or matter in a negative way. As illustrated in figure 1, change in efficiency by designing and implementing action strategies should be guided by considerations on effectiveness in terms of reflecting on the governing variables.

Figure 1. Model of how learning happens in practice (Argyris et al, 1985, p. 84)

For the purpose of researching double-loop learning, Argyris et al (1985) have suggested a particular type of action research they refer to as “action science”. The reason for doing so is because they felt that too much action research lacked theoretical focus. Systems theory, control theory, decision theory and the critical theory of the Frankfurt school exemplify theories of action (ibid, chapter 1). When applying this kind of theory, there are two types of empirical hypothesis to consider, they say, the first type is of the form “Agent a has disposition d” and the second type has the form “Action (or pattern of actions) a will lead to (be causally responsible for) consequences c” (ibid, p. 55). They also articulate the latter formula as “in situation s, to achieve a consequence c, do action a” (ibid, p. 81). As the motivational problem in this study has to do with the implementation of IS strategies for ensuring the development of NPM, canonical action research (CAR) is a more useful form of action research in the context of the IS community. Nevertheless, CAR and action science have much in common. For example, CAR hypotheses are expected to be formulated along the lines of “in situation S with salient features F, G and H, the outcomes X, Y, Z are expected from actions A, B and C” (Davison et al, 2004, p. 74). What this means in the context of the model in figure 1 is that the action research hypothesis is a causal statement dealing with governing variables, action strategy and consequences. Single-loop learning deals with the relationship between actions A, B and C and the consequences X, Y and Z when salient features F, G and H of the situation S (governing variables) are being left unquestioned. Double-loop learning deals with reflection about the situation S with salient features F, G and H in relation to what has been observed. In other words, single-loop learning is learning about the efficiency of the action strategy. If the action strategy was a strategy for developing NPM, single-loop learning would be concerned with issues like the time and cost for implementing NPM using this strategy. Double-loop learning, on the other hand, is concerned with issues like why one would be interested in NPM, whether the action strategy is based on a proper understanding of the situation S and whether the salient features F, G and H are the most useful features for describing S.


Another important issue in double-loop learning theory is the issue of how to implement double-loop and single-loop learning. The motivational position taken by Argyris and Schön (1978) is that organisations and individuals are generally good at single-loop learning as this usually means adapting to the norms of the organisation while they are generally bad at double-loop learning as this means questioning norms and culture. Other scholars, for example Lyytinen and Robey (1999), are more negative in terms of pointing out how cultural norms and politics may often hinder even single-loop learning. In the context of implementing information systems (IS), not only do organisations fail to learn when introducing IS, the IS development process becomes an acceptable means for explaining failure. Organisations learn to fail. Brunsson and Jacobsson (2000) have a similar view on organisational learning in the context of TQM. As they see it, problems like implementing double-loop learning cannot be understood in the context of looking at the organisation in isolation. Each organisation is tangled up in a political network of suppliers, customers, management consultants, quality standards and competitors. Often it is easier to explain why organisations are not learning through the use of TQM by considering who the important actors are. The outcome can be predicted by looking at what these actors have to gain or loose by complying with TQM standards as compared to pretending to comply. Brunsson and Jacobson believe that the market economy results in fake TQM in the sense that the most important actors in the TQM game (organisations, management consultants, TQM auditors, makers of TQM models, methods and standards) benefit from organisations being seen to comply with TQM even when they do not necessarily do so. The model in figure 1 is useful for conceptualising action research about NPM implementation as an action strategy. As mentioned above, it is quite likely that the consequences of the action strategy will be that nothing happens. According to the logic of the diagram, this may be cause for reflection about the governing variables that have been used for designing the action research. McKay and Marshall (2001) have argued the need for action research to be designed in such a double-loop manner to make sure that not only does the research result in learning about the client problem but also about effective research designs. Looking at the same problem within the context of software process improvement, Øgland (2007) argues that quality standards used for process improvement also define the nature of the kind of scholarly knowledge that needs to be developed. Integrating the EFQM assessment method with action research is an extension of this idea. Hypothesis 1. The EFQM self-assessment method is useful for learning about how to implement action research as double-loop learning. . The hypothesis above was motivated by the ideas that action research can be made more effective when viewed from the perspective of the action (process of organisational change) and research (producing new and relevant knowledge) being seen through a lens of doubleloop learning when both processes are being linked through the use of a quality standard like EFQM. The next two sections of the literature review will focus more specifically on what it means to integrate EFQM and action research through the double-loop learning model in the context of making action research effective and efficient.


2.2 How to make action research effective An example of an ineffective NPM action research would be a type of NPM implementation study that focuses on issues that are irrelevant for the purpose of implementing NPM. For example, if the action researcher questions the NPM ideology and focuses his research on explaining resistance to change based on the oppressive nature of NPM, he may produce an interesting case study, but it will be ineffective as action research. It is ineffective action research because the governing variables in the action research strategy are not aligned with what the client is trying to achieve in terms of implementing NPM. On the other hand, if the researcher is working for a client who wants to increase workplace democracy and reduce social injustice, a critical perspective on how the organisation is implementing NPM might be instrumental for making the action research effective. As TQM assessment models like the EFQM model are used for identifying the aims of an organisation, how it is performing with reference to such aims, and what kind of methods it is using to achieve the aims, assessing the action research against such TQM standards should in principle be useful for developing double-loop learning and increased effectiveness. An overview of the EFQM model is presented in figure 2. The model consists of nine criteria that are further divided into a set of 32 sub-criteria. As indicated by the diagram, the first five criteria on the left are described as enables as they are used for evaluating issues like leadership, people management, policy and strategy, resource management and process management. The final four criteria on the right are termed results criteria and are used for evaluating organisational performance through a “balanced scorecard” of people results, customer results, impact on society, and key performance business results.

Figure 2. EFQM model with weights and relationships between criteria

When measuring an organisation as a result of doing an EFQM assessment, the organisation is given a score between 0 and 1000 points. The percentages associated with each of the criteria in the diagram above are used as weights. According to EFQM theory it is assumed that different criteria have different levels of impact on overall excellence. The diagram presents the weights as they were defined in 1999. Although the EFQM model has been developed for assessing all kinds of organisations, universities and industrial research and development (R&D) departments sometimes argue that the research process is so fundamentally different from the manufacturing process that

11te konferanse for nettverk for organisasjonsforskning i Norge – NEON 2014 they question the validity of the TQM approach (Endres, 1997, p. 6). However, after struggling with how to interpret the language of the TQM assessment models in the context of R&D, Endres (1997, pp. 125-137) summarise industrial experience as positive. The TQM models help the R&D departments assess their current level of performance and structure long-term strategies for improvement. The EFQM model has also been extensively used among universities and institutions of higher education (Hides et al, 2004). The idea of using the EFQM model for improving the effectiveness of action research can be seen as an extension of how TQM assessment models have been used in industrial R&D units, assuming that action research and engineering research are sufficiently similar to allow this extension to work. With reference to how Argyris et al (1985) saw systems theory, control theory, decision theory and critical theory as special cases of action theory, action research and engineering research could be seen as tightly related. Hypothesis 2. Applying EFQM self-assessments on the action research process will increase the effectiveness of action research.

2.3 How to make action research efficient One way of characterising efficient research is to focus on features like low cycle times from initiation to publication of research, low cost in conducting research, getting good peer feedback in review and having the work widely cited after publication. While some would argue that ethics and moral awareness are important quality characteristics of action research (e.g. Boog et al, 2008), in the context of CAR it is more natural to see such aspects as part of the effectiveness of the approach rather than the efficiency. However, if the action research is carried out in a politically challenging environment, the duration of how long the action researcher is able to survive without selling out or getting fired could be used as an efficiency measurement. From a TQM perspective, increase in efficiency is achieved through the means of process improvement. Figure 3 gives a process perspective of CAR. The CAR process is cyclic and is made up of a sequence of five sub-processes for diagnosis, action planning, intervention, evaluation and reflection. The CAR process starts by entering the point of diagnosis, then cycles through the five steps as many times as needed, and exits after then final round of reflection has been completed.

Figure 3. CAR process model (Davison et al, 2004)

When considering how to integrate the EFQM model with the CAR process, there are several possibilities. One approach could be to apply the EFQM model at a meta-level by blackboxing the CAR process and using the RADAR logic of the EFQM model for investigating

11te konferanse for nettverk for organisasjonsforskning i Norge – NEON 2014 results, approaches, deployment, assessments and refinements in the way the CAR process is being implemented. Another approach would be to integrate the EFQM and CAR models at the same level by using the EFQM assessment method as the key tool at the evaluation stage of the CAR process. The approach used in this study consists of updating and using the results from EFQM assessments while going through the steps of the CAR process. What this means is that CAR and EFQM assessments are carried out in parallel through a sequence of five steps. The first step is the CAR diagnosis of NPM situation together with an EFQM assessment of the CAR situation. The second step is action planning for treating the NPM situation together with action planning for improving the CAR situation. The third step is to carry out the CAR intervention while recording the quality of the intervention through the use of the EFQM enabler criteria. The fourth step is to interpret the outcome of the CAR intervention while also assessing the EFQM results criteria. The fifth step is to consider the interpretation of CAR outcomes in terms of what to do next while also looking at the overall revised EFQM assessment score by comparing it with the reference EFQM score from the diagnosis stage. The chosen method of integrating the EFQM model with the CAR process has relevance for effectiveness in the sense that areas for improvement are “automatically” selected each time the CAR process goes through the action planning stage. The improvement process is driven by the EFQM results criteria and how the results criteria are weighted. This means that changes in focus are adjusted in a manner to increase efficiency in relation to the governing variables that are already in existence. In other words, the approach is expected to have an impact on effectiveness but the main drive is a concern for improving efficiency. In terms of defining governing variables for action research, much has been written about how to improve research efficiency without mentioning TQM. When Endres (1997) discusses the use of TQM for improving research and development (R&D) from a general perspective, he recommends the reduction of cycle time as goal for research units starting on the TQM journey. When translating this into the context of academic action research, cycle time could refer to the time between the initiation and publication of research. Another relevant goal could be to reduce the costs (man-hours) for writing and publishing scientific papers while making sure they get published in journals and conference proceedings of high merit. Some universities have classification systems where academic outlets are graded on scales like high level (2 points), normal level (1 point), and low level (0 points). Issues like citations, h-indexes and rankings of scholars are also relevant. The Google Scholar system is particularly useful because it automatically places results in a global context. Oakland (2003) gives a general introduction on how business measurements and processes in general can be analysed and improved through a TQM perspective. Whether the business process consists of selling sausages or publishing results from scientific studies does not matter in the context of TQM. In addition to methods for evaluating research, there has to be methods for enabling research. There is extensive literature on how to do action research and how to write and publish academic papers. Huff (1999) writes about scholarly writing for publication. Novak and Gowin (1984) write about research designs and formats of scholarly papers. Fisher and Ury (1981) have written about negotiations in a context that could be relevant for establishing and

11te konferanse for nettverk for organisasjonsforskning i Norge – NEON 2014 maintaining the research-client agreement in CAR. As the EFQM method of improving action research efficiency puts focus on a few parameters at a time, literature reviews have to be designed for figuring out how to improve the parameter in question. Hypothesis 3. Applying EFQM self-assessments on the action research process is an optimal strategy for improving action research efficiency.

3. Research methodology When trying to understand the impact of the EFQM assessment model on canonical action research (CAR) through real-world experiments, the mode of research is that of a CAR practitioner studying how to improve his own practice or what Donald Schön (1983) refers to as reflective practice. To make use of reflective practice as a research design, the approach taken in this study is the self-improvement action research methodology described by McNiff and Whitehead (2006).

3.1 Population and sampling procedure The target population in the study is the action research (AR) community in general and the canonical action research (CAR) community in particular. There are no assumptions about geography, age, gender and culture when defining the target population, but it is assumed that the action research and organisational change is carried out in a politically challenging organisation such as a public sector bureaucracy. The sampling procedure consists of the researcher using himself as a single-unit sample from the AR/CAR practitioner population.

3.2 Research instruments and measures Due to the nature of reflective practice research, the researcher himself is the main research instrument. It is through the researcher that data is being collected, analysed and interpreted. However, parts of the data will be collected through the use of the EFQM assessment tool. This will be done in two different ways. At the diagnosis stage at the beginning of the CAR cycle, existing CAR practice will be evaluated by assessing each of the 32 sub-criteria of the EFQM model on a scale from 0% (unable) to 100% (excellent). The scores reflect subjective interpretations with the average score of 50% is used to represent “I don’t know”. From the stage of action taking and onwards, the RADAR diagrams in the tables 1 and 2 below will be used for reassessing selected sub-criteria that were found particularly relevant during the diagnosis and action planning stages. Relevance & usability

Guidance

Scope & relevance

A coherent set of results, including key results, are identified that demonstrate the performance of the organisation in terms of its strategy, objectives and the needs and expectations of the relevant stakeholders. Results are timely, reliable and accurate. Results are properly segmented to provide meaningful insight.

Integrity Segmentation Performance Trends

Positive trends or sustained good

Unable to demonst rate

Limited ability to demonst rate

Able to demonst rate

Fully able to demonst rate

X

X X

X

Recogni sed as global role model


Targets

Comparisons

Confidence

performance over at least 3 years. Relevant targets are set and consistently achieved for the key results, in line with the strategic goals. Relevant external comparisons are made and are favourable for key results, in line with the strategic goals. There is confidence that the performance levels will be sustained into the future, based on established cause & effect relationships.

Scale Overall score

X

X

X

0%

25%

50% 43%

75%

100%

Table 1. Example of RADAR evaluation of a results sub-criterion

The use of the RADAR method for evaluating a results sub-criterion is illustrated in table 1 by setting an X in each of the seven parameters to be evaluated and calculating the overall score as the average of the percentage value given in the scale column. In case the subjective assessments are supported by objective data some of the seven parameters may be evaluated on a continuous scale. The RADAR evaluation of enabler criteria is exemplified in table 2. Approach

Guidance

Sound

The approaches have a clear rationale, based on the relevant stakeholder needs, and are process based. The approaches support strategy and are linked to other approaches as appropriate.

Integrated

Deployment Implemented Structured Assessment & refinement Measurement

Learning & creativity Improvement & innovation

Scale Overall score

Unable to demonst rate

Able to demonst rate

Fully able to demonst rate

Recogni sed as global role model

50%

75%

100%

X

X

The approaches are implemented in relevant areas, in a timely manner. The execution is structured and enables flexibility and organisational agility.

The effectiveness and efficiency of the approaches and their deployment are appropriately measured. Learning and creativity is used to generate opportunities for improvement or innovation. Outputs from measurement, learning and creativity are used to evaluate, prioritise and implement improvements and innovations.

Limited ability to demonst rate

X X

X

X

X

0% 7%

25%

Table 2. Example of RADAR evaluation of an enabler sub-criterion

The reliability and validity of the EFQM assessments are not expected to be high as part of the challenge of the research is to figure out how to use the EFQM model in the context of CAR.


3.3 Data collection Data will be collected while traversing one single cycle of CAR. This means that data will be collected by describing the process of diagnosis, action planning, taking action, evaluation and reflection while trying to aid the development of NPM in a public sector organisation. In addition to the qualitative and quantitative description of process and process outcome at the various stages of the cycle, numerical data will be produced through EFQM assessments. Some of these assessments will be based on objective measurements, but to a large extent they are dependent on subjective interpretations.

3.4 Data analysis The causal relationship common to all the three hypotheses is represented in figure 4 by a causal arrow in the upper part of the diagram and moderator arrows in the lower part. The causal arrow suggests that higher levels of maturity of the CAR strategy measured by the EFQM enabler criteria result in higher levels of CAR success measured by the EFQM results criteria. The moderator arrows suggest that that the environmental counterstrategies moderate the impact in the sense that a higher degree of EFQM enabler maturity is necessary to achieve a comparable level of EFQM results in environments that are more politically complex. CAR strategy assessed by EFQM enabler criteria

CAR outcome assessed by EFQM results criteria Environmental counterstrategies

Figure 4. Model of causal relationships in the self-improvement process

As the EFQM data are expected to have low reliability and validity, the mode of analysis will be that of observing the outcome of assessing the EFQM enablers and results before and after the intervention and then try to explain the meaning of the numbers rather than assuming that they can provide any conclusive evidence by themselves.

4. Analysis of results The analysis of results is structured by following the five steps of the action research process model and reflecting on the impact of the EFQM assessment model for each step.

4.1 Diagnosis In 2008 the Norwegian Directorate of Taxes (Skattedirektoratet, SKD) decided to reorganise the IT function by redesigning the traditional IT department and other service functions as a single unit (Skatteetatens IT- og servicepartner, SITS) that was still owned by SKD but was conceptually externalised and managed as though it were an external unit. The model in figure 5 shows how service level agreements (SLA) between SKD and SITS are used for daily supply-chain management while IT staff monitors and interacts with SITS from an information technology governance (ITG) perspective. The redesign at the Norwegian tax administration (NTAX) was based on strategic reasoning and not a response to IT failures. In fact, the use of IT at NTAX had been an ongoing success story and occasionally described as at role model for other Norwegian public sector organisations. However, there were also challenges along different dimensions, including the maintaining of a balance between user involvement from the professional bureaucracy and allowing the users to get “optimal solutions” without thinking about costs and complexity for

11te konferanse for nettverk for organisasjonsforskning i Norge – NEON 2014 the organisation as a whole (Wroldsen, 2008). Viewed from the perspective of the professional bureaucracy, or rather the machine bureaucracy of functionaries interacting between the professional bureaucracy and the IT department, there was discontent in how they had become totally dependent on decisions made by the IT department and thus had difficulty in seeing themselves as the real owners of the IT solutions. This was pointed out in an external study (Statskonsult, 2002). In the study it was further argued, based on the theory of five organisational structures (Mintzberg, 1983), that ITG run by a machine bureaucracy in the hands of the adhocracy of technological development may not be a good solution in the long run. CEO

IT staff

SKD

SLA

Audit

SITS

Figure 5. Organisational ITG strategy at the Norwegian tax administration

From the viewpoint of new public management (NPM), the model in figure 5 can be seen to incorporate all of the three fundamental NPM ideas of competition, contracts and control. Having the IT department and the other service departments conceptually externalised could be the first step towards making systematic use of the market. The SLA illustrates the use of contractual management, and by having SITS develop an ISO 9001-based quality management system to be audited by SKD (e.g. IT staff) illustrates the NPM idea of control. However, even though SITS has been trying to develop a quality management system based on ISO 9000 standards, and there are explicit agreements between SKD and SITS on how quality audits should be performed, there has been no progress in the development of an ISO 9001 audit process. Furthermore, the reason why there has been no progress is not because SKD has been perfectly happy with SITS performance. The latter part of a study on total quality management (TQM) implementation at NTAX dealing with the post-2008 period produced findings in support of the idea that quality management in certain processes, such as the COBOL software quality, was not only becoming more difficult to monitor from the SKD perspective but that the compliance with NTAX standards and procedures had gotten worse after the 2008 reorganisation (Øgland, 2013). The EFQM diagnosis in the table below represents the researcher’s subjective understanding of the situation above from the viewpoint of wanting to assist in the process of developing an ISO 9001 quality audit system to be run from IT staff as support in monitoring the SITS quality management system. This viewpoint means that EFQM result criteria like “customer results” refers to the relationship between the researcher and the client (head of IT staff), “people results” refers to how the research team evaluate their own performance and satisfaction, “society results” refers to remaining parts of NTAX, and “key performance results” refers to published research related to the development of an audit system. All the five enabler criteria refer to the management of the action research process from the research perspective. The table is sorted by having the criteria in greatest need of improvement come out on top.


EFQM Criterion 6. Customer results 4. Partnership and resources 9. Key performance results 5. Processes 7. People results 1. Leadership 2. Strategy and planning 3. People 8. Society results Total

Reverse weight (RW) 5,0 11,1 6,7 7,1 11,1 10,0 12,5 11,1 16,7

Score% 25 25 50 50 38 50 44 50 50 42

Priority = RW * Score% 1,25 2,78 3,35 3,55 4,22 5,00 5,50 5,55 8,35

Table 3. Total EFQM evaluation November 2013

The benefit of using the EFQM model in this manner is that it makes the action researcher a part of the system being diagnosed. Although the verbal diagnosis surrounding the diagram in figure 5 should be as objective as possible, the EFQM self-assessment stresses that the diagnosis is a model viewed from a certain perspective for the specific purpose of making interventions. The NTAX diagnosis can be thought of as trying to disclose a political game that prevents the organisation from designing and developing an ISO 9000 audit programme, but the nature of the CAR diagnosis is of a different kind. When using the NPM logic of the EFQM model in figure 2 to diagnose the CAR situation, the focus becomes managerial and economic in the sense of looking at issues like leadership and business results. Unlike the NTAX diagnosis, which might be viewed from a game theoretical perspective as it is concerned with the strategic interplay between several decision makers, when the action researcher makes use of EFQM self-assessments he is doing this for the purpose of controlling and improving his own process. Game theory may be one of several perspectives relevant in such a context, as there are customers and other stakeholders to deal with, but the overall problem is a control problem. By using the EFQM model to understand the situation, the result is having the problem framed as an engineering problem. The CAR process becomes something that can be studied from an industrial engineering perspective (e.g. Krick, 1962). As the purpose of the CAR diagnosis is to design control and improvement strategies, it is more important that the diagnosis captures the characteristics of the situation in a useful manner than being perfectly accurate. Although the data in the table above are based upon evaluating all the 32 sub-criteria of the EFQM model, the lack of practical organisational insights means that the diagnosis should be seen as an initial guess.

4.2 Action planning The purpose of action planning is to search alternative treatments for the diagnosis, compare such treatments and decide which one to implement. In the context of wanting to design a quality management audit system for IT staff to monitor SITS (figure 5), the first question becomes how to establish a researcher-client agreement (RCA) for getting started with the practical work and deciding how the practical work should be done. As was mentioned when commenting on how the EFQM model was used as a diagnosis tool, the table could be sorted in a manner that indicated the most important criteria to improve. According to the 80/20 rule (Koch, 1998), it should be sufficient to focus on the two weakest criteria (~20%) in the table as the majority of problems are often caused by a minority of causes. The summary diagnostics from table 3 consequently suggest that action planning

11te konferanse for nettverk for organisasjonsforskning i Norge – NEON 2014 should focus on customer results and partnerships/resources. In other words, the initial usefulness of the EFQM model for connecting the action research stages of diagnosis and action planning is that the focus of planning follows more or less automatically from the outcome of the diagnosis. When using the two highest ranking criteria in the EFQM diagnosis from table 3 as a basis for planning action in the context of redesigning enablers and getting better results, the enabler criterion dealing with partnerships can be investigated further by looking at the scores at the sub-criterion level, but in the case of the result criterion of customer results it is necessary to consult the EFQM theory explaining how the enabler and results criteria are causally connected. This illustrates the further usefulness of EFQM for controlling action research. According to the British Quality Foundation (2000), customer results are caused by phenomena addressed by ten sub-criteria distributed among all the five enabler criteria of the EFQM model. In table 4 the initial score for these sub-criteria have been listed and sorted according to scores and reverse weights in order to make it easier to start the improvement process by focusing on critical factors with poor performance. CAF/EFQM Criterion 4a Good supplier/partner relationships to satisfy customer 5b Improving processes to satisfy customers 5c Product and service development 5d Product and service delivery 5e Customer relationship management 1c Leaders’ involvement with customers 2a Establishing customers’ needs and expectations 2c Balancing customers’ needs and expectations 3b People have the skills and competence to deal with customers 3c People’s involvement with customers Total

Reverse weight (RW) 11,1

Score% 25

Priority = RW * Score% 2,78

7,1 7,1 7,1 7,1 10,0 12,5

50 50 50 50 50 44

3,55 3,55 3,55 3,55 5,00 5,50

12,5

44

5,50

11,1

50

5,55

11,1

50 46

5,55

Table 4. Ranking of EFQM enablers that should be addressed for improving customer results

Following the 80/20 principles outlined above, the first step of planning is to focus on developing good supplier/partner relationships (4a) and improving processes to satisfy customers (5b). In the context of a research-client agreement, the client (head of IT staff at NTAX) is the customer and academia is the supplier. Developing good supplier/partner relationships means to make sure that the client feels comfortable with the action research approach. In this case the head of IT staff at NTAX told the researcher (member of IT staff) that no research would be allowed unless external funding was provided for. A first step of action planning was consequently to discuss with academia how this problem could be solved. A more detailed account of the political challenges in developing a research-client agreement based on this story of writing research proposals and applying for funding is found in a study dealing with RCA-like problems from a game theoretical perspective (Øgland, 2014). A second step in the part of action planning that dealt with the academic side of the RCA was to identify academic journals and conferences that could be relevant for producing results and monitoring progress for the action research process. The conferences included in figure 6 are

11te konferanse for nettverk for organisasjonsforskning i Norge – NEON 2014 restricted to Nordic and European conferences to restrict the travelling budget from becoming an obstacle. ECIS (submit) NEON (conference)

Dec Nov

NOKOBIT (conference)

Jan

SCIS (submit) Feb

Oct

Mar IRIS (submit) Apr

NEON (submit)

Sep QMOD (submit) May

UKSS (conference)

Aug Jun

QMOD (conference)

UKSS (submit)

Jul

SCIS/IRIS (conference)

ECIS (conference)

NOKOBIT (submit) Figure 6. Relevant conferences for presenting research and meeting fellow action researchers

For most of the conferences in this diagram, papers are submitted during spring and the conferences themselves are held during autumn. The ideal situation would be to work on seven separate papers designed specifically for each of the seven conferences, submit each paper for each conference and go to the conferences where the papers have been accepted. As there may be overlap between conferences during autumn, it may not be possible to participate in all, so it may be necessary to select those conferences that are seen as most useful for the research. Table 5 gives a more detailed description of each of the most relevant conferences. When it comes to the sub-criterion 5b ranked as the second most important in table 4, the researcher asked for being allowed to start the development of an ISO 9001-based NTAX audit management system to be run by IT staff for the purpose of investigating the performance of the SITS quality management system. The answer, however, was that no research and development would be allowed until the issue concerning external funding had been settled. As a consequence of this, the only customer process of relevance is the communication between the member of IT staff and the head of IT staff concerning progress in getting external funding. The third part of action planning related to the RCA is consequently to develop a process of communicating progress in getting external funding on a regular basis. As time is spent reading scholarly literature, discussing with representatives of the scholarly community, writing research proposals, submitting application for research grants and preparing papers for journals and conferences in case the applications should be accepted, reporting to NTAX on a monthly basis through the use of emails seems like a reasonable approach. To summarise the contribution of the EFQM model at the stage of action planning, what the EFQM helps to clarify is that planning should be done for increasing the score of all the four EFQM result criteria. In addition to the technical planning of the quality audit system this includes planning for client satisfaction and academic success. The EFQM model provides links between enabler criteria and results criteria that gives instant suggestions on where to focus the planning in order to achieve results in the most critical domains.


Conference title

Submit

Conference

European Conference on Information Systems (ECIS) Scandinavian Conference on Information Systems Research (SCIS)

December

June

Rejection rate High

Comments

February

August

High

Quality Management and Organisations Development Conference (QMOD) Norsk Konferanse for Organisasjoners bruk av informationsteknologi (NOKOBIT) UK System Society International Conference (UKSS)

March

August

Medium

July

November

Medium

Possibility for meeting fellow ITG researchers and IS action researchers in Norway.

June

September

Low

Information Systems Research in Scandinavia (IRIS)

April

August

Low

Nettverk for organisasjonsforskning i Norge (NEON)

September

November

Low

Possibility for meeting fellow researchers doing action research with a focus on systems theory. Possibility for meeting fellow researchers doing IS action research on organisations in Scandinavia. Possibility for meeting fellow Norwegian scholars researching organisational development through the use of action research.

Possibility for meeting fellow IS and ITG researchers. Possibility for meeting fellow researchers doing IS action research on organisations in Scandinavia. Possibility for meeting international QMOD researchers.

Table 5. Overview of relevant conferences for presenting information systems action research

4.3 Execution of action strategy Although the action planning evolved over several months and is still in a process of evolving, the principle ideas described in the section above were used as a basis for formulating an action strategy made up of three basic steps. (1) Apply for research funding, (2) prepare papers for academic outlets while waiting for response, and (3) report progress to NTAX. As most of the planning was done in November 2013, the action strategy has been in constant use for eleven months. A chronological view of how the execution of the action strategy is given in table 6. 2013 Nov

2013 Dec

Member of IT staff 28.11. After having successfully defended his PhD thesis, the researcher reports back to NTAX and he asks for permission to continue his work by maintaining contacts with the academic network while doing practical TQM-related action research at NTAX.

Head of IT staff 29.11. Head of IT staff congratulates with completing the PhD and he agrees to meet at NTAX on December 5th.

5.12. The researcher meets with a group at NTAX concerned with tax research and the possibilities for building networks between NTAX and the scholarly community. As the head of IT staff was prevented from meeting, the researcher sends a mail summarising the meeting with the tax

06.12. Head of IT staff confirms the mail by saying that a post.doc project sounds interesting. 13.12. In a meeting at NTAX, the head of IT staff says the research proposal looks

11te konferanse for nettverk for organisasjonsforskning i Norge – NEON 2014 research group and says that they were, among other things, discussing the relevance of investigating the quality function at SITS and possibilities for doing post.doc research.

relevant and interesting but also says that it may be difficult to defend having a researcher working in his staff unless a part of the research is externally funded.

12.12. The researcher has discussed with various people at UiO and outlined a post.doc research proposal as input for discussion at NTAX the next day. 20.12. The researcher has discussed the situation with people at UiO. Their response is that a contractual collaboration between UiO and NTAX would be nice, but they have no budget at the moment. However, the matter will be discussed further and possible solutions will be outlined. 2014 Jan

26.01. There have been discussions at UiO. While a formal collaboration with NTAX is considered useful, the lack of budgets makes it impossible to contribute to the funding of the research. It is assumed that the situation will be better in 2015, but for 2014 it is recommended to apply for external funding outside of UiO. UiO will make sure that the researcher maintains access to UiO infrastructure. The researcher asks whether it would be possible for him to get computer access at NTAX and start doing the action research in an informal manner while the formalities are gradually being cleared out.

31.01. Head of IT staff acknowledges the problem in getting funding from UiO and suggests either taking contact with other universities or abandoning the project by becoming a regular office clerk at NTAX.

2014 Feb

10.02. After taking up the issue with UiO, the response from the university is that there should be no need for burning bridges. If the action research design for developing an audit system could be defined as a regular NTAX development project for 2014, it could be turned into an action research project in 2015 when the financial situation at UiO is expected to have improved.

No response.

2014 Mar

07.03. The researcher repeats his requests for getting an office at NTAX with computer access in order to start the audit system development, regardless of whether it will end up as a development project or action research, but also explains that he is spending time reading and trying to figure out how to collaborate with various people at UiO who might become important collaborators once the contract gets settled.

No response.

2014 Apr

03.04. The researcher informs that UiO has identified a relevant fund, and he has consequently spent his time since the 10th of March writing an application. A draft version of the application is appended to the mail. The deadline for submitting is on the 9th of April and a recommendation from NTAX is needed as an appendix to the application.

04.04. The head of IT staff says that there is too much going on at NTAX at the moment, but he would like to look through the application. He says that it should be sent regardless of him being able to read it or not.

08.04. The researcher reminds NTAX that the application is due for tomorrow and that he has not

08.04. The head of IT staff writes he is tied up in meetings, so the application should be sent without the recommendation.

11te konferanse for nettverk for organisasjonsforskning i Norge – NEON 2014 received the NTAX recommendation paper yet. 2014 May

07.05. The researcher informs that he is spending the time writing a paper for the UKSS conference to be submitted by the end of the month while waiting for the response from the application.

No response.

2014 Jun

06.06. The researcher informs that the UKSS conference was cancelled, so the paper was submitted to a journal instead (IJSS). He is now writing on a paper for the 3rd international innovation in information infrastructure workshop (IIIOS) to be submitted by the end of the month.

No response.

2014 Jul

04.07. The researcher informs that the two papers have been submitted. He has also submitted another paper to the NOKOBIT conference and is working on a journal paper (EJOLTS). However, he has also been informed that the application was rejected. He requests permission to rewrite and resubmit the application based on the feedback they got. He also reminds the head of IT staff that it would be useful if he could get a furnished office and be allowed to start the audit process when the internal audit group plan their audit of the quality function at SITS during the autumn.

No response.

2014 Aug

20.08. The researcher informs about a submitted paper being accepted for presentation at the IIIOS conference and that a paper for the NEON conference is in a final stage of development. He also informs about discussions with UiO on how to redesign and submit a new research grant application.

No response.

2014 Sep

15.09. The researcher informs about the NOKOBIT paper being rejected, the NEON abstract being accepted, and the current status in writing the RFFH application.

No response.

2014 Oct

22.09. The researcher sends a draft version of the RFFH project proposal for comments and asks for a confirmation document from NTAX. 01.10. As there is no response from NTAX the researcher sends a reminder. 08.10. As there is still no response from NTAX the researcher sends another reminder.

10.10. The head of IT staff emails a document that gives formal support of the project and some input on issues that have to be looked into to make sure research interests fits with NTAX interests.

10.10. As UiO starts to worry about written NTAX support to include in the project proposal, the researcher phones NTAX and is being informed that a confirmation document in support of the project is being written. 13.10. The final version of the grant application is discussed with UiO and submitted to RFFH. Table 6. Execution of the action strategy

11te konferanse for nettverk for organisasjonsforskning i Norge – NEON 2014 The execution of the action strategy matches with the three steps that make up the action strategy. The first step of the action strategy was to try to get external funding for the research. In the table it is seen how this theme is brought up in December 2013 and how it remains a central issue in all of the monthly status reports. The second step was to prepare for the actual research by reading, experimenting, discussing and writing papers for academic outlets. Although time was spent reading, writing and discussing from the beginning of the project in November 2013, it was only by the end of May 2014 that the papers had matured sufficiently to be ready to be submitted. The third step of the action strategy was to make sure that the client (head of IT staff) was getting progress reports. The table shows that progress reports were written at least once a month. The table also illustrates that it was difficult arranging face-to-face meetings or getting response in general. The table below gives an account of the scientific papers that have been worked on as part of the action research effort. For each paper the title of the paper, the time of initiation, and the motivation for writing the paper have been recorded. As the process continues, control statistics in terms of size of paper (number of words), cost of writing (minutes of editing) and cycle time by counting the weeks from initiation to termination are updated. Scientific paper

Start

Motivation

Can a critical systems approach compensate for lack of management commitment when implementing TQMbased organisational learning? Improving the effectiveness of IT governance in public sector organisations by bootstrapping quality control for supply-chain management The principle of Researcher-Client Agreement in Canonical Action Research

03.11.2013

Summary of PhD thesis using different data and analysis methods.

30.11.2013

#4

#5

#1

#2

#3

Size (words) 13531

Cost (minutes) 7853

Time (weeks) 33

Status

Research proposal

5945

1580

3

13.12.2014 Conditionally accepted by SKD: Partial funding from UiO required

14.12.2013

Getting the RCA in order

7967

3161

39

The effect of EFQM self-assessments on action research performance: New Public Management in a public sector financial organisation

15.12.2013

Monitor the CAR programme

22109

9811

42

IKT i offentlig forvaltning og New Public Management: Pacman-strategi for forbedring av kvalitet og sikkerhet i

19.04.2013

Research proposal (as part of grant application)

2832

2223

9

25.08.2014 Not accepted NOKOBIT conference (rejection rate = 38%) 10.11.2014 Accepted for NEON conference (session on governance, control, and learning) 23.06.2014 Rejected by RFFH

28.07.2014 IJSS journal confirms having received the paper for review


#6

#7

#8

#9

#10

#11

#12

#13

Skatteetatens IKTstyring Could constructive empiricism be more useful than critical realism for conducting action research on information infrastructure development? Canonical Action Research and Information Systems Development Learning how to implement Total Quality Management through the use of Pac-Man video game simulations Empirical constructivism makes action research similar to design science research Title not yet decided.

Mechanism Design for Total Quality Management: Using the Bootstrap Algorithm for Changing the Control Game The principle of Researcher-Client Agreement in Canonical Action Research New Public Management and ICT in public administration Using the bootstrap algorithm for cultivating IT governance Average

02.05.2014

Getting involved in philosophical debates concerning AR methodology

2589

2102

16

04.08.2014 Accepted for IIIOS conference. (Rejection rate = 11%).

11.05.2014

Drafting a book for summarising the prospective post.doc research. Getting involved with theory of organisational change and the AR practitioner community Getting involved with action research vs. design science debate. Writing a book to articulate political aspects of the research. Secondary PhD supervisor recommends having the thesis published as a book.

28686

2973

28

17947

6275

11

09.11.2014 In progress (publisher undecided) 07.10.2014 Preparing for EJOLTS journal

1295

59

16

01.09.2014 Preparing for ECIS conference

450565

9509

9

104977

1275

9

13.10.204 Preparing for publication (Lulu Press) 18.10.2014 Published (Lulu Press)

21.05.2014

03.07.2014

17.08.2014

21.08.2014

25.08.2014

Getting the RCA in order

7927

3205

5

02.09.2014

Research proposal (as part of grant application)

4570

1967

7

51611

3999

17

27.09.2014 Accepted for NEON conference (open track) 13.10.2014 Submitted to RFFH

Table 7. Production processes during period of action research (November 2013 – November 2014)

The development of products in the table above lists three research proposals, seven research papers and three books. As seen from the status column on the far right, the processes for the research proposals, one of the books and three of the research papers have been completed. For the remaining two research papers and two books the final outcome is not yet clear.

11te konferanse for nettverk for organisasjonsforskning i Norge – NEON 2014 The five enabler criteria of the EFQM model can be used for assessing the execution of the action strategy from a TQM point of view. Table 8 shows the results from evaluating the leadership process as executed by the action researcher with respect to his one-unit cell as an individual organisation. Criterion 1: Leadership 1a Leaders develop the mission, vision and values and are role models of a culture of excellence

Score 43

1b Leaders are personally involved in ensuring that the organisation’s management system is developed, implemented and continuously improved 1c Leaders are involved with customers, partners and representatives of society

50

1d Leaders motivate, support and recognise the organisation’s people Average score

50

18

Comments The mission, vision and values of action research at NTAX is to test and develop theories of ISO 9001 auditing as explained in the research proposal. Overall RFFH external evaluation of research proposal was 3/7 = 43%. This paper illustrates the way the action research is being developed, implemented and continuously improved. The story in table 6 explains how the researcher (leader) is involved with customers (NTAX management), partners (UiO), and representatives of society (colleagues at NTAX). Self-motivation.

40

Table 8. Evaluation of leadership

The evaluation of leadership illustrates the difficulties in subjective evaluations when there are few objective standards to measure against. The comparatively low score for criterion 1c is due to difficulties with understanding the needs and expectations of the main customer (head of IT staff). Leadership is difficult to evaluate and it maintains generally difficult to say whether the current status is dominantly good (>50%) or bad (