Applying Performance Management Principles to a Learning ...

Applying Performance Management Principles to a Learning Environment for Undergraduate Dissertations: a Case Study Applying Performance Management Principles to a Learning Environment for Undergraduate Dissertations: a Case Study Dr. Jacqueline Day and Dr. Milena Bobeva The Business School The Institute of Business and Law Bournemouth University Poole BH12 5BB

ABSTRACT In the UK an individual research work, often known as a ‘dissertation’, is a common requirement of undergraduate and taught postgraduate degrees in business, law and humanities subjects. However, changes in the business environment have placed heavy demands of dissertation students, their supervisors and academic departments. Amongst these challenges are increasing cohort sizes, individual supervision mode, coordination of many stakeholders and increasingly stringent quality regulations to be met within tight financial and resource budgets.

This research presents a response to these problems, by demonstrating how one Higher Education institution developed a Learning Environment (LE) specifically designed to manage dissertations. Implementation of this LE has lead, over several years, to apparent improvements to the student dissertation experience and achievement and also seemed to have enhanced the consistency and effectiveness of research supervisors. However, to justify further investment of time and money and to further develop operational management it became necessary to properly assess the performance and impact of the LE. The case study discussed in the paper explains how the performance management regime was designed and then examines how it will be used in the longer term to nurture a community of practice for all involved in dissertations. Keywords: student, dissertation, learning environment, performance management

Taken from The International Journal for Quality and Standards www.bsieducation.org/ijqs

Page 1 of 23

Dr. Jacqueline Day & Dr. Milena Bobeva

Applying Performance Management Principles to a Learning Environment for Undergraduate Dissertations: a Case Study

1. Introduction In commerce and industry business performance management (BPM) is a well accepted discipline that enables strategy to be executed in ways to meet organisational objectives (Becher, 2005; Clayton, 2005). However, the implementation of BPM within the Not-for-Profit organisations such as local government, health services or education is not so well understood (Rees and Gardner,2003). With respect to the latter, case studies and exemplars are rare and the limited research that has been carried out in the area of performance management with in Higher Education Institutions (HEIs) has been directed to executive or policy-making strata (Taylor, 2001) rather than at ‘nuts and bolts’ level of the organisation. The conventional view is that evaluation enables targets to be set, against which the work of individuals or groups will be judged and so will positively influence the behaviour of individuals or functions to meet these targets (Athiyaman, 1997). At the institutional level, performance indicators for Higher Education (HE), particularly those that involve a final aggregation in the form of league tables, have received a mixed response. However, the particular focus of the project was upon the internal settings, which by their nature are rarely published. In the absence of theoretical guidance, the pragmatic view endorsed by Taylor (2001, p.392) was used to guide design of performance management by judging the salience of measures for the problem domain by: ‘…the extent to which they improve the quality of the decisions made by the people who are exposed to them, whether this is the government, university management or the individual academic.’ Accordingly, the purpose of the present work is to address this lacuna in current knowledge by providing an ideographic study that reports experiences with applying BPM principles to the development and utilisation of an integrated learning environment (LE) to support undergraduate dissertations within UK Business School. These ideas have been tested in practice as they present one of the major challenges to the management of academic studies. The paper begins with a discussion of the nature of dissertations and the problems associated with these. Further it presents how these problems have been addressed in the form of a LE and introduced the phases of its development, grounded in the philosophy of BPM.


Page 2 of 23



1.1 What are Dissertations? An individual piece of work demonstrating understanding of the research process and critical understanding of a substantive knowledge domain is usually called a ‘dissertation’. This type of studies often commands a prominent position as the ‘flagship’ unit of most business, information systems, social science, humanities or law degrees and their completion is deemed by many institutions to be the element demonstrating the ‘honours worthiness’ of the award. In the realm of student-led research, academic prestige, funding and institutional pride are traditionally grounded in the demands of doctoral level studies (and for that matter taught Master’s dissertations) but in terms of the number, anecdotal evidence suggests that these are dwarfed by undergraduate studies. Despite this, undergraduate level research activities have attracted comparatively little academic attention. This neglect is surprising, considering the commitment bound up in this exercise and its implication for the development of those generic skills prized by industry and commerce. Furthermore, the dissertation being (usually) the most rewarding part of an undergraduate degree course (Ramsden, 1992) student experience of the research can be a factor in encouraging the individual to subsequently enrol for a higher degree. 1.2 Challenges for the Student What comes over particularly clearly from the existing studies is that it is at the undergraduate level where the change to the learning context for the student is perhaps at its most acute. Since it is here the budding researcher must adopt a more intense level of engagement with their chosen specialist subject area (Clauston and Whitcombe, 2005; Rowley, 2004). Indeed, the student finds her/himself having to deal with a set of profound and often uncomfortable transitions (Table I):

From To Interaction within the institution-----------------------------------------Æ External communication General knowledge--------------------------------------------------------Æ Knowledge in-depth Knowledge consumer-----------------------------------------------------Æ Knowledge provider Single truth ------------------------------------------------------------------Æ Multiple/provisional truth Structured time allocation-------------------------------------------------Æ Self management of time Delivery to a student group-----------------------------------------------Æ One-to-one with supervisor Directed----------------------------------------------------------------------- Æ Autonomous learning Pupil----------------------------------------------------------------------------Æ Academic collaborator Table I: The Impact of the Dissertation Upon Undergraduate Student In these matters as Todd et al (2006 p.171) rightly observe:


Page 3 of 23


Applying Performance Management Principles to a Learning Environment for Undergraduate Dissertations: a Case Study ‘while for the motivated and able student the dissertation can be an intellectually stimulating and rewarding experience which is a marker of student ability, for those who are weaker or less motivated, it can be an unpleasant and unsettling for both student and supervisor.’ Inability to cope with these changes, combined with natural doubts and uncertainties associated with any form of research are likely to have a negative impact upon student performance such that at its most extreme can cause failure of the degree (Gordan, 2003). 1.3

Challenges for Educational Institutions.

The limited studies of sub-doctoral research carried out to date have been directed toward specific pedagogical issues and problems. These include topics, such as improving the students knowledge and application of research methods (Booth and Harrington, 2003; Benson and Blackman, 2003; Edwards and Thatcher, 2004), analysis of student-supervisor relationship (Mackinnison, 2004) and assessment regimes (Hand and Clewes, 2000; Milton and Lyons, 2003; Webster et al, 2000). What has not really been looked at is a LE for dissertations that enables students to deliver improved quality outcomes reflecting higher levels of expectations about their learning experience and degree grades. Increased exposure to the business world and the introduction of fees in UK HE have fostered an attitude whereby students increasingly consider themselves as ‘clients’ of an educational service (Armstrong, 2003; Browne et al, 1998) and demand higher levels of value-for-money from the ‘provider’ (the academic team). This generates tension with the intention of the academic research exercise, as the novice researcher must be supported and encouraged to learn to deal independently with the problems associated with interpreting the literature, setting research objectives through development of problem-solving skills and a style of knowledge acquisition congruent with the precepts of self-directed learning (Mackinnon, 2004). Academics and administrative staff are also not immune from those forces that require the overseeing of more students by fewer supervisors. Furthermore, taught degree dissertations now come in a greater variety of forms including the more traditional empirical investigations, action research, consultancy projects or marketing investigations. These are all legitimate interpretations of what constitutes undergraduate research and which impact upon the supervisor–student relationship that lies at the heart of a successful dissertation. Therefore, it is clear unless an improved level of support is given, what is demanded may well prove impossible without damage to the supervisor’s own scholarly activity and research agendas (Doring, 2002). In this setting, the LE was made the locus of the effort to improve the performance of the dissertation management. It combines new organisational processes, information technology Taken from The International Journal for Quality and Standards www.bsieducation.org/ijqs

Page 4 of 23


Applying Performance Management Principles to a Learning Environment for Undergraduate Dissertations: a Case Study and pedagogical practices designed to deliver benefit to students, academic staff, administrators and the managers of a UK University Business School. The overarching vision for the LE was to develop something that supports the strategic objectives of the institution, including the need to improve the student learning experience and to better realise the research potential of the academic staff. In this regard the development of the learning environment addresses a paradoxical position of the university: it had few systems at operational level supportive of its prime mission to research and educate. 2 The General Design of the LE 2.1 The Existing Situation A range of sources were used to gather knowledge about the then current problems and where the future stress points were likely to occur (Appendix A). Anecdotal evidence was obtained from formal and informal feedback from supervisors and students. Documentary evidence included statistical analysis of marks and the results from investigations of formal student complaints and hearings dealing with academic offences. Comments from external examiners were also scrutinised and discussion groups employed to canvass the views of course managers and administrators. The findings from the theoretical review and the empirical evidence were integrated into a set of aspirations (soft objectives) for the future LE (Table II):


Page 5 of 23



Student Aspirations

Institutional Aspirations

Sustain a high level of motivation for students

Interaction between students and academic staff within a ‘community of scholars’

Good previous experience of research, such as dissertations, assignments and case studies.

The existence of a visible and accepted organisational strategy for undergraduate units to build up a knowledge base of investigations done by students.

Willingness to seek advice and guidance

Availability of a group of lecturers as a shared resource

A supportive and nurturing relationship with the supervisor

Good fit between the supervisor’s personal interpretation of the dissertations and the organisation.

A degree of self-awareness of learning styles

Provision of lecturer-led seminars using a panel of domain experts to guide the development of the conceptual framework.

Self-organisation and methodical approach to matters such as documentation.

The degree of flexibility in the organisational understanding of what constitutes a dissertation

A positive attitude toward critical inquiry in relation to the topic

Knowledge of the supervision arrangements and process

A willingness to engage in informal peer assessment

The active use of code of practice for defining expectations and responsibilities of the parties.

Ability to self-direct their learning based on feedback and guidelines

More accurate and transparent feedback and assessment

Awareness of the purposes and limitations of different communication channels

Ability to use consistently different communication channels

A realistic understanding of timescheduling and workloads

An accessible and responsive supervisor team, able to manage effectively students expectations

Table II: General Requirements for Dissertations LE

2.2

The Design of the LE

The requirements suggested that the design of the LE follows a ‘tight-loose’ form alluded to by Todd et al (2004). Prudent use of information technologies (Alavai et al, 1997; Clerehan et al, 2003) address the requirements of the parties delivered through improved work flow, the monitoring of support for dissertations, enhancement assessment regimes, better communication, as well as providing easy access and visualisation of relevant documents, such as official teaching and learning policy statements and guidelines. Ideally, the new LE would be able to rapidly adapt to the changing agendas of the university. However, it was recognised this need would be difficult to implement, not least because high level institutional strategies are often vague and frequently contradictory (Rowley, 2000). There is a whole set of constraints upon the design of any programme level management environment that originates in the general environment that impacts on the efficacy of learning environments (Aldridge and Rowley, 1998; Oldfield and Baron, 2000). For example, the usefulness of a Taken from The International Journal for Quality and Standards Page 6 of 23 www.bsieducation.org/ijqs


Applying Performance Management Principles to a Learning Environment for Undergraduate Dissertations: a Case Study book shop or the adequacy of university library stock, are outside the remit of either students or supervisors, at least in the short term. There was little that could be done about these factors but to try to take them into account when designing the performance component of the LE (see below). An eventual reconciliation of the requirements informed the overall design of the LE (Figure 2), which was conceived as being constitute from several interlinked elements: (1) Student context – student knowledge, motivation, understanding and skills (2) Learning context– the mode, content, scope of a dissertation and its learning objectives (3) Supervisory context – the knowledge, motivation understanding and skills of the tutor (4) Management context – the protocols, procedures and methods for dissertations To emphasise on the synergistic existence of these components, a fifth element was developed for reflecting the need for integration and acting as an essential nexus of information and communication. The sixth and final element recognises that performance management should not be linked to discrete elements, but be able to monitor and report upon activities across the total set of elements.

1. Student Context

2. Learning Context Indirect intervention

University controlled

Individually controlled 5. Integration Services

6. Performance Management

Direct intervention 3. Supervisory Context

4. Management Context

Figure 2: General Design of the Learning Environment

The figure above shows the elements of the design and distinguishes two taxonomies: one, based on the locus of control some components must fit into a schema laid down at an institutional level (2, 4, 5, 6) or at a programme management level (1,3,5); the other reflecting the type of intervention, i.e. direct, involving elements 3, 4, 5 and 6, and indirect, with elements 1 and 2. Taken from The International Journal for Quality and Standards www.bsieducation.org/ijqs

Page 7 of 23


Applying Performance Management Principles to a Learning Environment for Undergraduate Dissertations: a Case Study Much of what was needed for the LE existed in an operational form but the different elements were not integrated. In terms of management systems this problem was relatively easy to address, however, the element missing from this (and for matter other educational programmes) were the ways and means of monitoring the performance of the LE. Adopting the mantra, ‘what gets measured gets managed’, it became clear that the Performance component (6, Figure 2) was critical to success of an improved environment because these indicators and measures act as a bridge between the organisational conceptualisation of the LE and the IT and other types of system that practically implement the strategic intention 3

Developing the Performance Management Framework

In the main the plan for development of the LE performance management followed the steps suggested by Niven (2003). A few variations were necessary to reflect the particular needs of education and these are noted in course of the discussion below. Step 1 - Select the Performance Framework The need was for a performance assessment that would help to accommodate the different facets (contexts) of the LE and be transparent to all interested parties. Three perspectives were considered: the TQM philosophy (Widrick et al, 2002; Koch, 2003); the best-practice adoption ideas of the European Foundation for Quality Management (Calvo-mora et al, 2005) or the strategy-focused Balanced Score Card (BSC), (Kaplan and Norton, 2001). Of these general approaches, the BSC was chosen as the best framework because, through the use of strategy maps (Cardoso et al, 2005) it seemed very effective in translating the mission of the LE into a set of operational goals and actions (Dorweiler and Yakhou, 2005). Despite its conceptual limitations (Nørreklit, 2000) the BSC was considered to be the most easily adaptable to the specific organisational context of higher education. It allowed to be implemented in a mode allowing the cascade of performance management targets from higher organisational level to the lower level, that is, from the HEI, academic school and course team (Ruben, 1999). Finally, BSC reports could be partially automated enabling key performance indicators (KPIs) to be monitored as ‘dashboard’ controls thus providing a snapshot of progress at any point in time.


Page 8 of 23


Applying Performance Management Principles to a Learning Environment for Undergraduate Dissertations: a Case Study Step 2 - Customise the Performance Framework The starting point in any application of analytical model and frameworks is the grounding of the tool into the specific organisational setting. The interest of key stakeholders in the LE formed the basis for the scorecard (Atkinson et al, 1997) arrived at is discussed in the next section. Figure 3, shows the original form of the BSC adapted to reflect the particular contexts for the design and implementation of the dissertation LE. Financial Perspective (B) To succeed financially how should we appear to senior management 4. Management context

Customer Perspective (A) To achieve our vision, how should we appear to our students?

Balanced Scorecard – Vision and Strategy

1. Student context

2. Learning context

Processes Perspective(C) To satisfy our stakeholders what administrative processes must we excel at? 4. Management context

5. Integration

Learning & Growth Perspective (D) To achieve our vision, how will we sustain our ability to change and improve our academic quality?

3. Supervisory context

Figure 3: The BSC for the LE: adapted from Kaplan and Norton (1996)

The changes to the original form of the BSC are indicated in italics. The four generic BSC perspectives were modified in ways explained below. A) Customer Perspective -> Student Perspective This perspective is ‘first among equals’, in the sense that it is usually the starting point for the development of all the performance measurements. Generically, it specifies value of the product or service from the point of view of individuals, groups or organisations that consume and pay for the product or a service: the ‘customers.’ In the present case, the customers are the students and, for the LE in question, these are undergraduate dissertation students. At a strategic level within any commercial operation it is easy to see that if you don’t meet the needs of the customer they will seek other providers. Although this scenario is not applicable to a HE institutions with regard to the impact of financial perspective, (B), an analogous value proposition can be formulated in terms of the benefit derived from the knowledge capital Taken from The International Journal for Quality and Standards www.bsieducation.org/ijqs

Page 9 of 23


Applying Performance Management Principles to a Learning Environment for Undergraduate Dissertations: a Case Study transferred to the learner (Liebowitz and Suen, 2000). There are direct linkages between this perspective and the financial (B) and developmental (D) views. The second link is to the Learning and Growth, which concerns the inventiveness, skills, knowledge and experience directed to create and sustain this value. (B)

Financial Perspective -> Institutional Perspective

Historically, this view has been the main influence upon corporate planning for achieving the long term goals. Financial success is relatively easy to establish in terms of Return On Investment (ROI), cash net profitability etc. for a business but by definition will be of limited relevance to a not-for-profit operation (Baruch and Ramalho, 2006). The generic BSC suggested that this perspective is all about performance judged from the point of view of owners. The LE is owned by the University and the senior management team represent this proprietary interest. The stakeholders particularly interested in financial performance of the presented dissertation LE were Business School managers and, ultimately, the senior executives team of the university. This perspective is partially located in the Management context (4) component of the LE. (C)

Processes Perspective->Administrative Perspective

The importance of processes is that it is through these organisational systems and resources that benefits are provided to customers. These operational activities are concerned with operational knowledge provision in the form of service to students, academics (Pariseau and McDaniel, 1997) and the quality assurance of the products and processes affecting both internal and external customers. The latter responsibility includes management of the traditional and online modes of communication. The BSC identifies the quality of internal management processes and the deployment of resources to deliver what is expected by the customers. In the case of the LE, this is the job of unit leaders, coordinators and support staff who are responsible for the day-to-day conduct of the programme. This directly corresponds to the Management context (4) component of the LE. The performance framework shows that this is affected by the intentions and consequences of financial performance (B) but also by the emphasis placed upon the progress, advancement of understanding and innovation (D), the fourth and final view of the BSC. (D) Learning and Growth Perspective -> Faculty Perspective This perspective is important for the supervisors, since it acknowledges the effect of learning and creativity and thus acts as a window into organisational ‘know how’ enabling the creation of customer (student) value. This perspective acts as a window into the way the knowledge is Taken from The International Journal for Quality and Standards www.bsieducation.org/ijqs

Page 10 of 23


Applying Performance Management Principles to a Learning Environment for Undergraduate Dissertations: a Case Study created, refined and exploited to yield something valued by the customer. This is not just about the current situation but also include how technological, organisational capital can be mobilised to ensure the organisation can sustain and improve its knowledge competencies and capabilities in the face of environmental change. This, of course, is of primary importance for a university because it uses knowledge to produce its ‘product’ which is knowledge (Rowley, 2000). In the setting of the LE the stakeholders interested in, and indeed responsible for the maintenance of the knowledge base, are the Business School academics (the faculty), in particular those involved in the supervision the research undertaken by the students. Step 3 – Identify Stakeholders Ruben (1999 p.5) argues for evaluation scheme based upon the use of multiple measures identified from discussions with groups of stakeholder that are able to: “Contribute pertinent and useful insights and collectively these judgements yield a comprehensive and balance cluster of measures that help to address concerns associated with a reliance on any single perspective or measure.” For the present work this need was met by adapting the ideas of Olve et al (2004) concerning four generic BSC roles: scorecard owner, change agent, information agent and measurement (KPI) owner. Table III shows that there is a senior role within the BSC that equates to ownership of the BSC and also represents the interest of external stakeholders, such as the University’s Senior Management and the Senate, through a surrogate of the Head of Quality for the Business School. The role of change agent recognises the part that should be played by academics in the active feedback of performance into learning, knowledge growth and subject development. Information agent is the term used to label those who are responsible for the information underpinning each KPI.


Page 11 of 23



LE Stakeholders Deputy Dean of the Business School Dissertation Supervisors Dissertation Unit Leader Dissertation & Placement Officer Programme Leader Programme Administrator Placement Advisors Post-Dissertation Students Pre-Dissertation Students Head of Quality School Finance and Resources Officer Head of Undergraduate Programmes Head of Academic Group

BSC View Contributor All Faculty Faculty Administrative Customer / Admin Administrative Administrative Customer Customer Institutional Administrative Customer Faculty

Primary Evaluation Role BSC Owner Change Agent Information Agent Information Agent Information Agent Information Agent Information Agent Information Agent Information Agent KPI Set Owner KPI Set Owner KPI Set Owner KPI Set Owner

Id a b c e d f g h i j k l m

Table III: Stakeholders and their Roles in LE Performance Step 4 – Identify Critical Success Factors After the stakeholders had been identified they were invited to attend a workshop to identify the objectives for the areas that they represented. Each attendee was invited to suggested three – to five goals and then to prioritise these. When this had been accomplished the goals were represented as the first dimension of the Critical Success Factors (CSF) matrix. The second aspect, again, completed through the use of a workshop session, was to agree the main factors (activities and information and other resources) that most influenced the achievement or otherwise of the goals for the LE. The final task was to take the completed matrix and use it to understand the linkages between all the drivers for the new LE. It was found that ‘fishbone’ diagrams (Ishikawa 2007) were an intuitively simple way of articulating the initial cause – effect relationships, which then could elaborated in the form of a strategy map (Cardoso et al, 2005). Figure 2 illustrates the way academic capabilities and administrative processes help to deliver the learning experience for undergraduate researchers and then how satisfaction with the learning outcomes contributes to a perception of the HEI. The numbers shown for each objective correspond to the objective identifiers shown in Table IV. These views influence the subsequent academic activity directed toward growth and learning.


Page 12 of 23



Figure 2: LE strategy map of performance objectives and evaluative contexts

Degree results are improved 10

Supervisors Productivity is raised 11

Unit is more economical to run 09

Enhanced image of the Business School

Good level of

Consistent delivery and assessment 14

Dissertation ILO are met more effectively 13

Self-support for the novice researcher 15

Standardise d Dissertation procedures 03

Supervisors managed better

dissertation supervision 16

Improved Skills level 01

12

Institutional: ‘Move up the league tables!’

Student: ‘It is the mark that matters!’

04

Control of VLE expanded 02 Administrative: ‘We must follow the rules!’

Research skills Improved 05

Involvement with the community of practice

06

Higher Pedagogical competency

07

Subject specialism enhanced 08

Faculty: ‘Academics must be free to question! ’

Step 5 – Develop Performance Measures The identification of the key performance and key quality indicators was the most complex part of developing the evaluation framework for the LE. Two general considerations underpinned the design. Firstly, all indicators should be linked together as part of a cause and effect chain that represents the performance of the LE. The second principle is additional to the standard process and reflects the idea mentioned earlier about the importance of context, by adopting the suggestion made by Buytendijk and Flint (2002) that performance management should focus on the boundaries because these organisational interfaces is where the main failure points lie. It will be seen that intersections between the perspectives form dyads that map to the contexts within which performance evaluation will take place. The beliefs shared between relevant stakeholders form the contexts and are equally important to the design of the performance management system as the perspectives themselves, because the context affects the interpretation of results. That is, how the measurement as information, will become knowledge that can be used for decision-making. Taken from The International Journal for Quality and Standards www.bsieducation.org/ijqs

Page 13 of 23



This philosophy was operationalised by articulating the values of the KPI owners and associated information providers/users as a normative statement. The idea was that for the objectives (and KPIs) to be meaningful, in the sense of being specific, acceptable, relevant, the measures must be congruent with the behavioural norms of these groups, because these would guide the attention paid to the results of the assessment and consequent actions taken. In the setting of the LE, the KPI should flag up the extent to which collaborative or cooperative activities as a form of collective intelligence generated by the flow of knowledge across these interfaces (Busi and Bititci, 2006). For each of the KPI sets (Table 3) specific assumptions were made about the type of measurement that should be used and these are discussed below: •

Administrative The theme for this view was evaluating the level of service provided by the LE to the customer. The criteria were derived from the work of Abduallah (2006) on service performance. The key indicators were those that were easy to understand by the nonacademic staff supporting the delivery of the dissertation unit.

•

Faculty Of the four perspectives, this one is best matched with the generic BSC view of organisational learning, personal growth and innovation. However, by their nature these processes and outcomes are quite difficult to encapsulate in a measurement not least because some of the knowledge is personal. The stance taken was that the efficacy of the LE for the supervisors is the extent to which it enables the absorption, diffusion, generation and the exploitation of academic knowledge (Sprenger and ten Have, 1996).

•

Institutional These measures were based upon the work on institutional performance conducted by Tapinos et al, (2005). The choice reflected the agenda of the senior management to realise the potential of the staff of the university so as to improve the external image and prestige of the university. Thus LE would be a ‘success’ if it supported the goal of the institution to increase technical and allocative efficiency whilst maintaining the effectiveness of undergraduate research programmes as indicated by internal and external sources.

•

Student The measurements of the students’ views of the LE here were all related to satisfaction with their learning experience and outcomes (Tam, 2006). It was assumed that whilst the dissertations had a high instrumental value (encapsulated in the mark awarded) this was not all that satisfaction is predicated upon: the dissertation management process must be


Page 14 of 23


Applying Performance Management Principles to a Learning Environment for Undergraduate Dissertations: a Case Study perceived as being equitable and open. This perception was derived from the difference between expectation of the dissertation unit promulgated by the LE and what was actually experienced by the learner. What that needed measured here is quality so it is more accurate to call these measures for the student perspective Key Quality Indicators (KQIs) (Browne et al, 1998).

BSC Perspective: Id 01 02 03 04

KPI Objectives = Service Assurance Availability Reliability Responsiveness


KPI Objectives = Knowledge Absorption Diffusion Exploitation Generation


KPI Objectives = Realisation Contribution Effectiveness Productivity Credibility


KPI Objectives = Satisfaction Outcomes Preparation System Supervision

Administrative Metric Skills level of dissertation support staff Range of VLE facilities Consistency of the assessment processes Turnaround time of requests for help and guidance

Interest (Table 2 c,k c,k c,j c,k

Source Opinion Nominal Statistical Statistical

Faculty Metric Number of academic enrolling on RM courses Extent of communication of best research practice Extent of pedagogical ideas implemented Number of spin-off papers published

Interest (Table 2) j,m b,c c,l c,m

Source Nominal Opinion Opinion Nominal

Institutional Metric Ratio of administrative cost to total unit cost Average and spread of mark for the unit Ratio of dissertations to supervisors External Examiner Rating of student research skills

Interest (Table 2) a,l c,l a,l c,l

Source Financial Statistical Statistical Opinion

Student Metric Rating of the learning gain from the unit Effectiveness of the teaching of research methods Usefulness of the dissertation management system Effectiveness of the supervision

Interest (Table 2) c,j b,c c,k b,c

Source Opinion Opinion Opinion Opinion

Table IV: KPIs for LE Performance Evaluation

Step 6 - Specify Performance Metrics. Practicality was the key here by identifying simplest way of dimensioning the measurement. The process was to look at each of the objectives shown in the strategy map and then a list of candidate metrics were drawn up based upon existing sources of information or ones that Taken from The International Journal for Quality and Standards www.bsieducation.org/ijqs

Page 15 of 23


Applying Performance Management Principles to a Learning Environment for Undergraduate Dissertations: a Case Study could be easily adapted for monitoring performance against targets. Some of these were agreed by the unit delivery team and others were formally established by the management. The metrics were finalised by posing and answering several questions:

(i) How important is the metric for evaluating success of the LE? The answer revolved around the idea of whether an indicator was more ‘key’ than the others. The conclusion was that the satisfaction ratings within student perspective certainly met this criterion and this should be reflected in the aggregation of the points score (see (iv) below). (ii) Does the target time action to the LE? Some of the KPI/KQIs reports would be required each academic term (i.e. three per year), and the remainder linked to an annual planning cycle. (iii) What is the referent for the target? This question was about establishing whether a current snapshot useful or are trends in the direction of the results were important. This was a difficult issue to resolve before the LE was completely implemented because for some of the measures no data set existed so the default position was single observation. (iv) For this target can the results be aggregated with others? This decision was about differentiating between measures by weighting each one for its relative importance. The student satisfaction ratings, for example, could be double weighted. Each metric would then contribute its points reflecting the degree of attainment so that scores could aggregated per view and across whole score card to give an overview of the LE performance. At the time of writing a decision had not been made about this pending the validation of KPI/KQIs by the owners and users through internal position papers and further more informal discussions.

Step 7 - Implement Measurements of LE Performance The design of the measurement presentation ‘layer’ of the performance management regime will be crucial for the acceptance of the measurements not only for the effect on clarity of the results. Congruent with the earlier point about the critical nature of organisational interfaces, it is vital to provide ways and means to enable a common understanding to be gained and to take collaborative actions in response to the results of evaluating LE operations. Discussions have identified the following IT enabled functions: Taken from The International Journal for Quality and Standards www.bsieducation.org/ijqs

Page 16 of 23


Applying Performance Management Principles to a Learning Environment for Undergraduate Dissertations: a Case Study a. Integration of LE with general information infrastructure including university-wide content management and future multi-media package ‘Blackboard.’ Links to personal web pages for supervisors and those students with a common interest in a particular subject area will also be essential. b. Dissertation event calendars, FAQs, examples, contacts, issues lists, announcements. c.

Automated production of a balanced scorecard and appropriate ‘dashboard’ summaries.

d. Ability to provide a common view of all the KPIs and appropriate information ‘drill down.’ e. A feedback messaging facility enabling communication of commentaries and observations about the performance measurement issue and problem At the time of writing the thought, the intention for the core performance management component (c) will at least initially be developed as an Microsoft Excel-based application that will be driven by downloading data from the central LE. The advantage of doing this is that the application should have a high level of familiarity, accessibility, and functionality. This should create a high degree of stakeholder buy-in, because additional training needs will be minimised. There are no extra software platforms have to be purchased so the Total Cost of Ownership of the product will be relatively modest. Excel also has excellent links to other Microsoft Office products (e.g. MS Word) and downloads/upload capability to central serverbased systems.

4

Conclusions and Reflections.

The BSC has proved its worth as a guide for managing improvements to a LE. Early results evidenced by the expressions of satisfaction from students and supervisors; a higher degree of consistency in assessment practice and reduction in timescale and effort of administrative and support staff. However, the BSC has some limitations in the context of this project. Firstly, because it was developed for business shareholders, the financial element fails to translate well in not-for- profit organisations, such as Higher Education Institutions. That is, where the ‘shareholders’ are the wider community, and the HEI management represents external ownership. In the same vein, operationalisation of business strategy is not as easy to do as for a commercial organisation where corporate and departmental profit or turnover feature. In HEI the equivalent measurements are missing and what may be put in their place even, using activity-based costing model, is readily contested by the different parties. Conversely, the BSC underplays the value of human interactions, morale, motivations and relationships within a HEI. In this sense, the BSC was an ineffective pattern to follow for LE Taken from The International Journal for Quality and Standards www.bsieducation.org/ijqs

Page 17 of 23


Applying Performance Management Principles to a Learning Environment for Undergraduate Dissertations: a Case Study design. If it were possible to repeat this, it would best to shift the focus from financial capital to something that complements the Learning and Growth perspective by fully recognising social capital. Strategy maps (Symons et al. 2005) could be used to strengthen this aspect and to allow the BSC to capture the dynamics of the learning environment and its alignment with business strategy. The current project is driven by both the need provide the best for the learners and for educators and will help to nurture a common sense of identity between the dissertation tutors and the tutees. It is recognised that initially, only some, perhaps just a few of the more able and intellectually mature students will be able to actively to participate in the social network generated by involvement with research dissertations, the so called ‘communities of practice’ (Cross et al 2001). Not only would knowledge be more visible and accessible than at present but the development of dissertations would be driven by the collectivist views and will of the academic participants and not so much by managerial dictate. Furthermore, new supervisors should be able get up to speed quickly and for all academics the environment should help with developing their skills, particularly in the critical area of assessment. Co-operation and collaboration enabled through the LE, leading to better acquisition and utilisation knowledge of research methods and management is supportive of the growth within the University of crossdisciplinary sub-groups with an interest in research methods, the psychology of supervision or assessment strategies. In such an environment, the dissertation management team will be able to act more as knowledge brokers and facilitators, to those most directly involved in the LE. This should facilitate the development of supervisory skills; further developing the alignment of supervisor and student interests and resulting in more joint publications. The LE will also help to retain organisational memory of good ideas and help to avoid costly mistakes from knowledge loss caused by changes to personnel or lack of continuity when moving from one academic year to the next. In all this, performance evaluation will play a vital part because it acts as vital feedback mechanism for continued development of an LE that can deliver real benefits to the organisation. Some significant challenges lie ahead with regard to the operational deployment of LE performance management. The evidence from research on the implementation of performance management in commercial organisations (Nudurupat and Bititci, 2005) suggests that, increased visibility and transparency of knowledge management activities are likely to be subject to a form of ‘Uncertainty Principle.’ The act of measuring will affect the agendas, attitudes and behaviours of academic and administrative staff and as a form of structural modality eventually alter influence the institution (Jones and Hughes, 2001; Koskinen, 2003). Furthermore, it is likely that the implementation of the LE and its ongoing assessment will encounter resistance due to cultural discordance between the traditional


Page 18 of 23


Applying Performance Management Principles to a Learning Environment for Undergraduate Dissertations: a Case Study academic tutors and the e-oriented customers. In what way will happen, by whom and when, could form the subject of a separate paper. The longer-term significance of the performance system should be apparent in helping to improve the dissertation management, by linking back outcomes directly to actions. From an organisational point of view a significant payoff to a useful and usable LE is encouragement of a common sense of identity between the dissertation tutors and the tutees. Not only would knowledge be more visible and accessible than at present, but the development of dissertations would be driven by the collective views and will of the participants and not so much by management dictate. New supervisors could get up to speed quickly and for all academics the environment should help with developing their competencies, particularly in the fraught area of assessment. Co-operation and collaboration enabled through the LE should lead to better acquisition and utilisation knowledge of research methods and dissertation management. The dissertation management team (co-ordinator and officer) would then act more as knowledge brokers and facilitators, to the benefit of stakeholders most directly involved in the LE. This change should further facilitate the development of supervisory skills and may strengthen the alignment of supervisor and student interests, which could result in joint publications. It could also retain organisational memory of good ideas and help to avoid mistakes from any losses, e.g. those due to changes in personnel or moving from one academic cycle to another. In this way a truly complete learning environment for dissertations would evolve and flourish.


Page 19 of 23


Applying Performance Management Principles to a Learning Environment for Undergraduate Dissertations: a Case Study References Abduallah, F., 2006, “Measuring Service Quality in Higher Education: HEDPERF versus SERVPERF”, Marketing Intelligence and Planning, Vol. 24, No. 1, pp. 31-47. Alavai, M., .Yoo, Y. and Vogel, D.R., 1997, “Using Information Technology to Add Value to Management Education”, Academy of Management Journal, Vol. 40, No. 6, pp.1310-1333 Aldridge, S. and Rowley, J.,1998, “Measuring Customer Satisfaction in Higher Education”, Quality Assurance in Education, Vol. 6, No. 4, pp.197-204. Armstrong, M. J., 2003,“Students as Clients: a Professional Service Model for Business Education”, Academy of Management Learning & Education, Vol. 2, No. 4, pp. 371 – 383. Assessment & Education in Higher Education, Vol. 29, No.3, pp. 335 – 355. Athiyaman, A, 1997, “Linking Student Satisfaction and Service Quality Perceptions: the Case of University Education”, European Journal of Marketing, Vol. 31, No. 7, pp. 528-540. Atkinson, A., Waterhouse, J.A. and Wells, R.B. 1997, "A Stakeholder Approach to Strategic Performance Measurement", Sloan Management Review, Vol. 38., No.3, pp.25-37. Baruch,Y. and Ramalho, N., 2006, “Communalities and Distinctions in the Measurement of Organizational Performance and Effectiveness Across For-Profit and Non-profit Sectors”, Nonprofit and Voluntary Sector Quarterly, Vol. 35., No. 1, pp. 39-65. Becher, J.D., 2005, “Bridging the Gap between Strategy and Execution”, Business Performance Management, March, pp 11-17. Benson, A. and Blackman, D.,2003, “Can Research Methods Ever Be Interesting?”, Active Learning in Higher Education, Vol. 4, No. 1, pp.39 - 55. Booth, C. and Harrington, J., 2003, “Research Methods and Undergraduate Business Research: an Investigation”, International Journal of Management Education, Vol.3, No.3, pp19 -31. Browne, B., Kaldenberg, D., Borowne, W. and Brown, D., 1998, “Students as Customers: Factors Affecting Satisfaction and Assessment of Institutional Quality”, Journal of Marketing for Higher Education, Vol. 8, No.3, pp. 1-14. Busi, M. and Bititci, U.S., 2006, “Collaborative Performance Management: Present Gaps and Future Research”, International Journal of Productivity and Performance Management, Vol. 55, No.1, pp. 7-25. Buytendijk, F. and Flint, D. 2002, “How BAM Can Turn a Business into a Real-Time Enterprise, Gartner Research Notes, March. Calvo-mora, A., Leal, A. and Roldán ,J.L., 2005, “Relationships between the EFQM Model Criteria: a study in Spanish Universities”, Total Quality Management and Business Excellence, Vol. 16, No.6, pp. 741-770. Cardoso, E., Trigueiros, M.J. and Narciso, P., 2005, “A Balanced Scorecard Approach for Strategy and Quality Universities” ; in the Proceedings of the 11th International Conference of European University Information Systems, June, Manchester. Clauston,T.J. and Whitcombe, S.W., 2005, “An Emerging Person-Centred Model for ProblemBased Learning”, Journal of Further and Higher Education, Vol.29, No.3, pp. 265 - 77. Clayton,J., 2005, “Ask the Expert”, CIO, 23rd May Clerehan, R., Turnbull, J., Moore, T., Brown, A and Tuovinen, J., 2003, Educational Media International, Vol 40.,No.1/2, pp. 17 -32. Cross, R., Parker, A., Prusak and Borgatti, S.P., 2001, Organizational Dynamics, Vol. 30, No.2, pp.100-115. Doring, A., 2002, “Challenges to the Academic Role of Change Agent”, Journal of Further and Higher Education, Vol. 26, No. 2, pp. 139-148. Dorweiler, V.P. and Yakhou, M., 2005, “Scorecard for Academic Performance on the Campus”, Managerial Auditing Journal, Vol. 20, No.2, pp. 138-144. Edwards, D.F. and Thatcher, J., 2004, “A Student-centred Tutor—led Approach to Teaching Research Methods”, Journal of Further and Higher Education, Vol. 28, No. 2, pp.195 – 206. Gordan, P.J., 2003, “Advising to Avoid or to Cope with Dissertation Hang-Ups”, Academy of Management Learning & Education, Vol. 2, No. 2, pp.181 – 195. Hand, L. and Clewes, D., 2000, “Marking the Difference: an Investigation of the Criteria used for Assessing Undergraduate Dissertations in a Business School”, Assessment & Evaluation in Higher Education, Vol. 25, No. 1, pp.5 – 21. Taken from The International Journal for Quality and Standards www.bsieducation.org/ijqs

Page 20 of 23


Applying Performance Management Principles to a Learning Environment for Undergraduate Dissertations: a Case Study Ishikawa, 2007, Ishikawa Diagra [online] Available at http://mot.vuse.vanderbilt.edu/mt322/Ishikawa.htm Jones, S. and Hughes, J, 2001, “Understanding IS Evaluation as a Complex Social Process: a Case Study of a UK Local Authority,” European Journal of Information Systems, Vol.10, No.4, pp.189-203. Kaplan, R.S. and Norton, D.P., 1996, The Balanced Scorecard, Boston, MA, Harvard Business Press. Kaplan, R.S. and Norton, D.P., 2001, “Transforming the Balanced Scorecard: from Performance Measurement to Strategic Management: Part 1”, Accounting Horizons, V15., No.1, pp. 87-104. Kim Hua, K. Tan, K., Platts, K. and Noble, J., 2004, “Building performance through in-process measurement: Toward an ‘indicative’ scorecard for business excellence”, International Journal of Productivity and Performance Management, Vol. 53, No.3, pp.233-244. Koch, J.V., 2003, “TQM; Why is its Impact in Higher Education so Small?”,TQM Magazine, Vol.15, No.5, pp. 325-333. Koskinen, K., 2003, “Evaluation of Tacit Knowledge Utilization in Work Units”, Journal of Knowledge Management, Vol 7.,No.,5, pp. 67-81. Liebowitz, J. and Suen, C.Y., 2000, “ Developing Knowledge Management Metrics for Measuring Intellectual Capital”, Journal of Intellectual Capital, Vol.1, No.1 pp. 54 -67. Mackinnon, J., 2004, “Academic Supervision: Seeking Metaphors and Models of Quality”, Journal of Further and Higher Education, Vol.28, No.4, pp.395 - 405. Milton,J. and Lyons, J., 2003, “Evaluate to Improve Learning: Reflecting on the Role of Teaching and Learning Models”, Higher Education Research & Development, Vol.22, No.3, pp.297 – 312. Niven, P., 2003, Balanced ScoreCard: Step-by-Step for Government and Non-Profit Agencies, John Wiley and Sons, New York, N.Y. Nørreklit, H. 2000, "The Balance on The Balanced Scorecard – A Critical Analysis of some of its Assumptions", Management Accounting Research, Vol.11, No.1, pp. 65-88. Nudurupat, S.S. and Bititci, U.S.,2005, “Implementation and Impact of IT-Supported Performance Measurement Systems, Production Planning & Control, Vol.16, No.2, pp. 152-162. Oldfield, B. and Baron, S., 2000, “Student Perceptions of Service Quality in a UK Business and Management Faculty”, Quality Assurance in Education, Vol.8, No.2, pp. 85-95. Olve, N-G., Petri, C-J. and Roy, S., 2004, “Twelve Years Later: Understanding and Realizing the Value of Balanced ScoreCards”, Ivey Business Journal, University of Western Ontario, May/June, pp.1-7. Pariseau, S.E. and McDaniel, J.R. 1997, ``Assessing service quality in schools of business'', International Journal of Quality & Reliability Management, Vol.14 No.3, pp.204-218. Ramsden, P., 1992, Learning to Teach in Higher Education, London, Routledge. Rees, P. and Gardner, H 2003, “Best Value, Partnerships and Relationship Marketing in Local Government”, International Journal of Non Profit and Voluntary Sector Marketing, v.8, Iss.2, pp. 143-152. Rowley, J., 2000, “Is Higher Education Ready For Knowledge Management?”,International Journal of Educational Management Vol.14, No.7, pp. 325-333 Rowley, J., 2004,“What is the Future for Undergraduate Dissertations?”, Education & Training, Vol. 46, No. 4/5, pp.176-181. Ruben, B.D., 1999, “Toward a Balanced Scorecard for Higher Education: Rethinking the College and University Excellence Indicators Framework”, QCI Center for Organizational Excellence, Rutgers University at www.qci.rutgers.edu. Sanger, M, 1998, Supporting the Balanced Scorecard, Work Study, Vol.47, No.6,pp.197-200. Sprenger, C. and Have, S.ten, 1996,”The Role of Knowledge Management in the Learning Organisation, Holland Management Review, September-October, pp. 73 -89. Symons, C., Orlov, L., Bright, S. and Brown, K., 2005, IT Strategy Maps: A Tool for Strategic Alignment, Forrester Research, November 21. Tam, M., 2006, “Assessing Quality Experience and Learning Outcomes”, Quality Assurance in Education,Vol. 14,No.1, pp. 75-87. Taken from The International Journal for Quality and Standards www.bsieducation.org/ijqs

Page 21 of 23


Applying Performance Management Principles to a Learning Environment for Undergraduate Dissertations: a Case Study Tapinos,E., Dyson, R.G and Meadows, M., 2005, “The Impact of the Performance Measurement Systems in setting the ‘Direction’ in the University of Warwick”, Production Planning and Control, Vol. 16, No.2, pp. 189-198. Taylor, J., 2001, “Improving Performance Indicators in Higher Education: the Academic’s Perspective”, Journal of Further & Higher Education, Vol. 25, No.3, pp.379-393. Todd, M., Bannister, P. and Clegg, S., 2004, “Independent Inquiry and the Undergraduate Dissertation: Perceptions and Experiences of Final-Year Social Science Students”, Todd, M., Smith, K. and Bannister, P., 2006, “Supervising a Social Science Undergraduate Dissertation: Staff Experiences and Perceptions”, Teaching in Higher Education, Truyen, F. and Van Rentergem,L., 2005, “Preparing the University Information Architecture for Netcentric E-learning and Research: a case-study:” in the Proceedings of the 4th European Conference on e-Learning, November, Amsterdam.Vol.11, No.2, pp.161-173. Webster, F, Pepper, D. and Jenkins, A., 2000, “Assessing the Undergraduate Dissertation”, Assessment & Evaluation in Higher Education, Vol. 25, pp. 71 -80. Widrick, S.M., Erhan, M. and Delven, G., 2002,“Measuring the Dimensions of Quality in Higher Education”, Total Quality Management, Vol.13, No.1, pp.121-131.

Acronyms BPM BSC CSF FAQ HE HEI IT KPI KQI LE RM ROI TQM VLE

Business Performance Management Balanced Score Card Critical Success Factors Frequently Asked Questions Higher Education Higher Education institutions Information Technology Key Performance Indicators Key Quality Indicators Learning environment Research Methods Return On Investment Total Quality Management Virtual Learning Environment


Page 22 of 23



Appendix A - Problems with the Dissertation Learning Environment before the intervention

Tracking of supervisors’ performance Communication within the team Collection of data is not agreed with the supervisor Limited use of the Virtual Learning Environment Turnaround of assessment Sporadic conformance to requirements Tracking student performance Inconsistent marking Inconsistent feedback to students Inconsistent level of support amongst supervisors Documentation not standardised across courses Allocation of dissertations supervision happens last and is sometimes given to people with no/little research experience Limited transfer of best practice Inadequate documentation of the processes and procedures in place Supervisors with limited knowledge of research methods theory Workload allowance does not reflect the investment of time per student Working with students with Additional Learning Needs Limited knowledge of the support provided to students Limited knowledge of what information has been disseminated to students. Wide variety of topics with limited expertise in-house Expectations of supervisor’s availability and turnaround of feedback unrealistic. Difficulties with estimating the scope of the project and managing the work Cases of academic offence Inadequate knowledge of Research Methods Limited knowledge of the research process Choice of topics digress


Page 23 of 23
