THE UNIVERSITY of N O R T H C A R O L I N A at C H A P E L H I L L DEPARTMENT OF PUBLIC POLICY ABERNETHY HALL CAMPUS BOX 3435
T 919.962.1600 F 919.962.5824
[email protected]
CHAPEL HILL, NC
dddill.web.unc.edu
27599-3435
DAVID D. DILL Professor Emeritus
Capacity Building through Academic Audits: Improving “Quality Work” in the UK, New Zealand, Sweden, and Hong Kong1 David D. Dill Abstract: As countries convert from state to market-centered public policies, there is increasing interest in new forms of public accountability. Capacity building initiatives that reform institutional frameworks are useful policy instruments during this period of transition. What are the impacts and implementation problems characteristic of this approach? This article reviews the experience with “Academic Audit,” a capacity building accountability instrument for universities adopted in the UK, Sweden, New Zealand, and Hong Kong. Academic audits altered the incentives for cooperative behavior among faculty members to improve student learning. Identified implementation problems included: training for the new process, the uncertainty of capacity building benefits, and the central role of information.
Introduction As many countries make the transition from state-centered to market-centered public policies, there is increasing debate about the most effective forms of public accountability for assuring the public interest (Kettl, 1997). Some advocate the rapid adoption of performancebased funding, contracting, or organizational report cards (Horn, 1995; Gormley and Weimer, 1999), while others suggest that there is the need for some intermediate form of accountability to help publicly-funded organizations adjust to the new competitive realities. Recent research on developing countries for example has suggested that enhancing the performance of public sector organizations through capacity building interventions designed to alter the “rules of the game” by which they function can be a crucial means of managing the transition from state monopolies
1
Published in The Journal Of Comparative Policy Analysis, 2(2), 2000: 211-234 1
to market competition (Grindle, 1997a). What are the characteristics of such “capacity building” approaches to the reform of institutional frameworks and how can they best be implemented? Higher education is an example of a public sector that has been experimenting extensively with new forms of accountability. The rapid expansion of access to higher education in Europe and Asia (termed “massification” outside the US) stimulated public concern as to whether universities had in place sufficiently effective means for assuring the quality of their programs and degrees. As noted in a recent guide produced by the Quality Assurance Agency for Higher Education (QAA, 1998) in the UK: Universities and colleges have an obligation to protect the standing and good name of higher education. When the higher education system was small and largely uniform, and made a relatively small claim on public funds, reliance upon implicit, shared assumptions and informal networks and procedures may have been possible, and sufficient. But with the rapid expansion of the numbers of students and institutions, the associated broadening of the purposes of higher education, and the considerable increase in the amount of public money, more methodical approaches have had to be employed to provide the same guarantees. Institutions’ own internal mechanisms are important elements in providing these guarantees, but external scrutiny is also needed to confirm that institutions’ responsibilities are being properly discharged. The process of external scrutiny also makes an important contribution to the improvement of quality (pp. 6-7). Lacking the tradition of voluntary accreditation characteristic of North America, a number of countries have been experimenting with some novel forms of academic accountability designed to increase the quality of teaching and learning of undergraduate or first degree level university students (Brennan, De Vries and Williams, 1997; Scheele, Maassen, and Westerheijden, 1998). One of these new forms of academic accountability, termed “Academic Audit,” adopts a capacity-building approach to reform. Academic audits, first implemented in the UK, attempt to assess the processes that universities have in place for assuring the quality of student learning and the standards of their degrees. Interest in this new policy instrument has become worldwide. Concerns about the effectiveness of the existing quality assurance system of state regulation and voluntary accreditation in the US have led to a number of new experiments with academic audit (Dill, 2000). These audit processes are not directly tied to funding, but evaluate and provide public reports on the quality assurance processes by which colleges and universities exercise their responsibility to ensure academic standards and improve the quality of their teaching and learning. As observed in Sweden, the focus of academic audits is not on the quality of measured outcomes, but on “quality work” (Östling, 1997) -- how a university satisfies itself that its chosen academic standards are being achieved. The principal goal of the academic audits is organizational improvement through the stimulus of external accountability for academic quality assurance processes. Academic audit, from the perspective of current analyses of public sector reform, represents a working model of external accountability designed to achieve institutional reform, i.e., to alter the underlying rules or incentive structures that in turn influence the quality of teaching and student learning in higher education. Academic work is carried out within an 2
institutional framework that provides incentives and sanctions for individual and group behavior. These institutional arrangements include the rules and regulations of governments and educational ministries affecting higher education, professional accountability systems such as external examining and accreditation, the norms and beliefs of the academic disciplines, the nature of competition in academic markets, and the policies and practices of colleges and universities. Over time the relative influence of the components of this institutional framework have evolved altering the incentives for academic work (Dill, 1997). Governments have therefore sought means of reforming the existing institutional framework in a manner that provides greater incentives for the public interest. In order to clarify this focus on institutional reform in higher education, the arguments that follows will use the term “institutions” to refer to the institutional arrangements defined above and the term “academic organizations” to refer to colleges and universities. The sections to follow review the experience with academic audits in the UK, New Zealand, Sweden and Hong Kong, assessing the impacts and implementation problems characteristic of this capacity-building approach.i The first section suggests how capacity building can serve as a perspective for exploring academic audits. The second section suggests why academic audit can be conceived as a capacity building initiative for institutional reform and summarizes the impacts of the initiative upon academic behavior. The final section reviews the particular problems encountered in implementing this capacity building approach to academic accountability. A more extensive discussion of the organization and focus of the academic audit process is provided in Appendix 1. Capacity Building and Academic Audits Over the last decade as policymakers have turned to more market-oriented means of achieving the public interest there has been a growing interest in capacity building as an instrument of public policy (Honadle and Howitt, 1986; McDonnell and Elmore, 1987; Grindle, 1997b). While the concept of “capacity building” is ubiquitous in the policy literature, the meaning of the term is rather liquid. The Dictionary of Public Administration offers the following definition: Capacity building … includes among its major objectives the strengthening of the capability of chief administrative officers, departments and agency heads, and program mangers in general purpose government, to plan, implement, manage, or evaluate policies, strategies or programs designed to impact on social conditions in the community (quoted in Cohen, 1995, p. 409). Following the first half of this definition Cohen (1995) argues that capacity building in the public sector must focus on developing human resource capacity and therefore related interventions should be aimed at strengthening the training, recruitment, effective utilization and retention of skilled public sector personnel. Following the second half of this definition Honadle (1986) argues that capacity building in the public sector must focus on developing the management capacity of public organizations and agencies in a systemic sense. That is, the ability of an organization to anticipate change, to formulate relevant policy, to devise programs to implement
3
policies, to attract as well as manage resources, and to evaluate performance to guide future actions. In contrast to these earlier perspectives Grindle (1997b) suggests that the concept of capacity building must be reconciled with the new context of market-oriented policies characteristic of contemporary changes in the public sector. Therefore capacity building initiatives in her view involve not only development of human resources, and organizational strengthening, but also institutional reform (Table 1). Following North’s (1990) logic, Grindle suggests that capacity building initiatives that reform institutions alter the underlying rules of the game in which organizations and individuals make decisions and carry out activities. Such initiatives involve development of new mechanisms of public accountability, regulatory frameworks, and monitoring systems that help transmit information about the structure and performance of markets, governments, and public officials and thereby affect the incentive structure for economic and political interaction. This expanded framework of capacity building initiatives provides a useful perspective for understanding the emergence and implementation of academic audits as a new mechanism of accountability in higher education policy. For example, capacity-building initiatives are most likely to be pursued when there is a perception that failures of organizational performance will lead to the loss of future material, intellectual, or human benefits (McDonald and Elmore, 1987). As previously noted the rapid expansion of higher education over the last decade in Europe and Asia raised serious public concerns about the adequacy of existing institutions for sustaining academic quality and standards and led to the initiation of academic audit processes in a number of countries. For example, the first academic audit process emerged in the UK following persistently voiced concerns in the 1980s by the Thatcher government about the quality of teaching in the rapidly expanding university sector.ii As a result the
Dimension
Focus
Types of Activities
Human Resource Development
Supply of professional and technical personnel
Training, salaries, conditions of work, recruitment
Organizational Strengthening
Management systems to improve performance of specific task and functions; microstructures
Incentive systems, utilization of personnel, leadership, organizational culture, communications, managerial structures
Institutional Reform
Institutions and systems; macro structures
Rules of the game for economic and political regimes, policy and legal change, constitutional reform
Table 1: Dimensions and Focus of Capacity Building Initiatives (Adapted from Grindle, 1997b, p. 9). Committee of Vice Chancellors and Principals (CVCP), an independent association of the heads of the UK universities of that time, established a series of study committees to examine academic issues confronting the university sector. One of the first committees was an Academic Standards group. By virtue of the Royal Charters or private acts of parliament by which they were established, British universities are responsible for the degrees they award and for their 4
own academic standards. Nonetheless, the CVCP was concerned that if the universities were perceived as not monitoring their own standards, some other agency might be assigned the job by the government. The Academic Standards Group issued a report in 1986 that recommended codes of practice for critical academic processes such as subject examinations as well as means of monitoring academic standards, which provided the universities yardsticks for selfcomparisons. But a later CVCP study of the extent to which the universities had implemented the committee’s recommendations suggested that superficial compliance was masking less than strong commitment. Under further pressure from the government the CVCP’s Academic Standards Group recommended the creation of an Academic Audit Unit (AAU) to provide external and independent assurance that UK universities have adequate and effective mechanisms and structures for monitoring, maintaining, and improving the quality of their teaching. The unit would be “owned” by the universities themselves through the CVCP.iii The creation of the New Zealand, Swedish, and Hong Kong academic audit processes was similarly a response to pressure from government for greater accountability on academic quality in the university system. In New Zealand the 1990 Education Amendment Act established the New Zealand Vice Chancellor’s Committee as a statutory body with explicit responsibility for standards and qualifications in the university sector. The Act also strongly implied that there should be active monitoring of these standards and qualifications. The Committee therefore established an Academic Audit Unit in 1994 to audit the quality processes within the universities. In Sweden, the 1993 Higher Education Reform decentralized the higher education system and as part of the reform directed that all universities should have quality enhancement programs. A National Agency for Higher Education was created in 1995 that had a number of evaluative responsibilities including the auditing of quality in the universities and university colleges. In Hong Kong the University Grants Committee was established in 1965 to advise the government on the academic development and funding of the higher education sector. As the system expanded in size and cost in the 1990s it became the government’s and public’s expectation that the UGC would take an active role in assuring the quality of higher education provision in the academic organizations for which it had responsibility. Initially the UGC implemented a Research Assessment Exercise similar to one developed in the UK in the mid1980s. This process tied the amount of UGC research allocations to universities to an assessment of the quality of published research at the subject level in each university. Recognizing the clear signal that this assessment sent about the importance of research, the UGC then instituted in 1995 a process to audit the quality of teaching and learning -- Teaching Learning Quality Process Review (TLQPR). The evolution of academic audit as a new means of accountability in higher education therefore seems consistent with the emergence of capacity building initiatives in international development work (Grindle, 1997b) and in K-12 education in the US (Goertz, Floden, and O’Day, 1995). Of greater interest to those interested in policy analysis and design, however, is the way that academic audits illustrate what Grindle (1997b) terms institutional reform and the particular implementation problems associated with this type of capacity building initiative. Before examining these issues it is necessary to understand the rationale for classifying academic audit as an initiative for institutional reform.
5
Academic Audits as Institutional Reform In what way can academic audits be classified as a capacity building initiative designed to reform the institutional framework for academic quality? To understand this approach it is necessary to develop three arguments. First, to conceive of academic quality assurance in teaching and student learning as a problem of collective action. Second, to suggest how the changing institutional framework of academic quality assurance is affecting academic behavior. And third, to examine the impacts of academic audits on the cooperative efforts of faculty members to improve academic quality. Academic Quality as a Problem of Collective Action From North’s (1990) perspective institutions may be understood as mechanisms for addressing dilemmas of collective action through the structuring of cooperative behavior. “Dilemmas of collective action” are defined as those circumstances in which individuals may fail to coordinate their activity to produce a collective benefit, because it is not in their private interests to do so. That is, in the absence of credible commitments to one another, individuals may have incentives to defect from or shirk cooperative behavior in favor of individually beneficial activity. As North (1990) illustrates in his comparison of developed and underdeveloped countries, existing institutional structures do not always lead to productive human interaction. Unless institutional frameworks provide incentives for verifiable, enforceable, mutual commitments human interaction may be structured in a manner that is inefficient for the larger society. North’s perspective is thereby valuable for assessing the extent to which mediating institutions encourage individuals to cooperate for the common good. A first issue, therefore, is how can academic quality assurance be understood as a problem of collective action? An important and controversial issue in the development of quality assurance policies on teaching and learning is how quality assurance is conceptualized (Dill, 1992). Is academic quality primarily the product of the independent actions of individual professors, or is it significantly influenced by the cooperative efforts of program faculties and departments? Emerging but not yet definitive evidence suggests that academic quality assurance is best conceptualized in a holistic manner that is greater than the sum of the activities of individual teachers in separate classrooms (Ewell, 1988; Pascarella and Terenzini, 1991). As Tony Becher (1992) noted in developing a quality assurance system at the University of Sussex in the UK: ...the most important consideration in quality assurance must be a holistic rather than an atomistic one, namely the benefits students derive from the totality of their degree programmes, rather than the satisfactoriness or otherwise of their interactions with individual members of staff (p. 58). Similarly, research in the United States on teaching and learning (Pascarella and Terenzini, 1991) consistently reveals that, while student content learning and cognitive development is related to the quality of individual teaching students receive, it is also affected by the nature and sequence of students’ curricular experiences, and by the extent to which the curriculum faculty are collectively involved with and communicating with each other about the substance of teaching and learning. 6
This research suggests that more systematic efforts to improve the quality of student learning will likely require cooperative actions by faculty members to assess learning outcomes, to “restructure” the curriculum, and to better coordinate their individual efforts at instruction. The Changing Institutional Framework of Academic Quality Cooperative behavior in academic quality assurance within universities has traditionally been influenced by a number of interacting institutions. These include the rules and sanctions of academic organizations, the norms and incentives of the academic disciplines, and the influence of the competitive market for academic products and labor. There is increasing evidence that over time the traditional institutional framework is changing and has become less effective for motivating faculty collaboration in improving teaching and student learning within colleges and universities. The evolving institutional framework for higher education appears to provide strong incentives for faculty disengagement from the cooperative activities that have been linked to the improvement of academic quality particularly in undergraduate or first degree-level education. One example of this shifting framework is the growing “marketization” of higher education (Williams, 1995). Global economic forces are leading to increasing international competition for students, faculty members, and programs. Within countries and states this competition is being further fostered by the adoption of quasi-market and market-based policy instruments by governments as a means of achieving greater efficiency and innovation in their expanding higher education systems (Dill, 1997). Because of the structure of academic organizations, this shift in the competitive context can affect faculty members’ incentives for teaching versus research. In a series of influential papers James (1978, 1986, 1990; James and Neuberger, 1981) argued that universities are multi-product firms, producing both teaching and research, in which the faculty have significant control over the factors and consequently the costs of production. As Clotfelter (1996) recently observed: The university’s central and most distinctive activities – teaching, research, and public service – are carried out largely by its most distinctive sector of employees: the faculty. As a consequence, the decisions about how to allocate faculty effort are basic to the functioning of colleges and universities, and to their cost. ...most day-to-day decisions concerning these activities are entirely in the hands of departments and faculty members themselves ....(p. 179) James (1986) noted that faculty members, particularly in universities, tend to value research over teaching, because of its intrinsic interest, because of its clear contribution to unit reputation (which is a major proxy for academic quality), and because in increasingly competitive research and labor markets time spent on research can lead to enhanced grant revenue and future earnings for the individual faculty member. In this context, and in the absence of close monitoring and valid measures of student learning, faculty members both individually and collectively will choose to “satisfice” teaching quality, to limit their time investment in teaching and to maximize their time investment in graduate teaching and research. James (1986) tested this model by examining the trends in the percentage of self-reported faculty time devoted to teaching and in the average teaching loads at universities during the 7
period 1953-1965, when the US higher education system evolved from an elite to a mass system. Her analysis confirms in major universities a significant decline in reported time spent teaching and an increase in reported time spent on research. More recent studies (Clotfelter, 1996; Fairweather, 1996; Getz and Siegfried, 1991) suggest a continuation of this trend toward faculty disengagement in teaching and greater investment of time in research.iv Similar concerns about a shift in faculty time toward research have been noted in the UK and in Hong Kong following the implementation of policies featuring more competitive systems for the allocation of university research funds (Dill, 1997; Massy, 1997a). There is also some evidence that the influence of the norms and values of the academic disciplines on faculty collective behavior is eroding in the new context of “massification” and “disciplinary fragmentation.” The explosive growth of knowledge in the traditional disciplines as well as the evolution of new interdisciplinary fields such as molecular biology, public policy, and ethnic studies, have led to the rapid expansion of academic programs in universities throughout the world (Clark, 1996).v There is emerging evidence in the US system that this expansion of academic knowledge as well as related specialization is affecting the ability of faculties to reach consensus on what should be taught within academic curricula, or what constitutes an academic curriculum at the university level. Surveys on the views of faculty members in different disciplines (Lattuca and Stark, 1994) suggest declining normative influence from the various academic professions on the content of academic programs. In many disciplines faculty members did not easily agree on definitions of curricula content, nor were they in agreement that specified sequences of learning content were appropriate for students. Even in professional fields such as business and engineering, where faculty members reported the highest perceived consensus on the nature of academic knowledge, they also reported that defining the content of the professional courses was one of their most serious curriculum tensions. In several disciplines, faculty members expressed the belief that the field’s diversity precluded achieving a consensus on what students should know. Finally, the university itself may be conceived as an institutional structure that can influence academic quality. Potentially universities can promulgate rules and specify incentives or sanctions to improve the design of academic programs, increase the time commitment of faculty members to teaching, and stimulate collegial responsibility for academic quality. There is some evidence from the US that the influence of university-based rules and sanctions on faculty collaborative behavior is declining.vi Field research at the departmental level in US universities (Massy, Wilger, and Colbeck, 1994) has revealed a pattern of “hollowed collegiality” in which departments nominally appear to act collectively, but avoid those specific collaborative activities that might lead to real improvements in curricula cohesion. For example, faculty members readily reported informal meetings to share research findings, collective procedures for determining faculty promotion and tenure, and consensus decision making on what particular courses should be offered each term and who should teach them. But: Despite these trappings of collegiality, respondents told us they seldom led to the more substantial discussions necessary to improve undergraduate education, or to the sense of collective responsibility needed to make departmental efforts more effective. These vestiges of collegiality serve faculty convenience but dodge fundamental questions of task. This is especially the case, and is regrettable, with respect to student 8
learning: collegiality remains thwarted with regard to faculty engagement with issues of curricular structure, pedagogical alternatives, and student assessment (Massy, Wilger, and Colbeck, 1994, p. 19). A major contributor to this hollowed collegiality was an observed pattern of fragmented communication within departments in which traditions of individual autonomy and academic specialization have led to atomization and isolation among faculty members. Not only do faculty members do much of their teaching alone, but also because disciplinary sub-fields are defined quite narrowly, many faculty members find it almost impossible to discuss their teaching with other faculty members. The above analysis suggests that maintaining or improving academic quality represents a problem of collective action. The evolving institutional framework of market competition, professional norms, and university rules and sanctions appears to provide insufficient or negative incentives for cooperative academic activity leading to improved student learning. In this new context there appears to be an increasing rationale for government accountability initiatives such as academic audits designed to alter the existing incentive structure for academic quality. What then has been the impact of academic audits on faculty cooperation for improving teaching and student learning? The Impacts of Academic Audits on Academic Behavior The academic audit processes in the UK, New Zealand, Sweden, and Hong Kong have each been evaluated (Coopers & Lybrand, 1993; Brennan, et al, 1999; Stensaker, 1999; Meade and Woodhouse, 2000; Nilsson and Wahlen, 2000). These evaluations have identified the types of impacts of the first cycle of academic audits on academic behavior within universities. These initial academic audits have accomplished the following: •
•
•
Placed responsibility for improving teaching and student learning on university agendas. The public nature of the academic audit process and the publication of the audit reports placed responsibility for improvement of the processes designed to assure and improve the quality of teaching and student learning on the agenda of each audited college and university, in many instances for the first time (Coopers & Lybrand, 1993; Brennan, et al, 1999; Meade and Woodhouse, 2000; Nilsson and Wahlen, 2000). Because there were in every academic organization some faculty members who recognized the need for such improvement, this public attention provided an opportunity for changes in university policies and structures that were often described after the fact as “already underway.” Facilitated active discussion and cooperation within academic units on means for improving teaching and student learning. All the evaluations noted that faculty members at the departmental level reported an increased amount of collegial discussion and cooperative activity on academic quality assurance. This was particularly the case in the Swedish and Hong Kong versions of academic audit, which encouraged both organizational and departmental responses to the audit reviews (Brennan, et al, 1999; Stensaker, 1999). Helped to clarify responsibility for improving teaching and student learning at the academic unit, faculty, and organizational level. The academic audit process often revealed that a pervasive norm in many academic organizations is the avoidance of responsibility for the quality of teaching and student learning. University administrators often adopted an 9
•
“invisible hand” approach to academic quality assurance, either because they equated academic quality with recruiting the best faculty members and students and leaving them alone, or because they misinterpreted academic freedom to mean that individual faculty members have an unchallenged right to determine the content of their courses.vii In either case, the academic audit process was frequently reported as clarifying that the departments, faculties, and university leadership each have a responsibility for the active development of quality assurance and improvement processes for teaching and student learning (Massy and French, 1999; Stensaker, 1999; Nilsson and Wahlen, 2000). Provided information on best practices both within academic organizations and across systems. The process of academic audit often revealed to administrators, sometimes for the first time given the norm of the “invisible hand,” that there were “best practices” of academic quality assurance within their universities that could be disseminated as a means of academic improvement (Brennan, et al, 1999). Similarly, the UK developed a “quality enhancement” function designed to disseminate best practices discovered through the process of academic audit (Coopers & Lybrand, 1993).
There is some evidence from these countries that the academic audit initiatives have altered the institutional framework for academic quality assurance, at least in the short run, and provided some incentives and support for collective actions designed to improve the quality of teaching and student learning. In reaction to these reviews, the universities audited are changing their internal norms, policies, and structures with regard to “quality work.”viii In the process they are more closely approximating “learning organizations” (Dill, 1999a), building a capacity for developing and transferring knowledge for the improvement of their core academic processes. Policy Implementation The introduction of academic audit into the higher education sector also illustrated a number of special implementation problems characteristic of capacity building initiatives designed for institutional reform. These included whether the initiating agency has developed the knowledge and skills necessary to conduct process-oriented audits, the concern of policy makers and others about the intangible and uncertain results of capacity building activities, and the role of information in capacity building efforts designed to alter the structure of incentives. Training for Process Review A distinctive characteristic of academic audit is its focus on process. The Hong Kong TLQPR for example evaluated quality assurance processes in undergraduate or first degree teaching such as the means of designing curricula, evaluating teaching, assessing learning outcomes, and providing resources to quality assurance work. In each country, however, it quickly became apparent that even experienced academics – whether auditors or audited – may not have a clear conception of what constitutes academic quality assurance process. In each of the systems there were reports of confusion both among those audited and those conducting the audits as to the differences between academic substance and process, outcomes and process, objectives and process, etc. Auditors and faculties reviewed both reported that a great deal of learning took place about the nature of academic quality assurance processes and best practices in higher education. This discovery led in every country to a major emphasis on auditor training
10
and to preparatory visits designed to orient universities to the purposes and practices of academic audits. With the exception of Hong Kong, which utilized an intact team for all reviews that was composed primarily of members of the University Grants Committee and local university representatives, the other audit processes had to recruit auditors. Recruitment was often carried out through the professional contacts and experience of the individual directing the audit process, although university nominations and individual applications for auditor were invited in the UK, New Zealand, and Sweden. In the UK the Academic Audit Unit evolved a process of selection in which potential auditors were invited to participate in a day of role-play with audit materials. In this manner their knowledge of quality assurance processes, understanding of the goals of audit, and skills in working as a member of a team could be assessed prior to their actual selection as an auditor. Of particular concern to the AAU was weeding out “quality ideologues,” those who felt that there was “one best way” to assure academic quality assurance. New Zealand, Sweden and the UK all used unique teams of auditors, constructed for each audit visit. All but Sweden also created “banks” of trained auditors, individuals appointed for a period of time, who were then used for a number of different audits as a means of improving audit reliability. Hong Kong, by contrast, utilized an intact team over the course of a year for all its university audits as a means of enhancing auditor learning and the reliability of the process. The relatively small size of the Hong Kong system (seven colleges and universities at the time of the audits) made this a feasible approach. The employment of an intact team enabled the audit process to begin more quickly, with both the auditors and the universities learning about the process as it evolved. The UK, New Zealand, and Sweden also developed auditor training in which members of audit teams were given an opportunity to read and critique pilot audit reports, interview those involved in previous audits, and develop skills in questioning. Sweden, New Zealand and the UK developed audit protocols to help guide auditors in their visits. Audit guides including these protocols were made publicly available so as better to inform universities about how audits would be conducted. The openness of this process underscores the emphasis on development and improvement as a goal for the audit process. The Uncertainty of Capacity Building Benefits As McDonald and Elmore (1987) suggest, one limitation of capacity building initiatives is their future orientation and the concern this raises in the minds of policy makers about the benefits of the process. Building the capacity of universities to assure academic quality offers the potential of future returns, but because the benefits are distant, uncertain, and difficult to measure policymakers may have problems justifying such initiatives and supporting them over time. This issue was raised in a number of the external evaluations of academic audits (Coopers & Lybrand, 1993; University Grants Committee, 1997; Nilsson and Wahlen, 2000). That is, the logic of academic audits relies upon an underlying but difficult to prove assumption – that an improvement in quality assurance processes will eventually lead an improvement in academic outcomes. Understandably, academics and policy makers sought evidence confirming that such a relationship actually existed, evidence that academic audits with their focus on process could not easily provide. As academic audit matures, however, the reviews are responding to this criticism by focusing more sharply on the measures of learning outcomes employed by academic programs 11
and universities themselves as well as on the role such measures play in university quality assurance and improvement processes. In the second cycle of academic audit in the UK, “Continuation Audit” teams are pursuing the following question as a primary means of evaluating how effectively universities are discharging their responsibilities for the academic standards and quality of their programs and awards (QAA, 1998): “Can you convince us that the evidence that you are relying on for this purpose is sufficient, valid and reliable?” In both New Zealand (Meade and Woodhouse, 2000) and Hong Kong (Massy and French, 1999) it has been proposed that the second cycle of academic audit focus more on the outcomes of the quality assurance processes being audited by asking how the universities themselves measure such outcomes. In the US the recently designed program accreditation process being implemented by the Teacher Education Accreditation Council (TEAC, 1998), which is based upon the precepts of academic audit, will require that each program to be accredited submit an “Inquiry Brief,” which demonstrates and documents evidence: 1) about students’ learning and understanding of the curriculum, 2) that assessments of student learning are valid, and 3) that the program’s quality assurance system yields information that leads to program improvement. The Role of Information In contrast to traditional initiatives for capacity building, which have emphasized improvement in personnel or training in new management systems (Cohen, 1995), capacity building initiatives designed to reform institutions appear most effective when they emphasize the communication of information (Jacobs and Weimer, 1986). The success of academic audits must be attributed in part to their clear communication of public expectations that academics are both individually and collectively responsible for assuring academic quality. By conducting and publishing the academic audits, the auditing agencies clarified both the faculty’s professional authority for assuring academic standards and their accountability to the public for carrying out this responsibility. The academic audits directly challenged the prevailing norm of the “invisible hand” approach to academic quality assurance. The resulting debates within the audited universities about the appropriate balance between individual and collective responsibility for academic quality and student learning were generally seen to be valuable. The academic audit reports also helped to make public for the first time variations in quality assurance processes across units and universities and thereby provided an opportunity for improvement through transfer of best practices both within and across universities. Academic audits also helped to foster communication among academics on issues related to improving student learning (Dill, 1999a). First, the academic audits introduced a common language for discussing academic quality assurance, which facilitated collective discussion within and across academic departments. Second, academic audits themselves helped to create or strengthen channels of communication. For example, academic audits frequently stimulated the adoption of systematic feedback mechanisms from the clients of academic programs – whether stakeholders external to the universities or currently enrolled students. If the audit processes were designed to facilitate the development of academic quality assurance processes at both the university and the departmental level, as they were to differing extents in the UK, Sweden, and Hong Kong, they also helped to stimulate vertical communication on academic quality assurance between departments and central administrations as well as horizontal communication among faculty members within a department. In a number of cases, once 12
developed, these communication channels have been further strengthened as part of an ongoing improvement of quality assurance processes and procedures within the audited universities (Dill, 1999a). As noted above, perhaps the greatest contribution of academic audit may be in encouraging faculties to develop public measures of the value-added by academic programs. The lack of this information appears to be the most critical cause of the dilemma of collective action in academic quality assurance (Dill, 1999b). As game theorists have noted (North, 1990), even in a context characterized by small numbers and repeated dealings among individuals, such as an academic department, unless individuals have sufficient information to calculate relevant costs and benefits cooperative behavior for joint gain is unlikely to occur. But summative measures of student learning, such as comprehensive examinations or common projects, are becoming less common in all systems of higher education (Dill, 1999b). With the increasing privatization of academic work we lose the information that once made teaching and student learning “community property” (Shulman, 1993). Thus, if a measure of the value-added to students by an academic program is not publicly available, then individual faculty members will base their decision on the amount of time to commit to cooperative activity in teaching and curriculum improvement on the individual costs and benefits to themselves. The benefit of cooperating with other faculty members in the design and implementation of higher quality educational academic programs is therefore likely to receive little or no value. By the same logic, faculty members also have little incentive to invest time and effort in developing or maintaining measures of the value-added by academic programs, therefore the decline or rise in the quality of student learning becomes increasingly invisible and cannot act as a check on opportunistic faculty teaching behavior. By creating an incentive for universities as well as academic departments and programs to collectively assess and discuss student learning, academic audit helps to create the information necessary for cooperative behavior in professional work. Conclusion The shift from policies of state-supervision to market-based steering in the provision of public services such as higher education is an obvious trend in many countries of the world. How best to assure public accountability in these new systems is a point of continuing discussion and debate. Accountability instruments that adopt a capacity building approach, such as academic audit, appear to offer a number of advantages. First, if focused on information and communication, such capacity building initiatives have the potential for institutional reform by altering well-entrenched professional norms and incentive structures in public organizations that affect the ability of these organizations to respond to market signals. Second, capacity building initiatives for institutional reform can serve as effective transitional policy instruments that can be “faded” as markets become more perfectly competitive and thereby more effective in steering public sector organizations in the public interest. Finally, capacity building initiatives such as academic audit are a relatively flexible and generalizable mechanism of public accountability. The rapid diffusion of this new accountability mechanism across cultures and countries offers some insights into the characteristics of policy instruments that are more easily transferable. Capacity building approaches that focus on information and communication such as academic audit appear to be particularly effective in reforming the institutional framework of professionally based public organizations such as universities. In these settings the value of 13
professional specialization may come to dominate the needs of clients, customers, and the public interest. Prevailing professional norms and practices such as academic freedom may then provide cover for the pursuit of private interests. In the instance of organizations like universities that offer joint products this problem is further confounded by the evolution of differentially competitive markets for research and educational programs. This imbalance can create incentives for faculty members to cross-subsidize time for research with time for teaching. As will be noted below, this potentially inefficient institutional framework may be corrected in the long term by freeing and facilitating market competition for educational programs through policy instruments such as performance-based funding, organizational report cards, and studentbased funding. But in a sector dominated by publicly funded organizations, the failure of universities to adapt to these market signals may have significant consequences for equal access to quality education. Furthermore, professional values and norms are deep-seated and, given the necessary tradition of professional autonomy, difficult to alter through market signals alone. Information-based accountability initiatives such as academic audit can confront professional values and practices at the level of the basic production unit, utilizing peer-based discussions and reviews that are more likely to lead to behavioral change over time. If well designed, capacity building initiatives can encourage the development of new communication channels between the larger society and the focal organization, as well as restore communication links among the professionals themselves, helping to rebuild collegial bonds supportive of collective action on academic programs. Capacity-building accountability approaches such as academic audit also offer a valuable policy instrument for making the transition from systems of state supervision to more marketbased policies. Over time changes in the market structure for academic programs as well as in the underlying technology of teaching and learning will likely be essential for reforming the institutional structure influencing academic work. Academic audits are of necessity occasional and as such can sometimes be successfully resisted or compromised. The rapid development of international distance learning and the adoption of information technology in campus-based teaching already offer some evidence of the ways that market forces are influencing the structure of academic incentives (Massy, 1997b). Information technology for example helps to make the processes and activities of teaching and student learning more visible and thereby facilitates greater cooperation in the production of teaching materials and in the assessment of students. As the institutional framework for professional cooperation changes capacity building approaches such as academic audit can be phased down, focusing on those organizations most in need of review and emphasizing dissemination and sharing of best practice. Universities that demonstrate that they have developed a capacity for assuring the quality of their degrees and improving student learning, for example, can be granted greater authority and responsibility over their quality assurance processes. The diffusion across countries of an information-oriented capacity-building initiative such as academic audit also illustrates important characteristics of transferable policy instruments. Academic audit was initially developed in the UK and was rapidly adopted in the Scandinavian countries, New Zealand, and Hong Kong. In a reversal of the recent trade flow of policies in higher education, academic audits are now being adopted in the US (Dill, 2000). As a process-oriented accountability tool, academic audits are remarkably flexible. In Hong Kong the same process-oriented approach has been used to build the management capacity of universities in an increasingly market-oriented higher education context (Massy, French, and Thompson, 1999). Different countries have also been able to emphasize different values in their audit 14
reviews, thereby adapting the instrument to their own cultural and academic traditions (see Appendix 1). A second important dimension of academic audits is their compatibility with professional norms, including a reliance on peer review and professional communication as well as an emphasis on disseminating best practice. As a consequence the process has generally been more acceptable to professionals than accountability instruments emphasizing performance measurement or reporting. In countries with strong traditions of professional authority such as Sweden academic audits offered a politically more palatable approach for initiating public accountability. Capacity building instruments such as academic audit acknowledge, in a way that the simplistic distinction between governments and markets does not, that professional norms and ideologies remain a significant institutional structure in modern societies (Denzau and North, 1994). Designing policy instruments that can alter and strengthen the normative web of accountability that exists within a professional community may still offer a potentially powerful lever of institutional reform. Appendix 1: The Organization and Scope of Academic Audits Academic audits are external reviews of a college or university’s academic quality assurance processes. In designing the first academic audits in the United Kingdom the Academic Audit Unit (AAU) of the Committee of Vice Chancellors and Principals (CVCP) was influenced by the concept of “quality audits” as defined by the International Standards Organization: Quality audit is a systematic and independent examination to determine whether quality activities and related results comply with planned arrangements and whether these arrangements are implemented effectively and are suitable to achieve objectives (HEQC, 1993). In adapting this concept of a quality audit to higher education, the AAU defined the “planned arrangements” referred to in the ISO definition as “those processes by which academic institutions exercise their responsibility to assure academic standards and improve the quality of their teaching and learning.” Academic audits of universities are now being conducted on a comprehensive basis in the UK, New Zealand, Sweden, and Hong Kong. Each of the audit approaches has in common an organizational focus, an orientation to quality process, and a similar procedure for evaluation featuring peer review, a university submission, a site visit, and a published report. However, other aspects of the audit designs vary (Table 2). For example, the audit processes differ in the composition of the audit teams, the continuity of the teams in the reviewing process, the length of the audit visit, the means of distributing reports, and the follow-up activities. Some of these differences are likely due to the substantial variation in system size among the countries, ranging from the UK’s 500+ colleges and universities to New Zealand and Hong Kong with only 7 colleges or universities each. Other variations (e.g., the composition of audit teams) appear to be due to political or cultural considerations. Audit teams were composed primarily of experienced academics in every country (the majority of the team members representing the University Grants Committee in Hong Kong were also local academics). In Sweden there was a student member of every team, while in both
15
UK Date Begun Colleges and Universities Audited
Makeup of Audit Team
Team Continuity Audit Visit
Report Distribution Follow-up
1990 180 universities, colleges of higher education, and specialist academic organizations (4) 3 UK academics Secretary
Unique team Audit panel Two visits Second visit 2-4 days, varies by university size Sent to university Available to public Distributed Progress report Quality enhancement Published studies Codes of conduct Evaluation
New Zealand
Sweden
Hong Kong
1994 7 universities
1995 36 colleges and universities
1995 7 organizations (polytechnics, universities, and a college)
(7) 2 NZ academics 2 Industry/ Commerce 1 Overseas academic Director AAU Secretary Unique team Audit panel Two visits Second visit 3 days
(6) 3 Swedish academics 1 Industry/ Government 1 Student Secretary Project officer Unique team
(18) 8 UGC 8 HK academics 2 Overseas academics
Two visits Second visit 2-3 days, varies by university size Sent to university Available to public Distributed Internet Feedback visit Published studies
Two visits Second visit 1.5 days
Sent to university Available to public Distributed Voluntary progress report Evaluation
Intact team
Sent to university Internet Progress report Teaching workshop Evaluation
Table 2: Basic Organization of Academic Audits Sweden and New Zealand non-academics were also appointed. The UK, New Zealand, and Sweden all used a unique team for each audit, although the UK and New Zealand appointed for a period of years audit panels from which they could draw an audit team. In Hong Kong, a single large team was used to review all colleges and universities in the first cycle of academic audits. Audit teams generally visited each university twice: a first, briefer visit in which representatives of the audit team introduce the audit process to the academic community and answer questions, and a second visit during which the actual audit was conducted. Audit reports were sent to the relevant university for factual correction in each country and then were made public by the auditing agency by direct distribution, press releases, and/or by publication on the Internet. All the countries instituted feedback visits or progress reports to follow-up on the recommendations of the audit team. The UK, as well as Sweden and Hong Kong to a lesser degree, conducted quality enhancement efforts designed to disseminate examples of good practice in academic quality assurance identified through the audit process. Another source of variation among the audit approaches is the scope of the processes they assess. The widest scope is to be found in the audits carried out in Sweden, which focused on 16
evaluating university “quality enhancement” programs and their implementation. Therefore, Swedish quality audits examined: university strategies for quality implementation; how and to what extent teachers, researchers, administrative personnel and students were committed and involved in quality enhancement programs (“universal participation in quality enhancement”); the ways external stakeholders had been identified and their needs and demands determined (“cooperation among interested parties”); and methods for evaluating activities with particularly attention to staff and student recruitment and development (“evaluatory and follow-up systems”). Given the government’s desire to encourage university self- regulation, the Swedish audits placed particular emphasis on the exercise of organizational leadership in quality enhancement programs and on the strategies utilized to develop national and international contacts for improving professional activities (“external professional relations”). The New Zealand approach, modeled on the UK, focused attention on means of quality assurance in the appointment and development of staff, as well as quality assurance mechanisms in courses and programs. But the New Zealand approach added the additional topic of quality assurance in research with particular regard to its connection with teaching (Woodhouse, 1998). The UK academic audit in its earliest manifestation focused on university mechanisms for quality assurance in relation to academic staff, in the provision and design of courses and degree programs, in teaching and communication methods, and in taking account of the views of stakeholders. The Hong Kong TLQPR has the most carefully defined scope of the four processes reviewed, evaluating quality assurance processes in undergraduate or first degree teaching including the means of designing curricula, evaluating teaching, assessing learning outcomes, and providing resources to quality assurance. References Becher, T. (1992). “Making Audit Acceptable: A Collegial Approach to Quality Assurance.” Higher Education Quarterly 46, 47-66. Brennan, J., De Vries, P. and Williams, R. (eds.) (1997). Standards and Quality in Higher Education. London: Jessica Kingsley. Brennan, J., Dill, D., Shah, T., Verkleij, A., and Westerheijden, D. (1999). A Campaign for Quality: Hong Kong Teaching and Learning Quality Process Review. Hong Kong: University Grants Committee. Clark, B. R. (1996). “Substantive Growth and Innovative Organization: New Categories for Higher Education Research.” Higher Education 32, 417-430. Clark, B. R. (1997). “The Modern Integration of Research Activities with Teaching and Learning.” Journal of Higher Education 68, 241-255. Clotfelter, C. T. (1996). Buying the Best: Cost Escalation in Elite Higher Education. Princeton: Princeton University Press. Cohen, J. M. (1995). “Capacity Building in the Public Sector: A Focused Framework for Analysis and Action.” International Review of Administrative Sciences 61, 407-422. Coopers & Lybrand (1993). Review of Quality Audit: Higher Education Quality Council. London: HEQC. Denzau, A. T, and North, D. C. (1994). “Shared Mental Models: Ideologies and Institutions.” Kylos 47, 3-31.
17
Dill, D. D. (1992). “Quality by Design: Toward a Framework for Academic Quality Management.” In J. Smart (ed.), Higher Education: Handbook of Theory and Research, vol. VII. New York: Agathon Press. Dill, D. D. (1997). “Higher Education Markets and Public Policy.” Higher Education Policy 10, 167-186. Dill, D. D. (1999a). “Academic Accountability and University Adaptation: The Architecture of an Academic Learning Organization.” Higher Education 38, 127-154. Dill, D. D. (1999b). “Student Learning and Academic Choice: The Rule of Coherence.” In J. Brennan, J. Fedrowitz, M. Huber and T. Shah (eds.), What Kind of University? International Perspectives on Knowledge, Participation and Governance. Buckingham: Open University Press and Society for Research into Higher Education. Dill, D. D. (2000) “Is There an Academic Audit in Your Future?: Reforming Quality Assurance in US Higher Education.” Change Magazine, 32(4): 34-41. Ewell, P. (1988). “Outcomes, Assessment, and Academic Improvement: In Search of Usable Knowledge.” In J. Smart (ed.), Higher Education Handbook of Theory and Research, vol. IV. New York: Agathon Press. Fairweather, J. (1996). Faculty Work and Public Trust: Restoring the Value of Teaching and Public Service in American Academic Life. Boston: Allyn & Bacon. Getz, M., and Siegfried, J. J. (1991). “Costs and Productivity in American Colleges and Universities.” In C. T. Clotfelter, R. G. Ehrenberg, M. Getz, and J. J. Siegfried, Economic Challenges in Higher Education. Chicago: University of Chicago Press. Goertz, M. E., Floden, R. E., and O’Day, J. (1995). Studies of Education Reform: Systemic Reform. New Brunswick, NJ: Consortium for Policy research in Education, Rutgers University. Gormley, W. T. Jr. and D. L. Weimer, D. L. (1999). Organizational Report Cards. Cambridge, MA: Harvard University Press. Grindle, M. S. (ed.) (1997a). Getting Good Government: Capacity Building in the Public Sector of Developing Countries. Cambridge, Mass.: Harvard University Press. Grindle, M. S. (1997b). “The Good Government Imperative: Human Resources, Organizations, and Institutions.” In M. S. Grindle (ed.) Getting Good Government: Capacity Building in the Public Sector of Developing Countries. Cambridge, Mass.: Harvard University Press. Harcleroad, F. (1976). Educational Auditing and Accountability. Washington DC: The Council on Postsecondary Accreditation. Higher Education Quality Council (HEQC) (1993). Notes for the Guidance of Auditors. London: HEQC. Higher Education Quality Council (HEQC) (1994). Learning from Audit. London: HEQC. Honadle, B. W. (1986). “Defining and Doing Capacity Building: Perspectives and Experiences.” In B. W. Honadle and A. M. Howitt (eds.), Perspectives on Management Capacity Building. Albany, NY: State University of New York Press, 1986. Honadle, B. W. and Howitt, A. M. (eds.) (1986). Perspectives on Management Capacity Building. Albany, NY: State University of New York Press. Horn, M. J. (1995). The Political Economy of Public Administration: Institutional Choice in the Public Sector. Cambridge: Cambridge University Press. Jacobs, B. and Weimer, D. L. (1986). “Inducing Capacity Building: The Role of the External Change Agent.” In B. W. Honadle and A. M. Howitt (eds.) Perspectives on Management Capacity Building. Albany, NY: State University of New York Press 18
James. E. (1978). “Product Mix and Cost Disaggregation: A Reinterpretation of the Economics of Higher Education.” Journal of Human Resources 13, 157-186. James. E. (1986). “Cross Subsidization in Higher Education: Does it Pervert Private Choice and Public Policy?” In D. Levy (ed.), Private Education: Studies in Choice and Public Policy. New York: Oxford University Press. James. E. (1990). “Decision Processes and Priorities in Higher Education.” In . S. A. Hoenack and E. L. Collins (eds.), The Economics of American Universities: Management, Operations, and Fiscal Environment. Albany, NY: State University of New York Press. James. E. And Neuberger, E. (1981). “The University Department as a Nonprofit Labor Cooperative.” Public Choice 36, 585-612. Kettl, D. (1997). “The Global Revolution in Public Management: Driving Themes, Missing Links.” Journal of Public Policy Analysis and Management 16, 446-462. Lattuca, L. R. and Stark, J. S. (1994). “Will Disciplinary Perspectives Impede Curricular Reform?” Journal of Higher Education 65, 401-426. Massy, W. F. (1996). “Productivity Issues in Higher Education.” In W. F. Massy (ed.) Resource Allocation in Higher Education. Ann Arbor: University of Michigan Press. Massy, W. F. (1997a). “Teaching and Learning Quality-Process Review: The Hong Kong Programme.” Quality in Higher Education 3, 249-262. Massy, W. F. (1997b). “Life on the Wired Campus: How Information Technology Will Shape Institutional Futures.” In D. C. Oblinger and S. C. Rush (eds.), The Learning Revolution. Bolton, MA: Anker Publishing Company. Massy, W. F. and French, N. (1997). “Teaching and Learning Quality Process Review: A Review of the Hong Kong Programme.” Paper presented at the INQAAHE Conference, Berg-en-Dl, South Africa, May. Massy, W. F. and French, N. (1999). “Teaching and Learning Quality Process Review: What Has the Programme Achieved in Hong Kong.” Paper Presented at the International Network for Quality Assurance Agencies in Higher Education (INQAAHE) Meeting, Santiago, Chile, 2-5 May. Massy, W. F., French, N. G., and Thompson, Q. (1999). “Management Reviews: An Outline of the Hong Kong Program.” Paper Presented at the International Network for Quality Assurance Agencies in Higher Education (INQAAHE) Meeting, Santiago, Chile, 2-5 May. Massy, W. F., Wilger, A. K., and Colbeck, C. (1994). “Overcoming Hollowed Collegiality.” Change 26, 10-20. McDonald, L. M. and Elmore, R. F. (1987). “Getting the Job Done: Alternative policy Instruments.” Educational Evaluation and Policy Analysis 9, 133-152. Meade, P. and Woodhouse, D. (2000) “Evaluating the Effectiveness of the New Zealand Academic audit Unit: Review and Outcomes.” Quality in Higher Education 6, 19-30. National Agency for Higher Education (1996). The National Quality Audit of Higher Education in Sweden. Stockholm: National Agency for Higher Education. Nilsson, K.-A. and Wahlen, S. (2000). “Institutional Response to the Swedish Model of Quality Assurance”. Quality in Higher Education 6, 7-18. North, D. (1990). Institutions, Institutional Change, and Economic Performance. Cambridge: Cambridge University Press. Östling, M. (1997). “Self-Evaluation for Audit – A Tool for Institutional Improvement?” Paper presented at the INQAAHE Conference, Berg-en-Dl, South Africa, May. 19
Pascarella, E. T. and Terenzini, P. T. (1991). How College Affects Students: Findings and Insights From Twenty Years of Research. San Francisco: Jossey-Bass. Quality Assurance Agency for Higher Education (1998). Quality Assurance in UK Higher Education: A Brief Guide. Gloucester, UK: Quality Assurance Agency for Higher Education. Rosovsky, H. and Ameer, I.-L. (1998). “A Neglected Topic: Professional Conduct of College and University Teachers.” In W. G. Bowen and H. T. Shapiro (eds.), Universities and Their Leadership. Princeton: Princeton University Press. Scheele, J. P., Maassen, P. A. M. and Westerheijden, D. F. (eds.) (1998). To Be Continued Follow-up of Quality Assurance in Higher Education. Maarssen, The Netherlands: Elsevier/de Tijdstoom. Shulman, L. S. (1993). “Teaching as Community Property: Putting an End to Pedagogical Solitude.” Change 25: 6-7. Stensaker, B. (1999). “External Quality Auditing in Sweden: Are Departments Affected?” Higher Education Quarterly 53, 353-368. Teacher Education Accreditation Council (TEAC) (1998). Prospectus. Washington DC: TEAC. University Grants Committee (1997). “Report on TLQPR Seminar Held on 8 April 1997.” Hong Kong: University Grants Committee. Williams, G. L. (1995). “The “Marketization” of Higher Education: Reforms and Potential Reforms in Higher Education Finance.” In D. D. Dill and B. Sporn (eds.), Emerging Patterns of Social Demand and University Reform: Through a Glass Darkly. Oxford: Pergamon Press. Williams, P. (1992). Annual Report of the Director 1990/91: Academic Audit Unit. London: CVCP. Woodhouse, D. (1995a). Audit Manual: Handbook for Institutions and Members of Audit Panels. Wellington, New Zealand: Universities Academic Audit Unit. Woodhouse, D. (1995b). Institutional Responses to Quality Assessment: Qualifications and Quality in New Zealand. Paris: OECD. Woodhouse, D. (1998). “Auditing Research and the Research/Teaching Nexus.” New Zealand Journal of Educational Studies 33, 39-53.
Acknowledgements This article is a revised version of a paper presented at the annual meeting of the Association for Public Policy Analysis and Management (APPAM), Washington DC, November 1999. The paper was developed with the support of a grant from the Pew Charitable Trusts. I wish to give my sincerest thanks for useful criticism to David Breneman, Iris Geva-May, and Brian Kropp as well as to a large number of people involved with academic audit in the listed countries and agencies who provided valuable information and advice, most particularly to Nigel French, William Massy, Lars Niklasson, Malin Östling, Bjorn Stensaker, Staffan Wahlen, Peter Williams, and David Woodhouse. Notes i
The article focuses on academic audit activities in the UK, New Zealand, Sweden and Hong Kong because each of these countries has completed a full cycle of academic audits of their university sector and an evaluation of the process has been conducted. The analysis is based upon interviews and correspondence with administrators in the
20
various audit agencies as well as participation in an external evaluation of the Hong Kong TLQPR program. In addition the following sources were utilized: Hong Kong (Massy, 1997a; Massy and French 1997; UGC, 1997); New Zealand (Woodhouse, 1995a, 1995b); Sweden (National Agency for Higher Education, 1996; Östling, 1997); United Kingdom (Williams, 1992). ii An early advocate of the term “educational auditing” in the US was Fred Harcleroad, who suggested a model for reforming US academic accreditation that was drawn from the financial auditing system developed over time by the Securities and Exchange Commission (Harcleroad, 1976). iii The Academic Audit Unit (AAU) was established by the Committee of Vice Chancellors and Principals (CVCP) in 1990. In 1992, with the passage of the Higher and Further Education Act, UK higher education was substantially reorganized and Academic Audit was assigned to a new intermediary agency, The Higher Education Quality Council. In addition, Higher Education Funding Councils were created for England, Scotland, and Wales, part of whose responsibility was to initiate a separate system of assessments of teaching quality at the subject level in all UK universities. Following the publication of the Dearing Report on Higher Education in 1998, the recently elected Labor Government endorsed the creation of a new UK-wide Quality Assurance Agency (QAA), which now has responsibility for completing the cycles of the two preexisting quality processes as well as designing a new overall process. The analysis in this article focuses on the first cycle of Academic Audit as it was originally implemented under the CVCP. iv Many would argue that teaching and research, particularly in the university sector, are joint products -- faculty time spent on research can improve the quality of academic content all students learn (Clark, 1997). However as Massy (1996) notes, a faculty members’ commitment to the quality of teaching and the quality of research logically requires a tradeoff at some point. If a faculty member’s time commitment to research, unconstrained by institutional structures, increases beyond this point, it becomes a substitute for time committed to teaching. The quality of student learning inevitably will be affected negatively by this decline in the time faculty members can devote to assessing students, to the design of new curricula and teaching methods, and to teaching itself. v
Commenting on the contribution that disciplinary fragmentation makes to the complexity of higher education systems, Clark (1996) observed: in mathematics, 200,000 new theorems are published each year, periodicals exceed 1,000, and review journals have developed classification schemes that include over 4,500 subtopics arranged under 62 major topic areas. In history, the output of literature in the two decades of 1960-1980 was apparently equal in magnitude to all that was published from the time of the Greek historian Thucydides in the fourth century B.C. to the year 1960. In psychology, 45 major specialties appear in the structure of the American Psychological Association, and one of these specialties, social psychology, reports that it is now comprised of 17 subfields....In the mid1990s, those who track the field of chemistry were reporting that ‘more articles on chemistry have been published in the past 2 years than throughout history before 1900.’ Chemical Abstracts took 31 years to publish its first million abstracts, 18 years for its second million, and less than 2 years for its most recent million. An exponential growth of about 4 to 8 percent annually, with a doubling period of 10 to 15 years, is now seen as characteristic of most branches of science (pp. 421-422). vi
Similarly, the first cycle of academic audit in the UK discovered serious erosions in the well-established university process of external examining. The audits revealed substantial variations in examiner appointment and procedure across subject fields, ineffective monitoring by the universities, and a frequent failure to utilize examiners’ reports for educational improvement (HEQC, 1994). vii The relationship between academic freedom and the collective faculty responsibility for teaching and learning is thoughtfully explored in the US context by Rosovsky and Ameer (1998). viii For examples of the types of policy changes and structural adaptations made by universities participating in audit processes, see Dill (1999).
21
22
