Accountability and Quality Improvement: Examining ...

2018‐06‐08

Accountability and Quality Improvement: Examining the Implementation of Outcomes-based Quality Assurance Mechanisms in Ontario Postsecondary Education Qin Liu Ph.D. Researcher, University of Toronto Presentation at the CSSHE conference, Regina, May 29, 2018

This presentation • The tension between accountability and improvement • The quality assurance (QA) system for Ontario postsecondary education • Conceptualize the process of implementing QA mechanisms • Case studies of four processes at two universities and two colleges in Ontario

1

2018‐06‐08

Literature review and the problem Two views about the relationship between accountability and improvement in the context of quality assurance • Polarizing the two (Massy, 1997; Middlehurst & Woodhouse, 1995; Thune, 1996; Vroeijenstijn, 1995a, 1995b; Stensaker, 2003): mutually exclusive and independent of each other • Reconciling between the two (Dano & Stensaker, 2007; Genis, 2002; Stensaker, 2003): EQA can stimulate institutional improvement

Literature review (cont’d) Institutional strategies for managing the tension (Brennan et al., 1997; Dano & Stensaker, 2007; Newton, 2000; Smith & MacGregor, 2009; Stensaker, 2003; Thune, 1996; Weusthof, 1995):

• Self-regulation • Leadership • Organizational structures

2

2018‐06‐08

Research questions • What was the relationship between accountability and quality improvement in the processes of implementing quality assurance mechanisms at Ontario postsecondary institutions? • What strategies were used to manage the tension between accountability and quality improvement?

Outcomes-based quality assurance frameworks for Ontario PSE QA frameworks

Outcomes standards

The Quality Assurance Framework (QAF) for the university sector

The Degree Level Expectations

The Ontario College Quality Assurance Service (OCQAS) framework for the college sector

The Credentials Framework and provincial program standards

The Postsecondary Education Quality Assessment Board (PEQAB) framework for degree programs offered outside Ontario public universities

The degree standard in the Ontario Qualifications Framework

3

2018‐06‐08

Conceptualizing the process of implementing QA mechanisms Systemwide QA

Generative mechanisms

Institutional perspectives

Event

Changes

Individual perspectives

Individual actions

Departmental perspectives Individual perspectives

Case studies Within the university sector • University A: Technology-supported curriculum mapping process • University B: Implementing Senate-approved university learning outcomes Within the college sector • College A: New program development process • College B: Program review process

4

2018‐06‐08

Data collection Cases

Descriptions

University A

Technology-supported curriculum mapping process at a university

University B

Senate-approved University Learning Outcomes of a university

9

8

6

College A

New program development process at a college

4

2

6

College B

Program review process at a college

6

3

6

21

17

23

Total

Interviewees Related Academic Faculty Documents administrators members 2 4 5



The two university cases University A

University B

• Used a web-based curriculum mapping tool

• Used learning objectives as an institutional practice

• To ensure the alignment of programs and courses with the UDLEs

• Developed the University Learning Outcomes (ULOs) • Aligned programs and courses with ULOs

5

2018‐06‐08

The two universities: Institutional perspectives Commonalities: • Championship • Institutional efforts to harmonize accountability and quality improvement Different approaches to handling external pressure • University A: a reactive approach • University B: a pre-emptive approach

The two universities: Departmental perspectives University A It was recognized at the departmental level that the curriculum mapping process served the purposes of accountability and improvement. Challenges in the implementation process

University B Many departments took a ground-up approach to learning outcomes development while some also used accountability as a rationale or motive for moving things forward. Resistance in some departments

6

2018‐06‐08

The two universities: Individual perspectives and actions • Personal beliefs of about the use of the outcomes-based approach played a significant role in championship • Different effects of the accountability-focused versus improvement-focused view of implementing the initiative

The two college cases College A

College B

• Implemented a two-phase new program development process

• Implemented an eight-event program review process

• Streamlined from the previous process

• A major revamp since the previous system-mandated review

• Built upon a long-standing institutional practice

7

2018‐06‐08

Comparing the two college cases New program development at College A

Program review at College B

Institutional perspectives

Both demonstrated efforts to harmonize accountability and quality improvement The teaching and learning centre played an instrumental role in implementation

Departmental perspectives

Benefits of scaffolding Challenges in meeting some degree standards

Benefits for accredited and non-accredited programs Challenges in workload and time management

Individual perspectives and actions

The accountability mechanism was seen as an important step toward quality improvement

Desire for continuous improvement

Accountability and improvement in implementing QA mechanisms Home-grown institutional processes became a bridge to connect accountability and improvement. These processes were not free of challenges. • Two universities: confusion, resistance • Two colleges: workload, time management, meeting degree standards

8

2018‐06‐08

Strategies for managing the tension • Self-regulation within the system-wide QA context • Leadership and championship • Teaching and learning centres • Communication strategies: improvement > accountability • Different approaches: reactive versus pre-emptive

Thank you. Questions?

Qin Liu: [email protected]

9

2018‐06‐08

References Brennan, J., Frederiks, M., & Shah, T. (1997). Improving the Quality of Education: The impact of quality assessment on institutions. Bristol: Higher Education Funding Council for England. Danø, T. & Stensaker, B. (2007). Still balancing improvement and accountability? Developments in external quality assurance in the Nordic countries 1996–2006. Quality in Higher Education, 13(1), 81–93. Genis, E. (2002). A perspective on tensions between external quality assurance requirements and institutional quality assurance development: A case study. Quality in Higher Education, 8(1), 63-70. Massy, W. F. (1997). Teaching and learning quality-process review: The Hong Kong programme. Quality in Higher Education, 3(3), 249–262. Middlehurst, R., & Woodhouse, D. (1995) Coherent Systems for External Quality Assurance, Quality in Higher Education, 1(3), 257-268. Newton, J. (2000). Feeding the beast or improving quality? Academics’ perceptions of quality assurance and quality monitoring. Quality in Higher Education, 6(2), 154-163.

References (cont’d) Smith, B. L., & MacGregor, J. (2009). Learning communities and the quest for quality. Quality Assurance in Education, 17, 118-139. Stensaker, B. (2003). Trance, transparency and transformation: The impact of external quality monitoring in higher education. Quality in Higher Education, 9, 151–9. Thune, C. (1996). The alliance of accountability and improvement: The Danish experience. Quality in Higher Education, 2, 21–32. Vroeijenstijn, A.I. (1995a) Improvement and accountability: Navigating between Scylla and Charybdis. Guide for external quality assessment in higher education. London: Jessica Kingsley Publishers Ltd. Vroeijenstijn, A.I. (1995b) Governments and university: Opponents or allies in quality assurance? Higher Education Review, 27(3), 18-36. Weusthof, P.J.M. (1995). Dutch universities: an empirical analysis of characteristics and results of self-evaluation. Quality in Higher Education, 1(3), 235–248

10