Square Pegs in Round Holes - SAGE Journals - Sage Publications

Square Pegs in Round Holes The Relative Importance of Traditional and Nontraditional Scholarship in Canadian Universities

Science Communication Volume 28 Number 4 June 2007 501-518 © 2007 Sage Publications 10.1177/1075547007302213 http://scx.sagepub.com hosted at http://online.sagepub.com

Marie-Rose Phaneuf University of Ottawa, Ontario, Canada

Jonathan Lomas Chris McCutcheon Canadian Health Services Research Foundation, Ottawa, Canada

John Church Douglas Wilson University of Alberta, Edmonton, Canada

In May 2003 all university faculties of medicine, health sciences, nursing, business, and public administration in Canada were surveyed to document whether their promotion practices gave equal treatment to researchers with traditional disciplinary-based scholarly outputs versus those with nontraditional outputs flowing from an applied interest in interdisciplinary research, collaboration with nonacademics, and knowledge transfer. Forty-seven deans (response rate ⫽67 percent) and thirty-two promotion committee members (response rate ⫽51 percent) consistently rated research as more important than teaching and community service in promotion proceedings. Furthermore, in their considerations of research they accorded significantly more value in promotion to traditional than to nontraditional scholarly outputs. Among the deans, for example, the top ten ranked activities included seven traditional research outputs, two teaching-related outputs, and only one nontraditional research output. Scholars with nontraditional outputs were slightly less disadvantaged in health-related faculties compared to the nonhealth faculties. Despite broader acceptance of an evolving research landscape in Canadian universities, current reward structures may create barriers for academics interested in conducting nontraditional research characteristic of the emerging paradigm of applied scholarship.

Keywords: faculty rewards; survey; scholarship; applied research

501

502

Science Communication

F

or decades, schools of academic medicine have grappled with how to reward faculty whose academic activities do not easily fit into existing criteria for tenure and promotion. Mandated to achieve excellence in research, teaching, and patient care, these schools must recruit beyond research-focused faculty. The roles of clinician–educator and clinician– researcher are essential to meeting this tripartite mandate and have been recognized with the development of distinct career pathways in certain schools (Howell and Bertakis 2004). Nevertheless, a continuing and very real challenge is how to give recognition to these contributions through a reward system that strongly favors classic research granting and academic peer-reviewed publication. At the same time, a new thrust towards applied health research, different in many ways from traditional research, is emerging in the health sciences and related disciplines. Otherwise stated, a new paradigm of scholarship is rubbing against the grain of a paradigm already entrenched in university tradition and policy. This new paradigm, which emphasizes applied more than discoveryoriented research, brings with it at least three novel practices. The first is interdisciplinarity (Giacomini 2004). Complex health problems have multiple, interacting determinants, requiring varied perspectives and methods. Second, the very task of identifying problems has expanded collaboration to include nonacademic experts as equal partners in the research process. Clinical (patient-based), community, and health services researchers regularly partner with the potential users of their research to heighten relevance and open pathways for application (Denis and Lomas 2003). Third, since addressing a health problem really means getting research used by decisionmakers—clinicians, managers, policymakers, and the public— spending time in the transfer or translation of knowledge is now central to the applied research process (Canadian Health Services Research Foundation 2004). Canada and its health research funding agencies have been at the forefront of these changes. For instance, the federal government established the Canadian Health Services Research Foundation in 1997 with an explicit mandate to engage in applied health services research and knowledge Authors’ Note: Correspondence concerning this article should be addressed to Chris McCutcheon, Research Programs, Canadian Health Services Research Foundation, 1565 Carling Avenue, Suite 700, Ottawa, Ontario K1Z 8R1; phone (613) 728-2238; e-mail: [email protected].

Phaneuf et al. / Scholarship in Canadian Universities

503

transfer. This was followed in 2000 with the creation of the Canadian Institutes of Health Research (CIHR). Whereas previously under the Medical Research Council and related agencies funding was roughly segregated along disciplinary lines, CIHR’s mandate is to unite all health research under a single funding umbrella, with thirteen institutes acting as a virtual network of researchers focused on national health priorities instead of disciplines. In addition to promoting interdisciplinarity, it aims to translate knowledge “into improved health for Canadians, more effective health services and products and a strengthened Canadian health care system” (Canadian Institutes of Health Research 2004). Most recently, the Social Sciences and Humanities Research Council (2004) has embarked on a transformation “from granting council to knowledge council.” For Canadian health researchers based in universities these changes in scholarly practice, as graphically reflected in the transformed funding agencies, create new opportunities and a different set of pressures. The incentives of funders to encourage researchers to engage in novel and nontraditional scholarly practice have not been complemented at the university level by widespread reform of the tenure and promotion review process. This is a source of some anxiety for funding agencies and researchers alike because it is a potential major obstacle to achieving the new mandates and meeting changing expectations. The perception of young researchers appears to be that the new scholarly practice equals extra, unrecognized, and career-impeding work (Giacomini 2004; Jacobson, Butterill, and Goering 2004). This problem is not unique to Canada. The poor match between reward systems and actual faculty work has been well recognized for a number of years and across a number of countries (American Council on Education, American Association of University Professors, and United Educators 2000; Association of Universities and Colleges of Canada 1997; CommunityCampus Partnerships for Health 2005; Court 1998). The most influential critic is Ernest Boyer, who in his Scholarship Reconsidered (1990) does away with the promotion triad and links all the activities of a faculty member to a single enterprise—scholarship. His argument is that a broad range of “academic” activities demands the same training and expertise traditionally attributed only to research. Unfortunately, although there has been much hand-wringing and problem identification, very little empirical work has been done in this area. How, exactly, do traditional faculty reward systems treat nontraditional

504


practices such as outreach and knowledge transfer? Layzell (1999, 3–37) describes three categories of research on faculty work: 1. 2. 3.

faculty activity studies, documenting the number of hours spent on teaching, research, and community service; instructional workload analyses, focusing on teaching course loads and contact hours; and measures of noninstructional activity, enumerating scholarly output by quantity of articles published and cited.

Such inventories, however, do not tell us much about the merit accorded to different scholarly activities. The privileging of peer-reviewed journal publication over other forms of scholarly production is often highlighted as one barrier among many in the knowledge transfer literature (Crosswaite and Curtice 1994; Henke 2000). However, as is noted by Jacobson, Butterill, and Goering (2004), this literature “often presents information about such barriers anecdotally, in the context of research dealing with theory and practice” (p. 248). The research that has been done tells us to expect traditional measures of research excellence to heavily outweigh other measures in faculty reward systems. The ascendancy of research in the development of North American universities is well documented (Glassick, Huber, and Maeroff 1997). Surveys in medical schools repeatedly point to the precedence of peerreviewed publications (Jones and Froom 1994). Our study surveyed the promotion practices in all the faculties of medicine, nursing, and health sciences in Canadian universities, as well as in two other professional faculties—business and public administration. The purpose was to document whether promotion practices disadvantaged applied researchers working in professional schools, specifically those interested in interdisciplinary research, collaboration with nonacademics, and knowledge transfer. Our conjecture was that these professional faculties, where applied research is more widely practiced than elsewhere in the university, were the most likely to have overcome any disadvantage that may be present in the promotion process for those engaged in applied research. Thus, if we found that attitudes toward promotion favored traditional over nontraditional scholarship in these applied faculties, it was likely present in most other university faculties. We were also interested in whether the situation was better or worse for those researchers in health-related professional faculties versus those in nonhealth professional faculties, as represented by business and public administration. In order to meet this


505

objective, we looked at the relative value accorded to both traditional and nontraditional research outputs, as well as teaching and community service, in all these faculties.

Methods Between the months of February and May 2003, we mailed a two-part survey to the dean of every faculty of medicine, nursing, health sciences, business, and public administration in Canada. We also surveyed a member of a departmental or faculty promotion committee in these faculties. The study received ethics approval from the University of Alberta Health Research Ethics Board.

Sample The annual directory of the Association of Universities and Colleges of Canada (AUCC) was used to identify the faculties and their deans. The full population of relevant faculties in the country (n ⫽70), not just a sample, was the focus of the study. Using contact with the offices of these deans, we constructed a second sample of the individual promotion committee members in these faculties. When possible, the chair of the promotion committee was surveyed. In cases where no chair could be identified, we sought another individual with experience on the promotion committee—a current committee member or past chair. This proved challenging because of the variability in structures and processes across universities, but was eventually achieved for sixty-three of the seventy faculties under study.

Instrument Development The survey instrument was developed with two sections: a twentyseven-item inventory of activities and outputs that respondents rated for their importance to the promotion process, and a scenario question that compared the relative promotion chances for a scholar with nontraditional versus one with traditional outputs. The deans’ and the promotion committee members’ surveys were identical in all ways but one: the deans were asked to respond on the basis of the written promotion criteria (theory), whereas promotion committee members were asked to respond on the basis of how these written criteria were actually used in the assessment of individual cases for promotion (practice).

506


The twenty-seven outputs or activities in section one of the survey were each rated for their importance to the promotion process on a scale of 1 to 5 (not important, somewhat important, important, very important, extremely important), with deans considering their faculty’s written criteria for promotion from assistant to associate professor and promotion committee members the actual practice. Several inventory items were taken from an existing survey instrument (Conine, McPherson-Shilling, and Pierce 1985) and others were drafted based on the types of activities described in the literature under the headings of teaching, research, and community service. Finally, items were added that reflected the nontraditional forms of scholarship now expected by the major Canadian funding agencies described above— interdisciplinary research, collaboration with nonacademics, and knowledge transfer. The entire instrument was then piloted with six individuals knowledgeable about the promotion process in universities. Final modifications were made based on feedback from this pilot. Two of the authors (Phaneuf and Lomas) independently classified the twenty-seven inventory items into four categories: traditional research outputs, nontraditional research outputs, teaching, and community service. Immediate agreement on categorization was present for twenty-three items and a short discussion between the two reviewers resolved the discrepancy on the remaining four. Table 1 contains the list of eight traditional and eight nontraditional research items in full and Figure 1 lists all twenty-seven items in abbreviated form. Section two of the survey was the same scenario question for all participants and was designed to contrast how faculty members with an emphasis on traditional scholarship versus those with a nontraditional emphasis would fare under their faculty’s current system for promotion from assistant to associate professor. Respondents were asked to state whether faculty member A or faculty member B had a greater chance of being promoted, or whether they had an equal chance. The specific scenario presented was: Imagine two faculty members who have equivalent administrative and teaching loads and have made equivalent contributions to community service. Faculty member A conducts research aimed at building theory and developing knowledge in a particular discipline. This researcher’s output is directed largely at disciplinary peers. Faculty member B conducts research in a disciplinary area, but has made major additional commitments to seeing this research disseminated to and


507

Table 1 Traditional and Nontraditional Research Items Traditional Research 1. Publishing a frequently cited article in a peer-reviewed journal 2. Having strong letters of reference from recognized experts in your field 3. Receipt of a personnel award or career support from a federal or provincial peerreview funding agency 4. Being the first author of an article in a high-impact peer-reviewed journal 5. Receipt of a research grant from a federal or provincial peer-review funding agency 6. Being invited to present a scholarly paper at an academic conference 7. Being the first author of any kind of journal article 8. Organizing an international conference for other scholars Nontraditional Research 1. Producing a plain language document for the public or for decisionmakers outside the university 2. Producing a report specifically designed to influence public policy 3. Producing a commissioned research synthesis document 4. Creating and maintaining organizational structures that link the academy to the community in innovative partnerships 5. Facilitating interaction between researchers and potential users of research (e.g., workshops, briefings, conferences) 6. Having one’s contribution and impact formally recognized by public policy and management decisionmakers or their equivalent outside the university (e.g., unsolicited letters, awards, or appointment to an advisory committee) 7. Receipt of a research grant or personnel support from a peer-review funding agency requiring decisionmaker partnerships, and clear dissemination strategies 8. Organizing an international conference for practitioners and scholars

used by individuals outside the academy—for example, to practitioners or policymakers. Faculty members A and B have comparable productivity, but the nature of their outputs is different.

Survey Procedure The survey was sent by mail to all individuals in the sample. The initial mailing contained the survey, an information letter about the study, and a stamped return envelope. Three e-mail follow-ups were conducted. The first follow-up took place three weeks after the initial mailing. The second follow-up was eight weeks after initial mailing and a final e-mail follow-up was sent at twelve weeks. Every e-mail contained a copy of the information letter and survey instrument.

508


Figure 1 Mean Importance Ratings by Promotion Committee Members and Deans for Inventory Items 4.28 4.33

* First author of an article in a high-impact journal

4.28 4.25

* Research grant from a peer-review funding agency

3.80 4.13

* Strong letters of reference from experts in your field

4.20 4.06

* Frequently-cited article in a peer-review journal Research grant requiring decision-maker partnerships

3.72 3.91

Consistently favourable evaluation for teaching

3.96 3.81 3.54 3.81

* Career support from peer-review funding agency

3.43 3.68

* Being first author of any kind of journal article

3.58 3.55

Teaching an undergraduate course

3.09 3.25

External recognition for teaching/education excellence

3.57

* Invitation to present at an academic conference

3.09

Leadership in national/international organizations

2.98 3.00

Teaching a graduate course

2.84 2.94 3.04 2.91

Recognition by public policy decision-makers International conference for practitioners & scholars

2.70 2.78

Supervising a doctoral student

2.57 2.72

* Organizing an international scholarly conference

2.54 2.72

Promotion Committee Members

3.02 2.69

Leadership & membership on university committees

2.65 2.69

Commissioned research synthesis document Creating/maintaining linkages to the community

2.78 2.59

Student counselling and course co-ordinating

2.76 2.56

Supervise a master's student

2.74 2.55

Facilitating researcher/potential-user interactions

2.52

Service to governmental or public councils, boards

2.53 2.41

Administrative office in the department

2.46 2.28

Plain language document for public/decision-makers

2.43 2.28

2.93

2.72

Reports designed to influence public policy

2.25 0

* Traditional research items Non-traditional research items

Deans

1

2

3

4

5

Mean Importance (1 = not important, 2 = somewhat important, 3 = important, 4 = very important, 5 = extremely important)


509

Analysis and Results Respondent Characteristics and Response Rates Table 2 provides the numbers and response rates for deans and promotion committee members in total and across the health-related and nonhealth faculties. The most likely reason for the lower response rate in the promotion committee group (50.9 percent) versus the deans (67.1 percent) is that of the sixty-three survey requests sent to promotion committee members only thirty-five were mail surveys and the remaining twenty-eight were requests for a phone interview. All seventy of the requests sent to deans were mail surveys. The initial design of the study included a telephone administration of the inventory items and the scenario for half the promotion committee chairs in order to administer some additional qualitative questions on the political climate surrounding the issue. Difficulty in arranging such telephone interviews led us to abandon this part of the study. The inventory item and scenario responses from the seven telephone interviews that were conducted were included in the promotion committee sample. The three unknown respondents comprised two from the promotion committee sample where their faculties were not part of the original sample frame1 and one who returned the dean’s survey with the numerical code designating faculty type removed. With the exception of the overall analysis, these were considered as missing cases.

Psychometric Properties of the Inventory Items and Ratings As a further means of ensuring that the two reviewers’ categorization of inventory items was reliable, an alpha test was performed to measure the Table 2 Respondent Numbers and Response Rates Health-Related Faculties Deans Promotion committee members Total Response rate (%)

Nonhealth Faculties

Unknown

Total

Response Rate (%)

20

26

1

47

67.1

15 35

15 41

2 3

32 79

50.7 —

57.3

56.9

—

—

510


degree of internal consistency between traditional and nontraditional survey items. For the traditional research grouping, the alpha test revealed a .75 measure of reliability. For the nontraditional grouping, the reliability score was .89. This indicates there was a considerable degree of internal consistency for items in the traditional versus nontraditional groupings. The twenty-seven inventory items generated considerable spread in ratings by respondents (indicating good use of the scale), with a range from 2.25 at the low end (for promotion committee members’ rating of importance for producing a report designed to influence public policy) to 4.33 at the high end (also for promotion committee members, this time for their rating of being the first author of an article in a high impact journal). In the view of promotion committee members, therefore, being the first author of an article in a high impact journal falls in between very important and extremely important, whereas producing a report intended to influence public policy is only slightly better than somewhat important for promotion. Primary determinants of promotion from assistant to associate professor: Inventory items. A paired t-test comparing the mean ratings of importance for traditional versus nontraditional research outputs in determining promotion revealed a statistically significant difference for both the deans’ ratings against written criteria (mean ratings ⫽3.71 versus 2.88; alpha level ⫽.001, n ⫽46, t ⫽11.475, p ⬍.000, 99.9 percent C.I. 0.58 –1.09), and the promotion committee members’ ratings against actual practice (mean ratings ⫽3.76 versus 2.74; alpha level ⫽.001, n ⫽32, t ⫽10.419, p ⬍.000, 99.9 percent C.I. 0.66 –1.37)—see Table 3. Mean importance ratings on the specific inventory items by the deans and promotion committee members are presented in Figure 1. For both the deans (who were reporting on written criteria) and promotion committee members (who were reporting on actual practice), items related to publication in peer-reviewed journals and receipt of peer-reviewed research grants were given the highest measure of importance for achieving promotion. Items related to teaching and community service filled the mid range for importance. The lowest mean ratings were given to such nontraditional scholarship items as research in collaboration with nonacademics or producing a plain language publication for the public. Though deans and promotion committee members rated items similarly, the nontraditional items did not cluster as tightly at the bottom of the deans’ list. For deans, at least some of the nontraditional items appear to be intertwined with the midranking community service items.


511

Table 3 Respondents’ Mean Importance Ratings for Traditional versus Nontraditional Scholarship Items on the Inventory (1 ⴝ Not Important; 5 ⴝ Extremely Important) Traditional

Nontraditional

Deans (written criteria)

3.71

2.88

Promotion committee members (criteria in practice)

3.76

2.74

Health-related faculties

3.86

2.99

Nonhealth faculties

3.60

2.66

Paired t-Test Significance alpha level ⫽.001, n ⫽46, t ⫽11.475, p ⬍.000, 99.9% C.I. 0.58–1.09 alpha level ⫽.001, n ⫽32, t ⫽10.419, p ⬍.000, 99.9% C.I. 0.66–1.37 alpha level ⫽.001, n ⫽35, t ⫽10.783, p ⬍.000, 99.9% C.I. 0.61–1.24 alpha level ⫽.001, n ⫽41, t ⫽11.597, p ⬍.000, 99.9% C.I. 0.71–1.33

A literature-driven hypothesis that nontraditional research outputs fare better according to written criteria than in actual practice was tested by comparing the deans’ and promotion committee members’ inventory ratings in an independent sample t-test; the difference between the two groups did not achieve significance at the 0.05 alpha level. This similarity in responses for deans and promotion committee members led us to combine their responses for the analysis of health-related versus nonhealth faculties. This was done to minimize the occurrence of cells with small numbers for our statistical testing of differences. Table 3 summarizes the results from these statistical tests, along with those from the deans and the promotion committee members. Healthrelated faculties appear to give somewhat higher overall ratings compared to the nonhealth faculties, but other than that their ratings show the same kind of significant differences in favor of traditional research outputs as the nonhealth faculties. For health-related faculties the mean ratings of importance for traditional versus nontraditional inventory items were

512


significantly different at 3.86 versus 2.99; the same significantly different mean ratings for nonhealth faculties were 3.60 versus 2.66. Promotion from assistant to associate professor: Scenario responses. Respondents were asked to compare how two candidates for promotion from assistant to associate professor—equal in all respects except the traditional versus nontraditional research outputs—would fare under their faculty’s current promotion system. Figure 2 presents the results for both the deans and the promotion committee members. The dominant response to this question was that both candidates would have an equal chance of being promoted. This is in contrast to the significantly greater importance accorded to the individual inventory items related to the traditional researcher’s output. However, for those respondents who chose one candidate over the other, results were in line with the responses to the inventory items where high importance was given to peer-reviewed publication and grants. The promotion candidate with Figure 2 Judgements of Deans and Promotion Committee Members of the Relative Chances of Promotion for Faculty with Traditional versus Nontraditional Research Outputs Deans

30% 56% 14%

Promotion Committee Members

28%

66%

6%

Traditional fares better than nontraditional Nontraditional fares better than traditional Both have an equal chance


513

traditional scholarly output was judged to be more likely to be promoted than the one with nontraditional outputs—twice as likely according to the deans and nearly five times as likely according to the promotion committee members. The Fisher’s exact test was performed in lieu of a chi-square analysis to determine if health-related faculties versus nonhealth faculties were statistically different on the scenario question. (The Fisher’s exact test is sensitive to small counts in certain cells, which was the case in this analysis where the differences between deans and promotion committee chairs meant that their responses could not be combined for the comparison of faculty type.) There was a significant difference ( p ⬍ .016) between the groups. Although the faculty member with traditional output was favored over the one with nontraditional output more often by both groups of faculties, the respondents in nonhealth faculties indicated that the traditional candidate had a greater chance of promotion significantly more often than those in the health-related faculties.

Discussion First, the detailed inventory item results in Figure 1 show that research is greatly favored over teaching and community service in promotion reviews at Canadian universities. Although claims have been made to this effect before, this empirical demonstration confirms the prior assertions in the tenure and promotion literature. Of course, our interests lie more in how the new scholarly practices of applied researchers—interdisciplinary research, collaboration with nonacademics, and knowledge transfer—are counted in the promotion process. Our results do, in fact, show that these practices are valued less than the traditional indicators of research output. Also, these activities do not appear to have a clear home among the categories of the traditional promotion triad— research, teaching, and community service. Depending on whether we look at the ratings of the deans (written criteria) or the promotion committee members (applied criteria), nontraditional activities at best compete with teaching activities for the second tier. At worst, as in the case of the promotion committee members, most of the nontraditional activities were ranked at the bottom of the list. We say “at worst” not to minimize the importance of community service, but because the activities in question are viewed as fundamental to the research enterprise in many applied disciplines. The results of the deans’ survey actually show

514


a clustering of community service activities and nontraditional research activities, suggesting that the latter are perhaps seen as species of community service. Even taking for granted a system that privileges research over other kinds of scholarship, core research activities for applied health researchers do not appear to count as a genre of research among the people evaluating them. Looking at the inventory alone we might think the results point to a problem of awareness or appreciation; that the evaluation process does not value applied research. The results of the scenario question suggest the situation might not be so simple. Sixty-six percent of promotion committee members and 67 percent of deans said the nontraditional and traditional faculty member stood an equal chance of promotion. This result, even if it is attributable to a social desirability bias in the responses, may be a sign that at least deans and promotion committee members are aware of applied research as distinct from, but potentially as valuable as, discovery-oriented research. This might well reflect the fact that a large part of our sample came from health-related faculties where the need to reward different forms of scholarship has received a lot of recent attention, including reform of some promotion and tenure review systems. Indeed, this conjecture is supported by the finding that the faculty member with nontraditional scholarly outputs in our scenario situation was significantly less disadvantaged in health-related faculties than in the nonhealth faculties. Consequently, the significant discrepancy between the scenario and inventory results could be due to a discord between the perceived value of applied health researchers and awareness of the actual work they do. This would not be surprising. Despite the fact that the new scholarly practices of health research cut across disciplines, almost none of those disciplines assign a clear place to these new practices. Instead, activities that fail to fit under existing definitions of research and teaching tend to get lumped into community service. As Lynton points out in Making the Case for Professional Service (1995), this is a kind of curse because community service in academic departments has come to mean the more or less administrative tasks of institutional and disciplinary citizenship, not scholarship. There is a need for discipline-specific definitions of applied scholarship that outline both scholarly objectives and how specific activities contribute to those objectives. Lynton’s work, as well as that of Boyer (1990) and Glassick, Huber, and Maeroff (1997), goes a long way in framing how this might be done. Definition is one thing, measurement another. The results of this study should be considered in light of increasingly acknowledged deficiencies in


515

research evaluation. In recent years, the traditions for evaluating scientific quality have even been questioned, specifically peer-review and the practice of assigning impact ratings to academic journals (Diamond and Adam 1999, 99–122). The literature on tenure and promotion mentions the lack of clear criteria for assessing the quality of nontraditional research products (Jones and Froom 1994). Even attempts to implement reforms to faculty reward systems have been hampered by the challenge of matching concrete activities to unclear criteria (Schweitzer 2000). The Medical Committee of the Royal Netherlands Academy of Arts and Sciences deemed the predicament serious enough to commission work into new evaluation methods. They were satisfied with the standards for measuring science, but called for a response to the troubling lack of methods for measuring the social impact of research (Smith 2001), and offered its own list of possible impact indicators. In terms of the tenure and promotion review process, measuring impact is only one part of the problem. The activities of applied health researchers integrate the two aims of scientific quality and social (or health) impact, so both of these dimensions need to be evaluated. Efforts to resolve this dilemma are under way, but they need to be tested and disseminated (Aday and Quill 2000). This study provides only a small piece of a very large puzzle and therefore opens numerous possibilities for future research. We need to know more about the different reforms under way, as well as how deans and promotion committee members perceive the issues and the possibilities for change. More reforms need to be evaluated for their effectiveness. The framework and survey used for our study provide a useful baseline and could help faculties to conduct a needs assessment and postreform evaluation. The primary limitation of our study is the self-report nature of the data. Yet social desirability responses would likely lead to an overestimation of the value placed on society-oriented outputs compared to the more inwardly directed outputs of “pure academe.” Even with this potential bias in self-report our results nevertheless indicate a problem for faculty members focused on the more society-oriented outputs. The difficulties we had in obtaining a high response rate from members of promotion committees might also be considered a problem. However, the similarity in responses between them and the deans, who had a higher response rate, indicates that the results would probably have been no different even with a higher response rate. The differences between the health-focused disciplines and schools of business and public administration suggest other possibilities. Work needs

516


to be done to explain the differences and determine if they reflect more progress with a cultural shift in the health-focused faculties and departments. If so, we need to learn more about the facilitators of change. We should also go beyond the faculties included in this study to get a more comprehensive picture of the tenure and promotion process in universities. Do the professional schools we studied here represent, as we hypothesize, the best case in the university, or are there promising practices to be uncovered in the faculties we failed to study? We now have evidence that the faculty reward system in Canadian universities is a barrier to the goals of researchers and funding agencies that focus on the new applied scholarship practices of health research. So what should be done to reconcile this lack of congruence between emerging goals and historical assessment methods for promotion? In a way, the funding agencies have already played their hand and may see the responsibility for change lying with the universities. An alternative would be for them and others to play a more active role, diversifying the investment into new applied scholarship to include initiatives directed at the reform of faculty reward systems. For instance, they could fund the development of new evaluation methods and toolkits that assist faculty to fit their outreach activities into the evaluation guidelines of their host institution. They could also invest in further empirical research into the organizational barriers to this type of scholarship. Finally, faculty members themselves should begin making the case for new scholarly practices in their own disciplines. University administrators interested in seeing their institutions become increasingly relevant to society should consider more and more the policy changes needed to reach that goal. This may ultimately come down to a simple recruitment and retention problem. Schools that do not change will become less and less attractive as a place of employment for the researchers who are most capable of fulfilling the increasingly important outreach mission of a university.

Note 1. Phone calls to individual faculties revealed that at some universities there is no departmental or faculty promotion committee. At these universities, promotion decisions are made by a university-wide committee, often chaired by the vice-president academic. These individuals therefore received the survey and we were unable to assign them to a health-related or nonhealth faculty.


517

References Aday, L. A., and B. E. Quill. 2000. A framework for assessing practice-oriented scholarship in schools of public health. Journal of Public Health Management and Practice 6: 38–46. American Council on Education (ACE), American Association of University Professors (AAUP), and United Educators (UE). 2000. Good practice in tenure evaluation: Advice for tenured faculty, department chairs and academic administrators. http://www.acenet.edu/ bookstore/pdf/tenure-evaluation.pdf (accessed July 4, 2003). Association of Universities and Colleges of Canada (AUCC). 1997. The changing world of scholarly communication: Challenges and choices for Canada. Canadian Journal of Communication 22: 1–5. Boyer, E. L. 1990. Scholarship reconsidered: Priorities of the professoriate. Princeton, NJ: Princeton University Press. Canadian Health Services Research Foundation. Health services research and evidence-based decision-making. http://www.chsrf.ca/knowledge_transfer/pdf/EBDM_e.pdf (accessed August 12, 2004). Canadian Institutes of Health Research. Mandate. http://www.cihrirsc.gc.ca/e/about/7262.shtml# (accessed August 12, 2004). Community-Campus Partnerships for Health. 2005. Linking scholarship and communities: Report of the Commission on Community-Engaged Scholarship in the Health Professions. http://depts.washington.edu/ccph/pdf_files/Commission%20Report%20FINAL.pdf (accessed April 24, 2006). Conine, T. A., L. McPherson-Shilling, and E. R. Pierce. 1985. The relative importance of supportive data for promotion and tenure reviews. Journal of Allied Health 14: 183–90. Court, S. 1998. Opportunity blocks: A survey of appointment and promotion in UK higher education. London, UK: Association of University Teachers. Crosswaite, C., and L. Curtice. 1994. Disseminating research results—the challenge of bridging the gap between health research and health action. Health Promotion International 9: 289–96. Denis, J-L., and J. Lomas. 2003. Convergent evolution: The academic and policy roots of collaborative research. Journal of Health Services Research and Policy 8 (Suppl. 2): S1–S6. Diamond, R. M., and B. R. Adam. 1999. Documenting diversity: The professional portfolio and faculty rewards. In Faculty productivity: Facts, fictions, and issues, edited by W. G. Tierney, 99–122. New York: Falmer Press. Giacomini, M. 2004. Interdisciplinarity in health services research: Dreams and nightmares, maladies, and remedies. Journal of Health Services Research and Policy 9: 177–83. Glassick, C. E., M. T. Huber, and G. I. Maeroff. 1997. Scholarship assessed: Evaluation of the professoriate. San Francisco: Jossey-Bass. Henke, R. March 2000. Final report: International conference on “Social Science and Governance.” Zeist, The Netherlands: Netherlands Commission for UNESCO, Management of Social Transformations Programme. Howell, L. P., and K. D. Bertakis. 2004. Clinical faculty tracks and academic success at the University of California Medical Schools. Academic Medicine 79: 250–57.

518


Jacobson, N., D. Butterill, and P. Goering. 2004. Organizational factors that influence university-based researchers’ engagement in knowledge transfer activities. Science Communication 25 (3): 246–59. Jones, R. F., and J. D. Froom. 1994. Faculty and administration views of problems in faculty evaluation. Academic Medicine 69: 476–83. Layzell, D. T. 1999. Higher education’s changing environment: Faculty productivity and the reward structure. In Faculty productivity: Facts, fictions, and issues, edited by W. G. Tierney, 3–37. New York: Falmer Press. Lynton, E. A. 1995. Making the case for professional service. Washington, DC: American Association for Higher Education. Schweitzer, L. 2000. Adoption and failure of the “Boyer model” at the University of Louisville. Academic Medicine 75: 925–29. Social Sciences and Humanities Research Council of Canada. 2004. SSHRC Transformation–Consultation Documents. http://www.sshrc.ca/web/whatsnew/initiatives/ transformation/documents_e.asp (accessed August 12, 2004). Smith, R. 2001. Measuring the social impact of research. British Medical Journal 323: 528.

Marie-Rose Phaneuf is a medical student at the University of Ottawa. She completed a B.Sc. in health studies at the University of Waterloo, and a M.Sc. in public health sciences at the University of Alberta. Jonathan Lomas became the inaugural chief executive officer of the Canadian Health Services Research Foundation in 1998. His background includes work or training in psychology, health economics, epidemiology, political science, and management. From 1982 to 1997 he was a professor of health policy analysis at McMaster University, Canada, where he cofounded the Centre for Health Economics and Policy Analysis. The main focus of his work is the role and impact of research in healthcare systems. Chris McCutcheon is a senior program officer, Organizational Capacity Development, at the Canadian Health Services Research Foundation. In this role he tries to develop initiatives that improve Canada’s health-system research capacity by encouraging environments that are more supportive of the unique scholarly activities fundamental to applied health services research, such as knowledge transfer and exchange. He holds a master’s degree in social and political thought from York University. John Church is an associate professor in the Centre for Health Promotion Studies and the Department of Political Science at the University of Alberta. He teaches and conducts research on public policy processes and outcomes. Currently, he is co-investigator in two Canadian Institutes of Health Research–funded initiatives, including an Obesity NET and a cross-provincial study of health reform decision making, and a Canadian Population Health Initiative–funded knowledge synthesis on obesity. He was also research lead in the development of a Policy Analysis Framework for a Pan American Health Organization–funded initiative titled, “CARMEN Non-Communicable Disease Policy Observatory.” Douglas Wilson is professor emeritus in the Departments of Public Health Sciences and Medicine at the University of Alberta. A medical graduate of the University of Toronto, he was dean of the faculty of medicine at the University of Alberta from 1984 to 1994. There he was instrumental in establishing the interdisciplinary Centre for Health Promotion Studies with its graduate education and applied research programs.