ate the continued psychological and financial support of the Dean's. Office. ... service deliverers and customers' ratings would also be of con- siderable ...
Copyright 1994 by the American Psychological Association, Inc. 0021-90IO/94/$3.00
journal of Applied Psychology 1994, Vol. 79, No. 5,685-690
Do Customer Service Perceptions Generalize? The Case of Student and Chair Ratings of Faculty Effectiveness
This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
Benjamin Schneider, Paul J. Ranges, Harold W. Goldstein, and Eric P. Braverman We conceptualize faculty classroom behavior as service delivery and show that student (customer) evaluations of service effectiveness generalize to chairs' ratings of faculty effectiveness in other facets of the faculty role. We compared students' ratings of 132 professors in the classroom with their chairs' ratings of them in seven other roles. Findings suggest that students' ratings of faculty classroom behavior cluster with chairs' ratings of faculty colleagueship and service and that this cluster is relatively independent of chairs' ratings of faculty publication record, grant support, and professional activity. Results are interpreted in light of Gouldner's (1958a, 1958b) local-cosmopolitan distinction. Some implications for research on faculty effectiveness and customers' evaluations of service are presented.
is whether the behavior of service deliverers during a service encounter is related to the behavior of the same service deliverers in other aspects of their job. In fact, many facets of a service deliverer's job can be conceptualized as service encounters with different internal or external customers. Data concerning which other facets of employee behavior are related to customer perceptions would be important for several reasons. From an applied perspective, they would be important because it follows that these other behaviors are those that should be rewarded, supported, and expected as part of the creation and maintenance of a service climate (Schneider, 1990). For example, suppose that bank customer perceptions of teller service quality are later shown to be related to branch manager ratings of teller organization-citizenship behavior (OCB; Organ, 1988). Such information identifies potential internal organizational interventions (e.g., rewards to enhance tellers' OCBs) that might yield improvement in service quality to customers. The relationship between internal ratings of the behavior of service deliverers and customers' ratings would also be of considerable theoretical interest. This interest concerns the generalizability of behavior from situation to situation—an indicator of behavioral consistency. There is, of course, a large literature on behavioral consistency both in (Schneider, 1983) and out (Epstein, 1980) of the work place. We need not explore that literature here except to note that to our knowledge no previous literature relates end-user customers' perceptions of behavior to internal ratings of the behavior of employees. The present study attempts to examine such a relationship within the context of student evaluations of faculty behavior in the classroom and chairs' ratings of faculty behavior outside of the classroom.
The purpose of this study was to explore relationships among multiple facets of employee performance as rated by both customers and supervisors. The present study conceptualized teaching as the service being delivered, students as the customers being served, faculty as the service deliverers, and department chairs as the supervisors.
Teaching as Service According to the definition given by Bowen and Schneider (1988) and others (e.g., Parasuraman, Zeithaml, & Berry, 1985), teaching can be conceptualized as a service, in that (a) teaching processes and experiences are relatively intangible; (b) teaching is typically produced, delivered, and consumed simultaneously; and (c) teaching typically requires the presence of customers. Intangibility, simultaneity, and customer presence make service deliverers the key to service quality (Schneider & Bowen, 1992). One issue that has not been explored in the service literature
Benjamin Schneider, Paul J. Hanges, Harold W. Goldstein, and Eric P. Braverman, University of Maryland at College Park. This research was funded by the Dean's Office, College of Behavioral and Social Sciences, University of Maryland, College Park. We appreciate the continued psychological and financial support of the Dean's Office. The Computer Science Center at the University of Maryland, College Park, supported the data analyses connected with this article. Many people helped us make this research effort happen. Kathryn Niles, Beth Benjamin, and Pamela Carter helped in various ways on the present study. Over the years, David Schoorman, Jocelyne Gessner, Christy Douglas, Rene Morales, Gary King, and Gary Musicante helped the Course Feedback Survey program grow and mature so that the present study could be accomplished; we appreciate their help as well. Harold W. Goldstein is now at the Department of Psychology, Bowling Green State University. Eric P. Braverman is now at the Department of Psychology, Louisiana State University. Correspondence concerning this article should be addressed to Benjamin Schneider, Department of Psychology, University of Maryland, College Park, Maryland 20742-4411.
Student Evaluations of Faculty in the Classroom Some of the findings from the service marketing literature (see Bowen & Schneider, 1988, for a review) have been replicated in the student evaluation literature. For example, student evaluations of faculty classroom behavior have been shown to be positively related both to faculty self-evaluations (Blackburn & Clark, 1975; Breskamp, Caulley, & Costin, 1979; Centra, 685
This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
686
SCHNEIDER, HANGES, GOLDSTEIN, AND BRAVERMAN
1973; Doyle & Crichton, 1978; Marsh, 1987) and to ratings produced by external observers (Marsh, 1987; Murray, 1983). Thus, contrary to some faculty opinions and consistent with the service marketing literature, student evaluations of faculty are not simply popularity contests that yield data unrelated to other reports of classroom behavior (see Marsh, 1987, for an extensive review of the validity of student evaluations). A review of the literature on faculty evaluation in general, and student evaluations of faculty in particular (cf. Eble, 1971; Miller, 1972, 1987; Millman, 1981; Whitman & Weiss, 1982), indicates that the only nonclassroom faculty behavior that has been studied as a correlate of student evaluations has been research productivity of faculty (e.g., Feldman, 1987). There are obviously numerous other facets of the faculty role to which student evaluations might be related and on which faculty are evaluated for administrative purposes (e.g., Miller, 1987). For example, there is no research on how student evaluations relate to faculty behavior as a departmental citizen (Organ, 1988) or as a generator of grant money. The central purpose of the present effort was to explore these issues. More specifically, we were interested in the degree of relationship between student evaluations of faculty classroom behavior and seven facets of the faculty role. Although no formal hypotheses were tested, it follows from a behavioral consistency framework that faculty rated by students as delivering superior service in the classroom are likely to be rated highly by others (in the present case the department chair) for their behavior in other roles. Method The present study took place in a college of social sciences at a midAtlantic state university. The college contained nine departments. Both student and chair ratings were collected on a total of 132 faculty members.
Chair Ratings The various facets comprising faculty members' jobs were obtained from interviews with the Assistant Dean and the nine chairpersons of the departments in the college. Seven facets of faculty effectiveness were derived from these meetings. Four of these facets concerned behavior limited to the boundaries of the university: graduate educator effectiveness (graduate; e.g., serving on master's/doctoral committees, mentoring, etc.); undergraduate educator effectiveness (undergrad; e.g., all facets of teaching except student evaluations, counseling undergraduates); department, division, and university service (service; e.g., serving on committees and/or service positions); and colleagueship (colleagueship; e.g., level of informal contact from colleagues, recognition and support from colleagues). The three remaining facets of faculty effectiveness concerned behavior that extends beyond the university: publication history (publications; e.g., journal articles/books written, books edited, papers presented); outside support (grants; e.g., grants/ fellowships achieved, contracts awarded), and professional activities (activities; e.g., offices held in professional organizations, editorial positions, professional committees). In separate private sessions, the chairs were asked to take each facet of effectiveness and using a forced-distribution scale (Carroll & Schneier, 1982; Cascio, 1987), distribute their faculty along a continuum of effectiveness. This forced-distribution rating scale had five categories labeled with the appropriate number of ratees (faculty) representing 10%, 20%, 40%, 20%, and 10% of the total sample for each department.
The order of the seven dimensions of faculty effectiveness was randomized for each chair. In addition, all chairs rated faculty along an eighth dimension, overall effectiveness. Chairs rated their faculty on this eighth dimension either first (prior to rating faculty on the other seven dimensions) or last, depending on random assignment. Also, for each dimension on which chairs rated their faculty, the order in which faculty names were presented was randomized.
Student Ratings The teaching evaluation questionnaire was developed in 1982 through a series of interviews with department chairpersons, individual faculty members, and undergraduates. A factor analysis of the questionnaire indicated that seven factors are measured. The first five factors concern faculty classroom behavior: Organized (e.g., gives lectures that are easy to outline); Participative (e.g., encourages class discussion); Evaluation (e.g., gives exams that reveal strengths and weaknesses); Availability (e.g., is accessible outside of class); and Personal Style (e.g., demonstrates extensive knowledge of the subject). Two additional factors assess reactions to the course: Amount Learned (e.g., I really learned a lot) and whether or not students would Recommend the class to others (e.g., I would recommend this course and instructor to others). Finally, an Overall evaluation of the course is obtained by averaging student responses to all items on the survey. Further description of this questionnaire is provided in Hanges, Schneider, and Niles (1990). As reported in Hanges et al. (1990), these eight dimensions have considerable internal consistency (Cronbach's alpha ranged from .90 to .98). For the present study, student ratings on the eight teaching effectiveness dimensions were averaged across time (from the Spring 1982 semester up to the Fall 1988 semester) and courses taught for each of the 132 professors in the sample. This aggregation of student ratings follows Epstein's (1980) recommendation to increase the reliability of measures by aggregating over time (semesters in our case) and/or situations (courses in our case). Because the student data used in the present study to index faculty classroom behavior were composites, based on averaging student ratings over courses and semesters, the average reliability of student ratings was determined by applying the Spearman-Brown prophecy formula (Allen & Yen, 1979; Nunnally, 1978) to each professor's data and averaging these individual professor reliability estimates across all 132 professors. The eight teaching dimensions explored in the present study are quite stable. The reliability of the average student ratings ranged from .83 (Evaluation) to .90 (Organized).
Results Department Chair Ratings Table 1 shows the intercorrelation matrix of the department chair ratings of faculty effectiveness. The correlations among the seven specific scales ranged from .22 (correlation between undergrad and grants) to .72 (correlation between graduate and publications). Overall, the average intercorrelation among the seven specific faculty dimensions was .47 (after r to z transformation). Two observations can be made from this correlation matrix. First, the forced-distribution ratings procedure seems to have yielded the positive consequence of relatively modest interdimension correlations. Second, the chair ratings of specific facets of faculty effectiveness correlate with chair ratings of overall effectiveness from strongest to weakest as follows: graduate (r = .80), publications (r = .77), activities (r = .69), colleagueship (r = .66), grants (r = .63), undergrad (r = .45), and service (r = .43).
687
FACULTY EFFECTIVENESS Table 1 Inter correlations Among Chair Ratings and Correlations Among Chair and Student Ratings Chair ratings
This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
Measure Chair ratings 1. Publications 2. Graduate 3. Undergrad 4. Grants 5. Activities 6. Service 7. Colleague 8. Overall-C Student ratings 9. Overall-S 10. Org 11. Part 12. Eval 13. Avail 14. Style 15. Recorn 16. Learn
.72 .24 .68 .59 .24 .43 .77
.16 .09 .27** .02 -.08 .19* .11 .16
.42 .54 .64 .52 .60 .80
.22 .32 .59 .56 .45
.37** .25** .39** .19* .09 .37** .29** .41**
.45** .50** .15 .33** .19* .43** .50** .41**
.44 .30 .37 .63
.19* .11 .19* .10 -.05 .20* .17 .22*
.36 .35 .69
— .66 .43
— .66
—
.21* .13 .22* .04 .10 .21* .13 .29**
.29** .28** .09 .21* .26** .27** .27** .29**
.29** .32** .20* .17 .26** .25** .24* .27**
.32** .32** .28** .14 .08 .32** .26** .35**
Note. All intercorrelations among chair ratings are significant at p < .01. Publications = publication history; Graduate = graduate educator effectiveness; Undergrad = undergraduate educator effectiveness; Grants = outside support; Activities = professional activities; Service = department, division, and university service; Colleague = colleagueship; Overall-C = overall effectiveness (chair rating); Overall-S = sum of student ratings; Org = organized; Part = participative; Eval = evaluation; Avail = availability; Style = personal style; Recom = recommend to others; Learn = amount learned. *p
