Learning from Your Business Lectures: Using ...

Learning from Your Business Lectures: Using Stepwise Regression to Understand Course Evaluation Data Kaleel Rahman, The University of Sydney, New South Wales, Australia ABSTRACT Effective teaching in business schools is important because not only is “business administration” evaluated below average by students (Marks and O’Connell, 2003), but also business is one of the fastest growing academic fields, driven by strong industry demand both locally and internationally. This paper presents a case study from a pragmatic perspective, demonstrating an effective approach to using student evaluation data and gaining further insights into teaching practices in business schools. In understanding teaching evaluation data, “mean rating” is the norm in most cases. However, mean ratings can only be considered in isolation or relative to the mean ratings of other items. Whether a particular variable influences overall quality or satisfaction in a systematic fashion should be a key consideration when examining mean scores. Using a course evaluation dataset and stepwise regression technique on the SPSS statistical program, this paper provides a hands-on tool to use teaching evaluation data more effectively. INTRODUCTION Seeking feedback on teaching is an important part of the process of course and teaching evaluation. The process of (a) gathering information about the impact of learning and of teaching practice on student learning, (b) analysing and interpreting this information, and (c) responding to and acting on the results, is valuable for several reasons. When effectively carried out and appropriately used, evaluation has the potential to enhance student learning and the student experience, to facilitate and inform our professional development as educators and to promote quality assurance and improvement (Ramsden and Dodds 1989). Accordingly, a majority of tertiary institutions use student evaluations as one of their most important measures of faculty teaching and course effectiveness. However, some scholars have argued that students are not an appropriate or effective source of teacher evaluation. Tomasco (1980) argued that student evaluations of teaching are more likely to be personality contests than valid measures of teaching effectiveness. He also posited that student evaluations of teaching can lead to grade inflation and lowering of standards. Johnson (2000) argued that the motives for employing student evaluations of teaching or of courses are neither educationally sound nor focused on fulfilment of the goals of teachers or students. Johnson claimed that such evaluations exist primarily to serve the needs of bureaucracies in which a systematic reporting of feedback can be conducted on teaching quality standards. Despite these criticisms, there is ample evidence to suggest that students can provide useful information about the effectiveness of teaching methods (Stockham and Amann, 1994). Specifically, as I believe, an evaluation method that uses responses from a majority of students in a specific class setting can help to identify the nature of teaching- and course-related issues. Several researchers have investigated whether there are any differences between different disciplines in terms of student evaluations. In particular, I wanted an answer to the question “How does the business administration discipline compare with other disciplines in terms of student evaluation?” Marks and O’Connell (2003, p. 262), having reviewed the work reported by other authors, suggested that “business” was one of the worst players: “Feldman’s survey (1978,) of studies prior to 1978 found that student evaluation ratings were slightly above average in English, arts, language, and other humanities, average in the biological sciences, and somewhat below the mean in social sciences, physical sciences, mathematics, engineering, and business administration. A later study by Cashin and Clegg (1987) covering a great variety of courses spanning many institutions agreed with Feldman’s ordering.” [emphasis added]

Although differences between disciplines are relatively small (Marks and O’Connell, 2003), it is important to seek to improve the course and teaching effectiveness in business schools for several reasons. First, business is one of the fastest developing disciplines in the Western and Eastern academic worlds, driven by increased industry demand. Second, business schools reap a relatively higher level of revenue in terms of course fees, particularly from

The Journal of American Academy of Business, Cambridge * Vol. 9 * Num. 2 * September 2006

272

international students. Third, business disciplines constitute one of the ‘faculty of professions’, so that it is crucial to teach and train students in business schools to prepare for real-world practice. The key purpose of this paper is to demonstrate an effective approach to using student evaluation data to gain insights into teaching practices. Accordingly, in this paper, I present a case study of my own teaching experience in a business school, utilizing student evaluation data to understand my teaching approaches more effectively than by traditional methods. First I provide a background of my teaching endeavours, followed by an introduction to the Unit of Study Evaluation method used in my university. Then I introduce the stepwise regression technique and the stages involved in performing this technique in the SPSS statistical program, demonstrating the actual figures produced. Next I analyse and interpret my own data using the actual output produced. The conclusion suggests implications for other teachers. BACKGROUND I was allocated to develop and teach a third year undergraduate subject in a business discipline. Since this was going to be my first ever lecturing adventure (although I had considerable experience in small class teaching as a tutorial instructor), I spent much time preparing myself in terms of evaluating the syllabi of other teachers who had taught the same course, and perusing the code of conduct in terms of dealing with students, appropriate level of assessment, etc. In addition, I also took a short course on “principles and practice of university teaching and learning”. Help was available to me through different channels. There was a center for teaching and learning (teaching centre), a body that served the university in terms of teaching and learning development. There was a center for advanced learning in economics and business, a body that served the faculty to which I belong. And there were peers and mentors within my discipline. Since my school is an AACSB International accredited body, I had to pay considerable amount of attention to particular standards to which my course should adhere. Having obtained various types of help from the above constituents, and keeping AACBS standards in mind, I developed the course. Considered a large teaching class, my class consisted of 192 students from various backgrounds, of whom about 40% were international students. The format involved a 2-hour weekly lecture at which all students were expected to be present, although attendance was not assessed. Delivering the lectures relied heavily upon PowerPoint presentations slides, on which I devoted much time prior to the lectures. In addition to the weekly lecture there was a 1-hour weekly tutorial which was conducted by three tutorial instructors. I prepared a complete set of tutorial material in consultation with my peers and with the prior syllabi available to me. Assessment items included two individual assignments, a major project, tutorial participation marks, and a final examination. Apart from the considerable amount of time spent developing the unit of study, during the duration of the subject, I spent quality time in reading course materials and preparing lectures, responding to students’ queries, maintaining the blackboard (an online student administration platform), meeting with tutorial instructors, and of course delivering lectures. With my first-time experience, enthusiasm, and motivation to earn a good name, I attempted to deliver far more than I was required to do. Hence I expected a reasonable level of teaching/course evaluation from students. UNIT OF STUDY EVALUATION (USE) The Unit of Study Evaluation (USE) is the standard procedure in my university for teaching evaluation. Items included in the USE survey represent factors of various types, including clear goals and standards, good teaching, appropriate assessment, appropriate workload, overall satisfaction, etc. According to the teaching center that developed this survey, the items included in it are based on important components reflecting quality control and good teaching and learning practices. The USE survey includes 12 items (see Table 1). Each item is rated on a five point scale from ‘strongly disagree’ to ‘strongly agree’. Items 1 through to 11 measure various components of a course. Items 1 to 7 and 12 are standard across the university, while items 8 to 11 are selected by the individual faculty, and in my case these items were strongly reflective of AACBS International standards. Item 12 addresses overall satisfaction with the course. As a summary measure, item 12 is considered the most important component, that captures the overall performance of the course of study. Overall satisfaction as a single measure has widely been used in scholarly research (Tang, 1997). As Marks and O’Connell (2003) posited, an average rating on the overall satisfaction item is considered by

The Journal of American Academy of Business, Cambridge * Vol. 9 * Num. 2 * September 2006

273

some to be a surrogate for teaching effectiveness and often is used as a basis for decisions on compensations, tenure, and promotion. In this case the overall satisfaction item was also important because the class population included students from the wide range of cultures and countries that utilize this facility. A student who has no interest in research (item A11) may not be able to judge the effectiveness in terms of research-led teaching and learning. However, regardless of particular interests, any student should be able to rate “overall” satisfaction, based on a holistic abstraction that he/she obtained throughout the course of study. Items (factors) A1 A2 A3 A4 A5 A6 A7 A8 A9 A10 A11 A12

Mean Ratings

Standard Deviation

Clear goals and standards 3.7 Good teaching 3.1 Generic skills 3.5 Appropriate workload 3.1 Appropriate assessment 3.3 Relevance to my degree 3.9 Appropriate feedback 3.3 Feedback assisted learning 3.1 Engaged learner 3.3 Online learning 3.5 Research-led teaching 3.3 Overall satisfaction 3.2 Grand mean 3.4 Table 1 Unit of Study Evaluation (USE) items1 and mean evaluations

0.84 1.11 0.86 0.96 0.73 1.00 1.02 0.90 1.02 0.94 0.99 0.88

Data Collection At the end of the semester I decided to collect evaluation data in two forms. One form was the USE, that evaluated the course as whole. The other form was a qualitative teaching evaluation in which questions were essentially about the teaching skills of an effective teacher. This paper deals only with the first from of evaluation, the USE. For data collection, the survey was randomly distributed to two thirds of the students who attended the last lecture. (The other third of the students were asked to use the qualitative teaching skills survey that is not reported here.) Three students who volunteered to administer the survey sealed the completed surveys in an envelope to be sent to teaching center. After the final examination results were released I received the results of the survey including the original hard copies. Results The results forwarded by the teaching center included the mean ratings with standard deviation and the percentage distribution between disagreement, neutrality, and agreement for each individual item, as presented in Figure 1. These results were conducive to insight, demonstrating my strengths and weaknesses in terms of each of the items. For example, in terms of “relevance to my degree”, I could see that my average rating was 3.9, and only 5% of the students disagreed or strongly disagreed with this statement, 15% were unsure, and 80% agreed or strongly agreed with the statement. In comparison with my expectations, some of the items for which I expected to receive a better rating were indeed evaluated better. Looking at the mean ratings presented in Table 1, I was fairly satisfied that it was far better than the finding reported by Marks and O’Connell (2003) that “business administration” did below average. However, the teaching center encouraged me to further analyze my data using both quantitative and qualitative approaches. Furthermore, I genuinely wanted to improve on the items for which I received a relatively lower rating. Having consulted my supervisor and a mentor, I took further action to improve my teaching effectiveness.

Figure 1 A sample item from Unit of Study Evaluation (USE) results

It should be noted that mean ratings can only be considered in isolation or relative to the mean ratings of other items. Whether a particular variable influences overall satisfaction in a systematic fashion should be a key consideration when examining the mean scores provided by teaching centres. As Field (2000, p. 107) wrote, “mean is fairly useless as a model of a relationship between two variables – but it is the simplest model available”. However, my key consideration was item 12, which had the mean of 3.2. I wanted to know which items were

The Journal of American Academy of Business, Cambridge * Vol. 9 * Num. 2 * September 2006

274

strongly associated with and had an influence on this overall satisfaction item. Consequently I looked to correlation and regression analysis for some insight. STEPWISE REGRESSION Stepwise regression is a type of multiple regression model building technique that finds the best sub-set of predictor (independent) variables from a larger set (Field, 1999). Although there are many variations, the basic procedure is to find the single best predictor variable and add variables that meet some specified criteria. This procedure is automatically performed by computer programs, one of which is SPSS. These programs typically stop admitting predictor variables into the model when no predictor variable makes a contribution that is statistically significant at a level specified by the user. Thus, the regression equation is constantly reassessed to see whether any redundant predictors can be removed. Eventually the model that makes the largest contribution to R Square is considered the best model. Although the stepwise method may not be appropriate for confirmatory hypothesis testing (Cohen and Cohen, 1983; Miles and Shevlin, 2001), this method can be used effectively in “exploratory model building” (Wright, 1997, p. 181). Since I did not have any hypothesis in terms of which variables may predict overall satisfaction, stepwise regression was considered the most appropriate technique. Stepwise regression in SPSS I analysed my data using SPSS version 13 for Windows (SPSS Inc.). Previous versions of SPSS may also be used to run this analysis. First I entered my data by opening a new data file as shown in Figure 2. Hard copy questionnaires were given ID numbers from 1 to 96 (only 96 students had completed item 12), which was entered in the first column. The rest of the columns from a1 to a12 correspond to items presented in Table 1.

Figure 2 SPSS data sheet

Before running the regression2 I obtained a correlation matrix in order to find out which of the 11 items were strongly associated with the overall satisfaction item as presented in Table 2. I did this by clicking on analyse Æ correlate Æ bivariate and moving the variables into the dialog box on the right hand side, then clicking OK. Interestingly, as demonstrated in the last row of Table 2, all of the factors were significantly correlated with the overall satisfaction measure. The negative correlation of item 4 is reflective of its negative statement in terms of workload. The correlation matrix also showed that all the factors were positively related to one another. This demonstrates the validity of the measures and appears to be in agreement with Cashin’s (1995) conclusion that “in general, students’ ratings tend to be statistically reliable, valid, and relatively free from bias” (quoted in Marks and O’Connell, 2003, p. 259). I also checked the data against the assumptions of multiple regressions.3 Next, in order to perform the stepwise regression, I used the following stages in SPSS (see Figure 2): 9 9 9 9 9 9

Select Linear from the Analyze Æ Regression menu. Click on the dependent (a12) variable and then on the top arrow – a12 moves to Dependent box. Click on each of the predictor variables (a1 to a11) in turn and click on the middle arrow – all of them move to Independent box. Click on the down arrow next to Method and choose stepwise as the method. Click on statistics and tick estimates, model fit, R squared change, and collinearity diagnostics and click continue. Click OK when ready.

The Journal of American Academy of Business, Cambridge * Vol. 9 * Num. 2 * September 2006

275

Relationships of interest – correlations of factors 1 to 11 with overall satisfaction.

Table 2 Correlation matrix of all the variables

Figure 2 Stepwise regression analysis procedure in SPSS

The Journal of American Academy of Business, Cambridge * Vol. 9 * Num. 2 * September 2006

276

Stepwise regression adds and removes variables in steps until the optimum model is reached. The final model is given in the last step. An output window will produce a series of tables. The first table (not given here as nothing is usually reported from this table) produced in the output window consist of four columns: Model, Variables Entered, Variables Removed and Method. The step (model) number appears in the first column. Variables introduced at each step are detailed in the second column. Variables removed at any step are listed in the third column. The fourth column provides a summary of the rules used for inclusion and removal. In the model summary table (see Table 3) the R Square variable gives the proportion of variance that can be predicted by the regression model using the data provided. It is commonly reported as a percentage. Variables used in each step of the analysis are shown below the table. 'a' is step 1, 'b' is step 2, ‘c’ is step 3, and ‘d’ is step 4. Variables are added or removed only if the Sig. F Change value is significant (lower than .05).

Table 3 Model summary statistics from SPSS output window

Looking at my own case, the fourth model with R square of .645 and predictors of a2, a3, a10 and a9 was the model that best fit the data. This can be interpreted that 64.5% of total variance in overall satisfaction was explained by good teaching, generic skills, blackboard and engaged learner. It is interesting that although the correlation matrix showed that all of the 11 variables were positively associated with overall satisfaction, only four of them significantly predicted overall satisfaction. Four of these variables were included in the model justified by significant R square change (see justified by Sig. F Change statistics). Other predictor variables did not make a statistically significant contribution. Since the use of adjusted R square in SPSS is subjected to criticism (Field, p. 130), I did not consider that here. Next I turned to the ANOVA table (see Table 4). ANOVA statistics will be provided for each model. You would report only the final model in the ANOVA table. The F-ratio is given for the ANOVA. The significance (Sig) value is given in the final column. In my case, the large F value of 41.3 shows that it was significant. This tells me that the four predictor variables as a combination significantly predicted overall satisfaction.

Table 4 ANOVA table from SPSS output window

The Journal of American Academy of Business, Cambridge * Vol. 9 * Num. 2 * September 2006

277

Next I looked at the coefficient table (see Table 5). This was the most important table, because I wanted to determine the individual effect of these variables on overall satisfaction. Pay attention to Model 4. The t-test statistic (t) for each component of the model is given in the t column of the Coefficients Table. The p-value (Sig) for each component of the model is given in the Sig column. The coefficients of the final regression model are given in the B column of the Unstandardized and Standardised Coefficients. I could use these values to write the regression equation.

Table 5 Regression coefficients table from SPSS output window

In my case, the first thing I had to look for was any evidence of multicollenearity, because although the correlation matrix showed that the predictor variables were significantly correlated with one another, in multiple regression they must not be highly correlated with one another. This can be detected by examining Tolerance under Collinearity Statistics. If the Tolerance statistic is below .2, multicollinerarity can be biasing the results (Menard, 1995). My data were not subject to multicollenearity. Hence, on the basis of the unstandardized regression coefficients I could write a regression equation as follows: Satisfaction

= β0 + β1 good teaching + β2 generic skills + β3 blackboard + β4 engaged learner = -.16 + [.4 good teaching] + [.2 generic skills] + [.19 blackboard] + [.24 engaged learner]

Holding other variables constant, as good teaching increases by one rating unit, overall satisfaction increases by .4 rating units; as generic skills increases by one rating unit, overall satisfaction increases by .2 rating units; as blackboard increases by one rating unit, overall satisfaction increases by .19 rating units; as engaged learner increases by one rating unit, overall satisfaction increases by .24 ratings unit. The standardised coefficients provide a better insight into the relative contribution of each variable. They tell us the number of standard deviations that overall satisfaction will change as a result of one standard deviation change in the four predictors. DISCUSSION AND CONCLUSION From a pragmatic perspective, the key purpose of the present case study was to demonstrate an effective approach to using student evaluation data and gaining further insight into our teaching practices. Effective teaching in business schools is important because not only is “business administration” frequently evaluated below average by students (Marks and O’Connell, 2003), but also business is one of the fastest growing academic fields influenced by strong industry demand both locally and internationally. As teachers in universities and colleges, many of us do not look into the feedback provided by our students except for the results provided by a “teaching center” available to serve teaching and learning. Unfortunately, such centers do not have adequate resources to pay close attention to every teacher. Eventually, as time-poor teachers, we tend to stack up student evaluation data in file drawers hoping for a future opportunity for investigation or sometimes we even trash them. Having considered different approaches to improve my teaching effectiveness, I used stepwise regression to gain further insights into my teaching practices. When it comes my teaching, I now understand the factors driving students’ overall satisfaction. In future teaching practice I can focus strongly on these drivers because these factors

The Journal of American Academy of Business, Cambridge * Vol. 9 * Num. 2 * September 2006

278

contribute to their satisfaction far more than other factors. For example, good teaching was found to be the strongest driver of satisfaction, followed by engaged learner, generic skills and online learning respectively. If I had relied on the mean scores provided to me, I would not have found that these variables drive satisfaction. To illustrate, the highest mean score I obtained was in “relevance of the subject to degree” (3.9) but that has nothing to do with whether students are satisfied with the overall quality of the course. Similarly, I put a considerable amount of time and effort into research-led teaching and consequently obtained a good rating, only to find out that research-led teaching did not predict overall satisfaction. In other words, a particular student may have said, for example, “yes, you got us to do a busload of research-led activities, and I can be honest with you and give you a good rating (e.g., 5) on that for your hard work, but I was not satisfied with the quality of that research component and I am going to rate you as an average teacher (e.g., 3) on satisfaction”. On the other hand, I obtained a relatively low score (3.1) on the “good teaching” factor, and now I know that it is the most important determining factor in terms of students’ overall satisfaction. At the same time I will warn myself not to ignore the remaining seven factors that do not drive overall satisfaction. As I illustrated above with research-led teaching, I would also consider why certain factors like this do not predict students’ satisfaction. Maybe I am adopting a wrong approach in implementing such practices, and accordingly I will attempt to understand the reasons behind each of the factors. Alternatively, it is not only I who believe that all the factors are important, but my university also emphasizes that these factors are the reflection of various teaching quality standards adhered to by the university. Furthermore, the correlation matrix suggests that all of these factors are interrelated. Consequently I intend to run the same analysis after the next teaching project. If I find that in my teaching the same four variables drive overall satisfaction, I will work on my micro-model even more strongly. Finally, the present case is based on evaluation of a single teacher at a particular point in time. Therefore the techniques illustrated and conclusion reached in this paper should not be used for any other purposes except for evaluating one’s own teaching practices, although one can learn a few things about running a stepwise regression in SPSS that may be helpful in research projects far removed from the subject matter of this paper. ACKNOWLEDGMENT The author is grateful to Professor Charles Areni for his guidance in the statistical techniques from which this paper was inspired. He is also thankful to Peter McDonald for useful guidance and mentoring in teaching.

Notes: 1. The actual items were lengthy descriptions of the factors involved. 2. For a good discussion of the distinction between multiple correlation analysis and multiple regression analysis, see Huberty (2003) “Multiple correlation versus multiple regression”, Educational and Psychological Measurement, Vol 63, No. 2, pp 271-278 3. For a good discussion of multiple regression assumptions, see Gujarati, D. (1998) Essentials of Econometrics, McGraw-Hill: College

REFERENCES Cashin, W. E. (1995) Student Rating of Teaching: The research revisited. Center for Faculty Evaluation and Development, Kansas State University Cashin, W. E. and Clegg, V. L. (1987) “Are student ratings of different academic disciplines different?” Paper presented at the 1987 Annual Meeting of the American Educational Research Association, Washington, DC. (ERIC Document Reproduction Service No. ED 289 935)

Cohen, J. and Cohen, P. (1983) Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences, 2nd Edition, New Jersey: LEA Feldman, K. A. (1978) “Course characteristics and college students; ratings of their teachers: What we know and what we don’t”, Research in Higher Education, Vol. 9, pp. 199-242 Field, A. (2000) Discovering Statistics Using SPSS for Windows: Advanced Techniques for the Beginner, Sage Publications, London. Johnson, R. (2000) “The authority of the student evaluation questionnaire”, Teaching in Higher Education, Vol. 5, pp. 419-434 Marks, N. B. and O’Connell, R. (2003) “Using statistical control charts to analyse data from student evaluations of teaching”, Decision Sciences Journal of Innovative Education, Vol. 1 No. 2, pp. 259-272 Menard, S. (1995) Applied Logistic Regression Analysis. Sage series on quantitative applications in the social sciences, 07 – 106. Thousand Oaks, CA: Sage

Miles, J. and Shevlin, M. (2001) Applying Regression & Correlation, London: Sage Ramsden, P. & Dodds, A., 1989, Improving Teaching and Courses: A Guide to Evaluation, 2nd edn, Centre for the Study of Higher Education, The University of Melbourne, Parkville. Stockham, S. L. and Amann, J. F. (1994) “Facilitated student feedback to improve teaching and learning”, Journal of Veterinary Medical Education, Vol. 21, pp. 51-55 Tang, T. L. (1997) “Teaching evaluation at a public institution of higher education: Factor related to the overall teaching effectiveness”, Public Personnel Management, Vol. 26, No. 3, pp. 379-389 Tomasco, A.T. (1980). Student perceptions of instructional and personality characteristics of faculty: A canonical analysis. Teaching of Psychology, 7, 79-82

Wright, D. B. (1997) Understanding Statistics: an introduction for the social sciences, London: Sage

The Journal of American Academy of Business, Cambridge * Vol. 9 * Num. 2 * September 2006

279