Predictors of Academic and Clinical Performance in a ... - CiteSeerX

Predictors of Academic and Clinical Performance in a Baccalaureate Physical Therapy Program JOSEPH A. BALOGUN The purpose of this retrospective study was to determine the best predictors of academic and clinical performance in a physical therapy undergraduate program. The records of 42 graduates of the program were reviewed to obtain data concerning 1) preadmission cumulative grade point averages (GPAs), 2) written composition scores, 3) interview ratings (INTVs), 4) preprofessional faculty ratings, 5) mean Allied Health Professions Admission Test (MAHPAT) scores, and 6) scores on the comprehensive examination administered at the end of the educational program. The results of the comprehensive written and oral-practical examination were used as a measure of the students' academic achievement and clinical performance. Multiple regression analyses revealed that both academic achievement and clinical performance can be predicted reliably ( p < .001) from the preadmission requirements. The two viable predictors of academic achievement were GPA and MAHPAT. The GPA and MAHPAT accounted for 30.5% and 8.0%, respectively, of the total variance (40.6%) in academic achievement. The INTV and GPA were the two viable predictors of clinical performance. They accounted for 34.6% and 7.5%, respectively, of the total variance (44.8%) in clinical performance. Key Words: Education: physical therapist, admissions; Educational measurement.

The number of physical therapy education programs has increased steadily over the years, and staffing demands are projected to remain high.1 Because of the large numbers of qualified candidates competing for limited openings, physical therapy educators increasingly are demanded to document the criteria by which student selection decisions are made. One of the responsibilities of an admission committee is to design an efficient means of identifying candidates who will complete the educational program and become successful physical therapists. Currently, a paucity of literature exists on the admission process of physical therapy education programs.2 In 1976, French and Rezler commented on the "inadequate and poorly validated criteria" used in selecting students into allied health professions programs. They suggested that "the best combination of predictors should be locally established and may even vary from year to year, depending on local differences in curriculum, grading methods and the climate of learning."3 Clinical skill is the "backbone" of physical therapy practice; therefore, determining the preadmission requirement that is the most viable predictor of clinical performance is important. LITERATURE REVIEW

Previous academic achievement generally is assumed to be a fair and reliable indicator of the ability of students to succeed in any educational program. As such, cumulative grade point average (GPA) and standardized (aptitude) test scores are J. Balogun, PhD, is Lecturer 1, Department of Medical Rehabilitation, Faculty of Health Sciences, Obafemi Awolowo University, Ile-Ife, Oyo State, Nigeria, West Africa. He was Assistant Professor, Department of Physical Therapy, Russell Sage College, Troy, NY 12180, at the time this study was completed. This article was submitted March 24, 1986; was with the author for revision 35 weeks; and was accepted April 7, 1987. Potential Conflict of Interest: 4.

238

common criteria used in the selection of students into medical and allied health professions education programs.2-5 Other criteria used include personal interview ratings, personality or interest inventory scores, written composition (ESSAY) scores, letters of recommendation, motor dexterity test scores, and biographical information. Previous investigations have shown consistently that academic course work is the best predictor of scholastic (cognitive) achievement at the end of the professional education.2,6-11 Many studies indicate, however, that academic course work shows much lower or no correlation with clinical grades or performance on the job.3,4 Studies in nursing,12 occupational therapy,13 and medical programs14-16 have revealed weak correlations between academic (preprofessional and professional) course work and clinical performance. On the contrary, studies in physical therapy have shown significant correlation between academic achievement (in preprofessional and professional courses) and clinical performance.6,8,10 The best predictor of clinical performance currently is unclear. Researchers who have investigated the relationship between academic requirements and academic and clinical performance have used the overall GPA in professional courses, grades in clinical internship, or ratings of job performance by employers as the criterion variable.6-16 The use of GPA, clinical grades, and job rating scales as a measure of success, however, has a major psychometric weakness. Because the variables have a narrow range, they may not discriminate adequately among the usually homogeneous pool of candidates for admission to professional programs.3,4 Statement of the Problem

In 1983, Richardson advocated the need for physical therapy education programs to evaluate the proficiency of their graduates.17 In the fall of 1984, we introduced a policy in our PHYSICAL THERAPY

EDUCATION program that requires all physical therapy students to take a comprehensive written and an oral-practical examination at the end of their professional education program. The graduating class of 1985 was the first to take the comprehensive examination. The written component of the comprehensive examination was designed to test the students' theoretical knowledge of the basic and clinical sciences and physical therapy procedures. The oral-practical component of the comprehensive examination was designed to evaluate the students' communication, interpersonal, patient examination, and therapeutic intervention skills. In a recent study, my colleagues and I investigated the best predictor of academic achievement in our program, but we did not evaluate clinical performance at the end of the professional education program.11 The purpose of this retrospective study was to determine the best predictors of academic and clinical performance in our educational program. I hypothesized that the preprofessional GPA would be the most viable predictor of academic achievement and clinical performance in physical therapy.

achievement. The case studies presented during the oralpractical component of the comprehensive examination were limited to musculoskeletal, neurological, and cardiopulmonary content areas. A separate evaluation sheet was used for the three content areas. All evaluation sheets contained a list of specific behaviors to be assessed during the examination (Appendix). Each behavior, when correctly demonstrated or stated, was checked and later converted to numerical scores to reflect the weightings previously assigned to that particular behavior. To avoid bias, experienced clinical physical therapists administered the oral-practical component of the comprehensive examination, and students were assigned randomly to each case study. To control for interrater error, the same examiner evaluated all students assigned to each case study. I used the average of the scores of the different sections of the oral-practical component of the comprehensive examination (physical evaluation, problem and goal definition, therapeutic skills, and communication skills) to represent the students' clinical performance. Data Analysis

METHOD Subjects The subjects for this study consisted of 42 recent graduates of a four-year baccalaureate program in physical therapy. All subjects were admitted into the professional program during the "screening" in the fall term of their sophomore year and began the professional program in the fall of their junior year. The research protocol was approved by the Institution Review Board on the Protection of the Rights of Human Subjects. Procedure Each student's file was reviewed for information used during the screening: 1) overall preprofessional GPAs, 2) ESSAY scores, 3) interview ratings (INTVs), 4) preprofessional faculty ratings (FACEVs), and 5) mean Allied Health Professions Admission Test (MAHPAT) scores. The student's score on the comprehensive written and oral-practical examination also was recorded. The scores on the components of the comprehensive examination were widely distributed, thus, differentiating the students better than the restricted GPA and clinical internship grades. The preprofessional GPA was the mean of the grades for all courses taken in the first two years. The ESSAY was the student's score in a composition examination administered in the sophomore year. The INTV was the average of the ratings in a semi-structured interview. The FACEV was the mean of the ratings provided by two instructors from humanities and basic science departments. The MAHPAT was the average score for the five subsections (verbal ability, quantitative ability, biology, chemistry, and reading comprehension) of the Allied Health Professions Admission Test (AHPAT). The mean score, rather than the subscores of the test, was used because previous investigations showed that the MAHPAT is correlated most highly with academic achievement.11,18 The results of the comprehensive examination were used as a measure of the students' academic achievement and clinical competence at the end of their professional education program. The written component of the comprehensive examination consisted of two parts: 1) a basic-applied sciences section and 2) a clinical application section. The scores on both parts were averaged to represent the students' academic Volume 68 / Number 2, February 1988

In this investigation, the criterion (dependent) variable was defined as the performance on the comprehensive written and oral-practical examination administered at the end of the professional education program. Pearson product-moment correlation coefficients were computed to determine the relationship between the criterion variable and the predictor variables (preprofessional GPA, ESSAY, INTV, FACEV, and MAHPAT). The data were analyzed using multiple and stepwise regression models; both clinical and academic performances were used as dependent variables. The stepwise regression procedure selects the predictor variables in the order of their relative strength in predicting the criterion variable.19 The tolerance was set at an F ratio of 4.00 and at a probability level of .01. RESULTS The MAHPAT score for the subjects in this study was 58.2 (s = 17.5). This value is slightly higher than the normative MAHPAT score of 57.0 for female physical therapy students (J. R. Silvestro, unpublished data, 1983-1984). Presented in Table 1 are the Pearson product-moment correlation coefficients for the relationships between the criterion (academic and clinical performance) variables and independent (predictor) variables. Significant correlations were noted between academic performance and MAHPAT (r = .50, p < .01) and preprofessional GPA (r = .55, p < .01). Clinical performance was significantly related to preprofessional GPA (r = .34, p < .05) and INTV (r = .59, p < .01). The results of the multiple regression analyses for academic and clinical performance are presented in Tables 2 and 3, respectively. The analyses revealed that the predictor variables (ie, admission requirements) can be used reliably (p < .01) to forecast the academic and clinical performance at the end of the professional program. A summary of the stepwise regression analysis for academic performance is presented in Table 4. The results of the analysis revealed that preprofessional GPA is the best predictor of academic performance at the end of the professional program. It accounted for 30.5% of the total variability. The MAHPAT score is also a significant predictor (F = 5.38, p < .01) of academic performance. When MAHPAT was entered into the regression equation containing preprofessional GPA, they 239

both accounted for 38.5% of the total variability. The findings suggest that 8.0% of the variance in academic performance was explained by MAHPAT. All predictor variables combined accounted for 40.6% of the total variability. The findings suggest that FACEV, INTV, and ESSAY have minimal predictive utility. The three variables accounted for only 2.1 % of the variance in predicting academic performance (Tab. 4). Presented in Table 5 is the summary of the stepwise regression analysis for clinical performance. The results of the analysis revealed that INTV and preprofessional GPA were the two viable predictors of clinical performance. Both variables accounted for 42.1% of the total variability. The INTV and preprofessional GPA accounted for 34.6% and 7.5%, respectively, of the variance of predicting clinical performance. All predictor variables combined accounted for 44.8% of the total variability. The findings suggest that ESSAY, MAHPAT, and FACEV accounted for only 2.7% of the increase in the ability to predict clinical performance in an equation containing INTV and preprofessional GPA. DISCUSSION The results of this study revealed that academic performance was significantly related to preprofessional GPA and MAHPAT (Tab. 1). This finding supported the results of a previous investigation by Balogun and colleagues.11 They found significant (p < .01) correlations between academic achievement in the professional courses and preprofessional GPA (r = .63), ESSAY (r =.31), and MAHPAT (r = .28). In this study, however, the relationship between academic achievement and ESSAY (r = .08) was not statistically significant. The correlational findings of this study suggest that preprofessional GPA and MAHPAT may be viable predictors of academic achievement in our program (Tab. 1). The results of the stepwise regression analysis supported this speculation (Tab. 4). The analysis revealed that the most significant predictor of academic achievement was preprofessional GPA. The findings supported the research hypothesis and are consistent with the findings in previous research.2,6-11 The MAHPAT is a viable (p < .01) predictor of academic achievement, but it accounted for less than 8% of the total variance (40.6%). From a practical perspective, the contribution of MAHPAT toward predicting academic achievement is minimal. The MAHPAT is not a viable predictor of clinical performance (Tab. 5). The relationship between preadmission criteria and clinical performance (eg, internship grades, on-the-job ratings) has been a subject of considerable interest in the litera-

ture.6,8,10,12-16 Contrary to previous findings,12-16 the results in this study revealed significant correlations between cognitive admission requirements and clinical performance. Preprofessional GPA and INTV were significantly related to clinical performance (Tab. 1). The results of the multiple regression analysis revealed that clinical performance can be predicted reliably from the admission requirements considered in this study (Tab. 3). One of the primary objectives of this investigation was to determine the best predictor of clinical performance in our

TABLE 1 Correlation (r) Between Academic and Clinical Performance and the Predictor Variables (N = 42) Criterion Variable Predictor Variable

Academic Performance

Clinical Performance

.50a

.03

.55a

.34b

.08

.08

.28c .11

.02 .59a

Mean Allied Health Professions Admission Test Preprofessional grade point average Written composition scores Preprofessional faculty evaluation Interview ratings a

r value significant at the .01 level of confidence. r value significant at the .05 level of confidence. c r value significant at the .10 level of confidence. b

TABLE 2 Analysis of Variance for the Multiple Regression Equation for Academic Performance Source

df

SS

MS

F

P

Regression Residual

5 36

616.3 902.2

123.3 25.1

4.92

002

TABLE 3 Analysis of Variance for the Multiple Regression Equation for Clinical Performance Source

df

SS

MS

F

P

Regression Residual

5 36

2083.3 2569.0

416.7 71.4

5.84

.0005

TABLE 4 Summary of the Stepwise Regression Analysis for Academic Performance Showing Changes in Zero-Order (r) and Multiple Regression (R) Correlation Coefficients with Addition of Different Variables Predictor Variable Preprofessional grade point average (GPA) GPA + Mean Allied Health Profession Admission Test (MAHPAT) GPA + MAHPAT + FACEVb + INTVc + ESSAYd

Ra

R2

.55

.31

.62 .64

R2 Change

F

df

P

30.52

17.57

1,40

.001

.39

7.96

12.19

2,39

.001

.41

2.11

4.92

5,36

.001

(%)

a

R = r when only one predictor variable is being considered. FACEV = preprofessional faculty evaluation. c INTV = interview ratings. d ESSAY = written composition scores.

b

240

PHYSICAL THERAPY

EDUCATION TABLE 5 Summary of the Stepwise Regression Analysis for Clinical Performance Showing Changes in Zero-Order (r) and Multiple Regression (R) Correlation Coefficients with Addition of Different Variables Predictor Variable Interview ratings (INTV) INTV + preprofessional grade point average (GPA) INTV + GPA + ESSAYb + MAHPATC + FACEVd a b c d

Ra

R2

.59

.35

.65 .67

R2 Change

F

df

P

34.62

21.18

1,40

.001

.42

7.46

14.17

2,39

.001

.45

2.70

5.84

5,36

.001

(%)

R = r when only one predictor variable is being considered. ESSAY = written composition scores. MAHPAT = mean Allied Health Professions Admission Test. FACEV = preprofessional faculty evaluation.

program. The results in Table 5 showed that the most viable predictor of clinical performance was INTV. This finding did not support the research hypothesis and is inconsistent with the previous reports in physical therapy.6,8,20 Wiesseman found that preprofessional GPA and age were the strongest predictors of success in the professional board examination in physical therapy.20 The GPA and age accounted for 35% of the total variance (49%) found in Wiesseman's study. Peat et al found preprofessional GPA as the most viable predictor of clinical performance.8 It accounted for 9% of the total variance (12%) found in their study. Tidd and Conine reported the strongest correlation (r = .43) between clinical performance and GPA in physical therapy courses.6 Clinical performance also was significantly (p < .001) related to overall GPA (r = .39), preprofessional GPA (r = .29), GPA in biological and physical sciences (r = .28), and GPA in behavorial sciences (r = .23). The previous studies6, 8, 20 did not consider the same predictor variables in their design; thus, their findings cannot be compared objectively. Educational Implications College grades in preprofessional and professional courses and standardized aptitude test (AHPAT) scores, which are representative of cognitive measures, have been considered in previous studies in physical therapy.6-11,20 In this study, a noncognitive measure (INTV) was the most reliable predictor of clinical performance. It accounted for 34.6% of the variance in clinical performance. Cognitive knowledge (preprofessional GPA) accounted for 7.5% of the total variability (p < .01). This finding suggests that noncognitive characteristics and previous academic achievement have relevance as predictors of clinical performance. During the clinical practicum, students were expected to demonstrate knowledge in all three behavioral (cognitive, psychomotor, and affective) domains. The cognitive domain includes theoretical knowledge and problem-solving skills. The psychomotor domain deals with the student's manipulative and performance skills during "patient" evaluation and therapeutic procedures. The affective domain evaluates the student's attitudes and rapport with the patient (Appendix). In the current changing health care system, physical therapists are required to perform, in addition to the "traditional" clinical functions, more diverse activities such as organizing in-service training, staff supervision, and health promotion. The inclusion of the three behavioral domains in any instrument designed to evaluate clinical competency in physical therapy, therefore, is crucial. The five independent variables considered in this study accounted for 40.6% and 44.8% of the variance in predicting Volume 68 / Number 2, February 1988

academic and clinical performance, respectively. The unexplained variance (59.4% and 55.2% for academic and clinical performance, respectively) may be attributable to other cognitive admission criteria (high school grades and aptitude test scores) and noncognitive admission criteria (motor skills, motivation, attitude, and personality) not considered in this study. Because physical therapy is both a "science" (academic knowledge base) and an "art" (techniques of clinical practice), considering both cognitive and noncognitive measures is important when admitting students into physical therapy education programs. The predictive power of the subjective variables (ESSAY, INTV, and FACEV) can be improved by refining the measurement instruments.3,4,21,22 We recently standardized the method of administration of the INTV and revised the FACEV questionnaire. We shortened the INTV and FACEV rating scales by removing the items we considered ambiguous and increased the spread (range of continuum) of the Likert scales. We also modified our criteria for grading the ESSAY. The changes made in the measurement instruments were based on the findings of previous research by Balogun and colleagues11 and Balogun.23 We recently used the modified measurement instruments to test candidates for admission to our program and obtained interrater reliability coefficients of 49 (p < .01) and .84 (p < .001) for ESSAY and INTV, respectively. The r value for ESSAY is low, probably because some of the items included in the grading criteria are ambiguous and difficult to decipher and quantify. It is plausible that the refinement of our measurement instruments would improve the contribution of the subjective variables in predicting academic and clinical performance in our program. A followup study is needed to support this speculation. Limitations of the Study The findings of this study should be applied with caution because of the small sample size. The high unexplained variance in predicting academic achievement (59.4%) and clinical performance (55.2%) limits the generalization of the results of this study. Additional research is warranted before more conclusive recommendations as to the best predictor of success in physical therapy can be made. The reliability and validity of the instruments used in evaluating the proficiency of our graduates have not been determined. The test questions for the written component of the comprehensive examination were similar to those published by Hershey and Seibert24 and Hershey25 and cover all aspects of physical therapy; however, empirical documentation of its content validity is needed. 241

Implications for Future Research To date, few research studies relating to the prediction of academic and clinical performance in physical therapy have been published. In view of the increasing costs in graduate education17 and the desire of the profession to raise the entrylevel requirement to postbaccalaureate degree, additional research to determine the best predictors of academic and clinical performance in physical therapy is crucial to inspire the trust of both the public and the college administration. Results of such studies will be helpful in admitting the next generation of students into physical therapy education programs. Further studies are needed to determine the interrater and intrarater reliability and validity of the instrument used in our program for grading the oral-practical component of the comprehensive practical examination. Results of such studies may strengthen the validity of the findings in this study. Further interrater reliability studies are also needed after the interview forms are revised. These findings suggest that academic achievement at the time of graduation may be a viable predictor of future clinical competence; however, follow-up research is needed to determine the relationship between students' academic achievement and performance in actual practice. The contribution of biographical information and noncognitive factors (such as motivation, personality, and motor skills) toward predicting success was not evaluated in this study. This area might form the basis of further research. CONCLUSION A significant correlation exists between academic achievement and MAHPAT, preprofessional GPA, and FACEV. Clinical performance was significantly related to FACEV and INTV. The strongest predictors of academic achievement and clinical performance in the program studied were preprofessional

GPA and INTV, respectively. The findings suggest that prediction of success in physical therapy may be enhanced by considering cognitive and noncognitive measures. Acknowledgments. I express my appreciation to Dean Ursula Sybille Colby for providing the funds for computer analysis and a search of the literature. I give special thanks to John R. Silvestro, PhD, of The Psychological Corporation for providing literature on the reliability and validity of the AHPAT. Finally, I thank Fran Delaney for her contribution in the development of the oral-practical evaluation sheet and the multiple choice questions of the comprehensive examination.

APPENDIX Summary of Criteria for Grading Oral-Practical Examinationa Lower Quarter Screening History and interview, initial observation, active and passive lumbar spine movements, neurological tests, specific tests (straight leg raising, neck flexion, Ely's test), provocation tests for sacroiliac joint Specific Joint Evaluation Active and passive resisted movement tests, anthropometric measurements, special tests (stress, Apley's distraction and compression, Draws', McMurray's, Grind's, Tinel's sign) Patient Problem List and Therapeutic Goals Assessment for appropriateness, specificity to problem, logical progression, and comprehensiveness Treatment and Home Program Activities Assessment for appropriateness, specificity to problem, logical progression, comprehensiveness, and precautionary measures observed during treatment Communication Skills and Affective Behaviors Assessment for patient setup and instruction, body mechanics, and professional presentation a The format of the evaluation sheet has been changed to facilitate reproduction. Interested readers can write to the author for an original copy of the evaluation sheet.

REFERENCES 1. Johnson GR: Twentieth Mary McMillan lecture: Great expectations—A force in growth and change. Phys Ther 65:1690-1695,1985 2. McGinnis ME: Admission predictors for pre-physical therapy majors. Phys Ther 64:55-58, 1984 3. French RM, Rezler AG: Student selection in four-year programs. In Ford CW, Morgan MK (eds): Teaching in the Health Professions. St. Louis, MO, C V Mosby Co, 1976, pp 94-110 4. Chaisson GM: Student selection: Logic or lottery. J Allied Health 5:7-16, 1976 5. Gartland GJ: Synopsis of a study of admissions criteria for physical therapy programs. Physiotherapy Canada 29:6-10,1977 6. Tidd GS, Conine TA: Do better students perform better in the clinic? Relationship of academic grades to clinical ratings. Phys Ther 54:500505, 1974 7. Pickles B: Correlations between matriculation entry requirements and performance in the diploma program in physical therapy at the University of Alberta. Physiotherapy Canada 29:249-253,1977 8. Peat M, Woodbury MG, Donner A: Admission average as a predictor of undergraduate academic and clinical performance. Physiotherapy Canada 34:211-244,1982 9. Kerr KM: Pre-entry requirements and academic performance in primary degree courses in physiotherapy at the Ulster Polytechnic. Physiotherapy 71:468-472,1985 10. Olney SJ: Prediction of clinical competence of students of physiotherapy. Physiotherapy Canada 29:254-258,1977 11. Balogun JA, Karacoloff LA, Farina NT: Predictors of academic achievement in physical therapy. Phys Ther 66:976-980,1986 12. Taylor CW, Nahm H, Loy L, et al: Selection and Recruitment of Nurses and Nursing Students: Review of Research Students and Practices. Salt Lake City, UT, University of Utah Press, 1986

242

13. Anderson HE, Jantzen AC: A prediction of clinical performance. Am J Occup Ther 19:76-78, 1965 14. Benor DE, Hobfoll SE: Prediction of clinical performance: The role of prior experience. J Med Educ 56:653-658,1981 15. Hobfoll SE, Benor DE: Prediction of student clinical performance. Med Educ 15:231-236,1981 16. Hobfoll SE, Anson O, Antonovsky A: Personality factors as predictors of medical student performance. Med Educ 16:251-285,1982 17. Richardson RW: Thinking for tomorrow. Phys Ther 63:1795-1801,1983 18. Adams JR, Skinner DM: Can the AHPAT help evaluate comparable GPA from various academic institutions? Laboratory Medicine 11:258-264, 1980 19. Kim J, Kohout FJ: Multiple regression analysis. In Nie NH, et al (eds): Statistical Package for the Social Sciences, ed 2. New York, NY, McGrawHill Book Co, 1975, p 340 20. Wiesseman J: Does the AHPAT Add Enough Predictive Ability to the College GPA to Justify Its Use? Cleveland, OH, The Psychological Corporation (a subsidiary of Harcourt Brace Jovanovich Inc), 1984 21. Shepard KF: Use of small group interviews for selection into allied health educational programs. J Allied Health 9:85-94,1980 22. Dietrich MC: Putting objectivity in the allied health student selection process. J Allied Health 10:226-239,1981 23. Balogun JA: Predictive validity of the Allied Health Professions Admission Test. Physiotherapy Canada 39:39-42,1987 24. Hershey RA, Seibert HK: Physical Therapy Examination Review Book: Basic Sciences, ed 4. New Hyde Park, NY, Medical Examination Publishing Co Inc, vol 1,1984 25. Hershey RA: Physical Therapy Examination Review Book: Clinical Application, ed 2. New Hyde Park, NY, Medical Examination Publishing Co Inc, vol 2, 1973

PHYSICAL THERAPY