Assessment of student performance in problem-based

Research papers

Assessment of student performance in problem-based learning tutorial sessions Rosamar{Âa Valle, lleana Petra, Adrian Mart{Ânez-GonzaÂlez, Jose Antonio Rojas-Ramirez, Sara Morales-Lopez & Beatriz Pin~a-Garza

1,2 Objectives To assess student performance during tutorial sessions in problem-based learning (PBL). Design A 24-item rating scale was developed to assess student performance during tutorial sessions in problem-based learning (PBL) as conducted during the pre-clinical years of Medical School at the National Autonomous University of Mexico. Items were divided into three categories: Independent study, Group interaction and Reasoning skills. Fourteen tutors assessed 152 ®rst and second-year students in 16 tutorial groups. An exploratory factor analysis with an Oblimin rotation was carried out to identify the underlying dimensions of the questionnaire.

Results Factor analysis yielded four factors (Independent study, Group interaction, Reasoning skills, and Active participation) which together accounted for 76á6% of the variance. Their Cronbach reliability coef®cients were 0á95, 0á83, 0á94 and 0á93, respectively, and 0á96 for the scale as a whole. Conclusions It was concluded that the questionnaire provides a reliable identi®cation of the fundamental components of the PBL method as observable in tutorial groups and could be a useful assessment instrument for tutors wishing to monitor students' progress in each of these components.

Setting Medical School at the National Autonomous University of Mexico.

Keywords Problem-based learning; medical students; undergraduate medical education, *standards; educational measurement, *standards.

Subjects Medical students.

Medical Education 1999;33:818±822

Introduction The Faculty of Medicine at the National Autonomous University of Mexico (UNAM), founded in 1579, is the oldest medical school in America. It has a total undergraduate population of 5570 students and annually admits approximately 950 students. The undergraduate programme has a duration of six years (the ®rst two include mainly basic sciences followed by two years of clinical medicine, a year of internship and a year of medical practice in a community. A traditional, lecture-based approach is generally used in most of the courses. However, the latest curriculum revision, carried out in 1993, suggests the use of innovative teaching methods in medicine, such as problem solving, tutorial teaching, and interdisciplinary learning.1 Facultad de Medicina, DireccioÂn General de EvaluacioÂn Educativa, CoordinacioÂn del Sistema de Universidad Abierta y EducacioÂn a Distancia, Universidad Nacional AutoÂnoma de MeÂxico. Correspondence: Dr RosamarõÂa Valle. DireccioÂn General de EvaluacioÂn Educativa. Edi®cio de los Consejos AcadeÂmicos, Circuito Exterior, Ciudad Universitaria, Mexico, DF, 04510

818

Educational strategies based on problem-based learning methods have been used in some of the courses in the ®rst four years with the following objectives: (a) introducing a self-learning method using a problemsolving approach (b) promoting active student participation and (c) helping students to acquire reasoning and group interaction skills. To achieve these goals, students assemble in tutorial groups of nine assisted by one staff tutor and are given a medical problem to analyse. The ®rst session begins with a case presentation followed by clari®cation of concepts and facts that are new to the students. Students are then asked to identify and analyse the problem, and attempt to formulate hypotheses based on their previous knowledge. After this, students determine learning objectives and resources required to obtain the necessary information. After a period of 38±72 h during which they locate and collect information, students meet with their tutor for a second session to present their ®ndings, analyse the information they have gathered and discuss the case. This method is based on problem-based learning strategies as proposed by Schmidt,2 involving the

Ó Blackwell Science Ltd M E D I C A L ED U C A T I ON 1999;33:818±822

Assessment of student performance in PBL tutorial sessions

·

R Valle et al.

following seven steps: (l) clarifying terms and concepts not readily comprehensible; (2) de®ning the problem; (3) analysing the problem; (4) drawing up a systematic inventory of the explanations deduced from step three; (5) formulating learning objectives; (6) collecting additional information outside the group and (7) synthesizing and testing the newly acquired information. The UNAM's Faculty of Medicine has felt it necessary to assess students' progress during tutorial sessions and the achievement of PBL objectives. According to Hay,3 the evaluation of PBL students should be based on actual performance in their learning setting. This is where they should learn to make relevant hypotheses, identify their learning needs, use appropriate learning resources, provide evidence of learning and act as a responsible team member of the tutorial group. However, it has proved dif®cult for tutors to judge whether their students have acquired the desired competencies based on evidence obtained during tutorial sessions. Some tutors tend to ask open-ended questions which, since they are not very speci®c, provide little opportunity for accurately determining students' individual levels of understanding or to make comparisons.4 Some instruments evaluate group learning skills, knowledge and critical thinking;5 others refer to responsibility, information acquisition, communication, critical sense and self-assessment. The extensive reviews of PBL carried out by Albanese & Mitchell6 and Vernon & Blake7 do not include studies related to the assessment of student performance in tutorial sessions. Des Marchais & Vu8 assessed tutor rating instruments for evaluating students' communication, reasoning and group interaction skills, autonomy and humanism. They concluded that the validity of these types of instruments is still psychometrically questionable. With the exception of HeÂbert & Bravo's Tutotest (1996) no other standardized instruments were found that are designed to assess medical students' attitudes and skills acquired as a result of participating in PBL tutorials. Through factor analysis of their 44-item questionnaire, the authors identi®ed four factors: effectiveness within the group, communication and leadership skills, scienti®c curiosity and respect for peers. Despite the relatively low response (67á5%) and tutor participation (70%) rates, and variation in the number of times that some of the students were evaluated, the test was found to be a reliable and practical instrument, useful for evaluating students' skills and attitudes during tutorials. The aim of this study was to develop an instrument to assess student performance as a result of participation in PBL tutorial sessions, and evaluate its construct validity and reliability. It is part of a project that

819

includes the development of instruments and methods to evaluate PBL outcomes at UNAM's Faculty of Medicine.

Methods A team of medical teachers, psychologists and education specialists was established to develop an instrument enabling tutors to assess students' attitudes and competencies acquired through their participation in PBL tutorial sessions. This team included physicians trained in PBL educational strategies at the Schools of Medicine at McMaster University, Michigan State University and the University of Limburg, Maastricht. These physicians trained and supervised the rest of the team as well as other teachers applying PBL. The behaviours to be analysed were closely linked to those involved in PBL2,9±11 and the UNAM faculty's experience in PBL. Such behaviours were divided into three categories: Independent study, Group interaction skills and Reasoning skills. These categories were used as frame of reference for selecting the items to be included in the questionnaire. Certain criteria had to be met: all items should describe behaviours related to PBL, be observable by the tutor during the sessions, be stated in positive terms and unambiguously, and clearly belong to only one category. In keeping with this procedure, 39 items were written and their face validity analysed by a group of 12 teachers who were asked to assign each item to a category according to its content. Of these, 21 were accepted by at least 11 teachers, eight had to be rewritten and 10 were eliminated because they were apparently not relevant or their content had already been included in other items. The degree of clarity of the 29 remaining items was rated on a ®ve-point scale by ®ve independent referees familiar with the PBL method. As a result, minor corrections were made in six of the items. Lastly, these 29 items were included in the questionnaire in random order. They describe behaviours that the tutor has to rate on a six-point scale ranging from never (1) to always (6) to indicate current frequency. The tutors who participated in this study were teachers at the Faculty of Medicine who had participated in training sessions on PBL fundamentals, strategies and evaluation. In addition, they had been using this method of instruction for the previous two years and were trained in the use of this instrument. Assessments were performed at the end of the second tutorial session. An exploratory factor analysis was carried out to identify the underlying aspects. A principal component analysis was followed by an Oblimin rotation. This

Ó Blackwell Science Ltd ME D I C AL ED U C AT I ON 1999;33:818±822

820

Assessment of student performance in PBL tutorial sessions

·

R Valle et al.

procedure was chosen on the assumption that such components would not be independent.12,13

Results Fourteen tutors evaluated 152 ®rst- and second-year students in 16 tutorial groups. The sample was made up of 48á3% men and 51á7% women with a mean age of 19 years. All students attended PBL tutorials for at least 12 weeks before the study and agreed to participate in the study. The questionnaire was given once after each second tutorial group session and all 14 tutors returned the questionnaires. However, given that only fully completed questionnaires could be included in the analysis, 12 had to be eliminated, meaning that 140 questionnaires remained for analysis (Table 1). The initial factor analysis of the 29 item scale showed that two items had low load values (< 0á40) while three

loaded substantially on more than one factor (< 0á45). These ®ve items were therefore eliminated. Factor analysis of the remaining 24-item pool yielded 4 factors 3 with eigenvalues < 1á0 which together accounted for 76á6% of the variance, with each one accounting for 53á9%, 11á4%, 5á9% and 5á4%, respectively (Table l). On the basis of their content, factors were designated as Independent study, Group interaction, Reasoning skills and Active participation. Their Cronbach reliability coef®cients were 0á95, 0á83, 0á94 and 0á93, respectively, and the total scale reliability was 0á96. The ®rst factor, Independent study, comprises nine items linked to students' initiative, motivation and persistence in searching for information, studying and achieving the learning objectives and tasks agreed on by the group. Inter-item correlations of this subscale ranged from 0á57 to 0á91. The ®ve items referring to factor 2 (Group interaction) concern students' abilities to function in a group, such as openness to suggestions

7 Table 1 Factor analysis of students performance in PBL tutorial sessions questionnaire* Factor loads Factor 1 Independent study

Items 25. Uses different resources to obtain needed information 26. Uses additional references to those suggested by the programme 21. Presents well-organized information relevant to the case 28. Is persistent in the study of the case 27. Is motivated to know more 24. Implements activities to achieve the learning objectives 29. Shows initiative in the study of the case 23. Shows initiative in the search of information 9. Accomplishes tasks agreed by the group

Factor 2 Group interaction

0á968 0á836 0á824 0á812 0á808 0á775 0á717 0á706 0á653

) 0á004 ) 0á048 0á128 ) 0á120 ) 0á124 0á062 ) 0á160 0á067 0á195

5. Accepts suggestions about his/her work 2. Adjusts to different group roles 1. Shows respect to his/her peers 11. Accepts decisions made by the group 6. Listens attentively to other members of the group

) 0á132 0á056 0á101 ) 0á066 0á146

0á813 0á777 0á761 0á755 0á689

19. 16. 20. 15. 13. 14.

) 0á055 0á025 0á148 0á180 0á209 0á299 0á105 0á200 0á197 0á171

7. 3. 4. 8.

Identi®es his/her learning needs Asks questions related to the case Accepts or rejects hypothesis on well-based foundations Analyses different components of the case Clari®es facts, concepts and terminology Identi®es information relevant to the case Gives feedback (re¯ections, ideas and suggestions) to the group Helps her/his peers to clarify ideas Participates in case discussions Shares knowledge with the group

Eigenvalues Contribution to variance (%)

12á93 53á9

Factor 3 Reasoning skills

Factor 4 Active participation

0á103 0á011 0á145 0á138 0á107 0á074 0á215 0á113 0á092

0á012 ) 0á096 0á298 ) 0á132 ) 0á134 ) 0á161 ) 0á158 ) 0á193 0á012

0á086 ) 0á175 ) 0á130 ) 0á119 0á157

) 0á306 ) 0á023 0á217 ) 0á015 ) 0á094

0á146 ) 0á017 0á043 ) 0á087 0á022 ) 0á021

) ) ) ) ) )

0á882 0á812 0á769 0á669 0á578 0á498

) ) ) ) )

0á187 0á198 0á055 0á285 0á307 0á316

0á253 0á141 ) 0á029 0á136

) ) ) )

0á073 0á027 0á275 0á244

) ) ) )

0á741 0á740 0á639 0á627

2á74 11á4

) ) ) ) ) )

1á42 5á9

1á29 5á4

* Data derive from observations on 140 students made by 14 staff tutors. Ó Blackwell Science Ltd M E D I C A L ED U C A T I ON 1999;33:818±822

Assessment of student performance in PBL tutorial sessions

·

R Valle et al.

Table 2 Factors correlation matrix Factors Items Factor Factor Factor Factor

1: Independent study 2: Group interaction 3: Reasoning abilities 4 Active participation

1

2

3

0á343* 0á767* 0á734*

0á348* 0á430*

0á739*

4

n = 140. * p < 0á000, one-tail test.

and decisions, adjustment to different group roles and respect towards their peers. Inter-item correlations range from 0á57 to 0á66. The six items included in factor 3 (Reasoning skills) are related to students' ability to analyse cases, formulate hypotheses, and clarify concepts. The inter-item correlations ranged from 0á65 to 0á88. The last factor was called Active participation because the four items included re¯ect a speci®c form of interacting with the group that includes behaviours such as contributing, helping and sharing re¯ections, ideas and knowledge. Here inter-item correlations ranged from 0á69 to 0á83. Pearson product-moment correlations were computed to assess the interrelations between the four factors (Table 2). Although all correlations were signi®cant (p < 0á000, one-tail test), the highest were found between Reasoning skills and Independent study, and between Active participation and Reasoning skills and Independent study.

Discussion The implementation of educational innovations in the curriculum, such as problem-based learning, makes it essential to evaluate processes and outcomes and to develop the necessary instruments to achieve this goal. As part of these evaluation objectives, it was decided to elaborate and assess a standardized questionnaire that could be used by PBL tutors to monitor students' progress in the achievement of PBL objectives. Results of the analysis of the questionnaire yielded four factors re¯ecting essential components of problembased learning: Independent study, Group interaction, Reasoning skills, and Active participation.2,9±11 Identi®cation of these aspects enables tutors to assess PBL components separately or as a whole, as required. They may ®nd it useful for assessing their students during the process (formative evaluation) or at the end

821

of the course or period (summative evaluation), or for evaluating the extent to which changes in attitudes and target skills are being achieved through this educational strategy. The ®rst factor (Independent study) reveals an underlying strategy crucial to PBL; the training of students to manage their own learning. Two of the items (9 and 21) included in this factor had originally been assigned to the Group interaction and Reasoning skills categories, respectively. However, the content of these two items would seem to be more the result of Independent study than of the other factors. The second factor (Group interaction) permits the identi®cation of fundamental attitudes for team work, another essential component of PBL. The ability to analyse problems and propose or reject well-founded hypotheses is another fundamental aim of PBL expressed in the items comprising Reasoning skills. Finally, it is worth pointing out the distinction made between two factors: Group interaction and Active participation in the group suggest attitudes of co-operation and collaboration in the process of analysing the problem. The items included in the Group interaction and Active participation factors were originally conceived as part of a single category, but results have shown that two different aspects are involved. At the same time, the fact that the tutor is in a position to choose the type of group interaction he wishes to encourage will allow him to foster collaboration instead of competition, which is in keeping with the policies of both McMaster University14,15 and UNAM'S Medicine Faculty, which seek to encourage team work among students. On the other hand, it has been noted that the tutor should not intervene in his students evaluation since this may alter the social structure of a tutorial group.16 However, to determine the nature and extent of this in¯uence it would be necessary to carry out more research on the subject. The results of the correlation analysis show that the factors are signi®cantly interrelated, particularly Independent study, Reasoning skills and Active participation. Conversely, there is only slight correlation between Group interaction and Independent study, Reasoning skills and Active participation. These two sets are linked to intellectual activities and socio-affective activities, respectively, both crucial to the achievement of PBL objectives. The results obtained con®rm the construct validity of the questionnaire and the reliability (internal consistency) of each factor. Unlike the questionnaire devised by HeÂbert & Bravo,17 the items comprising each factor only have a high factor load in one of them. However, this study constitutes the initial phase of the

Ó Blackwell Science Ltd ME D I C AL ED U C AT I ON 1999;33:818±822

822

Assessment of student performance in PBL tutorial sessions

·

R Valle et al.

psychometric evaluation of the questionnaire. Further studies are indicated not only to corroborate the results obtained with the questionnaire but also to determine the link between PBL and students' performance in the acquisition of knowledge and other skills required for physicians' training. Lastly, the questionnaire may be used for the following purposes: to track the evolution of attitudes and skills during tutorials, to serve as a tool for providing useful feedback for students and to evaluate overall tutorial group performance at the end of the course.

Acknowledgements We would like to thank Dr Enrique PinÄa for his support to this project.

References 1 De la Fuente JR, PinÄa-Garza E, Gutierrez-Avila JH. La formacioÂn del meÂdico del siglo XXI y el Plan Unico de Estudios de la Facultad de Medicina de la Universidad Nacional AutoÂnoma de MeÂxico. EducacioÂn MeÂdica Y Salud 1994;28:331±40. 2 Schmidt HG. Problem-based learning: rationale and description. Med Educ 1983;17:11±6. 3 Hay J. Tutorial performance. In: Evaluation Methods: a Resource Handbook (eds G Norman, S Shannon.). Hamilton Ontario: McMaster University, 1995. 4 Graesser A, Person N. Questions asked during tutoring. Am Educational Res J 1994;31:104±37. 5 Woodward CA. Some re¯ections on evaluation of outcomes of innovative medical education programmes during the practice period. Ann Community-Oriented Educ 1992;5:181±91.

6 Albanese M, Mitchell S. Problem-based learning: a review of literature on its outcomes and implementation issues. Acad Med 1993;68:52±81. 7 Vernon D, Blake R. Does problem-based learning work? A meta-analysis of evaluative research. Acad Med 1993;68:550±63. 8 Des Marchais JE, Vu NV. Developing and evaluating the student assessment system in the pre-clinical problem-based curriculum at Sherbrooke. Acad Med 1996;71:274±83. 9 Barrows HS, Tamblyn RM. Problem-Based Learning: an Approach to Medical Education. New York: Springer, 1980. 10 Barrows HS. How to Design a Problem-Based Curriculum for the Pre-Clinical Years. New York: Springer, 1985. 11 Walton HJ, Matthews MB. Essentials of problem based learning. Med Educ 1989;23:542±58. 12 Tabachnik B, Fidell S. Using Multivariate Statistics. New York: Harper & Row, 1983. 13 Floyd F, Widaman K. Factor analysis in the development and re®nement of clinical assessment instruments. Psychol Assessment 1995;7:286±99. 14 Pallie W, Carr DH. The McMaster medical education philosophy in theory, practice and historical perspective. Med Teacher 1987;9:59±71. 15 Neufeld VR, Barrows HS. The McMaster philosophy: an approach to medical education. J Med Educ 1974;49:1040±50. 16 Van der Vleuter C, Verwijnen M. A system for student assessment. In: C Van der Vleuter, W Wijnen. eds. Problem± Based Learning: Perspectives from the Maastricht Experience. Amsterdam: CIP DATA Koninklijke Bibliotheek, Den Haag, 5 pp 27±49, 1990. 17 HeÂbert R, Bravo G. Development and validation of an evaluation instrument for medical students in tutorials. Acad Med 1996;71:488±94. Received 21 July 1997; editorial comments to authors 28 October 6 1997; accepted for publication 22 January 1999

4

Ó Blackwell Science Ltd M E D I C A L ED U C A T I ON 1999;33:818±822