The Process of Problem-Based Learning

The Process of Problem-Based Learning Elaine H. J. Yew

Copyright © 2009 E. H. J. Yew All rights reserved. No part of this thesis may be reproduced or transmitted in any form, by any means, electronic or mechanical, without the prior written permission of the author, or where appropriate, of the publisher of the articles. Cover design and layout by Wong Yin Tiong Printed by Ruby Printing Pte Ltd, Singapore

The Process of Problem-Based Learning Het proces van probleemgestuurd leren

Thesis to obtain the degree of Doctor from the Erasmus University Rotterdam by command of the rector magnificus Prof.dr. S.W.J. Lamberts and in accordance with the decision of the Doctorate Board The public defence shall be held on Tuesday 2nd of June 2009 at 13.30 hrs by Elaine Hau Jin Yew born in St. Albans, England

Doctoral Committee Promotor:

Prof.dr. H.G. Schmidt

Other members:

Prof.dr. R.M.J.P. Rikers Prof.dr. H.T. van der Molen Dr. D.H.J.M. Dolmans

Dedicated to my wonderful parents

Table of Contents Acknowledgements

9

Chapter 1:

Introduction

11

Chapter 2:

Evidence for constructive, self-regulatory, and collaborative processes in problem-based learning

21

Chapter 3:

How students learn in problem-based learning: A process analysis

53

Chapter 4:

Do the contents of learning activities in problembased learning predict student achievement?

83

Chapter 5:

Is learning in problem-based learning cumulative?

103

Chapter 6:

Summary and conclusions

127

References

151

Appendices

161

Curriculum Vitae

165

Acknowledgements I am very grateful to many people who have helped me during the course of my study. First and foremost, Professor Henk Schmidt, my promotor − thank you very much for your encouragement, guidance and time throughout the past three years. I am very privileged to have such a knowledgeable supervisor, who is at the same time so patient, helpful and kind. Your unwavering support and belief in what we can do has motivated me to persevere to the end! I am also very grateful to Dr W.A. M. Alwis for his help, advice and support in the course of my research. My reporting officers, Dr Terence Chong and Mr Ashley Chua have also given me their invaluable support and encouragement. Thank you very much for your patience and understanding. This study would also not be possible without the support of our current principal & CEO of Republic Polytechnic, Mr Yeo Li Pheow as well as our ex-principal & CEO, Professor Low Teck Seng. Thank you very much for your continual support. I am also very thankful for my immensely patient and generous colleagues, in particular, Mr Eric Kwek, Dr Wong Luh Cherng, Ms Nor Haslinda Mehad, Ms Maisarah Abu Samah as well as other staff from the School of Applied Science, Office of Information Services and Centre for Educational Development, Republic Polytechnic. Thank you so much for your assistance with classroom observations and other technical logistics required for my data collection and processing. I am thankful to have helpful colleagues and teammates − Mr Jerome Rotgans, Ms Magdeleine Lew, Ms Judith Williams and Ms Nachamma Sockalingam, to work together in the process of our research studies. Last but not least, a big thank you to my family − my husband, my parents, sisters and brother-in-law, for your constant support, encouragement and love. I’m so blessed to have all of you.

Introduction

Chapter 1: Introduction The theme of this thesis is how learning takes place in problem-based learning (PBL). The objectives of the studies presented were to gain insight into what and how students learn in all the phases of the PBL cycle, as well as to identify relationships between the learning activities of students (that is: what they know, say, and do) with their learning outcomes. Much of the research on PBL has been focused on effects of this approach on curricular outcomes (Dochy, Segers, Van den Bossche, & Gijbels, 2003; Schmidt & Moust, 2000). However, answers to the question of how the process of PBL produces these outcomes are still few. To contribute to our understanding of the nature of learning in PBL, I applied different methodologies to analyze and describe the PBL process in a naturalistic setting. In brief, PBL is a learning approach that seeks to create a learning environment where students learn in the context of meaningful problems, actively constructing mental models in the process, coconstructing ideas with peers in a collaborative fashion and developing self-directed learning skills in the process (Hmelo-Silver, 2004; Norman & Schmidt, 1992; Schmidt & Moust, 2000). Thus, PBL brings together four fairly new insights into learning, namely that learning can be considered a constructive, self-directed, collaborative, and contextual activity (Dolmans, De Grave, Wolfhagen and Van der Vleuten, 2005). This first chapter presents an introduction to PBL, a review of its theoretical background and relevant studies, an overview of the research questions of the study and an introduction to the thesis.

Learning as knowledge constructing Although there are several different versions of constructivism, the key belief in this philosophy is that learning is a process of knowledge construction, hence emphasizing the active role students must play in acquiring knowledge. One constructivist approach is cognitive constructivism which focuses mainly on the individual learner. Here, learning is viewed as the active organization of new relevant experiences

11

12

Chapter 1

into personal mental representations or schemata with the help of prior knowledge (Derry, 1996; Mayer, 1996). The construction of new meanings thus occurs as multiple links are made between information being acquired and existing knowledge of the learner. On the other hand, other constructivist approaches emphasize that learning is inherently social; hence, social interaction and activity are essential in mediating cognition (reviewed by Palincsar, 1998; Vygotsky, 1978). In the socio-constructivist theory, knowledge is viewed as being internalized in an individual as a result of their interacting with the social environment (Fetsco & McClure, 2005). In this theory, the focus is on the cognitive development of an individual within the context of social interactions. This is different from the socio-cultural theory, where learning is not viewed as taking place within an individual but deemed as a process of joint knowledge construction mediated by cultural tools, such as language, gestures, learning resources and computers (John-Steiner & Mahn, 1996; Palincsar, 1998). Hence sociocultural researchers are interested less in individual development but in the relationships between social interaction and individuals’ cognitive development (Visschers-Pleijers, 2006). In this study I assume that both social and individual learning processes are complementary and can be used to describe learning, in particular, PBL (Cobb & Yackel, 1996).

An introduction of problem-based learning The general model of PBL was first developed in medical education in the late 1960's. Since then, it has been adapted and adopted in an increasing number of different fields and education levels including business schools (Milter & Stinson, 1993), schools of education (Bridges & Hallinger, 1992); architecture, law, engineering, social work (Boud & Feletti, 1991); and high school (Kolodner et al., 2003). PBL is of interest to many educators as it provides a structured framework of active and collaborative learning, in line with current understanding of learning as a constructive and co-constructive activity involving social interactions (Glaser & Bassok, 1989; Palincsar, 1998). PBL always starts with a problem, for which students do not prepare beforehand. The problem description is provided to students

Introduction

prior to the problem analysis phase of a PBL tutorial. After the problem is presented to the students, they work as a group to analyze it, generating possible explanatory hypotheses, building on one another’s ideas, as well as identifying gaps in their knowledge with reference to the problem statement. These knowledge deficiencies are known as learning issues for students to explore further. After this period of teamwork, they disperse for a period of individual self-directed study (usually a few days to a week) to work on the learning issues they have identified as a group. They next meet as a team during what is called the ‘reporting phase’ of the PBL tutorial, where they are expected to share and discuss their findings, as well as refine their initial explanations based on what they have learned. Students would then move on to analyze a new problem or if new learning issues requiring further study are identified during this phase, the process described above would be repeated. PBL can thus be viewed as a cyclical process consisting of three phases: initial problem analysis, self-directed individual learning, and a subsequent reporting phase (Barrows, 1988; Schmidt, 1993). A tutor (also known as a facilitator) is present to guide students’ learning in the problem analysis and reporting phases of the PBL tutorial. The facilitator’s role is to facilitate students as they co-construct knowledge through discussions and sharing of ideas (Hmelo-Silver & Barrows, 2006; Schmidt & Moust, 2000). This can be viewed as a type of cognitive apprenticeship where good learning and thinking strategies are modelled for students in the process of probing them to think more deeply and asking them questions that they should be asking themselves when problem-solving (Collins, Brown, & Newman, 1989).

Review of research on PBL Several studies have focused on the problem analysis and reporting phases of PBL. Schmidt, De Volder, De Grave, Moust and Patel (1989), and De Grave, Boshuizen and Schmidt (1996) have demonstrated that students who discussed a problem in a small group before studying a relevant text learned more from that text relative to students who did not have the chance to discuss the particular problem. The opportunity for students to verbalize and elaborate on what they know or think about a subject during the problem analysis phase, before studying relevant

13

14

Chapter 1

resources helped them remember the concepts learned better. Surprisingly however, a study by Moust, Schmidt, De Volder, Belien and De Grave (1987) failed to find a relationship between the amount of verbal elaboration by students and their ability to do well on a test. The study by De Grave et al. (1996) explored the relationship between students’ verbal interaction and their cognitive change during the problem analysis phase of PBL. Although their findings suggest that the quality of what students articulate is likely to be related to their learning outcomes, the authors do not report any learning outcomes. A study by Capon and Kuhn (2004) examined the possible mechanisms by which PBL achieved its positive effect on student learning. Their findings suggest that the benefits of PBL lie in enabling a better integration of new concepts with existing knowledge, thus enhancing students’ understanding. As pointed out by the authors, other aspects of the PBL process, such as the social aspect (e.g. the role of peer collaboration, reporting of findings to peers etc) still require further study. Research investigating the group processes in PBL was carried out by Visschers-Pleijers and colleagues. One study by Visschers-Pleijers, Dolmans, Wolfhagen and Van der Vleuten (2004) analyzed the group interactions in the problem analysis and reporting phases of a PBL tutorial. They found that both elaborations and co-constructions occur during these phases of the PBL cycle although elaborations occurred less often compared to co-constructions. In a further study on students’ interaction processes, Visschers-Pleijers, Dolmans, de Leng, Wolfhagen and Van der Vleuten (2006) investigated how different types of verbal interactions such as cumulative reasoning, exploratory questioning, and handling of ‘cognitive conflicts’ were distributed over the group meetings. This study provided illustrations of the collaborative learning process in PBL and how questions, reasoning and conflict lead to elaborations and co-constructions by the students during the reporting phase of the tutorial. However, descriptions of what happens during the other phases of the PBL cycle were not provided. There was also no report on the relationships with the amount and content of subsequent learning.

Introduction

A naturalistic study of the PBL process by Koschmann, Glenn and Conlee (1997) involved a description and analysis of a segment of a PBL tutorial up to the point where a learning issue was generated. Another publication by the group focused on a detailed but short segment of the interaction in a PBL group to provide insight into how students interact in the process of presenting one’s own theory and responding to those of others (Glenn, Koschmann, & Conlee, 1999). While these studies enhance our understandings of the actual learning-oriented verbal interactions that occur within the PBL tutorial, limitations of these studies are that they only examined specific portions of the PBL tutorial and no relationships between student behaviour and achievement were shown. One recent study by Hmelo-Silver and Barrows (2008) analyzed in detail the knowledge building process in a PBL tutorial throughout both the problem analysis and reporting phase. Both students and facilitator discourse were examined and described to show how both groups played important roles in the collaborative and collective knowledge building. This study provided important insights into how an expert facilitator guided the group discourse with the use of open-ended metacognitive questions, and how students actively worked on enhancing and refining their collective knowledge throughout the group interaction portions of a PBL cycle. However this study again did not relate the quality of students’ verbal contributions with their learning outcomes. The studies cited above all focus on the problem analysis and reporting phases of PBL. Research dealing with the phase of individual, self-directed study is scarce. One study involving the self-directed learning phase was carried out by Dolmans, Schmidt and Gijselaers (1995) where the relationship between student-generated learning issues during problem analysis phase and what students actually studied during self-study time was investigated. Although the general assumption is that students would make use of the learning issues generated to determine their learning activities during self-directed study, this turned out to be the case only to some extent. The learning activities of the students during the self-study phase were found to also be determined by other factors such as the nature of tutor guidance and

15

16

Chapter 1

the learning resources available. Another study by Van den Hurk, Wolfhagen, Dolmans and Van der Vleuten (1999) focused on how students made use of their self-study phase in terms of learning issues previously generated and time spent on individual study. They found that higher-year students were more self-directed learners compared to first year students, and that those who studied beyond the learning issues generated by the tutorial group during problem analysis phase showed better achievement in tests. As both these studies relied on students’ retrospective self-report, the results may have been biased to some extent. The studies reviewed above all tend to focus on specific phases of the PBL cycle. There are fewer studies which investigate the entire PBL process inclusive of all phases. One causal model relating input variables such as the quality of problems, tutor performance and students’ prior knowledge, process variables such as group functioning and time spent on self-directed study, and learning outcomes was tested by Gijselaers and Schmidt (1990). They demonstrated that problem quality influences tutorial group functioning, which in turn had a strong influence on the amount of time spent in individual study. More time put in study also led to increased learning achievements. This model was further refined when Van den Hurk, Dolmans, Wolfhagen, & Van der Vleuten (2001) investigated in more detail what actually happens to learners in the processes of problem analysis, individual study and reporting. Here they found that the quality of learning issues generated during the problem analysis phase had an impact on the extent to which the learning issues were used during individual study. Increased usage of learning issues during self-directed study also influenced students’ research to be more explanation-oriented, which in turn led to a ‘deeper discussion’ during the reporting phase. Finally the ‘depth’ of reporting led to a higher score on an achievement test. Both of these causal models provide insight into the relationships between the variables important in the PBL process. However, as recognized by the authors, a limitation to both studies was that data was obtained from students’ perceptions and retrospective self-report. As argued by Dolmans & Schmidt (2006), and Hak & Maguire (2000), the

Introduction

research required to uncover the relationships between aspects of the tutorial process and students’ learning should be focused on the actual activities occurring in the various phases of PBL. Although a fairly detailed picture regarding the nature of the discussions in the tutorial teams has emerged from the studies reviewed, only one study has been conducted in which a relationship was studied between what is discussed and what is learned. This study by Moust et al. (1987) failed to disclose a relationship between the two activities. Since Moust et al. (1987) concentrated on how much was said in relation to achievement, there is a need for studies that relate what is said to achievement. In particular, in the studies described in this thesis, scientific concepts relevant to understanding the problem-at-hand were used as units of analysis. Since, as Solomon, Medin, and Lynch argued, “concepts are the building blocks of thought” (1999, p. 99), I suggest that the nature and the number of concepts articulated at any moment by a learner give an indication of the extent of his or her current understanding of the problem-at-hand. Secondly, there is a definite lack of research investigating what students actually do during self-directed study and how their activities influence the outcomes of their learning. Although in particular the experimental studies discussed above (De Grave et al., 1996; Schmidt et al., 1989) suggest that group discussion and elaboration play an important role in students’ learning in PBL, the way in which they affect learning remains to be clarified. Furthermore, as suggested, we do not yet know much about students’ learning processes during self-directed study periods, obviously because it is extremely difficult to directly observe a group of students engaged in individual study under naturalistic conditions. Studies on the learning activities and processes during self-directed study time inevitably relied on self-report (Dolmans, Schmidt, & Gijselaers, 1995; Van den Hurk, Wolfhagen, Dolmans, & Van der Vleuten, 1999). Self-reports however, tend to reconstruct activities from memory, hence limiting their validity. From the review above, it can also be seen that the different PBL phases are often studied in isolation, with few studies describing the causal relationships between the different phases of PBL and with learning achievement.

17

18

Chapter 1

Research questions A review of the literature regarding the PBL process above thus raises the following questions, which will be investigated in the studies presented in this thesis: 



 



Modern theories of learning assume that constructive, collaborative, and self-directed activities are important in the learning process. To what extent can these activities be observed in the PBL small-group tutorial? How can the learning process of PBL be described? Are there different phases that can be distinguished within the learning process? What is the relationship between the learning activities of students in PBL and their learning outcomes? What do students actually do during self-directed study and how do these activities influence their learning outcomes? Does learning in each PBL phase build on previous learning and if so, to what extent? What causal relationships exist between these phases that can help to explain the effects of PBL and students’ learning achievements? How can students’ learning be recorded and analyzed as it unfolds? Can an efficient method to capture and quantify students’ learning during the entire PBL process be developed?

Outline of thesis Here I will give a brief outline of the studies in this thesis and how they attempt to answer the research questions raised above. The studies described in Chapters 2 to 5 all seek a response to the research questions stated, albeit to different extents. All research was carried out in an authentic though somewhat unique PBL setting in a polytechnic in Singapore. The educational context here is special in that the problem analysis, self-directed learning and reporting phases of PBL all occur within one day. All students have a personal laptop that can be connected to the internet. First-year students generally rely mainly on internet resources for their research, and tend to remain in class instead of going to the library for their self-directed study. Thus it was possible

Introduction

to record and hence observe students’ learning activities for entire PBL cycles, even during the periods of self-directed study. For the studies in Chapters 2 and 3, the different PBL phases were classified similarly as in the education context used: First meeting (problem analysis phase), first SDL phase, second meeting (actually similar to an SDL phase, except that students meet with the tutor for a 20 minutes discussion in between this period), second SDL phase and lastly, third meeting (reporting phase). However for the studies in Chapters 4 and 5, the different SDL phases were analyzed as one unit since there were no significant differences between the three phases of first SDL, second meeting and second SDL phase. The main aim of the study described in Chapter 2 was to investigate whether and how PBL stimulates students towards constructive, self-directed and collaborative learning. Here the verbal interactions of one PBL group of five students throughout an entire PBL cycle were recorded and the verbatim transcript consisting of more than 1000 utterances was then analyzed. Next the study described in Chapter 3 sought to provide an account of how learning takes place in PBL, and to identify the relationships between the learning-oriented activities of students (the scientific ideas they discuss and encounter during selfstudy) with their learning outcomes. This study made use of voice and computer screen recordings of the entire PBL cycle for nine students, in order to identify relationships between the relevant scientific concepts articulated or studied individually with students’ learning outcomes, and to explore how students acquired different concepts over the different PBL phases. The studies in Chapters 4 and 5 were attempts to estimate the causal relationships between students’ learning activities and their learning outcomes, as well as how the different PBL phases influenced one another. The methodology for the study described in Chapter 4 was similar to that in Chapter 3, but with increased sample size of 35 students. The data here was then studied using structural equation modelling to clarify the relations between the different learning processes in the PBL cycle: the relevance of the verbal contributions during problem analysis phase, of verbal exchanges during self-directed

19

20

Chapter 1

learning, of individual study during self-directed learning, of verbal contributions during the reporting phase, and achievement. My hypothesis was that small-group collaboration is necessary in PBL, that it does influence individual study, and that it eventually influences achievement. For the final study as described in Chapter 5, the central thesis tested was whether learning in the different phases of PBL is cumulative− does the learning in each phase depend on the previous phase? Or are some phases of the PBL process more (or less) important than others? Secondly, I continued to seek to understand how students learn in the different phases of PBL in terms of concept acquisition and elaboration. A third objective was to devise an efficient and valid method to track students’ learning as it unfolds in the course of the PBL process. Here I used a modified word association exercise to quantify the number of relevant scientific concepts recalled by 218 students at the end of each PBL phase as units of analysis to provide an estimate for the quality of students’ learning during that phase. The data were analyzed using a structural equations modelling approach. The final chapter will then summarize and discuss the findings of the preceding chapters. Issues to be further explored by future research will also be suggested.

Learning processes in PBL

Chapter 2: Evidence for constructive, selfregulatory, and collaborative processes in problem-based learning 1 Abstract The goal of this study was to increase our understanding of the learningoriented verbal interactions taking place between students during the problem-based learning (PBL) cycle. The verbal interactions of one PBL group of five students throughout an entire PBL cycle were recorded in this data-intensive case study. The verbatim transcript consisting of more than 1000 utterances was analyzed to investigate whether and how PBL stimulates students towards constructive, self-directed and collaborative learning. Our results demonstrate the occurrence of all above-mentioned learning activities, with 53.3% of episodes being collaborative, 27.2% self-directed and 15.7% constructive.

Introduction Problem-based learning (PBL), as its name implies, always starts with a problem. This problem refers to an academically or professionally relevant issue of which students are supposed to learn more. Students work in small collaborative groups with guidance from a tutor. As students are not given the opportunity to prepare for the problem beforehand, they start their initial discussions based on their prior knowledge. During this time, they analyze the problem, generate possible explanations, build on one another’s ideas, as well as identify key issues to be studied further. This first phase of the learning process is often called the ‘problem analysis phase’. After this period of teamwork, students disperse for self-directed study in order to work on the learning issues identified. When they next meet, they are expected to share and discuss their findings, as well as refine their initial hypotheses 1 Published in: Yew, E. H. J., & Schmidt, H. G. (2009). Evidence for constructive, self-regulatory, and collaborative processes in problem-based learning. Advances in Health Sciences Education 14(2):251-273, DOI: 10.1007/s10459-008-9105-7.

21

22

Chapter 2

based on what they have learned. This phase is called the ‘reporting phase’. A tutor is present during the problem analysis and reporting phases to help facilitate and monitor students’ learning processes. PBL can thus be seen as being made up of three phases: problem analysis, self-directed learning (SDL) and reporting (Barrows, 1988; HmeloSilver, 2004; Schmidt, 1993). Thus PBL seeks to create a learning environment where students learn in the context of meaningful problems, actively constructing mental models, co-constructing ideas with peers in a collaborative fashion and developing self-directed learning skills in the process (Hmelo-Silver, 2004; Norman & Schmidt, 1992). According to Dolmans, De Grave, Wolfhagen and Van der Vleuten (2005), this characterization of PBL brings together four fairly new insights into learning, namely that learning can be considered a constructive, self-directed, collaborative, and contextual activity. Glaser (1991) also argues that the constructive nature of learning requires a focus on the influence of organization and representation of knowledge, on self-regulatory processes, and on the (social) context in which learning takes place. The constructive learning principle emphasizes that learning is an active process of constructing or reconstructing knowledge networks. Expert learners have been found to have a more extensive knowledge base organized coherently around key principles and concepts while novices represent their learning in terms of more surface features (Chi, Bassok, Lewis, Reimann, & Glaser, 1989). Self-regulated learning is a process involving the generation of thoughts, feelings and behaviours that are focused on achieving goals set by the individual (Zimmerman, 2002). Thus, students who exhibit selfregulated or self-directed learning, would demonstrate processes such as goal setting, planning and self-control in terms of time and taskmanagement (Zimmerman, 2002). As students select appropriate learning strategies and course of actions to achieve their goals, they would also be able to monitor and reflect on their own progress, exhibiting a kind of feedback loop in the process (Hmelo-Silver, 2004; Zimmerman, 1990). Research has shown a high correlation between the


students’ use of self-regulatory strategies and their academic achievement (Zimmerman, 1990; Zimmerman & Martinezpons, 1988). Collaborative learning involves student interaction with the purpose of achieving a common learning goal. Many studies have described the potential benefits of working collaboratively in small groups. Hmelo-Silver (2004), for instance, suggests that group work enhances critical thinking and encourages students towards a deep learning approach. Webb (1991) found that giving elaborated explanations within small groups was related to achievement gains although short unelaborated responses benefited neither the helper nor the recipient. Furthermore, it was found that helping behaviour in small group learning was most effective in enhancing learning when it led students towards greater constructive activity (Slavin, 1996; Webb, Troper, & Fall, 1995). As described above, the constructive, self-directed and collaborative learning processes are key features in current theories of learning. Glaser (1991) notes that the social context of learning enables students’ thinking processes, reasoning strategies and learning approaches to become observable. Thus, if PBL is an approach that has these three characteristics, they should be evident in the verbal interactions of students in the process of PBL. To what extent have constructive, self-directed, and collaborative processes been observed in PBL? Initially, research on PBL focused on effects of this approach on achievement. For instance, studies by De Grave, Schmidt and Boshuizen (2001) and Schmidt, De Volder, De Grave, Moust and Patel (1989) demonstrated that elaboration during problem analysis in a small group prior to studying problem-relevant new information led to increased knowledge acquisition and recall. However, answers to the question of how the process of PBL produces these positive outcomes are still few. As argued by Hak and Maguire (2000), process-oriented research that describes and qualitatively analyzes all the relevant phases of the PBL cycle in order to clarify the learning processes essential for students’ learning is lacking.

23

24

Chapter 2

One such process-oriented study was conducted by Koschmann, Glenn and Conlee (1997) where a segment of the PBL meeting was described and analyzed up to the point where a learning issue was generated. However, a limitation of this study is that only a small percentage of the PBL tutorial was examined. A study by Visschers-Pleijers, Dolmans, Wolfhagen and Van der Vleuten (2004) analyzed the group interactions in the problem analysis and reporting phases of a PBL tutorial. They found that both elaborations and co-constructions occur during these phases of the PBL cycle although elaborations occurred less often compared to coconstructions. In a further study on students’ interaction processes, Visschers-Pleijers, Dolmans, de Leng, Wolfhagen and Van der Vleuten (2006) investigated how different types of verbal interactions such as cumulative reasoning, exploratory questioning, and handling of ‘cognitive conflicts’ were distributed over the group meetings. This study provided illustrations of the collaborative learning process in PBL and how questions, reasoning and conflict lead to elaborations and coconstructions by the students during the reporting phase of the tutorial. However, descriptions of what happens during the other phases of the PBL cycle are lacking. This is problematic because the elaborative processes that occur during the reporting phase in PBL may not similarly occur during initial problem analysis, when students largely operate upon prior knowledge acquired in other contexts. Another limitation of the studies reviewed here is that they mainly focus on the process of individual and collaborative knowledge construction without looking into other components of the learning process such as setting priorities, planning, and monitoring that are important to successful learning as well. A study by De Grave, Boshuizen and Schmidt (1996) explored in greater detail both the cognitive and metacognitive processes during the problem analysis phase of a PBL tutorial by analyzing both verbal interactions and their thinking processes (using stimulated recall). Their findings indicate that although verbal interaction represents only a portion of the cognitive processes taking place in a student, it does quite well reflect the theory-building processes in the students’ learning.


The process-oriented studies described above provide information about what students do in PBL tutorials. However, they mainly focused on the problem analysis and/or the reporting phase as it is not possible to observe how students learn during their extended self-directed study time in a naturalistic environment. Studies on the learning activities and processes during self-directed study time inevitably relied on self-report (Dolmans, Schmidt, & Gijselaers, 1995; Van den Hurk, Wolfhagen, Dolmans, & Van der Vleuten, 1999). Self-reports however, tend to reconstruct activities from memory, hence limiting their validity. The present study is unique in that it takes place in an educational context where the self-directed learning phase can also be observed in its natural classroom setting, allowing us to gain insight into how students learn during this time. In the polytechnic where the study was carried out, the problem analysis, self-directed learning and reporting phases of PBL all occur within one day. All students have a personal laptop that can be connected to the internet. First-year students generally rely mainly on internet resources for their research, and tend to remain in class instead of going to the library for their self-directed study. Thus it was possible to record and hence observe students’ verbal interactions for the entire PBL cycle, even during the periods of selfdirected study. This study seeks to examine the cognitive activities and interactions involved in how students learn in PBL and to evaluate whether PBL does indeed achieve its objectives of stimulating students to engage in constructive, collaborative and self-directed learning. We hypothesize that PBL is an approach which has these three characteristics and expected to find evidence of these learning-oriented processes in the verbal interactions of students. To that end, we recorded group discussions about a problem in genetics for an entire PBL tutorial, from problem analysis to problem reporting phase, including two sessions of self-directed learning when students were engaged in individual study of resources. The resulting protocols were coded and analyzed based on an adaptation of the utterance and episodic coding scheme of learning-oriented interactions devised by Van Boxtel, Van der Linden and Kanselaar (2000). Having

25

26

Chapter 2

the data of all students’ verbal interactions for the entire day allowed us to analyze one complete PBL cycle. A similar complete analysis has, to our knowledge, not been carried out before. The methodology chosen here was an intensive and in-depth case study of one team with five students. The verbal interactions amounted to 7.5 hours of recording with a total of 1075 utterances and 349 episodes. Furthermore, by describing and analyzing the verbal interactions of students in the PBL cycle, we aim to increase our understanding of the cognitive activities − what students do − in all the relevant phases of PBL.

Method PARTICIPANTS Participants were five first-year students randomly selected from one Basic Science class of a polytechnic in Singapore. They were of comparable abilities to their peers based on their daily grades and test results for this module. In this polytechnic, all first-year students undergo a common curriculum regardless of their subject discipline. The five students (making up one team) were recorded on the fifth week of Semester Two. Students were not new to PBL as they had already completed Semester One (16 weeks of PBL classes from Monday to Friday). The tutor had several years of experience. Both students and tutor gave informed consent to be audio recorded for the entire day. EDUCATIONAL CONTEXT The PBL cycle in this polytechnic is a fairly unique ‘one-day-oneproblem’ approach where students work on one problem in a day. It takes place in a class setting consisting of 25 students and one tutor. The students are grouped into teams of five. The daily routine consists of three meetings with tutor interaction and two periods of self-directed study or teamwork without tutor involvement. Thus, this approach, though fairly unique because the learning cycle is short, has all the attributes of a true problem-based approach: (1) All learning starts with a problem, (2) students collaborate in small groups, (3) learning is selfdirected, that is: students are encouraged to formulate their own learning goals and find their own resources, and (4) no direct instruction


is provided; tutors facilitate learning but do not teach (Barrows, 1988; Schmidt, 1993). A brief description of the day’s process is shown below: 









First meeting (problem analysis phase) (approximately 1 hr): Tutor presents problem for the day. Students work in teams of five to identify their prior knowledge and learning issues. First self-directed learning (SDL) period (approximately 1 hr): Students do individual research or work with their teams on worksheets and other resources provided. Time is spent teaching one another within the team. Most of the individual research is done by reading up on online resources from the internet. Second meeting (approximately 1.5 hr): Students meet with tutor to share their progress and strategy of solving the problem. The tutor usually spends about 20 minutes with each team during this time, while the other teams continue with their research and/or discussion. Second SDL period (approximately 2 hr): Extended time where teams consolidate their research and formulate a response to the problem. Third meeting (reporting phase) (approximately 2 hr): Each team presents their consolidated findings and response to the problem, defending and elaborating based on questions raised by peers and the tutor. The team presentation is usually in the form of powerpoint slides. The tutor would also clarify key ideas if necessary.

PROCEDURE Verbal interaction was recorded using a digital audio recorder placed at the team’s table. The students were audio recorded twice before the study began to allow them to familiarize with the procedure on the actual day. The voice recording device was started at the beginning of the day when the problem was first shown to the students till the end of the day’s class (a total of 7.5 hours). To ensure an authentic recording that was representative of what usually happens during the self-directed study times, no tutor or observer was present during these periods. The room in which the recordings were carried out was the students’ regular classroom.

27

28

Chapter 2

As mentioned in the Introduction, the students chose to remain in class for online research and team discussion during self-directed study periods instead of going elsewhere to study. Thus in this educational context, self-directed study time would involve not only individual research but also interaction with peers since they are sitting in their teams in the classroom at this time as well. It was also therefore possible to continue recording students’ verbal interactions even during the periods of self-directed study. MATERIALS The problem for the day was entitled “Code of Life” and it was designed to introduce students to the concept of heredity and genes. The problem is part of a Basic Science module which aims to provide an introduction to foundational scientific principles and applications for all students in the polytechnic, regardless of their specific discipline of study. The range of topics in this general module is wide and includes concepts such as energy, electricity, atomic structure, structure of organic compounds, cells, recombinant DNA technology, Newtonian mechanics and special relativity. The problem statement is presented in Appendix A. This problem followed a previous one where students learned about cells and their organelles while in the following week, students would be exploring further the synthesis of proteins in cells. ANALYSIS A verbatim transcript of the verbal interactions of the team was produced. The transcript was examined and segmented into units of analysis termed episodes. An episode was defined as a series of interactions beginning when a student starts a topic of discussion by a statement or by raising a question, and ending when there was no immediate further response or when a new topic is introduced or question is raised. We adapted the episodic coding scheme by Van Boxtel et al. (2000). In the coding system of Van Boxtel et al., they focused on question, conflict and reasoning episodes. Each of these episodes was further distinguished by whether there was elaboration (in the case of question and conflict episodes) or whether the elaboration was constructed by one or more than one individuals (in the case of


reasoning episodes). The episodic coding scheme as described above was similarly used in our coding scheme, with some renaming of codes for easier reference in our context. For example, the episode of ‘Individual reasoning’ was named ‘Explanation’ and ‘Short answers questions’ was named ‘Basic question and answer’ episodes. As our verbatim transcript also included the self-directed learning phases, additional episodic codes involving planning, evaluation and monitoring of task progress were added. These categories were mutually exclusive and the adapted coding scheme is shown in Table 1. Table 1 also shows how the different episodes were classified as being indicative of the constructive, self-directed and collaborative learning processes. For constructive processes, only explanation episodes were included here. As described in the Introduction, effective collaborative learning would lead to increased constructive learning processes. Thus the two categories of learning processes − ‘constructive’ and ‘collaborative’ are likely to have some overlap. Although we recognize that co-construction and conflict are constructive in nature, we identified them as collaborative learning processes to distinguish between reasoning constructed together by two or more individuals with that which was individually constructed. Other episodes classified as ‘collaborative’ in nature involve input from more than one team members, with the exception of ‘unanswered question’. This was considered ‘collaborative’ because the question was directed at the teammates although no immediate response was observed perhaps due to no one knowing the answer at the moment. Finally, episodes that involved planning and evaluation, indicative of monitoring and evaluation were classified under ‘self-regulated or self-directed’ learning processes. Inter-rater agreement between two judges for the coding was 80.0%; differences in judgment between the judges were resolved by discussion.

29

30

Chapter 2

Table 1. Episodic coding scheme showing indicators of constructive, self-directed and collaborative learning episodes Constructive Explanation episode

initiated by a question and followed by an answer or series of answers that involve reasoning and elaboration by the student(s) answering

Collaborative Basic question and answer episode

initiated by a question and followed by one factual answer or a ‘yes’ or ‘no’ response

Sharing of information episode

initiated by the presentation of information related to the problem to one or more team members and followed by team members agreeing without further input of ideas

Co-construction episode

initiated by the presentation of information related to the problem to one or more team members and followed by further input of similar ideas and information from at least one other team member

Conflict episode (not elaborated)

initiated by the presentation of information related to the problem to one or more team members and followed by opposing views from at least one other teammate with no justification given

Conflict episode (elaborated)

initiated by the presentation of information related to the problem to one or more team members and followed by counter arguments or opposing views from at least one other teammate, with justification given

Unanswered question

a question or series of questions with no response given

Self-directed Planning episode

topic of discussion regarding a strategy to work on the problem

Evaluation of understanding episode

topic of discussion involving a judgement about the individual’s or someone else’s knowledge and/or understanding of the problem or concept

Evaluation of resources

topic of discussion involving a judgement about the quality of a resource

Monitoring of task progress

topic of discussion involving organizing the duties of each team member in regards to the specific area to research on or to prepare a powerpoint slide on

Other episodes

miscellaneous episodes that did not fit in the above categories


Results Table 2 shows the number of episodes for each category during the different learning phases of the day. The total number of episodes during the day was 349 and included 1075 utterances. The average length of an episode was 5 ± 0.3 statements. From Table 2, it can be seen that more than 50% of the students’ learning-oriented interactions involved collaborative episodes, about 30% was related to self-directed learning and about 16% included constructive episodes. It can also be observed that a significant proportion of student interactions occurred during the SDL phases. For the different phases of the PBL cycle, we found that the episodes that occurred most frequently in the problem analysis phase were basic question and answer (14 episodes) and sharing of information (10 episodes). There were relatively fewer episodes for the first SDL phase (26 episodes) compared to the other phases, with sharing of information and co-construction (both 6 episodes) taking place most often. In the case of the second meeting, the most frequent episodes were explanation (22 episodes) and basic question and answer (22 episodes) while for the second SDL phase, monitoring of task progress occurred the most often (28 episodes). Lastly, for the reporting phase, sharing of information (16 episodes) took place most frequently. In the next section, we describe a collection of examples of episodes that illustrate the presence or absence of constructive, self-directed and collaborative learning in the PBL cycle. Examples cited were edited slightly to increase clarity and readability, without compromising what the students were communicating. Appendix B shows an example of the unedited version of Example 7 to illustrate that there was minimal loss, if any, to the original meaning expressed by the students upon editing.

31

32

Chapter 2

Table 2. Number of episodes in each category during the learningoriented interaction of a PBL cycle 2nd meeting

2nd SDL period

3rd

meeting

1st SDL period

Explanation Explanation*

1 4

2 0

14 8

11 0

Total constructive

5

2

22

8 6

3 0

10

Category of episodes

1st

Total

%

7 8

35 20

10.0 5.7

11

15

-

15.7

11 11

14 0

9 5

45 22

12.9 6.3

6

9

9

16

50

14.3

4 2

6 0

8 1

8 0

1 1

27 4

7.7 1.2

1

0

3

2

1

7

2.0

0

0

2

4

1

7

2.0

2

3

3

6

3

17

4.9

0

0

7

0

0

7

2.0

33

18

55

43

37

-

53.3

0 1

1 0

10 9

9 0

0 0

20 10

5.7 2.9

1

0

7

9

2

19

5.4

0

2

2

0

0

4

1.2

0

0

14

28

0

42

12.0

Total selfdirected

2

3

42

46

2

-

27.2

Others

4

3

6

0

0

13

3.7

Total for each learning phase

44

26

125

100

54

-

-

meeting

Constructive

Collaborative Basic Q & A Basic Q & A* Sharing of information Co-construction Co-construction* Conflict (not elaborated) Conflict (elaborated) Unanswered question Unanswered question* Total collaborative Self-directed Planning Planning* Evaluation of understanding Evaluation of resources Monitoring of task progress

* tutor initiated


CONSTRUCTIVE PROCESS The constructive learning principle is based on the premise that learners must be actively involved in organizing and representing their knowledge network (Glaser, 1991). The constructive process should include the activation of prior knowledge and the formation of relationships between new concepts and existing knowledge (Dolmans et al., 2005). In our episodic coding scheme, we identified constructive processes as those that involved student reasoning and elaboration (explanation episodes). In Example 1 the group is in the initial stage of problem analysis and has just been instructed by their tutor to identify relevant concepts related to the problem statement. This activity would be expected to help students surface their prior knowledge. This example indicates that Student S has the most prior knowledge in terms of remembering relevant concepts. It is noteworthy that in the initial discussion here, there is no attempt to make connections between the concepts. The episodes classified here were mostly basic question and answer type or sharing of information, with minimal elaboration involved. EXAMPLE 1 Basic question and answer

Co-construction

L: Ok, what’s a gene? C: DNA or something like that S: DNA L: Ok, got some key words- like chromosomes, X and Y chromosomes S: Chromosomes, alleles, X and Y chromosome, phenotypes, genotype, dominant, recessive

Sharing of information

L: I only learned allele S: Allele…


F: What about gametes? S: Ah yes, gametes C: Cheem

33

34

Chapter 2

(Nb: Cheem is a Hokkien word meaning complicated/difficult) Basic question and answer

Co-construction

L: What are gametes? S: Gametes are your sperm and your egg … C: Genes are passed down from a generation S: Genes are passed down from generation to generation (typing) we know that children tend to express physical traits of their parents including eye colour

Basic question and answer

S: What else do we know? C: Talk about RNA

Conflict (not elaborated)

F: I think RNA is not related S: RNA is not related F: RNA is not related, only DNA

In Example 2, which was extracted during the second meeting, Student L raises a question to the team in an attempt to understand the relationships between the various keywords including DNA, gene and chromosome. He had actually spent most of the previous self-directed study time playing computer games and had only recently shown a renewed interest in the problem. His teammates’ answers suggest that they had already made their own connections between different concepts, although this was not indicated from any prior verbal interaction. EXAMPLE 2 Basic question and answer

L: Is DNA on the gene? S: The gene is on the DNA C: Yeah the gene


Explanation

L: Then the chromosome? The DNA is on the chromosome? S: The gene is on the chromosome and the chromosome is on the DNA … A: The DNA and genes are actually packed in the chromosome A: So the chromosome actually contains DNA and the genes F: Chromosome contains DNA and DNA is made up of genes… Genes are made up of DNA A: DNA are made up of genes F: They are parts together


L: So which is the biggest one? F: Chromosome


L: Then the second? F: DNA

Conflict (not elaborated)

L: DNA is found in chromosomes C: No the biggest one is protein L: Don’t confuse me please


C: Then followed by the chromosome, then gene, then DNA L: Chromosome, genes, then DNA… thanks thanks

Although Example 2 shows Student L trying to construct an understanding of how the different concepts are related, it can be seen that he is rather passively trying to absorb the information given by his teammates, without actively making sense of their answers. Interestingly, the tutor brings up a similar issue towards the end of his discussion with the team by asking them if they are able to relate all the different concepts mentioned in a “nice picture”. Although Student L confidently

35

36

Chapter 2

assures the tutor that he is able to, he immediately raises the question to his teammates once the tutor had left. He was able to evaluate that he still could not picture the relationship and eventually asks Student S to draw the structure out for him: EXAMPLE 3

Evaluation of understanding

Explanation

L: What’s the structure- can you draw for me the structure? S: Draw for you? L: Yeah. Because I don’t know how it looks like. … S: Is like your DNA right? Your double helix (drawing for L) L: So this is the chromosome. Then inside is the genesok thanks thanks

Examples 2 and 3 focus on how Student L tried to construct an understanding of how the different concepts are related. Although he took the initiative to ask his teammates, his initial learning attitude was passive, in the sense that he did not seek to truly understand what his teammates explained to him. However, later on after comments from the tutor, it appears that he experienced some cognitive conflict, leading him to seek a clearer explanation from Student S, by asking her to draw out her explanation. These two episodes show the full range of the constructive nature of the process of learning. On one hand, Students S and C most likely worked individually to organize the relationships between the different concepts and were thus able to elaborate upon their understandings when requested to do so by their teammate. On the other extreme was Student L who was dependent on others to explain and help him as he tried to make sense of how the different concepts were related. The constructive process includes not only the formation of relationships between new concepts but also relating new information to existing knowledge (Dolmans et al., 2005). Example 4 shows an excerpt taken during the first self-directed study period where Student F asks


Student A about how parents can pass their genetic material to their children. Student A’s explanation shows an attempt to relate what he had understood from a previous problem with the current problem. EXAMPLE 4


F: I ask you something- how will the parent chromosome be passed on to the children? A: In the egg and theS: Yeah

Explanation

F: But when the parent give to the child- then will it give to the child and so not have any more chromosome? A: Er I thinkS: It’s just a copy A: Yeah, I think that time we learned they actually can duplicate. Dean’s team presented right? The type of method that they can separate

This example suggests that students are more likely to make relations and connections to prior learning when a question or doubt is presented to them. This is in line with the findings by De Grave et al. (1996) that conceptual change occurs only when cognitive conflict is induced. SELF-DIRECTED PROCESSES Individuals demonstrate self-directed learning when they plan, monitor and reflect upon their learning process (Dolmans et al., 2005; Glaser, 1991). One aspect of self-directed learning is the setting of goals and learning objectives. At the end of the first meeting, the tutor asks the class to identify the possible learning objectives for the problem. Students’ answers were then recorded on the whiteboard and included: to learn more about genes, how genes carry out their task, how many genes are found in the body, how do genes mutate, where to find genes and so on.

37

38

Chapter 2

It is striking that there was no record of any team member making any reference to the learning objectives the class had come up with. As a group there was also no indication that a strategy or plan was actively devised. However episodes of planning could be observed. In Example 5, when Student A asked Student S during the first self-directed learning period on what they needed to research on, her reply indicated that she had already planned out a certain strategy to use for her research. EXAMPLE 5

Planning

A: So when we study on genes we have to link to DNA as well? S: Yeah, that’s why we must first explain about genes, then after that I link to chromosome, then come back to the definition of a gene

Similarly we can see some indication of a plan from Student A when Student S asked the team if anyone was doing the worksheet, a resource provided with the problem statement for the day. From Student A’s answer shown in Example 6, it can be seen that his strategy was to understand the relevant concepts involved in the problem before looking further at what the problem statement and worksheet required. EXAMPLE 6

Planning

S: Anyone looking at the worksheet? A: No L: I’m not doing the worksheet C: No L: I just looking at the keywords S: Look and see the keywords L: Yeah I already did that A: I’m planning to understand all these C: Yeah, understand first then later go on A: Then see what the worksheet wants us to do

Another example that shows students taking control of what they wanted to find out occurs when Student F found out that some


mathematics might be involved in understanding heredity (see Example 7). The team was not pleased at this discovery as they apparently do not like to deal with calculations and equations. They somehow came to a consensus not to bother with the information but Student F continued to read to find out a little more.

EXAMPLE 7


F: You know for the Mendel’s first law - I see something about the chi squre test A: Huh? How come there’s chi square? F: I don’t know

Planning

C: Does today’s problem involve equations? L: Let’s not do that lah (Nb: ‘lah’ is a word often used at the end of sentences in ‘Singaporean English’ for emphasis) S: Can we not do? C: Can we not do any calculations? A: Yeah or else we have to-


A few minutes later: F: I know why they talk about the chi-square thing already A: Why? F: To see which one is green which one is yellow; the chance of yellow- that sort of thing

During the team’s discussion with the tutor at the second meeting, the tutor asked them what they were planning to present during the reporting phase. Although the team’s response to the tutor indicated they had a plan for their presentation, this was not previously discussed as a group beforehand; the students were reporting what they had read about so far on their own.

39

40

Chapter 2

EXAMPLE 8

Planning (tutor initiated)





Tutor: Do you have any other points? L: Um, we are going to talk about Mendelian laws too L: Because that has to do with genes L: Yeah, because it’s quite prominent. The laws play quite a prominent role in understanding how genes work Tutor: Ok L: Pass down Tutor: Oh, that will tell you how genes are passed down, how genes work and get passed down L: Then we will talk about recessive and dominant genes in the family Tutor: Recessive and dominant. Ok I’ve heard your team discuss this already just now so I won’t ask you anymore. I’ll wait for your presentation L: We are going to include how genes can mutate too Tutor: Oh … Tutor: Ok, sounds good. Ok are all genes dominant or recessive- can we have two that are equal in dominance? L: Codominance is it? S: Codominance Tutor: What is codominance? Is there such a thing? … Tutor: So you all clear about what you are going to present? The whole team is clear? A: So what we say? Do we have any specific things to deliver? Tutor: Look at the problem statement … L: We link it back ah? Tutor: That will tell you if your points are related to the learning objectives. Ok, I go to the other teams first … if you have any questions, flag me


After discussing with the tutor during the second meeting, the team returns to their research online. By then the team realizes they have a lot of information already. Student A comments twice in a short span of time on the importance of understanding what they had found. Example 9 also contains an episode of monitoring task progress where Student L checks on the other teammates regarding the area of their research. Such episodes were rather frequent, especially during the second SDL period, just before the reporting phase. EXAMPLE 9 Monitoring of task progress

L: You are doing phenotype and genotype right? C: Mm


L: Um, A, what are you researching on? A: How the protein actually functions


L: Ah ok, then F? F: I am doing on RNA, mitosis, meiosis and the codominance of theL: Oh. You doing on the codominance one also ah. A: So actually what I’m doing is like- after she explain about the protein.and all that, then I will come in later L: Ok I do the mutation and the mutation and the Mendelian laws A: Later we really need to organize our work very well

Another aspect of SDL is that students would monitor and reflect on their learning process. Initially Student A was like his other teammates, being mostly preoccupied with reading up and understanding the various concepts related to the problem statement. However, towards the middle of the second SDL period, he re-examines the problem statement and starts puzzling over the meaning of the last paragraph: “Explore the concept of a gene and the role it plays in an organism. Is it possible that the gene is represented by an identifiable molecule, one that is able to carry information akin to a line of code, giving it the ability to execute highly detailed tasks? Determine the

41

42

Chapter 2

qualities such a molecule should have.” Example 10 shows how he evaluates his own understanding and realizes it is lacking. It also shows how he grapples with the question more or less on his own, as his team members do not find the issue as important as he does. At that moment they appeared to be more concerned about getting the powerpoint slides for their presentation during the reporting phase ready. EXAMPLE 10 Monitoring of task progress





A: You are doing the RNA, the mRNA and the …? F: Yeah…erm ok I do mRNA and tRNA. I don’t think I will touch on rRNA ‘cause it’s not really- wait, is it? A: Because I have to find the molecular properties… L: You doing meiosis right? F: Yeah… What you saying- what the molecular? A: Because I want to find the molecular properties of a gene… F: Uh… A: But I’m not really sure what does it mean by molecular properties F: What does that mean? A: Yeah that’s why F: What do you mean by we have to find the molecular properties of the gene? A: The problem statement says so… F: Does it? A: Yeah…determine the qualities such a molecule should have… the problem statement says: is it possible that the gene is represented by a molecule and you have to determine the qualities lah F: Is a gene a molecule? A: Not really... but they have molecular qualities


Planning

F: Can we just talk about the functions? A: I don’t know…

Some time later: S: You’re done?(in reference to the powerpoint slides Monitoring of for the reporting phase) task progress A: Haven’t yet… I still have to find the qualities of a gene


Some time later: L: You’re doing concept is it? (in reference to the powerpoint slides for the reporting phase) A: I try to find… L: How many slides you have? A: Not a lot. Only two. I haven’t done concept of a gene …


A: But I really don’t understand the question of “can a gene be represented by an identifiable molecule?” I really don’t understand


L: Means- ok, ‘cause we need to find out what exactly is a gene C: There’s a website with this information right?

Unanswered question

L: So what exactly is a gene? Is it a molecule? …


Some time later: A: I know what it means already! I think… …

A study by Dolmans, Schmidt and Gijselaers (1995) showed that student-generated learning issues were not the major determining factor of what students decided to study during self-study time. The items cited

43

44

Chapter 2

by the team of students in Example 8 were generally in line with the learning issues generated as a class and recorded on the whiteboard. However there were no indications from the verbal interactions recorded in the day that the team actually discussed together a plan of study. It is therefore unclear if individual students did refer to the learning issues previously generated as a class as a guide for doing their research, or that the items happen to match by coincidence. In Example 8, the tutor asks a question, which led the students to ask if he was referring to the concept of ‘codominance’. This was not an issue they had researched much on prior to this. However after the question was raised, Student F began to read about it, and indicated to the team (in Example 9) that she had researched on this topic. On one hand, the tutor encouraged the students to identify on their own what they should present during the reporting phase (end of Example 8) without directing them on what to research on. However the question regarding the possibility of two genes being equally dominant that was brought up, also induced one of the students to find out more about this issue. In our opinion this is not a contradiction of the principle of self-directed learning as the students were free to decide for themselves if they wanted to research further on the topic, and the question raised was perhaps to challenge the students to go a little further in their research. Example 7 also shows the team determining on their own what they wanted to pursue further. A study by Lloyd-Jones and Hak (2004) indicated that in their education context, self-directed learning was not apparent. Instead, learning was agreed amongst peers and driven by what they thought to be faculty objectives. Although there was a question by Student A to the tutor on whether they had “any specific things to deliver” (Example 8), there were no other indications by this team of being particularly concerned in meeting faculty-set objectives. Student A’s later preoccupation with answering the problem statement (Example 10) seems to be driven by his own desire to clarify what he found to be lacking in his understanding. Even though his teammates were less interested than him to find a solution to the question, he continued in his quest until he thought of a solution that made sense to him. This shows that his learning was not overly influenced by peer opinion either. Student A also shows the ability to evaluate his learning


progress when he understands the need to organize the team’s findings and to understand what they had found “very well” (Example 9). Selfdirectedness is characterized by the ability/tendency to generate learning objectives, select and manage learning strategies and resources and periodically evaluating one’s learning process (Dolmans et al., 2005; Miflin, Campbell, Price, & Miflin, 2000). COLLABORATIVE PROCESS Collaborative learning involves interaction between students with a shared goal and responsibility. It takes place when students depend on one another to construct their knowledge, and involves elaborations, coconstruction, confirmation and criticism of ideas (Dolmans et al., 2005; Van Boxtel et al., 2000). One of the instances of co-construction and sharing of information is shown in Example 11 during the first self-directed study period. Student A had just read some information from a website that he found useful. As the rest of the group discusses the ideas together, it can be seen from their verification questions and confirmations that the discussion process was helping them make sense of the concepts. EXAMPLE 11


Co-construction

A: Actually genes are made up of DNA ‘alphabets’- you want to read? C: I’m reading now... “what is chromosomes?” (a website) right? A: No mine is “what is DNA?” (a website) A: Then the alphabets match up with different alphabets and then they form words. Alphabets they form into words and then into a sentence C: Sentence A: And then those sentences are the genes L: Oh ok so actually genes are made up of alphabets… A: They call it DNA alphabets- they are actually like genes L: Like a code ah A: Yeah, they are a code

45

46

Chapter 2

Co-construction

… A: They are different in structure also F: Yeah made up of one long pair of letters. But it will form into words and then make into sentences A: So these are the genes

Example 11 demonstrates how the team reasoned together and affirmed one another’s understandings. Less common were examples of conflicts. Example 12 shows an instance of a difference in opinion which took place shortly after the excerpt in Example 10. Here Student S does not agree with Student A’s conclusion on the answer to the question in the problem statement. She raises a critical question that causes Student A to elaborate on his reasoning. However she does not continue to argue for or defend her own opinion after that. EXAMPLE 12

Explanation

Conflict (elaborated)

A: (reading the problem statement) “Is it possible that a gene be represented by an identifiable molecule, one that is able to carry information, akin to a line of code giving the ability to execute...” I think here it actually means protein because protein is actually a molecule and it represents the information given out by DNA and it is able to carry out specific tasks L: So the identifiable molecule is actually protein? A: Yeah S: Are you sure protein? Genes make protein- how can they be protein? (...) S: They make protein A: They represent protein. Eh no, they represent DNA L: Represent DNA A: They are notL: Actually they makeS: They make- no genes. Genes are spiral thread-like particles made up of DNA. So DNA is deoxyribonucleic acid


Co-construction

… A: Ok, what do we know about genes? Genes are actually- you know the line of code put together, so they are actually instruction S: Yeah A: Ok. So protein, they actually have the instruction to carry out the specific tasks so the protein can actually represent it C: Yeah A: Yeah. As a molecule. Because protein is actually a molecule and it is able to- because of the instruction right, it is able to carry out the specific task C: Yeah L: So basically the whole thing is the gene, but it is expressed by the protein A: Protein

Since student collaboration in small groups is a characteristic of PBL, it is not surprising that examples of student elaboration and coconstruction were not difficult to find from students’ verbal interactions in this study. Example 11 shows Student A sharing what he had just read with the team. This was a common occurrence throughout the day, with team members verbalizing their findings as they found out new information. Van Boxtel et al. argued that there are three group interaction types that stimulate student elaboration and co-construction. These include asking and answering questions, reasoning, and resolving conflict (Van Boxtel et al., 2000). In this study, we did not find any evidence of effective conflict resolution. In Example 12, Student S raised a relevant question to query the accuracy of Student A’s deduction. Student A was able to elaborate upon his reasoning, but Student S did not follow through with her criticism by continuing to identify possible loopholes in his conclusion. This finding was similar to that by Visschers-Pleijers et al. (2006) where they found very little time (7% ± 2.3%) spent by students in handling conflicts. Very few other instances where there were disagreements were found throughout the day. Table 2 indicates 4.0% of the episodes involved differences in opinion. However

47

48

Chapter 2

we found that the team members tended to ignore differences in opinion and did not attempt to work out whose idea was more accurate through critical discussion. As reasoned by Visschers-Pleijers et al. (2006), such a lack of reasoned conflict resolution could be due to time constraints (needing to reach a consensus by the team quickly). However, since cognitive conflict is an important means to stimulate learning (De Grave et al., 1996), it is somewhat disturbing to find that students were not better able to resolve conflicts using critical questions, reasoning and counter-arguments. It is also of interest that there were many episodes of student discussion during the SDL periods. Even though students were using this time to research, there was a tendency for them to articulate what they had found (Example 7 and 11) in the form of sharing of information with no further input by their team or leading to further co-construction of ideas. These episodes possibly provided a form of rehearsal and repetition for the students as they tried to make sense of what they had read.

Discussion The purpose of the present study was to provide an account of whether and how PBL stimulates students towards constructive, self-directed and collaborative learning. Also, we aimed to clarify what students do in all the phases of PBL by providing a detailed description of their actual activities (through their verbal interactions) in the PBL cycle. Since the first meeting (problem analysis phase) was to enable students to identify learning issues in a collaborative setting and to build on one another’s prior knowledge, we expected high levels of sharing of information and co-construction episodes. While we did observe high levels of sharing of information during this phase, we found that co-construction episodes where ideas or concepts raised were built upon by others occurred less frequently than expected. In the case of the SDL phases, we expected relatively high occurrences of planning and monitoring of task progress episodes during the self-directed learning phases as these were the periods for students to research and discuss as a team. Our results showed that the


planning occurred less frequently than expected during these phases, although monitoring of progress took place very often especially for the second SDL phase just before the reporting phase. As each team is required to present a final response to the problem during the reporting phase, we postulated that there would be high numbers of explanation and co-construction episodes during the self-directed phases since students were likely to co-teach one another to ensure the whole team is able to understand what they were going to present. The second meeting was similar to the SDL phases except for a twenty minute discussion with the facilitator and so we expected to find similar proportions of episodes for this phase as well. Our results did show higher numbers of explanation episodes for this phase although there were more basic question and answer episodes compared to co-construction episodes. Finally, as expected, the reporting phase consisted of significant numbers of sharing information episodes, since this period was for students to present their findings. However although we also predicted that there would be relatively high numbers of explanations and elaborated conflict here since the different teams were likely to challenge one another’s ideas and responses, this was not observed in our data. In general, we found lower levels of individual constructive learning processes (explanations) and co-constructions than expected for the different PBL phases. This suggests that students do not spontaneously make connections between concepts, and when left to discuss on their own, spend more time sharing information and ideas with less elaboration occurring than hoped for. In the case of the self-directed learning, while we did find high frequencies of monitoring of task progress episodes especially in the second SDL phase, there was relatively low numbers of planning episodes. Although the SDL phases in this context are different from traditional SDL done outside of classroom time, the learners were selfdirected in that they need not remain in the classroom during this time. They could also choose the resources and reading materials online or from the library for their research and individual study. For this group of students, as is typical of the Basic Science students, they chose to do their research from the internet and to remain in the classroom. While it

49

50

Chapter 2

is true that self-regulated learning processes would mostly take place internally and need not be verbalized, since students were supposed to come up with a group response for the reporting phase, it is reasonable to expect more planning episodes especially during the first SDL period. However there were hardly any planning episodes here. It appears that students were more task-oriented and focused on delegating duties towards the later part of the day. One possible reason for this could be the time constraints involved in this ‘one-day-one-problem’ PBL system. Since students only have about 3 hours of SDL time, they may prefer to spend more time doing individual research or preparation of the powerpoint slides, and view working on a team plan and strategy as less urgent or important. It can be concluded from the results that it was possible to describe examples of constructive, self-directed and collaborative learning in the PBL cycle under study, albeit to different extents. The percentage of constructive episodes was the lowest − 15.7% while the collaborative episodes were the most prevalent – 53.3%. Out of the collaborative episodes, the largest proportion was of basic questions and answers, which did not involve elaboration or explanation. Co-construction episodes (8.9%) appeared also to a lesser extent than sharing of information episodes (14.3%). These data indicate that the elaboration on, reasoning about, and making connections with knowledge occurred much less frequently and readily than stating and/or sharing facts. Of course, while elaboration and reasoning are higher-order learning processes and therefore important, sharing of information and stating of facts are also important activities necessary for learners. The percentage of self-directed episodes here was rather high − 27.2%. It can be seen from Table 2 that about one-third of the planning episodes were initiated by the tutor. This means that in those episodes, the students were responding to the tutor’s questions on what their approach or plan was. This shows that the modelling of self-directed processes is possible and could be helpful to less mature students in learning how to plan and set goals and objectives for their learning. The largest proportion of the self-directed processes (12.0%) was related to the monitoring of task progress. This indicates that students were strongly driven to be ready


for the reporting phase and thus spent a lot of time checking on one another’s research area and the powerpoint slides they were preparing. One point to note is that the Basic Science module is one which all students have to go through regardless of their discipline of study (e.g. arts, IT, sports etc). Also most of the students have not learned Biology prior to this. These could be possible reasons for the relatively slow progress and rather rudimentary level of understanding reached by the team at the end of the day. The present study has several limitations. First, the number of participants for this study (N = 5) was very small. Since our intent was to investigate and describe the processes of PBL in detail, we only studied a limited number of students intensively. It would be useful to replicate the study with a larger sample size which would enable more detailed analyses and possibly uncover more subtle information regarding the relationships between students’ verbal interactions, individual study, facilitator contributions, and student learning. Second, PBL described here is a unique ‘one-day-one-problem’ approach. While we have argued that this approach is indeed PBL as it contains all the prerequisite characteristics of PBL (Barrows, 1988; Schmidt, 1993), the short learning cycle, in particular with the SDL phases being only 3 hours in total, is likely to impact students’ learning processes. Thus our findings in this study may not be readily extrapolated to PBL in other contexts. Third, students’ verbal interactions were recorded using an audio recorder placed at their table for the entire day. Although students were recorded a few times beforehand in order to desensitize them to the presence of the recorder, it is possible that the students’ interactions were affected when they knew they were being audio-taped and hence were less representative of their usual group discussions. However when listening to the audio recording, it could be observed that students often forgot they were being recorded and did not have much qualms talking about their personal matters unrelated to class work. In conclusion, with regards to the first aim of our study, we have shown that the PBL cycle does contain evidence that PBL encourages

51

52

Chapter 2

students towards constructive, self-directed and collaborative learning activities. This study shows for the first time observational data that selfdirected processes do occur in PBL and in relatively high proportions. However, the proportions of interaction for constructive processes were less compared to those for collaborative and self-directed activities. Moreover, in the case of the collaborative process, students may need more guidance by tutors to help them deal more effectively with conflicts in knowledge and ideas. Second, by describing and analyzing the verbal interactions of students in the PBL cycle, we intended to contribute to current understanding of how all the relevant phases of PBL are carried out in practice. The qualitative descriptions of what students say in the process of PBL are useful both at the practical and theoretical level (Koschmann et al., 1997). At the practical level, suggestions that are relevant to educational practice can be derived from our observations. Students tend to pick up cues from the tutor during the limited interaction time. Thus, one way to encourage students to formulate more critical questions, and seek clearer explanations and reasoning during their group interaction is for tutors to actively model such questions when interacting with students. At the theoretical level, this article illustrates the different types of group interactions during the entire PBL cycle, including those during the self-directed study time, to help us understand better the learning processes involved in PBL.

How students learn in PBL

Chapter 3: How students learn in problembased learning: A process analysis Abstract The goal of this study was to provide an account of how learning takes place in problem-based learning, and to identify the relationships between the learning-oriented activities of students (the scientific ideas they discuss and encounter during self-study) with their learning outcomes. The verbal interactions and computer resources studied by nine students for an entire PBL cycle were recorded. The particular scientific concepts articulated by each student in the group and studied individually while working on the problem-at-hand were identified as units of analysis and counted to provide an account for the growth in concepts acquired over the different learning phases. We identified two distinct phases in the PBL process- an initial terminology articulation phase, where students are exposed to and articulate new concepts, and a later terminology repetition phase. Our results also show that what students do during self-directed study plays a larger role in their learning outcomes than their verbal interactions. Finally, the breadth of new scientific concepts articulated and used by students to understand the problem-at-hand seems to be more important to students’ learning than the repetition of previously acquired concepts.

Introduction What exactly do students learn in problem-based learning (PBL)? And how do they learn during group discussions and self-directed study? While there has been considerable research into various aspects of PBL over the past twenty years, answers to these questions are still lacking (Dolmans & Schmidt, 2006; Hak & Maguire, 2000). The goal of this paper is to report on a first attempt to provide a comprehensive account of the learning-oriented activities of students − what they do and say − throughout the PBL learning process, as well as to identify relationships between the learning activities of students with

53

54

Chapter 3

their learning outcomes. Problem-based learning, as its name implies, always starts with a problem. This problem refers to an academically or professionally relevant issue that students are supposed to learn more about. Students do not prepare for the problem beforehand and therefore begin their initial discussions based on their prior knowledge. As a group, the students analyze the problem, generate possible explanations, build on one another’s ideas, as well as identify key issues to be studied further. The purpose of this exercise is to construct a shared initial explanatory theory or model explaining the problem-at-hand (Schmidt, 1983). After this period of teamwork, they disperse for a period of self-directed study to work on the learning issues identified. When they next meet as a team, they are expected to share and discuss their findings, as well as refine their initial explanations based on what they have learned. A tutor is present during the team discussions to help facilitate the learning processes. Thus, the cycle of PBL can be seen as being made up of three phases: initial problem analysis, followed by self-directed learning, and a subsequent reporting phase (Barrows, 1988; Hmelo-Silver, 2004). Described this way, PBL can be considered a constructivist approach to instruction, emphasizing collaborative and self-directed learning, and being supported by flexible teacher scaffolding (Schmidt, Loyens, Van Gog, & Paas, 2007). Thus PBL is clearly a highly relevant approach in science classrooms. Science educators have long advocated that science classrooms be active learning environments which allow students to construct meaning collaboratively (Erickson & MacKinnon, 1991; Roth, 1990). Fosnot (1993) also argued that science education should emphasize both personal construction of knowledge as well as the social processes involved in making sense of new information. Since the PBL approach is well-suited for learning science, understanding how students learn in the PBL process will have practical implications for science educators. These implications will be discussed further later in this paper. A number of studies have focused on the effects of the initial problem analysis on learning. Schmidt, De Volder, De Grave, Moust and


Patel (1989), and De Grave, Boshuizen and Schmidt (1996) have demonstrated that students who discussed a problem in a small group before studying a relevant text learned more from that text relative to students who did not have the chance to discuss the particular problem. It was found that the opportunity for students to elaborate on what they know or think about a subject before studying relevant resources helped them remember the concepts learned better. Surprisingly however, a study by Moust, Schmidt, De Volder, Belien and De Grave (1987) failed to find a relationship between the amount of verbal elaboration by students and the ability to do well on a test. These studies however did not examine the group discussions in terms of their content, which may have yielded more conclusive information on whether or how group discussions contribute to students’ learning. A similar study, comparing problem-based discussion with expository teaching, was conducted by Capon and Kuhn (2004). They found that students who experienced PBL demonstrated superior explanations and understanding as compared to students in the lecture/discussion group. Their results support the hypothesis that the advantages of PBL over traditional lecture-based instruction lies in its ability to help students integrate new concepts with existing knowledge. While this study suggests a possible mechanism of how PBL enhances understanding, it does not provide us with information on how the students’ verbal interactions during the 3 hour session studied helped students learn in the process. The study by De Grave et al. (1996) explored the relationship between students’ verbal interaction and their cognitive change during the problem analysis phase of PBL. Their findings indicate that although verbal interaction represents only a portion of the cognitive processes taking place in a student, it does reflect the theory-building processes in the students’ learning. This suggests that observational studies of the PBL tutorial process can provide valid data regarding students’ thought processes involved in learning. It also suggests that the quality of what students articulate is likely to be related to their learning outcomes. However, De Grave et al. do not report any learning outcomes. In a recent study on students’ interaction processes, VisschersPleijers, Dolmans, de Leng, Wolfhagen and Van der Vleuten (2006)

55

56

Chapter 3

investigated time spent on different types of verbal interactions during group discussion sessions. Using video-recordings of four tutorial group sessions, they found that time spent on learning-oriented verbal interaction was very high − around 80%, and also identified how different types of verbal interactions such as cumulative reasoning, exploratory questioning, and handling of “cognitive conflicts” were distributed over the group meetings. However, no relationships with the amount and content of subsequent learning were reported. A naturalistic study of the PBL process by Koschmann, Glenn and Conlee (1997) involved a description and analysis of a segment of a PBL meeting up to the point where a learning issue was generated in order to demonstrate the actual events taking place in a PBL tutorial. Another publication by the group focused on a short segment of a PBL group’s interaction to provide insight into how students interact in the process of presenting one’s own theory and responding to those of others (Glenn, Koschmann, & Conlee, 1999). One limitation of these studies is that they only examined specific portions of the PBL tutorial and did not relate the behavior of students to later achievement. The studies cited above all focus on the problem analysis and reporting phases of PBL. Research dealing with the phase of individual, self-directed study is scarce. One study involving the self-directed learning phase was carried out by Dolmans, Schmidt and Gijselaers (1995). They investigated the relationship between student-generated learning issues during problem analysis phase of a PBL classroom and what students actually studied during self-study time. Even though it is generally assumed that students would make use of the learning issues generated to determine their learning activities during self-directed study, this turned out to be the case only to some extent. It seemed that the learning activities of the students during the self-study phase were also determined by other factors such as the nature of tutor guidance and the learning resources available. However, since the measurement of what students actually were studying was based on retrospective selfreport, the results may have been biased to some extent. Another study by Van den Hurk, Wolfhagen, Dolmans and Van der Vleuten (1999) focused on how students made use of their self-study phase in terms of


learning issues previously generated and time spent on individual study. They found that higher year students were more self-directed learners compared to first year students, and that those who studied beyond the learning issues generated by the tutorial group during problem analysis phase showed better achievement in tests. This study also relied on selfreport data. Although a fairly detailed picture regarding the nature of the discussions in the tutorial teams has emerged from these studies, only one study has been conducted in which a relationship was studied between what is discussed and what is learned. This study by Moust et al. (1987) failed to disclose a relationship between the two activities. Since Moust et al. (1987) concentrated on how much was said in relation to achievement, there is a need for studies that relate what is said to achievement. In addition, there is a definite lack of research investigating what students actually do during self-directed study and how their activities influence the outcomes of their learning. Although in particular the experimental studies discussed above (Capon & Kuhn, 2004; De Grave et al., 1996; Schmidt et al., 1989) suggest that group discussion and elaboration play an important role in students’ learning in PBL, the way in which they affect learning remains to be clarified. Furthermore, as suggested, we do not yet know much about students’ learning processes during self-directed study periods, obviously because it is extremely difficult to directly observe a group of students engaged in individual study under naturalistic conditions. The first purpose of the present study, therefore, was to increase our understanding of the learning processes in all the phases of the PBL cycle, including the self-directed study periods. The study reported here is, as far as we know, unique in that it has taken place in an educational context where the self-directed learning phase can be observed in its natural classroom setting, allowing for insights into how students learn during this time. In the polytechnic where the study was carried out, the problem analysis, self-directed learning and reporting phases of PBL all occur within one day. All students have a personal laptop that can be connected to the internet. First-year students generally rely mainly on internet resources for their research, and remain in class for their self-

57

58

Chapter 3

directed study. Thus it was possible to record and hence observe students’ verbal interactions and internet activities for the entire PBL process, even during the periods of self-directed study. Second, unlike previous studies (e.g. Capon and Kuhn, 2004; Moust et al. 1987), the present study focused on the contents of what was discussed and learned, and attempted to relate these contents to subsequent achievement. To that end, we recorded group discussions about a problem in genetics, logged all individual study activities of the students while they were using online resources, and recorded contributions of the facilitator. In addition, we measured prior knowledge of the subjects and actual learning gains. The resulting protocols were analyzed to understand the growth in usage and study of relevant concepts over the different learning phases and its effect on achievement. The units of analysis were the scientific genetics-related concepts or terminologies that the students articulated and studied while gaining insight in the problem at hand. Concepts such as ‘DNA’. ‘alleles’, or ‘meiosis’ can be considered micro-theories (Murphy & Medin, 1985) that students use in the course of trying to learn about genetics. We hypothesized that the frequency with which these concepts were used by students while discussing the problem and studying subjectmatter, could be considered an indicator of the learning-oriented activities going on and would determine subsequent achievement. (Of course, this is not to say that the usage of a concept in itself implies understanding: however we hypothesize that increased usage over time would result in increased learning.) The study to be reported here involved nine participants collaborating in two groups. We followed these two groups throughout the day, resulting in roughly 16 hours of discussion protocols and 70 hours of individual internet log files. The need to conduct a fine-grained study of learning in PBL precluded involving more students. The study therefore must be considered a case study in PBL. In the light of the fact that it is, to our knowledge, a first attempt to chart the entire learning process of students in a natural environment, this limitation to our findings may be acceptable. In summary, the research questions addressed were: (1) how are


the numbers of relevant concepts verbalized and studied from resources (and the frequencies of their occurrence) distributed over the different learning phases of the PBL process? What does this distribution of concepts over time suggest about the nature of the learning process in a problem-based curriculum? And (2) how is student achievement influenced by their verbalization and by their individual study?

Method PARTICIPANTS Participants were nine first-year students from a polytechnic in Singapore. In this polytechnic, all first-year students undergo a common curriculum regardless of their subject discipline. The nine students (making up two different teams) were from the same Basic Science class and were being recorded on the fifth week of Semester Two. Students were not new to PBL as they had already completed Semester One (16 weeks of PBL classes from Monday to Friday). The facilitator had several years of experience. Both students and facilitator gave informed consent. EDUCATIONAL CONTEXT The PBL process in this polytechnic is unique in that a ‘one-day-oneproblem’ approach where students work on one problem in a day is adopted. It takes place in a class setting consisting of 25 students and one facilitator. The students are grouped into teams of five. The daily routine consists of three meetings with facilitator interaction and two periods of self-directed study or teamwork without facilitator involvement. A brief description of the day’s process is shown below:



First meeting (problem analysis phase) (approximately 1 hour): Facilitator presents problem for the day. Students work in teams of five to identify their prior knowledge and learning issues.



First self-directed learning (SDL) period (approximately 1 hour): Students do individual research or work with their teams on worksheets and other resources provided. Time is spent teaching one another within the team. Most of the individual research is done by reading online resources from the internet.

59

60

Chapter 3



Second meeting (approximately 1.5 hours): Students meet with facilitators to share their progress and strategy of understanding the problem. The facilitator usually spends about 20 minutes with each team during this time, while the other teams continue with their research and/or discussion.



Second SDL period (approximately 2 hours): Extended time where teams consolidate their research and formulate a response to the problem.



Third meeting (reporting phase) (approximately 2 hours): Each team presents their consolidated findings and response to the problem, defending and elaborating based on questions raised by peers and the facilitator. The team presentation is usually in the form of powerpoint slides. The facilitator would also clarify key ideas if necessary.

Although the PBL approach in this context has been adapted to suit the learning needs of the particular students and takes place within one day, it is to be classified as PBL based on the ‘six core characteristics of PBL’ as described by Barrows (1996). These characteristics include student-centred learning with small groups working under the guidance of a tutor who acts as a facilitator. Students work on authentic problems with no prior preparation so as to achieve the required knowledge. In addition, no direct teaching takes place; all learning is student generated. Finally, it is through self-directed learning that students acquire new information (Hmelo-Silver, 2004). Dochy, Segers, Van den Bossche, and Gijbels (2003) used the same criteria in their meta-analysis on the effects of PBL for inclusion in their study. PROCEDURE The class in which the nine students were being observed was in the fifth week of a 16-week semester. The teams had been working together for the Basic Science module for two weeks before being recorded for this study. Although only two teams were being observed, there were a total of four teams in the class for that day.


Verbal interaction was recorded using a digital audio recorder placed at each team’s table. Students’ computer usage was tracked using the program Camtasia Studio Screen Recorder (TechSmith Corporation, Okemos, MI) installed on each student’s laptop. The students were audio-recorded and had their computer usage recorded twice beforehand to allow them to be familiar with the procedure on the actual day. Both recording devices (voice and computer) were started at the beginning of the day when the problem was first shown to the students till the end of the day when the facilitator finished presenting the 6th presentation (a total of about 8 hours). To ensure an authentic recording that was representative of what usually happens during the self-directed study times, no facilitator or observer was present during the SDL sessions. The room in which the recordings were carried out was the students’ regular classroom. MATERIALS The problem statement for the day was entitled “Code of Life” and it introduced students to the concept of heredity and genes. Students were to explore the role and properties of the gene which is able to transmit information from parents to children. The problem is part of a Basic Science module which aims to introduce foundational scientific principles and applications for all students in the polytechnic, regardless of their specific discipline of study. This is a general module that includes a wide range of concepts such as energy, electricity, atomic structure, structure of organic compounds, cells, recombinant DNA technology, Newtonian mechanics and special relativity. The problem statement is presented in Appendix A. Two weeks before the problem, students were given a pre-test consisting of two parts: free recall essay questions as well as a concept recognition test. The free recall essay test was administered first and consisted of the following instructions: “Write in detail your answers to the following questions. Include all the ideas that you think are relevant. 1. Why do children tend to resemble their parents in terms of their physical traits? 2. What do you know about the structure and function of a gene?”

61

62

Chapter 3

The concept recognition test was a simplification of the concept mapping technique (Novak, 1998). This test consisted of a list of 34 concepts that are more or less closely related to the central topic of heredity. This set of 34 concepts cover the domain as a whole and was based on an exhaustive review of relevant literature from textbooks and internet resources. Examples of such concepts include ‘gametes’ and ‘chromosomes’. A number of concepts not related to heredity (“fillers”) were interspersed in the list. Examples include ‘water’ and ‘oxygen’. Students were instructed to rate the extent to which the concepts listed were related to heredity using a scale of 1 to 5, where 1 = not at all related; 2 = a little bit related; 3 = to some extent related; 4 = quite closely related and 5 = very closely related. This concept recognition test was given to the students after they had submitted their answers for the free recall essay test so that they were not able to use the concepts listed to help them answer the essay questions. No time limit was set for the tests. The same free recall essay and concept recognition tests were administered as post-tests immediately after the day’s problem in the same sequence as described for the pre-test. The free recall essay tests were analyzed for accuracy using the “idea unit” as the entity for scoring (Meyer, 1985; Schiefele & Krapp, 1996). The answers were segmented into idea units. An idea unit was defined as a statement which ends with a comma, period, or ‘and’. Each idea unit was given either a score of 2, 1 or 0. A score of 2 was given for a completely correct idea unit, 0 for a completely incorrect idea unit and 1 when the idea unit was only partially correct. For the concept recognition test, two colleagues with expertise in the field of molecular biology were asked to identify the most appropriate answers independently. Inter-rater agreement was 83.8%. Where there were differences in rating, a third opinion from a similar expert was sought. Student rating of each concept scored 2 points if it was the same as the expert answer, 1 point if it differed by ±1, and 0 for any other answer.


Together, the concept recognition and free recall tests measured the breadth as well as the depth of the students’ scientific vocabulary. Breadth refers to the extent of the knowledge distribution while depth refers to the ability to describe the relationships between the concepts (Alao & Guthrie, 1999). In this study, the former is measured through the recognition of relevant concepts while the latter is assessed through the free recall essay tests. It was assumed that the both the breadth and validity of the learners’ scientific concepts give an indication of their understanding of the theory (Alao & Guthrie, 1999; Tsai & Huang, 2002). ANALYSIS Verbatim transcripts of a total of 16 hours of verbal interactions of the two teams (approximately 8 hours per team) were produced. The computer screen recordings of the students were viewed to identify the websites they had accessed. This amounted to around 70 hours of screen recording as each student was online for about 7-8 hours when working on the day’s problem. The unit of analysis was the number of relevant propositions or concepts related to the theme of heredity, as identified by the exhaustive review of literature. The concepts articulated and studied from online resources were counted for each student for each learning phase (i.e. first meeting, first SDL period and so on). The total frequency of concepts refers to the total number of relevant concepts verbalized or studied, including those that were repeated in one session. On the other hand, the total number of different concepts did not include those that were repeated during the same session. Newly emerged concepts were those that were not previously mentioned by the individual in any prior learning phase of the day. The following excerpt from a discourse of one team during the first meeting (problem analysis phase) is shown to demonstrate how the relevant propositions verbalized by each student were counted. Excerpt taken from initial part of the first meeting: (The facilitator has just given the teams 10 minutes to come up with relevant keywords or ideas related to the problem statement.) L: Ok, what’s a gene?

63

64

Chapter 3

C: DNA or something like that… S: DNA… L: Ok, got some key words right- like chromosomes, X and Y chromosomes. S: Chromosomes, alleles, X and Y chromosome, phenotypes, genotype, dominant, recessive… L: I only learned allele. S: Allele... should have brought my entire textbook. SF: What about gametes? S: Huh? Ah, gametes… C: Cheem… (Nb: ‘Cheem’ is a Hokkien term which means complex/difficult to understand) L: What are gametes? S: Gametes are your sperm and your egg. L: Gametes, ok… is there anything like opposite to gametes? S: No, gametes are male and female. C: Both are called gametes. S: Both are called gametes ah? Male gamete, female gamete. The relevant concepts counted for this excerpt would include ‘gene’, ‘DNA’, ‘chromosomes’, ‘X and Y’, ‘alleles’, ‘phenotype’, ‘genotype’, ‘dominant’, ‘recessive’, ‘gametes’ and so on. The total number of relevant concepts articulated by student L would include one count of ‘gene’, two counts of ‘chromosome’ and three counts of ‘alleles’. However the number of different relevant concepts uttered would only count ‘gene’, ‘chromosome’ and ‘alleles’ once each. If L were to mention the proposition ‘gene’ again in say, the second meeting, it would still be counted once as the number of different relevant concepts uttered for that meeting. However for the counting of newly emerged concepts, any concepts mentioned here would not be counted again in subsequent meetings or SDL periods. The amount of time spent on-task was deduced from the students’ computer screen recordings as well as the audio recordings. T-tests were used to compare differences in pre- and post-test results. One-way ANOVA was used to find out if there were significant


differences in the mean number of concepts verbalized or studied during each learning phase. Correlation analysis was performed to examine the relationships between students’ learning outcomes with the following: verbal interactions, self-directed learning, time spent on-task and facilitator’s contributions.

Results and Discussion Results of mean student performance for the free recall essay and concept recognition pre- and post-tests showed improved scores for the post-tests. The average difference between the pre-test and post-test scores for the free recall essay questions was 9.3 (SD = 4.21), indicating a significant increase in achievement at the end of the learning process, t (8) = 6.65, p < .01. Similarly for the concept recognition test, the nine students had an average difference from pre-test to post-test of 7.9 (SD = 7.15), showing a significant improvement in their scores, t (8) = 3.31, p < .05. Correlations for the pre- and post-tests are shown in Table 1. Performance in the two different pre-tests and post-tests were significantly correlated, indicating that both measurements are valid indicators of knowledge level of students. Pre-tests and post-tests results were also significantly correlated. This demonstrates the importance of prior knowledge in students’ learning outcomes in this context. Such a finding is in line with the many studies (reviewed by Dochy, Segers, & Buehl, 1999) that demonstrate the positive effect of prior knowledge on learning. The concepts verbalized by students during the different learning phases were counted in three different ways. First, the total number of relevant concepts articulated were scored, including propositions that were repeated during each meeting or SDL time. The distribution of the average number of total relevant concepts verbalized for each of the learning phases of the PBL process for the nine students is shown in Figure 1. Second, repetition of concepts within a learning phase was excluded in the count. Figure 2 is a distribution of the average number of different concepts verbalized for each learning phase of the day by each team. Lastly, only completely new concepts were counted, that is:

65

66

Chapter 3

only when verbalized by the particular student for the first time in the day. Concepts verbalized in earlier learning phases were not included. The average number of these newly emerged concepts for each learning phase is shown in Figure 3.

Table 1. Correlation analysis between pre-test and post-test scores Tests

Free recall essay pretest

Recognition pre-test

(M = 2.9,

SD = 2.8)

Free recall essay posttest (M = 12.2,

.60*

.78**

.45

−

.41

.60*

−

.73*

(M = 35.7,

SE = 0.7) Free recall essay pre-test

−

Recognition pre-test Free recall essay posttest Recognition post-test

*p < .05, **p < .01

Recognition post-test (M = 43.6, SD = 2.5)

SD = 1.9)

−


Figure 1. Distribution of the mean number (+ SE) of total relevant concepts (includes repetitions) articulated over the different learning phases of the PBL process (N = 9)

Figure 2. Distribution of the mean number (+ SE) of different relevant concepts (excludes repetitions within each learning phase) articulated over the different learning phases of the PBL process (N = 9)

67

68

Chapter 3

Figure 3. Distribution of the mean number (+ SE) of newly emerged relevant concepts articulated for the first time in the day over the different learning phases of the PBL process (N = 9)

The one-way ANOVA revealed that the concepts verbalized differed significantly as a function of the different learning phases: for total number of concepts, F(4, 40) = 3.40, p < .05; for number of different concepts within each learning phase, F(4, 40) = 3.68, p < .05 and for number of newly emerged concepts for the day, F(4, 40) = 3.64, p < .05. Post-hoc analyses using the Games-Howell test showed that the number of newly emerged concepts verbalized was significantly higher for the second meeting (M = 8.3, SD = 4.39) as compared to the second self-directed learning (SDL) phase (M = 3.0, SD = 2.06) (p < .05) while the number of different concepts articulated during the second meeting (M = 17.0, SD = 7.43) and third meeting (M = 16.0, SD = 6.58) were significantly higher than for the first SDL phase (M = 6.9, SD = 3.48) (p < .05). Relevant concepts accessed via online resources were counted similarly. The distribution of the average number of total relevant concepts, different concepts within each meeting or SDL period and newly emerged concepts for the day that were accessed online for each of the learning phases are shown in Figures 4, 5 and 6 respectively.


Figure 4. Distribution of the mean number (+ SE) of total relevant concepts (includes repetition) accessed online over the different learning phases of the PBL process (N = 9)

Figure 5. Distribution of the mean number (+ SE) of different relevant concepts (excludes repetition within each learning phase) accessed online over the different learning phases of the PBL process (N = 9)

69

70

Chapter 3

Figure 6. Distribution of the mean number (+ SE) of newly emerged relevant concepts accessed online for the first time in the day over the different learning phases of the PBL process (N = 9)

The one-way ANOVA revealed significant differences in the numbers of concepts accessed online for the 5 learning phases: for total number of concepts, F(4, 40) = 2.75, p < .05; for number of different concepts within each learning phase, F(4, 40) = 4.42, p < .05 and for number of newly emerged concept for the day, F(4, 40) = 4.85, p < .05. Post-hoc comparisons using the Games-Howell criterion for significance showed that the number of newly emerged concepts accessed online was significantly higher for the first (M = 12.7, SD = 10.48) and second meeting (M = 13.4, SD = 10.54) compared to the third meeting (M = 0.56, SD = 1.13) (p < .05) while the number of different concepts accessed online during the second meeting (M = 25.3, SD = 10.25) was significantly higher than for the third meeting (M = 5.1, SD = 7.87) (p < .05). No other specific post-hoc contrasts were significant. The results above show that the highest number of newly emerged concepts occurred during the first meeting. This suggests that many of the relevant concepts are already known in one way or another (as also indicated by the response on the recognition pre-test) and that the initial discussion of the problem serves to reactivate those concepts. However, the highest total number of relevant concepts and number of different concepts articulated by the students occurred during the second meeting.


This suggests that the second meeting is particularly rich in terms of articulation and repetition of the concepts learned previously. Interestingly, the total number of concepts studied online was also the highest during the second meeting, whether it was the total number or the number of different concepts or the number of newly emerged concepts that was being considered. Thus, it appears that students’ research activities as well as verbal interactions were highest during the second meeting. The second meeting is the learning phase where the facilitator meets with each team to find out their progress. Since the facilitator only spends about 20 minutes with each team during this time, results here indicate that students spend the remaining time of about 40 minutes to 1 hour on-task, researching individually and discussing as a team. Thus it can be seen that in the present context, the distinctions between student-facilitator interaction, group discussion and self-study phases are blurred. This is due in part to students having their personal laptop computers with them throughout the day and are able to continually access internet resources, whether it is the discussion or SDL phase of the PBL process. Excerpts of the two teams taken from the second meeting are shown below to demonstrate the contexts in which students articulate relevant terminologies. In Team 1, students A and SF are coconstructing their understanding about RNA. It can also be seen from the example of Team 2 that the students tend to share what they have read up from online resources and in the process of further discussion, make sense of the new ideas they have read. Team 1: A: So RNA is actually information within the DNA SF: Mm, actually DNA transcribes the RNA, then RNA produces the peptide chain to become protein A: So it’s the cell reads DNA and thenSF: Actually RNA is part of DNA… (unclear) A: So it’s like the messages… like the message within the DNA SF: Yeah A: And then RNA and then the cell produces protein

71

72

Chapter 3

Team 2: ZW: We have to find out more about genes, right? About how does it play in the organism J: It says that genes produce all the protein ZW: Yeah J: Control all your proteins ZW: Yeah yeah yeah correct. Yeah I read that. You go to worksheet, first question, there is a link already. J: No I read from this. Easier cos animation. ZW: Blood contains lots of red blood cells that transport oxygen for our body. The cells use the proteins called haemoglobin to capture and carry the oxygen. From our 40000 genes, only a few contain instructions for making haemoglobin proteins. The remaining genes contain instructions for making other parts of our body J: That means we have 50000 genes ah? Z: No forty over thousand … ZW: Wow. Cool. Yeah the animation. J: You go to this website. It’s easier to understand I think. The data shown in Figures 1 to 6 also give insight into what students do during the self-directed study periods. It can be seen that generally the new concepts verbalized emerge in the first meeting and SDL phase, while there is a greater number of repetitions in the second meeting and second SDL period. A similar trend occurs for the concepts accessed online. This suggests that in the first phases of the learning process in PBL, students focus on “initial terminology articulation” while in the later phases of the process they focus on “terminology repetition”. Although we did not do a qualitative content analysis of the students’ discourse, a brief excerpt from a typical episode of verbal interaction during this learning phase demonstrates to some extent the elaboration and repetitions involved in the process. L: Can just give me a brief explanation of meiosis? SF: MeiosisA: Meiosis simplySF: Is a-


A: I try to explain- meiosis is simply the division of chromosomes but is different from mitosis as in, you know we have 46 chromosomes … so in mitosis rite they actually duplicate themselves L: Mm hm A: And then split into two and then both have 46 chromosome L: Mm hm. A: But meiosis it actually divides itself and has 23 chromosomes only- so 23 chromosomes are just for reproduction, soL: Oh for reproduction of the same cell… … SF: You say after meiosis rite, it become mitosis A: Yeah because just now I was trying to say that meiosisSF: Yeah A: The main reason it happens is because- for the reproduction of the baby. So whenSF: When the- so like they join together A: Then mitosis starts SF: The er… the zygote will undergo mitosis A: Mitosis SF: To split into many cells A: Yeah Furthermore, a comparison of the trends in Figures 1 and 4 suggests that what students do during the two SDL periods differ to some extent: the first phase involves less verbalization of concepts and more research, while students engage actively in both discussion and research during the second period. Finally, Figures 4 to 6 show that the number of relevant concepts accessed during the third meeting to be very low. This is to be expected as students would be listening to other teams present their findings during this reporting phase, and unlikely to still be actively searching for relevant resources to study. Thus the first conclusion of this study is that in the process of PBL studied here, two different phases can be observed – an initial

73

74

Chapter 3

terminology articulation phase - consisting mainly of the first meeting and first SDL period, and characterized by the emergence of new concepts articulated and studied online, and secondly, a terminology repetition phase (mainly the second meeting and second SDL period) where relevant concepts are repeated. We also found that the most extensive on-task activity occurs half-way in the process, during the second meeting; in this phase most verbal interaction and online research were taking place. It seems that students first need a certain period “to warm up” before they deeply engage in the study of subjectmatter. This could also indicate that most of the students do not fully utilize the first period of self-directed study, tending to spend it on offtask matters and getting more serious about the task-at-hand when the facilitator is present in class during the second meeting. We will now present and discuss the results for the second aim of this study investigating how student achievement is influenced by their verbalization and by their individual study. Since the number of students studied was limited, multiple regression analysis of verbalizations in the different phases of the learning process on the learning outcomes was not considered meaningful. Therefore simple descriptive correlation analysis was conducted. The results of correlation analysis between students’ post-test scores and the total number of relevant concepts articulated during the different learning phases are shown in Table 2. The results with respect to concepts uttered during the verbal interactions were as follows: The total number of relevant concepts articulated during the different learning phases for the whole day was significantly correlated with students’ post-test essay scores but not with concept recognition scores. In particular, it was the total number of concepts articulated during the meeting times (excluding the SDL phases) that correlated most significantly with students’ post-test scores. Of the three meetings, it was the total number of concepts articulated during the third meeting (reporting phase) that showed the highest correlations with student learning. This shows that increased repetition of concepts discussed leads to increased learning. Similar correlations were found when the number of different concepts articulated was considered instead. However in this case, the

Table 2. Correlation analysis between post-test scores and concepts articulated during the different learning phases

How students learn in PBL 75

76

Chapter 3

significant correlations with student learning were even higher than when the total concepts were considered. For example, the total number of different concepts for the day correlated with both the post-test essay (r = .80, p < .01) as well as the recognition test (r = .59, p < .01). This shows that students who use more different concepts in the discussions over time also learn more. Thus our findings show that both the total repetition of concepts discussed throughout the PBL process as well as the breadth of terminologies articulated play significant roles in the learning process. This is in contrast with the findings of Moust et al. (1987), who found that the amount of talk by students as such did not predict achievement. No significant correlations were found between student performances in post-tests and the number of newly emerged relevant concepts. The observation that both the total number and the number of different relevant concepts articulated during the last meeting (the reporting phase) correlate strongly with students’ learning (r ranging from .77 to .87, p < .01) indicates that students who have learned tend to share more of their findings and understandings during the reporting phase. This is a useful finding when assessing student progress at the end of the PBL process as it indicates that the amount of information articulated by the students at the end of the PBL process gives a good representation of their learning gains. The results of correlation analysis between students’ post-test scores and the total number of relevant concepts encountered online during the different learning phases are shown in Table 3.

Table 3. Correlation analysis between post-test scores and concepts studied from online resources during the different learning phases

How students learn in PBL 77

78

Chapter 3

Significant correlations were found between the total concepts studied online during the SDL phases and student achievement suggesting that individual research on online resources during SDL was important to student learning. The amount of study during the first SDL phase appeared to play a greater role in predicting students’ final learning outcomes compared to the second SDL phase as the correlations between student learning and post-test scores were statistically significant for the former while not for the latter. The total number of concepts accessed online during the first meeting (problem analysis phase) was also significantly correlated with student achievement. Similar correlations of statistical significance were found for the number of different concepts studied online during SDL times as a whole with students’ post-test results, with the breadth of concepts studied in the first SDL phase being of particular importance. The total number of different concepts accessed online during the first meeting (problem analysis phase) was also significantly correlated with student achievement. These results show that both the amount and breadth of self-directed research undertaken from the study of online resources are highly indicative of students’ learning from the PBL process. In the case of newly emerged concepts accessed from online resources, there was a significant positive correlation between total number of new concepts read throughout the day with scores only for concept recognition post-test. Significant positive correlations exist between the number of new concepts read in the first meeting (problem analysis phase) and their post-test scores as well as between that studied in the first SDL phase. A significant negative correlation exists between newly emerged concepts read during the second meeting with post-test scores. These results indicate that the more new concepts accessed or read during the earlier part of the learning process was related to student achievement, while coming across new concepts only towards the later part of the day was indicative of lesser learning by the end of the day. Students who were less on-task during the first period of selfdirected study would naturally have more newly emerged concepts when they read online resources during the second meeting compared to


others who had made use of the self-directed study time to acquire new concepts. Thus, these results show that students who made use of the initial knowledge acquisition phase to research and uncover new concepts in the process achieved more learning at the end of the day. This deduction is in line with the finding that time spent on-task correlated significantly with results for the concept recognition post-test (r = .64, p< .05). Since all the students were generally on-task during meeting times when the facilitator was present, time spent on-task was relevant mainly during the self-study periods. The other element of the PBL process which also contributed to learning was the extent of the facilitator’s contributions (recognition: r = .62, p < .05; essay: r = .62, p < .05). This result is particularly striking because the facilitator’s interventions and contributions were limited in number. As our current data cannot be used to determine the causeand-effect relationship between facilitator’s verbal interaction and students’ learning, further investigation is necessary to better understand the effect of facilitator contribution to students’ learning in the PBL context. Thus in regards to the second question to which the present study sought an answer we report that it is both the repetition as well as the range of different concepts articulated throughout the different learning phases, which correlate most strongly with students’ learning. Results of this analysis suggest that between the two, breadth of concepts verbalized plays a greater role in student learning than repetition of similar ideas. In the case of students’ self-directed study, it was revealed that being exposed to a greater number of different relevant concepts as well as having increased exposure to the same concepts correlated with students’ learning outcomes. Comparing this result with that of the correlations of the total number of students’ verbal interactions during SDL with achievement (r = .20 and .47, p > .05) suggests that individual study plays a greater role in their learning than verbal interactions with peers during the SDL phases.

79

80

Chapter 3

Conclusion and implications In conclusion, the process of PBL is characterized by two distinct phases: a phase in which there is a high degree of terminology articulation and a later terminology repetition phase. We have also shown that it is what students do during self-directed study that plays the key role in their learning outcomes. Studying a greater number of relevant concepts as well as having increased exposure to the concepts had strong correlation with students’ achievement. The concepts students articulate correlate slightly less strongly, though still significantly, with their learning, although the correlation becomes more significant when the number of different relevant propositions (i.e. breadth and variety of relevant concepts) is considered instead of the frequency (repetition of relevant concepts). At the research level, one value of this study is in demonstrating that it is possible to observe and describe what students do during selfdirected study time in a naturalistic setting by making use of a computer screen-recording program. Furthermore, by providing an account of what actually happens during an entire PBL process, the learning activities that result in learning outcomes in PBL can be better understood. This study also has implications at the practical level in science education. Researchers in the science education community have highlighted the importance of discussion and the role of talk in learning science. For example, a study by Rivard and Straw (2000) demonstrated that talk combined with writing enhanced knowledge retention while Bianchini (1997) also found that students who talked more during group work learned more. Others have also argued for science education to create learning environments which give opportunities for students to ‘talk’ science, instead of just ‘hearing’ science (Lemke, 1990; Pea, 1993). Thus the account of the scientific concepts students articulate and are exposed to in this PBL context demonstrate a plausible learning approach that can be used in the science classroom. In the study by Bianchini (1997) one problem that surfaced was the exclusion of students of lower status in terms of perceived academic


ability and popularity in group discussions and opportunities for on-task talk. This in turn had negative consequences on the students’ learning. Although classroom equity is an important issue that must be dealt with by the educator, our present study shows that giving students access to resources for individual study may help in reducing the impact of lower on-task talk. Our findings show that what students read during individual research influenced their learning outcomes more than what they verbalized. Thus one possible way to minimize differences in opportunities to learn is to provide equal access to individual learning resources. The present study has several limitations. First, as units of analysis it focused on individual scientific concepts articulated and read rather than on more integrated, higher-level units of knowledge such as theories. Thus our results can only offer limited insight into how the quality and coherence of the verbal interactions influenced student learning. An alternative to this approach may be the kind of qualitative labeling of learning-oriented interactions as conducted by VisschersPleijers et al. (2006). They have made a distinction between exploratory questioning, cumulative reasoning, and handling cognitive conflicts as basic units of a problem-based discourse. Classifying the ideas verbalized by students in a similar manner would possibly highlight more clearly the quality of students’ verbal interactions. Second, the number of participants for this study (N = 9) was fairly small. Since our intent was to investigate the processes of PBL at a microlevel, we only studied a limited number of students intensively; in fact we analyzed for each of these students more than seven hours of learning activities. Despite the small size of our sample, we found statistically significant effects on learning. Given the fact that the power of the statistical tests used was extremely small due to sample size, the significant results are the more telling. They strongly suggest that our findings are meaningful and likely to be valid. However, it may be useful to replicate the study with a larger sample size. This would enable more detailed analyses using multiple linear regressions and possibly more subtle information regarding the relationships between students’ verbal interactions, individual study, facilitator contributions, and student

81

82

Chapter 3

learning would be uncovered. In addition, it would enable us to study in more detail the influences exerted by results of one learning phase on the next. For instance, if small-group discussion influences self-directed learning activities, as is suggested by Dolmans and Schmidt (1994), then one would expect these relationships to show up in the kinds of analyses we have conducted. Third, learning during the self-directed learning phase was measured by tracking the websites accessed by the students. It is possible that students did not actually read through the information on the web pages, but were talking or doing other things at that time. While it is theoretically possible to have eye-tracking devices to monitor if students were actually reading through the resources, this is not feasible in a natural learning environment. However, the high correlation coefficients between the numbers of concepts counted using the website tracking method and students’ learning outcomes strongly suggest that this is a valid measure of students’ actual on-task activity. This methodology can now be further developed and used to answer new questions such as how different problems influence students’ online research activities or how different tutoring approaches would influence learning.

Learning activities and achievement in PBL

Chapter 4: Do the contents of learning activities in problem-based learning predict student achievement? Abstract The aim of this study was to identify relationships between the contents of the learning activities of students and their achievements in a problem-based learning context. The variables under investigation were students’ verbal interactions during different phases of the problembased learning cycle, self-directed study, and achievement. All verbal interactions engaged in, and resources studied, by 35 students were recorded for an entire problem-based learning (PBL) cycle. The particular scientific concepts articulated by each student in the group or studied individually while working on the problem-at-hand were identified as units of analysis to provide an estimate for the adequacy of the contents of students’ verbal interactions and self-directed learning. In this particular education context, verbal interactions occur in all phases of PBL − problem analysis, self-directed study and reporting. The data were analyzed using a structural equations modelling approach. Our results show that students’ verbal contributions through collaborative discussion during the initial problem analysis phase strongly influenced the extent of their verbal contributions during selfdirected learning and the reporting phases. The contents of verbal contributions and individual study influenced the nature of the contributions during the reporting phases, to a similar extent. Greater contribution of relevant concepts verbalized during the reporting phase also led to higher achievement at the end of the PBL cycle. It was found that collaborative learning plays a significant role in the PBL process, and may even be more important than individual study in determining students’ achievement.

Introduction Problem-based learning (PBL), as its name implies, uses problems as a

83

84

Chapter 4

stimulus for learning. These problems are academically or professionally relevant issues that students are to learn more about. As a group, the students analyze the problem, generate possible explanatory hypotheses, build on one another’s ideas, as well as identify key issues to be studied further. These activities allow students to construct a shared initial explanatory theory or model explaining the problem-at-hand (Schmidt, 1983). After this period of teamwork, they disperse for a period of selfdirected study to work on the learning issues identified. When they next meet as a team, they are expected to share and discuss their findings, as well as refine their initial explanations based on what they have learned. If new learning issues requiring further study are identified during this phase, the process described above is repeated. Thus, PBL can be seen as a cyclical process that is made up of three phases: initial problem analysis, followed by self-directed learning, and a subsequent reporting phase (Barrows, 1988; Hmelo-Silver, 2004). A tutor is present during the team discussions to help facilitate the learning processes. In PBL, both group and individual learning processes are recognized to play important roles in students’ learning (Schmidt & Moust, 2000; Van den Hurk, Dolmans, Wolfhagen, & Van der Vleuten, 2001). Group processes would include activities such as small group discussions, giving and receiving explanations, asking for clarification, elaboration on ideas voiced, formulation of learning issues and so on. Schmidt, De Volder, De Grave, Moust and Patel (1989), and De Grave, Boshuizen and Schmidt (1996) have demonstrated that small group discussions prior to studying relevant new information enhanced students’ learning and ability to recall new concepts learned. Webb found that giving elaborated explanations within small groups was related to achievement gains although short unelaborated responses did not benefit the helper nor the recipient (Webb, 1991). Research into the phase of individual, self-directed study has been limited. One study that gives insight into individual study in PBL and how it affects students’ learning was carried out by Van Den Hurk et al. (2001). They found that the quality of learning issues generated during the problem analysis phase had an impact on the extent to which the learning issues were used during individual study. Using learning issues


during self-directed study also influenced students’ self-study to be more explanation-oriented, which in turn influenced the ‘depth’ of subsequent reporting (as they call it), and finally led to a higher score on an achievement test. However, as recognized by the authors, a limitation to the study was that the results may have been biased to some extent due to the data being based on students’ retrospective self-report. Although studies as described above have provided some insight on how group processes and self-directed study (separately) affect student achievement, research on the function and influence of both collaborative learning and self-directed study in the PBL process is still lacking. This seems strange given that both phases are believed to be essential aspects of PBL. Without understanding their mutual influence and how they each affect student learning, important questions about how PBL works remain unanswered. As noted by Capon and Kuhn (2004), it is critical to examine the various components of PBL to ascertain what is and what is not essential for PBL to take place. For example, can students learn just as effectively if they were to focus on self-directed learning and research, without collaborative problem analysis and discussion, and vice versa? How, if at all, does each phase of PBL influence the next? Which influences students’ learning to a greater extent − collaborative group discussions or individual self-study? As argued by Hak and Maguire (2000), lack of understanding regarding the relationships between these learning processes of PBL with students’ learning outcomes means that there is “no research base for giving pertinent advice to students and tutors about how to conduct PBL tutorials (p. 771)”. Furthermore, research required to uncover which aspects of the tutorial process are crucial to students’ learning should be focused on the actual learning activities occurring in the various phases of PBL (Dolmans & Schmidt, 2006; Hak & Maguire, 2000). However, such direct-observation studies are rare as most studies on the activities that occur in PBL were conducted using student self-report, which may be biased to some extent (Schmidt & Moust, 2000; Van den Hurk et al., 2001). One observational study that did focus on the learning-oriented interactions of students was conducted by Yew & Schmidt (2009). Here

85

86

Chapter 4

the verbal interactions taking place during an entire PBL process were analyzed qualitatively and the results demonstrated that PBL stimulates constructive, self-directed, and collaborative learning processes as defined in the relevant literature. However, no relationships between the content of their interactions with subsequent learning were reported. The goal of this study, therefore, was to identify the relationships between the actual learning activities of students and their learning outcomes using data from “on-line” observations. We also sought to clarify the relations between the different learning processes in the PBL cycle: the relevance of the verbal contributions during problem analysis phase, of verbal exchanges during self-directed learning, of individual study during self-directed learning, of verbal contributions during the reporting phase, and achievement. Our hypothesis was that small-group collaboration is necessary in PBL, that it does influence individual study, and that it eventually influences achievement. Figure 1 summarizes the hypothesized relations in terms of a causal model. Figure 1. Theoretical model on the learning processes involved in PBL


As can be deduced from Figure 1, we expected the adequacy of verbal contributions in discussions as well as individual study to influence students’ achievement indirectly, through the adequacy of the students’ contributions in the reporting phase. The approach taken in the present study is new for this field in that the model was tested on data derived from actual observations in a natural classroom setting rather than on student self-report. The educational context for this study was such that even the self-directed learning phase could be observed in its natural classroom setting, allowing for insights into how students learn during this time. In the polytechnic where the study was carried out, the problem analysis, selfdirected learning and reporting phases of PBL all occur within one day. All students have a personal laptop that can be connected to the internet. First-year students generally rely mainly on internet resources for their research, and remain in class for their self-directed study. Thus it was possible to record and hence observe students’ verbal interactions and internet activities for the entire PBL process, even during the periods of self-directed study. Second, unlike previous studies (e.g. Moust, Schmidt, De Volder, Belien, & De Grave, 1987), the present study focused on the contents of what was discussed and learned, and attempted to relate these contents to subsequent achievement. To that end, we recorded group discussions about a biology problem (on blood transport around the body), logged all individual study activities of the students while they were using online resources, and recorded contributions of the facilitator. The methodology chosen was data-intensive, resulting in verbal interactions that amounted to about 60 hours and computer screen recordings of approximately 260 hours. The resulting protocols were analyzed to identify the number of concepts relevant to the theme of blood transport around the body verbalized or studied by each of the students during the different learning phases of PBL. We hypothesized that the frequency with which these concepts were used by students while discussing the problem and studying subject-matter, could be considered an indicator of the learning-oriented activities going on and would determine subsequent achievement. While we do not mean to say that the usage of

87

88

Chapter 4

a concept in itself implies understanding, we hypothesize that increased usage over time would be the result of increased learning. We believe that, in the process of learning, the students gradually build up a cognitive representation of the explanations for the problem at hand consisting of a semantic network of concepts related by links. The more students have learned about a topic, the richer and more detailed this particular network would be (Glaser & Bassok, 1989). The richer the semantic network, the more concepts the student has available to understand the issues at stake. Hence measuring the number of relevant concepts articulated and studied by the students in each learning phase gives an indication of the adequacy of these students’ learning. An indication of the validity of this approach can be deduced from a previous study using the same methodology. Here it was found that increased exposure to relevant concepts through individual study as well as increased verbalization of relevant concepts were significantly correlated with students’ achievement (Yew & Schmidt, 2008). Similarly Van Boxtel, Van der Linden and Kanselaar (2000) found that increased articulation of relevant “propositions” was significantly correlated with increased achievement. In summary, the research questions addressed were: (1) Do the contents of learning activities carried out by students in PBL predict student achievement? (2) What are the function and influence of both collaborative learning and self-directed study in the PBL process? (3) How do they affect student learning and achievement? (4) How do the learning activities in one phase of the PBL cycle affect the next phase? To this end, data from eight tutorial groups were analyzed using a structural equations modelling approach to determine the extent to which they fit the theoretical model hypothesized.

Method PARTICIPANTS Participants were 35 first-year students from a polytechnic in Singapore. In this polytechnic, all first-year students undergo a common curriculum regardless of their subject discipline. The 35 students were from two different Basic Science classes and were being recorded on the


seventh week of Semester One. Students were not new to PBL as they had already completed 6 weeks of PBL classes (from Monday to Friday) prior to this. Students and facilitators gave informed consent. EDUCATIONAL CONTEXT The PBL process in this polytechnic is unique in that a ‘one-day-oneproblem’ approach where students work on one problem per day is adopted. It takes place in a class setting consisting of about 20 students and one facilitator. The students are grouped into teams of five. The daily routine consists of meetings with facilitator interaction and periods of self-directed study or teamwork without facilitator involvement. A brief description of the day’s process is shown below:



Problem analysis phase (approximately 1 hour): Facilitator presents problem for the day. Students work in teams of five to identify their prior knowledge and learning issues.



Self-directed learning (SDL) period (approximately 4 hours): Students do individual study or work with their teams on worksheets and other resources provided. Time is spent helping one another within the team when necessary. Most of the individual study is done by reading online resources from the internet. Students meet with facilitators for about 20 minutes in between this period to share their progress and strategy of understanding the problem.



Reporting phase (approximately 2 hours): Each team presents their consolidated findings and response to the problem, defending and elaborating based on questions raised by peers and the facilitator. The team presentation is usually in the form of powerpoint slides. Here the facilitator would also clarify key ideas if necessary.

Although the PBL approach in this context has been adapted to suit the learning needs of the particular students and takes place within one day, it is to be classified as PBL based on the ‘six core characteristics of PBL’ as described by Barrows (Barrows, 1996). These characteristics include student-centred learning with small groups working under the guidance of a tutor who acts as a facilitator. Students work on authentic

89

90

Chapter 4

problems with no prior preparation so as to achieve the required knowledge. In addition, no direct instruction takes place; all learning is student-generated. Finally, it is through self-directed learning that students acquire new information (Hmelo-Silver, 2004). Dochy, Segers, Van den Bossche, and Gijbels (2003) used the same criteria in their meta-analysis on the effects of PBL for inclusion in their study. One point to note is that a unique feature of the self-directed learning phase in this context is that it also involves peer consultation and collaboration, and not just individual study alone. PROCEDURE Verbal interaction was recorded using a digital audio recorder placed at each team’s table. Students’ computer usage was tracked using the program Camtasia Studio Screen Recorder (TechSmith Corporation, Okemos, MI) installed on each student’s laptop. The students were audio-recorded and had their computer usage recorded once beforehand to allow them to be familiar with the procedure on the actual day. Both recording devices (voice and computer) were started at the beginning of the day when the problem was first shown to the students till the end of the day (a total of about 7 hours). To ensure an authentic recording that was representative of what usually happens during the self-directed study times, no facilitator or observer was present during the SDL phase. The room in which the recordings were carried out was the students’ regular classroom. MATERIALS The problem statement for the day was entitled “Heart Matters” and it introduced students to the concepts of the circulatory system and blood pressure. The problem statement is presented in Appendix C. To measure students’ achievement, a concept recognition test was administered at the end of the day after the learning process. This test was a simplification of the concept mapping technique (Novak, 1998) and consisted of a list of 47 concepts that are more or less closely related to the central topic of blood flow in the body. This set of concepts covers the domain as a whole and was based on an exhaustive review of relevant literature from textbooks and internet resources. Examples of


such concepts include ‘blood pressure’ and ‘valves’. Some 18 concepts not related to blood flow in the body (“fillers”) were interspersed in the list. Examples include ‘siphoning’ and ‘MRI scan’. Students were instructed to rate the extent to which the concepts listed were related to blood flow around the body using a scale of 1 to 5, where 1 = not at all related; 2 = a little bit related; 3 = to some extent related; 4 = quite closely related and 5 = very closely related. No time limit was set for the test. Two teachers with expertise in the field of biology were asked to identify the most appropriate answers independently. Inter-rater agreement was 81.2%. Where there were differences in rating, a third opinion from a similar expert was sought. Student rating of each concept scored 2 points if it was the same as the expert answer, 1 point if it differed by ±1, and 0 for any other answer. The concept recognition test is a measure of the extent and adequacy of the students’ scientific vocabulary. It was assumed that the accuracy in which they could identify the relevant scientific concepts gives an indication of their understanding of the theory (Alao & Guthrie, 1999; Tsai & Huang, 2002). As discussed in the Introduction, the theoretical basis of this test is that students learn by building up a cognitive representation of the circulatory system. This would consist of a semantic network of concepts linked to one another at different levels (Glaser & Bassok, 1989). Students who have learned more throughout the day would have built a richer, more detailed and accurate network, which would allow them to give more accurate responses in the recognition test (Collins & Quillian, 1969). ANALYSIS Verbatim transcripts of a total of approximately 56 hours of verbal interactions of the eight teams (approximately 7 hours per team) were produced. The computer screen recordings of the students were viewed to identify the websites they had accessed. This amounted to around 245 hours of screen recording as each student was online for about 7 hours when working on the day’s problem. The unit of analysis was the number of relevant concepts related to the theme of blood transport

91

92

Chapter 4

around the body, as identified by the exhaustive review of literature. The concepts articulated and studied from online resources were counted for each student for each learning phase (i.e. problem analysis, SDL, and reporting). The total frequency of concepts refers to the total number of relevant concepts verbalized or studied, including those that were repeated in one session. Verbal contribution during problem analysis phase was measured using the total frequency of relevant concepts uttered by each individual student during the problem analysis phase, while verbal contribution during self-directed learning phases was measured using the total frequencies of relevant concepts uttered during those phases. As described earlier in ‘Education Context’, student activity during the selfdirected learning phases is not limited to individual study and search, but also includes group discussion, peer consultation and collaboration. The extent of individual search and study conducted by students was estimated using the total frequency of relevant concepts accessed online during the self-directed learning phase. We are aware that this method would not distinguish between what students only browse through and what is actually read and studied. However, we consider it a “best guess” about what is learned in a natural setting. Alternative methods, such as asking students as to what is learned, would intrude upon the students’ learning process and perhaps distort it; our recording method does not. Furthermore, a previous study using the same methodology showed that the number of concepts accessed online during self-directed learning was significantly correlated with student achievement (r = .79) (Yew & Schmidt, 2008). Contribution during reporting phase was measured using the total frequency of relevant concepts uttered during this time. Student achievement was measured by the recognition post-test. The data were analyzed using a structural equations modelling (SEM) approach. SEM is a statistical technique used to test causal hypotheses among multivariate data. This procedure generates several statistics that enable the investigator to assess how well the empirical data fit the theoretical model and to estimate the strengths of the causal relations hypothesized. In evaluating the goodness-of-fit of the models to the sample data, we used four indicators suggested in the literature:


the Cmin/df index of fit, Chi-square, the Comparative Fit Index (CFI), and the Root Mean Square Error of Approximation (RMSEA) (Arbuckle, 2006; Browne & Cudeck, 1993; Hu & Bentler, 1999). The level of significance (p) computed from Chi-square and degrees of freedom should be higher than 0.05. The Cmin/df index of fit, yielded by dividing the minimum discrepancy (C) by its degrees of freedom should be lower than 3 and preferably close to 1 (Arbuckle, 2006). CFI values larger than 0.95 and RMSEA scores below 0.06 can be considered as indicators of good fit (Browne & Cudeck, 1993).

Results Table 1 shows the intercorrelations, means, and standard errors of the variables. There were 35 students from eight teams involved in this study. Table 1. Intercorrelations, means and standard errors of the variables (N = 35)

1.

Verbal contribution during problem analysis

2.

Verbal contribution during selfdirected learning

3.

Individual study

4.

Verbal contribution during reporting phase

5.

Achievement

1

2

3

4

5

−

.73*

.16

.75**

.29*

−

.08

.68**

.23

−

.32*

.36*

−

.35*

−

Mean

16.00

77.23

1039.48

35.17

60.31

Standard error

4.95

15.95

130.33

4.22

2.81

** p < .01; *p < .05

93

94

Chapter 4

The model displayed in Figure 1 was tested against the data, yielding the following results: Chi-square = 12.41, df = 4, p < 0.05; the minimum discrepancy, C, divided by the degrees of freedom, Cmin/df = 3.10; the square root of the population discrepancy corrected by the complexity of the model RMSEA = .25; and the Comparative Fit Index (CFI) = .86. These statistics indicate that the model does not adequately represent the data. Figure 2 shows the relevant path coefficients that are statistically significant. An inspection of the modification indices and the expected parameter statistics revealed that a slightly modified version of the original model would fit the data much better. Chi-square for this model is = 3.87 (df = 5, p = .57); Cmin/df = .77; RMSEA = .00; and the CFI = 1.00. Figure 3 shows the relevant path coefficients. Only statistically significant path coefficients are displayed.

Figure 2. Path model of the hypothesized model on the learning processes involved in PBL (error terms are omitted for readability and only statistically significant path coefficients are displayed)


Figure 3. Path model of the learning-oriented activities affecting student achievement

Discussion The purpose of the present study was to investigate the relationships between the contents of the learning activities of students in a PBL process and their academic achievement. We also sought to investigate the influence of the different learning phases in PBL on student achievement, and how the learning activities of each phase influenced the next. To this end, students’ discussions and screenshots of web pages accessed online during the entire PBL process were recorded. Relevant concepts, verbalized or encountered, related to the problem of the day were then quantified for each student in each phase and analyzed using a structural equations modelling approach. We found a model, based on existing theory with regards to PBL (Dolmans & Schmidt, 2006; Schmidt & Moust, 2000; Van den Hurk et al., 2001) to fit the data quite well. We will discuss the interrelations found one at a time and relate them to existing theory and empirical

95

96

Chapter 4

findings by others. Subsequently, we will assess the significance of our findings as a whole. The first relationship is the strong influence of the contents of students’ verbal contributions during problem analysis phase on the verbal contributions during the self-directed learning phase. Verbal contributions during problem analysis are likely to reflect students’ prior knowledge. However, the extent of students’ verbal contribution during problem analysis would also be influenced by students’ efforts to activate what they already know about the problem-at-hand (Schmidt 1983; 1992). We know from other studies that prior knowledge, once activated, influences subsequent learning (see for a review: Dochy, Segers, & Buehl, 1999). This was what was found in experimental studies of PBL as well (De Grave, Schmidt, & Boshuizen, 2001; Schmidt et al., 1989). These studies showed that groups that activated prior knowledge through problem analysis learned more from a subsequently presented problem-relevant text than those who did not. Our original hypothesis was that the extent to which students contribute in discussion during the problem analysis phase would mainly have an impact on the next learning phase − that of self-directed study. However our findings showed that students’ verbal contribution during problem analysis influenced not only the SDL phase but also the reporting phase. Hence active engagement in knowledge building and sharing during the problem analysis phase also influences that during the reporting phase. Students who had raised more ideas during the initial knowledge sharing process would perhaps also be more motivated to share and clarify more during the reporting phase, when their knowledge networks are more established. Thus the first phase of the PBL cycle is clearly a very important one since the extent of a student’s learning in the PBL process is largely determined at this point. Other factors that could influence this phase include student interest or motivation induced by the problem or the tutorial group itself (e.g. whether it is a productive or dysfunctional group). Contrary to our original hypothesis though, is that the extent of verbal contributions during problem analysis does not directly influence


the amount of individual study in the subsequent learning phase. We will discuss possible reasons for this observation later on. The next relationship of interest is a relatively strong influence of students’ verbal contribution during self-directed learning on the extent of their verbal contribution during the reporting phase. One unique feature of the self-directed learning that occurs in this education context is the collaboration students undertake during this time. While studying individually, students consult each other and interact about the topics at hand. Studies on the self-directed study phase generally investigate only issues related to the individual learning such as time spent on individual study, extent of literature search and extent of studying of literature (Van den Hurk et al., 2001; Van den Hurk, Wolfhagen, Dolmans, & Van der Vleuten, 1999). However in this context, we note that students also discuss and share ideas during self-directed learning, and this collaboration also influences their achievement indirectly through the verbal contributions in the reporting phase. It is quite surprising to note however, that unlike our original hypothesis, there is no direct relationship between verbal contributions during problem analysis phase and the amount of individual study, nor any direct relationship between individual study and verbal contributions during the selfdirected learning phase. One possible reason could be due to students who are reserved or are not comfortable to voice out their ideas readily, but do put in significant amounts of individual study during the learning process. Since every student is generally required to share some of their findings and response to the problem during the reporting phase, those who have done more individual study would also be able to contribute more during this phase. Another possible reason for the results that deviate from our original hypothesis could be due to students’ perceptions of what makes learning effective in PBL. Although they are not new to PBL, having been in a PBL environment for six weeks of a semester, all of them have come from traditional ‘teacher-centred’ learning environments. Hence there could be significant numbers of students who do not see the benefits of or are not used to the idea of talking and sharing what they have learned spontaneously during selfdirected learning time. Another possibility could be due to students not

97

98

Chapter 4

wanting to share what they have learned during self-study time with their peers. Several studies have shown that peer discussion is important for sharing knowledge and increasing the “collective knowledge of a group” (Hmelo-Silver & Barrows, 2008; Rivard & Straw, 2000). Some students may not like to share their findings with their teammates, preferring only to share what they have read up on during the reporting phase when the facilitator is present, in an attempt to show that they have done more than their peers. As these are only possible speculations, further studies are needed to investigate the possible effects of students’ perceptions of effective learning in PBL on their learning activities. Another relatively strong relationship is the impact of the extent of individual study on students’ verbal contribution during the reporting phase. A study by Van den Hurk et al. (2001) showed that the quality of student individual study (based on whether learning was done in an ‘explanation-oriented way’) influenced the depth of students’ reporting. This seems similar to our findings where individual study influenced verbal contributions during the reporting phase. However a surprising finding in our context is that there was no direct path from individual study to achievement. Individual study influences achievement indirectly, through verbal reporting. Similarly there is no direct relationship between students’ prior knowledge as indicated from their verbal contribution during problem analysis phase to achievement. These findings underline the importance of actively constructing through verbalization for learning to emerge. Our model thus clearly shows the importance of verbalizations throughout the PBL process. In trying to make sense out of the problem, students produce explanations, initially based on prior knowledge, but in later phases also based on what was learned from fellow students and from the materials studied on the internet. In an earlier study using a similar “online” methodology, we have demonstrated that more than 50% of the learning-oriented verbal exchanges in the small group were collaborative in nature (i.e. involving co-construction, sharing of information etc). Visschers-pleijers et al. similarly found that more than 60% of the verbal interaction during the reporting phase of a PBL tutorial consisted of ‘cumulative reasoning’ processes. Although no


relations to achievement were reported in the studies cited above, Rivard & Straw (2000) showed that giving opportunities for collaborative exploratory talk significantly improved students’ post-test scores as compared to the groups which did not have peer interaction. Chi, De Leeuw, Chiu and Lavancher (1994) have also demonstrated that eliciting self-explanations by students results in increased learning. They found that the more students self-explained, the deeper the understanding of the topic that was achieved. Thus the significant role played by verbalizations of ideas in explaining student achievement in our study may be attributed to the PBL approach, which provides opportunities for learning-oriented discussion, and encourages students to verbally interact. One may argue that students have different levels of knowledge to start with, that those with more prior knowledge tend to contribute more verbally, and that these students would eventually also do better on the achievement test. In this line of thought, the path coefficients found would only reflect that initial differences in aptitude tend to replicate themselves in the different phases of the process and are in itself not an indication of learning. However if this were so, there would likely to be a direct influence of the verbal contributions during the problem analysis phase on students’ achievement in the path model. Our results show that this is not the case. Moreover, each phase in the learning process appears to have a unique contribution to learning in the next phase. Our findings also suggest that learning in each phase of the PBL cycle is a precondition for subsequent learning, thus providing support for the PBL cycle of initial problem analysis, followed by self-directed learning, and a subsequent reporting phase as described by various authors (Barrows, 1988; Hmelo-Silver, 2004). Moreover, it appears that collaborative learning is to some extent dominant over individual study in predicting students’ performance in this PBL context. This can be seen by the strong path coefficients between the verbal contributions at the different phases and eventually to achievement. Hence, the answer to the question raised in the introduction − Can students learn just as effectively if they were to focus on problem-based, self-directed learning

99

100

Chapter 4

and research without collaborative problem analysis and discussion? − is no, collaborative learning has been shown to be an integral part of the PBL process. The present study has several limitations. First, our structural equation modelling approach has been what is described by Jöreskog & Sörbom (1996) as a ‘model generation’ approach, where an initial model is fit to data and then modified as necessary until it fits adequately well. MacCallum and Austin (2000) warned in their review that such modifications may sometimes lack validity and are susceptible to chance effects. We recognize that our resulting model is data-driven to some extent, but would also argue that our modifications are meaningful in the context and do not contradict existing theory. The main difference in our modified model is that the role of verbal contribution during problem analysis phase influences not only the next SDL phase but also the reporting phase. Such a model is still in line with our original theoretical basis and also with the findings demonstrating the importance of prior knowledge (reviewed by Dochy, Segers and Buehl, 1999), and that of active integration of new knowledge with prior knowledge (Chi et al., 1994). Second, and related to the first issue, the number of participants for this study (N = 35) was by necessity rather small (although sufficient to demonstrate the expected effects). The data-intensive approach chosen, with its emphasis on the detailed analysis of verbal protocols and internet log files, precluded the involvement of more students. Therefore, we were not able to test our model against a new group to check whether it would survive cross validation. Cross validation is also necessary to check whether our findings are limited to the particular brand of PBL studied or has a broader significance (MacCallum & Austin, 2000). The reader is reminded here that PBL as practiced in this study is a specific ‘one-dayone-problem’ approach. While we have argued that this approach is indeed PBL as it contains all the defining characteristics of PBL (Barrows, 1988; Schmidt, 1993), the short learning cycle, in particular with the SDL phases being only about 4 hours in total, is likely to have an impact on students’ learning processes. Thus our findings in this study may not necessarily be easily extrapolated to PBL in other contexts.


While this study does have several shortcomings, it is a first attempt at developing and testing a model of the process of PBL in an actual educational context using naturalistic data. In particular, we think this study represents the first attempt to provide an insight into the selfdirected learning phase of student learning using an analysis of usage of online resources. While we recognize possible limitations to this methodology, our results show that students’ learning can be predicted by counting the number of relevant concepts they have accessed during the individual study period. Of course, the use of concept counting only provides an estimate of the quantity of students’ learning without providing information regarding the quality or depth of their understanding. We have however shown that this method, while not without flaw, does fairly accurately predict students’ achievement. However studies in which students’ verbal contributions are analyzed more qualitatively should be carried out to further verify our findings.

Conclusion In conclusion, we have shown that the contents of learning activities undertaken in PBL play important roles in predicting student achievement. Both collaborative learning (verbal interactions) and selfdirected study appear to have similar degrees of influence on and importance for students’ learning. Moreover, our findings suggest that group processes in PBL provide opportunities for active construction of what is learned during self-study. In order for what is learned during self-study to lead to eventual achievement, verbalization of ideas within the group appears to be essential. Finally we have shown that the learning in each phase of the PBL process is dependent on the earlier phase, thus providing support for the idea that PBL is indeed a process of sequential steps each building upon the other as described by various authors (Schmidt, 1993). Our study has thus provided insights into the active learning within the PBL process through a naturalistic approach.

101

Cumulative learning in PBL

Chapter 5: Is learning in problem-based learning cumulative? Abstract The process of PBL is generally organized in three phases (problem analysis, self-directed study and reporting), which involve both collaborative and self-directed learning processes. The central thesis tested in this study is whether learning in the different phases of PBL is cumulative such that the learning in each phase depends on the previous phase. This was done by tracing and analyzing the learning process of students throughout all the phases of PBL. We also sought to understand how students learn in the different phases of PBL in terms of concept acquisition and elaboration. Thirdly, we also aimed to develop an efficient method to capture and quantify students’ learning during the PBL process. The scientific concepts recalled by 218 students during a concept recall exercise at the end of each of the three PBL phases were identified as units of analysis to provide an estimate for the quantity of students’ learning during that phase. The data were analyzed using a structural equations modelling approach. Our hypothesis was that students’ existing knowledge would influence the relevant concepts students learn in the problem analysis phase. The learning in the problem analysis phase would subsequently influence that in the selfdirected study phase, and this in turn, would affect that in the reporting phase. Our results show our hypothesized model to generally fit the data well while alternative hypotheses such as that students’ achievement is predicted only by collaborative learning or only by self-directed learning did not. Hence we conclude that the learning in each phase of the PBL process is cumulative, and is strongly influenced by the earlier phase, thus providing support for the PBL cycle of initial problem analysis, followed by self-directed learning, and a subsequent reporting phase. Our results also suggest two distinct phases in the PBL process, the first being dominated by initial terminology articulation and the second by terminology repetition. Lastly we demonstrate that it is possible to

103

104

Chapter 5

capture and quantify students’ learning during the PBL process in an efficient way − a finding that may facilitate future studies into the process of PBL.

Introduction Educators have long been advocating ‘active’ learning whereby students are engaged in meaningful activities as part of their learning process. Active learning has been generally defined as any instructional strategy that involves “students in doing things and thinking about what they are doing” (Bonwell & Eison, 1991, p. 2). Given such a broad definition, active learning can be viewed as encompassing a wide variety of instructional methods. Although various studies have demonstrated the effectiveness of promoting student engagement using interactiveengagement methods compared to those in traditional courses (reviewed by Michael, 2006; Prince, 2004), questions about how students learn while being actively engaged, both individually and when in collaborative small groups, remain to be further investigated. Generally learning is thought to be a cumulative process where new learning builds upon knowledge acquired in a previous phase. In the case of active learning, it is generally assumed that both collaborative learning episodes and individual self-directed study phases play important roles in students’ learning. Although the idea that new learning is dependent on what has been learned previously is almost universally accepted, demonstrations of its truth have been largely confined to the psychological laboratory, particularly in the field of text processing (e.g. Bransford & Johnson, 1972; Kintsch & Van Dijk, 1978). To our knowledge, no natural classroom demonstration of the cumulative nature of learning exists to date. Moreover, since social constructivism suggests that knowledge is mainly constructed by means of collaborative interactions (e.g. Cobb, 1994; Driver, Asoko, Leach, Mortimer, & Scott, 1994), it is possible that the effects of active learning on achievement are really only due to the group interactions and coconstruction of knowledge. Alternatively, since research on selfregulated learning has shown that the use of self-regulated learning strategies strongly influences academic achievement (Zimmerman, 1990), it can be argued that it is the individual self-directed learning


phase that is most important to students’ learning. The purpose of this paper therefore is find the extent to which active learning is cumulative and whether it involves both collaborative and self-directed learning, in the context of problem-based learning (PBL). PBL is an example of an active-learning approach in which students are given the opportunity to learn independently as well as collaboratively, while understanding an ill-structured problem. It was originally developed in medical schools to help students integrate basic science and clinical knowledge, as well as to develop clinical reasoning and lifelong learning skills (Barrows, 1986). However it is now of increasing interest to educators of various levels and disciplines (Gallagher, Stepien, & Rosenthal, 1992; Kolodner et al., 2003) as it provides a structured framework of active and collaborative learning, in line with current understanding of learning as a constructive and coconstructive activity involving social interactions (Glaser & Bassok, 1989; Palincsar, 1998). As will be described in greater detail later on, PBL involves a sequential series of learning phases that emphasizes collaborative and individual self-directed learning at different points in time. The assumption underlying PBL is that learning in the PBL process is cumulative − learning in one phase is dependent on the previous, and also that both co-construction with peers and individual construction of concepts during self-directed study contribute to student learning (Schmidt, 1983). We therefore seek to test the assumptions regarding the nature of learning in PBL, by tracing the learning process of students throughout all the phases of PBL. The central thesis to be tested is whether learning in the different phases of PBL is cumulative − does the learning in each phase depend on the previous phase? Or are some phases of the PBL process more (or less) important than others? Secondly, we also seek to understand how students learn in the different phases of PBL in terms of concept acquisition and elaboration. A third objective is to devise an efficient and valid method to track students’ learning as it unfolds in the course of the PBL process. We will first give a brief introduction to PBL. PBL always starts with a problem, for which students do not prepare beforehand. After the description of the problem is given to small groups of students, they first

105

106

Chapter 5

analyze the problem, generate possible explanatory hypotheses, build on one another’s ideas, as well as identify key issues to be studied further. These activities allow students to construct a shared initial explanatory theory or model explaining the problem-at-hand based on their prior knowledge (Schmidt, 1983). After this period of teamwork, they disperse for a period of individual study to work on learning issues they have identified as a group. When they next meet as a team during what is called the ‘reporting phase’, they are expected to share and discuss their findings, as well as refine their initial explanations based on what they have learned. Students would then move on to analyze a new problem, or if new learning issues requiring further study are identified during this phase, the process described above would be repeated. Thus, PBL can be seen as a cyclical process consisting of three phases: initial problem analysis, self-directed individual learning, and a subsequent reporting phase (Barrows, 1988; Hmelo-Silver, 2004; Schmidt, 1993). A tutor is present to guide students’ learning in the problem analysis and reporting phases. The tutor’s role is to facilitate the processes involved when students co-construct knowledge through discussions and sharing of ideas (Hmelo-Silver & Barrows, 2006). In PBL, both group and individual learning processes are recognized to play important supplementary roles in students’ learning (Schmidt & Moust, 2000; Van den Hurk, Dolmans, Wolfhagen, & Van der Vleuten, 2001). Various studies have focused on how students learn in the different phases of the PBL cycle. The initial problem analysis activates students’ prior knowledge and allows them to relate new information in the problem to their existing knowledge. Hearing what other students elaborate upon could also serve to activate or uncover the less accessible prior knowledge in the listeners. Studies by De Grave, Schmidt, & Boshuizen (2001) and Schmidt, De Volder, De Grave, Moust and Patel (1989) have demonstrated that elaboration during problem analysis in a small group prior to studying problem-relevant new information resulted in increased knowledge acquisition and recall. As argued by De Grave, Boshuizen and Schmidt (1996), such elaboration and activation of existing knowledge are instrumental in restructuring and transferring concepts resulting in the construction of new knowledge and ideas. The process of discussion during the problem analysis phase would also


result in students realizing the gaps between their existing knowledge and what they are required to know in order to respond to the problem. Thus students would identify these gaps as learning issues to be studied further during the self-directed learning phase. This individual study phase is a key feature in PBL, in line with its underlying “studentcentred” philosophy of enabling students to take responsibility for their own learning by deciding what to study and to what extent. Through the self-directed learning phase, students learn important skills such as goal setting, planning and self-control in terms of time and task-management (Zimmerman, 2002). As students implement their course of actions to achieve their goals, they would also have to monitor and reflect on their own progress, thus exhibiting a kind of feedback loop in the process (Hmelo-Silver, 2004; Zimmerman, 1990). When the tutorial group reconvenes to report their findings and the results of their individual study, opportunities are given to students to present, explain and defend their ideas, and in the process, to restructure or refine their own knowledge networks (Schmidt & Moust, 2000). The discussions during the reporting phase are centred on students’ response to the problem statement given in the problem analysis phase. Studies have shown that group interactions such as elaborations and co-constructions take place during this phase, allowing for collaborative knowledge construction (Hmelo-Silver & Barrows, 2008; Visschers-Pleijers, Dolmans, Wolfhagen, & Van der Vleuten, 2004; Yew & Schmidt, 2009). A few studies have examined and tested how the variables thought to be active in PBL influence and relate with one another and students’ learning outcomes. Gijselaers and Schmidt (1990) tested a path model relating input variables such as the quality of problems, tutor performance and students’ existing knowledge, process variables such as group functioning and time spent on self-directed study, and the outcomes of learning. They demonstrated that problem quality influences tutorial group functioning, which in turn had an influence on the amount of time spent in individual study. More time put into individual study led to increased academic achievement. This model was further refined by Van der Hurk, Dolmans, Wolfhagen, & Van der

107

108

Chapter 5

Vleuten (2001). They investigated in more detail what actually happens to learners during problem analysis, individual study and reporting. They found that the quality of learning issues generated during the problem analysis phase had an impact on the extent to which the learning issues were used during individual study. Increased usage of learning issues during self-directed study also influenced students’ research to be more explanation-oriented, which in turn led to a “deeper discussion” during the reporting phase. Finally the depth of reporting led to a higher score on an achievement test. Both of these tests of a causal model provide insight into the relationships between the variables important in the PBL process and hence into how students learn in PBL. In particular, the study by Van der Hurk et al. suggests that learning in PBL is indeed cumulative. Their study demonstrates that learning in the problem analysis phase influences individual study, which in turn influences the reporting phase, and finally achievement. However, as recognized by the authors, a limitation to both studies was that data were obtained based on students’ perceptions and retrospective self-report rather than on their actual behaviors. As argued by Dolmans & Schmidt (2006), and Hak & Maguire (2000), the research required to uncover the relationships between aspects of the tutorial process and students’ learning should be focused on the actual activities occurring in the various phases of PBL. Some studies have used direct observational methods to examine how and what students learn during PBL. One observational study focusing on the content of the learning-oriented interactions of students was conducted by Yew and Schmidt (2009). Here the verbal interactions taking place in an entire PBL process were audio recorded and analyzed qualitatively. While the results demonstrated that PBL stimulates constructive, self-directed and collaborative learning processes, no relationships between the content of their interactions with subsequent learning were reported. In addition, due to the data- and time-intensive nature of the methodology involved, the sample size used in the study was limited, thus making statistical analysis difficult. A recent study by Hmelo-Silver and Barrows (2008) analyzed in detail the knowledge building process in a PBL tutorial by examining the discourse of


students and facilitator throughout both the problem analysis and reporting phase of a PBL tutorial. This was carried out by videotaping five students as they worked on a problem over 5 hours in two sessions. The study demonstrated how an expert facilitator guided the group discourse with the use of open-ended metacognitive questions, and how students actively worked on enhancing and refining their collective knowledge throughout the group interaction portions of a PBL cycle. However this study again did not relate the quality of students’ verbal contributions to outcomes of their learning. There have been several other attempts to trace the learning process in PBL. Visschers-Pleijers, Dolmans, de Leng, Wolfhagen, & Van der Vleuten (2006) made use of video recording while other researchers have made use of stimulated recall (De Grave et al., 1996), and thought sampling (Geerligs, 1995) to provide qualitative descriptions of the actual behaviors and activities in a PBL tutorial. The difficulty of such approaches is that they do not easily allow for the quantification of learning. In addition, they are so data-intensive that studying larger numbers of students becomes almost impossible. A case in point is our own previous attempt to identify the relationships between learning activities of students in PBL with their learning outcomes (Yew & Schmidt, 2008). We recorded all verbal interactions of two groups of students for an entire PBL cycle. In addition, we logged all their individual study activities, which were conducted through the use of computers. The resulting protocols, consisting of around 72 hours of material were segmented into ‘idea units’ consisting of the scientific ideas that were exchanged and studied (Meyer, 1985). The units of analysis selected were the relevant scientific concepts found in the protocols as expressed by the individual students during discussion and encountered during individual study on the internet (below more about the relevance of scientific concepts for studying learning on-line). We identified and counted the relevant scientific concepts articulated by each student during the different PBL phases and those they studied individually while working on the problem-at-hand. By analyzing the number of concepts acquired over the different learning phases for the nine students we identified two distinct phases in the PBL process − an

109

110

Chapter 5

initial concept articulation phase, in which students are exposed to and articulate new ideas, and a later concept repetition phase, in which ideas acquired seem to be repeated and elaborated upon. Given the small number of students involved, however, further statistical analysis of the data proved impossible. A second study using the same methodology included a larger sample size of 35 students and thus enabled us to analyze the quantitative relationships between students’ verbal interactions during different phases of the problem-based learning cycle, self-directed study, and achievement, using a structural equations modelling approach. Our results showed that students’ verbal contributions through collaborative discussion during the initial problem analysis phase strongly influenced the extent of their verbal contributions the reporting phase. Greater contribution of relevant concepts verbalized during the reporting phase also led to higher achievement at the end of the PBL cycle. The methodology as used in these studies assumes that exposure to (from computer screen recordings of internet study resources) or the articulation of a concept during discussion can be considered a proxy of the learning taking place. However it is possible that students may not really understand the concepts they were verbalizing, or could be simply scanning the computer screens without seriously studying the material before them. In addition, the recording and transcription of all the learning activities throughout a PBL cycle turned out to be extremely time-consuming, thus limiting the sample size that could be utilized for each study. In this present study, we therefore sought to devise a more efficient and potentially more valid method to track students’ learning as it unfolds in the course of the PBL process. For this purpose, we used a concept recall procedure to capture and quantify students’ learning during the PBL process. Students were asked to spontaneously recall and list all the concepts they considered relevant to the problem-at-hand at the end of each PBL phase (i.e. at the end of problem analysis, selfdirected learning, and reporting phase). The assumption behind this procedure is that recall is probably a better – or at least more conventional – measure of what is actually learned compared with


counting the number of times ideas were uttered or encountered. Scientific concepts relevant to understanding the problem-at-hand were used as units of analysis. Since, as Solomon, Medin, and Lynch argued, “concepts are the building blocks of thought” (1999, p. 99), we suggest that the nature and the number of concepts recalled at any moment by a learner represent the extent of his current understanding of the problem-at-hand. The underlying assumption here is that, while learning, students structure knowledge in semantic networks of related concepts connected by associational links (Collins & Quillian, 1969; Hovardas & Korfiatis, 2006). A beginner’s initial network would consist of a few isolated concepts or ideas that are poorly connected. Therefore, if asked to retrieve relevant concepts from these cognitive structures, his memory will be limited. The more students have learned about a topic, the richer, more coherent, and more detailed this particular network would be (Glaser & Bassok, 1989). As learning progresses, more linkages and integration between new and existing ideas are constructed. Therefore, students who have learned more effectively would be able to recall more concepts and would do that more easily (Collins & Quillian, 1969; Rumelhart & Norman, 1978). Hence, measuring the number of relevant concepts students were able to recall relevant to the problemat-hand at the end of each learning phase gives an indication of the extent and quality of students’ learning in that particular phase. To our knowledge, no natural classroom demonstration of the cumulative nature of learning exists to date. Moreover, since social constructivism suggests that knowledge is mainly constructed by means of collaborative interactions (e.g. Cobb, 1994; Driver et al., 1994), it is possible that the effects of active learning on achievement are really only due to the group interactions and co-construction of knowledge. Alternatively, since research on self-regulated learning has shown that the use of self-regulated learning strategies strongly influences academic achievement (Zimmerman, 1990), it can be argued that it is the individual self-directed learning phase that is most important to students’ learning. The purpose of this paper therefore is find the extent to which active learning is cumulative and whether it involves both collaborative

111

112

Chapter 5

and self-directed learning, in the context of PBL. Figure 1 summarizes our hypothesized relations in terms of a causal model. We hypothesized that learning in PBL is a cumulative process where the learning in each new phase builds upon knowledge acquired in a previous phase. The process is initially driven by the prior knowledge that students bring with them to the classroom and the learning in each of the PBL phases influences student achievement. As mentioned earlier, it could be argued that the effects of active learning on achievement are mainly due to the group interactions and co-construction of knowledge or alternatively, that it is the individual self-directed learning phase that is most important to students’ learning. We therefore test our hypothesis against these alternative hypotheses: 1. Learning in PBL is only influenced by phases involving collaborative learning and co-construction; 2. Learning in PBL is only influenced by self-directed study and 3. Learning in PBL is influenced by both collaborative learning as well as self-directed study, but not in a sequential cumulative manner. These alternative models are summarized in Figure 2 below. Secondly, we hypothesize that the different PBL phases would involve the acquisition of new ideas (concepts) and the elaboration of previously acquired concepts to different extents. In an earlier study involving only nine students, we have shown that two different phases of the PBL process could be observed: an initial terminology articulation phase − consisting mainly of the problem analysis phase and initial SDL period, and characterized by the emergence of new concepts articulated and studied online, and secondly, a terminology repetition phase (mainly the later part of the SDL phase) where relevant concepts are repeated (Yew & Schmidt, 2008). Here we aim to test this “acquisitionelaboration theory” of learning in PBL again, this time using a larger sample size. Finally, an important auxiliary issue is: How can students’ learning be recorded as it unfolds? Through this study, we also aimed to develop and evaluate an efficient method to capture and quantify students’ learning during the PBL process so that causal relationships in the PBL process can be identified through path analysis.


Figure 1. Hypothesized model on the relationships between different learning phases of PBL

113

114

Chapter 5

Figure 2. Alternative models on the relationships between the different learning phases of PBL


Method PARTICIPANTS Participants were 218 students from 11 randomly selected classes. The students were in their second year in the School of Applied Science at a polytechnic in Singapore. Data was collected from these students during the third week of their Molecular Cell Biology class. As they had already completed one year of study in the polytechnic, students were not new to the PBL approach described below. Students and facilitators gave informed consent. EDUCATIONAL CONTEXT The PBL process adopted at this polytechnic is somewhat unique in its ‘one-day-one-problem’ approach. Here students work on one problem per day. Each class has a maximum of 25 students working together in teams of five. A brief description of the day’s process is described below:



Problem analysis phase (approximately 1 hour): The facilitator presents the problem for the day. Students work in teams of five to identify their prior knowledge and learning issues.



Self-directed learning (SDL) period (approximately 4 hours): Students do individual study or work with their teams on worksheets and other resources provided. They are also able to access other resources from the internet or textbooks. Time is spent helping one another within the team when necessary. Students meet with their facilitator for about 20 minutes in between this period to share their learning progress and strategy of understanding the problem.



Reporting phase (approximately 2 hours): Each team presents their consolidated findings and response to the problem, defending and elaborating based on questions raised by peers and the facilitator. The team presentation is usually in the form of powerpoint slides and the facilitator would also clarify key ideas if necessary.

Although the PBL process in this institution was adapted to suit the learning needs of the students and is completed within one day, it

115

116

Chapter 5

remains classified as PBL based on the ‘six core characteristics of PBL’ described by Barrows (1996). The characteristics include studentcentred learning whereby students work in small groups under the guidance of a tutor who facilitates the learning process. Problems are used as the stimulus for students’ learning with no opportunities to prepare beforehand. Furthermore, facilitators do not provide direct instruction. Instead, students construct their own understanding through self-directed learning (Hmelo-Silver, 2004). An additional feature of the PBL approach in this context is that instead of only individual study during the self-directed learning phase, peer consultation and collaboration also take place during this time. PROCEDURE A concept recall exercise was designed to estimate the number of relevant concepts that students were able to recall at the end of each PBL phase: problem analysis, self-directed learning and reporting. Our assumption is that as students engage in problem analysis, self-directed learning, group discussions, and/or peer teaching, they would be building networks of concepts related to the different learning issues as well as making relations between their prior knowledge and new ideas (Glaser & Bassok, 1989). As learning progresses, more specific terminologies would be used to articulate the newly acquired knowledge. Students who have learned more effectively would also be able to retrieve the information more easily during the concept recall exercise (Gijselaers, 1996). Hence measuring the number of relevant concepts students were able to recall at the end of each learning phase gives an indication of the quality of students’ learning, as well as the concepts they were exposed to, either from what they had read or discussed during that phase. The concept recall exercise was given to the students three times in the day − at the end of the problem analysis phase, self-directed learning and reporting phase. It consisted of the following instruction: “List all the keywords or terminologies that are related to DNA and/or RNA.” (Understanding the structure of DNA and RNA was the focus of the particular day’s learning.) Students were instructed to only list concepts or keywords they thought were relevant, and not write in paragraphs or


sentences. They were not allowed to discuss their answers or to refer to any resources when completing the exercise. MATERIALS The problem statement for the day was entitled “Made for the Job” and introduced students to concepts related to the structures and functions of DNA and RNA. A week prior to the problem, students were given an essay pre-test consisting of the following instruction: “Describe and explain as much as you know about the structure of DNA and RNA.” This was to measure students’ prior knowledge in regards to the topic. The same essay question was administered as a post-test immediately after the day’s problem to measure students’ learning achievement. No time limit was set but students were instructed to complete the test on their own without referring to any resources. The problem statement is presented in Appendix D. The “idea unit” was used as the entity for scoring the free recall essay tests for accuracy (Meyer, 1985; Schiefele & Krapp, 1996). Answers were segmented into idea units, which was defined as a statement ending with a comma, period, or ‘and’. A score of 2, 1 or 0 was awarded to each idea unit. A score of 2 was given for a completely correct idea unit, 1 for a partially correct idea unit and 0 when the idea unit was completely incorrect. The first and second authors independently scored about 20% of the tests with interrater correlation of r = .91. The remaining tests were scored by the first author. ANALYSIS Students’ answers to the concept recall procedure were analyzed by awarding 1 point to each relevant concept given by the student. Rating was done by both first and second author and differences in opinion were solved by discussion. Total scores of each student for each PBL phase were then further analyzed. The relevant concepts listed were counted for each student for each learning phase (i.e. problem analysis, self-directed learning and reporting). The total number of concepts refers to the total number of relevant concepts recalled, including those that were repeated in one

117

118

Chapter 5

session. Newly emerged concepts were those that were not previously mentioned by the individual in any prior learning phase of the day. Repeated concepts were those that were previously recalled in an earlier learning phase. For the problem analysis phase, newly emerged and repeated concepts were deduced by comparing concepts listed at the end of the phase during the concept recall exercise with the concepts written in the pre-test answers. T-tests were used to compare differences in pre- and post-test results. One-way ANOVA was used to find out if there were significant differences in the mean number of relevant concepts recalled at the end of each learning phase. The data was also analyzed using structural equations modelling (SEM), a method that is able to test causal hypotheses among multivariate data. The pre- and post-test results as well as the total number of relevant concepts recalled by the students at the end of each PBL phase were analyzed for this structural equation modelling analysis. The method generates several statistics that enable the investigators to assess how well the empirical data fits the theoretical model and to estimate the strengths of the causal relations hypothesized. Four indicators suggested in the literature were used to evaluate the goodness-of-fit of the models to the sample data, namely, the Cmin/df index of fit, Chi-square, the Comparative Fit Index (CFI), and the Root Mean Square Error of Approximation (RMSEA) (Arbuckle, 2006; Browne & Cudeck, 1993; Hu & Bentler, 1999). The level of significance (p) computed from Chi-square and degrees of freedom should be higher than 0.05. The Cmin/df index of fit yielded by dividing the minimum discrepancy (C) by its degrees of freedom should be lower than 3 and preferably close to 1 (Arbuckle, 2006). CFI values larger than 0.95 and RMSEA scores below 0.06 can be considered as indicators of good fit (Browne & Cudeck, 1993).

Results and Discussion Results of mean student performance for the free recall essay pre- and post-tests showed improved scores for the post-test. The average difference between the post-test and pre-test scores for the free recall essay questions was 4.88 (SD = 3.88), indicating a significant increase in


achievement at the end of the learning process, t (217) = 21.31, p < .01. The pre- and post-tests were significantly correlated at r = .44, p < .01. The relevant concepts recalled by students at the end of each learning phase during the concept recall exercise were counted in three different ways − the total number of relevant concepts including those which were repeated, newly emerged concepts as well as repeated concepts. The distribution of the average number of these relevant concepts is shown in Figure 3.

Figure 3. Distribution of the mean number (+ SE) of total, new and repeated relevant concepts recalled at the end of the different learning phases of the PBL process (N = 218)

The one-way ANOVA revealed that the concepts verbalized differed significantly as a function of the different learning phases. The assumption of homogeneity of variance was violated so the Brown-

119

120

Chapter 5

Forsythe F-ratio is reported. There was a significant effect of the learning phase on the total number of concepts, F(2, 618.13) = 55.59, p < .01; number of newly emerged concepts, F(2, 609.93) = 79.32, p < .01 and repeated concepts, F(2, 497.73) = 156.06, p < .01. Post-hoc analyses using the Games-Howell test showed that the total number of relevant concepts recalled was significantly higher after the self-directed learning phase (M = 9.88, SD = 4.86) as compared to after the problem analysis phase (M = 15.71, SD = 6.52) (p < .05) and the reporting phase (M = 12.15, SD = 5.94). The total number of relevant concepts recalled after the reporting phase was also significantly higher compared to after the problem analysis phase. For the number of newly emerging concepts, these were significantly higher in the problem analysis phase (M = 8.25, SD = 4.10) and self-directed learning phase (M = 8.85, SD = 4.06) compared to the reporting phase (M = 4.66, SD = 2.99), while for the repeated concepts, these were significantly higher in the self-directed learning phase (M = 6.86, SD = 4.19) and reporting phase (M = 7.49, SD = 4.71) compared with the problem analysis phase (M = 1.68, SD = 1.85). These significant differences are indicated in Figure 3. The distributions of concepts indicate that the self-directed learning phase is rich both in the acquisition of new concepts as well as the reiteration and repetition of concepts previously exposed to. That there was a high number of new concepts at the end of the problem analysis phase (as compared to students’ pre-test answers) suggests that the discussion during this phase helped to activate students’ prior knowledge, as previous studies have suggested (De Grave et al., 2001; Schmidt et al., 1989). We also observe that the reporting phase is characterized more by repetition of concepts rather than being exposed to new ones. This result is similar to the findings by Yew & Schmidt (2008) who identified two distinct phases of initial terminology articulation and a later terminology repetition in the PBL process from a group of students’ online research data and verbal interactions. Our results strengthen their findings, which were limited due to small sample size.


One surprising observation from the distribution of concepts in Figure 3 is that the total number of concepts recalled during the reporting phase is less than that in the self-directed learning phase. One would expect that by the end of the whole PBL cycle, students would be able to recall more relevant concepts. Possible reasons for this observation could be that students have already started to forget some of the concepts learned within the day, or they could also be mentally drained by the end of an intensive day’s work. Table 1 shows the intercorrelations, means and standard deviations of the variables used in the structural equation model. Prior knowledge as measured by the essay pre-test is significantly correlated to students’ learning achievement and the total number of concepts recalled after each of the PBL phases. Students’ achievement is also significantly correlated to the concepts recalled at the end of each PBL phase. It can also be seen that the concepts recalled at the end of the different PBL phases are highly correlated with one another. The hypothesized model displayed in Figure 1 was tested against the data, yielding the following results: Chi-square = 7.84, df = 5, p = .17; the minimum discrepancy, C, divided by the degrees of freedom, Cmin/df = 1.57; the square root of the population discrepancy corrected by the complexity of the model RMSEA = .05; and the Comparative Fit Index (CFI) = .97. Figure 4 displays the path diagram of the model, showing the significant paths. The parameter estimates for the model were all statistically significant. These findings show that the model fits the data adequately. The following alternative hypotheses proposed in the introduction were also tested against the data: 1. Learning in PBL is only influenced by phases involving collaborative learning and co-construction; 2. Learning in PBL is only influenced by self-directed study and 3. Learning in PBL is influenced by both collaborative learning as well as self-directed study, but not in a cumulative manner. Table 2 shows a comparison of the results of the indicators of goodness of fit for the different models tested.

121

122

Chapter 5

Table 1. Intercorrelations, means and standard errors of the variables (N = 218) 1 1.

2.

3.

4.

5.

Pre-test results (Prior Knowledge)

2

3

4

5

−

Total number of concepts recalled after problem analysis

.44**

−

Total number of concepts recalled after individual study

.37**

.74**

−

Total number of concepts recalled after reporting phase

.26**

.59**

.75**

−

Post-test results (Achievement)

.41**

.34**

.40**

.34**

−

2.02

9.88

15.71

12.15

6.90

2.05

4.86

6.52

5.94

3.66

Mean Standard deviation

** significant at the 0.01 level


Figure 4. Path model of the hypothesized model on relationships between different PBL phases

123

124

Chapter 5

Table 2. Comparison of results for different models tested using structural equation modelling Model\Indicators of goodness of fit

Cmin

df

Cmin/ df

P value

CFI

RMSEA

Hypothesized model:

7.84

5

1.57

.17

.97

.05

70.18

3

23.39

.00

.19

.32

26.33

1

26.33

.00

.41

.34

103.12

6

17.19

.00

.12

.27

Learning in PBL is influenced by both collaborative learning and self-directed learning in a cumulative manner

Alternative model 1: Learning in PBL influenced only by collaborative learning

Alternative model 2: Learning in PBL influenced only by selfdirected learning

Alternative model 3: Learning in PBL is influenced by both collaborative and selfdirected but not in a cumulative manner


Results from Table 2 clearly show that our hypothesized model best fits the data obtained. This model shows that the impact of students’ prior knowledge on the concepts students were able to recall after the problem analysis phase is equal to .44. Students’ prior knowledge also influenced their achievement directly (.34). This finding is in line with a previous study by Gijselaers and Schmidt (1990) who found that amount of prior knowledge influenced students’ achievement by .37. The number of relevant concepts recalled at the end of the problem analysis phase strongly influenced the number recalled at the end of the selfdirected learning phase, which similarly influenced the number of concepts recalled at the end of the reporting phase. Finally being able to recall more relevant concepts at the end of the reporting phase influenced students’ learning achievement significantly (.26). Results from the alternative hypotheses tested as tabulated in Table 2 also show that learning in PBL cannot be described only in terms of collaborative learning and teamwork, nor only in terms of self-directed learning. The lack of fit of the models with the data also demonstrates the importance of the sequential influence of learning from one phase to the next. This is important evidence showing that the three phases of PBL: problem analysis, self-directed learning, and reporting phase, play specific roles in influencing students’ learning achievements. Since our model enables us to predict student achievement very well, this also indicates the validity of our methodology as a means of keeping track of students’ learning in the course of the learning process. Thus our method appears to be a useful and efficient way to overcome the typical difficulties faced in data collection of large samples for naturalistic studies. One limitation of this present study is that the units of analysis focused on individual scientific concepts students were able to associate with the topic-at-hand and to recall at the end of each PBL phase, without connecting propositions demonstrating how the different concepts were linked. This then limits the deductions we can draw about the depth and accuracies of students’ understanding of the different concepts. However despite this shortcoming, our findings from

125

126

Chapter 5

our model fit also show that this method does provide valid insight into students’ learning.

Conclusion In conclusion, we have shown that all the phases in the PBL process are necessary to understand how students learn in PBL. The learning in each phase of the PBL process is shown to be strongly influenced by the earlier phase, thus providing support for the PBL cycle of initial problem analysis, followed by self-directed learning, and a subsequent reporting phase as described by various authors. Alternative hypotheses where students’ achievement is predicted only by collaborative learning or selfdirected learning were shown to be insufficient to explain the data observed. Secondly, we have identified two distinct phases of initial terminology articulation and a later terminology repetition in the PBL process, thus providing further insight into the process of learning in PBL through a semi-naturalistic approach, instead of depending on student self-report. Lastly, we have described a useful and efficient method to keep track of students’ learning throughout the PBL process.


Chapter 6: Summary and conclusions The research presented in this thesis was focused on the learning process in problem-based learning (PBL). I developed and applied different methods for use in a naturalistic PBL setting to gain insight into what and how students learn in all the phases of the PBL cycle, as well as to identify the relationships between the learning activities of students (what they say, do, and remember) with their learning outcomes. All research was carried out in an authentic though somewhat unique PBL setting in a polytechnic in Singapore. The educational context here is special in that the problem analysis, self-directed learning and reporting phases of PBL all occur within one day. Students thus work on one problem per day, in a class of 25 where they are divided into teams of five. A brief description of the day’s process is described below: 

Problem analysis phase (approximately 1 hour): The facilitator presents the problem for the day. Students work in teams of five to identify their prior knowledge and learning issues.



Self-directed learning (SDL) period (approximately 4 hours): Students do individual study or work with their teams on worksheets and other resources provided. They are also able to access other resources from the internet or textbooks. Time is spent helping one another within the team when necessary. Students meet with their facilitator for about 20 minutes in between this period to share their learning progress and strategy of understanding the problem.



Reporting phase (approximately 2 hours): Each team presents their consolidated findings and response to the problem, defending and elaborating based on questions raised by peers and the facilitator. The team presentation is usually in the form of powerpoint slides and the facilitator would also clarify key ideas if necessary.

127

128

Chapter 6

Although the PBL process in this institution was adapted to suit the learning needs of the students and is completed within one day, it remains classified as PBL based on the ‘six core characteristics of PBL’ described by Barrows (1996). The characteristics include studentcentred learning whereby students work in small groups under the guidance of a tutor who facilitates the learning process. Problems are used as the stimulus for students’ learning with no opportunities to prepare beforehand. Furthermore, facilitators do not provide direct instruction. Instead, students construct their own understanding through self-directed learning (Hmelo-Silver, 2004). An additional feature of the PBL approach in this context is that instead of only individual study during the self-directed learning phase, peer consultation and collaboration also takes place during this time. Moreover, all students have a personal laptop that can be connected to the internet. First-year students generally rely mainly on internet resources for their research, and tend to remain in class instead of going to the library for their self-directed study. In this context, it was possible to record and hence observe students’ learning activities for entire PBL cycles, even during the periods of self-directed study, so as to better understand the learning process in PBL. In this chapter, I will summarize my findings in regards to the research questions identified in Chapter 1, as well as discuss the limitations of the research, implications to educational practice and suggestions for further research.

Main conclusions In this section I will give an overview of the findings from the studies in Chapters 2 to 5, as well as relate them to the research questions identified in the Introduction. C H A PT ER 2 As mentioned previously, PBL is seen to be made up of three phases: problem analysis, self-directed learning (SDL) and reporting (Barrows, 1988; Hmelo-Silver, 2004; Schmidt, 1983). The learning environment which PBL seeks to create is one in which students learn in the context


of meaningful problems, actively constructing mental models in the process, co-constructing ideas with peers in a collaborative fashion and developing self-directed learning skills in the process (Norman & Schmidt, 1992). Thus, according to Dolmans, De Grave, Wolfhagen and Van der Vleuten (2005), PBL brings together four fairly new insights into learning, namely that learning can be considered a constructive, self-directed, collaborative, and contextual activity. The main aim of the study described in Chapter 2 therefore was to investigate the extent to which PBL indeed stimulates students towards constructive, selfdirected and collaborative learning, since theories of learning assume that these learning activities are essential in the learning process. To this end, the verbal interactions of one PBL group of five students throughout an entire PBL cycle were audio recorded and the verbatim transcript was then obtained. The resulting protocols were coded and analyzed based on an adaptation of the utterance and episodic coding scheme of learning-oriented interactions devised by Van Boxtel, Van der Linden and Kanselaar (2000). The methodology chosen here was an intensive and in-depth case study of one team with five students. The verbal interactions amounted to 7.5 hours of recording with a total of 1075 utterances and 349 episodes. Furthermore, by describing and analyzing the verbal interactions of students in the PBL cycle, we aimed to increase our understanding of what students do in all the relevant phases of PBL. Table 1 shows how the different episodes were classified as being indicative of the constructive, self-directed and collaborative learning processes. For constructive processes, only explanation episodes were included here. Episodes classified as ‘collaborative’ in nature involve input from more than one team members, with the exception of ‘unanswered question’. This was considered ‘collaborative’ because the question was directed at the teammates although no immediate response was observed perhaps due to no one knowing the answer at the moment. Finally, episodes that involved planning and evaluation, indicative of monitoring and making a judgment were classified under ‘self-regulated or self-directed’ learning processes.

129

130

Chapter 6

Table 1. Episodic coding scheme showing indicators of constructive, self-directed and collaborative learning episodes Constructive Explanation episode

initiated by a question and followed by an answer or series of answers that involve reasoning and elaboration by the student(s) answering

Collaborative Basic question and answer episode

initiated by a question and followed by one factual answer or a ‘yes’ or ‘no’ response

Sharing of information episode

initiated by the presentation of information related to the problem to one or more team members and followed by team members agreeing without further input of ideas

Co-construction episode

initiated by the presentation of information related to the problem to one or more team members and followed by further input of similar ideas and information from at least one other team member

Conflict episode (not elaborated)

initiated by the presentation of information related to the problem to one or more team members and followed by opposing views from at least one other teammate with no justification given

Conflict episode (elaborated)

initiated by the presentation of information related to the problem to one or more team members and followed by counter arguments or opposing views from at least one other teammate, with justification given

Unanswered question

a question or series of questions with no response given

Self-directed Planning episode

topic of discussion regarding a strategy to work on the problem

Evaluation of understanding episode

topic of discussion involving a judgement about the individual’s or someone else’s knowledge and/or understanding of the problem or concept

Evaluation of resources

topic of discussion involving a judgement about the quality of a resource


topic of discussion involving organizing the duties of each team member in regards to the specific area to research on or to prepare a powerpoint slide on

Other episodes

miscellaneous episodes that did not fit in the above categories


Since the problem analysis phase was to enable students to identify learning issues in a collaborative setting and to build on one another’s prior knowledge, I expected high levels of sharing of information and coconstruction episodes. While I did observe high levels of sharing of information during this phase, I found that co-construction episodes where ideas or concepts raised were built upon by others occurred less frequently than expected. In the case of the SDL phase, I expected relatively high occurrences of planning and monitoring of task progress episodes during the selfdirected learning phases as these were the periods for students to research and discuss as a team. The results however showed that planning occurred less frequently than expected during these phases, although monitoring of progress took place very often especially during the later part of the SDL phase just before the reporting phase. As each team is required to present a final response to the problem during the reporting phase, I postulated that there would be high numbers of explanation and co-construction episodes during the self-directed phases since students were likely to co-teach one another to ensure the whole team is able to understand what they were going to present. My results did show higher numbers of explanation episodes for this phase although there were more basic question and answer episodes compared to co-construction episodes. Finally, as expected, the reporting phase consisted of significant numbers of sharing-information episodes, since this period was for students to present their findings. However although I also predicted that there would be relatively high numbers of explanations and elaborated conflict here since the different teams were likely to challenge one another’s ideas and responses, this was not observed in the data obtained. In general, there were lower levels of individual constructive learning processes (explanations) and co-constructions than expected for the different PBL phases. This suggests that students do not spontaneously make connections between new concepts and their prior knowledge, and when left to discuss on their own, spend more time sharing information and ideas with less elaboration occurring than

131

132

Chapter 6

suggested by the existing literature. In the case of the self-directed learning, while high frequencies of monitoring of task progress episodes were found especially in the later part of the SDL phase, there was relatively low numbers of planning episodes. Although the SDL phase in this context is different from traditional SDL done outside of classroom time, the learners were selfdirected in that they need not remain in the classroom during this time. They could also choose the resources and reading materials online or from the library for their research and individual study. For this group of students, they chose to do their research from the internet and to remain in the classroom. One might argue that self-regulated learning processes would mostly take place internally and need not necessarily be verbalized or observed. On the other hand, since students were supposed to come up with a group response for the reporting phase, it is reasonable to expect more planning episodes especially during the initial portion of the SDL period. However there were hardly any planning episodes here. It appears that students were more task-oriented and focused on delegating duties towards the later part of the day. One possible reason for this could be the time constraints involved in this ‘one-day-one-problem’ PBL system. Since students only have about 4 hours of SDL time, they may prefer to spend more time doing individual research or preparation of the powerpoint slides, and view working on a team plan and strategy as less urgent or important. In conclusion, out of the three types of activities, collaborative episodes were the most prevalent – 53.3% while the constructive episodes were the lowest- 15.7%. For the collaborative episodes, the largest proportion was of basic questions and answers, which did not involve elaboration or explanation. Co-construction episodes (8.9%) appeared also to a lesser extent than sharing of information episodes (14.3%). These data indicate that the elaboration on, reasoning about, and making connections with knowledge occurred much less frequently and readily than stating and/or sharing facts. Of course, while elaboration and reasoning are higher-order learning processes and therefore important, sharing of information and stating of facts are also important activities necessary for learners. The percentage of self-


directed episodes here was rather high − 27.2%. About one-third of the planning episodes were initiated by the tutor. This means that in those episodes, the students were responding to the tutor’s questions on what their approach or plan was. This shows that the modelling of selfdirected processes is possible and could be helpful for less mature students in learning how to plan and set goals and objectives for their learning. The largest proportion of the self-directed processes (12.0%) was related to the monitoring of task progress. This indicates that students were strongly driven to be ready for the reporting phase and thus spent a lot of time checking on one another’s research area and the powerpoint slides they were preparing. Finally then, to answer the question on the extent to which constructive, collaborative, and self-directed activities can be observed in the PBL tutorial, I conclude that all three activities can be observed in the PBL cycle under study, albeit to different extents, with 53.3% of episodes being collaborative, 27.2% self-directed and 15.7% constructive. CHAPTER 3 As mentioned in the preceding chapters, most studies on the learning activities of students in the PBL phases focus on either the problem analysis or reporting phase. One of the key features of the studies in this thesis is that I examined the entire PBL process, including self-directed study phases, in an attempt to uncover the learning-oriented activities involved. While the results of Chapter 2 demonstrated that PBL stimulates constructive, self-directed and collaborative learning processes, no relationships between the content of their interactions with subsequent learning were reported. In addition, due to the dataand time-intensive nature of the methodology involved, the sample size used in the study was limited, thus making statistical analysis difficult. For the study in Chapter 3, I made use of voice and computer screen recordings of the entire PBL cycle for nine students collaborating in two groups, in order to identify relationships between the relevant scientific concepts articulated or studied individually with students’ learning outcomes, and to explore how students acquired different concepts over the different PBL phases. Thus the units of analysis were

133

134

Chapter 6

the scientific concepts or terminologies that the students articulated and studied while gaining insight in the problem at hand. Concepts such as ‘DNA’, ‘alleles’ or ‘meiosis’ (these are genetics-related concepts since the problem was one on genetics) can be considered micro-theories (Murphy & Medin, 1985) that students use in the course of trying to learn about genetics. The concepts articulated and studied from online resources were counted for each student for each learning phase (i.e. problem analysis phase, SDL period and reporting phase). The total frequency of concepts refers to the total number of relevant concepts verbalized or studied, including those that were repeated in one session. On the other hand, the total number of different concepts did not include those that were repeated during the same session. Newly emerged concepts were those that were not previously mentioned by the individual in any prior learning phase of the day. I hypothesized that the frequency with which these concepts were used by students while discussing the problem and studying subject-matter, could be considered an indicator of the learning-oriented activities going on and would determine subsequent achievement. By recording the group discussions and individual internet log files of two groups of students (nine students in total) throughout the day, I made a first attempt to chart the entire learning process of students in a natural environment. Students’ learning achievement was measured using two tests: (1) a free recall essay test and (2) a concept recognition test. The free recall essay test was administered first and consisted of the following instructions: “Write in detail your answers to the following questions. Include all the ideas that you think are relevant. 1. Why do children tend to resemble their parents in terms of their physical traits? 2. What do you know about the structure and function of a gene?” Answers for the test were analyzed for accuracy using the “idea unit” as the entity for scoring (Meyer, 1985; Schiefele & Krapp, 1996). The answers were segmented into idea units. An idea unit was defined as a statement which ends with a comma, period, or ‘and’. Each idea unit was given either a score of 2, 1 or 0. A score of 2 was given for a completely


correct idea unit, 0 for a completely incorrect idea unit and 1 when the idea unit was only partially correct. The concept recognition test was a simplification of the concept mapping technique (Novak, 1998). This test consisted of a list of 34 concepts that are more or less closely related to the central topic of heredity. This set of 34 concepts cover the domain as a whole and was based on an exhaustive review of relevant literature from textbooks and internet resources. Examples of such concepts include ‘gametes’ and ‘chromosomes’. A number of concepts not related to heredity (“fillers”) were interspersed in the list. Examples include ‘water’ and ‘oxygen’. Students were instructed to rate the extent to which the concepts listed were related to heredity using a scale of 1 to 5, where 1 = not at all related; 2 = a little bit related; 3 = to some extent related; 4 = quite closely related and 5 = very closely related. This concept recognition test was given to the students after they had submitted their answers for the free recall essay test so that they were not able to use the concepts listed to help them answer the essay question. For the concept recognition test, two colleagues with expertise in the field of molecular biology were asked to identify the most appropriate answers independently. Inter-rater agreement was 83.8%. Where there were differences in rating, a third opinion from a similar expert was sought. Student rating of each concept scored 2 points if it was the same as the expert answer, 1 point if it differed by ±1, and 0 for any other answer. Together, the concept recognition and free recall tests measured the breadth as well as the depth of the students’ scientific vocabulary. Breadth refers to the extent of the knowledge distribution while depth refers to the ability to describe the relationships between the concepts (Alao & Guthrie, 1999). In this study, the former is measured through the recognition of relevant concepts while the latter is assessed through the free recall essay tests. It was assumed that the both the breadth and validity of the learners’ scientific concepts give an indication of their understanding of the theory (Alao & Guthrie, 1999; Tsai & Huang, 2002). Results showed two distinct phases in the PBL process − an initial

135

136

Chapter 6

terminology articulation phase, consisting mainly of the problem analysis phase and the initial part of the SDL period, where students are exposed to and articulate new concepts; and a later terminology repetition phase, consisting mainly of the second part of the SDL period where relevant concepts are repeated and elaborated upon. It was also found that the most extensive on-task activity occurs half-way in the process, during the second part of the SDL period; in this phase most verbal interaction and online research were taking place. As the sample size for the study in Chapter 3 was again small (9 students), I was only able to use simple descriptive correlation analysis to find the relationships between students’ learning achievement (posttest scores) and the number of relevant concepts articulated or studied during the different learning phases. The results show that it is both the total number of concepts (i.e. including repetition of concepts) as well as the different numbers of concepts (i.e. excluding repetition of concepts within the same PBL phase) articulated throughout the different learning phases, which correlate most strongly with students’ learning. Between the two, breadth of concepts (different concepts) verbalized (r = .80, p < .01) has a higher correlation with student learning (as measured by the free recall essay test) compared to the repetition of similar ideas (r = .59, p < .05). Thus the findings show that both the total repetition of concepts discussed throughout the PBL process as well as the breadth of terminologies articulated play significant roles in the learning process. This is somewhat in contrast with the findings of Moust et al. (Moust, Schmidt, De Volder, Belien, & De Grave, 1987), who found that the amount of talk by students as such did not predict achievement. In the case of students’ self-directed study, the findings indicate that being exposed to a greater number of different relevant concepts as well as having increased exposure to the same concepts (total concepts) correlated with students’ learning outcomes (as measured by free recall essay test) (r = .78, p < .01 and r = .82, p < .01 respectively). These results show that both the amount and breadth of self-directed research undertaken from the study of online resources are highly indicative of students’ learning from the PBL process. The strengths of the


correlations here also suggest that individual study plays a greater role in students’ learning achievements compared to verbal interactions with peers during the SDL phases (total concepts articulated: r = .27 and different concepts articulated: r = .47, p > .05). Thus, despite the small sample size of this study, statistically significant effects on learning were found. Given the fact that the power of the statistical tests used was extremely small due to sample size, the significant results are the more telling. They strongly suggest that the findings are meaningful and likely to be valid. Chapter 3 therefore answers the research question on whether there are different learning phases in the PBL process, as well as how the learning activities in the different PBL phases are related to students’ learning outcomes. However the small sample size involved in the study was a limitation. CHAPTER 4 The study in Chapter 4 attempted to overcome the limitations of small sample size faced in the previous study. With a larger sample size of 35 students and using the same methodology as in Chapter 3, I was able to investigate how students’ verbal interactions during different phases of the problem-based learning cycle, self-directed study, and achievement were related using the statistical approach of structural equation modelling. This allowed information regarding possible causal relationships between what students say and do with their learning achievements to be obtained. The objective of this study, therefore, was to identify the relationships between the actual learning activities of students and their learning outcomes using data from observations of students’ learning process. Such an approach is new for this field because the model was tested on data derived from actual observations in a natural classroom setting rather than on student self-report. I also sought to clarify the relations between the different learning processes in the PBL cycle: the relevance of the verbal contributions during problem analysis phase, of verbal exchanges during self-directed

137

138

Chapter 6

learning, of individual study during self-directed learning, of verbal contributions during the reporting phase, and achievement. My hypothesis was that small-group collaboration is necessary in PBL, that it does influence individual study, and that it eventually influences achievement. Figure 1 summarizes the hypothesized relations in terms of a causal model. As can be deduced from Figure 1, I expected the adequacy of verbal contributions in discussions as well as individual study to influence students’ achievement indirectly, through the adequacy of the students’ contributions in the reporting phase. In this study, my assumption was that measuring the number of relevant concepts articulated and studied by the students in each learning phase gives an indication of the adequacy of these students’ learning.

Figure 1. Theoretical model on the learning processes involved in PBL


The resulting model (shown in Figure 2), based on existing theory with regards to PBL (Dolmans & Schmidt, 2006; Schmidt & Moust, 2000; Van den Hurk, Dolmans, Wolfhagen, & Van der Vleuten, 2001) was found to fit the data quite well. The first relationship is the strong influence of the contents of students’ verbal contributions during problem analysis phase on the verbal contributions during the selfdirected learning phase. Verbal contributions during problem analysis are likely to reflect students’ prior knowledge. However, the extent of students’ verbal contribution during problem analysis would also be influenced by students’ efforts to activate what they already know about the problem-at-hand (Schmidt, 1983). It is known from other studies that prior knowledge, once activated, influences subsequent learning (see for a review: Dochy, Segers, & Buehl, 1999). This was what was found in experimental studies of PBL as well (De Grave, Schmidt, & Boshuizen, 2001; Schmidt, De Volder, De Grave, Moust, & Patel, 1989). These studies showed that groups that activated prior knowledge through problem analysis learned more from a subsequently presented problem-relevant text than those who did not. My original hypothesis was that the extent to which students contribute in discussion during the problem analysis phase would mainly have an impact on the next learning phase − that of self-directed study. However the findings showed that students’ verbal contribution during problem analysis influenced not only the SDL phase but also the reporting phase. Hence active engagement in knowledge building and sharing during the problem analysis phase also influences that during the reporting phase. Students who have raised more ideas during the initial knowledge sharing process would perhaps also be more motivated to share and clarify more during the reporting phase, when their knowledge networks are more established. Thus the first phase of the PBL cycle is clearly a very important one since the extent of a student’s learning in the PBL process is largely determined at this point.

139

140

Chapter 6

Figure 2. Path model of the learning-oriented activities affecting student achievement

Contrary to my original hypothesis though, is that there is no direct relationship between verbal contributions during problem analysis phase and the amount of individual study, nor any direct relationship between individual study and verbal contributions during the selfdirected learning phase. One possible reason could be due to students who are reserved or are not comfortable to voice out their ideas readily, but do put in significant amounts of individual study during the learning process. Since every student is generally required to share some of their findings and response to the problem during the reporting phase, those who have done more individual study would also be able to contribute more during this phase. Another possible reason for the results that deviate from our original hypothesis could be due to students’ perceptions of what makes learning effective in PBL. Although they are not new to PBL, having been in a PBL environment for six weeks of a semester, all of them have come from traditional ‘teacher-centred’ learning environments. Hence there could be significant numbers of students who do not see the benefits of or are not used to the idea of talking and sharing what they have learned spontaneously during self-


directed learning time. As these are only possible speculations, further studies are needed to investigate the possible effects of students’ perceptions of effective learning in PBL on their learning activities. The next relationship of interest is a relatively strong influence of students’ verbal contribution during self-directed learning on the extent of their verbal contribution during the reporting phase. One unique feature of the self-directed learning that occurs in this education context is the collaboration students undertake during this time. While studying individually, students consult each other and interact about the topics at hand. Studies on the self-directed study phase generally investigate only issues related to the individual learning such as time spent on individual study, extent of literature search and extent of studying of literature (Van den Hurk et al., 2001; Van den Hurk, Wolfhagen, Dolmans, & Van der Vleuten, 1999). However in this context, students also discuss and share ideas during self-directed learning, and this collaboration also influences their achievement indirectly through the verbal contributions in the reporting phase. Another relatively strong relationship is the impact of the extent of individual study on students’ verbal contribution during the reporting phase. A study by Van den Hurk et al. (2001) showed that the quality of student individual study (based on whether learning was done in an ‘explanation-oriented way’) influenced the depth of students’ reporting. This seems similar to our findings where individual study influenced verbal contributions during the reporting phase. However a surprising finding in our context is that there was no direct path from individual study to achievement. Individual study influences achievement indirectly, through verbal reporting. Similarly there is no direct relationship between students’ prior knowledge as indicated from their verbal contribution during problem analysis phase to achievement. These findings underline the importance of actively constructing through verbalization for learning to emerge. My model thus clearly shows the importance of verbalizations throughout the PBL process. In trying to make sense out of the problem, students produce explanations, initially based on prior knowledge, but in later phases also based on what was learned from fellow students and

141

142

Chapter 6

from the materials studied on the internet. In the study in Chapter 2, I have demonstrated that more than 50% of the learning-oriented verbal exchanges in the small group were collaborative in nature (i.e. involving co-construction, sharing of information etc). Visschers-pleijers et al. similarly found that more than 60% of the verbal interaction during the reporting phase of a PBL tutorial consisted of ‘cumulative reasoning’ processes. Although no relations to achievement were reported in the studies cited above, Rivard & Straw (2000) showed that giving opportunities for collaborative exploratory talk significantly improved students’ post-test scores as compared to the groups which did not have peer interaction. Chi, De Leeuw, Chiu and Lavancher (1994) have also demonstrated that eliciting self-explanations by students results in increased learning. They found that the more students self-explained, the deeper the understanding of the topic that was achieved. Thus the significant role played by verbalizations of ideas in explaining student achievement in my study may be attributed to the PBL approach, which provides opportunities for learning-oriented discussion, and encourages students to verbally interact. In conclusion, in regards to the question of the relationship between the contents of the learning activities of students in PBL and their learning outcomes, my findings show that both the contents of selfdirected learning and verbal interaction play a role in predicting students’ learning outcomes. While the findings from this study suggest that collaborative learning is to some extent dominant over individual study in predicting students’ performance in this PBL context, the findings from Chapter 3 also demonstrate the importance of individual study in relation to students’ achievement. My findings in this chapter also suggest that learning in each phase of the PBL cycle is a precondition for subsequent learning, thus providing support for the PBL cycle of initial problem analysis, followed by self-directed learning, and a subsequent reporting phase as described by various authors (Barrows, 1988; Hmelo-Silver, 2004). C H A PT ER 5 The study in chapter 4 investigated to some extent how the learning in each PBL phase influences that in the next and to what degree using a


similar methodology as in Chapter 3, but with a larger sample size of 35 students. Although the methodology of concept counting used in these two studies was able to provide an estimate of the quantity of students’ learning, it could not provide information regarding the quality or depth of their understanding. However, the results showed that the data obtained fit the model well and that the model was also relatively accurate in predicting students’ achievement. Another limitation of the methodology as described above is that it is possible that students may not understand the concepts they are verbalizing, or could be simply scanning the computer screens without seriously studying the material before them. Moreover, recording and transcribing all the learning activities throughout a PBL cycle was very time-consuming and severely limited the sample size that could be utilized for each study. In the study described in Chapter 5, I therefore sought to devise a more efficient and effective method to track students’ learning as it unfolds in the course of the PBL process. For this purpose, a concept recall exercise to capture and quantify students’ learning during the PBL process so that causal relationships in the PBL process can be identified through path analysis was used. Students were asked to spontaneously recall and list the concepts they considered related to two keywords relevant to the problem-at-hand at the end of each PBL phase (i.e. at the end of problem analysis, self-directed learning and reporting phase). The assumption behind this procedure is that recall is probably a better – or at least more conventional – measure of what is actually learned than counting the number of times ideas were uttered or encountered. Scientific concepts relevant to understanding the problem-at-hand were used as units of analysis. Since, as Solomon, Medin, and Lynch argued, “concepts are the building blocks of thought” (1999, p. 99), I suggest that the nature and the number of concepts recalled at any moment by a learner represent the extent of his current understanding of the problem-at-hand. The underlying assumption here is that, while learning, students structure knowledge in semantic networks of related concepts connected by associational links (Collins & Quillian, 1969; Hovardas & Korfiatis, 2006). A beginner’s initial network would be

143

144

Chapter 6

largely disconnected, consisting of a few isolated concepts or ideas that are poorly connected. Therefore, if asked to retrieve relevant concepts from these cognitive structures, his memory will be limited. The more students have learned about a topic, the richer, more coherent, and more detailed this particular network would be (Glaser & Bassok, 1989). As learning progresses, more linkages and integration between new and existing ideas are constructed. Therefore, students who have learned more effectively would be able to recall more concepts and would do that more easily (Collins & Quillian, 1969; Rumelhart & Norman, 1978). Hence, measuring the number of concepts relevant to the problem-athand that the students were able to recall at the end of each learning phase gives an indication of the extent and quality of students’ learning in that particular phase. I believe that this methodology proposed and used in this study is more effective than the previous methodology described in Chapters 3 and 4. This is because the concepts listed are generated actively by the students at the end of each phase and hence likely to be indicative of their learning. Moreover, using this procedure, only a representative ‘snapshot’ of students’ concepts at the end of each learning phase needs to be taken instead of a continuous recording of the whole PBL cycle. This makes the process more efficient and allows for a much larger sample size to study causal effects. With this modified methodology in Chapter 5, I thus carried out further analyses using a larger sample size of 218 students. For this study, students’ prior knowledge was also measured by giving them an essay pre-test a week prior to the problem. The same essay question was also administered as a post-test immediately after the day’s problem to measure students’ learning achievement. Figure 3 summarizes my hypothesized relations in terms of a causal model. I hypothesized that learning in PBL is a cumulative process where the learning in each new phase builds upon knowledge acquired in a previous phase. The process is initially driven by the prior knowledge that students bring with them to the classroom and the learning in each of the PBL phases influences student achievement. This hypothesis was also tested against the following alternative hypotheses: 1. Learning in PBL is only influenced by phases involving collaborative


learning and co-construction; 2. Learning in PBL is only influenced by self-directed study and 3. Learning in PBL is influenced by both collaborative learning as well as self-directed study, but not in a sequential cumulative manner.

Figure 3. Path model of the hypothesized model on relationships between different PBL phases

In this final study, I also aimed to use the larger sample size to test the “acquisition-elaboration theory” of learning proposed from the findings in Chapter 3. Finally, through this study, I also aimed to develop and evaluate an efficient method to capture and quantify students’ learning during the PBL process so that causal relationships in the PBL process can be identified through path analysis.

145

146

Chapter 6

Using this new methodology of concept counting, I was able to test how the concepts relevant to students’ learning were distributed over the entire PBL cycle. Similar to the findings in Chapter 3, I found that the self-directed learning phase is rich both in the acquisition of new concepts as well as the reiteration and repetition of concepts previously exposed to. The reporting phase is also characterized more by repetition of concepts rather than being exposed to new ones. Thus the two distinct phases of initial terminology articulation and a later terminology repetition in the PBL process is demonstrated once again, using a different methodology and larger sample size. In regards to the causal relationships between the different PBL phases and students’ learning, my results showed that the impact of students’ prior knowledge on the concepts students were able to recall after the problem analysis phase is equal to .44. Students’ prior knowledge also influenced their achievement directly (.34). This finding is in line with a previous study by Gijselaers and Schmidt (1990) who found that the amount of prior knowledge influenced students’ achievement by .37. The number of relevant concepts recalled at the end of the problem analysis phase strongly influenced the number recalled at the end of the self-directed learning phase, which similarly influenced the number of concepts recalled at the end of the reporting phase. Finally being able to recall more relevant concepts at the end of the reporting phase influenced students’ learning achievement significantly (.26). Results from the alternative hypotheses tested also show that learning in PBL cannot be described only in terms of collaborative learning and teamwork, nor only in terms of self-directed learning. The lack of fit of the models with the data also demonstrates the importance of the sequential influence of learning from one phase to the next. In conclusion, all the phases in the PBL process seem necessary to understand how students learn in PBL. The learning in each phase of the PBL process is shown to be strongly influenced by the earlier phase, thus providing support for the PBL cycle of initial problem analysis, followed by self-directed learning, and a subsequent reporting phase as described by various authors. Furthermore, the validity of the new methodology proposed and used in this chapter as a means of keeping track of


students’ learning in the course of the learning process is supported by the results obtained, since the hypothesized model was able to predict student achievement very well. This method of concept recall thus appears to be a useful and efficient way to overcome the typical difficulties faced in data collection of large samples for naturalistic studies.

Critical reflections and directions for further studies Two key features of the studies described in this thesis are that they were process-oriented and carried out in a natural educational environment. By describing and analyzing the actual activities in all the relevant phases of the PBL process, I was able to obtain rich and authentic information about the causal effects of the different aspects of the phases of PBL. Some limitations of my research are presented below together with suggestions for further studies. First I recognize that using this particular education context where a ‘one-day-one-problem’ implementation of PBL has both advantages and disadvantages. An advantage of basing my research on this context is that I was able to analyze the entire PBL process within a day. This is a significant feature of my research as it enabled me to effectively analyze the causal effects between the different phases of PBL, including that of the self-directed learning phase using actual observations instead of student self-report. This has, to my knowledge, not yet been done by other researchers. However, using this relatively unique educational context also restricts, to some extent, the generalizability of my results. It would be interesting to investigate if the different phases of PBL influence one another similarly when the phases are separated over a few days or even a week. Using the method developed for the study described in Chapter 5, it would be possible to capture students’ learning in such a different PBL setting. Secondly, the unit of analysis in the studies of Chapters 3 and 4 was the scientific concepts students were exposed to via computer resources, or articulated in the process of discussion. As discussed in the respective chapters, this methodology assumes that exposure to (from computer screen recordings of internet study resources) or the articulation of a

147

148

Chapter 6

concept can give an estimation of the learning has taken place, at least to some extent. While this assumption appears valid based on my findings, I sought to refine it in the study in Chapter 5 by using the concepts that students were able to spontaneously recall and list at the end of each PBL phase (i.e. at the end of problem analysis, self-directed learning and reporting phase). The assumption behind this method is that the concepts recalled would reflect the learner’s mental representations of the keywords used as the stimulus (Collins & Quillian, 1969; Hovardas & Korfiatis, 2006). Although this second method has been shown to be useful and efficient to overcome the typical difficulty faced in data collection of large samples for naturalistic studies, it also has certain limitations. Since the quality of understanding involved in each concept was not determined, the deductions that can be drawn about the depth and accuracy of students’ understanding of the different concepts were limited. The relationships that exist (or not exist) between concepts recalled, and hence the cognitive structures developed by the students while learning, could not be studied directly. A possible strategy to use for further studies would be to require students to make concept maps that may more accurately represent students’ knowledge structures. Finally this series of studies focused on observational research on the process of PBL. Further controlled studies designed to systematically test variables potentially influencing the learning process in PBL should be carried out.

Implications of the findings At the theoretical level, these studies have provided insight into the influences of the different phases of PBL on students’ learning, as well as described and analyzed what actually happens during the PBL process of problem analysis, self-directed study and reporting. Some further implications of the findings for educational practice are suggested below. My studies have demonstrated the importance of verbalizations throughout the PBL process. In trying to make sense out of the problem, students produce explanations, initially based on prior knowledge, but in later phases also based on what was learned from fellow students and from the materials studied on the internet. For the study described in


Chapter 1, more than 50% of the learning-oriented verbal exchanges in the small group were collaborative in nature (i.e. involving coconstruction, sharing of information etc). Visschers-pleijers et al. similarly found that more than 60% of the verbal interaction during the reporting phase of a PBL tutorial consisted of ‘cumulative reasoning’ processes. The study in Chapter 3 also showed that the extent of verbalization during the reporting phase directly influences students’ learning results. At the practical level, suggestions that are relevant to educational practice can be derived from these observations. Facilitators need to encourage students to verbalize their ideas, especially during the reporting phase. In a typical tutorial class, it is likely that there are some students who are more vocal and tend to elaborate more. Facilitators should help to ensure that opportunities are given to all students to present their ideas to an audience and to elaborate upon or defend their positions. My findings suggest that in order for what is learned during selfstudy to lead to eventual achievement, verbalization of ideas within the group is essential. This observation is also important to the facilitator in the classroom. Students need to recognize the importance of collaborative learning. For students who are unaccustomed to working in a PBL or collaborative learning environment, guidance on how to work in teams, how to communicate effectively and constructively are all important issues the facilitator should consider. On a similar note, observations from the studies in Chapter 2 show that students tend to pick up cues from the tutor during the limited interaction time. Thus, one way to encourage students to formulate more critical questions, and seek clearer explanations and reasoning during their group interaction is for tutors to actively model such questions when interacting with students. Since it was observed that students tend to avoid conflicting ideas and proper resolution of differences in opinions, it is also recommended that students be guided to understand that collaborative learning does not only consist of cooperative teamwork, but also requires mutual questioning and challenging in order to stimulate effective learning.

149

150

Chapter 6

Thus in examining the relationships between the various phases of the PBL cycle and the effects of their interactions on students’ learning, it is possible to refine and guide the development and the implementation of PBL in everyday practice.

References

References Alao, S., & Guthrie, J. T. (1999). Predicting Conceptual Understanding with Cognitive and Motivational Variables. The Journal of Educational Research, 92(4), 243-254. Arbuckle, J. L. (2006). Amos 7.0 User's Guide. Chicago: SPSS. Barrows, H. S. (1986). A taxonomy of problem-based learning methods. Medical Education, 20, 481-486. Barrows, H. S. (1988). The Tutorial Process. Springfield Illinois: Southern Illinois University School of Medicine. Barrows, H. S. (1996). Problem-based learning in medicine and beyond: a brief overview. In L. Wilkerson & W. H. Gijselaers (Eds.), New directions for teaching and learning (Vol. 68, pp. 3-11). San Francisco: Jossey-Bass Publishers. Bianchini, J. A. (1997). Where knowledge construction, equity, and context intersect: Student learning of science in small groups. Journal of Research in Science Teaching, 34(10), 1039-1065. Bonwell, C. C., & Eison, J. A. (1991). Creating Excitement in the Classroom Washington, DC: George Washington University, School of Education and Human Development. Bransford, J. D., & Johnson, M. K. (1972). Contextual prerequisites for understanding: Some investigations of comprehension and recall. Journal of Verbal Learning and Verbal Behaviour, 11, 717-726 Browne, M. W., & Cudeck, R. (1993). Alternative ways of assessing model fit. In K. A. Bollen & J. S. Long (Eds.), Testing Structural Equation Models (pp. 136-162). Newbury Park, CA: Sage. Boud, D., & Feletti, G. (1991). The Challenge of Problem Based Learning. New York: St. Martin’s Press. Browne, M. W., & Cudeck, R. (1993). Alternative ways of assessing model fit. In K. A. Bollen & J. S. Long (Eds.), Testing Structural Equation Models (pp. 136-162). Newbury Park, CA: Sage.

151

152

References

Bridges, E., & Hallinger, P. (1992). Problem Based Learning For Administrators: University of Oregon. Capon, N., & Kuhn, D. (2004). What's so good about problembased learning? Cognition and Instruction, 22(1), 61-79. Chi, M. T. H., Deleeuw, N., Chiu, M. H., & Lavancher, C. (1994). Eliciting Self-Explanations Improves Understanding. Cognitive Science, 18(3), 439-477. Chi, M. T. H., Bassok, M., Lewis, M. W., Reimann, P., & Glaser, R. (1989). Self-Explanations - How Students Study and Use Examples in Learning to Solve Problems. Cognitive Science, 13(2), 145-182. Cobb, P. (1994). Where is the mind? Constructivist and sociocultural perspectives on mathematical development. Educational Researcher, 23, 13-20. Cobb, P., & Yackel, E. (1996). Constructivist, emergent, and sociocultural perspectives in the context of developmental research Educational Psychologist, 31(3 & 4), 175-190. Collins, A., Brown, J. S., & Newman, S. E. (1989). Cognitive apprenticeship: Teaching the crafts of reading, writing, and mathematics. In L. B. Resnick (Ed.), Knowing, learning and instruction: Essays in honor of Robert Glaser. Hillsdale NJ: Lawrence Erlbaum Associates. Collins, A. M., & Quillian, M. R. (1969). Retrieval Time from Semantic Memory. Journal of Verbal Learning and Verbal Behavior, 8, 240-247. De Grave, W. S., Boshuizen, H. P. A., & Schmidt, H. G. (1996). Problem based learning: Cognitive and metacognitive processes during problem analysis. Instructional Science, 24(5), 321-341. De Grave, W. S., Schmidt, H. G., & Boshuizen, H. P. A. (2001). Effects of problem-based discussion on studying a subsequent text: A randomized trial among first year medical students. Instructional Science, 29(1), 33-44.

References

Derry, S. J. (1996). Cognitive schema theory in the constructivist debate. Educational Psychologist, 31(3 & 4), 163-174. Dochy, F., Segers, M., & Buehl, M. M. (1999). The relation between assessment practices and outcomes of studies: The case of research on prior knowledge. Review of Educational Research., 69(2), 42. Dochy, F., Segers, M., Van den Bossche, P., & Gijbels, D. (2003). Effects of problem-based learning: a meta-analysis. Learning and Instruction, 13(5), 533-568. Dolmans, D. H. J. M., & Schmidt, H. G. (1994). What drives the student in problem-based learning. Medical Education, 28(5), 372-380. Dolmans, D. H. J. M., & Schmidt, H. G. (2006). What do we know about cognitive and motivational effects of small group tutorials in problem-based learning? Advances in Health Sciences Education, 11(4), 321-336. Dolmans, D. H. J. M., Schmidt, H. G., & Gijselaers, W. H. (1995). The relationship between student-generated learning issues and selfstudy in problem-based learning. Instructional Science, 22(4), 251-267. Dolmans, D. H. J. M., De Grave, W. S., Wolfhagen, I. H., & Van der Vleuten, C. P. M. (2005). Problem-based learning: future challenges for educational practice and research. Medical Education, 39(7), 732-741. Driver, R., Asoko, H., Leach, J., Mortimer, E., & Scott, P. (1994). Constructing scientific knowledge in the classroom. Educational Researcher, 23(5-12). Erickson, G. L., & MacKinnon, A. M. (1991). Seeing classrooms in new ways: On becoming a science teacher. In D. A. Schön (Ed.), The reflective turn: Case studies in and on educational practice (pp. 15-36). New York: Columbia University Teachers College. Fetsco, T., & McClure, J. (2005). Educational psychology: an integrated approach to classroom decisions. New York: Pearson Education, Inc.

153

154

References

Fosnot, C. T. (1993). Rethinking science education: A defense of Piagetian constructivism. Journal of Research in Science Teaching, 30, 1189-1201. Gallagher, S. A., Stepien, W. J., & Rosenthal, H. (1992). The effects of problem-based learning on problem solving. Gifted Child Quarterly, 36(195-200). Geerligs, T. (1995). Students' thoughts during problem-based small-group discussions. Instructional Science, 22, 269-278. Gijselaers, W. H. (1996). Connecting Problem-based Practices with Educational Theory. New Directions for Teaching and Learning, 68, 1321. Gijselaers, W. H., & Schmidt, H. G. (1990). Towards a causal model of student learning within the context of a problem-based curriculum. In Z. Norman, H. G. Schmidt & E. Ezzat (Eds.), Innovation in Medical Education. An Evaluation of its Present Status (pp. 95-113). New York: Springer Publishing Company. Glaser, R. (1991). The maturing of the relationship between the science of learning and cognition and educational practice. Learning and Instruction, 1(2), 129-144. Glaser, R., & Bassok, M. (1989). Learning theory and the study of instruction. Annual Review of Psychology, 40, 631-666. Glenn, P. J., Koschmann, T., & Conlee, M. (1999). Theory presentation and assessment in a problem-based learning group. Discourse Processes, 27(2), 119-133. Hak, T., & Maguire, P. (2000). Group process: The black box of studies on problem-based learning. Academic Medicine, 75(7), 769-772. Hmelo-Silver, C. E. (2004). Problem-based learning: What and how do students learn? Educational Psychology Review, 16(3), 235-266. Hmelo-Silver, C. E., & Barrows, H. S. (2006). Goals and Strategies of a Problem-based Learning Facilitator. The Interdisciplinary Journal of Problem-based Learning, 1(1), 21-39.

References

Hmelo-Silver, C. E., & Barrows, H. S. (2008). Facilitating Collaborative Knowledge Building. Cognition and Instruction 26, 48-94. Hovardas, T., & Korfiatis, K. J. (2006). Word Associations as a Tool for Assessing Conceptual Change in Science Education. Learning and Instruction, 16(5), 416-432. Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Structural Equation Modeling, 6(1), 1-55. John-Steiner, V., & Mahn, H. (1996). Sociocultural approaches to learning and development: A Vygotskian framework. Educational Psychologist, 31(3 & 4), 191-206. Jöreskog, K. G., & Sörbom, D. (1996). LISREL 8 User's Reference Guide. Chicago: Scientific Software International. Kintsch, W., & Van Dijk, T. A. (1978). Toward a Model of Text Comprehension and Production. Psychological Review, 85(5), 363-394. Kolodner, J. L., Camp, P. J., Crismond, D., Fasse, B., Gray, J., Holbrook, J., et al. (2003). Problem-Based Learning Meets Case-Based Reasoning in the Middle-School Science Classroom: Putting Learning by DesignTM Into Practice. Journal of the Learning Sciences, 12(4), 495-547. Koschmann, T., Glenn, P., & Conlee, M. (1997). Analyzing the Emergence of a Learning Issue in a Problem-Based Learning Meeting. Retrieved February 14, 2007, from http://www.med-ed-online.org/res 00003.htm. Lemke, J. L. (1990). Talking science: Language, learning and values. Norwood, NJ: Ablex. Lloyd-Jones, G., & Hak, T. (2004). Self-directed learning and student pragmatism. Adv Health Sci Educ Theory Pract, 9(1), 61-73. MacCallum, R. C., & Austin, J. T. (2000). Applications of Structural Equation Modeling in Psychological Research. Annual Review of Psychology, 51, 201-226.

155

156

References

Mayer, R. E. (1996). Learners as information processors: Legacies and limitations of educational psychology's second metaphor Educational Psychologist, 31(3 & 4), 151-161. Meyer, B. J. F. (1985). Prose analysis: Purposes, procedures, and problems. In B. K. Britton & J. Black (Eds.), Analyzing and understanding expository text (pp. 11-64, 269-304). Hillsdale, N.J.: Erlbaum. Michael, J. (2006). Where's the evidence that active learning works? Advances in Physiology Education, 30, 159-167. Miflin, Campbell, Price, & Miflin, B. (2000). A conceptual framework to guide the development of self-directed, lifelong learning in problem-based medical curricula. Medical Education, 34(4), 299-306. Milter, R. G., & Stinson, J. E. (Eds.). (1993). Educating Leaders for the New Competitive Environment. London: Kluwer Academic Publishers. Moust, J. H. C., Schmidt, H. G., De Volder, M. L., Belien, J., & De Grave, W. S. (1987). Effects of verbal participation in small group discussion. In J. T. E. Richardson, M. W. Eysenck & D. W. Piper (Eds.), Student Learning. Research in Education and Psychology (pp. 147-154). Great Britain: Open University Press. Murphy, G. L., & Medin, D. L. (1985). The role of theories in conceptual coherence. Psychological Review 92(3), 289-316. Norman, G. R., & Schmidt, H. G. (1992). The Psychological Basis of Problem-Based Learning - a Review of the Evidence. Academic Medicine, 67(9), 557-565. Novak, J. D. (1998). Learning, Creating, and Using Knowledge: Concept maps as facilitative tools for schools and corporations. Mahwah, N.J.: Lawrence Erlbaum Associates. Palincsar, A. S. (1998). Social Constructivist Perspectives on Teaching and Learning. Annual Review of Psychology, 49, 345-375.

References

Pea, R. D. (1993). Learning scientific concepts through material and social activities: Conversational analysis meets conceptual change. Educational Psychologist, 28, 265-277. Prince, M. (2004). Does active learning work? A review of the research. Journal of Engineering Education, 93(3), 223-231. Rivard, L. P., & Straw, S. B. (2000). The Effect of Talk and Writing on Learning Science: An Exploratory Study. Science Education 84(5), 566-593. Roth, K. J. (1990). Developing meaningful conceptual understanding in science. In B. F. Jones & L. Idol (Eds.), Dimensions of thinking and cognitive instruction. Hillsdale, NJ: Lawrence Erlbaum. Rumelhart, D. E., & Norman, D. A. (1978). Accretion, tuning, and restructuring: Three modes of learning. In J. W. Cotton & R. L. Klatzky (Eds.), Semantic Factors in Cognition (pp. 37-54). Hillsdale, NJ: Erlbaum. Schiefele, U., & Krapp, A. (1996). Topic Interest and Free Recall of Expository Text. Learning and Individual Differences, 8(2), 141-160. Schmidt, H. G. (1983). Problem-based learning: rationale and description. Medical Education, 17, 11-16. Schmidt, H. G. (1993). Foundations of problem-based learning Some explanatory notes. Medical Education, 27(5), 422-432. Schmidt, H. G., De Volder, M. L., De Grave, W. S., Moust, J. H. C., & Patel, V. L. (1989). Explanatory models in the processing of science text: The role of prior knowledge activation through small-group discussion. Journal of Educational Psychology, 81(4), 610-619. Schmidt, H. G., Loyens, S. M. M., Van Gog, T., & Paas, F. (2007). Problem-based learning is compatible with human cognitive architecture: Commentary on Kirschner, Sweller, and Clark (2006). Educational Psychologist, 42(2), 91-97. Schmidt, H. G., & Moust, J. H. C. (2000). Factors affecting smallgroup tutorial learning: a review of research. In D. H. Evensen & C. E.

157

158

References

Hmelo-Silver (Eds.), Problem-based learning: A research perspective on learning interactions (pp. 19-52). Mahwah, NJ: Lawrence Erlbaum. Slavin, R. E. (1996). Research on Cooperative Learning and Achievement: What We Know, What We Need to Know. Contemporary Educational Psychology 21, 43-69. Solomon, K. O., Medin, D. L., & Lynch, E. (1999). Concepts do more than categorize. Trends in cognitive sciences, 3(3), 99-105. Tsai, C. C., & Huang, C. M. (2002). Exploring students' cognitive structures in learning science: a review of relevant methods. Journal of Biological Education, 36(4), 163-169. Van Boxtel, C., Van der Linden, J., & Kanselaar, G. (2000). Collaborative learning tasks and the elaboration of conceptual knowledge. Learning and Instruction, 10(4), 311-330. Van den Hurk, M. M., Dolmans, D. H. J. M., Wolfhagen, I., & Van der Vleuten, C. P. M. (2001). Testing a causal model for learning in a problem-based curriculum. Advances in Health Sciences Education, 6(2), 141-149. Van den Hurk, M. M., Wolfhagen, I., Dolmans, D. H. J. M., & Van der Vleuten, C. P. M. (1999). The impact of student-generated learning issues on individual study time and academic achievement. Medical Education, 33(11), 808-814. Visschers-Pleijers, A. J. (2006). Tutorial group discussion in problem-based learning: Studies on the measurement and nature of learning-oriented student interactions. Unpublished PhD thesis, Universiteit Maastricht, Maastricht. Visschers-Pleijers, A. J., Dolmans, D. H. J. M., de Leng, B. A., Wolfhagen, I., & Van der Vleuten, C. P. M. (2006). Analysis of verbal interactions in tutorial groups: A process study. Medical Education, 40(2), 129-137. Visschers-Pleijers, A. J., Dolmans, D. H. J. M., Wolfhagen, I. H., & Van der Vleuten, C. P. (2004). Exploration of a method to analyze group

References

interactions in problem-based learning. Medical Teacher, 26(5), 471478. Vygotsky, L. S. (1978). Mind in Society: The Development of Higher Psychological Processes. Cambridge, MA: Harvard University Press. Webb, N. M. (1991). Task-related Verbal Interaction and Mathematics Learning in Small Groups. Journal for Research in Mathematics Education, 22(5), 366-389. Webb, N. M., Troper, J. D., & Fall, R. (1995). Constructive Activity and Learning in Collaborative Small-Groups. Journal of Educational Psychology, 87(3), 406-423. Yew, E. H. J., & Schmidt, H. G. (2008). How students learn in problem-based learning: A process analysis. Unpublished manuscript. Yew, E. H. J., & Schmidt, H. G. (2009). Evidence for constructive, self-regulatory, and collaborative processes in problem-based learning. Advances in Health Sciences Education. 14(2), 251-273, DOI: 10.1007/s10459-008-9105-7. Zimmerman, B. J. (1990). Self-Regulated Learning and Academic Achievement: An Overview. Educational Psychologist, 25(1), 3. Zimmerman, B. J. (2002). Becoming a self-regulated learner: An overview. Theory into Practice, 41(2), 64-70. Zimmerman, B. J., & Martinezpons, M. (1988). ConstructValidation of a Strategy Model of Student Self-Regulated Learning. Journal of Educational Psychology, 80(3), 284-290.

159

Appendices

Appendices Appendix A: Basic Science problem that students worked on for the day Code of Life I am the family face; Flesh perishes, I live on, Projecting trait and trace Through time to times anon, And leaping from place to place Over oblivion. From “Heredity” by Thomas Hardy (First published in Moments of Vision and Miscellaneous Verses, Macmillan, 1917) The idea of the gene came first. The gene is the thing that carries information about the living organism. The gene tells if one’s hair is black and eyes are blue. The gene tells if one can curl one’s tongue. The gene carries the ‘family face’ that goes ‘through time to times anon’ from mother to daughter, father to son, or the other ways across, over time. Is the gene a substance you can find in your body, or a kind of a soul-like invisible thing? Explore the concept of a gene and the role it plays in an organism. Is it possible that the gene is represented by an identifiable molecule, one that is able to carry information akin to a line of code, giving it the ability to execute highly detailed tasks? Determine the qualities such a molecule should have.

161

162

Appendices

Appendix B: Unedited version of Example 7: F: You know right for the Mendel’s first law right- I see something about chi squre test, I mean the chi square test A: Huh, how come got chi squre? F: I don’t know C: Today got equation meh? (Nb: ‘meh’ is a word used at the end of questions in ‘Singaporean English’ to add a tone of disbelief) L: Let’s not do that lah (Nb: ‘lah’ is a word often used at the end of sentences in ‘Singaporean English’ for emphasis) S: Can we not do? C: Can we not do any calculations? A: Yeah or else we have to… A few minutes later: F: I know why they got the chi-square thing already A: Why? F: To see which one is green which one is yellow… the chance of yellow...that sort of thing lor (Nb: ‘lor’ is a word often used at the end of sentences in ‘Singaporean English’ for emphasis and often carries a sense of resignation)

Appendices

Appendix C: Basic Science problem that students worked on for the day Heart matters A young adult participating in a parade fainted during a ceremony after standing through a long speech. The medical personnel who attended to him told him that fainting was due to lack of blood supply to the head and advised him to wriggle his toes when standing still for long periods of time. Examine the following issues: 1.

What could be the reason for the head receiving more blood as a result of wriggling one’s toes?

2.

What is the difference in this context, between standing still and lying down? (Do people faint unknowingly during their sleep? They could be sleeping without wriggling their toes.)

163

164

Appendices

Appendix D: Molecular cell biology problem that students worked on for the day

Made for the job Living things use the DNA molecule to store their genetic information and to pass this information to their offspring. Analyze the structure of DNA, and determine why it is suitable to assume this role.

165

Curriculum Vitae Elaine H. J. Yew was born in St. Albans, England on 24 July, 1973. She completed her primary and secondary education in Singapore. In 1996, she graduated from the National University of Singapore (NUS) with 1st Class Honours, majoring in Chemistry. Her interest in teaching, as well as her enjoyment of sharing ideas and interacting with young people led her to pursue a teaching career after obtaining a postgraduate diploma in education from the National Institute of Education, Singapore in 1997. Although the next six years of teaching both in a neighbourhood school and a premier girls’ school were extremely fulfilling, Elaine decided to take a break to delve into another area of her interest – research. She took up a scholarship with NUS (Faculty of Medicine, Biochemistry) to do her Master’s study on the effects of proteasome inhibition on neuronal cells. Realising she is still a teacher at heart, Elaine joined the School of Applied Science in Republic Polytechnic as an academic staff. It was here that she was given the opportunity to facilitate classes as well as work on her PhD study on the process of problem-based learning under the supervision of Professor Henk Schmidt, Erasmus University. She is currently Senior Manager (Faculty Development) in the School of Applied Science, Republic Polytechnic.

The Process of Problem-Based Learning

The Process of Problem-Based Learning

Suggest Documents

a problembased learning curriculum in undergraduate dental education

Students Perception of Interdisciplinary, ProblemBased ... - CEAD

Learning the limiting process

The learning process

Understanding the Learning Process - Oxford Learning Institute

The Learning Process of Scientific Imagineering ...

The Learning Process of Globalization: How ...

Learning Process in the Development of Business

Psychopedagogists' Conceptions of the Learning Process and

Learning: A Process of Enculturation into the

Assessment tasks: center of the learning process

Problem-Based Learning: A Process for the Acquisition of Learning ...

students' learning process

students' learning process

Learning Process through Knowledge

Engaging Students in the Learning Process: the learning journal

The Evaluation of the Cognitive Learning Process of ...

the improvement of the learning process of basic disciplines ... - iNEER

Undergraduates' learning experience and learning process - CiteSeerX

Bridging Digital Divides in the Learning Process

Analysis of Science Learning Process by Using Learning Module of

Supporting the Learning Process with ... - Stanford University

Improving the Teaching-Learning Process through ...

Reviewing the Learning Process through Creative