Q 2007 by The International Union of Biochemistry and Molecular Biology
BIOCHEMISTRY AND MOLECULAR BIOLOGY EDUCATION Vol. 35, No. 6, pp. 392–396, 2007
Articles You Be the Examiner! ‘‘MODEL ANSWERS’’ THAT REQUIRE CRITICAL THINKING* Received for publication, June 11, 2007, and in revised form, July 26, 2007 Peter C. Farley‡¶, Natisha Magan‡, Cynthia L. Charron‡, Nicholas I. Broomfield‡, and A. Fay Farley§ From the {Institute of Molecular Biosciences, Massey University, Palmerston North, New Zealand and §CUPELS, Massey University, Palmerston North, New Zealand
‘‘You be the examiner!’’ is an online approach to providing students with immediate, readily accessible, and nonthreatening feedback on their understanding of key biochemical concepts. The feedback aims to affirm correct understanding and, where further study appears necessary, direct the student to the relevant sections of their textbook and/or lecturer-provided study notes. Rather than providing model answers to previous examination questions, ‘‘You be the examiner’’ asks the students to evaluate typical ‘‘student’’ answers to such questions. Instead of a single ‘‘correct’’ answer, students encounter a range of answers that they must assess for accuracy and appropriateness. Keywords: Computer-assisted learning, peer-assessment, undergraduate education. Since January 2002, the phrase ‘‘model answers’’ has appeared just twice in Biochemistry and Molecular Biology Education, once in the context of reviewing multiple choice questions [1] and once in the context of selfassessment questions that require a written answer [2]. This is an interesting contrast to the frequent request for model answers made by first and sometimes secondyear undergraduate students, at least amongst those studying biochemistry at Massey University! The expectation of such model answers appears to be that they will provide the ‘‘correct’’ answer that can then be committed to memory and reproduced in the subsequent assessments. Such memorization is not the outcome desired by teaching staff who are therefore reluctant to accede to requests for model answers. At Massey University, biochemistry papers (known elsewhere as courses) are offered at all levels of undergraduate study. The paper for which the ‘‘You be the examiner!’’ material was developed is offered, at both the Palmerston North and the Albany campuses of Massey University, to students in their first year of university study who have already passed papers in chemistry and biology. The major focus of this paper is on how energy is obtained and utilized by living organisms, with particular reference to the pathways of glycolysis, the citric acid cycle, oxidative phosphorylation, and gluconeogenesis. Students taking this paper (between 139 and 195 students each year) are not only expected to focus on
understanding and explaining biochemical concepts rather than memorization, but also to direct their own learning using the textbook Biochemistry and Molecular Biology [3] to complement the taught material. The paper is taught by means of lectures (three per week) and laboratory classes (seven 3-hour classes) over a period of 12 weeks, and tutorials are offered at various times during the semester. Assessment is done by written and practical examinations. A handful of the students who took this paper are talented and keen to perform well. These students achieve high marks (the highest mark has been between 86 and 96% over the last 6 years). However, the paper is a greater challenge for many other students (the lower quartile was 46% 6 3% over the last 6 years). Lowachieving students are known to be unable to accurately assess their own performance, or that of their peers, tending to overestimate the quality of their own work [4– 6]. These students, therefore, require assistance beyond that which is available in lectures, laboratory classes, occasional tutorials, and appointments with staff. The purpose of the ‘‘You be the examiner!’’ project was to develop materials that would provide these students with the opportunity for early and regular selfassessment, and that would stimulate students to think about the material they are studying, to use the textbook and to practice thinking about what was actually asked in the written examination questions.
THE CONCEPT
* This work was supported in part by a grant from the Massey University Fund for Innovation and Excellence in Teaching. } To whom correspondence should be addressed. Tel.: þ64-6350-5515; Fax: þ64-6-350-5688. E-mail:
[email protected]. DOI 10.1002/bambed.112
Students would be provided random access to a database of typical examination questions and an associated ‘‘student’’ answer. The answers would reflect the range
392
This paper is available on line at http://www.bambed.org
393
FIG. 1. Screen shot of a typical ‘‘student’’ answer as presented within WebCT. [Color figure can be viewed in the online issue, which is available at www.interscience.wiley.com.]
of typical student answers (from ‘‘excellent’’ to ‘‘irrelevant’’) but not be an actual student’s answer. The student would be asked to use their understanding of the topic to assess the appropriateness of the answer. On the basis of their assessment, they would be provided feedback that reinforced correct understanding and/or directed them to the relevant sections of their textbook and lecturer-provided study notes where further study appeared necessary. No factually incorrect information would be given in the answers.
Implementation Massey University has adopted WebCT to provide an online component to teaching across the university and therefore the material was written for use in the WebCT environment. Dynamic HTML (JavaScript plus Cascading Style Sheets) was used to provide functionality not available from the WebCT quiz tool. The approach described here could, however, be implemented using other commercially available or in house packages. Students access the material through a WebCT content page that introduces the material and explains the role reversal they are being invited to adopt and the nature of the feedback they can expect. This page also provides links to i) a set of instructions from the ‘‘Chief Examiner’’ outlining the criteria on which they should mark the ‘‘student’’ answers, ii) the metabolic pathways diagrams that are provided with the written examinations for this paper, and iii) a list of topics. Students select a topic and are provided with either a single answer to mark or two different answers to compare (the question being answered
and the total marks for the question are also displayed). In the WebCT environment, this was achieved using a hidden HTML page containing a JavaScript that randomly selected ‘‘student’’ answers from a pool of questions. The topics cover the major divisions of the lecture course and provide a way for students to access a relevant subset of the ‘‘You be the examiner!’’ material from early in the semester. One topic covers material from the laboratory course. A total of 76 ‘‘student’’ answers, between 4 and 15 per topic, are currently available. When presented with a single answer, students select a mark from a preset range, using buttons in a web form interface, and feedback from the ‘‘Chief Examiner’’ is then displayed at the bottom of the page. The number of marks that can be selected by the student (between 3 and 6) varies with the complexity of the question and when not all marks in the range are available this is shown by use of inactive buttons (Fig. 1). The database contains only a limited number (about 15%) of answers of this type that the ‘‘Chief Examiner’’ considers worthy of full marks. For questions where the student is provided with two different answers to compare, they are asked to indicate whether or not both answers are worth full marks and are provided feedback from the ‘‘Chief Examiner’’ according to their response. In the WebCT environment, the feedback from the ‘‘Chief Examiner’’ was displayed using a JavaScript that interacts with the web form buttons. This JavaScript also makes the web form buttons invisible, so that the student can only see the feedback from the ‘‘Chief examiner’’ associated with the mark they selected. Examples of each type of marking
394
BAMBED, Vol. 35, No. 6, pp. 392–396, 2007 TABLE I Student Use of ‘‘You be the examiner!’’ Compared with Their Achievement in the Paper
Final grade
Number of topics accessed
A
B
C
Total number of students who passed
Total number of students who failed
0 1 ‡2
1 5 10
16 14 22
22 16 14
39 35 46
27 13 10
assignment are given in the Appendices. How the student awarded mark, compared with what might be awarded by the ‘‘Chief Examiner,’’ is indicated in words (Appendix A). Although no real student answers are used, inspiration for many of the ‘‘student’’ answers came while marking final examination scripts. Only a few ‘‘student’’ answers that could be considered excellent are included for two reasons. First, to avoid providing material that might encourage memorization and second, because the feedback associated with such answers will be less informative for low achievers who tend to score their peers generously [4]. The few that are included encourage students to think of answers in their own words rather than considering the answer they have just read to be the only correct answer. In 2006, ‘‘You be the examiner!’’ was introduced to students enrolled in the paper on the Palmerston North campus both by advertising the material in lectures and by inviting students to access the topic relating to their laboratory course during class time. Student feedback was invited via the WebCT discussion postings toward the end of the semester but before the final examination. This feedback was encouraging, indicating that the students found it helpful and were thinking about the content of the ‘‘student’’ answers rather than just surfing for a good answer to memorize. For example, one student wrote, ‘‘I reckon it’s good! It gave me an idea on how to construct my answers in the semester test and also gave me points on what I need to include in my answer for certain questions, which is probably why I did better in the test than expected.’’ One negative comment was posted. Interestingly, it was from a student who had failed the semester test but went on to pass the final examination! WebCT records the student use of the ‘‘You be the examiner!’’ pages. This data show that, of the 170 students enrolled in the paper in 2006 at the Palmerston North campus, 33% accessed two or more topics (Table I). These repeat visits suggest that these students perceived the material to be helpful. Statistical analysis of the data showed that there was a statistically significant relationship between use of ‘‘You be the examiner!’’ and student performance in the paper. The null hypothesis that a pass in the paper is independent of a student’s use of ‘‘You be the examiner!’’ was tested using the v2 test on a 2 3 3 contingency table [7]. Assuming independence, the probability of obtaining data as different from the expected values as that in Table I is less than 0.02 and, therefore, the null hypothesis can be rejected.
Students enrolled in the paper (%) 66 48 56
39 28 33
There is a statistically significant relationship between a pass in the paper and use of ‘‘You be the examiner!’’ Furthermore, the null hypothesis that a student’s grade (A, B, C, or Fail), and use of ‘‘You be the examiner!’’ is independent was tested using Fisher’s exact test on a 4 3 3 contingency table [7]. In this case, the probability of obtaining data as different from the expected values as that in Table I assuming independence is less than 0.01, and so this null hypothesis can also be rejected. There is a statistically significant relationship between grades achieved in the paper and use of ‘‘You be the examiner!’’ It could be argued that these relationships arise because only the higher performing students made use of ‘‘You be the examiner!’’ However, this was not the case. Students accessing two or more ‘‘You be the examiner!’’ topics had overall grade point averages (excluding this biochemistry paper) ranging from 1 to 8. This also suggests that students with a wide range of academic ability perceived the material to be helpful. Furthermore, there was no statistically significant correlation between the use of ‘‘You be the examiner!’’ (as indicated by the number of topics accessed by a student) and the student’s grade point average (Pearson’s correlation coefficient r ¼ 0.137, n ¼ 170, p > 0.05 [8]). These data should provide a strong incentive for our students to use ‘‘You be the examiner!’’ to sharpen their understanding of key biochemical concepts and suggest that this approach could be useful in other courses as well. Acknowledgment— Support and encouragement from the late Darrylin O’Dea, in the early stages of this project, and from Associate Professor John Tweedie, throughout this project, is gratefully acknowledged.
REFERENCES [1] S. A. Azer (2003) Assessment in a problem-based learning course: Twelve tips for constructing multiple choice questions that test students’ cognitive skills, Biochem. Mol. Biol. Educ. 31, 428–434. [2] A. J. M. Verhoeven, M. Doets, J. M. J. Lamers, J. F. Koster (2005) Metabolic interrelationships software application: Interactive learning tool for intermediary metabolism, Biochem. Mol. Biol. Educ. 33, 437–442. [3] W. H. Elliott, D. C. Elliott (2004) Biochemistry and Molecular Biology, 3rd Ed., Oxford University Press, Oxford. [4] V. Langendyk (2006) Not knowing that they do not know: Selfassessment accuracy of third-year medical students, Med. Educ. 40, 173–179. [5] B. Hodges, G. Regehr, D. Martin (2001) Difficulties in recognizing one’s own incompetence: Novice physicians who are unskilled and unaware of it, Acad. Med. 76, S87–S89. [6] J. Kruger, D. Dunning (1999) Unskilled and unaware of it: How difficulties in recognizing one’s own incompetence lead to inflated selfassessments, J. Pers. Soc. Psychol. 77, 1121–1134. [7] Tools for Science, Available at http://www.physics.csbsju.edu/ [8] Free Statistics Software (version 1.1.21-r4): http://www.wessa.net/
395 APPENDIX A
Example 2
Two examples of a single answer and the feedback associated with each of the possible marks that could be chosen by the student acting as examiner.
The question you set was ‘‘Explain how enzymes are regulated by (a) allosteric effectors and (b) covalent modification and give details of one example of each type of regulation. [10 marks].’’ Student N gave the following answer, how many marks would you give student N? An example, of an enzyme regulated by allosteric effectors, is phosphofructokinase. Its negative allosteric effectors are ATP and citrate (both decrease phosphofructokinase’s activity), and its positive allosteric effector is AMP (which increases phosphofructokinase’s activity). Covalent modification is illustrated by the enzyme pyruvate kinase that, in the liver, is phosphorylated (this decreases its activity) or dephosphorylated (this increases its activity). Feedback for Students Who Gave the Answer 0, 2.5, or 5 Marks—This begins with either ‘‘The chief examiner does not agree.’’ (0 marks), ‘‘You are a tough marker.’’ (2.5 marks) or ‘‘The chief examiner agrees with you.’’ (5 marks) and continues as follows: This student has attempted half the question and given relevant details of two examples, one each for covalent modification and allosteric regulation (although only three out of five allosteric effectors of phosphofructokinase are given). What is missing is an explanation of how phosphorylation or the allosteric effectors effect a change in enzyme activity. How would you explain this? If necessary, you should read the relevant sections in Chapter 16 of Elliott and Elliott. Allosteric regulation is described on pages 258–260. Allosteric effectors of other key enzymes in glycolysis and gluconeogenesis are found in Fig. 16.19 and the accompanying text on page 268. Covalent modification is discussed under the heading ‘‘Control of enzyme activity by phosphorylation’’ on pages 260–261, and the example described by this student is described on pages 270–271. Feedback for Students Who Gave the Answer 7.5 or 10 Marks—The chief examiner does not agree. This student has given relevant details of two examples, one each for covalent modification and allosteric regulation, but has only attempted half the question. There is no explanation of how either phosphorylation or the allosteric effector ATP effects a change in enzyme activity. How would you explain this? If necessary, you should read the relevant sections in Chapter 16 of Elliott and Elliott. Allosteric regulation is described on pages 258– 260. Allosteric effectors of other key enzymes in glycolysis and gluconeogenesis are found in Fig. 16.19 and the accompanying text on page 268. Covalent modification is discussed under the heading ‘‘Control of enzyme activity by phosphorylation’’ on pages 260–261 and the example described by this student is described on pages 270–271.
Example 1 The question you set was ‘‘Describe the regulation of the enzyme phosphofructokinase AND explain how this enzyme is important in controlling the rate of glycolysis. [5 marks].’’ Student U gave the following answer, how many marks would you give this student? Glycolysis is the process by which glucose is converted to pyruvate, with an overall net production of ATP. Phosphofructokinase is the enzyme that catalyses the phosphorylation of fructose-6-phosphate producing fructose 1,6-bisphosphate. This is an important enzyme for controlling glycolysis. Plenty of ATP indicates a ‘‘high energy charge’’ and since ATP cannot be stored glycolysis is slowed down. High levels of AMP indicate a ‘‘low energy charge’’ and this increases the rate of glycolysis. Feedback for Students Who Gave the Answer Zero Marks—The chief examiner agrees with you. This student has not answered the question. Can you recall the positive and negative allosteric regulators of phosphofructokinase? Find pen and paper and write them down now as you would in an exam (hint: there are five of them). Now write an explanation of how phosphofructokinase is important in controlling the rate of glycolysis. If necessary, refer to the textbook [Elliot and Elliot, pp. 268–270] and your study notes. Elliott and Elliott (pp. 258–260) may be useful background reading about allosteric regulation. Feedback for Students Who Gave the Answer Between 1 and 5 Marks—You are [a generous (1 mark)/a very generous (2 or 3 marks)/an extremely generous (4 or 5 marks)] marker. The chief examiner does not agree. This student has not answered the question. The first sentence describes glycolysis but this was not asked for. Sentences 2 and 3 repeat information provided to the student in the exam [phosphofructokinase is described as an enzyme in the question itself and the reaction catalyzed by this enzyme (phosphorylation of fructose-6phosphate) is given to the students (click here to see the pathways they are given)]. The last two sentences indicate the importance of ATP/AMP levels in controlling glycolysis but do not say how this is affected through phosphofructokinase. Can you recall the positive and negative allosteric regulators of phosphofructokinase? Find pen and paper and write down as many as you can now (hint: there are five of them). Now write an explanation of how phosphofructokinase is important in controlling the rate of glycolysis. If necessary, refer to the textbook [Elliot and Elliot, pp. 268–270] and your study notes. Elliott and Elliott (pp. 258–260) may be useful background reading about allosteric regulation.
APPENDIX B
Example of a request to compare two answers and the feedback associated with either a ‘‘yes’’ or ‘‘no’’ answer. The question set was ‘‘Explain the term ‘futile cycle’ [3 marks].’’ Two different students wrote the answers shown below. Would you give both students full marks?
396
BAMBED, Vol. 35, No. 6, pp. 392–396, 2007 Student A
Student B
A futile cycle is the result of two different enzymes (F and M), simultaneously catalyzing opposite and irreversible reactions. The net result of this simultaneous activity is that the energy (in the form of ATP) is wasted. Futile cycles can also involve whole pathways.
Feedback for Students Who Said ‘‘Yes’’—The chief examiner disagrees. The students were asked for an explanation. Only student A provided an explanation of the diagram. Using a figure is a good idea, but it is no good if there is no legend to explain what the figure is illustrating.
Feedback for Students Who Said ‘‘No’’—The chief examiner agrees. The students were asked for an explanation. Only student A provided an explanation of the diagram. Using a figure is a good idea, but it is no good if there is no legend to explain what the figure is illustrating.
