Critical thinking and skills in de®ning problems have been among the goals of a computers and society course that had its credits reduced and student number ...
Education and Information Technologies 3 101±117 (1998)
Teaching critical thinking and problem de®ning skills JENS J. KAASBéLL
Department of Informatics, University of Oslo, P.O. Box 1080 Blindern, N ± 0316 Oslo, Norway. E-mail: jens.kaasboll@i®.uio.no
Critical thinking and skills in de®ning problems have been among the goals of a computers and society course that had its credits reduced and student number increased. In an attempt to prevent worsening the students' learning, four measures were taken. The results actually improved from a failure rate of 21.5% to 0.7%. This was mainly due to tighter project structure and additional student work. Reduced course material and improved teacher preparation did not seem to have any effect, while more focused project teaching may have contributed to the decreased failure rate. Further improvements may be gained through time estimation in the problem de®nition process. KEYWORDS: Higher education; creativity; interdisciplinary; problem solving; social issues; teaching methods.
INTRODUCTION Employers and managers complain that computer science graduates lack practical competence, e.g. for writing and teamwork (Hartmanis and Lin, 1992). Denning (1992) argues that the engineers of the future need more communicative skills, and that the students should be given more responsibility for their own learning. Scragg et al. (1994) argue that students also have to learn creativity, and that this comes from insight into the facts, methods and paradigms of the ®eld. We have observed a similar lack of skill among students who embark on their Master's theses. Then they have to de®ne their own research problems in co-operation with their supervisors, ®nd their methods, and question the literature and their own empirical data. This requires skills in both critical thinking and the creativity needed to de®ne their own problems. We will call the latter `problem de®ning skills'. In most undergraduate courses, the teacher de®nes the student assignments, which 1360±2357 # 1998 IFIP, published by Chapman & Hall Ltd
102
Kaasbùll
can be solved within the time given, at the department's computer, and by means of the method lectured the same week. This kind of teaching does not speci®cally support critical thinking and problem de®ning skills. In the Department of Informatics, University of Oslo, we have for more than 20 years also taught an undergraduate course in computers and society where critical thinking has been encouraged. Students have had to expose themselves to the world of practical tasks outside the university, de®ne their own problems based on the illstructured situations they encounter, work with the problems without knowing how their conclusions will look, and write comprehensive reports of their work (Kaasbùll, 1981; Nygaard, 1982; égrim, 1991). However, only a minority of undergraduates attended this course. Consequently, many students started working on their Master's theses without this training. Therefore, the department decided to make this course compulsory for all students wanting to study at least one year of informatics. Concurrently, the credits for the course were reduced from 0.5 workload of a 13 weeks' semester to 0.3, putting much more stress on the ef®ciency of the course. The number of students increased from approximately 60 to 143, giving the lecturers less time for interaction with each student. We made several efforts to tackle this problem, the major ones being preparing the teachers, tightening the course structure, and providing more project directed teaching. The research problem addressed in this paper is how these efforts were effective in improving computer science students' skills in critical thinking and problem de®nition. Because we actually changed the course in the process of ®nding answers to the research question, we carried out action research. As often happens in action research, action takes precedence over research, making it dif®cult to interpret the effects of the actions. Recent evaluators of adult education suggest that information be collected from many sources (Athanasou, 1995); this is something we tried to achieve. Our sources consisted of written student assignments, grades given to the students by external examiners, examiners' statements, informal reports from the tutors and the students during the course, and answers to questionnaires. Unfortunately, the response rates to the questionnaires were 40 and 34%, disabling most generalizations from the responses. Therefore, the responses have been used only when no better information has been available. Previous research in teaching critical thinking and problem de®nition skills is described in the next section. Then the actions taken to change the course are described and how the number of students who passed the course compares with previous years. After a discussion of whether this measure re¯ects improvements in students' abilities of critical thinking and de®ning problems, the effects of the different changes are discussed in the remaining sections. PREVIOUS RESEARCH Training students in the skills needed to set up and carry out their own research project can of course be done through a course having this as its main activity. In
Critical thinking and problem de®ning skills
103
fact, we run the course this way. In addition to this similarity between means and ends, educational science provides another reason for choosing this teaching strategy.
Principles of instruction Critical thinking and the ability to de®ne problems both belong to the category of general cognitive skills. Pascarella and Terenzini (1991) have carried out a survey of research on what students learn at college. They refer to research that suggests that institutions with a broader curriculum in¯uence positively the development of general cognitive skills (p. 137). They de®ne `critical thinking' as: the individual's ability to do some or all of the following: Identify central issues and assumptions in an argument, recognise important relationships, make correct inferences from data, deduce conclusions from information or data provided, interpret whether conclusions are warranted on the basis of the data given, and evaluate evidence or authority. (p. 118)
Even if `identify central issues' and `recognize important relationships' also are parts of problem de®nition, their concept of critical thinking does not include the ability to act in ill-structured situations, which is necessary in order to de®ne problems. We adopt this de®nition of `critical thinking', except that `identify central issues' and `recognize important relationships' are given less emphasis, because these are also included in `problem de®ning skills'. Pascarella and Terenzini (1991, p. 144) only found one experiment that signi®cantly correlated a particular form of instruction with a positive in¯uence on critical thinking, i.e. the integrated teaching of humanistic disciplines that otherwise were taught separately. Problem de®ning skills constitute one out of ®ve features of practical competence (Pogson and Tennant, 1995). This is the competence needed to cope with real world situations, as opposed to intelligence, which correlates with the ability of theoretical studying. Problem de®ning skills do not seem to correspond to one category of cognitive skills. However, the categories `re¯ective judgment' and `conceptual complexity' seem to be preferable abilities. `Conceptual complexity is the extent to which a person is capable of attending to a large variety of cognitive stimuli and organising her or his dealings with the external environment in increasingly abstract, complex, and varied ways' (Pascarella and Terenzini, 1991, p. 126). Re¯ective judgment has been measured through subjects having to justify their position in an illstructured problem, which is represented by two contradictory points. A speci®c curriculum where courses were organized inductively from the concrete to formal reasoning in¯uenced conceptual complexity positively (Pascarella and Terenzini, 1991, pp. 142, 145). The more general factors of student involvement and student ± teacher interactions also have positive in¯uences on general cognitive skills.
104
Kaasbùll
In summary, the following principles of instruction seem to foster critical thinking and problem de®ning skills: · · · ·
student involvement, student ± teacher interaction, inductive teaching, and integration of disciplines.
Problem based learning Problem based learning is an inductive educational strategy that aims at increasing student involvement and integrating knowledge from different sources, and it requires interaction between the student and teacher. Therefore, this strategy may be effective for learning the skills at which we aim. A main principle of problem based learning is that the students work on real life problems or constructed problems that mimic the complexity of the practical world. In such situations, the students have to think critically through all the information available to sort out the relevant material. When students have the opportunity to de®ne their own problems, they become more involved in their work, and this involvement increases motivation for learning. Because students are assumed to obtain a more profound understanding of the subject area, assessment of problem based learning should focus more on the students' skills in handling an ill-structured situation than on recalling the textbook. Problem based learning or similar approaches have been used in software development courses. The literature con®rms that this approach to learning is feasible. Kirsch's (1996) report of a human±computer interface course, where students could choose their problems themselves, gives three major assignments plus weekly explorations of techniques to be used in the project. Kirsch gives an enthusiastic report of the students' achievements. However, he does not provide rigorous comparison with more traditional teaching methods. Ryan and Quinn (1994) report of a course in which the students are given an incompletely speci®ed design task. The students evaluate the course positively, and an increasing number of students have been attracted to the course. Baldwin (1996) reports on courses conducted as `discovery learning', where the instructor sets the project goal. The students in a computer graphics course following these principles were able to construct a sophisticated ray tracer and a three-dimensional game, while students at previous lecture based courses could hardly make up a ray tracer for a single object and a two-dimensional game. In Adams's (1993) course, the instructor has the role of project manager, who plans deliverables and weekly due dates and reviews. Even if the students have a more subordinate role in this course, Adams reports that the students became engaged. These experiences from computer science education unanimously give a positive evaluation of the principles of problem based learning. However, the evidence is
Critical thinking and problem de®ning skills
105
episodic rather than scienti®c. It is worth noticing that frequent follow ups on the students' progress are mentioned as an important part of the course structure by most of the authors. COURSE STRUCTURE The computers and society course, in the Department of Informatics, is usually taught by two lecturers. The lecturers responsible for the new course had also taught the old course a total of four times during the ®ve last years. In addition, the course has tutors, each responsible for one or two classes of 15±25 students. Three of the ®ve tutors have teaching experience of the old course. The students have to complete a project in groups of four±six, supervised by their tutor. The typical student attending the course is male, between 20 and 25 years old, his mother tongue is Norwegian, and he has completed two semesters of study in informatics and one semester with mathematics and statistics. The students carry out ®eld studies of computer systems in organizations. The students have to ®nd suitable organizations and arrange contacts themselves. Summaries of previous projects can be found in (Kaasbùll et al. 1993) and (Kaasbùll and égrim, 1994).
Changed conditions The main changes of the conditions for the course were reduced credits, and the fact that it became mandatory for computer science students. The latter means that students who are not interested in this particular subject also attend the course. Among the 57 students who responded to the survey made after one month of teaching, six ticked lack of interest as their main reason for putting less work than required into the course. Without any knowledge about the 100 students who did not respond, it is not possible to say how widespread the low motivation was. However, in an optional course, one would not expect that any student had low motivation. Apart from students' motivation, we have no reason to suspect any other changes in the population of students attending the course.
Course changes in response to changed conditions The number of lecturers, the ratio of students per tutor and the required project work were unchanged. The modi®cations made to adapt to changing conditions concerned course material, structure, teacher preparation and sessions. An overview of the changes is given in this section, while more details are provided later, where we try to explain their consequences. The material for the new course consisted of parts of a textbook on management research (Easterby-Smith et al., 1991) and a textbook on management and technology (Levin et al., 1994). The remaining material was available through the World Wide Web (égrim and Kaasbùll, 1996). A collection of articles used in the old course was abandoned in the new one.
106
Kaasbùll
The course schedule allocated 2 h for lecturing (reduced from 4 h) and 3 h in class per week for 13 weeks. There were six lectures concerned about issues within the subject of computers and society, and ®ve lectures aimed at guiding the project work. In addition there were three class exercises concerning the subject and three aimed at the project. The remaining six classes were used for supervision of the projects. Given an average of 40 h studying time per week, 0.3 full-time credits over 13 weeks gives a total of 156 h of student work during the course. Assuming that the students spent 10 h on subject related activities and activities of no relevance to the project, the time allotted for the activities in the course can be separated as shown in Table 1. The previous course was scheduled to occupy twice the amount of time (300 h) during the semester, including a three±four week period of time allotted for study between delivery of the project and the oral exam. The exam was abandoned in the new course, and replaced with grades based on the project work. In total the amount of time for project work was reduced 33%, and the time for project work outside the scheduled lessons was reduced by 43%. In addition to these changes, we also initiated and carried out training of the tutors prior to the course. Four of the ®ve tutors attended this training.
ASSESSMENT
Criteria for grading The projects were marked according to four main areas: subject matter, empirical study, reporting and working process. These areas were further divided into subareas, see Table 2. Both the subject matter and the empirical study had to be passed in order to obtain a passing grade. Problem de®ning skills are covered in the sub-area `problem de®nition' under `empirical study'. It emphasizes that the problem is grounded. The sub-area of `relevance' under `subject matter' requires the ability to recognize important issues. Table 1. Student hours allotted for course activities
Activities Lecturing Class exercises Supervision
Previous course
New course
Project
Project
Subject
Subject
10 8 21
30 32
10 12 18
12 9
Organized teaching in total Unorganized studying (rough estimate)
39 149
62 50
40 85
21 10
Total organized and unorganized hours
188
112
125
31
Total for the whole course
300
156
Critical thinking and problem de®ning skills
107
Table 2. Criteria for grading projects (IT information technology) Area Subject matter Insight in relations between IT and society Relevance Empirical study Problem de®nition Coherence between problem and empirical method Quality of data collection Coherence between data and analysis=conclusion Coherence between problem and data=conclusion Report Structure and disposition Quality of presentation Language Presentation and analysis of other literature Process Organization of work
Management of controversies Planning and following up Constructive self-critique
Criteria Re¯ection on the in¯uence of IT, and understanding of others' relations to IT Of interest to speci®c groups The problem and the partial problems are grounded, precisely formulated, and can be answered during the course Methods and selection of units to be studied are grounded in the problem, and reasons for not choosing other methods are given Follow the methods as thoroughly as time permits. Different sources are compared Analysis and conclusion are based on data Data is relevant for problem, and conclusion answers problem Logically grounded sequence of chapters that focuses the main points Documents method, presents data, and conveys units of study, analysis and conclusion in suf®cient depth and precision Well formulated and easy to read Refer to others' work, relate to it, and show its function in own work Account for roles, areas of responsibility and division of tasks, and explain why the group decided this structure based on individual differences and the principle that one ought to learn the skills of which one is ignorant. Corroborated with quotations from minutes of group meetings Account for how the group has handled controversies and possible con¯icts. Corroborated with quotations from minutes of group meetings Account for how the group has handled delays and changes of plans Account for what the group would have done otherwise given they could redo the project
Analyses of data, comparing sources and drawing conclusions require the skills of critical thinking, and these are included in `empirical study'. However, critical reference to other literature is not included because this is categorized under the `report' area of assessment. With this exception, the areas `subject matter' and
108
Kaasbùll
`empirical study' cover problem de®ning skills and critical thinking as we intended in the course, and also, as far as we can see, in accordance with (Pascarella and Terenzini, 1991). The previous course had no such detailed list of requirements. However, the requirements made up for the new course were based on our experience of assessment in the old course, such that the students were assessed in the same main areas. The level of expectation of student performance remained, except in the process area, where the new requirements constituted a distinct rise of expectation. In the previous course, the students were given pass or not pass based on their project and an oral examination where both the project and subject matters were discussed. In the new course, only 20% of the students were given an oral examination, and this examination concerned only their project. The faculty's numerical grading scale consisting of 31 steps of passing grades from 1.0±4.0 was used.
Results The students were organized into 28 projects, and some report titles were: · Commerce on the Internet ± a survey of Norwegian net users' attitudes towards buying and selling on the net. · Change of an application system at Telenor Research & Development. · Will e-mail increase the possibility for changes of power structures? · Who cares where you come from? Employers' conceptions of IT education. Out of 143 students delivering their projects, only one student failed. This student was expelled from a group and was tested in an oral exam. The average grade of those who passed was 2.6. A comparison with the total number of students who took the previous course in the previous ®ve years is given in Table 3. Because the actual distributions of the pass=fail grades of both the old and new courses are known, a chi-square test between the two will show whether they correlate. The chi-square of the distributions is 33.3. A score above 6.64 indicates, with a probability of 0.99, that the distributions do not correlate. Because the grading is based on projects rather than students, the corresponding number of projects is shown to the right in Table 3. The chi-square of the projects is 7.25; this is also above the 0.99 level. Hence, we conclude that the proportion of students who passed the new course has improved signi®cantly compared with the old course. Table 3. Assessment of students (the previous course columns show the accumulated numbers of the last ®ve years) Number of students
Number of project groups
Previous course
New course
Previous course
New course
Passed Failed
229 63
142 1
46 13
28 0
Total
292
143
59
28
Critical thinking and problem de®ning skills
109
We set out to improve students' critical thinking and problem de®ning skills, and we have seen a signi®cant improvement of students' abilities to pass the course. In order to pass, the areas of assessment called `subject matter' and `empirical study' have to be passed. We have argued that these areas include problem de®nition and critical thinking, excluding critical reference to other literature. We therefore conclude that with this exception, the students who passed have the required skills in critical thinking and problem de®nition. In order to show an improvement in these skills, we must also establish that those students who failed in the previous course failed in these issues. The requirements for the new course were based on the old, with no intention of changing the areas of `subject matter', `empirical study' and `report'. The failures in the old course were commonly due to poor coherence between problem de®nition, method, analysis and conclusion, and the poor coherence could often be tracked down to poor problem de®nition. Therefore, we believe that the grades constitute a valid measure of critical thinking and problem de®ning skills. The projects were graded by examiners external to the university. Out of the four examiners of the new course, two had also been examiners for several years in the previous course. Together, the two experienced examiners marked ten projects, all of which passed. This number is too small to yield any statistically signi®cant increase in the number of projects they marked as passed. The chi-square value corresponds to the 0.9 level of con®dence. One of the experienced examiners said: The ®ve reports I had were in general at a much higher level than before. None of them were bad, and one was exceptionally good.
Adding the fact that the only student who failed was judged by the new examiners, makes us believe that the result is also a reliable measure of student improvement. Taking the reduction in credits and increase in the number of unmotivated students into consideration, we were very pleased with the improvement shown by the students. We identify the following factors as possible reasons for the improvement, the ®rst four of which constitute our deliberate changes, and we will subsequently discuss each factor. · · · · ·
reduced course material, improved teacher preparation, tighter project structure, project directed teaching, and increased amount of student work.
REDUCED COURSE MATERIAL The course material was reduced to partly make up for the reduced credits of the course. Our impression from previous years is that the students put their main effort on reading between the end of the project and the oral exam; see our estimate of unorganized studying in Table 1. Because they presumably spent little time on
110
Kaasbùll
reading during the project period of the old course, there was little time saved for project work this way. Some of the responses to the open questionnaire re¯ected this, e.g. ``I don't understand how one can remove one book and say this is a 3 credit course. The project report is the same. Same amount of work.'' We therefore rule out reduction in course material as a reason for the improved results. PREPARING THE TEACHERS We arranged a ten day pre-course training for the tutors two months before the course commenced. Four of the ®ve tutors attended the course, three of whom had acted as tutors before. We spent 14 h together with the students, partly lecturing, but mainly discussing the issues with the tutors. The main assignment negotiated was to make up guidelines for tutors on how to tackle the most dif®cult part of their teaching. The guidelines they wrote up were useful, and the tutors were awarded a good mark by the external examiner. The tutor who did not follow the training course supervised nine projects with a total of 49 students, all of whom passed. This is a better result than the group of tutors who attended the training course had! However, we cannot generalize any ®nding based on one tutor. In addition to the training course, we had weekly teacher meetings with the tutors, where we discussed the students' progress and possible ways for the tutors to react to actual problems. From educational science, we know that recurrent sessions on a topic are more effective than one shot training. The weekly discussions may therefore explain why the tutor who did not follow the training course had such a good result. This is also in line with Carbone et al.'s (1996) observation that regular discussions between lecturers and tutors on the tutoring sessions of a programming course have a greater impact on teaching quality than a preliminary course for tutors. Another possible explanation is that the course gave too little practical training on how to guide students during a project. We have no knowledge of the selection of students in different classes, and only one tutor did not follow the course. Consequently, we ®nd no effect of the training course on the tutors' competence, as judged by the results of their students. However, the ®ndings suggest that the weekly meetings are important. TIGHTER PROJECT STRUCTURE We tightened the structure of the project in four major areas: time schedule, project organization, criteria for evaluation and numerical grading.
Frequency of deliverables In the previous course, there were three written and one oral mandatory progress reports on the project during the semester. Experiencing that the students did not spend their time effectively during the ®rst half of the project, we doubled the num-
Critical thinking and problem de®ning skills
111
ber of written deliverables. We also tried to make them like draft versions of sections of their ®nal report, e.g. problem formulation, units of study, method and results. The emphasis in the previous course had been more on status reports than on the students' understanding. The ®rst questionnaire that was handed out after a month showed that two-thirds of the students who answered said that they spent more than the allotted 12 h per week on the course. This indicates that the students had a high level of activity from the start of the project. Many comments on the deliverables were positive, e.g.: Our tutor had more assignments than the lecturers had decided. Could have had even more One needs to start working on the project at an early stage.
The tutors who checked and accepted the deliverables were unanimously in favour of the tight control, saying that the students thereby were forced to carry out the tasks required in their projects. The intermediate reports were also effective in giving early warnings of projects that were not on track. During the weekly teacher meetings, we had numerous discussions on which actions to take, based on the students' reports. Also the external examiners emphasized that the groups that worked consistently throughout the semester produced a better report than those who started off late. It seems that the frequent deliverables, at least every two weeks, helped the students through the projects. Other comments suggest that the schedule was too rigid: Too many deliverables, many of which were irrelevant for the project. They [the deliverables] were too in¯exible to ®t each group.
These comments may indicate that the deliverables offer the students too little freedom in their scheduling of work tasks at a particular stage. However, we do not think it had a major impact on their choice of project and research method, because the deliverables requested general issues like problem formulation, literature, units of study, etc. To provide more ¯exibility, one could require the students to submit a partial product every two weeks, and leave the actual choice of deliverable to the students and the tutors.
Project organization The students were required to set up a structure for their project work. Sets of work tasks had to be assigned to roles, and the persons could take up roles, some of which might be changed during the course. They were required to adapt roles from which they could learn new skills. They were also required to agree on procedures of work, in particular how to make decisions and how to follow up when plans were not met. In the previous course, the students were encouraged but not required to set up a formal project organization.
112
Kaasbùll
Responses to the ®rst questionnaire indicated that the groups who had decided upon a clear leadership had structured their work more tightly than those who did not want to have a leader. Whether having a clear leader role or not, the responses indicated that the more engaged and extrovert students had a greater say in decisions. In the previous course, project reports with poor cohesion, consequently making the students fail, were often produced by groups with major internal con¯icts. Although one group had a split-off of one member, other con¯icts in other groups were resolved or controlled. Considering general knowledge of con¯icts, we found it likely that groups who had agreed on procedures for decision making and internal control were more able to prevent and control con¯icts. Consequently, we believe that the tighter project organization helped students learn more, through preventing and controlling possible obstacles to learning.
Criteria for assessment Mature learners are more inclined to aim at goals of learning than youngsters, who would rather follow the teacher's instructions timely. We therefore assumed that introducing rather detailed criteria for grading would have an impact on what the students tried to achieve. The assessment shows that the students met the majority of the criteria at an acceptable level. Two observations indicate that the students also considered the criteria during their projects. The reports in general had a better cohesion between problem, description, discussion and conclusion than before. They also included the important elements which are assumed to be in a report. The criteria for evaluation were obviously used a lot during writing, making some of the reports a little bit too mechanical. And what was not included in the criteria was not in the reports, even if it could have ®t their topic. (one of the external examiners)
There is obviously a risk that the students' work becomes more in¯uenced by standard criteria than the projects' problem. Although the criteria seem to have prevented poor reports, students who master their projects should not feel that they get bad marks due to not following the guidelines in every respect. The ®rst criterion, `re¯ection on the in¯uence of IT, and understanding of others' relation to IT' was less operational than most of the other criteria. It is our general impression that the students on the new course showed no progress compared with the previous course in this respect. The other criteria on subject matter, empirical study and report were more operational, and the students seemed able to meet them. A way to try to achieve improved results also in this respect would be to reformulate the ®rst criterion in more operational terms. The criterion may be easier to ful®l if expressed like, e.g. `Explain how your ®ndings are aligned with our general knowledge of the in¯uence of IT in society' and `Report how the persons in your empirical study conceived and understood the IT.' The other indication that the criteria were effective on student behaviour, was a
Critical thinking and problem de®ning skills
113
rather frank suggestion for improvement found among the responses to the ®nal questionnaire: Nearly all the process criteria are completely behind target! Consequently the work load and the requirements become overwhelming, so the students tailor a report according to these requirements. Lie a little bit here and there . . .
This admitting of opportunistic behaviour indicates that the students regarded the criteria so seriously that they even tried to cheat in order to ful®l them. In addition, the response also indicates that the process criteria should be reconsidered completely. When students admit fabricating data in this respect, one may also wonder whether they have conducted the empirical work described in their reports. The tutors followed the progress of the students' work closely, without becoming suspicious of any cheating. In fact, we believe that it is easier to cheat in courses where the students can copy reports over the Internet than in this course where work progress is monitored frequently. Although we do not know whether the criteria or the frequency had greater impact on student learning, the observations strongly suggest that the criteria have aided the students.
Numerical grading Changing from pass=fail to the numerical scale was done to make the students try to achieve more than just a poor `passed', and to honour those who did well. Out of the 43 students who responded in the ®rst questionnaire that they spent more than the allotted time on the course, grading was their most important motivation. This result indicates that the numerical grades were effective in making the students work hard. PROJECT-DIRECTED TEACHING Project relevant class teaching was the only type of teaching that had more hours. In some of the previous courses, the projects were focused on one area of study, e.g. Internet applications. Then some of the lessons concentrated on that particular area, while all the lessons in the new course were aimed at general skills useful for the project work.
Problem de®nition Russell et al. (1994, p. 64) suggest that problem de®ning skills should be taught in two phases. First, the students select a topic, carry out a literature study, and write up their understanding of the area. Second, they develop the question they would like to research. Our approach mixed problem de®nition and empirical studies to a larger degree. During the ®rst week, the students were required to write up a page on the area of interest and the problem on which they wanted to work. Afterwards, they explored the empirical area and read some literature, whereupon they re®ned and possibly revised their questions.
114
Kaasbùll
In a training session on project de®nition, the students were given two problem formulations from earlier reports, and the following criteria that a problem de®nition should ful®l: · · · · ·
new and partly unknown, motivating for the students, of interest to others, can be realized, and manageable during the course.
They compared the given examples with these criteria, and then proceeded to assess their own projects. Subsequently, they reworked their problem de®nitions several times during the project period, often after feedback from the tutors on deliverables. In the ®rst questionnaire response, this session was the only main activity that the students thought had contributed to their skills in de®ning problems. In response to the second questionnaire, some students said that they would have liked there to have been more guidance on de®ning problems, separation into partial problems, and how to limit their work. The training should probably focus more on the process of de®ning problems. Russell et al. (1994, pp. 65±7) mention drawing the situation and second-hand reports as suitable techniques. Research on the effects of teaching directed towards cognitive skills and intellectual growth suggests that a cumulative set of mutually reinforcing experiences over an extended period of time is better than a single lesson (Pascarella and Terenzini, 1991, p. 159). Instead of generating more sessions, the students' recurrent reworking of their own problem de®nitions may therefore be more effective in learning problem de®ning skills. Requirements for resubmitting the problem de®nition could be included in two or three of the deliverables.
Critical thinking Lectures and class lessons on research methods and analysis of results were carried out. The latter was added in the new course. Methods of empirical enquiry include reading documents, hands-on experience with the systems, studying logs of use, observations of use, interviews, and possibly also questionnaires. The students were encouraged to try at least two methods of enquiry, and most of them followed that advice. Student response to the last questionnaire mentioned particularly that they would have liked more training on how to structure a report. This should also be included in teaching. We cannot evaluate the contributions of the individual sessions on the overall performance of the students. The effects of each class exercise were discussed at the weekly teacher meetings. In addition to minor hints for improving the exercise, the tutors typically said:
Critical thinking and problem de®ning skills
115
I don't think the students became very good at interviewing, but, you know, every training session helps.
Our judgment is that there should be a training session for each major assessment criterion. AMOUNT OF STUDENT WORK The planned time for unorganized project work was reduced from 149 to 85 h, see Table 1. Thirty-seven out of the 45 students responding to the questionnaire said that the work load exceeded the credits. The 28 students who answered the same question in the last year of the previous course also complained about too high a work load. The report of a high work load had increased, but not signi®cantly, for the new course. Unfortunately, only a minority of the students have responded to the questionnaires. Introduction of numerical grading was mentioned as a reason for the students' eagerness. We believe that the hours put into the project work was not reduced much compared with the old course, and that the students' efforts contribute to explaining their good results. Increasing the credits is a simple solution for aligning the amount of work with the course requirements. However, this solution may require other changes to the curriculum. If wanting to reduce the requirements such that less time is needed to complete the course, we should have had more knowledge on which tasks consume the most time. Our general impression is that much time is spent on co-ordination, empirical enquiry and report writing. Reducing group size will reduce the need for co-ordination. However, there will also be fewer persons to share the burden, and the group will be more vulnerable to quitting group members. Some groups carry out a large number of interviews or they administer a questionnaire, while others are capable of passing with four interviews and reading appropriate literature. A way of reducing work load can be to consider the amount of empirical study needed more thoroughly during problem de®nition. Less empirical material could also reduce the length of the reports, which often reach the limit of 50 pages. CONCLUSIONS Managers and researchers have called for computer science graduates to have better skills in teamwork, writing and communication, and that they should also take more responsibilities for their own learning. Problem de®ning competence constitutes the skills that are needed to cope with real world situations, where the abovementioned skills are important ingredients. In order to strengthen students' learning of critical thinking and problem de®ning skills in a course, four main changes were introduced: reduced course material, im-
116
Kaasbùll
proved teacher preparation, tighter project structure and more project-focused teaching. While we found no direct effects of the two ®rst mentioned changes, tighter project structure seemed to be the main cause of the actual improvement observed. The credits of the course were reduced, and we believe that the students worked more than the scheduled 12 hours a week, and that their extra efforts also contributed to the improvement of learning. Approaches reported in the literature (Adams, 1993; Baldwin, 1996; Kirsch, 1996) follow up the projects in a weekly or bi-weekly frequency. Our study con®rms that tight follow up is essential for students' learning. When we allowed students to de®ne problems of their own, we required the problems to be new and partly unknown, motivating for the students, of interest to others, realizable and manageable during the course. Considering the high workload reported by the students, the students' problems did not ful®l the last of these requirements. A way of improving this issue could be to provide the students with estimates of the time required for single tasks like making an interview and ®nding a literature reference. Even if the actual time spent may differ greatly from the estimate, work that exceeds allotted time considerably should make the students rethink their problem. In a software engineering course, estimation techniques could be used in the same manner. Detailed criteria for assessment prevented poor reports. However, the individuality of projects should also be honoured, even if this implies that some criteria are not met. Pascarella and Terenzini's (1991) literature review concluded that student involvement, student±teacher interaction, inductive teaching and integration of disciplines were effective in fostering general cognitive skills. These principles were followed to the extent possible, and we have found no indications that they should be abandoned. While learning critical thinking has attracted research, less knowledge exists on how students acquire problem de®ning skills. We have con®rmed that the motivational factors (interesting for students and for others), and the research factors (partly unknown and realizable) are valuable for learning problem de®ning skills. Our case also indicates that management of the problem solving activity should be taken into account in problem de®nition. If not, students may drown in tedious work that does not necessarily foster learning. While software project assignments seem to be common in information systems teaching, we have only encountered one other case (Kirsch, 1996) where the students had to ®nd the problems themselves. In most of the areas taught in informatics, e.g. programming, software engineering, systems analysis and design, human±computer interaction, communication, multimedia, there are ample opportunities for letting students both de®ne and solve problems, thus increasing their responsibility for learning and strengthening their motivation. The study reported in this paper shows that student de®ned projects are feasible provided that there is tight follow up by the teachers.
Critical thinking and problem de®ning skills
117
REFERENCES Adams, E. J. (1993) A project-intensive software design course. SIGCSE Bulletin 25(1), 112±16. Athanasou, J. (1995) Issues in the evaluation of adult education. In Foley (ed.) Understanding Adult Education and Training. St Leonards, Australia: Allen & Unwin. Baldwin, D. (1996) Discovery learning in computer science. SIGCSE Bulletin 28(1), 222±6. Carbone, A., Mitchell, I. and Macdonald, I. (1996) Improving teaching and learning in ®rst year computer science tutorials. In A. Christie, P. James and B. Vaughan (eds) Proceedings of the thirteenth annual conference of the Australian Society for Computers in Learning in Tertiary Education, Adelaide, South Australia. Denning, P. (1992) Educating a new engineer. Communications of the ACM 35(12), 83±97. Easterby-Smith, M., Thorpe, R. and Lowe, A. (1991) Management Research: An Introduction. London: Sage. Hartmanis, J. and Lin, H. (eds) (1992) Computing the Future: A Broader Agenda for Computer Science and Engineering. Washington, DC: National Academy Press. Kaasbùll, J. (1981) Computers and society: a part of informatics. In Raasio and Fontell (eds) Education on Data Processing and Cooperation in Computer Technology. Helsinki: Kaasbùll, J., Braa, K. and Bratteteig, T. (1993) User problems concerning functional integration in thirteen organizations. In D. Avison, J. Kendall and J.I. DeGross (eds) Human, Organizational, and Social Dimensions of Information Systems Development. IFIP Transactions A-24, pp. 61±81. Amsterdam: North Holland. Kaasbùll, J. and égrim, L. (1994) Super-users: hackers, management hostages, or working class heroes? A study of user in¯uence on redesign in distributed organizations. In P. Kerola, A. Juustila and J. JaÈrvinen (eds) Proceedings of the Seventeenth Information Systems Research Seminar in Scandinavia. University of Oulu, Department of Information Processing Science, Research Paper Series A 21, Part II, pp. 784±98. Kirsch, R. P. (1996) Teaching OLE automation: a problem-based learning approach. SIGCSE Bulletin 28(1), 68±72. Levin, M., Fossen, é. and Gjersvik, R. (1994) Management and Technology. Introduction to Organization and Management. (in Norwegian) Oslo: Universitetsforlaget. Nygaard, K. (1982) Computers and society. In H.L.W. Jackson (ed.) Teaching Informatics Courses: Guidelines for Trainers and Educationalists. Amsterdam: North-Holland. Pascarella, E. T. and Terenzini, P. T. (1991) How College Affects Students: Findings and Insights from Twenty Years of Research. San Francisco: Jossey-Bass. Pogson, P. and Tennant, M. (1995) Understanding adult learners. In Understanding Adult Education and Training. Foley (ed.), St Leonards, Australia: Allen & Unwin. Russell, A. L., Cready, D., Davis, J. (1994) The use of contract learning in PBL. In S. E. Chen, R. M. Cowdroy, A. J. Kingsland and M. J. Ostwald (eds), Re¯ections on Problem Based Learning. Australian Problem Based Learning Network. pp 57±72. Sydney. Ryan, G. L. and Quinn, C. N. (1994) Cognitive apprenticeship and problem based learning. In S. E. Chen, R. M. Cowdroy, A. J. Kingsland, and M. J. Ostwald (eds), Re¯ections on Problem Based Learning. Australian Problem Based Learning Network. pp. 15±33. Sydney. Scragg, G., Baldwin, D. and Koomen, H. (1994) Computer science needs an insight-based curriculum. SIGCSE Bulletin 26(1), 94. égrim, L. (1991) Project work in system development education. In P. van den Besselaar, A. Clement and P. JaÈrvinen (eds) Information System, Work and Organization Design. pp. 165±72. Amsterdam: North-Holland. égrim, L. and Kaasbùll, J. (1996) Computers and Society Course. (in Norwegian) URL: http:==www.i®.uio.no=,in165=h96=
