Learning within incoherent structures: the space of online ...

Journal of Computer Assisted Learning (2002) 18, 351-366

Learning within incoherent structures: the space of online discussion forums M.J.W. Thomas School of Aviation, Massey University (now at the University of South Australia)

Abstract Online discussion forums are an increasingly common use of new information and communication technologies in education. As a tool for promoting conversational modes of learning, it has been suggested that online discussion forums can lead to enhanced learning outcomes for students. However, there is a need to explore further the implications of the highly mediated nature of computer–based interaction on student learning within these virtual learning environments. This paper presents results from a detailed study of students’ learning outcomes and patterns of interaction within an online discussion forum. The findings suggest that the typical nonlinear branching structure of online discussion may be insufficient for the realisation of truly conversational modes of learning. The paper discusses the implications of these findings in relation to students’ learning. Keywords: Case study; Conferencing; Discourse analysis; Environmental studies; Online; Questionnaire; Student-centred; Undergraduate Introduction Communication, rather than individual knowledge acquisition, has recently been emphasised as a means to increase the quality of instruction. However, as we move towards increasing use of information and communication technologies as a means of supporting student learning, it is necessary to evaluate the discontinuities between the structure of familiar conversation and asynchronous text-based communication. Contemporary explorations of the role of internal and interactive dialogue in students’ construction of knowledge have lead to the development of a conversational model of learning (Laurillard, 1993; 1999). Enhanced learning outcomes such as increased motivation and engagement in the learning task, deeper levels of understanding, increased metacognition, the development of higher-order thinking skills and divergent thinking have been shown to result from the adoption of conversational modes of learning (Blumenfeld et al., 1996; Flynn & La Faso, 1972). The development of new information and communication technologies has in turn offered new means by which conversational modes of learning can be achieved. As Harasim (1990) has suggested, the introduction of online education may open unprecedented opportunities for educational interactivity. Similarly, it has been argued that information and communication technologies offer new forum in which Accepted 11 May 2002 Correspondence: Matthew J. W. Thomas, University of South Australia, City West Campus, North Terrace, Adelaide SA 5000 Email: [email protected]

 2002 Blackwell Science Ltd

351

352 M.J.W. Thomas collaborative knowledge construction can take place (Littleton & Häkkinen, 1999). The common conception of the online discussion forum is that it is a virtual learning environment in which students are likely to learn as much from one another as from course materials or lectures. What they learn can be seen not so much as a product, but as a creative cognitive process of offering up ideas, having them criticised or expanded on, and being able to reshape ideas in the light of peer discussions (Rowntree, 1995). Similarly, it is believed that by reflecting on peers’ contributions in online discussions, students engage in higher-order processing of information, then through the act of articulating their own emergent understanding, are led towards the construction of personal meaning which is not individualistic, but rather a product of the students’ interaction (Bates, 1995). These descriptions of the process of online learning do highlight some tangible benefits of the virtual learning environment. It has been shown that online discussion forums increase participation and collaborative thinking through the provision of asynchronous, nonhierarchical and reciprocal communication environments (Ruberg et al., 1996). Furthermore, online discussions provide a perfect forum for an academic discourse which promotes increased student engagement, critical analysis and reflection, and the social construction of knowledge (Warschauer, 1997; Dehler & Parras-Hernandez, 1998). Recent research continues to re-emphasise these potentials of online discussion forums and has examined further possibilities for enhanced learning. It has been suggested that new technologies can support learning that reflects the multiplicity of knowledge forms through multiple modes of representation and configuration (Koschmann et al., 1994). Therefore, technologymediated learning becomes more than the transfer of declarative knowledge, and promotes the construction of complex knowledge structures. Secondly, technology has been shown to support active learning and, in particular, learning where students must articulate and negotiate their developing knowledge structures (Greening, 1998). Through the use of online discussions, students participate in learning where they were confronted by a variety of different perspectives, engaged in critical reflection and many change their outlook as a result of the exercise (Harrington, 1992). However, the social dynamics of computer-mediated communication are quite different from those of traditional face-to-face communication. As early as the mid1970s, it was hypothesised that the emergent forms of telecommunications media vary in their degree of social presence and, therefore, have affects on group interaction (Short et al., 1976). Computer-mediated communication has been shown to place the focus on what is said and remove seemingly extraneous aspects of faceto-face communication (Sproull & Keisler, 1991). Further, it is often suggested that the lack of nonverbal cues, and the associated depersonalising of communication, allows for a more egalitarian mode of communication (Willis, 1991; Ruberg et al., 1996). As research has also found, introverted students are more likely to benefit from computer-mediated communication than extroverted students, as introverted students find it easier to express themselves in the depersonalised forum (Straus & McGrath, 1994). Therefore, whilst there appear to be many possible benefits, it must be recognised that computer-mediated communication is just that: it is highly mediated. Whilst some of the recent research has taken this mediation to be beneficial, it is possible that the nature of computer-mediated communication may impact on students’  2002 Blackwell Science Ltd, Journal of Computer Assisted Learning, 18, 351-366

Learning within incoherent structures

353

learning. The study presented in this paper was designed to provide a structured analysis of students’ contributions in an online discussion forum in order to understand better the nature of online discussion. Method Participants The sample for this study consisted of 69 students enrolled at both the first and second year-level on a one semester undergraduate university course. There were 40 females and 29 males in the sample with 83% of the students under the age of 25. Of the students, 91% stated that they used a computer weekly or daily and 59% stated that they used the Internet daily or weekly. Only one student had never used a computer before and 10 students had never used the Internet. Software The online discussion forum was created with Lotus Notes and placed on a central University server and was accessible over the Internet by using any computer with an Internet connection and web-browsing software. The online discussion was designed in a generic format in order to represent the typical structure and interface features found in many similar ‘off-the-shelf’ computer-based threaded discussion tools. Each student’s contribution to the discussion was stored as a separate file in a database. The web-interface presented students’ contributions in the following structure: Discussion thread Main topic Response Response Students were able to access the online discussion at any time during the semester, either from a computer at home or office, or from the public computer laboratories within the university. Of the total activity within the discussion forum, 48% occurred from computers within the campus laboratories and the remaining 52% occurred from students’ private computers. Procedure The online discussion forum was integrated into an existing undergraduate Environmental Studies course, and designed specifically to provide a virtual learning space in which students could engage in a conversational mode of learning towards higher order learning outcomes. Three discussion themes were run consecutively through the semester for a period of approximately five weeks each. These discussion themes did not attempt to elicit specific ‘correct’ responses from students, but rather outlined the major issues students were expected to explore in their discussions. Most importantly, the themes were designed to promote student discussion and were worded to promote students’ critical reflection on issues central to the course of study. The online discussion was explicitly embedded within the programme of lectures and each theme reflected a specific segment of the course. Participation in the online discussion was compulsory for all students and contributed to their overall grade for the course. On enrolment in the course, students were given detailed instructions on the operation of the online discussion forum, including information on logging-in, navigating within the forum and step-by-step instructions on how to define a new discussion thread, create a main topic or write a  2002 Blackwell Science Ltd, Journal of Computer Assisted Learning, 18, 351-366

354 M.J.W. Thomas response. Students were introduced to the idea of an online discussion forum and given an outline of the potential learning benefits, the desired types of participation and the criteria on which the quality of students’ participation would be assessed. Students were explicitly urged to engage in interactive discussion with one another. Discussion Theme 3: Sustainability Expires 27 June In recent years policy-making and planning for ‘sustainability’ have become key activities in the effort to minimise adverse environmental impacts associated with various forms of development. Critics of the concept of ‘sustainable development’ often advocate the reverse of development, meaning substantially reducing all forms of resource consumption by First World countries and stabilising all forms of resource consumption by Third World countries at or below current levels. • What does the concept of ‘sustainability’ mean to you? • Has your thinking about sustainability changed as a result of what you have learned from this subject so far? If so, how? • How do you think we should go about the task of achieving sustainable development? Do you think it is achievable Fig. 1. An example of a discussion theme.

At the beginning of each theme one of the lecturers or tutors in the course acted as a facilitator of the discussion, encouraging students and ‘seeding’ the discussion with initial input. However, as the study was concerned with students’ interaction within the online discussion forum environment, this participation was kept to a minimum. Analysis The study adopted an evaluative methodology which sought to describe and interpret the activities of students as they undertook the learning task (House & Howe, 1999). In line with suggestions from previous research examining computer-mediated communication, a variety of information types was sought, and both quantitative and qualitative data collection and analysis techniques were employed (McAteer et al., 1997). The study used a content analysis of students’ discourse as the major form of data collection and analysis. This content analysis procedure was designed to provide quantitative measures of the quality of students’ writing and interaction. For two of the three discussion themes over the course of the semester, each student’s contribution was coded under categories which included the level of cognitive engagement with the course material, the level of critical and reflective thinking, and the level of interaction between students. The level of students’ cognitive engagement with material in the online discussion forum was established through analysis of their contributions under the Structure of Observed Learning Outcome (SOLO) taxonomy. The SOLO taxonomy, developed by Biggs & Collis (1982) provides a clear indicator of the level of a students’ cognitive engagement, through a 5-level taxonomy relating to the students’ expression of their understanding of a topic area. Each of the SOLO levels relate to a qualitatively different mode of knowledge application and represent increasing levels of conceptual complexity in students’ written work. This technique offers a criterionreferenced means of assessing the quality of student writing (Boulton-Lewis, 1994). The SOLO taxonomy has been used recently in the evaluation of technology-based learning activities and was found to be an effective tool in the analysis of student activity in the online learning environment (Jackson, 1998). The five levels of the SOLO taxonomy and their criterion are as follows:  2002 Blackwell Science Ltd, Journal of Computer Assisted Learning, 18, 351-366

Learning within incoherent structures

355

prestructural: the task is engaged but the learner is distracted or mislead by irrelevant aspects. • unistructural: the learner focuses on the relevant domain, and picks up one aspect to work with. • multistructural: the learner picks up more and more relevant or correct features, but does not integrate them. • relational: the learner integrates parts with each other, so that the whole has a coherent structure and meaning. • extended abstract: the learner generalises the structure to take in new and more abstract features, representing a higher mode of operation. The level of critical thinking undertaken by students was established using a model and taxonomy of critical thinking developed by Norris & Ennis (1989). This taxonomy has been validated for use within the online discussion forum environment (Bullen, 1997). Messages were coded according to three levels of critical thinking each of which utilise a comprehensive set of criteria to establish the following: • Low: messages in this category displayed a lack of critical thinking. • Medium: messages in this category displayed only minimal evidence of critical thinking. • High: messages in this category displayed a high level of critical thinking. The level of students’ interaction was initially established through an analysis based on a simple taxonomy of participation in computer conferences, where messages are coded as either independent or interactive (Henri, 1992). Preliminary content analysis revealed that more than a simplistic binary categorisation was required and accordingly messages were coded under four categories. • Independent: message makes no reference to other student’s messages. • Quasi-Interactive: message refers to other student’s messages, but only as a preliminary point of reference before student continues with their own isolated analysis. • Interactive (elaborative): message refers to another student’s message and further develops the theme. • Interactive (negotiating): message refers to another student’s message and engages in negotiation or debate. Qualitative data was collected from students and informed the interpretation of students’ use of the online discussion forum. At the end of the semester, a questionnaire was given to all participants in order to gather information as to their perceptions of the online discussion environment as a learning tool. The questionnaire contained open-ended questions pertaining to the positive and negative aspects of the online discussion forum, student’s perception of its benefits to their studies, and their perception of the differences between face-to-face and online discussion. Analysis of the access logs for the online discussion forum served as another form of data collection and analysis, establishing the patterns of student usage. •

Results Students’ use of the online discussion forum At the beginning of each theme students were urged in their weekly classes to access the online discussion and begin participation. Students proposed the threads as an individual response to the discussion ‘theme’ and each thread simply began as an initial posting to the discussion forum. To this end, students were able to control  2002 Blackwell Science Ltd, Journal of Computer Assisted Learning, 18, 351-366

356 M.J.W. Thomas both the structure and content of discussions with minimal input from the tutors. Each of the discussion themes was similar in size and structure, with an average of 224 individual messages, 13 discussion threads and 17 messages per thread. Analysis of the access logs provided details of the students’ use of the online forum. Distinct patterns of usage were evident in the first two themes, which were characterised by limited activity in the first weeks and a noticeable increase in activity during the final weeks. Activity during the third theme showed a similar pattern, although the increase in activity before the deadline was less pronounced. The graphs in Fig. 2 illustrate the amounts of student activity within the online discussion forum. Activity is shown in terms of hits, which denote the total reading, navigating and message composition activity, and also in terms of the number of messages posted each day. 40

1500 hits

30

1000

messages

20 500

10 0

0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 2 4 6 8 10 12 14 16 18 20 22 24 26

March

April

May

June

Fig. 2. User activity within the online discussion forum

During the first six weeks of the first theme, there were proportionately more hits relative to the number of messages. This can be interpreted as students engaging in exploratory activity. During this time students were familiarising themselves with the operation of the online discussion forum and gaining confidence in logging on and navigating the forum. Also, an amount of nervousness in contributing to the online discussion forum can be inferred, with students waiting to see a variety of other messages before contributing their own. This form of inertia appears to be difficult to overcome in an isolated and asynchronous online discussion forum. As the deadline for the first theme approached, activity within the forum increased, which shows the influence of cut-off date for the assessment period. However, as the number of messages increased, the number of hits proportionately decreased. This indicates that students were engaging in less exploratory activity within the online discussion forum and reading fewer messages before contributing. In total there were 197 messages contributed to the first theme, 70% of which were in the final two weeks. Of these 197 contributions, a total of 136 were subjected to coding under the content analysis procedure as they were posted by students who gave their consent to participate in the study. A similar pattern is evident in the second theme where there were 245 contributions, 80% of which were in the final two weeks. Again, it was evident that assessment and the sanctions associated with failure to contribute to each forum, worked as a significant form of extrinsic motivation for participation. Students’ use of the online discussion forum during the third theme was characterised by a more constant level of activity after the first two weeks. In total there were 250 contributions to this theme. Qualitative data from students suggests that the more regular level of activity was a result of pressures associated with the end of semester. The final weeks of this theme coincided with a dedicated study period and examinations. Students indicated that they contributed much earlier in this forum so that they could focus on the other elements of their studies. Accordingly, it was another  2002 Blackwell Science Ltd, Journal of Computer Assisted Learning, 18, 351-366

Learning within incoherent structures

357

form of extrinsic motivation which influenced participation in the third theme. Students’ level of cognitive engagement It was found that students achieved relatively high levels of cognitive engagement. The majority of messages in both the first and third themes were coded under levels three and four of the SOLO Taxonomy, representing the multistructural and relational categories (see Table 1). Theme Two was not analysed due to the large volume of data, and the need to establish the difference between students' initial postings and their postings as more experienced online learners. This finding suggests that students grasped the complexity of the discussion topics and were often able to form complex conceptual relationships between elements of the topic. Table 1. The distribution of messages, as a % of the total under the SOLO Taxonomy. Prestructural Theme One Theme Three

2.94 (n = 4) 0.57 (n = 1)

SOLO Taxonomy Unistructural Multistructural

Relational Extended Abstract

12.50 (n = 17) 15.43 (n = 27)

36.03 (n = 49) 24.00 (n = 42)

41.18 (n = 56) 57.14 (n = 100)

7.35 (n = 10) 2.86 (n = 5)

Previous research has shown that this predominance of multistructural and relational content is typical of the levels of cognitive engagement of university students in formal writing exercises (Boulton-Lewis, 1994; p. 390). However, Biggs & Collis, 1989) have argued that undergraduate students should ideally achieve relational and extended abstract levels in their learning activities. These levels of the taxonomy characterise writing in which students’ form integrated understandings, rather than simple conglomerates of unrelated declarative knowledge. Accordingly, the number of messages coded under the multistructural category suggests levels of cognitive engagement in the online discussion forum which are somewhat less than desirable. In the third theme it was anticipated that students’ level of cognitive engagement would increase, primarily as a result of their familiarity with the subject matter. In fact, it was found that there was a statistically significant decrease in the mean SOLO level of students’ responses (t(61) = –2.502, p = 0.015). As evident in Table 1, there was an increase in multistructural content, yet a decrease in both relational and extended abstract content. This suggests that in the third theme students were not integrating concepts related to the discussion topic, nor achieving a level of personal meaning that could be abstracted. Comparative analysis of data suggests three main factors that influenced the overall lack of relational and extended abstract writing and the apparent decrease in cognitive engagement over the semester. Firstly, the complexity and integration of students’ knowledge structures is a product of their familiarity and experience in the field. The students in this case were enrolled in an introductory subject and for many students it was the first time they had considered issues such as ‘sustainability’ and ‘social justice’. At this level students would not have extremely well developed knowledge structures, and this would be reflected in their messages, resulting in the dominance of the multistructural responses. Secondly, students reported that external pressures of increased study load and the imminent examinations impacted on their participation during the third theme. As Dart & Clarke (1991, p. 318) suggest, students are more likely to adopt strategic and surface approaches to studying when faced with such pressures as work overload and time constraints. Therefore, as the adoption of surface and strategic approaches to  2002 Blackwell Science Ltd, Journal of Computer Assisted Learning, 18, 351-366

358 M.J.W. Thomas studying results in poorer cognitive engagement, these external pressures on students can be seen to have caused the decrease in performance during the third theme. Finally, the decrease in mean SOLO level can also be accounted for, at least partially, by a change in some of the students’ discourse. Students’ familiarity with the online discussion forum and its particular mode of learning, resulted in a shift away from an overtly academic and highly structured discourse. Therefore, some students’ messages in the third theme were less like mini-essays on the discussion topic and were more familiar in their tone. Furthermore, there was an increase in short messages, where a student simply made a brief supportive comment to another, or passed on a reference. It is possible that when students engage in more interactive discourse, they are less likely to provide evidence of the complexity of their knowledge structures and are more likely to communicate in a less integrated or abstracted manner. Accordingly, the perceived decrease in cognitive engagement may be an artefact of an improvement in their use of the online discussion forum, rather than an actual decrease in the students’ quality of learning. This suggests a potential problem for the application of the SOLO Taxonomy in anything other than specifically ‘academic’ discourse. In summary, the online discussion forum did promote good levels of cognitive engagement, but students’ unfamiliarity with the field, external pressures and a possible shift towards less academic discourse resulted in a lack of conceptual complexity desired in academic writing. Students’ level of critical and reflective thinking In line with the relatively high levels of cognitive engagement, there was also evidence of high levels of critical thinking within the online discussion forum. Close to half the messages in each theme showed evidence of high levels of critical thinking. Of the remainder, the majority were coded as showing medium levels of critical thinking (Table 2). Accordingly, the online discussion forum was shown to be a learning environment which promoted students to think critically about the issues raised in the discussion themes. There was evidence of Table 2. The distribution of messages, as a % of the total under an increase in the the three levels of critical thinking. amount of critical Level of critical thinking thinking across the Low Medium High semester, but this was Theme One 16.18 (n = 22) 39.71 (n = 54) 44.12 (n = 60) not statistically signifiTheme Three 12.00 (n = 21) 38.29 (n = 67) 49.71 (n = 87) cant. However, a number of students did comment explicitly that the online discussion forum did help in the development of critical evaluation and thinking skills, as well as promoting reflection on the ideas presented by other students. This further reinforces the conclusion that the online discussion forum was a means to promote critical thinking by students. Level of student interaction The content analysis of students’ messages found that the students’ interaction increased over the course of the semester. However, as will be discussed, the quality of that interaction was often insufficient to promote the levels of interpersonal communication necessary for a truly conversational mode of learning. The data in  2002 Blackwell Science Ltd, Journal of Computer Assisted Learning, 18, 351-366

Learning within incoherent structures

359

Table 3 illustrate the distribution of messages across the four categories of content analysis related to student interaction. Table 3. The distribution of messages, as a % of the total under the four levels of interaction

Theme One Theme Three

Level of student interaction in messages Independent QuasiInteractive interactive (elaborating)

Interactive (negotiating)

32.35 (n = 44) 10.29 (n = 18)

13.97 (n = 19) 30.29 (n = 53)

21.32 (n = 29) 17.71 (n = 31)

32.35 (n = 44) 41.71 (n = 73)

In the first theme, 54% of the messages were either completely independent, or only quasi-interactive. As shown by the data in Table 3 there was a higher level of interaction in the third theme, suggesting that students came to increase the interactivity of their messages as their familiarity with the online discussion forum increased. In the third theme the amount of independent messages halved, with only 28% of the messages showing no evidence of interaction. This suggests that interactive modes of participation are themselves something which need to be learned by students. This was contrary to the initial expectations, based on the published research in the field, that the virtual learning environment would itself promote interaction between students. The increase in student interaction within the online forum over the semester can be explained with reference to the changes in students’ level of cognitive engagement as discussed above. It was apparent that students’ contributions to the online discussion shifted from academic mini-essays and became more familiar in tone. It is apparent that this shift also contributed to an increase in students’ interaction in the online discussion. This issue highlights the possibility that the attainment of a discourse that is both interactive and academic in nature is difficult within the online learning environment of the traditional threaded discussion forum. Although the level of interactivity significantly increased during the third theme, there was still a serious limitation in relation to the realisation of interactive learning within the online discussion forum. On examining each thread of discussion, rather than merely the individual messages in isolation, it became apparent that there was a lack of coherent structure in the discussion threads. Whilst there was evidence of some interaction, with students building upon or presenting arguments against other students’ contributions, there was no real co-operative development of ideas between groups of students. The conceptual incoherence of the discussion threads can be illustrated by mapping the development of typical discussion threads. Figure 3 shows the structure of three typical discussion threads. In the figure each student contribution is numbered sequentially, and data is provided on the date and time of posting, as well as the number of ‘hits’ which represents the times the message was read by another student. While each thread is unique in content and in length, a structural analysis of each thread yields a number of defining characteristics of online discussion. Firstly, it is evident that the online discussion forms a conceptually fluid structure, where there is a nonlinear development of ideas. As evident in each of the threads in Fig. 3, a threaded discussion is one which branches and becomes progressively more fragmented as it evolves. From the perspective of the content of messages, there was evidence of both repetition and duplication in students’ comments as each thread branches. Students were not united in discussion, but drawn along constantly diverging paths.  2002 Blackwell Science Ltd, Journal of Computer Assisted Learning, 18, 351-366

360 M.J.W. Thomas Psychology of Change Thread

MESSAGE ONE

MESSAGE TWO

MESSAGE FIVE

MESSAGE EIGHT

DATE: 7 April TIME: 1:02pm HITS: 56

DATE: 12 April TIME: 6:11pm HITS: 30

DATE: 15 April TIME: 10:40am HITS: 17

DATE: 16 April TIME: 6:58pm HITS: 3

MESSAGE FOUR

MESSAGE SIX

MESSAGE THREE

DATE: 14 April TIME: 3:10pm HITS: 25

DATE: 16 April TIME: 11:21am HITS: 4

DATE: 13 April TIME: 4:23pm HITS: 11

MESSAGE SEVEN DATE: 16 April TIME: 4:40pm HITS: 2

Implications of Overpopulation Thread

MESSAGE ONE

MESSAGE EIGHT

DATE: 24 March TIME: 10:58am HITS: 70

DATE: 14 April TIME: 4:56pm HITS: 14

MESSAGE TWO

MESSAGE SIX

MESSAGE NINE

MESSAGE FIFTEEN

MESSAGE TWELVE

DATE:25 March TIME:9:25am HITS: 77

DATE: 13 April TIME: 3:25pm HITS: 13

DATE: 15 April TIME: 12:35pm HITS: 3

DATE: 18 April TIME: 9:51pm HITS: 1

DATE: 17 April TIME: 10:21pm HITS: 2

MESSAGE THREE

MESSAGE 10

MESSAGE SEVEN

DATE: 1 April TIME: 2:04pm HITS: 57

DATE: 16 April TIME: 10:04pm HITS: 1

DATE: 14 April TIME: 2:30pm HITS: 20

MESSAGE FOUR

MESSAGE FIVE

MESSAGE THIRTEEN

DATE: 6 April TIME: 11:54am HITS: 41

DATE: 12 April TIME: 1:48pm HITS: 29

DATE: 18 April TIME: 1:25am HITS: 5

MESSAGE ELEVEN DATE: 16 April TIME: 10:23pm HITS: 19

MESSAGE FOURTEEN DATE: 18 April TIME: 10:31am HITS: 0

Sustainability and Ecology Thread

MESSAGE ONE

MESSAGE TEN

DATE: 02 June TIME: 7:21pm HITS: 66

DATE: 18 June TIME: 2:30am HITS: 22

MESSAGE TWO

MESSAGE FOUR

MESSAGE SEVEN

MESSAGE EIGHT

MESSAGE TWELVE

MESSAGE FIFTEEN

MESSAGE SEVENTEEN

MESSAGE ELEVEN

MESSAGE THIRTEEN

DATE: 3 June TIME: 2:20pm HITS: 19

DATE: 6 June TIME: 8:30pm HITS: 14

DATE: 9 June TIME: 8:30pm HITS: 2

DATE: 13 June TIME: 10:06pm HITS: 5

DATE: 21 June TIME: 1:22am HITS: 6

DATE: 25 June TIME: 3:50pm HITS: 2

DATE: 26 June TIME: 5:37pm HITS: 3

DATE: 18 June TIME: 5:55pm HITS: 10

DATE: 22 June TIME: 5:48pm HITS: 12

MESSAGE THREE

MESSAGE FIVE

MESSAGE SIX

MESSAGE SIXTEEN

MESSAGE NINETEEN

DATE: 4 June TIME: 7:38pm HITS: 14

DATE: 7 June TIME: 2:43pm HITS: 1

DATE: 9 June TIME: 5:04am HITS: 5

DATE: 26 June TIME: 12:41am HITS: 2

DATE: 27 June TIME: 8:12pm HITS: 1

MESSAGE NINE

MESSAGE EIGHTEEN

MESSAGE FOURTEEN

DATE: 15 June TIME: 10:33pm HITS: 1

DATE: 27 June TIME: 2:32am HITS: 1

DATE: 25 June TIME: 3:19pm HITS: 1

Fig. 3. Structural analyses of discussion threads

Secondly, there is a significant decrease in the number of times a message was read in relation to its position in the thread. Obviously, the messages that were contributed early in the ‘discussion’ were accessed a greater number of times. Further, as the thread branched, the number of times a message was read decreased  2002 Blackwell Science Ltd, Journal of Computer Assisted Learning, 18, 351-366

Learning within incoherent structures

361

rapidly, and many of the messages that terminated branches were often read less then five times. This unequal weighting of hits demonstrates that an evolving interactive discussion did not take place. Thirdly, it can be seen that there was a large proportion of messages terminating branches. In each of the discussion themes over half of students’ contributions received no response. In effect, the majority of messages were isolated, and mostly unrelated, conclusions to the one thread. Furthermore, very few students read the messages that terminated these branches. Accordingly, the students did not necessarily benefit from access to a wide variety of perspectives. What could be suggested is that they often stated their own opinions, yet did not further develop these in light of other students’ contributions. Finally, it is obvious that the discussion threads evolved over a considerable period of time. There was little on-going development and exchange of ideas in any of the discussion themes. Rather, the disjointed and fragmented individual contributions were abstracted in space and time from other students’ contributions. While on occasion a contribution was answered within a 24-hour period, the total cumulative time in the evolution of a thread was often a period of several weeks. Together, the branching structure, the large proportion of messages that terminated branches, and the abstracted nature of student interaction demonstrate an overall incoherence in online discussion. This incoherent structure of the discussion threads is not compatible with a truly conversational mode of learning. From this analysis it is evident that the virtual learning space of the online discussion forum does not promote the interactive dialogue of conversation, but rather leads students towards poorly interrelated monologues. When asked to evaluate the relative merits of the online discussion forum compared to face-to-face discussions, the forum was criticised by students for being disjointed, stilted and less spontaneous. The majority of students found the online discussion forum a worthwhile element of their studies, but only 51% of students stated that they enjoyed using the facility. Similarly, only 52% of students stated that they would like to participate in an online forum again. Most importantly, when asked whether they found the forum better than a face-to-face tutorial discussion, only 14% agreed. When asked to evaluate the relative merits of the discussion forum compared to face-to-face discussions, 51% of students suggested that face-to-face discussions allowed for more interaction, more feedback and more personal modes of learning than could be offered by a forum. Indeed, many students questioned whether the forum actually allowed for discussion. Students found the discussion forum to be far less immediate and interactive, more time consuming and more difficult. As one student said, capturing the essence of the online discussion forum: In tutorials the discussion is much more alive and direct. My ideas can be changed, influenced and appreciated in a more integrated environment. The online discussion forum felt too much like monologue vs. monologue. It needs to be a discussion.

Discussion and conclusion The findings of this study demonstrate that while online discussion forums promoted high levels of cognitive engagement and critical thinking, the virtual learning space of an online forum did not promote the coherent and interactive dialogue necessary for conversational modes of learning. This finding can be elaborated upon by examining the typical patterns of interaction expected in a face-to-face discussion.  2002 Blackwell Science Ltd, Journal of Computer Assisted Learning, 18, 351-366

362 M.J.W. Thomas As Pincas (1998; p. 14) has stated, rather intuitively: for effective collaborative learning to take place in a virtual learning environment, students need to engage in what they can perceive as normal discussion. This study has demonstrated that such ‘normal discussion’ did not occur. Three major factors can be suggested as influencing this lack of discussion: the isolated mode of participation, the structural organisation of messages, and the conflict between the written form and oral function of technology-mediated interpersonal communication. Firstly, the lack of discussion within the virtual learning space can be seen to be a product of the asynchronous and isolated mode of participation. Students never metaphorically ‘came together’ to learn, but rather they were isolated by the technology which offered several levels of abstraction from normal discussion. In the virtual learning space, students interact not with another student, but with another student’s writing. Further, this writing is removed in space and time from the parties involved in the discussion. Each message is placed in an external database and is thus effectively severed from dialogue. A significant proportion of the messages that were ‘submitted’ to the discussion forum were never viewed by another person. Therefore, it would be more appropriate to conceptualise students’ messages as data stored for potential access by other students, rather than contributions to an on-going dialogue. In this way the online discussion forum promoted an individualistic mode of learning rather than an interactive mode. A second means by which discussion is inhibited involves the structural organisation of the messages. As analysed above, the online discussion forum places students’ contributions in a structure that lacks the cohesion of a normal face-to-face discussion. From one ‘main topic’ the misnamed ‘discussion’ branches endlessly, until no more responses are submitted. It seems rather inappropriate, from the perspective of a conversational model of education, that this amorphous structure of a discussion thread has been taken to benefit interactive learning in much of the recent research. As Shank & Cunningham (1996; p. 30) suggest, a typical discussion thread is: . . . a series of computer-generated communications that start on a particular topic, but that might lead in many different directions before the discussion is done. Once a thread has been started though, it is no longer under sender control. This is because the mechanics of Internet response do not require turn taking. From the oral side, it is as if everyone who is interested in talking can all jump in at once, but still their individual voices can be clearly heard.

According to Shank & Cunningham, and many other proponents of technologymediated communication, this structure of online discussion provides a democratic and egalitarian mode of participation, as everyone’s voice can be heard and is given equal standing. However, the findings of this study conflict with this interpretation of online discussion. The number of individual voices which were ‘clearly heard’ as part of a discussion in the online discussion forum was small. As contributions to a database, each student’s messages had the potential to be given equal standing in an egalitarian discussion. However, as many messages in the online discussion forum received no response, and were not interactive in any way, the number of individuals’ voices which could be said to have been ‘heard’ was limited. As students’ participation in the online forum was characterised more by the insertion of their opinion into a illstructured collection of other students’ messages, the possibility of interactive dialogue was limited.  2002 Blackwell Science Ltd, Journal of Computer Assisted Learning, 18, 351-366

Learning within incoherent structures

363

A third means by which a rich form of discussion was inhibited in the virtual learning space pertains to the conflict between the written form and oral function of technology-mediated communication. Written discourse has developed in a highly specialised manner, and is different in both form and function to the spoken language of face-to–face interaction. To call again upon current understanding of language and discourse, face-to-face discourse is fundamentally interactional in nature, while written discourse is generally transactional in nature. Accordingly, the text-based medium of the online discussion forum does not simply present a technologymediated alternative to face-to-face dialogue. Rather, it attempts to serve the function of face-to–face interaction through the sole use of the written form. The fact that students felt they could not effectively communicate with each other through the text-based medium of the online forum, and the obvious lack of interaction in their messages, suggests that some major difficulties arise from this conflict between form and function. These difficulties have potential ramifications for student learning. As Postman (1992; p. 17) suggests, the introduction of technology into the realm of education breaks the truce between orality and the printed word. He asks: Orality stresses group learning, cooperation and a sense of social responsibility. . . Print stresses individualised learning, competition and personal autonomy. Over four centuries, teachers, while emphasising print, have allowed orality its place in the classroom and have therefore achieved a kind of pedagogical peace between these two forms of learning. Now comes the computer, carrying a new banner of private learning and individual problem solving. Will the widespread use of computers in the classroom defeat once and for all the claims of communal speech?

In face-to-face group discussions the learners share a common focus. The contributors in group discussions generally interact with the group as a whole, and most students participate in the one developing discussion. While this discussion may shift in emphasis or topic, students are engaged together in the development of a shared understanding. Although messages in an online discussion forum might appear to be interactive, in as much as they make reference to a previous message, the branching structure of threads promotes an incoherent development of ideas amongst the group of students. Individual students do not engage with a group, but with the isolated contributions of individual students. The important role of a tutor or moderator in an online discussion cannot be underestimated. This study has demonstrated that the rich interactive discussion of the tutorial environment does not necessarily come naturally to students as they work in a virtual learning environment. The facilitation of discussion must be a focus of further research and the mechanisms by which instructors are able to assist groups of students in creating vibrant online discussion made explicit. Recently, Hara et al. (2000) have discussed one method to enhance the structure of online discussion called the ‘starter-wrapper’ technique. This technique attempts to increase the coherence of online discussion by assigning individual participants specific roles in the discussion. Typically on a rotating basis, one participant is assigned the role of ‘starter’ which involves responsibility for the initial development of online discussion. Similarly, another participant is assigned the role of the ‘wrapper’ and is responsible for reading all the postings on a particular theme, summarising key aspects of the discussion and providing a synthesis of the debate. While typically performed by students, the starter-wrapper technique can also be used by a tutor or moderator as a means of shaping their participation in online discussion.

 2002 Blackwell Science Ltd, Journal of Computer Assisted Learning, 18, 351-366

364 M.J.W. Thomas This simple technique has significant potential to overcome some of the issues involved in the incoherent structure of online discussion at the macro level. However, from the perspective of individual student’s developing their own personal understanding of a broad topic, problems still exist. The starter-wrapper technique enables some gestalt of discussion to emerge. However, this gestalt is only one student’s interpretation of the debate and discussion in a topic area. In the synchronous multifaceted exchange of the traditional tutorial each student is exposed to a multitude of perspectives, and each develops their own understanding through an active process of information interpretation and evaluation. The starter-wrapper technique only promotes this type of analysis in one participant, and does not necessarily eliminate the fragmentation and inherent incoherence embedded within the traditional structure of the threaded discussion list. Perhaps the most promising method by which to tackle the problem of structure in online discussion involves the development of innovative supportive interfaces and redesigned curricula that are designed specifically for collaborative knowledge building. While this research suggests that the current design features of online discussion forums inhibit truly conversational modes of learning, possibilities exist to improve the virtual learning space. The challenge is for interface design which promotes a more coherent structure and true many-to–many interaction in the virtual learning space. Through innovative graphic representation of student contributions a more authentic group discussion may be able to occur online. One possibility could be the construction of a three dimensional ‘concept-map’ of contributions to the discussion forum which may provide unique opportunities for students’ development of an on-going discussion. With such a system the potential also exists for students to understand better the complex relationships which exist in many academic knowledge domains, by exploring the relationships between students’ contributions. Recently, a prototype system called ConverSpace has been developed which utilises such an innovative interface to assist students in the collaborative learning process (Popolov et al., 2002). Another approach entails the development of radically different tools for online collaboration that extend beyond the task of online discussion. While the online discussion forum has become a ubiquitous element of Internet-supported flexible delivery of education, it is apparent that it might not be the best technology to support the interactive and collaborative processes essential to a conversational model of learning. These new developments must involve the redesign of both the technological support tools and curriculum structures to support collaborative learning processes. Accordingly, such innovation would emphasise the implementation of learning tasks that promote collaborative engagement towards knowledge development and problem solving. It is perhaps this route that may prove to be the most productive means of realising truly conversational modes of learning, given the inherent problems involved in traditional online discussion. Acknowledgements Data collection and preliminary analysis for this research was undertaken in the Department of Geographical and Environmental Studies at the University of Adelaide. Thanks go to the reviewers and Editor of JCAL for their detailed and insightful remarks on successive drafts of this paper.

 2002 Blackwell Science Ltd, Journal of Computer Assisted Learning, 18, 351-366

Learning within incoherent structures

365

References Bates, A.W. (1995) Technology, Open Learning and Distance Education. Routledge, London. Biggs, J. & Collis, K. (1982) Evaluating the Quality of Learning: the SOLO Taxonomy. Academic Press, New York. Blumenfeld, P.C., Marx, R.W., Soloway, E. & Krajcik, J. (1996) Learning with Peers: From Small Group Cooperation to Collaborative Communities. Educational Researcher, 25, 8, 37-42. Boulton-Lewis, G. (1994) Tertiary Student’s Knowledge of their Own Learning and a SOLO Taxonomy. Higher Education, 28, 3, 387-402. Bullen, M. (1997) A Case Study of Participation and Critical Thinking in a University level Course Delivered by Computer Conferencing. Unpublished Doctoral Dissertation, University of British Columbia, Vancouver. Dart, B.C. & Clarke, J.A. (1991) Helping Students Become Better Learners: A Case Study in Teacher Education. Higher Education, 22, 3, 317-335. Dehler, C. & Parras-Hernandez, L.H. (1998) Using Computer-Mediated Communication (CMC) to Promote Experiential Learning in Graduate Studies. Educational Technology, 38, 3, 52-55. Flynn, E.W. & La Faso, J.F. (1972) Group Discussion as a Learning Process. Paulist Press, New York. Greening, T. (1998) Building the Constructivist Toolbox: An Exploration of Cognitive Technologies. Educational Technology, 37, 2, 23-35. Hara, N., Bonk, C.J. & Angeli, C. (2000) Content analyses of online discussion in an applied educational psychology course. Instructional Science, 28, 2, 115-152. Harasim, L.M. (1990) Online Education: An Environment for Collaboration and Intellectual Amplification. In Online Education: Perspectives on a New Environment. (ed. L.M. Harasim) pp. 39-64. Preager, New York. Harrington, H. (1992) Fostering Critical Reflection through Technology: Preparing Prospective Teachers for a Changing Society. Journal of Information Technology for Teacher Education, 1, 1, 67-82. Henri, F. (1992) Computer Conferencing and Content Analysis. In Collaborative Learning Through Computer Conferencing. (ed. A. Kaye) pp. 117-136. Springer-Verlag, London. House, E.R. & Howe, K.R. (1999) Values in Evaluations and Social Research. Sage Publications, Thousand Oaks, CA. Jackson, B. (1998) Evaluation of Learning Technology Implementation. In Evaluation Studies. (ed. N. Mogey) pp. 117-136. Learning Technology Dissemination Initiative, Edinburgh.(http://www.icbl.hw.ac.uk/ltdi/evalstudies/cont.htm) Koschmann, T.D., Mayers, A.C., Feltovich, P.J. & Barrows, B.S. (1994) Using Technology to Assist in Realizing Effective Learning and Instruction: a Principled Approach to the use of Computers in Collaborative Instruction. Journal of the Learning Sciences, 3, 3, 227264. Laurillard, D. (1993) Rethinking University Teaching: A Framework for the Effective Use of Educational Technology. Routledge, London. Laurillard, D. (1999) A Conversational Framework for Individual Learning Applied to the 'Learning Organisation' and the 'Learning Society'. Systems Research and Behavioural Science 16, 113-122. Littleton, K. & Häkkinen, P. (1999) Learning Together: Understanding the Processes of Computer-Based Collaborative Learning. In Collaborative Learning: Cognitive and Computational Approaches. (ed. P. Dillenbourg) pp. 20-29. Pergamon, Amsterdam. McAteer, E., Tolmie, A., Duffy, C. & Corbett, J. (1997) Computer-Mediated Communication as a Learning Resource. Journal of Computer Assisted Learning 13, 4, 219-227.

 2002 Blackwell Science Ltd, Journal of Computer Assisted Learning, 18, 351-366

366 M.J.W. Thomas Norris, S.P. & Ennis, R.H. (1989) Evaluating Critical Thinking. Critical Thinking Press & Software, Pacific Grove, CA. Pincas, A. (1998) Successful Online Course Design: Virtual Frameworks for Discourse Construction. Educational Technology and Society, 1, 1, 14-25. Popolov, D., Callaghan, M. & Luker, P. (2002) Tying models of learning to design of collaborative learning software tools. Journal of Computer Assisted Learning, 18, 1, 4647. Postman, N. (1992) Technopoly. Vintage Books, New York. Rowntree, D. (1995) Teaching and Learning Online: a Correspondence Education for the 21st Century? British Journal of Educational Technology, 26, 3, 205-215. Ruberg, L.F., Moore, D.M. & Taylor, C.D. (1996) Student Participation. Interaction, and Regulation in a Computer-Mediated Communication Environment: a Qualitative Study. Journal of Educational Computer Research, 14, 3, 243-268. Shank, G. & Cunningham, D. (1996) Mediated Phosphor Dots. Toward a Post-Cartesian Model of Computer-Mediated Communication Via the Semiotic Superhighway. In Philosophical Perspectives on Computer-Mediated Communication (ed. C. Ess.) pp. 2741. State University of New York Press, Albany, NY. Short, J., Williams, E. & Christie, B. (1976) The Social Psychology of Telecommunications. Wiley, New York. Sproull, L. & Keisler, S. (1991) Connections: New Ways of Working in the Networked Organisation. MIT Press, Cambridge, MA. Straus, S.G. & McGrath, J.E. (1994) Does the Medium Matter? The Interaction of Task Type and Technology on Group Performance and Member Reactions. Journal of Applied Psychology, 79, 1, 87-97. Warschauer, M. (1997) Computer-Mediated Collaborative Learning: Theory and Practice. Modern Language Journal, 81, iv, 470-481. Willis, J. (1991) Computer Mediated Communication Systems and Intellectual Teamwork: Social Psychological Issues in Design and Implementation. Educational Technology, 31, 4, 10-20.

Forthcoming JCAL papers: Correcting computer-based assessments for guessing:

R. Harper, University of Luton

An experiment on scientific discovery learning in computer simulations: D.J. Reid, J. Zhang & Q. Chen,,Universities of Manchester, Tsinghua & Beijing Normal

The effect of metacognitive training on the mathematical word problem solving of S.K. Teong, National Institute of Education, Singapore low achievers: School experience course with multimedia in teacher education: an example from A. Asan, Karadeniz Technical University, Turkey Turkey: Using computer algebra systems in mathematical classrooms: B. Kramarski & C. Hirsch, Bar- Ilan University, Israel

 2002 Blackwell Science Ltd, Journal of Computer Assisted Learning, 18, 351-366