Jack Burston
Exploiting the Potential of a ComputerBased Grammar Checker in Conjunction with Self-Monitoring Strategies with Advanced Level Students of French* Jack Burston Temple University ABSTRACT The purpose of this paper is to report on a study of grammar checker usage which surveys the steps taken to successfully integrate it into the curriculum of an advanced level French course. In so doing, the pedagogical limits of computer-based grammar checking are identified, practical applications explored, short-term and long-term effectiveness measured, and the role of self-monitoring for grammatical accuracy considered.
KEYWORDS Grammar Checker, Errors, Focus on Form, Input Processing, French
INTRODUCTION Of the most commonly taught languages, including English as a Second Language (ESL) as well as foreign languages, French stands head and shoulders above the rest in both the number and quality of grammar checkers available for it. Indeed, the best French grammar checkers are currently at the cutting edge of what is technologically possible. Notwith*A preliminary version of the research upon which this article is based was first presented at the 1999 CALICO symposium at Miami University in Oxford, Ohio. The author wishes to publicly acknowledge his appreciation of the Australian National Teaching Development Grant (CUTSD) and the School of European Languages and Cultures (Monash University) whose funding made this teaching project possible. © 2001 CALICO Journal
Volume 18 Number 3
499
Computer-Based Grammar Checker and Self-Monitoring standing, unlike the widely encouraged use of word processors, the use of French grammar checkers within the Français Langue Etrangère (FLE) curriculum is much less common and is even actively discouraged by some instructors. Likewise, while most students of French would consider access to a word processor a definite advantage when writing essays, the same cannot be said about the use of a grammar checker. The limited acceptance of grammar checkers in the FLE context is in large part the result of the lack of appeal of their English language counterparts, which have a well established reputation of being of little use to native speakers and even less to ESL students (Brock, 1990; Knowles, 1992; Levy, 1992). A second, contrary, reason for resistance to their usage is the concern that it could be pedagogically counterproductive, encouraging language learners to rely on the computer to take care of grammatical details. In reality, even the most advanced French grammar checkers require considerable user intervention to work effectively, a feature which both students and teachers alike can find off putting.
THE USE OF CORRECTEUR 101 AT MONASH UNIVERSITY Interest in the exploitation of French grammar checker technology began at Monash University in 1995 in response to a comprehensive evaluation of available products which revealed them to be far more suitable to student usage than anything produced for English (Burston, 1996). This usefulness is, however, subject to limits imposed by computer-based natural language processing. Current grammar checkers, for French or any other language, are restricted in their operation by their inability to perform semantic analysis. They function almost entirely on the basis of surface level morphosyntactic relationships. For example, even the most sophisticated French grammar checker is incapable of detecting basic errors in determiner usage (e.g., definite vs. partitive articles), anaphoric reference (e.g., demonstrative vs. personal pronouns), or verb tense (e.g., imperfect vs. passé composé). While some warnings can be given about the use of homonyms, regionalisms, or anglicisms, any general determination of lexical appropriateness is impossible. Nonetheless, within these operational constraints, one product in particular, Le Correcteur 101, proved to be especially good at detecting and correcting fautes bêtes ‘low level errors,’ that is, spelling (especially accents), noun gender, adjective/ past participle agreement, subject/verb agreement, verb conjugation, and incomplete negations. Because these kinds of errors occur with such regularity in the written work of even advanced level language learners, the prospect of exploiting a grammar checker to ferret them out led to the introduction of Le Correcteur 101 in second year (semester 7-8 equivalent) French language classes in 1996. 500
CALICO Journal
Jack Burston Initially, students were given two training sessions on the operation of Le Correcteur 101 and encouraged to use it to proofread their compositions before submitting them. Such unsupervised ad hoc usage had little discernible effect on the grammatical accuracy of student work. On the basis of this experience it became obvious that, for its pedagogical potential to be realised, the use of Le Correcteur 101 needed to be better integrated into the curriculum. The following year, correction exercises, based on actual student compositions, were prepared and added to the grammar checker training program. This prompted greater use of the grammar checker in the lab, and it was immediately apparent to instructors which compositions had been checked before submission and which had not. Voluntary usage of Le Correcteur 101, however, remained the exception rather than the rule and, disappointingly, it was the most error prone students who made the least (or least effective) use of it. Of equal concern, improvements in grammatical accuracy evidenced in compositions submitted as homework assignments did not carry through to essays written under strict examination conditions, that is, where no grammar books or dictionaries were permitted. In 1998, the effort to develop practical remediation strategies to address the problem of persistent low level errors in the written work of advanced learners was combined with a formal investigation into the exact nature of the phenomena and its causes. An analysis of over 200 previous student exam essays was undertaken, the results of which revealed that more than two thirds of all mistakes were surface level morphosyntactic errors of the type detectable by Le Correcteur 101 (Mogilevski & Burston, 1999). Further research with an experimental group (n = 12) demonstrated the extent to which students were quite literally blind to basic spelling and grammatical mistakes. Working with compositions derived from the examination database, which had been corrected to eliminate syntactic and semantic errors leaving only morphosyntactic mistakes, participants in the experimental group were on average initially only able to achieve a 40% correction rate. When subsequently presented with the same composition, this time with all mistakes underlined in advance, the correction rate more than doubled. The disparity between the failure of language learners to detect errors on their own, compared to their performance in correcting mistakes once these were made salient, indicated clearly that the key to improving morphosyntactic accuracy in the written work of students lay in training them to systematically monitor critical surface features. A series of exercises was designed to target recognition of the highest frequency grammatical errors, that is, those involving various forms of agreement, verb conjugation, and negation. These exercises were piloted with the experimental group during the second semester of 1998, at the end of which students were required to write an in-class composition under examinaVolume 18 Number 3
501
Computer-Based Grammar Checker and Self-Monitoring tion conditions. Analysis of these compositions revealed the nearly total absence of most of the targeted error types. The same level of performance was observed in the examination essays later written by this group. Grammar checker usage in the curriculum continued throughout 1998 very much as it had the previous year, with one important difference. The assessment parameters for all major written work (i.e., 250-300 word discursive essays) were changed to allocate 20% of the grade to morphosyntactic accuracy. Each low level error in an essay, up to a maximum of 20, reduced the percentage result by one point. Though the use of Le Correcteur 101 was not obligatory, the weighting thus accorded to formal precision acted as a strong incentive to utilize it. The outcome of those who did so was encouraging, with low level errors reduced by about half compared to those who did not. Still, with the exception of the experimental group, only a slight improvement was noted in the morphosyntactic accuracy of examination essays.
LESSONS LEARNED The insights derived from the research undertaken in 1998, and experience gained from the assessment procedures implemented, allowed some conclusions to be drawn regarding the persistence of low level errors in the written work of advanced level students and the contribution which a grammar checker could make to their remediation. First, extrinsic reward plays an important role in the motivation to eliminate morphosyntactic errors and, hence, to use a grammar checker. Unless sufficient weighting is given to formal precision in the assessment of essays, students are disinclined to make the effort to systematically check their work for errors. Motivation alone is not, however, the only factor influencing the effective use of a grammar checker. Whatever their motivation, most students were unable to fully exploit the resources of Le Correcteur 101 without guidance. Further, it is precisely the language learners most in need of improving the formal accuracy of their writing who, left to their own devices, derived the least benefit from its use. The reason for the minimal benefit that these learners obtained from the grammar checker involved the multiple errors they made in sentences. The computer algorithms underlying any grammar checker rely heavily on the assumption of conformity with native speaker syntax. Successful error detection is critically sensitive to the degree of error saturation and the soundness of underlying sentence structure. Compound mistakes make it very difficult, if not impossible, to determine the base forms to use for syntactic analysis, without which even the best French grammar checkers can do little more than spell check. 502
CALICO Journal
Jack Burston With student compositions, deviations from the syntactic norms very frequently result in incomplete analyses. When incomplete analyses occur, the usual computer response is to inform the user that only a partial analysis was performed and that no grammatical errors could be identified. To the unwary and inattentive student, this message equates to an “all clear.” In fact, just the opposite is almost always true. Recognition of the warning conveyed by a “partial analysis” message was thus the first step in improving the effectiveness of grammar checker usage. The second step was to teach students to selectively check subparts of a sentence for correctness. One of the strong points of the version of Le Correcteur 101 in use (Version 3 Pro) was its ability to work with sentence fragments. All students needed to do was highlight a selected string and begin the normal verification process. Since the likelihood of completing a syntactic analysis varies in inverse proportion to the number of mistakes in a sentence, every error corrected by fragment analysis improves the ability of a grammar checker to detect remaining problems. As attested by the continuing high level of morphosyntactic errors in examination essays, more effective use of a grammar checker does not in itself lead to improvement in self-monitored writing. The reason for this problem derives from the fact that, when used as an ex post facto correction tool, a grammar checker at best only alleviates the symptoms of a learning difficulty; it does not affect the underlying source of the problem. As the work of the experimental group demonstrated, inattention to orthographic and morphological detail is not simply a matter of neglect, or low motivation, but stems in large measure from the intrinsic lack of the saliency of these features to students. Such findings are consistent with input processing theory (Van Patten, 1996), which posits that language learners attend to the transmission of meaning first. Concomitantly, they only pay attention to formal linguistic features to the extent they have surplus language processing capabilities and motivation to do so. In the case of our students, it was difficult to avoid the conclusion that long years of exposure to communicative methodologies and a primary emphasis on “getting the message across” had deeply ingrained in them a disregard for orthographic and morphological precision.
CURRENT STRATEGIES Strategies to improve formal accuracy in the writing of advanced level students of French led to significant changes in pedagogical approach in 1999 which aimed to increase the saliency of morphosyntactic features through fuller integration of grammar checker usage into class activities and greater focus on active intervention in the writing process itself. Volume 18 Number 3
503
Computer-Based Grammar Checker and Self-Monitoring On-going evaluation of French grammar checkers resulted in the introduction of a new product, Antidote 98, into the curriculum. Antidote was selected as a replacement for Le Correcteur 101 because of its overall superior performance, simpler user interface, and inclusion of an extensive on-line dictionary and reference grammar. In order to more closely supervise student writing, normal tutorial groups (20-22 students each) are divided in half one hour per week (out of three class hours per week) and are taught in computer-equipped language labs where students learn how to use Antidote, do text correction exercises, and then work on major writing assignments under the guidance of their instructor. All compositions must be word processed and drafted in two stages, each of which is submitted for separate assessment. The first draft of each essay is produced out of class and is worth 80% of the mark. Extrinsic motivation continues to be provided by the “carrot and stick” approach to formal accuracy, that is, 20% of the first draft mark (equal to 16% of the total mark for the composition) is directly determined by the number of low level errors. The assessment weighting of first drafts is intended to encourage students to submit essays which, though far from perfect, are as devoid as possible of self-correctable mistakes. First drafts are returned to students a week after submission with two types of markups, residual low level errors are highlighted and other difficulties (maximum of 20) are underlined without comment. The differential nature of this markup procedure is specifically intended to focus attention on the distinction between mistakes and errors (Corder, 1981). Morphosyntactic mistakes are not annotated precisely because they entail performance problems as opposed to comprehension difficulties. Students know very well what is wrong and are quite capable of making corrections without explanation. In the case of syntactic and semantic errors, simple underlining is motivated by several considerations, theoretical as well as pragmatic. First, the absence of comments obliges students to analyse their writing to discover what the problem is and prepares the groundwork for attentive reception of instruction. Second, minimal correction by the instructor has the advantage of not obscuring the student’s text with a flood of red ink and thus helps to keep the “affective filter” down. Equally important, it also greatly reduces the amount of time needed to correct first drafts, which is critical since second corrections would otherwise increase the teacher’s workload beyond feasible limits. The restriction of identifying a maximum of 20 errors is likewise designed to keep marking workloads within reason. However, since our students’ essays at this level contain on average between 30 and 40 syntactic/semantic errors, it also requires the teacher to focus on the most systematic difficulties. So, too, limiting the number of error identifications increases the saliency of the problems which receive attention. Though mistakes and errors are indicated without annotation, first drafts are not devoid of comment. To prevent composition 504
CALICO Journal
Jack Burston correction from becoming a totally negative experience, a special point is made to provide positive feedback on content, discourse structure, and vocabulary usage. Marked-up first drafts form the basis of writing activities in the computer lab tutorials. Sessions begin with an examination of morphosyntactic inaccuracies which escaped correction in the first draft. In part, this initial review involves teaching students how to make more effective use of the grammar checker (e.g., doing fragment analyses and consulting the online dictionary and verb conjugator). It also entails teaching language learners to recognize the limits of computer-based composition correction. In particular, grammar checkers are better at detecting errors than in identifying their source, so students need to develop the habit of carefully considering proposed changes before accepting them. The head-on “focus on forms” methodology (i.e., directed correction) used to attack basic spelling and grammar mistakes is complemented by a “focus on form” approach (Doughty & Williams, 1998) when dealing with the correction of syntactic and semantic errors in student essays. Concentrating on the intended meaning of troublesome passages, the instructor’s task is to guide students to appropriate resources for help. The Antidote program itself contains an on-line monolingual French dictionary and synonym look up. Students also have available to them the on-line Grand Robert Electronique and French On-CALL grammar review program as well as the prescribed course reference grammar and printed Le Nouveau Petit Robert dictionary. Since the instructor previews students’ first drafts in advance of class, it is possible to discuss with the whole group (in French) problems common to a number of compositions. Students are given a week to individually complete the revision of their first draft, which is corrected using a standardized code and whatever individual feedback is required.
1999 RESULTS Our approach to the problem of morphosyntactic errors was evaluated relative to two critical parameters: student attitudes and practical effectiveness. Student attitudes were gauged throughout the year by observation of grammar checker usage patterns and informal conversations with students. In addition, a formal student questionnaire relating to error correction and grammar checker usage was completed during the second half of the subject. Practical effectiveness was evaluated through a detailed analysis of student compositions and examination essays written during both semesters of the course. The 1999 student cohort numbered 54 in the first semester and 40 in the second. Volume 18 Number 3
505
Computer-Based Grammar Checker and Self-Monitoring With respect to error correction, learner attitudes towards morphosyntactic accuracy did not change overnight. Despite knowing that a substantial percentage of their course results hinged on grammatical precision, less than half of the class (49%) used Antidote to verify their first composition. Conversations with students revealed a number of different reasons for not using the grammar checker. Some students just could not be bothered. A few, quite good students, felt they could manage on their own. Many students decided on the basis of class correction exercises that using Antidote demanded too much effort. Whatever the initial reasons for letting pass low level errors, the 16% weighting for grammatical precision in compositions proved too substantial to be ignored, and usage of Antidote on the second and third compositions in the first semester increased to 67% and 87%, respectively. Grammar checker usage during the second semester was virtually 100%. The attitude questionnaire (n = 33) given after a semester’s use of Antidote revealed that 79% of the students found the program easy to use, 9% were neutral, and 12% thought it was difficult. The major criticisms of the software related to its perceived under correction and over correction. 42% of students complained that Antidote allowed mistakes to slip through undetected, while 39% took it to task for flagging errors where none existed. Significantly, this perception was very much at variance with comprehensive tests done with Antidote (Burston, 1998) on previous student compositions in the course which demonstrated that only 7% of morphosyntactic errors went undetected and that just 2% of correct expressions were falsely identified as erroneous. Likewise, close inspection of first semester essays which contained high numbers of low level errors, written by students who claimed to have used Antidote, showed that in the hands of the instructor the grammar checker in fact detected nearly all uncorrected morphosyntactic mistakes. Two explanations for this discrepancy suggest themselves. The first, as previously observed, is that students presupposed that partially analysed sentences, in which no mistakes were identified, were error-free. Second, despite the effort to explain the limits of grammar checker correction, many students apparently still expected Antidote to correct semantic and syntactic ill-formedness. As far as the perception of frequent false correction is concerned, student comments in the questionnaire relating to the time and effort required to use the grammar checker helped to shed light on the most probable cause of the complaint. By default, whenever Antidote has any hesitation in determining the correctness of a sentence, it underlines dubious portions and invites the user to verify whether or not they are correct. Many students reported finding this procedure time consuming and, upon determining that nothing was wrong, viewed these warnings as false corrections. As the discrepancies between student attitudes towards the accuracy of Antidote and objective measures of its correcting ability demonstrate, 506
CALICO Journal
Jack Burston perceptions of the usefulness of Antidote are very much influenced by expectations of what a grammar checker can do. Not surprisingly, the greater the expectations, the greater the dissatisfaction with its limitations, and the less inclined students are to use it voluntarily. Without exception, students who expressed the most positive attitudes towards Antidote were those who recognized that its usefulness was limited to ferreting out low level errors and accepted that this process required their active intervention. Student attitudes towards the long term benefit of grammar checker usage, that is, its effect upon their ability to independently detect and correct errors, was equally divided between those who perceived an improvement (39%) and those who did not (39%). Of those reporting an improvement, about a third associated the improvement with greater grammatical accuracy in their writing, while the remaining two thirds indicated only an increased awareness of the need to systematically check for low level errors. Objective measurements of the effectiveness of the use of Antidote in improving morphosyntactic accuracy in assigned compositions were overwhelmingly positive. Students who used Antidote during the first semester to proofread their first essay scored on average 72% (14.3/20), compared to only 20% (4/20) for those who went it alone. The results of the second essay showed an increase in basic grammatical accuracy to 85% (16.9/20) for Antidote users compared to only 54% (10.7/20) for the rest of the class. It should be noted that although well below the results obtained with the aid of a grammar checker, the morphosyntactic accuracy of students who relied on their own devices more than doubled between their first and second essays. Since effective self-monitoring is the ultimate goal of grammatical precision, this outcome was not unwelcome. The morphosyntactic accuracy of the third composition, which was intrinsically a more difficult assignment, was slightly lower than that of the previous essay, 82% (16.3/20) for those who had used Antidote but substantially lower, 44% (8.7/20), for those who had not. While considerable progress was thus observed during the first semester both in the usage rate of Antidote and the corresponding degree of morphosyntactical precision in assigned compositions, analysis of first semester exam essays appeared to show no improvement in the average number of low level errors: 8.6 per 100 words, compared to 8.5 per 100 words for a composition on an identical exam topic in 1997. Median scores were in fact worse: 8.1 per 100 words in 1999 compared to 7.6 in 1997. Closer inspection of these results, however, revealed a quite distinct distribution of scores (see Figure 1).
Volume 18 Number 3
507
Computer-Based Grammar Checker and Self-Monitoring Figure 1 Comparison of Error Rate Per 100 Words in Exam Topic in 1997 and 1999 Histogram 1999 Exam (n = 49) Average = 8.6 Median = 8.1 Standard Deviation = 3.6
16
16
14
14
12
12
Frequency 8 6
10 Frequency 8 6
4
4
2
2
More
Errors per 100 words
16.4
17.6
14.2
14.4
9.8
11.2
12.0
8.0
7.6
4.7
3.2
1.5
5.4
0
0
More
10
Frequency
Frequency
Histogram 1997 Exam (n =43) Average = 8.5 Median = 7.6 Standard Deviation = 4.4
Errors per 100 words
Two noticeable differences were immediately apparent. First, the 1997 cohort included a few exceptionally good students whose grammatical error rates were well below the best students in 1999. Second, the spread of scores, as reflected in the standard deviation, was substantially greater in 1997 than 1999. When compared against 1997 results normalized by the removal of exceptional students (Figure 2), slight improvements are observable in 1999 (see Figure 2). Figure 2 Comparison of Normalized Error Rate Per 100 Words in Exam Topic in 1997 and 1999
Histogram 1997 Exam (Normalized, n= 39) Average = 9.2 Median = 8.6 Standard Deviation = 4.1
Histogram 1999 Exam (n = 49) Average = 8.6 Median = 8.1 Standard Deviation = 3.6
14
16
12
14 12 Frequency
8 6
Frequency
Frequency
10
10 Frequency 8 6
4
4 2
16.4
More
14.2
17.9
12.0
15.1
9.8
12.2
7.6
9.4
5.4
6.5
3.2
0 3.6
More
2
0
Errors per 100 words
Errors per 100 words
508
CALICO Journal
Jack Burston Average errors per 100 words drops from 9.2 to 8.6, and the median from 8.6 to 8.1, and the standard deviation decreases from 4.1 to 3.6. Also, as is evident from the 1999 exam histogram, there is a marked shifting in the direction of fewer mistakes in the most error prone third of the group. Although encouraging, the improvements noted in morphosyntactic accuracy under examination conditions where much less than those observed in submitted homework assignments during the first semester. Grammar checker usage demonstrably made students more aware of their mistakes and helped eliminate errors from assigned compositions, but it appeared to have little long term effect on self monitoring. To promote more effective self-monitoring, a detailed quantitative analysis of first semester exam essays was undertaken to determine which low level errors accounted for the greatest number of morphosyntactic mistakes. This analysis was accompanied by a qualitative evaluation of the relative ease or difficulty involved in correcting these problems. Error analysis showed that, of a total 1,927 errors, 956 (50%) were purely morphological or involved only the most basic concordance relationships (see Figure 3). Figure 3 Quantitative Analysis of Error Types
Individual Student Exam Results Average % of Morphosyntactic Error Types (Total errors = 956/1927: 50%) 0%
14%
27%
Other Errors
6%
Accent Errors
Number Agreement Gender Agreement
Noun Gender Capital.
Multiple Character Mispelling
Single Character Misspelling
11%
17% 15%
Noun Gender / Adjective Agreement = 32%
Volume 18 Number 3
3%
7%
Spelling = 54%
509
Computer-Based Grammar Checker and Self-Monitoring Spelling mistakes were by far the most frequent, accounting for 54% of all morphosyntactic errors. Half of all spelling mistakes were related to wrong or missing accents. Trailing a distant second and third place among morphosyntactic errors was adjective (including past participle) agreement (17%) and noun gender (15%). The remaining 14% of low level errors were thinly spread among a number of categories. In order to determine the predictability, and ultimately the self-monitoring correctability, of mistakes, a detailed analysis was undertaken of the two major morphological error categories: spelling and gender. With the exception of capitalization, misspelling unrelated to accents proved to be quite random. Incorrect capitalization, on the other hand, was almost exclusively related to a single phenomenon: the noun/adjective distinction in names of nationality (e.g., un Français/l’histoire française ‘A Frenchman [noun]/the French [adj.] story’). A breakdown of accent mistakes revealed a very skewed distribution of errors (see Figure 4). Figure 4 Quantitative Analysis of Accent Error Types
Fully 84% of incorrect accents involved just two forms: é and è. Of these two forms, a missing acute accent on vowels pronounced [e] accounted for over two thirds (68%) of all accent errors. The incorrect use of é for vowels pronounced [E] and [´] accounted for an additional 6%. A miss510
CALICO Journal
Jack Burston ing grave accent on vowels pronounced [E] represented another 10% of the total accent errors. It addition, it turned out that over half of all gender errors (54%) related either to nouns with morphologically marked gender or very basic vocabulary items (see Figure 5). Figure 5 Analysis of Gender Errors
Semester 1/1999 Final Exam: Composition Gender Errors (n = 146/956)
13%
-sion / -tion
Other
16%
(21% masculin ending in -e)
Other Gender Endings Common Feminin Nouns
46%
Common Masculin Nouns
19%
6%
e.g. monde, nombre, problème, type
e.g chapeau, dîner, magasin, menu, mur, temps
e.g. chanson, chose, cuisine, fin, pièce, pierre, table, ville
Surprisingly, after the equivalent of at least seven semesters of French instruction, the largest single category of gender misassignment (13%) involved nouns with the archetypical feminine ending: -sion/-tion. Other reliably marked gender endings (e.g., -ure, -ance, -isme, -age) accounted for another 16% of misassignments. With regard to elementary vocabulary, the failure to assign the correct gender of common feminine nouns amounted to 19% of all gender errors compared to 6% for common masculine nouns. Among the remaining 46% of gender errors, by far the most frequent mistake (21%) was the treatment of masculine nouns ending in -e as feminine.
MODIFIED PEDAGOGICAL STRATEGY On the basis of the error analysis of the first semester exam essays, it was apparent that the maximum return on investment could be obtained Volume 18 Number 3
511
Computer-Based Grammar Checker and Self-Monitoring by focusing on the elimination of predictable, hence avoidable, accent and gender mistakes. Successfully mastering the link between pronunciation and correct accent in the use of é and è and morphological gender marking in the case of -sion/-tion alone had the potential to reduce low level errors by 25%. Closer attention to adjective/past participle agreement could likewise bring the improvement in grammatical precision to over 40%. However, by the time the error analysis of the first semester exam essays was complete, only four weeks remained in the second semester. In order to impress upon students the need to develop self-monitoring for grammatical accuracy and to provide them with feedback on their progress during the semester, it had already been decided to remove the essay component from the second semester exam and replace it by two compositions written in class under examination conditions (what in French is known as a travail sur table [TST]). A one hour tutorial, based on the findings of the error analysis and previous experience gained from work with the 1998 experimental group, was given to all students the week preceding the second TST. A comparison of the results of the first TST (with no prior tutorial preparation) and the second TST showed the most significant improvements in grammatical precision to date (see Figure 6). Figure 6 Comparison of Error Rates Per 100 Words in TST 1 and TST 2 Histogram TST 2 (n = 40) Average = 5.2 Median = 4.8 Standard Deviation = 2.8
Histogram TST 1 (n = 44) Average = 8 Median = 6.5 Standard Deviation = 3.8
14
20 18
12
16 10
Frequency
Frequency 12 10 8 6
Frequency
14
Frequency 8 6 4
4 2
2 0
0 2.0
4.7
7.5
10.2
13.0
Errors per 100 words
15.7
More
1.1
3.0
4.8
6.7
8.6
10.5
More
Errors per 100 words
With an average of 8 errors per 100 words and a median score of 6.5, the first TST itself attested to a falling error rate compared to the results of the first semester exam essay (see Figure 2 above). The distribution of scores, as measured by the standard deviation, remained essentially the same. In comparison, the second TST revealed an average morphosyntactic error rate of 5.2 per 100 words, a mean of 4.8, and a substantially reduced standard deviation of 2.8.
512
CALICO Journal
Jack Burston DISCUSSION Outside of clinically controlled (and quite artificial) testing environments, links between cause and effect can never be unequivocally demonstrated. Real life classroom teaching, by its very nature, involves numerous variables which are difficult to specify and even harder to control. In particular, the personal influence of the instructor and relationships with students play a significant role in pedagogical results. Notwithstanding these qualifications, the evidence gives good reason to believe that the observed improvements in the grammatical accuracy of our students derives from the methodological strategies elaborated over the past two years. As part of an overall program to reduce morphosyntactic mistakes, the integration of grammar checker technology into the curriculum has proven very effective in improving the grammatical precision of submitted essays. By means of insistence, assessment weighting, and tutorial instruction, clear signs are emerging of improved self-monitoring in compositions written under examination conditions. The results of the end of year TST are especially encouraging in this respect. It needs to be acknowledged, however, that some of this gain may simply stem from a process of natural selection to a certain extent, that is, the loss of students who failed the first semester of the course. However, not all the decrease in enrollments between first and second semester can be attributed to the elimination of weak students. While the special tutorial which preceded the second TST is the most probable factor influencing the outcome, it is also possible that the observed improvement was the result of the cumulative effects of exposure to the language. From a practical standpoint, it could be argued that, whatever the precise contribution of the variables in the course, the important result is that improvements in grammatical accuracy are beginning to show themselves, both in submitted homework and examinations. It is also important to bear in mind that this reduction in morphosyntactic errors takes place in an advanced level language course in which the primary focus is on discursive and argumentative writing skills. In contrast to typical communicative methodologies, our students are having to learn that they cannot “get the message across” without due regard to formal correctness. At this level, writing in French necessarily means writing à la française (i.e., with no fautes bêtes).
CONCLUSION The lessons learned from our 1999 experiences are being put into practice in the 2000 academic year. The use of Antidote is continuing, with increased emphasis on making students more aware of the need to work Volume 18 Number 3
513
Computer-Based Grammar Checker and Self-Monitoring through partial analyses and to recognize that warnings of possible errors are not over corrections. Two TST are being introduced into the curriculum from first semester accompanied by a series of self-monitoring grammar checking exercises. A one hour tutorial, modelled on the one given before the second TST in 1999, is likewise being added to the program. Finally, error analysis of the TST is being undertaken for comparison to the 1999 results with a view to demonstrating the effectiveness of our pedagogical strategies. To the extent that the second semester results of 1999 can be duplicated by the end of the first semester in 2000, we should have objective evidence that improvements are in fact directly related to the methodology used and not simply the result of extraneous factors such as the attrition of weak students or cumulative exposure to the language.
REFERENCES Brock, M. (1990). Customizing a computerized text analyzer for ESL writers: Cost versus gain. CALICO Journal, 8 (2), 51-60. Burston, J. (1998). Review of Antidote 98. CALICO Journal, 16 (2), 197-212. Burston, J. (1996). Software report: A comparative evaluation of French spelling/ grammar checkers. CALICO Journal, 13 (2-3), 104-111. Corder, S. P. (1981). Error analysis and interlanguage. Oxford: Oxford University Press. Doughty, C., & Williams, J. (1998). Focus on form in classroom second language acquisition. Cambridge: Cambridge University Press. Knowles, S. (1992). Review of Grammatik IV. On-CALL, 6 (3). Levy, M. (1992). Writing rules with Grammatik IV. On-CALL, 6 (3), 37. Mogilevski, E., & Burston, J. (1999). Morphosyntactic accuracy in the written compositions of advanced university level students of French. Australian Review of Applied Linguistics, 21 (2), 1-20. VanPatten, B. (1996). Input processing and grammar instruction. Norwood, NJ: Ablex Publishing Corporation.
514
CALICO Journal
Jack Burston AUTHOR’S BIODATA Jack Burston (B.A., M.A. in French; Ph.D. in Linguistics) is the Director of FLIT at Temple University. He is particularly interested in authoring languages and systems for courseware development and is currently completing a multimedia CD-ROM/WWW project on French sociolinguistics. His major research interest is in second language acquisition in instructed environments. Since 1996 he has been the Software Review Editor of the CALICO Journal.
AUTHOR’S ADDRESS Dr. Jack Burston Director of FLIT College of Liberal Arts 524 Anderson Hall Temple University 1114 W. Berks Street Philadelphia, PA 19122 Phone: 215/204-3678 Fax: 215/204-3731 E-mail:
[email protected]
Volume 18 Number 3
515
