Detecting, diagnosing and correcting low-level problems when editing with and without computer aids
Rickard Domeij Interaction and Presentation Laboratory (IPLab) Dept. of Numerical Analysis and Computing Science Royal Institute of Technology Sweden
[email protected] January 16, 1997
Abstract Can a tool for computer-aided editing be used to make bureaucratic writing more clear and less formal? The question is explored in a research project of which the study presented here is a part. In the study, 16 university students were asked to revise a letter in bureaucratic style, first using pen and paper, then using computer aids. The letter was prepared to contain 26 problems in mechanics and style, all of which could be analysed by the computer tool. The design made it possible to compare the number of changes subjects made to planted problems in mechanics and style with and without computer support. In avarage, subjects changed 99 % of the mechanical problems when using the computer tool, compared to 52 % without it. In contrast, subjects changed only 63 % of the style problems when using the computer tool, compared to 27 % without it. Thus, the computer tool had a strong influence on the total amount of changes made to mechanical problems; for problems in style, the influence was considerably weaker. Interestingly, the influence of the tool on the number of changes made in style varied greatly between different subjects. While some writers changed many problems in style both with and without computer support, other writers made almost no changes in style, even though they were urged to attend to them by the computer tool. Among the writers who made few or no changes in style without aid, some were strongly influenced to change in style by the tool. It is suggested that these differences may be related to different revision strategies employed by the writers, some of which match the strategy embodied in the computer tool, some of which do not. Possible negative effects are discussed.
Practical background The study presented here is part of a cooperation project between the Royal Institute of Technology and the Swedish County Government Board of Stockholm. The aim is to explore the possibilities of designing a tool for computer-aided editing as a means of making public letters from the authorities more reader-oriented. Public letters are traditionally characterized by bureaucratic language, a highly formal and incomprehensible writing style used within the authorities and in other 1
organisations as well. The authorities' new policy is to reform bureaucratic language. One strategy has been to publish black lists with especially bad examples of bureaucratic expressions, and style guides propagating a less formal and more comprehensible writing style. Style guides and other handbooks are the traditional aids available for writers at work. Since some years back, there are also other means available for writers to check their writing at the computer (see Kuchich 1996 for a good overview of available techniques). Today's advanced word processors have language checking functions for spelling, grammar and style with great potential for writers at work despite technical limitations (Kohut et al. 1995). However, while big languages – especially English – are quite well served in this respect, smaller languages – such as Swedish – are not (a spelling checker for Swedish is presented in Domeij 1995). If a tool for computer aided editing were available for Swedish, perhaps it could contribute to make bureaucratic writing more comprehensible and less formal in style. To this end, rules for detecting and correcting problems in bearaucratic writing have been implemented in a prototypical tool for computer-aided editing. Writers use the tool interactively and problems are presented with remedial instructions (Domeij 1996). Changes can sometimes be made just by accepting a concrete suggestion made by the tool, but writers can also edit directly in the text as they wish. When presented to a problem, a writer can either choose to edit the problem, or continue to the next problem without changing. So the writer's choice here is to change or not to change. In this study, I wanted to look deeper into this situation of choice, trying to explore how well writers are enabled to change effectively with support from computer aids.
Theoretical background Word processors have made it much easier for writers to revise their texts compared to when they had to use traditional techniques such as pen and paper or typewriter. While this may be a good thing for experienced writers who have the knowledge and skills to perform revision effectively, it is of limited help for inexperienced writers who may lack these skills. Therefore, functions that support the skills involved in revision have great potential. So, what are these skills? Hayes et al. (1987) present an elaborate model of revision based on their own think-aloud studies and previous research. In Flower (1986) they emphasize the role of three crucial subskills in the review process: detecting, diagnosing and revising. Their findings suggest that inexperienced writers not only detect less problems than experienced writers, they also have problems diagnosing them correctly and choosing effective strategies for dealing with them. Even if a writer detects a problem, he may fail to diagnose it and be unable to change – or he may misdiagnose and change ineffectively. Hayes et al. also observed that, in general, inexperienced writers detected, diagnosed and revised almost exclusively local problems, disregarding global problems in structure and intention. This local scope was found to be related to how writers represented their task, the overall strategy of revision. In general, experienced
2
writers represented it as a wholetext task involving intention and reader-orientation, while inexperienced writers represented it as a mere hunting for low-level problems. Consequently, computer-aided editing may have a role in supporting writers to detect, diagnose and revise local and global problems in their writing according to the demands from the overall task of revision. How well computers may support writers revising problems in bureaucratic style is a question explored in practice by this study. The study is inspired by two similar studies on revision of planted low-level problems (Hayes et al. 1987 and Hill et al. 1991). However, these studies did not involve computer aids.
Aim of the study: do they change or not? The aim of the study was to understand more about how computer-aided editing may influence writers' ability and willingness to edit relevant problems in bureaucratic writing. The following qustions were addressed: 1. Do writers change when being urged to attend to relevant problems by the computer tool, or not? 1a. If yes, are they really enabled to change effectively when supported by the instructions from the computer tool? 1b. If no, why? Are writers sometimes not enabled to change with the computer tool because of insufficient instructional support, or are there other reasons – perhaps they are not willing to change because they disagree with the instructions from the program? In order to explore these questions, I also needed to know what problems writers edited without aid – in particular among those problems that can be detected with computer aid.
Method: editing with and without aid The study was designed as a comparative study combining quantitative and qualitative methods with the aim of exploring the questions addressed above. 16 university students were asked to edit a real letter from the authorities showing many characteristics of bureaucratic language. The letter was prepared to contain 26 planted problems which could be analysed with computer-aid (for convenience I call them "planted", even if some of them were there from the start). The letter also contained other, more global problems of bureaucratic style that could not be detected by the computer tool. The students were given two tasks: Task 1: to edit the letter on paper Task 2: to edit the letter again using computer-aid, being presented to all 26 planted problems by the tool.
3
Task 1 was presented as a whole-text task open for global revision of the letter, while, in task 2, writers were told only to respond to the planted problems presented by the computer tool. The design made me able to observe what additional problems writers were influenced to change with aid compared to what problems they changed without aid.
Problems planted in the text The problems planted in the letter were 13 problems in mechanics and 13 problems in style. The mechanical problems were of the following types: Character level: punctuation, typographical conventions, capitalization, use of apostrophe Words and phrases: date format, abbreviation, word segmentation, phrase construction. The 13 style problems in diction and usage included both problems that were consistent to the overall formal, bureaucratic style of the document and problems in informal style that were inconsistent to the overall style (note that the letter also contained many problems in bureaucratic style other than the planted local style problems, for example global problems in bureaucratic style that could not be detected by the computer). The style problems were of the following types: Bureaucratic problems (style-consistent): formal/ancient word variants, long prepositions, formal pronoun, incomprehensibleexpression, nominalisation, passive voice Informal problems problems (style-inconsistent): informal word variant, informal pronoun case, sentence initial "and" The source used for the mechanical problems were the book of Swedish writing rules, a style-guide highly agreed upon among Swedish writers. The sources used for the bureaucratic problems were different style guides produced within the authorities. The style-inconsistent, informal problems are among those mentioned in commercially available style-guides.
Analysis and results After the 16 students had completed their two tasks (manual and computer-aided editing of the letter with the 26 planted problems), all changes made to planted problems in each task were collected, counted and compared for frequency. The general results are presented below.
4
Average number of planted problems changed
in mechanics (of 13)
in style (of 13)
Task 1: without aid
6,8 (52%)
3,5 (27%)
Task 2: with aid
12,8 (99%)
8,1 (63%)
Table 1. Avarage number of changes made to planted problems in mechanics and style when not using computer aid compared to when using computer aid.
Results suggest that writers are strongly influenced by the computer to change problems in mechanics with computer aid, while, for problems in style, the influence is not as strong (see table 1). Without the tool writers changed in avarage 52 % of all planted mechanical problems. When using the tool, they changed almost all mechanical problems, 99 % in total. For these mechanical problems, then, the computer seem to complement human editing rather well. In contrast, the results for changes to problems in style give a more complicated picture, as shown in table 1. First, without the computer tool, only 27 % of the planted style problems were changed in avarage. Among the most infrequently changed problems were problems in beauracratic style. Although the writers were urged to attend to the remaining planted problems in style when using the tool, writers only changed 63% in total. Why is that? Even if the relative influence of the tool is rather substantive, there seem to be some problem with computer supported checking of style compared to the checking of mechanics. Are writers not given enough instructional support to be able to change problems in style by aid, or are they just not willing to change?
Discussion There seems to be some evidence for the claim that writers are not always enabled to change problems in style with aid. For some problems, writers did not seem to get enough instructional support to be able to correct problems of style. Problems were less often changed with aid if they were presented without concrete suggestions. Moreover, when changed, problems without concrete suggestions were often seen to result in ineffective changes. This often seemed to be the case for problems in passive voice, which can be quite difficult to handle. From what it seems then, the computer tool does not always enable writers to change effectively, even if they seem quite willing to do so. However, there is a lot of variance in changes between writers that can not be explained by insufficient instruction. While some writers made very few changes in formal style both with and without aid, others started changing heavily with aid, although they had made very few changes in style without aid. And the only writer who changed all problems in style with aid, also made changes most frequently without aid. How can these differences be explained?
5
Some of these differences seem to be related to different revision strategies. Writers having a revision strategy that conforms to the overall strategy reflected in the advice from the program, may be more willing to change problems in bureaucratic style because this is what they are already trying to do. The computer tool supports them in their mission to change for more clarity and a less formal tone. This may explain why some writers willingly made changes in bureaucratic style both with and without aid. In contrast, other writers do not seem very willing to change in style at all, especially not when that would be inconsistent with the overall style of the document. They seem to have an initial, overall style-preserving strategy which they keep to when using the computer tool, disregarding advice from the tool. One example of such an overall style preserver is writer 9 whose changes can be seen in table 2. This writer made 6 changes in mechanics without aid, and only 2 changes in style. Both changes in style were to problems in informal style which were inconsistent with the overall bureaucratic style of the letter. When presented to all planted problems with aid, the writer changes all additional problems in mechanics, but only one more problem in style – the third informal, style-inconsistent problem. For this writer, not changing problems in bureaucratic style seems consistent to his or her overall strategy of revision. Writer 9: style-consistency strategy Problem Type use of apostrophe mechanics capitalisation within word mechanics date format 1 mechanics punctuation 1 mechanics capitalisation in proper noun mechanics phrase construction mechanics segmentation: split word mechanics punctuation 2 mechanics segmentation: concatenation mechanics abbreviation mechanics punctuation 3 mechanics date format 2 mechanics typographical convention mechanics long preposition 1 beauraucratic nominalisation beauraucratic formal/ancient word beauraucratic incomprehensive expression beauraucratic formal pronoun beauraucratic long preposition 2 beauraucratic passive voice construction 1 beauraucratic wrong pronoun case informal passive voice construction 2 beauraucratic passive voice construction 3 beauraucratic informal word informal sentence initial "and" informal formal/ancient word beauraucratic
Without aid 1 1 0 0 1 0 0 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0
With aid 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 1 0 0 1 1 0
6
Another group of writers seems strongly influenced to change in style with aid, although they made very few changes in style without aid. Consider writer 5 who is an example of such a writer (see table 3). This writer only made one change in style without aid. When using aid, writer 5 start changing almost all problems in style. Unlike previously observed writers, this writers responses do not seem consistent with a whole-text revision strategy. Just as Hayes et al. (1987) often observed with unskilled writers, writer 5 may not have a strategy other than that of hunting for local-scope problems. Is this writer really enabled to change effectively by the program, or just willing? It is hard to say. Writer 5: enabled or just willing? Problem Type use of apostrophe mechanics capitalisation within word mechanics date format 1 mechanics punctuation 1 mechanics capitalisation in proper noun mechanics phrase construction mechanics segmentation: split word mechanics punctuation 2 mechanics segmentation: concatenation mechanics abbreviation mechanics punctuation 3 mechanics date format 2 mechanics typographical convention mechanics long preposition 1 beauraucratic nominalisation beauraucratic formal/ancient word beauraucratic incomprehensive expression beauraucratic formal pronoun beauraucratic long preposition 2 beauraucratic passive voice construction 1 beauraucratic wrong pronoun case informal passive voice construction 2 beauraucratic passive voice construction 3 beauraucratic informal word informal sentence initial "and" informal formal/ancient word beauraucratic
Without aid 1 0 0 1 1 1 0 1 1 1 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 1
With aid 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 0 1 1 1 1
One can certainly see some dangers here. If skills are missing, just calling attention to complex problems may lead to increased cognitive effort with poor or no result. On the other hand, if unskilled writers just click yes to concrete suggestions without thought, it may lead to unmotivated changes and global inconcistency problems in style. This is were willingness and ability seem to converge in a dilemma. Computers can only identify some surface problems in style, which gives no global support for revising bureaucratic style consistently. Hence, while less skilful, revisers with a local scope may not be given enough support to be able to perform revision of overall style, more skilful revisors with a global concern for consistency may be unwilling to perform revision of overall style because they prefer global consistency before clarity. Only 7
writers who are already trying to change the overall bureaucratic style to become more clear and reader-oriented without aid, may be both able and willing to use the advice from the computer tool adequately, because their strategy match the style revision strategy embodied in the tool.
Summary and future research The quantitative results support what may be expected: writers do not seem to find advice on style as useful as advice on mechanics. Nevertheless, the influence was observed to be relatively strong for problems in style, considering the few changes made to these problems without aid. However, there are reasons to believe that these additional changes may not be only for good. Some ineffective changes were observed in the study, especially when the presented problems were not supported with concrete remedial suggestions. Furthermore, lack of whole-text support for style makes global effects uncertain – a problem that deserves more research. The qualitative exploration of the data, also pin-pointed other problems that would be interesting to explore further. First, if unskilled writers are urged to attend to a complex problem by aid and they are not given enough support to diagnose and revise the problem effectively, it seems probable that some writers will spend time and effort trying to revise the problem even if they eventually give up. In this hypothetical case, the tool will trigger mental effort that do not result in actual changes. Thus, computer-aided editing may sometimes cause unskilled writers more trouble than they already have. I would like to explore this question further. Second, it would also be interesting to further explore the relation between computer-aid and writers' revision strategies. Third, the notion of style and genre seem more complex and socially-situated than assumed in the hand-books used in the construction of computer-aid (cf Freedman & Medway 1994). Not only are the advice on style often simplistic, its relation to genre is not as straith-forward as traditionally assumed. Genre is not an ideal type of text with certain text features that all writers are willing to adhere to. As suggested by the study, there may be conflicting views among writers on what the style of the genre is and what motivates it. Therefore, the authorities may have trouble in introducing their new policy in writing style among the clerks, at least if they only depend on the influence of a computer tool. It seem essential to consider computeraided editing in a wider context. The study presented here is experimental and, as such, unnatural in setting. To complement it, a good thing would be to leave the laboratory for a while in order to interview people and observe how they use computer tools in a realistic setting (cf Sharples 1996). This seem the right approach to get a wider perspective and a deeper understanding of the problems involved in designing and using a tool for computeraided editing at work.
8
References Domeij, Rickard, Hollman, Joachim & Kann, Viggo, (1994). Detection of spelling errors in Swedish not using a word list en clair. Topical Paper. In: Proceedings of the 2:nd International Conference on Quantitative Linguistics, Moskva, september 1994. Domeij, Rickard (1996). Detecting and presenting errors for Swedish writers at work. IPLab-rapport 108, TRITA-NA-P9629, IPLab, NADA, KTH. Flower, L., Carey, L., Hayes, J. R., Schriver, K., & Stratman, J. (1986). Detection, diagnosis and revision. College Composition and Communication, 37, 16–55. Freedman, A. & Medway, P. (1994). Genre and the new rhetoric. London: Taylor and Frances LTD. Hayes, J. R., Flower, L., Schriver, K., Stratman, J. & Carey, L. (1987). Cognitive processes in revision. In: S. Rosenberg (Ed.), Advances in applied psycholinguistics: Vol. 2. (pp. 176–240). New York: Cambridge University Press. Hill, C., Wallace, D. & Haas, C. (1992). Revising on line. Computers and Composition, Vol. 9. No. 1. Kohut, G. & Gorman, K. (1995). The effectiveness of leading grammar/style software in analysing business students’ writing. JTBC pp. 341-361. July 1995. Kukich, K. (1992). Techniques for automatically correcting words in text. ACM Computing surveys, Vol. 24, No. 4, pp. 377–439. Sharples, M & van der Geest, T (Eds). (1996). The New Writing Environment. London: Springer.
9
