Noteworthy Observations about Note-Taking by Professionals

J. TECHNICAL WRITING AND COMMUNICATION, Vol. 35(3) 317-329, 2005

NOTEWORTHY OBSERVATIONS ABOUT NOTE-TAKING BY PROFESSIONALS

MARK MELENHORST THEA VAN DER GEEST MICHAËL STEEHOUDER University of Twente, The Netherlands

ABSTRACT

In this article we focus on professional readers who have to write recommendations in an online environment. We address the question whether taking notes on screen influences the reading process and the quality of the recommendations in terms of applicability, completeness, and persuasiveness. Seven participants each composed two pieces of advice on technical communication issues. They could use an electronic Notepad whenever they wished. Taking notes appeared to influence advice quality negatively, which may be caused by attention shifts from reading to taking notes on screen. Although we could not find a relationship between the contents of the notes and advice quality, we noted differences in note-taking approaches between the participants.

TAKING NOTES DURING PROFESSIONAL WRITING-FROM-SOURCES TASKS Taking notes is common for professional readers who have to accomplish a writing-from-sources task, in which multiple documents are read to come up with a substantiated piece of writing. When professionals are engaged in such tasks, they copy citations, write notes that relate multiple sources to one another, refer to previous notes, reread previous notes, and incorporate notes into the eventual piece of writing [1]. Both writing-from-sources and note-taking are usually 317 Ó 2005, Baywood Publishing Co., Inc.

318

/

MELENHORST, VAN DER GEEST AND STEEHOUDER

studied within educational settings with the purpose of improving learning outcomes. But learning is not a goal in itself in many other settings. Little research has been done on note-taking during a writing-from-sources task in noneducational settings. The rhetorical situation, the often diverse and sometimes contradictory materials, as well as the different writing goals may affect how readers take notes. From previous educational research on note-taking we know that the quality of the eventual piece of writing is influenced by how learners structure their notes and whether they formulate their own notes or copy passages verbatim from the sources. For instance, Kiewra et al. studied how different note-taking formats influenced student note-taking during lectures [2]. They found that undergraduate college students performed better in terms of recall and drawing relationships between elements when they used a hierarchical outline to structure their notes compared to a two-dimensional matrix framework and just a blank sheet of paper. Slotte and Lonka observed 226 college students who took notes on a single source, which had to be used to write an essay [3]. Students who frequently formulated notes themselves wrote essays that were more complete and better structured than students who took fewer notes and copied-and-pasted passages. Additionally, reviewing notes had a positive effect when deep level processing was required for the writing task. The explanation for these results is that when students fit information into a framework or when they formulate paraphrases, they put more cognitive effort into processing the information. In terms of [4], they transform knowledge rather than tell knowledge resulting in superior pieces of writing. However, it remains to be seen whether these results apply to non-educational settings as well. Is there a similar relationship between the quality of the eventual piece of writing and how readers take notes in a non-educational setting? One of the few studies in which note-taking (taking notes on separate pieces of paper) and annotation (notes taken on the source document itself, close to the passage it refers to) are studied in a non-educational setting is a small-scale exploratory study by O’Hara and Sellen [5]. They compared how hard-copy readers and on-screen readers write a summary based on one article. Note-taking and annotation were considered important to comprehend the text and to form a plan for writing the summary, because the notes not only helped readers to draw attention to the main points, but also to explicate the structure between these points: readers found that taking notes helped them to collect information from dispersed locations. In an ethnographic study O’Hara et al. found similar results [1]. They took a Distributed Cognition perspective on the writing-from-sources process [6], which emphasizes that the interaction with artifacts (such as notes) influences the cognitive processes. They videotaped 10 professionals who were engaged in writing-from-sources tasks that were part of their jobs, ranging from a researcher who had to write a conference paper to a contract lawyer who had to write a reply

OBSERVATIONS ABOUT NOTE-TAKING /

319

to a client. In a debriefing session after the task, participants were asked to explain their actions based on parts of the tape. Readers appeared to arrange documents on their desks in such a way that they can read, compare, and contrast information from different sources, while writing the eventual text at the same time. These desktop arrangements facilitate the use of notes during composition. With respect to note-taking, O’Hara et al. found that their participants primarily write notes close to the passage the note referred to [1]. These notes functioned as cues to more detailed knowledge the reader had acquired while reading the passage. Consistent with a study by O’Hara and Sellen [5], they found that the notes helped readers to reread the materials and to integrate information from disparate sources, which in turn generated new ideas. A quick shift in attention suffices to scan the notes they have already taken. O’Hara et al. did not report on the use of notes on separate sheets of paper, neither during reading nor during composing [1]. To summarize, writing notes on the sources close to the text helps readers to off-load some of the cognitive effort from the writing-from-sources task to paper [1], while in an educational context taking notes is assumed to result in additional processing of information and thus to superior pieces of writing. TAKING NOTES ON ONLINE DOCUMENTS Taking notes on online documents is different from taking notes on paper. When documents are read from screen, the advantages of a flexible physical organization of documents disappear. Gaining an overview of a document or comparing-and-contrasting information from documents is hard when documents have to be read from screen. The ease with which readers can take notes is one of the main reasons to prefer reading from paper over reading from screen [5]. When readers want to take notes on screen, they have to shift from reading to taking notes deliberately, because they have to activate the corresponding function with their mouse. O’Hara and Sellen noted that this complex interaction with keyboard and mouse refrained readers from annotating documents onscreen, even though annotating was a natural activity for readers to perform [5]. Instead, three of the 10 participants started to take notes in a separate window. The downside of this approach is that readers have to activate the separate window, which moves the sources out of view and disrupts task execution. A second disadvantage is that the context the note refers to has to be recreated in the notes: when readers reread their notes, the notes will be incomprehensible when it is unclear to which information they refer [7]. In writing-from-sources tasks, these shifts in attention interfere with task execution [1]. However, the consequences of these attention shifts in terms of the quality of the eventual piece of writing were not addressed by [7] and [1]. Online note-taking projects have focused on the development of annotation tools. Wolfe has reviewed 25 of such annotation tools [8]. These tools have various

320

/


features, ranging from basic highlighting to hyperlinks within annotations. In general, the tools seek to emulate how documents are annotated on paper while they try to make use of the opportunities the online environment has to offer. However, interacting with their interfaces is complex and requires substantial effort. Instead of focusing on technology, we will direct our attention to how the process of note-taking onscreen affects the writing-from-sources process within non-educational settings as it is unclear both whether professional readers benefit from taking notes as well as which note-taking practices are the most effective [9]. More specifically, we will address the following issue: can the quality of a substantiated piece of writing be traced back to the process and contents of the notes? METHOD To answer this research question we set up a study in which participants were asked to write substantiated recommendations based on multiple online sources. The design of the study is described below. Participants Seven graduate students (four male, three female) of Technical Communication participated in the study, most of which were employed as technical communication professionals. In this article, we gave the seven participants fictitious names, running from A through to G. Tasks The participants were given two writing-from-sources tasks. The first task, which we called the Animation task, dealt with the use of animations in an educational Web site for 12-year-old students. The participants were asked to give substantiated recommendations to a Web site designer on the appropriateness of animation (rather than static visuals) for explaining the effects of eating too much fat food. In the second task, called the Intranet task, the participants were asked to provide substantiated recommendations about the structure of an Intranet site. A fictitious client had to be convinced whether an Intranet home page should contain many or just a few menu items. The order of tasks was balanced between participants. Both tasks were carried out during the same session with a short break in between them. Materials In both tasks, the participants had a set of electronic articles in different genres (research reports, columns, and a handbook chapter) with diverse and sometimes


321

contradictory information. The participants were familiar with the subjects of the task. During their two writing-from-sources assignments, the participants had permanent access to the task description and the articles through a permanently visible navigation bar. They could activate Microsoft’s Notepad whenever they wanted to make notes, which resembles a blank sheet of paper, which is common in research on writing-from-sources in a paper environment. Procedure and Data Collection The study was conducted in the University of Washington’s usability lab of the University of Washington, Seattle. After an explanation of the procedure, the participants started reading and making notes. They were not forced to take notes and they were free to choose their own approach. Participants were familiar with the Notepad application as it resembles a very basic word processor. They did not show any difficulty in using Notepad. They were asked to verbalize whatever they were reading and writing, mixing their evaluative comments in a natural way. If they fell silent, they were prompted to resume thinking aloud. To emulate the time pressure under which professionals carry out writing-from-sources tasks, we allowed the participants to spend only 30 minutes on reading the sources for each task. Thus, selective reading was required, which is common for professionals. After reading, they were asked to write their recommendations in a new window. Participants could make use of their notes during writing. The following data were collected: • Think-aloud protocols, recorded with video cameras directed at the screen. • Notes and recommendations, as written by the participants. The protocols of the participants were transcribed, divided into episodes and then coded, based on the diverse activities readers employed during reading and note-taking. Each sentence expressed or each interaction with the environment (such as activating Notepad or pasting a passage copied from the sources) was counted as one episode. The sentences in the notes and the recommendations were divided into clauses. Ranking Procedure for Advice Quality To be able to analyze the relationship between the notes, the piece of advice, and advice quality, we asked 21 experienced Web designers to rank the recommendations on quality dimensions. We tentatively formulated three dimensions that constitute advice quality: 1. Completeness: the degree to which the piece of advice addresses every issue from the task description;

322

/


2. Applicability: the degree to which the advice can be used as guidance for design decisions; and 3. Persuasiveness: the degree to which the piece of advice convinces the recipient of the credibility and accurateness of its arguments. Ten experienced Web designers ranked the recommendations for the Animation task, while the other 11 ranked the recommendations for the Intranet task. The order of the quality dimensions was counterbalanced across the raters. After a rater was done with one dimension, the experimenter asked him to explain why a piece of advice was ranked on the top, the middle, or the bottom of the pile. After explaining the last dimension, the raters were asked whether they felt other important dimensions of advice quality were lacking. THE PROCESS OF TAKING NOTES In this section we will answer the question whether the quality of the recommendations can be traced back to note-taking activities and their frequency. Advice Quality The seven recommendations in the Animation task were ranked by 10 raters, the seven recommendations in the Intranet task were ranked by 11 raters. Thus, the worst piece of advice was ranked with a one, the best piece with a seven. To correct for the difference in the number of raters, we computed the sum of the average rankings in the two tasks. Values can thus range from 2 to 14, as they sorted the recommendations of seven participants. The results can be seen in Table 1. The raters’ perspectives on applicability and completeness were relatively consistent with each other, indicated by the difference between the lowest and the highest value as well as the relatively low standard deviations for most pieces of Table 1. Summated Average Advice Quality Rankings Completeness

Applicability

Persuasiveness

Anne

7.6 (1.1)

6.3 (1.1)

8.0 (2.3)

Bill

6.7 (1.6)

6.4 (1.0)

5.3 (1.4)

Charles

2.1 (.3)

2.5 (1.2)

5.4 (1.6)

Derrick

7.5 (1.8)

7.6 (2.1)

6.3 (1.3)

Emily

7.0 (1.1)

9.3 (.8)

6.1 (1.4)

Fred

9.2 (1.4)

8.2 (2.3)

6.3 (2.0)

Gail

2.8 (1.0)

3.4 (2.2)

5.8 (1.6)

Note: Standard deviations between parentheses.


323

advice. Charles’ recommendations were seen as the least applicable and complete, while Fred’s recommendations received the best rankings on these two dimensions. However, no recommendations were unanimously considered to be of the best quality as the summated values did not exceed 9.3, while the highest possible ranking was 14. The raters had different perspectives on what persuasive recommendations were, indicated by the moderate values clustered around 6 and the high standard deviations. Only Anne’s recommendations were seen as more persuasive than the others. Table 2 shows the correlations between the dimensions of advice quality. Completeness is significantly related to both applicability and persuasiveness. Based on the explanations readers offered for their rankings, we assume that the raters considered completeness a prerequisite for recommendations to be applicable and persuasive. However, the correlations between the three dimensions are not strong enough to capture advice quality in one value. Therefore, in order to discuss the relationship between note-taking and advice quality we will relate aspects of the note-taking process to each of these dimensions separately. Effect of Frequency of Taking Notes on Advice Quality Readers differed with respect to the frequency in which they took notes during reading. For each task, we transcribed what they had read, their evaluative comments, and their interactions with Notepad and with the other parts of the reading environment. We computed the percentage of the episodes in which readers interact with Notepad. Formulating notes, rereading notes, copying and pasting clippings on Notepad, and opening and closing the Notepad window make up this percentage. These percentages are displayed in Table 3. The two most important components of the note-taking percentage are the shifts from reading to annotating and the frequent copying and pasting of passages. It was particularly Charles who copied and pasted citations frequently and thus had to shift back and forth from reading to note taking frequently. Charles was also rated lowest on the advice quality dimensions.

Table 2. Correlations between Advice Quality Dimensions

Completeness

Completeness

Applicability

Persuasiveness

—

.60***

.36***

Applicability Persuasiveness ***Correlations significant at .001 level.

—

.16 —

324

/


To test whether the total percentage of note taking-related episodes influences advice quality, we computed the correlation between this percentage and the three average advice quality rankings. These correlations are shown in Table 4. Apparently, taking notes frequently is negatively correlated to both completeness and applicability, but not to persuasiveness. One explanation may be that the attention shifts caused by taking notes in a separate window as pointed out by [5] influences the degree to which readers can focus on their task and thus how well they are prepared for the writing task. THE CONTENTS OF THE NOTES In the previous section we focused on the process of taking notes. This section addresses the effect of different note-taking formats and structures on advice quality. Effect of Formulation on Advice Quality Participants also differed in the formulation of their notes: while some of them resorted to writing notes themselves, others copied information verbatim from the sources. Table 5 shows the number of clauses for each category.

Table 3. Percentage of Note-Taking in the Protocols Participant Anne Bill

Total % 9.8 7.8

Charles

21.0

Derrick

12.8

Emily

7.2

Fred

4.0

Gail

20.0

Table 4. Correlations between Total % of Note-Taking and Advice Quality

Note-taking % *p < .05.

Completeness

Applicability

Persuasiveness

–.633*

–.554*

.224


325

Table 5. Number of Copied and Transformed Clauses in the Notes Participant Anne

Copied

Transformed

Total

19

45

64

18

18

Charles

111

23

134

Derrick

8

20

28

Bill

Emily

35

Fred

48

Gail

49

35 48 16

65

Note: Numbers indicate the number of clauses.

As is clear from Table 5, participants differed with respect to the degree of effort they put into transforming information from the sources to the notes. Charles frequently copied citations, only writing the headings himself. These headings consisted of references to the articles. In contrast, Bill wrote keywords like “equipment” and “software” that may have had the purpose to trigger knowledge acquired from the articles. Anne’s notes consisted of paraphrases, relevant citations, draft advice like “consider having Flash in a pop-up window,” and background knowledge (“Shneiderman, interface design guru”). To observe the effect of the degree of transformation, we computed the correlation between the percentage of transformed clauses and the rankings of the recommendations on each of the three dimensions of advice quality. The three correlations were not significant (p > .41): transforming clauses was not related to advice quality. This disproved our expectations and previous research [3], which stated that formulating one’s own notes results in superior performance compared to verbatim copying. Within the context of writing-from-sources in a professional setting, we did not find such an effect. Effect of Organization on Advice Quality We analyzed the notes with respect to the influence of their structure (the organizing principle the reader applied in order to group notes together) on advice quality. We found three organizing principles in the notes. Notes were either organized according to the task’s core issues or according to the articles, while some notes were not organized at all. Table 6 shows how the participants organized their notes. Only Anne in her Animation task and Gail in her Intranet task organized their notes according to the issues from the task description such as “Instructional

326

/


Table 6. Number of Clauses Organized and Unorganized Clauses in the Notes Participant Anne

Issue

Article

15

41

Bill Charles

123

Derrick

28

Unorganized

Total

8

64

18

18

11

134 28

Emily

35

35

Fred

48

48

Gail

22

43

65

Note: Numbers indicate the number of clauses.

issues” or “Depth vs. Breadth.” Apart from not organizing the notes at all, preparing a section in the notes for each article was the most common organizing principle. The purpose of organizing the notes according to this principle was to be able to make proper references to information from the articles. Derrick stated “I want to be sure to connect . . . these articles with their authors this time around, so I can make better reference to them.” Previous research within an educational context suggests that organizing notes results in better student performance than not organizing notes at all [2]. The question is whether advice quality is indeed affected negatively when readers do not organize their notes in a non-educational setting. To answer this question, we computed correlations between the relative number of unorganized clauses, clauses organized according to the articles, and clauses organized according to the task’s issues. Again, these correlations were not significant (p > .70), indicating that the way in which readers organized their notes is not related to the quality of the recommendations. Although no direct relationship was found between the manner in which readers organize their notes and the quality of the recommendations, note-taking did have an observable effect on the reading process. When readers organized their notes according to the task’s issues, we saw that it helped them to plan their reading process and monitor its progress. For instance, while looking at her notes Anne said: “But we haven’t had a comparison of learning with animation versus learning with straight text and pictures.” Gail changed the headings of her notes whenever the text she read raised an issue she felt was central to the task. Ultimately, “hierarchy or not hierarchy” and “breadth vs. depth” were the two issues that defined her Intranet task. She then tried to find information that resolved these issues. Thus, the notes helped


327

her to keep focused on the most important issues of the task, triggered by the information she read. However, using the notes to monitor the task’s progress did not affect the writing-from-sources process to such an extent that it resulted in higher advice quality, as correlations between advice quality and the organization of the notes did not reach statistical significance. Based on the results described above, we could not find evidence for a direct relationship between the contents of the notes and the quality of the piece of writing. DISCUSSION The aim of this study was to assess whether the quality of a piece of writing could be traced back to how readers take notes. The frequency of note-taking turned out to have a negative influence on completeness and applicability. There was no relationship between the contents of the notes and advice quality. How can this negative influence of taking notes frequently be explained? It may be caused by the extra cognitive effort that note-taking on screen requires. Although participants are able to copy and paste clippings with only a few mouse clicks or key strokes, they have to shift attention from reading to notetaking deliberately. In observing paper note-taking practices, O’Hara et al. concluded that the attention shifts from reading to note-taking are very small, with the notes serving as indicators to knowledge acquired from the sources [1]. But with onscreen note-taking in a separate window, the attention shifts appear to be larger, disrupting task execution. The large attention shifts may have prevented the participants’ ideas to evolve from a rough outline as laid out in the task description to a detailed view about what the eventual recommendations should contain. Such an unfocused or incomplete task representation is likely to affect advice quality negatively. Neither the structure nor the formulation of the notes had an influence on advice quality, which is in contrast to previous research. Kiewra et al. found in the context of note-taking during lectures that structuring notes using a hierarchical outline or a matrix results in superior performance [2]. Slotte and Lonka argued that when students formulate their own notes, their performance is better than when they copy passages verbatim [3]. Both the knowledge of the domain and the experience with the genre of the documents may offer an explanation for this contrast with previous research. The participants in our study had more topic knowledge and more experience with the genres of both the source documents and the text they had to write than participants in the educational studies had. This enabled them to process and remember the information more easily, which may have reduced the necessity of structuring the notes. Alternatively, the phasing of the reading and writing process may explain why the formulation and structure of the notes do not influence advice quality in

328

/


this case. Participants had to write their recommendations right after evaluating the four or five sources. There was no delay between reading and writing. Given their assumed domain knowledge, the readers might well have been able to remember what they have read, and thus it was not essential for them to organize their ideas into notes for writing the recommendations successfully. The phasing of the process may also explain why the formulation of the notes did not affect the quality of the recommendations. As the number of the sources to be evaluated was relatively small and there was no time lapse between evaluating these sources and writing the recommendations, it was not necessary to recreate the context of the clippings in the notes: participants could remember from which source the clippings were copied, reducing a potential effect of the formulation on the quality of the recommendations. Although we could not find a significant influence of different organizing principles or formulations on performance within the current context, we saw differences in how readers take notes on screen and what role the process of taking notes plays during reading. Copying-and-pasting clippings was the most dominant note-taking approach. This may have been an affordance of the online environment. The ease with which they could copy and paste clippings may have invited them to do so, while in contrast formulating one’s own notes required much more effort. In terms of Distributed Cognition (see p. 318), the ease with which readers could deal with external artifacts influenced how information was evaluated, a result which is consistent with O’Hara et al. [1, p. 296], who concluded that “Cognitive processes . . . have all been seen to be highly dependent upon the way that people can manipulate and interact with external representations, which in turn is dependent upon a particular representation’s material characteristics.” It was shown that onscreen artifacts like the notes in this study not only enable readers to off-load cognitive effort, but that interacting with these artifacts can frequently result in lower performance, presumably/assumedly because of the attention shifts between reading and taking notes.

REFERENCES 1. K. P. O’Hara, A. Taylor, W. Newman, and A. J. Sellen, Understanding the Materiality of Writing from Multiple Sources. International Journal of Human-Computer Studies, 56, pp. 269-305, 2002. 2. K. A. Kiewra, S. L. Benton, S. I. Kim, N. Risch, and M. Christensen, Effects of Note-Taking Format and Study Technique on Recall and Relational Performance, Contemporary Educational Psychology, 20, pp. 172-187, 1995. 3. V. Slotte and K. Lonka, Review and Process Effects of Spontaneous Note-Taking on Text Comprehension, Contemporary Educational Psychology, 24, pp. 1-20, 1999. 4. C. Bereiter and M. Scardamalia, The Psychology of Written Composition, Lawrence Erlbaum Associates, Hillsdale, New Jersey, 1987.


329

5. K. P. O’Hara and A. Sellen, A Comparison of Reading Paper and On-Line Documents, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Association for Computing Machinery, Atlanta, Georgia, pp. 335-342, 1997. 6. J. Hollan, E. Hutchins, and D. Kirsh, Distributed Cognition: Toward a New Foundation for Human-Computer Interaction Research, ACM Transactions on Computer-Human Interaction, 7:2, pp. 174-196, 2000. 7. B. Schilit, G. Golovchinsky, and M. N. Price, Beyond Paper: Supporting Active Reading with Free Form Digital Ink Annotations, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Los Angeles, California, Association for Computing Machinery, New York, pp. 249-256, 1998. 8. J. L. Wolfe, Annotation Technologies: A Software and Research Review, Computers and Composition, 19, pp. 471-497, 2002. 9. J. L. Wolfe and C. M. Neuwirth, From the Margins to the Center. The Future of Annotation, Journal of Business and Technical Communication, 15:3, pp. 333-371, 2001.

Other Articles On Communication By These Authors M. Hall, M. de Jong, and M. Steehouder, Cultural Differences and Usability Evaluation: Individualistic and Collectivistic Participants Compared, Technical Communication, 51:4, pp. 489-503, 2004. J. Karreman and M. Steehouder, Some Effects of System Information in Instructions for Use, IEEE Transactions on Professional Communication, 57:1, pp. 34-43, 2004. M. Coney and M. Steehouder, Role Playing on the Web. Guidelines for Designing and Evaluating Personas Online, Technical Communication, 47:3, pp. 327-340, 2000. Th. van der Geest and N. Loorbach, Testing the Visual Consistency of Web Sites, Technical Communication, 52:1, pp. 27-36, 2005. B. Lehman, J. Nduna, T. van der Geest, and C. Winberg, A Meta-Analysis of the Teaching of Technical Writing to Students for Who English is Not a First Language, poster presented at the College Composition and Communication Conference, San Francisco, March 11-15, 2005. Th. van der Geest, Online Reading and Note-Taking, Information Design Journal + Document Design, 12:1, pp. 19-23, 2004.

Direct reprint requests to: Mark S. Melenhorst University of Twente Faculty of Behavioral Science Department of Communication Studies P.O. Box 217 7500 AE Enschede The Netherlands e-mail: [email protected]