Proceedings of the ACM Conference on Human Factors in Computing Systems (Boston, April 24-28, 1994).
The Cost-of-Knowledge Characteristic Function: Display Evaluation for Direct-Walk Dynamic Information Visualizations Stuart K. Card, Peter Pirolli, and Jock D. Mackinlay Xerox Palo Alto Research Center 3333 Coyote Hill Road Palo Alto, California 94304 E-mail:
[email protected],
[email protected],
[email protected]
ABSTRACT
In this paper we present a method, the Cost-of-Knowledge Characteristic Function, for characterizing information access from dynamic displays. The paper works out this method for a simple, but important, class of dynamic displays called direct-walk interactive information visualizations, in which information is accessed through a sequence of mouse selections and key selections. The method is used to characterize a simple calendar task for an application of the Information Visualizer, to compute the changes in characterization as the result of possible program variants, and to conduct empirical comparison between different systems with the same function. KEYWORDS: Information visualization, dynamic displays, methodology, evaluation, 3D user interfaces, Information Visualizer. INTRODUCTION
The personal computer is changing from a device into which information is mainly put for authoring or analysis (e.g., desktop publishing) to a device from which information is mainly accessed (e.g., CD-ROM encyclopedias and on-line data services). This trend can only increase as the national data superhighway comes into being and as local memory costs continue to drop. Tennant and Heilmeier [8] estimate, for example, that the amount of information available through one’s computer by 1995 will be more than 10,000 times greater than the information available at the time of their writing, about 1991. A challenge of the 1990s is to develop ways of
using emerging technologies to manage the complexity inherent in accessing and utilizing such vast quantities of information. A key observation is that information in an information system has a cost structure, that is, a set of different costs for the information in different parts of the system [3]. Information retrieval and other information-handling systems reorganize this cost structure of information relative to some task. For example, retrieving paper documents from filing cabinets and placing them on a desk reduces the time costs substantially for a task in which the documents must be repeatedly referenced. A goal in designing information access systems is to rearrange this cost structure in beneficial ways. In previous papers, we have reported the designs of experimental programming systems whose interfaces were designed for this purpose [3][4]. But methods are needed for conceptualizing and measuring the abilities of these and other systems to bring about the desired result. In this paper we propose an abstraction, the Cost-ofKnowledge Characteristic Function, for characterizing the effect of the design of a dynamic display or humancomputer dialogue on the cost structure of information. The purpose of the paper is to work out and measure empirically this abstraction for a simple case of information access, "direct-walk" information visualizations, by which we mean the use of the mouse to point to and gesture over displays of an information structure so as to navigate from one place in that structure to another. THE COST-OF-KNOWLEDGE CHARACTERISTIC FUNCTION
We have argued that, at least in a world of abundant information, but scarce time, the fundamental information
access task is not finding information, but the optimal use of a person’s scarce time in gaining information [7]. That is, the important thing is to maximize information benefits per unit cost (The unit of cost considered in this article is primarily the user’s time). To aid in doing this, we need to know how much additional information becomes available for each additional amount of time expended. We call the curve this notion defines the Cost-ofKnowledge Characteristic Function.
NUMBER OF DOCS
Fig. 1 shows a schematic plot of this function. Curve A shows a hypothetical office in which information is hierarchically arranged: Small amounts of information are placed on the low-cost access desktop, larger amounts of information are placed in the more expensive-to-access, but more capacious, desk file drawer, and large quantities of information are in the file cabinet. For simplicity, it is assumed that the average time to access the information within each of these categories is the same, and we have ignored the staircase function produced by the fact that the repositories are of discrete sizes. CURVE B (HYPOTHETICAL IMPROVEMENT) (a)
Filing Cabinet
We define a direct walk to be a task in which a user navigates from a starting point to a goal point in an information structure by a series of mouse points or other direct-manipulation methods. Examples would be the series of mouse clicks and button choices required to operate the Macintosh hierarchical file system or a typical HyperCard stack or many help systems. The essence of a direct-walk is that an information structure is displayed and the user points to, flies to, or gestures over some part of this visible structure resulting in a new display at which time the cycle is repeated until the goal is found. We do not believe the Cost of Knowledge Characteristic Function is limited to this class of dialogues. We simply choose these dialogues as a simple case on which to develop the methodology. In the remainder of this paper, we perform a series of analyses as exercises to establish the feasibility and utility of this concept. It should be noted that the emphasis is on the development of the methodology itself and that while useful information may be revealed about the systems we analyze (after all, that is the purpose!), our analyses do not constitute a complete evaluation of these systems, since that would require considering what other figures of merit might be relevant as well. EMPIRICAL CHARACTERIZATION OF A SYSTEM
(b)
CURVE A (HIERARCHICAL OFFICE)
Desk File Desk
TIME COST (s) Fig. 1. Cost of Knowledge Characteristic Function
Now suppose that we were able to invent some device or procedure that improves information access. The Cost of Knowledge Characteristic Function should show that improvement by having at least some portion above the original curve (e.g., Curve B in Fig. 1). Notice that we can harvest this benefit in two ways, as shown by the arrows. If we keep the time cost the same, we could access more documents (arrow a). On the other hand, if we keep the number of documents the same, we can access them for a lower cost (arrow b). In this way, the Cost-of-Knowledge Characteristic Function is intended to help us reason through more complicated consequences of system improvements than just thinking that one system is better than another. Direct Manipulation Walk of Information Structures
We now attempt to measure the function in Fig. 1 for an actual system. To keep the analysis simple, we investigate a basic, but important, information access task, which we dub a direct walk of an information structure.
In our first analysis, we empirically measure the function of Fig. 1 for one particular task done by users of the Spiral Calendar, an application of the Information Visualizer [6] developed by Mackinlay and DeLine [5]. A user can access the schedule for various calendar dates by selecting objects that represent the appropriate period, then selecting the unit within that period, and so on. For example, to select June 4, 1982, the user would select the year 1982 within the Decade 1980-1989 object, this would cause the Year 1982 object to fly up and grow large. Then the user would select June from that Year object. This would cause the June Month object to fly up. The user would select 4 from the June month object causing the containing Week object to fly up. Finally, the user would select 4 from the week object, causing the daily schedule for that day to appear. There could be fewer selections or more selections depending on the condition of the display from which the user started.
experience. Two had never used the Information Visualizer, one was one of the designers of the Spiral Calendar. Procedures
As a warm-up, the user first performed a set of 11 accesses to dates different from, but similar to in time, those actually used in the experiment. These served to help the user assimilate the procedures of the experiment, to learn how to operate the Spiral Calendar, and to ask questions. As a limitation on the prototype, only data from 1993 was actually contained in the database and the calendars said 1993, regardless of the simulated year. Users were simply told of this limitation and that this data was being used to simulate a larger range of years. Fig. 2. The Spiral Calendar.
This system was measured using the following procedure: Task
The general task measured can be described as follows: A user is looking at the detailed hourly calendar for a certain day, and he or she wishes to view the daily calendar for another day. How long will it take to do this?
The set of 11 tasks was randomized into a block of trials. Each user performed 5 of these blocks, each block separately randomized, for a total of 55 trials/user. Users were allowed to take a break between blocks if they wished. On each trial, the user flipped a page in a notebook asking him or her to navigate through the calendar to the day display of a specific date. The trials were videotaped and the time measured from when the user had turned from the notebook and was facing the display until the day page was done displaying. Empirical Results
Users had to position the calendar to a set of 11 different days related to the starting date (chosen to be September 7, 1993). These days were chosen to lie on a logarithmic scale: We specifically chose such a large range because we want to understand how the interaction scales with size. Users
The results of the measurement are in Table 1, column (4). Plotting time to access a date as a function of number of days back (column 2) gives Fig. 3. As might be expected, the time required increases with the number of days back. This is obviously because there are more steps in the dialogue to reach distant dates. How much additional time can be understood by analyzing the predominant method users utilized.
The measurement was done on four users, members of the professional staff of the lab. Users varied in their Table 1. Computation of Cost of Knowledge Characteristic Function for Spiral Calendar. (1) TASK
1 2 3 4 5 6 7 8 9 10 11
(2) DAYS BACK
1 3 10 30 100 300 1000 3000 10,000 30,000 100,000 300,000 1,000,000
(3) DATE
Sep. 6, 1993 Sep. 4, 1993 Aug. 28, 1993 Aug. 8, 1993 May 30, 1993 Nov. 11, 1992 Dec. 12, 1990 Sep. 25, 1982 Apr. 22, 1966 Jul. 20, 1911 Nov. 23, 1719
(4) ACCESS TIMEa MEAN ±SD (S)
5.6±0.97 11.1±1.80 14.3±0.49 14.6±0.77 14.4±0.46 16.6±0.39 17.8±0.35 21.1±0.28 21.2±0.69 20.7±0.46 24.3±1.40
(5) METHOD
(6) N O. CYCLES
(7) COST METHOD TIME (S)
(8) COST FROM
(9) SELEC-
MODELb (S)
BRANCH FACTOR
TION
1 7 4.25
(10) ACCESS NO. OF DAYS ACCESSIBLE
Day Week Month
1 2 3
5.6±0.97 11.1±1.80 14.4±0.36
6.9 10.4 14.0
1 7 30
Year
4
17.2±0.35
17.5
12
365
Decade
5
21.0±0.31
21.0
10
3,562
Century Millennium Era
6 7 8
24.3±1.5
24.6 28.1 31.6
10 10 10
36,525 365,250 3,652,500
a
Each mean is based on 4 users x 5 repetitions = 20 data points. b Computed using Time = 3.346 + 3.535 * NCycles Methods
The cost in time for accessing some date can be characterized in terms of the major methods available to users. Let us take the most extreme case, accessing the date November 23, 1719. A GOMS model [2] for this procedure would be CENTURY-METHOD = GOAL: DO-TASK GOAL: GET-DATE TURN-TO[MANUSCRIPT] GET-DATE GOAL: ACCESS-DAY-CALENDAR GET-YEAR • • • if necessary GOAL: SELECT-CENTURY (1700's) POINT-TO (Century=1700-1790s)) Century-display GET-YEAR • • • if necessary GOAL: SELECT-DECADE (1710's) POINT-TO (1710-1719)) Decade-display GET-YEAR • • • if necessary GOAL: SELECT-YEAR: (1719) POINT-TO (1719)) Year-display GET-MONTH • • • if necessary GOAL: SELECT-MONTH: (November) POINT-TO (November)) Month-display GET-DAY • • • if necessary GOAL: SELECT-WEEK: [??] POINT-TO [23] Week-display GET-DAY • • • if necessary GOAL: SELECT-DAY: [23] POINT-TO (23)) Day-display
Neglecting the initial part of this method that has to do with our experimental procedure, the method can be summarized in terms of seven cycles of pointing and display–one each for Century, Decade, Year, Month, Week, and Day. Other methods used in our measurement are the same, except that the larger units of time, such as the century, or the decade, or even the month, are eliminated if the date is close enough. To make the methods easy to talk about, we name them by the largest unit of time selected. The method for each task is listed in column (5) of Table 1.
30
Century 25
20
Decade Year
15
Month 10
Week Day
5 0 1
100
10000
1000000
DAYS BACK Fig. 3. Time as a function of number of days back.
Model for Access Time
On Fig. 3, we have circled those data points that are done with the same method and collapsed the data cells to one data point per method in Table 1, columns (5) and after. At this point, we can use our data to fit a simple model in order to characterize the direct walk time. To a first order of analysis, the time (in seconds) to select a date is just proportional to the number of cycles required (Fig. 4). By a regression analysis, Time to Access = 3.3 + 3.5 * NCycles.
(1)
This model allows us to give a smoother characterization than the individual data points and we list the model times in Table 1, column (8). Number of elements accessed
Finally, we compute the number of days that it is possible to reach in less than or equal to a certain amount of time. This is done by computing the number of days existing within the different periods serviced by the methods in Table 1. We list the result in Table 1, column (10). With the model, we can make reasonable estimates of what the data would be for other dates not actually measured (e.g., 1,000,000 days distant). Cost-of-Knowledge Characteristic Function
We can now plot the Cost-of-Knowledge Characteristic Function of Fig. 1 by plotting the number of elements accessible (column 10) as a function of the cost in time (column 8) in Table 1. This is done in Fig. 5 (Curve A: Spiral Calendar). This graph, the first actual calculation of this concept we have been able to achieve, shows how,
as would be expected in a reasonable system, the small amounts of knowledge can be accessed quickly, larger amounts of knowledge require longer times. For simplicity we have omitted the stair-case detail of the curve that would track abrupt representation shifts (e.g., from month to year). The metric is roughly linear, in semi-log coordinates, indicating that the items accessed increase exponentially. This linear shape of the curve may be a natural form for describing accessibility with cost for well-designed systems.
30 25 20 15 10 5 0 0
2
4
6
NUMBER OF SELECT-DISPLAY CYCLES
Fig. 4. Time as a function of the number of selection-display cycles. COMPUTATIONAL CHARACTERIZATION OF DESIGN IMPROVEMENTS
With the Cost-of-Knowledge Characteristic Function conceptually in hand, we can now use it to help reason
about and discover variants in the system design. Let us discuss briefly the effect of some design changes on the system measured in the last section. One possibility is to speed up the system response. We notice (from Eq. 1) that the time per picking cycle is on the order of 3.5 s. This would seem to be relatively long. Assuming a mouse point of around 1 s [2] and a response animation time of 1 s [3] suggests that 2 s/cycle should be possible The discrepancy suggests re-examination of the animation algorithms. We now replot (as recalculated from Eq. 1) the Cost of Knowledge Characteristic Function that would have resulted from a faster 2 second user-action cycle (Curve B in Fig. 5). The curve is tilted upwards indicating an improvement in the system (if it can be achieved computationally and if some other phenomenon does not intervene). Notice that in this case we have plugged the results from previous models into our new model to cascade the speed with which we can think about design variants. Another possibility is to eliminate the Week display (or probably better, to integrate it into the same display as the Day) in order to reduce the number of action-display cycles required. If we were to do just this, then the Cost of Knowledge Characteristic Function Fig. 5 grows a bump at the bottom (Curve C), because of the larger branching factor at the Month level. In Curve D, we combine both variants. With the Cost of Knowledge Characteristic Function, we are led to view variants in terms of their effect on the coststructure of access rather than just on a single point.
10000000 C: No Week B: 2s A: Spiral Calendar
1000000
D: Both E: CM
100000 10000
1000 100 10
1 0
20
40
60
80
100
120
COST (s)
Fig. 5. Cost of Knowledge Characteristic Function for several variants of the Spiral Calendar and the CM calendar program. EMPIRICAL COMPARISON AMONG SYSTEMS
In addition to using calculations to do paper comparisons Table 2. Computation of Cost of Knowledge Characteristic Function for Sun CM Calendar. (1) TASK
(2) COST TASK TIME MEAN±SD (S)
(3) METHOD
(4) COST METHOD TIME MEAN±SD (S)
(5) COST FROM MODELb
2.3±0.34
2.8
(S)
(6) ACCESS NO. OF DAYS ACCESSIBLE
1
2.4±0.56
2
2.2±0.25
3
5.0±0.60
4
4.3±0.53
5
9.0±0.90
Year
9.0±0.55
8.1
365
6
8.7±0.90
Year2
8.7±0.90
4.5
731
7
10.03±0.80
Year4
10.0±0.80
10.2
1461
8
12.8±0.70
Year11
12.8±0.70
12.8
4018
a
Month
among system variants before they are built, we can also use the Cost of Knowledge Characteristic Function to do comparisons between competing systems. As an example, we repeat our measurement and calculation, but this time for the Sun calendar program CM. Procedure
The tasks and the procedure were exactly the same as for the Iris Spiral Calendar above. Four users participated in this measurement, two of these were users measured in the previous Spiral Calendar. The results are shown in Table 2, comparable to Table 1.
30
The GOMS analysis [2] of the methods is summarized in short-hand form in Table 3 column (4). In this analysis, there are three operators: Month2
4.6±0.14
4.2
61
Each mean is based on 4 users x 5 repetitions = 20 data points. b Time = 1.340 + 3.889 m + 1.412 P + 0-.362 B.
m P B
point, menu pull-down, and select point and select press a button (not including pointing)
These operators include the system response time of this particular system. They do not count the mental preparation time [2]. The time taken by the different methods is given in Table 3, which analyzes the predominant method used. (Remember, methods were restricted to be direct walk). Table 3. Analysis of methods for CM calendar task.
METHOD DISPLA
ACTION
ANALYSIS
Select Date
P
Y
Month Month2
Year
Year(n)
Month display Month display Month display Year Month Month display Year Year Month
Select PREV button 2P Select Day Select year on m + 2P VIEW pulldown menu Select Month Select Day Select year on m + 3P + VIEW pulldown (n-2)B menu Select PREV button (n-1) times Select Month Select Day
In this case, we are not trying to predict method times, but to analyze them, so, as before we use regression analysis to assign numbers to the operators. The regression gives Time = 1.3 + 3.9 m + 1.4 P + 0.36 B .
(2)
This equation (the equivalent of Eq. 1 for CM) is used to determine a smoother version of the Cost of Knowledge Characteristic Function in Table 2, column (5). Finally, the number of days accessible within a given iso-cost contour is determined from an inspection of the program displays and summarized in Table 2, column (6). As before, we plot number of items accessible (column 6) against cost (column 5) in Curve E in Fig. 5. A comparison between the Cost of Knowledge functions in Fig. 5 shows a rather dramatic contrast. The Spiral Calendar uses a uniform direct-walk method to access all dates and costs go up logarithmically. The CM program is at an advantage for lower numbers of items, but costs radically increase for numbers of items over about 10,000. (Remember, however, we are not characterizing the programs themselves, only certain methods. For example, we are not considering methods such as typing in the date directly that would be useful for larger numbers of items.)
is to weight items by a probability density function describing their frequency of use. Anderson and Schooler [1] have shown that for many different kinds of information (e.g., news articles in the New York Times or messages in electronic mail) the probability an item D days old will be needed is given by Pr{needed|D days ago} = A / (A + DC), where where A and C are constants. For the case of electronic mail, A = .34 and C = 0.83, hence Pr{needed|D days ago} = 0.34 / (0.34 + D0.83). If we multiply this function with the Cost of Knowledge Characteristic Functions in Fig. 5 to obtain a curve expressing the expected cost of accessing different numbers of items (see Fig. 6). In the case of the calendars, it expresses the fact that the user is likely to access recent dates much more frequently. The area under the curves is related to the total costs of using the two programs and the curve also shows in what area the costs are concentrated. Fig. 6 shows that the expected cost for the user is heavily contained in the in the most recent hundred days. For this reason, the Spiral Calendar prototype tested would be more expensive to use than CM (if only the direct walk feature were considered). On the other hand, if the task involved reference to historical dates, then a different probability density function would be appropriate.
1.8 1.6 1.4 1.2 1 0.8 0.6 0.4 0.2 0
Spiral Calendar
CM 1
100
10000
1000000
100000000
NUMBER OF DAYS ACCESSED
The Spiral Calendar suffers because of its high cost intercept. Clearly there is payoff in concentrating effort at the low end of the curve to shift it to the left. The CM program suffers because users must shift through a more complex space of methods, changing methods for different regions of the space. PROBABILITY DENSITY FUNCTION
Finally, it should be noted that whereas we have determined the Cost of Knowledge Function, we have not as yet weighted the items accessed by their value. There are several such weightings, but one of the most important
Fig. 6. Expected probability-weighted associated with retrievals.
costs
CONCLUSION
This paper has introduced the Cost-of-Knowledge Characteristic Function as a method for analyzing interactive information visualizations. The metric was measured for direct-walk dialogue taken from two calendar programs. We have also done calculations showing the likely consequences for proposed variants of these systems in terms of the metric. Finally, we
introduced the next step in the development of this analysis, taking into account the frequency of access for different items of information.
available access methods is taken into account. This would be interesting to know as well as knowing what are characteristic values for the exponents.
While we have explored direct-walk information access because of its simplicity, other more complex information access dialogues should be able to be analyzed with this basic method and it should be possible to elaborate the analysis for more insight.
ACKNOWLEDGEMENTS
The purpose of the paper was to move some of the qualitative reasoning behind the design of the Information Visualizer and other information access systems closer to a measurable, computable methodology. In this way, we hope to be able to understand more precisely the consequences of design decisions in this area. Indeed, in this case, we discovered the initial design of the Spiral Calendar, while good for handling very large time periods, was under-optimized for the frequent close-to-present dates. While obvious to the designers after the measurement, it was previously under-appreciated. A number of design suggestions have ensued, many couched in terms of what was required to move various pieces of the curves in Fig. 5. It should be noted that individual differences, as indicated by the standard deviation in Tables 1 and 2, are relatively small, suggesting the results are not very sensitive to individual users. The method is therefore usable by an individual system builder with a stop watch, timing himself or herself. Indeed, a pilot of this experiment with one of the authors timing himself with a stopwatch while performing the role of the user yielded a very similar curve to that plotted in Fig. 5 (with 5 s/cycle instead of 3.5 s/cycle). This is a significant finding, because, while it is important to do formal user testing in building systems, it is also important to have inexpensive methods designers can use rapidly as they work to reduce the number of more expensive user tests required. This is similar to the way many experienced system builders now routinely perform system timings as they work. Finally, according to Fig. 5, the number of information items accessible by these systems increases logarithmically with time cost, roughly at Number of item accessible = A e0.5 t , where t is the cost of access in seconds. This is as expected, since the user has a succession of choices, each with a similar branching factor. But the cost structure for other information systems may also tend to arrange themselves logarithmically, when the user’s shift among
The Spiral Calendar was designed and implemented in the Information Visualizer by Robert DeLine (University of Virginia and Carnegie-Mellon University) as a summer project at Xerox PARC. An improved version has now been integrated into the standard Information Visualizer release by George Robertson (Xerox PARC). REFERENCES
1. Anderson, J. R. and Schooler, L. J. Reflections of the environment in memory. Psychological Science 2(6 November), 1991: 396-408. 2. Card, S. K., Moran, T. P., and Newell, A. The Psychology of Human-Computer Interaction. Hillsdale, New Jersey: Erlbaum, 1983. 3. Card, S. K., Robertson, G. G., and Mackinlay, J. D. The Information Visualizer, an information workspace. In Proceedings of CHI ’91 ACM Conference on Human Factors in Computing Systems (New Orleans, Louisiana, April 27–May 2, 1991). ACM, New York, 1991, pp. 181-188. 4. Henderson, D. A., Jr. and Card, S. K. Rooms: The use of multiple virtual workspaces to reduce space contention in a window-based graphical user interface. ACM Transactions on Graphics 5 (3, July 1986)., 211243. 5. Mackinlay, J. M. and DeLine, R. Designing calendar visualizes for the Information Visualizer. Research Report, Xerox PARC, Palo Alto. 6. Robertson, G. G., Card, S. K., and Mackinlay, J. D. Information visualization using 3D interactive animation. Communications of the ACM, 36 (4, April), 1993, 57-71. 7. Russell, D. M., Stefik, M. J., Pirolli, P., and Card, S. K. The cost structure of sensemaking. In Proceedings of CHI ’93, ACM Conference on Human Factors in Software (April 24-29, Amsterdam). New York: ACM, 1993, pp. 269-276. 8. Tennant, H. and Heilmeier, G. H. Knowledge and equality: Harnessing the tides of information abundance. In Leebaert, D. (ed.), Technology 2001: The Future of Computing and Communications. Cambridge, Massachusetts: The MIT Press, 1991.
