The Role of Cognitive Modeling in an Automated System for Understanding Bar Charts? Stephanie Elzer1 and Richard Burns2 and Sandra Carberry2 1
Dept of Computer Science, Millersville University, Millersville, PA 17551 USA
[email protected] 2 Dept of Computer Science, Univ. of Delaware, Newark, DE 19716 USA {burns,carberry}@cis.udel.edu Abstract. Information graphics (such as line graphs, bar charts, and pie charts) are an inherently visual medium which relies on a viewer’s spatial reasoning abilities to facilitate the comprehension of complex data. Our research group has developed an automated system for understanding the intended message of simple bar charts, and we are currently working to expand this system to handle more complex types of information graphics. A key component of our system is a cognitive model for estimating the effort required to perform various tasks within a given information graphic. This novel application of cognitive modeling (including spatial reasoning) within an artificial intelligence framework represents a promising example of the potential synergy between these fields.
1
Introduction
Our work involves a novel application of cognitive modeling within an artificial intelligence framework. Our research group has developed an automated system for understanding the intended message of simple bar charts, and we are currently working to expand this system to handle more complex types of information graphics. When evaluating the impact of various components of our system for understanding simple bar charts, we found that our module for estimating the perceptual effort for various tasks within a given information graphic played a critical role [1]. Information graphics rely on a viewer’s spatial reasoning abilities to facilitate the comprehension of complex data, and so our cognitive modeling component must incorporate facets of spatial cognition in order to determine which tasks are easier to perform on a given information graphic. In order to expand our work to recognize the messages conveyed by more complex graphics, we have found it necessary to also expand the depth and detail of our cognitive modeling. After briefly describing the underlying hypotheses and the application domains of our work, this paper focuses on the evolution of the cognitive modeling component within our system. ?
This material is based upon work supported by the National Science Foundation under Grant No. IIS-0534948.
2
The Importance of Understanding Information Graphics
Information graphics (such as bar charts, line graphs and pie charts) that appear in newspapers, magazines, and other popular media, are frequently utilized not only to serve as a communication medium for an analysis of data, but also to convey a specific message regarding that data. This message generally supports the overall purpose of an article, but, as we found in a corpus study, the message is usually not fully reiterated within the graphic’s caption or the surrounding context [2]. Figure 1 shows a grouped bar chart from NewsWeek. The primary message conveyed by the graphic is ostensibly that ‘the percentage of pirated software in China is much higher than in the world as a whole’. The article itself is about Microsoft’s commitment to China and the issues of pirated software. The closest the article comes to mentioning the graphic’s message is: “Ninety percent of Microsoft products used in China are pirated.” No comparison is ever made between piracy in China and the world. This example illustrates that in order to fully comprehend a multimodal document, the knowledge conveyed within information graphics cannot be ignored. Percentage of Software in Use Which is Pirated 97
1994
92
2002
49 39
China
World
Fig. 1. Graphic from NewsWeek, “Microsoft’s Cultural Revolution,” June 28, 2004. The goal of our research is to identify the message conveyed by an information graphic. Our work has two applications that we are pursuing: 1) better summarization of documents in digital libraries, and 2) access to multimodal documents for individuals with impaired eyesight. Currently, research on summarization has focused on an article’s text. Our goal is to integrate the messages conveyed by an article’s information graphics into a richer summary of a multimodal document. In the case of individuals with sight impairments, we want to provide access to an article’s information graphics by providing a brief summary (rendered via speech) of the high-level content of the graphic, centered on its overall message, and then responding to followup questions from the user.
3
Utilizing a Cognitive Model
In addition to words and phrases, language includes any deliberate action or signal (or lack of) that has a communicative intent [3]. Under this broad view, information graphics in popular media are a form of language. A graphic designer has an intended message to communicate to the viewer, and thus will use specific signals in attempting to convey that message. Our research group was the first to apply language understanding techniques to information graphics. We developed a system [4] which recognizes the intended message of a simple bar chart, where a simple bar chart is one that “displays the values of a single independent attribute and the corresponding values for a single dependent attribute” [5]. The system extracts communicative signals from a simple bar chart via image recognition [6] and uses them as evidence in a Bayesian network that hypothesizes the message conveyed by the graphic. The system was shown to have a success rate of 79.1% on a corpus of bar charts whose messages had been previously identified by human annotators. We posit that the relative effort involved in performing various tasks on a given information graphic may constitute a communicative signal because, given a set of data, the graphic designer has many alternative ways of designing a graphic. As Larkin and Simon note, information graphics that are informationally equivalent (all of the information in one graphic can also be inferred from the other) are not necessarily computationally equivalent (enabling the same inferences to be drawn quickly and easily) [7]. Peebles and Cheng [8] further observe that even in graphics that are informationally equivalent, seemingly small changes in the design of the graphic can affect viewers’ performance of graph reading tasks. Much of this can be attributed to the fact that design choices made while constructing an information graphic will facilitate some tasks more than others. Following the AutoBrief work on generating graphics to achieve communicative goals, we hypothesize that the designer chooses a design that facilitates the tasks that are important to conveying his intended message [9].
4
Modeling Task Effort in Simple Bar Charts
In order to identify the perceptual tasks that the graphic designer has best enabled in a simple bar chart, our methodology was to construct a set of rules that estimate the effort required for different perceptual tasks within a given information graphic. To develop these rules, we applied the results of research from cognitive psychology. In doing this, we constructed a model representing the relative ease or difficulty with which the viewer of a simple bar chart could complete various perceptual tasks. The goal of this model is to determine whether a task is easy or hard to perform with respect to other perceptual tasks that could be performed on an information graphic. In order to estimate the relative effort involved in performing a task, we adopted a GOMS-like approach [10], decomposing each task into a set of component tasks. Following other cognitive psychology research, we take the principal
measure of the effort involved in performing a task to be the amount of time that it takes to perform the task. Wherever possible, we utilize existing time estimates (primarily those applied in Lohse’s UCIE system) for the component tasks [11]. Our set of rules for estimating the effort of tasks in simple bar charts has been validated by our eye tracking experiments that are described in [12]. While our system for recognizing the intended message of a simple bar chart effectively utilizes a wide variety of communicative signals including highlighting, annotations, and caption information [13], we found that relative effort was the communicative signal that had the greatest impact on our system’s performance [1]. Therefore, we anticipate that the ability to model the relative effort of performing various tasks will also play a vital role as we expand our system to include grouped bar charts.
5
Modeling Task Effort in Grouped Bar Charts
Whereas the messages conveyed by simple bar charts include a simple trend (increasing, decreasing, or stable), a comparison of two entities, and the rank of an entity with respect to the other entities depicted in the graphic, the messages conveyed by grouped bar charts3 are more complex. Our analysis of a corpus consisting of approximately 100 grouped bar charts has identified 14 categories of possible messages. On occasion, it appears that multiple messages can exist. Figure 2 is a graphic from Business Week, which shows that the percentage of households with internet access increases with income for both rural and urban households and that it is less for rural areas than for urban areas at every income level. From its design, it seems that the inter-trends, internet access increasing by income (inter meaning across all groups), is the primary message of the graphic, although the intra-relationship, rural areas having less internet access than urban areas (intra meaning within a group), may be important to the overall message. As an example of a different kind of message, consider Figure 3. Its message falls into the Gap-Decreasing category: the gap between the percentage of men and women involved in illicit drug use shrunk between 1999 and 2000. To gain insight into the features that impact task effort in grouped bar charts, we performed a preliminary set of experiments with human subjects and an eye tracker, to identify fixations, measure fixation durations, and compute the time required to process different graphics. The graphics presented in the experiment varied in size, number of groups, number of bars per group, number of exceptions (bars that deviate slightly from a trend), the presence of a trend, and the type of trend (if one was present). Although we were only interested in trend tasks for this experiment, other types of graphical tasks were included, so that 3
A grouped bar chart is a bar chart that consists of two or more visually distinguishable groups of bars. Groups must share the same ontology, and bars must share the same ontology. For example, the grouped bar chart in Figure 1 has two groups (China and World) and two bars per group; parts of the universe and years are the ontologies respectively for the groups and the bars.
By Income
$10,000 − $14,999
Rural Urban
$15,000−$19,999 $20,000−$24,999 $25,000−$34,999 $35,000−$49,999
$50,000−$74,999 $75,000 PLUS 0
15
30 Percent
45
60
75
Fig. 2. Graphic from Business Week, “A Small Town Reveals America’s Digital Divide,” October 4, 1999. subjects did not become accustomed to repeating the trend task. After these initial experiments, we hypothesized that our model for estimating the relative effort of tasks in grouped bar charts would need to include factors such as size and density of the graphic, visual clutter, and exceptions to trends (bars that deviate slightly from a trend). Although the GOMS-based approach worked well for modeling relative effort in simple bar charts, it is inadequate for grouped bar charts since the tasks required have a greater cognitive aspect to them. Thus we are modeling task effort for grouped bar charts using ACT-R [14], a programmable cognitive modeling framework developed as a model of higher level cognition. For modeling perception in ACT-R, a visual module is available, and we also use EMMA, an add-on to ACT-R [15], to capture aspects of peripheral vision. Once we had developed a model which estimates the relative effort required to recognize a trend based on features in a grouped bar chart, we then performed a set of eye tracking experiments to validate this model.4 In validating our model, we have demonstrated how graphic size, density and the presence of visual clutter and exceptions affect the effort involved in recognizing a trend in a grouped bar chart. We hypothesize that our observations of eye movements and fixation patterns may also have implications with respect to the way that viewers are performing spatial reasoning while carrying out these tasks.
6
Conclusion
Information graphics are an inherently visual medium which relies on a viewer’s spatial reasoning abilities to facilitate the comprehension of complex data. In our research on methods for the automated understanding of information graphics, we have benefited from the application of results from cognitive psychology in order to estimate the effort of performing tasks within information graphics. This novel approach represents a promising example of the potential synergy between 4
More information on the model and validation experiments can be found in [16].
the fields of artificial intelligence and spatial reasoning. We believe that further understanding of models of spatial reasoning could strengthen the models within our system, and that some of our results could be applied within the discussion of strengths and weaknesses of various spatial models.
References 1. Carberry, S., Elzer, S.: Exploiting evidence analysis in plan recognition. In: Proc. of the 11th Int’l Conf. on User Modeling. (2007) 7–16 2. Carberry, S., Elzer, S., Demir, S.: Information graphics: An untapped resource for digital libraries. In: Proceedings of SIGIR 2006, Seattle, WA (2006) 3. Clark, H.: Using Language. Cambridge University Press (1996) 4. Elzer, S., Carberry, S., Zukerman, I., Chester, D., Green, N., Demir, S.: A probabilistic framework for recognizing intention in information graphics. In: Proc. of the 19th Int’l Joint Conf. on Artificial Intelligence (IJCAI). (2005) 1042–1047 5. Elzer, S., Carberry, S., Demir, S.: Communicative signals as the key to the automated understanding of simple bar charts. In: Proc. of the 4th Int’l Conf. on the Theory and Application of Diagrams. (2006) 25–39 6. Chester, D., Elzer, S.: Getting computers to see information graphics so users do not have to. In: Proc. of the 15th Int’l Symposium on Methodologies for Intelligent Systems, LNAI 3488, Springer-Verlag (2005) 660–668 7. Larkin, J.H., Simon, H.A.: Why a diagram is (sometimes) worth a thousand words. Cognitive Science 11 (1987) 65–99 8. Peebles, D., Cheng, P.C.H.: Modeling the effect of task and graphical representation on response latency in a graph reading task. Human Factors 45 (2003) 28–46 9. Kerpedjiev, S., Roth, S.F.: Mapping communicative goals into conceptual tasks to generate graphics in discourse. In: Proc. of Intelligent User Interfaces. (2000) 157–164 10. Card, S.K., Moran, T.P., Newell, A.: The Psychology of Human-Computer Interaction. Lawrence Erlbaum Associates, Inc., Hillsdale, NJ (1983) 11. Lohse, G.L.: A cognitive model for understanding graphical perception. HumanComputer Interaction 8 (1993) 353–388 12. Elzer, S., Green, N., Carberry, S., Hoffman, J.: A model of perceptual task effort for bar charts and its role in recognizing intention. User Modeling and User-Adapted Interaction 16(1) (2006) 1–30 13. Elzer, S., Carberry, S., Chester, D., Demir, S., Green, N., Zukerman, I., Trnka, K.: Exploring and exploiting the limited utility of captions in recognizing intention in information graphics. In: Proc. of the 43rd Annual Meeting of the Assoc. for Computational Linguistics (ACL). (2005) 223–230 14. Anderson, J.R.: Problem solving and learning. American Psychologist 48 (1993) 35–44 15. Salvucci, D.D.: An integrated model of eye movements and visual encoding. Cognitive Systems Research 1 (2001) 201–220 16. Burns, R., Elzer, S., Carberry, S.: Estimating effort for trend messages in grouped bar charts. In: Proc. of the 5th Int’l Conference on the Theory and Application of Diagrams. (to appear)
