The Creativity Support Index

CHI 2009 ~ Spotlight on Works in Progress ~ Session 1

April 4-9, 2009 ~ Boston, MA, USA

The Creativity Support Index Erin A. Carroll

Celine Latulipe

Abstract

University of North Carolina

University of North Carolina

at Charlotte

at Charlotte

Charlotte, NC 28233 USA

Charlotte, NC 28233 USA

Dept. of Software &

Dept. of Software &

Information Systems

Information Systems

We present a draft survey tool called the Creativity Support Index (CSI). The CSI is similar to the NASA Task Load Index Survey but is designed specifically for evaluating creativity support tools, based on concepts and theories from creativity research.

[email protected]

[email protected]

Keywords Creativity, creativity evaluation, standardized surveys, engagement, flow, expressiveness, play.

ACM Classification Keywords H.5.2 Evaluation/Methodology User interfaces

Introduction

Copyright is held by the author/owner(s). CHI 2009, April 4 – 9, 2009, Boston, MA, USA ACM 978-1-60558-247-4/09/04.

While the concept of creativity is something that people understand intuitively, it is not easily defined, and it is therefore difficult to measure. If creativity itself is difficult to measure, the ability of a tool to support, promote, or enable creativity will also be difficult to measure. Due to the challenges of measuring creativity, there are many different approaches to its evaluation. Most commonly, Creativity Support Tools (CSTs) are evaluated using qualitative methods, such as observation and interview techniques. While these techniques can be extremely rewarding, they are also time-consuming and expensive. The use of qualitative methods also makes it more difficult to compare results between tools and publish statistically significant results

4009


that please conference and journal reviewers [10]. Due to these constraints, many HCI researchers use a blend of qualitative and quantitative methods.


time pressure, and frustration, focusing on tasks that have clearly defined objectives [3].

One promising method for quantitative analysis of CSTs is biometrics, which have been used successfully to measure entertainment applications [6]. However, at present, such methods tend to be somewhat invasive and expensive compared to survey methods. Even though surveys are limited to self-report, we are interested in them because they are inexpensive and convenient. Unfortunately, there are no standardized surveys designed for measuring CSTs, so surveys are either borrowed from other disciplines, or are customized for the activity, and therefore not general enough to be used by other researchers. We have designed a measurement tool, the Creativity Support Index (CSI), which we hope will be adopted in the future as a standard survey for evaluating CSTs. The goal of the CSI is to aid researchers in evaluating CSTs, as a complement to qualitative methods. In this paper, we explain the difficulty in evaluating tools that support creative expression and open-ended problem solving. We give a brief summary of the work that has been done in the area of evaluating creativity and Creativity Support Tools. We present the Creativity Support Index that we propose as a new standard tool and describe how we would expect the survey to be used.

The NASA Task Load Index The Creativity Support Index that we present was modeled after the NASA Task Load Index (TLX), since it is a familiar tool in the HCI community (See Figure 1). The NASA TLX measures factors like work performance,

Figure 1. The NASA Task Load Index, factor ratings.

While the NASA TLX is a good tool for evaluating productivity applications, creative activities differ significantly from typical productivity work. For example, artistic individuals do not necessarily set themselves time restrictions for their creative work, so

4010


temporal demand is likely to be a confusing question for an artist. If an artist spends a long time on an activity, that may indicate a high level of creative engagement, rather than an ineffective tool or interface. Similarly, mental demand may be high due to the creative energy and intensity of being fully engaged in a creative work, but that is reflective of the work, not necessarily an indication of a poor tool.

Sample reporting of individual NASA Task Load Index factors from a 3condition within-subjects study of interaction techniques for manipulating digital images [5].

While the metrics of the NASA TLX are clearly inappropriate for the evaluation of CSTs, the NASA TLX has a number of excellent features: it is easy for researchers to use, it is relatively quick for participants to fill out (making it suitable for within-subjects studies, where it has to be filled out for each study condition), and finally, it generates both categorical ratings (see sidebar) and a single overall rating of workload. The fact that the NASA TLX is a standardized survey tool also makes it appealing because researchers can report results based on the NASA TLX without having to explain the details of the measurement tool, and the results reported are meaningful to other researchers. Thus, while the NASA TLX provides a good model of a standardized survey for evaluating interfaces and tools, it is obvious that the particular factors measured in the NASA TLX are not appropriate for evaluating CSTs. Our goal was to create a survey that provides the same benefits to researchers as the NASA TLX but is appropriate for evaluating how well Creativity Support Tools actually support creative work.

Creativity Theory Researchers have been thinking about creativity for many years, and there are a number of promising theories, ideas, and concepts that we can draw upon


[11]. One of the most respected theories is by wellknown psychologist Mihaly Csikszentmihalyi and focuses on the concept of enjoyment. He studied people of all ages and backgrounds that enjoyed their occupation or hobby but were not rewarded with money or fame [2]. He found that their enjoyment came from deep but effortless involvement, calling this optimal experience or flow. There are nine elements of flow, which also correspond to the flow of creativity: 1. 2. 3. 4. 5. 6. 7. 8. 9.

There are clear goals every step of the way. There is immediate feedback to one’s actions. There is a balance between challenges and skills. Action and awareness are merged. Distractions are excluded from consciousness. There is no worry of failure. Self-consciousness disappears. The sense of time becomes distorted The activity becomes autotelic, [the meaning of the activity is within itself].

Not only is creativity related to flow, but we also believe there is a connection to play. Specifically, we believe that creative activities are more similar to play than work tasks. Interestingly, many of the concepts of play intersect with the elements of flow. The most relevant play theory is that of Rubin et al., who explain play as disposition [9]. They describe six factors that define play: intrinsic motivation, attention to means rather than ends, active engagement of the individual, freedom from external rules, nonliterality, and behavior dominated more by the individual than by the environment.

4011


Read et al studied play and documented three dimensions of children’s fun – expectations, engagement, and endurability [8]. Of particular interest is their use of engagement, which was measured by observing facial expressions, and endurability, which was the willingness to continue or repeat an activity.


to use a new piece of software. We have designed the Creativity Support Index (CSI) as a survey tool, using all of these concepts and factors related to creativity.

The Creativity Support Index

The concept of engagement seems particularly important, as it is used by Rubin et al and Read et al and it relates to many of the flow elements described by Csikszentmihalyi. For example, engagement could be responsible for distractions being excluded from consciousness, from self-consciousness disappearing, and from the sense of time being distorted. Endurability is also related to engagement, since a desire to repeat the activity reflects a person’s enjoyment of the activity. In addition to the theories from psychology and early childhood education, Schneiderman provides the HCI community with four principles for designing CSTs: support exploratory research, enable collaboration, provide rich history-keeping and design with ‘low thresholds, high ceilings, and wide walls’ [10]. We are particularly interested in supporting exploratory search because we believe exploration is a component of play and is related to idea of flow. While early play research suggests that play and exploration are distinguishable behavior [4], we believe they are strongly related. Theory from the Montessori educational philosophy suggests that exploration of the world is the main motivation for play in small children, and that this exploration is the ‘work of the child’ [6]. We also see from common parlance that the concept of exploration is often linked with play, as in the phrase “I’m just playing around with it”, used to describe learning how

Figure 2: The Creativity Support Index

Given the many concepts, ideas, categories, and definitions used to describe people engaged creatively, it’s possible to create a very long survey that addresses all of these issues. However, such a survey would be tedious for participants and especially difficult to use in a within-subjects study. We used a card sorting

4012



exercise with a small number of individuals to see if it was possible to place the many terms related to creativity theory into some general categories. We ended up with six main categories. Using these categories as the main variables that contribute to supporting creative work, we have designed the Creativity Support Index shown in Figure 1: a short survey consisting of six statements about a participant’s activity with Likert scales from 1 to 20 (Figure 2). In the survey, participants are asked to indicate their level of agreement with each statement. Each statement is given a factor keyword, which is referred to in the second part of the survey. The six factors are Exploration, Collaboration, Engagement, Effort/Reward Tradeoff, Tool Transparency, and Expressiveness. Table 1 synthesizes how the various definitions, concepts, and theories of creativity and play relate to the six main evaluation factors in our Creativity Support Index.

Second part of Creativity Support Index: Factor Rankings.

In the second part of the survey, each factor will be compared against the other five, to assess the relative importance of these factors to the participant for the particular activity under study (See Sidebar). For example, they will rate whether Tool Transparency or Exploration was more important to them while doing the activity. Because each category needs to be compared with every other category, we have an additive factorial combination of questions (so for the 6 factors, there are 15 comparison questions). We realize that the factor comparison could be tedious for a participant. However, we believe that the factors in the CSI should be ranked independently of the actual conditions, and

Table 1. How creativity and play factors in the literature relate to the Creativity Support Index factors.

4013


Sample CSI Calculation Participant Ratings Exploration: 15 Collaboration: 8 Engagement: 17 Effort/Reward Tradeoff: 15 Tool Transparency: 13 Expressiveness: 16 Factor Comparison Counts Exploration: 3 Collaboration: 0 Engagement: 4 Effort/Reward Tradeoff: 3 Tool Transparency: 2 Expressiveness: 3 CSI Calculation: CSI = (15*3 + 8*0 + 17*4 + 15*3 + 13*2 + 16*3)/3 CSI = 77.3


therefore the factor comparison part of the survey can be administered just once, at the end of a study.

References

To calculate the creativity support index, multiply each factor ranking from the first part of the survey (which is out of 20) by the count for that factor in the second part the survey and sum these together. Then, divide that sum by 3 to arrive at an index out of 100. The sidebar shows a sample calculation of the CSI. The best results are obviously those that are averaged over many users. The rankings from the first part of the survey can also be useful to look at for a particular individual in a within-subjects study, to see how that individual rated the different conditions with respect each factor. These individual factor results are also interesting when averaged across study participants (similar to the sidebar on page 2 which shows an example of factor rankings from the NASA TLX).

[2] M. Csikszentmihalyi. Creativity: Flow and the Psychology of Discovery and Invention. Harper Perrennial, 1997.

Future Work

[1] E.G. Carmines and R.A. Zeller. Reliability and Validity Assessment. SAGE Publications, 1979.

[3] S.G. Hart and L.E. Staveland. Development of the NASA TLX (Task Load Index): Results of empirical and theoretical research. In P. Hancock and N. Meshkati, editors, Human Mental Workoad, pages 239-250. North Holland Press, Amsterdam, 1988. [4] C. Hutt. Exploration and play in children. In Play, exploration, and territory in mammals, volume 18, pages 61-68, London, 1966. Symposium of the Zoological Society. [5] C. Latulipe, I. Bell, C. Clarke and C. Kaplan. symTone: Two-Handed Manipulation of Tone Reproduction Curves. GI’06 Proceedings, pages 9-16. [6] R.L. Mandryk. Objectively evaluating entertainment technology. In CHI 2004, pages 1057-1058. ACM Press, 2004.

At present, we are in the process of validating the CSI factors using a process similar to the NASA TLX in its early stages, when participants were instructed to rate factors that were equivalent to workload, similar to workload, or completely unrelated [3]. In the future, we plan to focus on the reliability and validity of the CSI using the test-retest method and criterion-related validity [1].

[7] M. Montessori. The Absorbent Mind. Henry Holt and Company, New York, 1995.

Conclusion

[10] B. Schneiderman. Creativity support tools: Accelerating discovery and innovation. Communications of the ACM, 50(12):20-32, 2007.

We have presented a new survey tool for evaluating the effectiveness of CSTs in supporting creative work. We believe this survey provides a strong starting point, and we hope it will be of use to researchers. With feedback from others, we expect to present the survey formally in a final version within the next few years.

[8] J. Read, S. MacFarlane, and C. Casey. Endurability, engagement, and expectations: Measuring children’s fun. In Interaction Design and Children, pages 53-64, Shaker Publishing, 2002. [9] K. Rubin, G. Fein, and B. Vandenberg. Play. In P. Mussen and E. Hetherington, editors, Handbook of Child Psychology, pages 693-774. Wiley, New York, 1983.

[11] B. Schneiderman, G. Fischer, M. Czerwinski, B. Myers, and M. Resnick. Creativity Support tools: Report from a US National Science Foundation sponsored workshop. International Journal of Human-Computer Interaction, 20(2):61-67, 2006.

4014