Temporal integration of interactive technology in ... - Semantic Scholar

Temporal Integration of Interactive Technology in Dance: Creative Process Impacts Sybil Huskey and Melissa Word Department of Dance University of North Carolina at Charlotte {sdhuskey, mword}@uncc.edu

Celine Latulipe, David Wilson and Berto Gonzalez HCILab University of North Carolina at Charlotte {clatulip, davils, agonza32}@uncc.edu

clear temporal process that is followed to move the production to opening night readiness. We liken the performance production process to the flowering of a plant: for the most exciting bloom, all of the necessary elements have to be provided at the right time.

ABSTRACT

Interdisciplinary projects bridging technology and the arts face challenges across many dimensions. In this paper, we reflect on the issue of temporal constraints on integrations of interactive technology within dance production. We describe lessons learned and illustrate practical impacts on stakeholders, creative process and outcomes from over three years of experience in developing dance productions at our university. Specifically, we reflect on perceptions of how the timing of the technology integrations impacted the creativity of the stakeholders involved. We separate results that are idiosyncratic to our project and highlight those that we expect are more generally applicable to other art and technology projects. Finally, we provide a set of temporal integration considerations that others can use to assess when it is best to integrate technology into an arts project.

The goal of a dance production, as with any other art, is the creative exploration of an idea. Within dance, this exploration takes place through the choices made regarding choreographic expression, musical accompaniment, costuming, lighting, scenic elements, and props [8, 16, 14, 7]. Moreover, for our project and others, these choices also consider interactive technology. A single dance production is thus a pinpoint in a large, multi-dimensional space of possible design choices. Each of these dimensions may itself have multiple facets: what to use, how to use it, and when to bring it into the production process. While choreography is the cornerstone of the dance production and a rehearsal schedule anchors the process timeline, the other production elements can vary in the timing of their integration into the process.

Author Keywords

interdisciplinary integration, arts, creativity, dance, temporal effects

The production process in dance is a generative dialogue between the varying ‘actors’ of movement, sound, lighting, props, costuming, scenic elements and digital effects, though not all of these actors play a role in every dance. The dance production ‘conversation’ that takes place between all of these ‘actors’ is determined by the timing of entry of each — if the accompanying soundtrack or orchestra is brought in towards the end of a dance production, it has a more limited ability to impact the choreographic development. If props are available for use at the beginning of the process, they are more likely to be well-integrated into the choreography. Thus, the timing of entry of different elements into an arts production impacts the creative process in significant ways. We call these phenomena ‘temporal integration effects’ and in this paper we focus on the temporal integration effects of technology on dance.

ACM Classification Keywords

J.5 Computer Applications: Arts and Humanities—Performing Arts General Terms

Human Factors. INTRODUCTION

There are long-term research projects that cross many disciplinary boundaries, and how the inter-disciplinary integration occurs varies according to the needs of the component disciplines. In performing arts productions that involve technology, the question of temporal integration arises. Performing arts productions are complex and temporally driven. Because of the deadline imposed by ‘opening night’ there is a

At UNC Charlotte, the Dance.Draw project has been integrating technology and dance over the past three years. Our interdisciplinary team has been involved in six different dance production cycles, integrating various technologies, at various stages, into dance productions [10, 12, 11]. The different technology integrations have focused on sensing dancer movement to drive interactive visualizations that

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. C&C’11, November 3–6, 2011, Atlanta, Georgia, USA. Copyright 2011 ACM 978-1-4503-0820-5/11/11...$10.00.

107

by the built-in lighting, sound and stage systems. This concept is illustrated by the diagram in Figure 1. These situated technologies prescribe the timing of their integration with the arts production: when the piece moves from a rehearsal space onto the stage, these technologies are integrated into the production. While the production crew (lighting designer, scenic designer, sound designer) may have visited the piece in rehearsal ahead of time to see the production and plan the effects, the majority of this type of technological integration happens once the piece moves into the theater. In most cases, this means that there is only a week or two in which to convert the theoretically planned effects into actual working effects. The fixed timeline for integration of traditional theater technologies is an important consideration for temporal integration of less conventional digital effects: it is very stressful to leave all of the technological integration to the last week or two of a production cycle. This also means that the choreographer or director may have difficulty visualizing the final production effects early on in the cycle, and this could have a negative impact on decision making.

Figure 1. The use of built-in technology systems in a theater space constrains what is possible for a temporal arts production, and the production can only make use of what is available.

are projected as part of the choreography. In this paper we address the question: If technology is to be integrated into an arts production and play an important role, during which part of the production process should the technology enter? We do not attempt to provide a correct answer, as that will vary between projects and circumstances. Rather, we reflect on the impact of bringing in technology at various points in the production process, based on our experience in dance production. We show that the timing of the technology integration has considerable impacts on the creative process, and we expect that these lessons learned will be useful to others.

Interactive Technologies in Dance

There is a significant amount of work, both within academia and in the professional dance world, that involves the integration of interactive, reactive or non-active multimedia technology in dance productions [5, 6, 13]. This is the subdomain that is of special interest to us and that is causing excitement in dance communities world-wide, as seen by the very active communities on sites such as dance-tech.net. Various research groups have been working on ways to capture the movement of dancers and turn that movement into digital interaction [9, 1]. While there have been interesting reports on this work, none have focused attention on the temporal aspects of technological integration in the dance production process. There has also been significant work in the creativity research community about the integration of art and technology in general, such as the work by Edmonds and Candy [4]. However, to our knowledge there has not been a thorough, methodical investigation of the effects of the temporality of technology integration into the arts in general or dance in particular.

This paper is structured as follows: first we discuss previous work in the study of art and technology integration, highlighting the temporal aspects. Next, we describe our experiences in our interdisciplinary dance project outlining the impacts of the timing of the technology integration for each production cycle. These experiences have been gathered through a combination of design team reflection, interviews with choreographers and post-performance focus groups with dancers. Based on our experiences and lessons learned, we develop a set of general considerations for the temporal integration of technology that we expect will be relevant to other researchers, artists and technologists considering such cross-disciplinary work. BACKGROUND Technology Integration in Traditional Dance Processes

To say that bringing digital interactive effects into dance is the ‘introduction of technology’ belies the incredible amount of technology that is already in use in current performing arts processes. Much of modern stage mechanics, such as the effects used in shows by Cirque de Soleil or in Broadway productions, is incredible in its technological sophistication. Even in productions that are on a smaller scale with limited budget and thus less commercially successful, there is still a significant amount of technology at work in operating sound systems and sound effects, lighting systems and lighting effects and in staging mechanisms. What is common about all of these systems is that they are ‘situated technologies’ that live in a theater space. Because of their permanent installation, the artistic productions that use these spaces adapt to these technological systems. In this sense, the production integrates itself to work with the effects that can be provided

TEMPORAL INTEGRATION EXPERIENCES

In this section we describe our experiences integrating interactive technologies into six different dance production cycles, and we highlight some of the lessons learned. All of these productions took place within a university setting, though some productions traveled beyond the university. A chart that illustrates the temporal integration of various elements of a dance production across these six different production cycles is shown in Figure 8. Exquisite Interaction

In our pilot production, which we called “Exquisite Interaction,” the technology used was wireless gyroscopic computer mice, which the dancers held in their hands as they danced. The motion data from these mice were transmitted

108

rehearsals involved a ‘visual show-and-tell’ that allowed the visual artist and the technologist to show previews of interactive visualizations that were being worked on for future sections of the dance, in order to get feedback and know which ideas were worth pursuing. Lesson: Tandem development of interactive visualizations and choreography requires the continual presence of all stakeholders at rehearsals.

Figure 2. Project pilot performance.

“A Mischief of Mus musculus” was the first production to be performed in a formal dance concert setting. This was the technology team’s first experience with the stage-readiness aspects of a dance concert setting. A costumer was used to create costumes especially designed for this piece, but the costumes did not require any special technology accommodation features. This meant that costuming ran as a separate, parallel production activity, with the costumer attending one or two early rehearsals and then worked on her own or in outof-rehearsal sessions with the choreographer and individual dancers. Lighted display boxes were used to highlight the mice on stage before initial pick-up by dancers, but were not used as part of the dance movement. In addition, there was a 30 minute delay between opening the house for patrons and the start of the performance. Because the mice would go into a ‘sleep mode’ after a certain period of inactivity, it was necessary to have a stage technician ensure they were in an active state just prior to the start of the piece.

to a central laptop and used to control a projected visualization. From the very first rehearsal with the three dancers, the technology was used and worked well. The main visualization used was ready immediately and another visualization was added during the very short three week production period before the first performance. The pilot dance was performed at three different venues, and there was no specialized lighting or other theatrical effects in use. The soundtrack was chosen before the first rehearsal. The dance was three minutes long and the choreography was developed in response to the technology, highlighting the fact that the dancers were carrying the mice and trying to demonstrate that the dancers were controlling the visualizations by using the mice. Because the dancers were carrying mice, the costuming did not need to be adapted or specialized to handle the technology, and so the costuming was simply designed to enhance the movement and spirit of the dance. Figure 2 shows an image from the pilot performance.

The technology used was not impacted by moving from the rehearsal space to the theater in terms of stage spacing or prop usage. However, this was our first encounter with formal theater lighting, and given that the main component of our technology is communication of kinetics through projected visuals, the impact of stage lighting on projected visuals was very important. It is typical in stage performances for a set of lighting cues to be developed for each scene, section, or movement phrase during a process called ‘lighting cue-to-cue’. A typical five minute dance may have 10 different lighting cues. When projected visuals are a part of the performance, it is necessary to adjust each lighting cue to ensure that the projections are visible and not washed out. Typically, the lower the lighting, the more vibrant the projections. However, it is not usually possible to have very low lighting levels as dancers need to be able to see and be seen. Seeing is particularly important for dancers — it is very dangerous for dancers to spin or move quickly in the dark, as they become easily disoriented without the ability to focus on a visible anchor point. This could easily lead to injuries. Thus, the integration of projected visuals with stage lighting requires fine-tuning the trade-off between safe lighting levels and visible projections. Given that most choreographic pieces in a university setting have only one or two hours in which to do lighting cue-to-cue, this is a very difficult and stressful part of the technological integration. In our production, we made use of a good, research-quality projector in the dance studio, and then made use of a much more powerful, performance-quality Christie projector that is built into the theater space. This required calibrating the visualization brightness and colors to work with the Christie pro-

Lesson: Existing technology integration can be the idea of the performance and drive the entire creative process. A Mischief of Mus musculus

The production, “A Mischief of Mus musculus,” involved technologists, an artist, a choreographer and six dancers, and was an 11-minute, pure movement piece (see Figure 3). In this case, the technology was the same as in the pilot – dancers held wireless gyroscopic mice in their hands while they danced, but they also made use of the mouse buttons to actuate particular visualizations. In addition, the dancers handed the mice off to each other throughout the dance, as three pairs of mice were used by six dancers. A visual artist was hired to develop more aesthetic visualizations, still making use of the movement data from the mice carried by the dancers. While the visual artist had some coding experience, a significant portion of the technologist’s time was devoted to coding visualizations throughout the production cycle in order to fulfill the artistic vision. Thus, in this case, the choreography and the visualizations were developed in tandem. Some of the choreography that was developed inspired the artist to design new visuals and some artist-created visuals inspired new choreography. The visual artist and the technologists needed to be present at all rehearsals in order for this tandem development to work, and much time was devoted to trying out different movement sections with different visualizations. The development process mainly followed a chronological ordering, in which the choreography and the visuals for the first part of the dance were designed first, and then the choreography and visuals for the next part of the dance were designed, etc. However, most

109

A single working prototype of the technology the dancers wore was developed before the production cycle began, but complete technical instrumentation needed to be constructed for each of the seven dancers involved. This meant that at the beginning of the production cycle the technology was not completely ready. This lead to a number of temporal integration issues. One of the technology issues that couldn’t be dealt with until multiple dancer packs were in use was networking the sensors. The visualizations and choreography were meant to be developed in tandem, as had been the case in the previous production. However, the visualizations were dependent on the hardware technology, and while they could be developed, they could not be tested until the hardware was completely in place. This meant that the development of visuals lagged behind the development of the choreography. The impact of this was that the choreography was set by the time the hardware was in place and the visuals could be tested. This meant that visuals which didn’t work as anticipated had to be reworked, but there was no opportunity for the choreography to change in response to the visuals because of the looming performance deadline. In this case, the visuals were reworked more to suit the hardware and to suit the narrative storyline than to suit the choreography, leading to some visuals that didn’t necessarily enhance the kinetic connection with the dancers.

Figure 3. “A Mischief of Mus musculus” performance.

jector, and this had to be done before the lighting cue-to-cue could proceed. It is important to note that once cue-to-cue occurs, it is highly impractical to make any changes to visual effects, other than fine-tuning. Thus all visual interactive effects had to be completed before moving into the final production space. An interesting event occurred during this production, that was disappointing, but proved to be educational: on the final performance day, the Christie projector in the performance venue stopped working just before the dance was about to be performed for a packed theater of 300 patrons. The technology team did not realize the projector had stopped functioning until the dance had begun and the projector shutter would not open. To their credit, the dancers performed as if everything was fine, and most audience members were unaware that there was any issue. This highlights an interesting issue: as with any performance element (lighting, sound, etc.), decisions sometimes have to be made about whether a performance should go forward if a particular system is not functioning. Given that technological systems are fallible, we have learned that it is wise to have a backup plan or a decision made about whether the performance should go ahead if the interactive technology fails.

Another issue that arose because of the late introduction of the hardware was that the dancers were not wearing the instrumentation initially, and they learned much of the choreographic material without wearing the technology. The technology setup was somewhat cumbersome for the dancers to wear and it impacted the quality of their dancing when worn during later rehearsals. While the dancing did improve with the worn technology, the choreographers noted that it did not reach the level of the choreography as originally developed.

Lesson: Decide ahead of time whether a performance should run if the technology fails, and who should make that call.

As with the previous production, there were particular issues that arose when moving this piece from the rehearsal studio onto the stage. Some of the choreography could not really be tried and set until the ramps and platform were in place on stage, and so the tracking/sensing technology used in those parts of the performance could not be tested until that time. In addition, the wireless networking required by the accelerometer setups was affected by electronic noise in the theater, where there is a lot of built-in technology. In an initial rehearsal on stage, the tracking did not work adequately because of networking issues. Due to this issue, in combination with the fact that the dancers performed better when not wearing the technology, we decided to run the performances using logged motion data from the dancers. This was disappointing, but it highlighted the importance of the timing of technology integration. With complicated technical systems, the systems need to be in place and used from the beginning of the production cycle, and more time needs to be allotted to moving the technology into the stage space.

Whispering to Ophiuchus

The third production, “Whispering to Ophiuchus” was significantly different from the first two productions in almost every aspect (see Figure 4). Our technology team expanded and our hardware technologist developed a set of wired 3D accelerometer sensors that each dancer could wear at their wrists. These sensors sat at the wrists, with wires that ran up the dancers’ arms and down their torsos, connecting to a battery and wireless transmitter pack worn at the sternum. This technology required specially designed costuming to secure, stabilize, and hide the instrumentation. The production process for “Whispering to Ophiuchus” was a massive parallel effort, with choreography, artistic storyline, musical score, hardware, interactive visualizations, costumes, scenic design and props all being developed simultaneously. This was a significantly more ambitious piece, totaling 22 minutes in length, with seven dancers performing. The piece was a modern, narrative dance, based on a myth about the thirteenth constellation of the zodiac. The complexity of the narrative demanded scenic pieces and props including ramps, an altar, a vessel, platforms, glass bowls, ribbons and a chair.

Lesson: Late integration of technology at best limits creative conversation and at worst may degrade creative outcomes.

110

Figure 4. “Whispering to Ophiuchus” performance. Figure 5. “Bodies/Antibodies” performance.

wireless sensor boxes were complete at the beginning of the production. However, having learned how important it is for the dancers to accommodate the technology as early as possible, we used empty blue boxes for some of the dancers while the real blue boxes were being completed.

Bodies/Antibodies

The fourth production, “Bodies/Antibodies,” was different yet again (see Figure 5). This was a restaging and expansion of a student-choreographed dance based on the concept of community, in this case, the community of cells in the human body. In this dance, we began synthesizing sensors. The student choreographer was an undergraduate research assistant on the project who was aware of the project goals and the technology we were using. We used the wireless gyroscopic mice employed in earlier productions, but this time they were embedded in a padded pocket sewn into the sports bra part of the costume worn by dancers. In addition, wireless, self-contained versions of the accelerometers (called ‘blue boxes’) were deployed and worn by the dancers in pockets located in the forearm of the costume. The software used data from both the mice and accelerometers from each dancer as input for the interactive visualizations. While the initial choreography was set, the dance was lengthened from an original 5 minutes to 11 minutes, so a significant portion of new material was developed. When the dance was originally staged it was not performed with any interactive (or non-interactive) technology enhancements. Because the existing choreography was designed to narrate a biological phenomenon, the visuals in this dance were designed in reaction to the choreography and the phenomenon. However, there was still some adaptation of choreography to fit the technology. For example, there were significant sections of floor work in the original choreography that required the dancers to press their whole front torsos against the stage. This had to be adapted in order not to have the dancers pressing the mice embedded in the front of their torsos against the floor, both because there was a fear of hurting the dancers and a fear of damaging the technology. In addition, in this dance, the performers responded to the technology explicitly, watching a visualization of cell division and dividing themselves in response. This was clearly a change from the original choreography and arose serendipitously during one of the many rehearsals involving the technology.

An Instance Of...

The fifth production in the project was a shorter dance that was modular in nature, consisting of a trio that could be deconstructed into three solos or various duets. The modular, flexible nature of this choreography led to the dance being titled, “An Instance of ...” (see Figure 6). The technology consisted of synthesizing audio feed data with the blue box sensors. The dancers wore the blue box sensors in sweatbands at their ankles and could trigger changes in visualizations by stomping that foot, allowing ‘personalization’ of the visualizations by any of the three dancers at any time. The dancers wore performance-quality radio mics, which consisted of a wireless transmitter worn at the hip, with a wire mic that was threaded through the back of the costume and up to the head, with the end of the mic sitting at the hairline. The temporal integration of visualizations and choreography was significantly different in this piece. The piece was choreographed collaboratively over a summer by four different people, with the official production cycle beginning in the fall. Thus, the choreography was developed before any software visualizations were created. The conceptual theme behind the dance evolved slowly and continued to evolve throughout the process. When the software visualizations were developed in the early fall, they were designed to react to the sounds the dancers were making with their bodies. One of the interesting temporal issues that arose in this production was related to the evolving theme of the dance. The costumer was initially called in to see the choreography and developed a costuming concept based on the cello music that accompanied the piece and the types of movements she observed. However, one of the changes that took place during the production cycle was the decision to have the dancers wear iPod Shuffles so that only they would hear the cello sound-track, allowing the audience members to focus on the body sounds of the dancers and the visualizations created by those sounds. In order to make this relationship clear to the audience, the waveform of the cello soundtrack was projected below the dancer-generated visuals, and it was decided that the iPod Shuffles and ear buds should be evident and not hidden under costuming. This change did not work with the existing costuming, resulting in a full costume redesign at some expense to the project and inconvenience to the costumer. To adapt to this new conceptual structure the

Lesson: Using technology throughout the rehearsal cycle can give rise to serendipitous opportunities for exploration. “Bodies/Antibodies” was performed in various venues but not in a formal theatre setting, thus a singular lighting level always sufficed for performance. The dance had no scenic pieces or props, and so the main concern was setting a lighting level in each performance venue that would allow the dancers to see and be seen and ensure that the visuals were not washed out. A similar constraint occurred as in “Whispering to Ophiuchus,” in that not all of the self-contained

111

Figure 6. “An Instance of ...” performance. Figure 7. “The Angled Angels Assembly” performance.

choreography also had to be reworked somewhat. However, this particular set of choreographic movements had been designed to be modular and flexible, and did prove to be very flexible in adapting to these conceptual changes.

cess while contributing a context within which the choreographers could evolve the content for the piece. However, the introduction of this new technology did pose some temporal challenges. The choreographers needed every possible minute of rehearsal to develop, teach and clean the material, and so fine-tuning and testing the tracking of the overhead camera took away valuable time from this process. There were times when the technology could have used more exploration, testing and fine-tuning, but that time had to be sacrificed in order to get through the choreographic development. Similarly, there were times when choreographic development time was sacrificed in order to figure out how to best make use of the particular tracking algorithm.

Lesson: Even when technology is integrated early, changes in the way the technology is used can cause issues for other elements of the production. When this piece was moved from the rehearsal studio to the stage, the Christie projector in the theater was not operational. Thus, our research quality projector had to be used in the very large theater space, which meant that the visualizations were not as bright as expected. This led to extra time being needed to light the dance, again to try to deal with the tradeoff between visuals being washed out and the dancers being able to see and be seen. In addition, it was necessary to fine-tune and adjust the audio levels from the microphones the dancers wore. It was critical to the piece that the audience be able to hear the breathing, stomping and other percussive sounds made by the dancers, amplified through the sound system in the theater, especially since the audience was not hearing the soundtrack that the performers were dancing to. However, care was needed to ensure that audio feedback didn’t occur, and so this required extra time in the stage-readiness phase.

The inclusion of various lengths of 3-inch wide black elastic as props was suggested by the strong horizontal line imagery of the initial visualizations. The use of the elastic added a third dimension to the 2-dimensional visualizations, allowing the elastic to reflect the lines and right-angled geometric shapes seen on the screen. The elastic props were used in each of the three sections, providing interesting choreographic tools and serving as one of the bridges between the physical dancing and the visualizations. The addition of this prop only occurred because the visualizations pre-existed; there would not have been time to incorporate them in response to a visualization development later in the process.

The Angled Angels Assembly

Our more recent production (see Figure 7), titled “The Angled Angels Assembly,” used an overhead fish-eye lens camera, and software developed to track the positions of the dancers using a combination of background subtraction and K-means clustering. With this piece, the hardware technology was ready and in place at the beginning of the production cycle and about 60% of the visualizations had been developed before the production cycle began. Another 20% of the interactive visualizations were developed in the first month of the production, with the remaining being added and fine-tuned throughout the rehearsal process. In particular, the order of the different visualizations and the transitions between them were the main tasks performed by the software team during the production cycle. The original 14 minutes of choreography was divided into three distinct sections. Because there were two choreographers working on this piece, the movement material was developed, taught and cleaned in an overlapping, non-chronological manner. For example, a sequence in the third section would be developed and taught while material in section one was being cleaned. Given that the choreography was not developed ahead of the production cycle, the existence of the majority of the visualizations at the beginning was helpful to the production pro-

Lesson: Early technology integration allows for the inspiration, design and deployment of relevant props. Finally, the use of an overhead camera required additional technical rehearsal time when the piece moved from the rehearsal studio to the stage. The camera had to be installed, tested and adjusted to get it at the right height to maximize stage coverage. It was critical that this installation and testing be done ahead of the spacing rehearsal, so that the spacing adjustments could take into account the visualizations and the visual field of the camera. DISCUSSION

Figure 8 shows a temporal integration comparison across all six of our productions, aligned to the all-important opening night deadline. There are clear patterns that emerge, some of which are independent of the technology. For example, the three choreographic phases of development, teaching/learning and cleaning always occur in that order with some level of overlap. Similarly, lighting is always set in the final week before performance, when the piece moves

112

Temporal Spanning of Elements of Dance Productions

1

1

2

3

Pilot (Jan 2008) Choreographic Development Choreographic Learning Choreographic Cleaning Movement Sensing Hardware Visualization Sound Costuming Props Set Lighting Performance

1

2

3

4

5

6

7

8

9

10

11

Mus Musculus (Fall 2008) Choreographic Development Choreographic Learning Choreographic Cleaning Movement Sensing Hardware Visualization Sound Costuming Props Set Lighting Performance

1

2

3

4

5

6

7

8

9

10

11

Whispering (Fall 2009) Choreographic Development Choreographic Learning Choreographic Cleaning Movement Sensing Hardware Visualization Sound Costuming Props Set Lighting Performance

2

3

4

5

6

7

8

9

10

11

12

Bodies/Antibodies (Spring 2010) Choreographic Development Choreographic Learning Choreographic Cleaning Movement Sensing Hardware Visualization Sound Costuming Props Set Lighting Performance

1

2

3

4

5

6

7

5

6

7

8

9

10

11

1

2

3

4

An Instance Of … (Fall 2010) Choreographic Development Choreographic Learning Choreographic Cleaning Movement Sensing Hardware Visualization Sound Costuming Props Set Lighting Performance

Angled Angels Assembly (Spring 2011) Choreographic Development Choreographic Learning Choreographic Cleaning Movement Sensing Hardware Visualization Sound Costuming Props Set Lighting Performance

Figure 8. The temporal integration of production elements into six dance production cycles (numbers represent weeks), with hardware shown in purple and interactive visualizations shown in pink. The green vertical line designates the beginning of the formal rehearsal schedule, while the red line designates the end of rehearsal and the beginning of the formal performances.

113

ographic selection and editing process. The same is true for interactive elements. In productions where the interactive visualizations were being developed throughout the production cycle, there were instances of choreographic movements being discarded or adapted to work with the new visualizations. In “The Angled Angels Assembly” no choreography that was created for the visualizations went unused, but some general movement sequences created early were discarded by the choreographers. There were also visualizations developed before the production cycle and early in the production cycle that were subsequently discarded, as they did not work with the choreographic concept, or they demanded too much visual attention. Similarly, in “An Instance of...”, costumes developed early in the production cycle were not used and were replaced with new costumes to better fit the evolved theme and technological focus. Thus, elements developed earlier are more likely to be discarded or changed, while elements introduced later are designed to fit what exists and are therefore less likely to be discarded or require reworking.

from the rehearsal studio to the stage. Costuming typically occurs outside of the rehearsal space and parallel to the process, with the costumer attending an early rehearsal in order to plan costuming, followed by subsequent design meetings with the choreographer and fittings with the dancers. Dress rehearsals occur in the final week before performance. When special modified costumes are required in order to house technological sensors, markers or devices, the costuming schedule changes and the costumes need to be available throughout the production cycle, as seen in the “Whispering to Ophiuchus” and “Bodies/Antibodies” productions. It is interesting to note the variation across productions in level of parallelism of various elements. This gives a visual sense of how much time was allotted in which the choreographic development and the interactive visual effects development could impact each other. What is also striking is that the “Whispering to Ophiuchus” performance had a high level of parallel activity across almost all elements of the production, and this was certainly the most stressful production for all stakeholders. This is an example of trying to do too much at once, and the stakeholders agree that from a process perspective, this was our least successful production.

Physical Impedance

Every technology that is used to create interactive dance will have some impact on the choreography. It is especially true that if dancers are going to be required to wear some type of technology (sensors, markers, devices), this technology needs to be in place early so that dancers can adapt to wearing it. There are a variety of reasons: the dancers need to become accustomed to the worn technology so that it becomes a ‘part of their bodies’. Also, depending on where the technology is worn or held on the body, it may impact the choreographic vocabulary. For example, in our productions in which dancers were required to hold mice, the choreography could not include any floor work, since the dancers would not be able to brace themselves. Similarly, there could not be any risky leaps, because the dancers would not be able to brace themselves in case of a fall. It is not possible to integrate technology that would require this type of choreographic adaptation late in the production cycle.

Deadlines

Most projects thrive on deadlines, but in performance disciplines the deadlines completely drive the work. Stage time is often scheduled more than a year in advance, well before the performance team is put together. For performance productions, opening night is a hard-stop deadline. Production managers, theater directors and choreographers work backwards from that date, scheduling what needs to be done by what date in order for everything to be ready. Team Participation in Rehearsals

Both the number of team members attending rehearsals and the frequency of that attendance had direct impact on the choreographic process. The more people attending the more likely it was that multiple side conversations about different aspects of the work would occur. The more conversations, the more difficult it was for the choreographer and dancers to concentrate. In the first few productions there was also an issue of the dancers feeling as though they were being ‘watched’ by the large technology team, and that was uncomfortable for some of the dancers. When the visualizations and hardware were being developed throughout the process it was necessary to have technologists at all rehearsals, but we learned that it was possible to minimize the size of the technology team present. In the most recent production, “The Angled Angels Assembly,” where the hardware was ready to use immediately and the interactive visuals were largely developed ahead of time, two technologists attended each of the early rehearsals. Later in this cycle, it was not necessary for the technologists to be at every rehearsal, as by that time the visualizations were set. This successfully minimized disruption and distraction.

Impacts on Choreography

Every type of sensing or technological tracking of dancers will have some impact on the choreography. For example, an instrumented floor will define a space in which dance movements that activate the technology can take place. A camera-based tracking system will similarly define a ‘viewable’ space in which the dancers can be successfully tracked. These types of technologies then have the same type of impact as stage size does in traditional dance. In normal dance productions, spacing takes place when the production moves from rehearsal studio to the performance stage, typically a week or two before opening night in academia. The choreographers know the final stage dimensions ahead of time and can sometimes tape off those dimensions in the rehearsal space so that the performers can practice adapting to the actual stage size. This can also be done for technological interventions that affect spacing.

Unused Resources

Other technological interventions may impact the dancers’ movements more directly. Having dancers wear equipment impacts the types of movements made and the level of en-

It is common for movement sequences to be developed for a piece and then not used. This is a typical aspect of the chore-

114

nificantly more time when moving from rehearsal space to the stage. However, new and experimental works often have less time. Some bigger dance companies have constant access to their own stage, which makes some of the issues we discuss here less salient. Professional dance companies also employ professional dancers who are able to create, learn and retain material more quickly, which allows more time for technology interfacing. However, the timing of the introduction of the interactive technology (both hardware and software) and how that impacts choreography or vice-versa is still quite salient.

ergy employed. Thus, it is imperative that the technology to be worn is ready at the beginning of the production cycle and is available to be worn at every rehearsal. In our early productions that involved dancers holding wireless mice, the choreography had to be designed around the dancers holding the mice, since they were not hidden. Dancers need to get comfortable with the weight and feel of the equipment and learn how the devices handle after performing and sweating. Costuming

In cases where the technology being integrated is worn by the performers, or where the costumes worn affect how the technology works, the temporal integration needs to take into account the costuming schedule. Costumes are typically conceptualized early but created mid-way through the production cycle, tailored towards the end of the production cycle and used in the dress rehearsals in the final week before the show. The costumer is usually present at one or two early rehearsals to get a sense of the choreographic concept and vocabulary and discuss costuming ideas with the choreographer. If the costumes need special features in order to accommodate sensors or techno-props, it is imperative that this is known early, at the costume conceptualization phase, since not only will the costumes need to be fitted, but they will also need to be tested with the technology, early enough to make adjustments for the technology if necessary. As an example of what can go wrong in costuming technology, multiple layers of costuming can inhibit wireless data transfer from sensors worn under the costume. Additionally, adding extra layers of costuming to hide technology can elevate core body temperatures, making the dancers uncomfortable. Determining early on what the parameters are for adequately hiding technology while ensuring the technology still works, and is comfortable for the dancers, is important.

Collaborative Production Software

Integrating interactive technologies into a dance performance is one way to bring technology into dance, but there are other aspects of technological integration. For example, there have been efforts to enable 3D choreographic modeling through software such as LifeForms [2, 3]. We have also developed a software tool that is aimed at allowing outof-studio collaboration for the dance rehearsal process [15]. The use of collaboration software to enhance the process is relevant to this paper in that such systems may be able to mitigate some of the issues we have uncovered, where the integration of interactive technologies requires more rehearsal time or more time when moving to the stage. TEMPORAL INTEGRATION CONSIDERATIONS

In this section, we present a list of considerations (and the potential impacts of these considerations) when planning a project that integrates technology into dance or perfomance in general. Hardware Integration Does the technology being used involve hardware setup in the space or on the performers? Hardware is probably the most critical component that needs to be in place at the beginning of a production cycle. The dependencies, in terms of setup, costuming, tracking and testing all point to the necessity of having hardware in place before production begins.

In the “Whispering to Ophiuchus” production, where dancers were wearing wired wrist sensors with battery packs on their torsos, all hidden under their costumes, the gadgetry had to be designed so as to limit the discomfort experienced by the dancers. Getting the dancers into the worn gadgetry was time-consuming at each rehearsal (taking away from actual movement rehearsal time), but was necessary to help the dancers feel comfortable dancing while wearing the technology. The choreographer noted that in rehearsals when the technology was not worn, the dancers performed more articulately. The choreographer felt that with time that could improve, but that this particular technology setup was cumbersome enough that the performance would always be slightly hindered by it. This suggests that more cumbersome technology worn by performers may actually require a longer production cycle, and certainly requires that the technology and costuming be ready to use at the beginning of the cycle.

Interaction Design Is the technology interactive, in that it responds to the performers and/or the performers respond to it? The best temporal integration for the design of interactive elements varies according to how that development takes place. If the dancers are to respond through the choreography to the digital elements, then the digital elements need to be in place early. If the digital elements are to respond to the dancers, then the choreography needs to be set and the digital elements can be designed in response. In the most compelling designs the interaction is bidirectional and requires tandem development. Costuming Does the technology require being hidden, or alternatively, being exposed, by the costuming of the performers? There are issues of safety, comfort, and aesthetics that come into play when embedding technology into costumes. It is our experience that costuming takes time, and that dancers need significant rehearsal time to adjust to costumes that embed technology, thus embedding tech in costumes needs to be done very early in the process.

Academic versus Professional Dance Settings

While the experience we have reported on here has taken place within the academic context, one of the authors has extensive experience working with professional dance companies. The biggest difference is that professional dance companies have longer rehearsal hours in general, which means there is more time for experimentation. They also have sig-

115

Lighting and Projection Systems Will the interactive technology performance make use of projections? There is a trade-off between being able to see the dancers, allowing the dancers to be able to see well enough to be safe, and the projected visuals being washed out by lighting. Since the actual stage lighting cannot be tested until the work is moved to the stage, scheduling additional technical rehearsal time for a piece with projections is often essential. Computers running visuals should be calibrated with the theater projector early on. We have found it useful to test the projector before each performance, as projection settings are sometimes reset or altered by others.

computing systems, pages 115–122, New York, NY, USA, 1993. ACM. 4. L. Candy and E. Edmonds. Explorations in Art and Technology. Springer-Verlag, 2002. 5. K. de Spain. Dance and technology: A pas de deux for post-humans. Dance Research Journal, 32(1):2–17, 2000. 6. K. Farley. Digital dance theatre: The marriage of computers, choreography and techno/human reactivity. Body, Space and Technology, 3(1):39–46, 2002. 7. R. Grove, C. Stevens, and S. McKechnie, editors. Thinking in Four Dimensions: Creativity and Cognition in Contemporary Dance. Melbourne University Press, 2004.

Sound Does sound play an important interactive role? If so, extra time is needed when the piece is moved to the stage in order to test and understand the acoustics in the stage space. Sound systems in rehearsal studios are typically very limited compared to those in professional venues.

8. J. Hanna. To Dance is Human: A Theory of Nonverbal Communication. University of Texas Press, 1979.

Rehearsal Participation What is the appropriate team balance for rehearsals? Managing the team such that a small number of technology team members are at rehearsals and only when necessary, appears to be the best temporal integration strategy to ensure minimum disruption.

9. J. James, T. Ingalls, G. Qian, L. Olsen, D. Whiteley, S. Wong, and T. Rikakis. Movement-based interactive dance performance. In Proceedings of the 14th Annual ACM international Conference on Multimedia, pages 470–480, New York, NY, 2006. 10. C. Latulipe, E. Carroll, and D. Lottridge. Evaluating longitudinal projects combining technology with temporal arts. In CHI ’11: Proceedings of the 29th International Conference on Human Factors in Computing Systems, pages 1835–1844. ACM, 2011.

CONCLUSION

We have presented our experiences with integrating technology into six different dance productions at UNC Charlotte over the past three years. This has provided us valuable experience with a variety of models for the temporal integrations that employ different hardware and software systems for the creation of interactive dances. We have found that the timing of the introduction of the technology has salient effects on the dance production as a whole and on the choreography in particular. We have created a list of temporal integration considerations which will be very useful to others attempting to integrate interactive technology into dance.

11. C. Latulipe and S. Huskey. Dance.Draw: Exquisite interaction. In People and Computers XXII: Proceedings of HCI 2008, pages 47–51. British Computer Society, 2008. 12. C. Latulipe, D. Wilson, S. Huskey, M. Word, A. Carroll, E. Carroll, B. Gonzalez, V. Singh, M. Wirth, and D. Lottridge. Exploring the design space in technology-augmented dance. In CHI EA ’10: Proceedings of the 28th of the international conference extended abstracts on Human factors in computing systems, pages 2995–3000, 2010.

ACKNOWLEDGEMENTS

We would like to acknowledge other Dance.Draw team members and past contributors: Erin Carroll, Vikash Singh, Adam Harris, Art Carroll, Mike Wirth, Nathan Nifong, Danielle Lottridge, Caroline Deeprose and Jordan Stevens. This work was supported by NSF CreativeIT (IIS-0855882).

13. W. S. Meador, T. J. Rogers, K. O’Neal, E. Kurt, and C. Cunningham. Mixing dance realities: collaborative development of live-motion capture in a performing arts environment. Comput. Entertain., 2(2):12–12, 2004.

REFERENCES

1. R. Aylward and J. Paradiso. Sensemble: a wireless, compact, multi-user sensor system for interactive dance. In Proceedings of the 2006 conference on New interfaces for musical expression, page 139. IRCAMCentre Pompidou, 2006.

14. R. Mitchell and M. Gallaher. Embodying music: Matching music and dance in memory. Music Perception, 19, 2001.

2. T. Calvert. Animating dance. In Proceedings of Graphics Interface 2007 (GI’07), pages 1–2, New York, NY, 2007. ACM.

15. V. Singh, C. Latulipe, E. Carroll, and D. Lottridge. The choreographer’s notebook – a video annotation system for dancers and choreographers. In To appear in: Proceedings of the 2011 Conference on Creativity & Cognition, C&C ’11, 2011.

3. T. W. Calvert, A. Bruderlin, S. Mah, T. Schiphorst, and C. Welman. The evolution of an interface for choreographers. In Proceedings of the INTERACT ’93 and CHI ’93 conference on Human factors in

16. C. Stevens, S. Malloch, S. McKechnie, and N. Steven. Choreographic cognition: The time-course and phenomenology of creating a dance. Pragmatics & Cognition, 11, 2003.

116