Interactivity and Narrativity in Screen-Media - CiteSeerX

3 downloads 78166 Views 77KB Size Report
Each production is made with bespoke software written by software developers. The experts in narrativity, who are the actual authors of the interactive.
Interactivity and Narrativity in Screen-Media Marian F. Ursu1, Maureen Thomas2, Mika L. Tuomola3, Terrence Wright4, Doug Williams5 and Vilmos Zsombori1 1 Narrative and Interactive Media, Department of Computing, Goldsmiths, University of London 2 University of Cambridge, 3University of Art and Design Helsinki, 4University of Ulster, 5BT [email protected]

Abstract In this paper we explore the relationship between interactivity and narrativity in the context of screen-media, where by screen media we mean mainly time-based audio-visual material. Despite a few successful productions, interactive time-based screen media has not yet become an established storytelling medium and the opportunities it presents to media consumption are yet to be explored. Here, we discuss possible forms of interaction made possible by a genre independent paradigm for developing interactive narratives, called ShapeShifting Screen Media. For validation, we present two ShapeShifting screen media productions.

1. Introduction and Context There appears to be a conflict between interactivity and narrativity, in that neither a good quality screenmedia narrative production, such as a movie, is customarily regarded as being able to be interactive, nor a highly interactive screen media experience, such as a game, is regarded as having a strong narrativity aspect. In spite of the view (e.g. Roger Ebert) that the freedom offered by interactivity and the coherence demanded by serious narratives are inherently opposed, we, taking as reference oral storytelling, believe that this tension is not intrinsic to the terms themselves, and is more or less historical and due to the restricted capabilities of the traditional forms of screen media and the way they have emerged. We believe that given appropriate software support screen media can be authored to convey complex narrations and be interactive at the same time. Interactivity should add to the power of storytelling. In the commercial world, interactive screen media narratives occur mainly as interactive television or

games. Television, the main embodiment of screen media narratives, has moved towards interactivity, but interaction occurs essentially “on the side”, trough satellite services, rather than being part of the narrations themselves. Games, representative for interaction, in general disregarded narrativity. However, a number of genres did put story front and centre, including adventure games (e.g. Grim Fandango, The Pandora Directive, Blade Runner), text adventures (e.g. A Mind Forever Voyaging, Anchorhead) and interactive movies (e.g. Tending Loving Care, Phantasmagoria), a handful of them (e.g. The Last Express, Discworld Noir, Farenheit) displaying quite complex narrative structures. The research world, too, has also developed a number of prototype interactive screen media productions. Surveys of both commercial and research productions can be found in [1], [2], [3] and [4]. All this work does demonstrate that narrativity and interaction could work together and illustrate new forms of engager experiences, but it presents two major drawbacks. One refers to the technical difficulty of the implementation. Each production is made with bespoke software written by software developers. The experts in narrativity, who are the actual authors of the interactive productions, have limited means to think about, experiment with and explore new forms of expression. Normally, they employ some generic presentation tools to simulate and model their ideas, which are not dedicated to and thus not really suitable for modelling interactive narrativity. Then, these models are communicated to the software developers for their implementation in a production. This process is quite cumbersome, expensive to iterate, and thus hinders creativity. Even if the end result is a good quality and engaging interactive narrative, the technology that supports it is normally quite expensive to develop, and,

most importantly, not reusable in other productions. This reduces significantly its impact on the field. The other drawback refers to simplicity. As the means that authors have to model, experiment and realize are fairly limiting, the natural consequence is to implement simplistic interactive narrative structures, which can be anticipated to work without requiring specialized tools for modelling interactivity. However, obviously, simplicity does not carry impact. The natural conclusion is that the area of interactive screen media narratives cannot make significant advances without dedicated tools and technologies [5]. Pioneering research has been carried out in developing technologies for interactive narratives made with recorded essence, which includes systems dedicated to specific productions, such as Terminal Time [6] and Mindful Camera [7], and generic authoring tools, such as those developed by Agmanolis and Bove [8], Gordon et al. [9] and Wages et al. [10]. They demonstrate the positive impact that dedicated technology could have, but they do this under significantly limiting assumptions (a succinct review of this work is presented in [11]). A general paradigm for interactive screen media narratives made with recorded content – ShapeShifting Screen Media [12] – has been developed within the EU Integrated Project NM2 [13]. ShapeShifting Screen Media introduces a production and genre independent computational representation language, called Narrative Structure Language (NSL), and an associated authoring and delivery software system. NSL [14], [11] is founded on a primitive set of entities and aggregation structures, and represents the medium in which ShapeShifting Screen Media productions are conceived. However, authors do not have to manipulate NSL directly. They are provided with an authoring environment [11] in the form of a comprehensive set of tools, implemented as visual interfaces. A number of productions both inspired and validated the paradigm. We believe that this is still merely the start in exploring this new type of media – interactive screen media narratives made with recorded essence – but, at the same time, we believe that this new paradigm constitutes a powerful basis for further developments.

2. ShapeShifting Screen Media A ShapeShifting Screen Media production is an interactive screen media narrative, authored, made with recorded essence, and able to respond to input from viewers/engagers. A ShapeShifting production could

lead to different experiences depending on the engager’s interaction. A manifestation of a ShapeShifting production for one engager and during one performance or screening is called a narrative thread. Narrative threads are automatically compiled/edited at delivery time on the basis of the engagers’ interaction. Narrative threads are made of media items, such as video and audio clips. Conversely, media items are automatically edited together at delivery/viewing time into narrative threads. Production configurability or adaptivity is achieved by (automatically) rearranging media items in narrative threads. Media items represent the atomic elements of content – they can be rearranged, but their content is fixed. In ShapeShifting Media the essence is separated from the symbolic representations. Essence is ingested in a media pool as files and is represented as media items; a media item is a symbolic representation of a contiguous fragment from an ingested file, defined by an in- and out-point. Media items also have associated structured descriptions in the form of metadata, relating to both low-level features, such as dominant colour, and high-level features, such as the characters who figure in the item. The media items’ definitions together with their metadata are part of the narrative space. For each production, the terminology for describing high level features has to be grounded in a formally represented ontology, which, too, is part of the narrative space. The narrative space has another component – the production’s narrative structures – which specifies the flow of the narration with reference to the possible engager interactions. In other words, the narrative structures specify the order in which the media items are to be edited into narrative threads in the context of any possible performance. Media items are referred to in the narrative structures either directly, via their identifiers, or indirectly, via their characteristics (such as “a media item that introduces the character Juulia”). More details on these can be found in [15]. The narrative structures are expressed in NSL. Simplistically, NSL supports the recursive aggregation of media items into (structured) narrative objects and provides means for specifying the order in which the narrative objects should be ‘played’ on the basis of the engagers’ interaction. Examples of structures available in NSL are the link, the layer and the selection group. The NSL aggregation structures are accompanied by sophisticated mechanisms for the specification of different composition features and are fully recursive,

in that they can be combined in any order and to any level of depth. These make NSL a very powerful specification language. However, NSL is still a core language – it is closer to the mechanics of choosing and editing media items together, than to aspects of rhetoric or style. Nevertheless, the latter don’t yet exist for interactive screen media. Hence our approach: develop a core (low level) language and refine it, through experimentation, with high level narrative structures. NSL is described in more detail in [11] and [14]. To summarize, the narrative space consists of the pool of media items and their associated metadata descriptions, the associated ontology and the production’s narrative structures. When the metadata is not in focus, the term ‘narrative space’ may be used to refer solely to its most important component, the production’s narrative structures. An authoring environment provides dedicated tools for the creation of narrative spaces. Narrative spaces are manually authored. For each production, the narrative space (fully authored) is the artefact on whose basis the narrative threads are automatically compiled, edited and delivered to the engagers. For each engager, this is accomplished via an iterative process that happens, loosely speaking, either between two consecutive interactions, or after a period of time has elapsed. Each iteration produces a fragment of the narrative thread. The sequence of all the fragments represents the overall narrative thread. Narrative threads are automatically compiled by the Realisation Engine and reach the engagers through specific Client Applications, which display their content, captures (new) interaction from the engagers, and send them back to the Realisation Engine. Timing is crucial in the iterative communication between the Realisation Engine and the Client Applications: new narrative thread fragments have to be produced and delivered sufficiently early such that they reach the Client Applications before the previous fragments were completely played. The ‘engager interaction’ during one screening is a sequence of time-stamped multimedia objects sent by the Client Application. In general, they contain only textual information that describes the engager’s intention or preferences, and which is accordingly interpreted by the Realisation Engine. However, they can also be fragments of content that could be used in the production itself; this is the case of user generated content. More about interaction will be said in the next section.

3. Authoring and Consuming Screen Media Narratives Linear, non-interactive screen media narratives tell one story in one way; a storyworld is narrated as a linear sequence of events. The choice of the events and the order in which they are juxtaposed is left entirely to the creators, but they are always linearly arranged. For example for a dramatic production, the scriptwriter creates the script with a linear narration in mind – events are recounted in a sequence, even if they jump in time and space in the storyworld. The director, subsequently, imagines how the story could be represented visually and, even if he/she experiments with alternatives, the thinking is focused on creating a linear sequence of scenes. Finally, the editor, whilst ensuring that the juxtaposed clips tell the story well, both in terms of plot and aesthetics, may also rearrange the clips, but still with the linear story in mind. These processes – writing, directing, shooting, and editing – raise challenges with each production, and on auspicious occasions they bring about innovation. However, on one hand, paradigms exist, in the form of plot, shot and screen grammars that lay the foundations for them. On the other hand, and more importantly for the argument in this section, because they refer to linear objects, they are much easier to be carried out in the mind of the creators, though even in this case external support, such as file cards, is necessarily required. In interactive screen media, the storyworld does not have to be linearised. More than one facets of the same storyworld could be told by the same interactive screen media object. Instead of being the passive recipient of an account of the story the engager can explore the storyworld [16]. The challenge for creators, in this context, is about how to craft an explorable media object such that the engagers’ explorations result in narrative threads of the same quality as if they were told directly by the creators themselves. The challenge is considerable, as the storyworld is virtually impossible to hold in the mind of the creators as a union of all the possible narrative threads. Without dedicated computational representations and software tools, the effective production of interactive narratives is not possible. ShapeShifting Screen Media brings them in the form of NSL and an associated Authoring Environment. Note the above use of the term ‘exploration’. Interactivity is not restricted, for example, to making a choice between two possible outcomes (such as to fall

in love or not), as initial work in interactive narratives in the form of branching stories was (problematically) taken to suggest. ShapeShifting storytelling is not linear storytelling with interaction; it is a new form of expression and consumption. Exploring a storyworld may be regarded, simplistically, as having a one to one conversation with the media. However, this is not the only possibility of interaction; through ShapeShifting screen media people may converse with each other. The cardinality of the relationship is one interactive screen media object to many engagers or consumers. We could speculate that interactions between the media objects themselves may also be possible, but, as yet, such experiments have not been carried out in ShapeShifting media. The exploration of a storyworld is achieved through the interaction means provided by the production itself. Interaction may be regarded as having a form and an outcome. In the ‘form’ we include the cue provided by the system to the engager and the multimedia object sent back to the system (refer to the previous section). The ‘outcome’ relates to the meaning assigned to the interaction, to its expected result in terms of shaping the narration. For example, the engager may choose the name of a character when it appears on the screen – the form – which will result in the narration changing its focus to the selected character – the outcome. The main problem in creating interactive narratives is not about the syntax of the interaction – the form –, but about the potential response in the narration – the outcome. In the remainder of this section, interaction is taken to refer to ‘outcome’. For genres such as news and documentaries, interaction could refer to aspects such as the relevance, accuracy and trust, and conciseness of the information, to the discovery of new interpretations, and to the ability to contribute with input and engaging in conversations with others. For example, a documentary could allow the exploration of issues at different levels of depth, the request for explanations, the reiteration of certain aspects already presented, either for enjoyment or for clarification, and the exploration of opposing or related viewpoints. Further, engagers may be allowed to contribute with their own recordings and commentaries. The same may be possible in news. Also, through carefully constructed narrative structures, current news may be hooked into archived material, thus opening up the possibility of new interpretations through emerging but unplanned juxtapositions. In drama, taking as reference the Aristotelian coordinates of plot as action, place and time, the

interaction could refer to movements through place and time, following different characters and/or events. For example, in order to understand the reactions of a character, the engager may be asked to be taken back in time or, alternatively, to a different location, to the events that could provide an explanation. Once there, new aspects may be unveiled which could incite the curiosity of the engager and generate other similar questions. However, from any point reached in the storyworld, the narration should always be able to resume and conclude. NSL does not provide direct support for such highlevel interaction modes. For example, it contains no predefined narrative structures that would change the focus of a narration from one character to another. However, via NSL’s interaction annotation [11], it does provide basic but powerful means to implement such behaviour. The next section will give examples of such behaviour implemented in two ShapeShifting Screen Media productions. Interaction modes such as those discussed above cannot be implemented literally. They are further constrained by the requirement of ensuring the informative, entertaining, engaging and aesthetic qualities of the production – i.e. of all the possible narrative threads. The ability to interact should not impede upon them. How can all these characteristics be guaranteed in productions which may result in hundreds or even thousands of narrative threads? The grammars from the linear productions may be useful, partly, but they are not sufficient. Grammars for interactive screen media productions have been described in a number of ways [17], but few have been tested in practice, for lack of sufficiently powerful supporting computational languages. We believe that NSL represents a possible basis for such a language and thus will become a means for capturing rhetorical and stylistic guidelines. There is a need to describe new grammars for interactive screen media productions, which have been tried and tested in practice. However, even if such guidelines existed, the quality of the final outcome/production – i.e. all the possible narrative threads – would still have to be tested and validated. To this end, ShapeShifting Screen Media proposes a solution in terms of semantic tests – i.e. dedicated procedures implemented in the Authoring Environment that can extract characteristics of the narrative space relevant to the quality of the possible narrations [11]. However, this work is only just beginning.

The following section describes two ShapeShifting productions, one drama and one documentary, as illustrations for new ways of media consumption provided by interactive screen media.

4. ShapeShifting Screen Media Productions 4.1. Accidental Lovers Accidental Lovers (AL) [18] is a musical comedy that explores a love relationship between an old cabaret singer, Julia, and a young rock star, Roope. It was broadcast in 12 screenings on TV1 of the Finnish Broadcasting Company YLE between 28 December 2006 and 5 January 2007, and it reached over one million overall viewers (Finland has approximately 5.3 million people). AL has a cyclic structure similar to, for example, ‘Groundhog Day’ [19] – events are recounted over and over again, to bring new meanings to the story. However, as opposed to Groundhog Day, where the events are hard coded in a linear structure, AL automatically constructs new narratives on the basis of engager input. Interaction in AL is carried out through unstructured SMS messages. Engagers are invited to send their thoughts regarding the developing drama via texts. They determine the unfolding of the story though a keyword logic. A more sophisticated natural language processing module could have been implemented instead, but it was not deemed necessary. AL is thus a conversation between engagers and the dramatic production, but also, as an inherent effect, between engagers themselves. AL is constructed as a superimposition of voiceovers on video clips. The logic of interaction determines the choice and sequence of the video clips, and, to a greater extent, the choice of the voiceovers. New meanings can emerge by overlaying different voiceovers with different video clips. AL is configurable on two levels. On a micro level, there is an immediate response to aggregated engager input which affects the nuances of the story, but not the plot itself. For example, an issue that is captured from the engagers’ input could be initiated for discussion between the protagonists. As a more subtle interaction ‘outcome’, the atmosphere of the story could be affected by choosing video clips and voiceovers of a mood corresponding to that extracted from the engagers’ SMSs. On a macro-level, engagers can influence the plot. Cumulated engager input determines

the main events of the story. The micro level responses are substantially more frequent than the macro level responses. The production’s statistics showed that engagers had taken the opportunity to get involved in the themes presented in AL and the destinies of the fictional characters, which is a very encouraging result for interactive screen media narratives [20].

4.2. Interactive Village Interactive Village (IV) [21] is a research demonstrator of a reconfigurable ‘documentary’ about village life in Czech Republic. IV aimed to escape the traditional models of the ‘documentary’, ‘film’, ‘portrait’, ‘programme’, etc., and devise an “interactive profile” which could be explored on different levels including observational, didactic and ‘taking an issue’. The observational mode is essentially ‘watch and listen’. Interaction is limited to choices of topic (e.g. place, event, and person) and depth/length of presentation. The didactic mode includes commentaries in the narration, as explanations provided by the anthropologist, or extra footage that brings more details to the topic. There is also a choice of the perspective of the anthropologist, as male, female and with expertise in masquerade. The ‘taking an issue’ mode is about attitudes expressed on a number of subjects or issues. The material, representing scenes, interviews, activities and commentaries, has been recorded in a “’forensic’ approach” – it was gathered recognizing that its significance would be continuously redefined through engager exploration; “story potentials” were recorded rather than definite storylines. The aforementioned exploration modes – observational, didactic and ‘taking an issue’ – are not explicitly presented in the interface. They emerge from the use of the more basic interaction means, which allow engagers to increase and decrease the level of detail of the presentation, to require more explanations on the current topic in the form of voiceovers, and to respond to prompts presented in the narration, such as cutaways that illustrate an interview. Such interactions happen within a chosen topic and the narration flows seamlessly – the appearance of a continuous programme is preserved. Continuity between topics is not ensured. The narrative space of IV would support well user generated content and links to archives. For example, personal video diaries and images captured on digital cameras could be incorporated dynamically into the

production. They would undergo the same associative mechanisms as the material captured professionally by the researchers. However, for this, the narrative structures would have to denote media items entirely in terms of their characteristics/metadata. This is one of the directions towards which IV may be developed.

5. Conclusion We believe that ShapeShifting Screen Media provides a powerful foundation for exploring new forms of media “consumption” as interactive screen media productions.

6. Acknowledgements This work was supported in part by the EU FP6 Integrated Project, IST-2003-004124 “NM2: New Media, New Millennium”. The authors would like to thank to the entire NM2 consortium for their indirect input to this paper.

7. References [1]

M. Rieser, and A. Zapp, The New Screen Media: Cinema/Art/Narrative, BFI Film Classics, 2002.

[2]

S. Dixon, Digital Performance: A History of New Media in Theatre, Dance, Performance Art and Installation, MIT Press, 2007

[3]

B. Atkins, More than a Game. The Computer Game as a Fictional Form, Manchester and New York: Manchester University Press, 2003.

[4]

J. Wyver, “D5.16: Interactive and Configurable Models in Popular Media Genres”, Project deliverable in the EU FP6 Integrated Project IST-2003-004124 “NM2: New Media, New Millennium”, 2006. Available at: http://www.istnm2.org/publications/deliverables/deliverables.html.

[5]

J. Murray, “Towards the Holodeck,” Session at the International Broadcast Conference (IBC), Amsterdam, The Netherlands, 7–12 September, 2006.

[6]

M. Mateas, P. Vanouse, and S. Domike “Generation of Ideologically-Based Historical Documentaries,” in Proceedings of AAAI 2000, Austin, TX, 2000, pp. 36–42

[7]

B. Barry, “Mindful Documentary,” PhD Thesis at the Media Lab MIT, Massachusetts, USA, 2005.

[8]

S. Agmanolis, and M. Boye Jr., “Viper: A Framework for Responsive Television,” IEEE Multimedia, 10 (1), 2003, pp. 88–98.

[9]

A. Gordon, M. van Lent, M. van Velsen, M. Carpenter, and A. Jhala, “Branching Storylines in Virtual Reality Environments for Leadership Development,” in Proceedings of the Sixteenth Innovative Applications of Artificial Intelligence Conference (IAAI-04), San Jose, CA. Menlo Park, California, USA, 2004

[10] R. Wages, B. Grützmacher, and S. Conrad, “Learning from the Movie Industry: Adapting Production Processes for Storytelling,” in S. Göbel, U. Spierling, A. Hoffmann, et al. (eds.) Technologies for Interactive Digital Storytelling and Entertainment: Second International Conference, TIDSE 2004, Darmstadt, Germany, LNCS 3105, Springer-Verlag, 2004, pp. 119–125 [11] M. F. Ursu, J. J. Cook, V. Zsombori, and I. C. Kegel, “A Genre Independent Approach to Authoring Interactive Screen Media Narratives (Proceedings—Accepted for publication),” AAAI 2007 Fall Symposium on Intelligent Narrative Technologies, Westin Arlington Gateway, Arlington, Virginia, November 911, 2007, to be published. [12] D. L. Williams, M. F. Ursu, J. J. Cook, V. Zsombori, M. Engler, and I. C. Kegel, “ShapeShifted TV – Enabling MultiSequential Narrative Productions for Delivery over Broadband,” presented at the 2nd Institution of Engineering and Technology Multimedia Conference, London, 29–30 November 2006. Available at: http://conferences.theiet.org/multimedia/ithd_digest [13] NM2: New Media, New Millennium, EU FP6 Integrated Project, IST-2003-004124, www.ist-nm2.org. [14] M. F. Ursu, J. J. Cook, V. Zsombori, I. Kegel, D. Williams, M. Hausenblas, and M. L. Tuomola, “ShapeShifting Screen Media: A Declarative Computational Model for Interactive Reconfigurable Moving Image Narratives,” in the Proceedings of the AAAI Artificial Intelligence and Interactive Digital Entertainment Conference 2007 (AIIDE 07), Stanford, USA, 6–8 June 2007, pp. 101–104. [15] M. F. Ursu, J. J. Cook, V. Zsombori, R. Zimmer, I. Kegel, D. Williams, M. Thomas, J. Wyver, and H. Mayer, “Conceiving ShapeShifting TV: A Computational Language for TrulyInteractive TV,” In P. Cesar et al. (Eds): EuroITV 2007, LNCS 4471, 2007, pp. 96–106. [16] M-L. Ryan, Narrative as Virtual Reality: Immersion and Interactivity in Literature and Electronic Media. Baltimore: Johns Hopkins University Press, 2001, pp 254-256. [17] G. Liestol, A. Morrison, and T. Rasmussen, eds., Digital Media Revisited: Theoretical and Conceptual Innovation in Digital Domains, Cambridge, Mass.: MIT Press, 2003. [18] M. L. Tuomola (director), L. E. Saarinen (writer), and M. J. Nurminen, Accidental Lovers (‘Sydän kierroksella’), Crucible Studio, Helsinki University of Art and Design Finland, broadcast on Channel TV1 by YLE, The Finnish Broadcasting Company, 2006/2007. [19] H. Ramis (director), Groundhog Day, Columbia Pictures Corporation, USA, 1993. [20] L. E. Saarinen, M. J. Nurminen, and M. L. Tuomola, “D3.7: Accidental Lovers Production Report,” Deliverable (restricted access) in the EU FP6 Integrated Project IST-2003-004124 “NM2: New Media, New Millennium”, 2007 (permission required). [21] T. Wright (director), Interactive Village, University of Ulster, UK, 2007. Available at: http://www.istnm2.org/media_productions/new_page_3-flash.htm.