... Uschi, Jasmien,. Simon, Jonas (x3), Koen, Ike, Birgitte, Wouter, Ilse (x2), Lizzy, Shirley, Carina, Marlen, .... âWriting about Music is like Dancing about Architectureâ ...... concept and the research field is in constant evolution and revolution.
Moving Sounds and Sonic Moves Exploring Interaction Quality of Embodied Music Mediation Technologies through a User-centered Perspective
Alexander Deweppe Proefschrift voorgelegd tot het behalen van de graad van Doctor in de Kunstwetenschappen Dissertation submitted in partial fulfillment for the Degree of Doctor in Art Science
Decaan Faculteit Letteren & Wijsbegeerte Prof. dr. Marc Boone
Rectoren Universiteit Gent Prof. dr. Anne De Paepe (2012 – 2017) Prof. dr. Paul Van Cauwenberge (2008 – 2012)
Faculty of Arts and Philosophy Department of Art, Music and Theater Science Institute for Psychoacoustics and Electronic Music Doctoral School of Arts, Humanities and Law Ghent University
iii
dedicated to Sofie and Max – and to Kashmir and Borrel
v
Moving Sounds and Sonic Moves: Exploring Interaction Quality of Embodied Music Mediation Technologies through a User-centered Perspective
Alexander Deweppe
Dissertation presented in partial fulfillment of the requirements for the Degree of Doctor in Art Science Proefschrift voorgelegd tot het behalen van de graad van Doctor in de Kunstwetenschappen
2017
vii
Promoter: prof. dr. Marc Leman Doctoral guidance committee: prof. dr. Lieven De Marez dr. Micheline Lesaffre Members of the reading committee: VUB-chancellor prof. dr. Caroline Pauwels prof. dr. Marcelo M. Wanderley prof. dr. Kris Rutten Members of the examination committee: prof. dr. Francis Maes dr. Edith Van Dyck President of the examination committee: dean prof. dr. Marc Boone Scholarships: BOF-GOASeMA (2008 – 2011) BOF-Emcometecca (2011 – 2012)
I certify that this thesis, and the research to which it refers, are the product of my own work, that any ideas or quotations from the work of other people, published or otherwise, are fully acknowledged in accordance with the standard referencing practices of the discipline. Any copyrighted or trademarked materials that might be included, resort under the guidelines of academic fair use. This thesis is published for non-educational purposes. All rights reserved. No part of this book may be reproduced in any form by any electronic or mechanical means (including photocopying, recording, or information storage and retrieval) without written permission of the author.
© Alexander Deweppe
Cover image: Picture of the demonstrator use case “SoundFlocks”, developed in the context of the “SoundField”-project (discussed in chapter 9 of the thesis).
ix
Dankwoord
Het heeft net geen tien jaar in beslag genomen om deze thesis tot een aanvaardbaar einde te brengen; niet zonder slag of stoot en zeker niet zonder menig voet in de aarde. Velen hebben me met hun niet-aflatende aanmoedigingen gesteund en gaande weten te houden. Zo zijn er in eerste instantie mijn promotor, de leden van mijn doctoraatsbegeleidingscommissie en de mensen van de verschillende onderzoeksgroepen (IPEMUGent, iMinds-SMIT-VUB, CEMeSo-SCOM-VUB en JRN-Howest) waar ik deel van uitmaak(te), die me de kans gaven om dit project aan te vatten en me, ondanks wat omzwervingen, met expertise en advies op het juiste pad hebben gehouden. Ook de inzichten van de vele professoren, doctores, doctorandi, studenten en lieden uit andere geledingen van het academische bestel waarmee ik met genoegen van gedachten wisselde, waren voor mij zeer waardevol om van mijn – inmiddels ongeveer vijftien – jaren aan de universiteit een aangename en inspirerende periode te maken. Het is evenwel vooral het aanhoudende enthousiasme van mijn familie en mijn huisgenoten, van mijn vrienden, vriendinnen en maten, van mijn ex-collegae, van mijn oude studiemakkers en van mijn talrijke kameraad-muzikanten dat me uiteindelijk gemotiveerd heeft om door te zetten; het is zeker niet altijd even makkelijk geweest! Ik weet ieder van jullie dank voor de gewaardeerde hulp, de steun en de interesse: zonder jullie was het vast niet gelukt om dit omstandige liedje tot het einde uit te zingen. Bedankt – en nog een paar keer “sorry” voor het feit dat het zo lang geduurd heeft! Gent, 26 maart 2017 Alexander Henry William Carl Deweppe
xi
Abstract and Keywords (EN)
This research project deals with the user-experience related to embodied music mediation technologies. More specifically, adoption and policy problems surrounding new media (art) are considered, which arise from the usability issues that to date pervade new interfaces for musical expression. Since the emergence of new wireless mediators and control devices for musical expression, there is an explicit aspiration of the creative industries and various research centers to embed such technologies into different areas of the cultural industries. The number of applications and their uses have exponentially increased over the last decade. Conversely, many of the applications to date still suffer from severe usability problems, which not only hinder the adoption by the cultural sector, but also make culture participants take a rather cautious, hesitant, or even downright negative stance towards these technologies. Therefore, this thesis takes a vantage point that is in part sociological in nature, yet has a link to cultural studies as well. It combines this with a musicological frame of reference to which it introduces empirical user-oriented approaches, predominantly taken from the field of human-computer-interaction studies. This interdisciplinary strategy is adopted to cope with the complex nature of digital embodied music controlling technologies. Within the Flanders cultural (and creative) industries, opportunities of systems affiliated with embodied interaction are created and examined. This constitutes an epistemological jigsaw that looks into 1) “which stakeholders require what various levels of involvement, what interactive means and what artistic possibilities?”, 2) “the way in which artistic aspirations, cultural prerequisites and operational necessities of (prospective) users can be defined?”, 3) “how functional, artistic and aesthetic requirements can be accommodated?”, and 4) “how quality of use and quality of experience can be achieved, quantified, evaluated and, eventually, improved?”. Within this multi-facetted problem, the eventual aim is to assess the applicability of the foresaid technology, both from a theoretically and empirically sound basis, and to facilitate widening and enhancing the adoption of said technologies. Methodologically, this is achieved by 1) applied experimentation, 2) interview techniques, 3) self-reporting and survey research, 4) usability evaluation of existing xiii
devices, and 5) human-computer interaction methods applied – and attuned – to the specific case of embodied music mediation technologies. Within that scope, concepts related to usability, flow, presence, goal assessment and game enjoyment are scrutinized and applied, and both task- and experience-oriented heuristics and metrics are developed and tested. In the first part, covering three chapters, the general context of the thesis is given. In the first chapter, an introduction to the topic is offered and the current problems are enumerated. In the second chapter, a broader theoretical background is presented of the concepts that underpin the project, namely 1) the paradigm of embodiment and its connection to musicology, 2) a state of the arts concerning new interfaces for musical expression, 3) an introduction into HCI-usability and its application domain in systematic musicology, 4) an insight into user-centered digital design procedures, and 5) the challenges brought about by e-culture and digitization for the cultural-creative industries. In the third chapter, the state of the arts concerning the available methodologies related to the thesis’ endeavor is discussed, a set of literature-based design guidelines are enumerated and from this a conceptual model is deduced which is gradually presented throughout the thesis, and fully deployed in the “SoundField”-project (as described in Chapter 9). The following chapters, contained in the second part of the thesis, give a quasichronological overview of how methodological concepts have been applied throughout the empirical case studies, aimed specifically at the exploration of the various aspects of the complex status quaestionis. In the fourth chapter, a series of application-based tests, predominantly revolving around interface evaluation, illustrate the complex relation between gestural interfaces and meaningful musical expression, advocating a more usercentered development approach to be adopted. In the fifth chapter, a multi-purpose questionnaire dubbed “What Moves You” is discussed, which aimed at creating a survey of the (prospective) end-users of embodied music mediation technologies. Therefore, it primarily focused on cultural background, musical profile and preferences, views on embodied interaction, literacy of and attitudes towards new technology and participation in digital culture. In the sixth chapter, the ethnographical studies that accompanied the exhibition of two interactive art pieces, entitled "Heart as an Ocean" & "Lament", are discussed. In these studies, the use of interview and questionnaire methodologies together with the presentation and reception of interactive art pieces, are probed. In the seventh chapter, the development of the collaboratively controlled music-game “Sync-In-Team” is presented, in which interface evaluation, presence, game enjoyment and goal assessment are the pivotal topics. In the eighth chapter, two usability studies are considered, that were conducted on prototype systems/interfaces, namely a heuristic evaluation of the “Virtual String” and a usability metrics evaluation on the “Multi-Level Sonification Tool”. The findings of these two studies in conjunction with the exploratory
xiv
studies performed in association with the interactive art pieces, finally gave rise to the “SoundField”-project, which is recounted in full throughout the ninth chapter. The integrated participatory design and evaluation method, presented in the conceptual model is fully applied over the course of the “SoundField”-project, in which technological opportunities and ecological validity and applicability are investigated through userinformed development of numerous use cases. The third and last part of the thesis renders the final conclusions of this research project. The tenth chapter sets out with an epilogue in which a brief overview is given on how the state of the arts has evolved since the end of the project (as the research ended in 2012, but the research field has obviously moved on), and attempts to consolidate the implications of the research studies with some of the realities of the Flemish culturalcreative industries. Chapter eleven continues by discussing the strengths and weaknesses of the conceptual model throughout the various stages of the project. Also, it comprises the evaluation of the hypotheses, how the assumptions that were made held up, and how the research questions eventually could be assessed. Finally, the twelfth and last chapter concludes with the most important findings of the project. Also, it discusses some of the implications on cultural production, artistic research policy and offers an outlook on future research beyond the scope of the “SoundField” project.
xv
Keywords
Embodiment, new media technology, cultural studies, cultural and creative industries, usability, participatory design, collaborative control interactive and new media art, quality of experience
xvii
Samenvatting en kernwoorden (NE)
Dit onderzoeksproject handelt over de gebruikerservaring van muzikale bemiddelingstechnologieën die aspecten benutten van lijfelijkheid. Meer specifiek worden problemen behandeld die bestaan rond het toepassingsgebied en het beleid aangaande nieuwe (artistieke) mediatoestellen in de cultureel-creatieve sector, ten gevolge van de beperkte ergonomische kwaliteiten van de bemiddelaars die op heden nog steeds een vlotte ingebruikname bemoeilijken. Sedert de opkomst van nieuwe draadloze bemiddelaars en besturingstoestellen voor muzikale expressie is er een uitdrukkelijke wens van de creatieve sector en diverse onderzoekscentra om dergelijke technologie in verschillende deelgebieden van de culturele sector binnen te loodsen. Het aantal toepassingen en het gebruik ervan is in het laatste decennium dan ook exponentieel toegenomen; anderzijds hebben veel van de toepassingen ernstige gebruiksvriendelijkheidsproblemen, waardoor niet enkel de culturele sector maar ook veel cultuurparticipanten vaak een afwachtende, eerder terughoudende of zelfs negatieve positie innemen ten aanzien van de betreffende technologieën. Binnen een referentiekader van de systematische muziekwetenschap, eigent deze thesis zich daarom naast een sociologisch uitgangspunt, ook het perspectief van de ‘cultural studies’ toe. Daarenboven worden empirische benaderingswijzen en gebruikersonderzoeksmethodes ontleend aan het veld van de computerwetenschappen (meer specifiek, de studie van de mens-systeeminteractie). Deze interdisciplinaire strategie wordt aangenomen om in staat te zijn het hoofd te bieden aan de uitermate complexe natuur van digitale lijfelijke muziekmediërende technologieën. Binnen de realiteit van de Vlaamse cultureel-creatieve industrie worden opportuniteiten, gelieerd aan lijfelijke muziekbemiddelingstoepassingen, geobserveerd en gecreëerd. Binnen deze methodologische puzzel wordt bekeken 1) “welke belanghebbenden verschillende vormen en gradaties van betrokkenheid, interactief potentieel en artistieke mogelijkheden veronderstellen”, 2) “hoe artistieke aspiraties, culturele veronderstellingen en operationele benodigdheden als noodzakelijk voor de gebruiker kunnen worden gedefinieerd”, 3) “hoe aan functionele, artistieke en esthetische
xix
vereisten tegemoet kan worden gekomen”, en 4) “hoe de kwaliteit van de interactie en de kwaliteit van de gebruikservaring bij deze toestellen kan worden gegarandeerd, gekwantificeerd, geëvalueerd en uiteindelijk verbeterd”. Binnen dit gelaagde probleem is het de uiteindelijke bedoeling om het toepassingsgebied van de besproken technologie vanuit een theoretisch en empirisch gefundeerde basis vast te stellen, en het gebruik ervan te verbreden en te verbeteren. Methodologisch wordt hiernaar gestreefd aan de hand van 1) toegepaste experimenten, 2) interviewtechnieken, 3) zelf-rapportering en survey-onderzoek, 4) ergonomieonderzoek van bestaande bemiddelaars, en 5) methodieken ontleend aan de menssysteem-interactie, toegespitst en aangepast aan lijfelijke muziekbemiddelingstechnologieën. Binnen dit studiegebied worden concepten ontwikkeld en getest die verwant zijn aan bruikbaarheidsonderzoek, flow, presence, doelstellingsinschatting en vermaak. Deze worden binnen het referentiekader van de vigerende technologieën getest en onderzocht, zowel aan de hand van taak-gebaseerde als ervaringsgeoriënteerde heuristieken en meetinstrumenten. In het eerste deel van de thesis, dat drie eerste hoofdstukken behelst, wordt de algemene context gegeven. In het eerste hoofdstuk wordt een inleiding tot de onderwerpen gepresenteerd en worden de problemen die de ontwikkeling op heden domineren op een rij gezet. In het tweede hoofdstuk wordt een breder theoretisch kader geschetst van de concepten waarop de thesis steunt, namelijk 1) het paradigma van het lijfelijkheidsonderzoek en de implicaties ervan voor de muziekwetenschap, 2) een overzicht van de gangbare technologieën die gebruikt worden voor het maken van lijfelijke muziekbemiddelingstoepassingen 3) een overzicht van de voor musicologisch onderzoek beschikbare en toepasbare mens-computer-interactie en bruikbaarheidsmethodologieën, 4) een introductie in gebruikersgebaseerde digitale ontwikkelingsprocedures, en 5) enkele inzichten betreffende de aardverschuivingen die digitalisering en e-cultuur in het cultureel-creatieve landschap teweegbrengen. Na de voorstelling van deze concepten, worden in het derde hoofdstuk 1) de voor deze thesis beschikbare methodologieën behandeld, worden 2) de uit de literatuur gehaalde ontwerprichtlijnen opgesomd, en wordt van daaruit 3) een conceptueel model opgebouwd dat in zijn volledigheid wordt ontplooid en toegepast in de bespreking van het “SoundField”-project (zie hoofdstuk 9). De daaropvolgende hoofdstukken, vervat in deel twee, geven een quasi-chronologisch gestructureerd overzicht van hoe de methodologische concepten stelselmatig werden toegepast doorheen een reeks van empirische casestudies, specifiek toegespitst op de exploratie van verschillende aspecten van de complexe status quaestionis. Het vierde hoofdstuk beschrijft een reeks applicatie-gebaseerde tests, die voornamelijk op de evaluatie van de gebruikte bemiddelaars focussen, illustreert de complexe relatie tussen
xx
bewegings-gebaseerde interfaces en zinvolle muzikale expressie en vormt een pleidooi voor een meer gebruikersgeoriënteerde ontwikkelingsprocedure. In het vijfde hoofdstuk wordt een vragenlijst van een surveyonderzoek besproken, genaamd “Wat beweeg(t) je”. De vragenlijst had meerdere doelstellingen en was gericht op het verwerven van een overzicht van de belangrijkste eigenschappen van de eindgebruikers van lijfelijke muziekbemiddelingstoepassingen. De vragen doelden voornamelijk op culturele achtergrond, muzikaal profiel en muzikale voorkeuren, opinies over lijfelijke interactie, kennis van en attitudes met betrekking tot nieuwe technologie en participatie in de digitale cultuur. In het zesde hoofdstuk worden twee kleinschalige etnografische studies gepresenteerd die samen met de voorstelling van twee interactieve kunstwerken, “Heart as an Ocean” & “Lament”, werden uitgevoerd. In deze studies werden interviews en vragenlijsten gebruikt om te peilen naar de perceptie van interactieve kunstwerken binnen de context waarin ze gepresenteerd worden. In het zevende hoofdstuk wordt de ontwikkeling van een collaboratief controleerbaar muziekspel uit de doeken gedaan en worden interface-vergelijking, presence, spelvermaak en doelstellingsinschatting als voornaamste thema's naar voor geschoven. In het achtste hoofdstuk worden twee bruikbaarheidsstudies voorgesteld die werden uitgevoerd op systeem-prototypes, namelijk een heuristische evaluatie van de “Virtuele Snaar” en een bruikbaarheidsanalyse van de “Multi-niveau Sonificatie Applicatie”. De bevindingen uit deze bruikbaarheidsstudies in combinatie met de exploratieve studies die de interactieve installatie-kunstwerken begeleidden, gaven aanleiding tot het op touw zetten van het “SoundField”-project, dat uit de doeken wordt gedaan in het negende hoofdstuk. Het volledig geïntegreerde participatieve ontwerp en de evaluatiemethode, die werd gepresenteerd in het conceptuele model, werd integraal gebruikt als basis voor de gehele duur van het “SoundField”project, waarin technologische mogelijkheden aan ecologische validiteit en artistieke toepasbaarheid worden afgetoetst aan de hand van de gebruikers-gestuurde ontwikkeling van veelvuldige casestudies. Het derde en laatste deel van de thesis geeft de finale conclusies van het onderzoeksproject weer. Hoofdstuk tien begint met een korte epiloog over hoe de status quaestionis sinds het einde van het onderzoeksproject is geëvolueerd (gezien dat reeds in 2012 afliep, maar het onderzoeksveld sindsdien geenszins ongewijzigd is gebleven). Voorts worden pogingen ondernomen om de implicaties van de geschetste studies te consolideren met het cultureel-creatieve landschap in Vlaanderen. Het elfde hoofdstuk vervolgt het verhaal met een opsomming van de sterke en zwakke elementen van het conceptuele model doorheen de verschillende stadia van het onderzoek. In dit hoofdstuk worden ook de hypothesen opnieuw onder de loep genomen, wordt geëvalueerd hoe goed de gemaakte veronderstellingen overeind blijven, en hoe de onderzoeksvragen uiteindelijk dienen te worden beantwoord. In het twaalfde en laatste hoofdstuk, wordt tenslotte een overzicht gegeven van de belangrijkste conclusies, bevindingen en de implicaties voor
xxi
cultuurproductie gegeven. Bovendien worden mogelijke beleidsaanbevelingen van deze soort artistiek onderzoek beschouwd, en wordt een blik geworpen op een bredere context van toekomstig artistiek onderzoek, maar voortbouwend op de verworvenheden van het “SoundField”-project.
xxii
Kernwoorden
Lijfelijkheid, nieuwe media technologie, culturele studies, cultureel-creatieve industrieën, bruikbaarheidsonderzoek, participatief ontwerp, gemeenschappelijke controle, interactieve en nieuwe mediakunst, ervaringskwaliteit
xxiii
Preface and Acknowledgements
Back in August 2006, I promised myself never to go through the arduous process of knowingly and willingly writing a thesis again. Soon after, however, I started preparing my research funding applications for further study on the topic of my master's dissertation – i.e. classical quotation in heavy metal music. After several proposals being rejected, I abandoned my aspirations to pursue any PhD-ambitions and further academic pipe dreams. After I graduated and already started working a regular job, it was coincidence that lead me back to the IPEM, and, as fortune would have it, at the beginning of 2008, prof. dr. Marc Leman eventually offered me the opportunity to start working in a still largely to-be-assembled research team, which would ultimately enable me to obtain my PhD in systematic musicology (of which I knew relatively little at the time). The title for my part of the project – and for a long time the working title of my thesis – was “Sociomusicological Investigation into the Cultural-Creative Applications of Embodied Mediation Technology in the Music Sector”. I was to be funded by the remainder of the GOASeMA-project, but would effectively be working on the new, super-secret, large and amply funded “EmCoMeTeCCA”-project. My part of the project, though initially only loosely delineated, was set out to be a preparatory study for cultural policies pertaining to new media technologies of the Flemish government. Its goal was to report on changes in the artistic landscape and cultural reality brought about by the rise of multi-, new and unstable media arts using embodied music mediation tools (EMMTs) and augmented and virtual reality environments (AI&VREs). Within that scope, the aim was to define the problems, ascertain opportunity and eventually valorize the cultural use of EMMTs, as well as to assess properties of new media adoption, new media literacy, new media readiness and eagerness, specifically within a regional scope of the cultural and artistic sector. Very ambitious indeed! And, something that would quickly become apparent, it was also a relatively unobtainable goal. The subject matter was something altogether new for me, which made me second-guess all my steps well into the second year of the scholarship. Whilst in progress, I had to master a lot of new theorems, methods, concepts and technologies. Meanwhile, I had to define a research area where I could make a xxv
somewhat valuable contribution, both in the broad inter/multi-disciplinary field that is known as cognitive, systematic, computational or comparative musicology, as well as in the main ongoing project at IPEM, i.e. the EmCoMeTeCCA (Embodied Music COgnition MEthusalem MEdiation TEchnology for Cultural and Creative Applications) project, the Methusalem-grant funded by UGent BOF, awarded to professor Leman. Within that project framework, this research is a part of the fourth work package, focusing on interaction. The goals set forth in the "Social Interaction" work package entail predominantly qualitative research based on observations, interviews and subject ratings attached to case studies (e.g. studies of modification of actions, perceptual and emotional effects of human body movement in artistic communication), ontological considerations, modeling of concepts taken from social interaction theories (e.g. symbolic interactionism) and the implementation of a structural framework from user-provided annotations. The requirements made in the work package assumed the creation of an annotated subject pool. The purpose of the subject pool would be (1) to provide a ready supply of subjects for experiments, and (2) to develop of a database with background information on these subjects. The test subjects would be scouted by means of an online questionnaire with the target population, facilitated by partners in the cultural-creative sector. Moreover, this would help define user profiles, based on subjects' personal background and ratings by subjects on their perception of bodily and affective impact of music. Essentially, the related topics are 1) interaction and communication types and levels (Human-Human / Human-Technology / Technology-Human / Human-Technology-Human), 2) body gestures types and characteristics, 3) bodily experience of movement and music, and 4) acquaintance with mediation technology / social networking / gaming. A first set of experimental studies was quite unsuccessful, compelling me to change the scope of the thesis, as the required technologies were not readily available and the general lines that were set out, appeared to be a dead end. And so, it became abundantly clear that some of the formulated goals, requirements and presumptions were too advanced for the technologies at hand. As all the work packages were at that point equally advanced, some considerable issues pervaded the project. The measurements that were conducted and the data that was gathered, pertained to heavily flawed, at times seriously malfunctioning or even downright non-existing technologies. Then, the database that was being developed contained predominantly prospective users that were either not very eager to use the said technologies, or simply reluctant to do so. Moreover, no user-profiles could be discerned, since in most of the cases, they would be connected to at best conceptual technologies. And finally, in view of the more standardized and more reliable applications, this concerned the interaction and gestural experience with a heavily diversified range of applications, with varying interaction strategies and altogether different objectives. Trying to quantify the properties of these applications meant either describing a very specific case (i.e. a singular sensor technology requiring a particular gestural schema, generating a
xxvi
distinct data stream onto a specific mapping strategy with a fixed data selection protocol and specified sonic representations) from which essentially no broader assessments could be drawn that go beyond the single instantiation of the technology, or reveling in commonplace and complacent high-level disquisitions about the embodied interaction principle that went in no way into the specificities of the application at hand. So, with considerably more problems than answers, both theoretically as well as methodologically, I was trapped between rock and a hard place. Then, in preparation for the 2008 International Summer School for Systematic Musicology, conveniently organized by IPEM at Ghent University, I started going through some usability literature, spurred on by Marcelo Wanderley’s “Borrowing tools from HCI”-paper. Though its impact was not radically paradigm-shifting for my research altogether, I found it to be quite a novel idea, which would at least enable me to bridge the high-level embodiment theories with the bottom-up installation development and evaluation. I distinctly remember three separate instances in the aftermath of the summer school where I found my enthusiasm snubbed when I shared my newly gained ideas with a number of authorities in the field. When I informed my colleagues about my plans at an IPEM think tank meeting, prof. dr. Van Noorden asked me with a great deal of concern why I suddenly had chosen to become ‘the marketing guy’ of the group. At the IPEM open house event leading up to the first Methusalem evaluation round, prof dr. Godøy confided to me that there definitely was promise in my chosen path, but that I should not let the vastness of the HCI-literature discourage me. After the Methusalem hearings, prof. dr. Schneider recommended me by all means to go through with it, since doing a PhD was what he called “Lustverzicht” anyway. Three separate telltale signs that this was not going to be a walk in the park, and indeed, I have oftentimes in various degrees felt either intimidated, detached, superfluous, doleful, dazed and confused about the whole undertaking. Another mention-worthy obstacle in an otherwise already bumpy road is that, during this period, my doctoral guidance committee – “DBC” in Dutch – underwent a series of changes prior to it being in its definitive form (with cameo appearances from prof. dr. D. Moelants, prof. dr. J. Van Looy and the late prof. dr. J. Vincke), and aside from the volatile nature of the committee, substantial reforms in the rules, conditions and regulations of the Doctoral Schools further complicated matters. Yet, with the support of my promoter and DBC, I was able to turn the scope of the thesis around. Instead of defining and categorizing relevant gestures, we focused on a more sorely needed earlier development stage. It became apparent that, in order to meet the requirements of the EmCoMeTeCCA-project, additional, different and more exploratory methodologies would be necessary, as the project balances somewhere between the fields of human computer interaction and embodied music cognition. Since the Methusalem project essentially aims at successfully connecting human movement to musical expression, it has to draw on theories from computer science, system engineering, musicology, sports xxvii
science, social sciences and cultural anthropology to do so. In order to enable measurement and design of embodied music interaction tools and to address the embodied and socially interactive nature of musical behavior, a user-oriented approach to embodied cognitive musicology is necessary, which focuses on social aspects of music behavior and the role of music in society. In other words, successful development of EMMTs relies heavily on defining the proper opportunities for the technology and on pinpointing the issues that matter to the end-users. Given this thoroughly end-user-oriented development, some additional aims can be defined, like the technology-aided 1) support of personal expressiveness and creativity, 2) stimulation of social interaction and musical bonding, and 3) facilitation of automatic recognition of interaction patterns and body movement. So, rather than organizing random experiments, that could loosely be described as music-based and movement-related, around a given test-subject pool, there is a direr need for the development of a conceptual model that observes the multiple realities relevant to person-music interactions. Only after that is established, systematic musicology can start focusing on the assessment of action-relevant movement-cues that form the basic vocabulary for automated gesture recognition and personal expressiveness in embodied music mediation technologies. As much in the beginning as now, in the end, I feel this thesis has a sense of not being fully finished, as it serves as only one small juncture in (what is supposed to be) a bigger journey. Even though apposite to tackle most of the specific issues it describes, it is flawed in giving the all-encompassing answers in the much larger and obscure realms of embodied music cognition. But that would be, moreover, beside the point. As a PhDthesis itself is something that is – in my case relatively – limited in time, it should not continue until the sorcerer’s stone is finally discovered; one of the realizations that took about nine years to sink in. So, in keeping with family traditions, 97 years after my late great-grandfather Henri de Sagher – ex-professor at Ghent University – I also have come to conclude my PhD-thesis a few months shy of a decade; no WWI to blame for the delay in my case, though. That being said, I must admit that at times, and especially near the end, finalizing the thesis felt like an exercise in futility (in reference to the bigger scheme of things, and the things in life that matter); the utter untimeliness of the project and the ensuing ineptitude of the technology to live up to the expectations set forth by the embodiment paradigm, further exacerbated my reluctance resentment towards the whole endeavor. To boot, after six years as a student, nine years as a doctoral student and ten years working for university, I've also gotten to see, on multiple occasions, the less pretty side of (what has apparently become not all that much more than) the industry of proposing projects and producing papers. Whether it was the unabating abuse of positions of power, the incessant exploitation of intellectual resources of subordinates, the unremitting unscrupulousness concerning intellectual property (infringements), the at times grove misappropriation of government funds and unmitigated abuse of tax payers' hard-earned xxviii
bucks, the tendentious, self-serving, improper and opportunistic appropriation of scientific methodologies and paradigms or the purposive misrepresentation of experimental figures, the downright egocentric and malevolent work-ethos of some academics and the glaring disrespect for human capital in academia in general, or the blatant display of patronism and partisanship, any and all remonstrance was systematically shrugged off with strategic and stereotypical academic indifference, and complaints somehow got lost between the academic echelons or swept under the university's malodorous rug. The fact of the matter is that it is the university context itself that requires such outlandish behavior, yet a lot of these trespasses – most of which I luckily witnessed from a reasonable distance – went without being given proper attention. It left me with a sense of disenchantment with academic hierarchy and its structures, as well as a suspicion of scarcely concealed rivalry with and disingenuity of some of my peers. Scholars, I gathered, were reckoned to be diligent, morally sound and intellectually superior people, supposedly motivated predominantly by the all-encompassing quest for knowledge; in my utopia, therefore, academia would benefit from having a few more principled and ethical (wo)men in its ranks. And so, I have come to find this period a bit short of a decade, spent together with academics and immersed in academia, an at times somewhat demoralizing one. It does moreover beckon the question whether the inability to beat them automatically and inadvertently means that one would have better joined them. Hopefully, these sentiments will now be appeased by the end of this seemingly never-ending juncture in my academic trajectory. Moreover, I digress! Most of the malpractices inherent in this system are more eloquently, quite humorously and most poignantly described by the beautiful academic minds that brought us “PhD Comics”, which helped me put things into a more relativistic perspective on numerous occasions.
More importantly, I was fortunate enough to get all the intellectual and spiritual guidance I needed to help me through these trying times along the way, and for that, some words of thanks are in order, because without the much appreciated help and support of the following people, I probably wouldn't have pulled through. First and foremost, I need to extend my profound gratitude to my promoter, prof. dr. Marc Leman, for giving me this great opportunity, for granting me the freedom to execute this research project, for putting his confidence in the favorable outcome of my thesis and for his unparalleled patience. Closely linked, I have to express my indebtedness to the members of my doctoral guidance committee, dr. Micheline Lesaffre and prof. dr. Lieven De Marez. Even though the formation of my committee went through some serious transformations, the eventual members gave me the motivation, inspiration and guidance to see it through. Then, I would like to express my gratitude to some highly inspirational figures in the research field whom I had the pleasure of encountering; prof. dr. Marcelo Wanderley, xxix
prof. dr. Alexander Refsum Jensenius, dr. Maarten Grachten, prof. dr. Ian Cross, prof. dr. John Sloboda, prof. dr. David Huron, prof. dr. Rolf Inge Godoy and basically all the professors, doctors and researchers whom I had the pleasure of talking to during summer schools and international conferences. I would like to thank my esteemed colleagues and former colleagues at IPEM, for the interesting talks, the fun moments and the help I received. Special acknowledgement goes to the ladies in my office (dr. Leen, dr. Edith, Kathelijne and dr. Katty), to dr. Luc and dr. Muzaffer (for the interesting and motivational conversations we had), and to the technical and administrative staff (Aagje, Katrien, Raven and Ivan). Moreover, I would like to thank some of the members of the department, prof. dr. F. Maes and prof. dr. M. Martens, whom I always had pleasant conversations with, the always inspirational librarian Mario Floryn, as well as a cornucopia of other colleagues from our own faculty, other faculties and the central administration of the university; you know who you are! I would like to thank my former colleagues of the UZ pediatrics department 5K6 (the lovely office ladies, prof. dr. D. Matthys, the witty em. prof. dr. E. Robberecht, Leen Poupeye and Thijs Verbeurgt) who – all in their very own particular way – graciously enabled me to leave their department and pursue my ‘academic dreams and ambitions’. I want to express my vast and sincere gratitude as well to VUB-chancellor prof. dr. Caroline Pauwels and MP prof. dr. Katia Segers, the two additional supervisors I had during my stint (2012–2016) at iMinds/imec–SMIT–VUB and CEMeSo–VUB, for showing me a different, new and exciting side to academia, for believing in my capacities – probably more than I did myself, for presenting me with new challenges, for giving me a number of great responsibilities and for their unwavering patience during the most intense transitional period of my life – in which I tried virulently to get my two halftime jobs done for the policy research centers for culture and media, finish my thesis to finally leave my student days behind once and for all, rebuild a house, whilst becoming a daddy; I only managed partially within the given time-frame… In addition to the promoters, I’ve had the pleasure of making the acquaintance of a large and tremendous team of inspiring (meanwhile ex-)colleagues at iMinds–SMIT–VUB. I’ve shared many great moments with quite a lot of you; you made me feel at home and appreciated, you provided an extremely inspirational environment for me, and – even though most of you to date have no clue about what it is precisely I did before arriving at SMIT – you helped me to apply the skills and experiences I had gained from this research project to other challenges. Hence, I would like to thank – in no particular order, and apologetic for those I might have forgotten – Tim & Tom (aka the 'friends of Valerio' or ‘Dixie after six’-crew), Eva, Rob, Ralf, Jan, Karen (x3), Jo, Leo, Katleen, Katriina, Catalina, Sanne, Elias, Paulien, Wim (x2), Hans, Olivier (x2), Jana, Uschi, Jasmien, Simon, Jonas (x3), Koen, Ike, Birgitte, Wouter, Ilse (x2), Lizzy, Shirley, Carina, Marlen,
xxx
Bram, Olga, An (x3), Joke, Sander, Sven, Sarie, Yoni, Evelien, Iris, Sarah (x2) and Dana. And some people, who weren’t colleagues, but whom I had the pleasure of working with: dr. Jessy, prof. dr. John, prof. dr. Ignace, Joeri and Gert. It was and is a great place to be and to work at; I miss you guys dearly (although I absolutely don’t miss the commute). To eliminate the risk of forgetting anybody who was involved in my personal history over the last nine-and-a-half years, I would like to start by thanking all my friends, acquaintances, relatives and family that played an indispensable part in my life during this period. That being said, I would like to thank some special people personally. I would like to start by expressing my gratefulness for the sumptuous ‘decade of decadence’ I spent as a student enrolled at Ghent University. In this era, I had the magnificent delight of meeting numerous fabulous people from all echelons of university. Most dear to me, are the inspirational figures I met during my time in FaculteitenKonvent Gent, the p(l)easant times I had with the scoundrel-musicians of Studentenfanfare Ghendt and the lasting friendships I made and retained during my time in KunstHistorische Kring. To name but a few, thank you dr. Thomas, Karlien, Melanie, Sarah, Frauke, Jo, Evi, dr. Devi, Jeroen (x2), Babs, Jen, Tho, Mathieu, Thijs, Frank and pro-chancellor prof. dr. Paul. Also, I would like to pay homage to the phenomenal people I have had the exquisite pleasure playing in bands with, both past and present, for creating that great diversion I so sorely needed, for inspiring me, for making me a better musician and for helping me to keep my eye on the ball. After all; we are in it for the music, aren't we? Consequently, I would like to apologize profoundly to friends and acquaintances that had the audacity of asking me what I had been doing exactly during those nine years. You patiently endured my lengthy and incoherent ramblings, that were necessary for me to figure it out for myself and to make a feeble attempt at explaining; even though it was probably cruel and unusual punishment to you, it meant a whole lot to me. And on a side note, I have gotten a lot better at it than I was in the beginning. On a personal note, I would like to thank my two fantastic co-PhD-thesis-writing friends, meanwhile dr. Bram De Geyter and dr. Lieven Haenebalcke, their spouses Maia and dr. Vicky, and the little miracles that have since joined the party, Lou(s)isa and Clara. They were there for me, they walked alongside me for most of the adventure – even though I walked for a considerable while longer – and gave me a different view on the horizon when I needed one. They got me out of my comfort zone, repeatedly, when they thought that was appropriate, and I can be nothing but grateful for that. Much in the same vein, I would like to thank my LA Woman, Elke Pauwels, her utterly French surfer-dude-adventurer Clément, and their little rascal Tibo, for much the same reasons; we will always have Vegas, baby!
xxxi
On another personal note, I am much indebted to two fabulous ex-colleagues, that over the course of the last eight years have become genuine friends, dr. Pieter Coussement and dr. Nuno Diniz. I closely and fruitfully collaborated with them, lived together with them during some positively flabbergasting times at Dworp, Switzerland, Amsterdam and Finland, communicated and brainstormed with them, had lengthy discussions with them on research, on musicology, on politics, bureaucracy, computer tech support, administration, (guitar) technology and on life in general, shared minor victories and weathered major disappointments with them, built massive installations and performed strange (beautiful) music, went on ski and road trips with them, watched some very peculiar concerts with them, did more than a bit of fine dining with them and their significant others Marijke and Elvire, shared the occasional drink and genuinely had a lot of fun in their presence. In a sense, I owe a large portion of my PhD-thesis to working together with such fine and esteemed gentlemen. On an even more personal note, I owe a lot to my in-laws, Jean-Marie and Lut, to godfather Thomas and T(ante N)athalie, for helping out with the ever-so-important logistics; for making sure we all can live in a cozy, well-groomed and at least halffinished house. Thank you also Kirsten, Dorian and Roxane, sister in-law, nephew and niece, for always putting a sincere smile on my face! I want to express my profound gratefulness to my parents, Sabine and Alex, and to my brother Walter; for life, for the good – and also a little for the bad – character traits, for looking after me and taking care of me for genuinely as long as I can remember, and more than that, for the ardent yet enjoyable conversations, for the loving and unwavering support, and for being on my side unremittingly at those times when the going got tough. In view of the multiple moments of solitary confinement that I spent practically alone with my thesis over the last decade, I have to give credit where credit is due, and acknowledge our two cats, Kashmir and Borrel. They kept me company and tried very hard to make me put things into perspective every once in a while, also occasionally deleting a paragraph by unapologetically crouching on my keyboard. Basically, during this arduous process, they kept me happy and – more or less – sane. Similar gratitude goes for my godchildren Lily and Dorian – with the exception of sitting on my keyboard. The same (i.e. the very much happy and just a few inches short of insanity) also goes for my tiny little Max. I almost can’t remember the times when you weren’t around yet, creating intricate Duplo- or Bumba-based diversions and traps, pulling my leg – literally – when it was the direst time to get away from that infernal computer. It makes me so grateful and proud – and positively petrified – to see that another creature in this world might possess the same idiosyncrasies, quirks and interests as me. I hope you are better and faster than I at locating those things that bring you joy and fulfilment in your life, and
xxxii
if that time ever comes, there is a small chance that I might not react overly enthusiastic when you tell me you are considering getting a doctoral degree. Finally, my wholehearted gratitude goes out to my girlfriend, Sofie. Over the past seventeen years – and counting, she has been everything from chauffeur, housekeeper, trip advisor, laundry lady, realtor to personal secretary, confidant, maid, therapist, accountant, forlorn stylist, mental coach, moral compass, reviser and mommy to our son Max. She has shown me so much kindness, made so many sacrifices and bore with me for the entire ride; and I have given back ever so little… I am ashamed! Nonetheless, she is my immortal beloved, my inspiration and my very best friend. Thank you, Sofietje! We're going to Disneyland!
xxxiii
“Writing about Music is like Dancing about Architecture” Frank Zappa
Statement of Original Authorship: The work contained in this thesis has not been previously submitted to meet requirements for an award at this or any other higher education institution. To the best of my knowledge and belief, the thesis contains no material previously published or written by another person, except where due reference is made .
xxxv
List of Abbreviations
AI&VRE CAPI DAW D2M DKO EMMT FFT GUI HaaO HCI MIR MLST NIME PaS PCi PRC QoM QoE RITE RtD SF SIT UC UCCi UE UX VR VS WMY
Augmented, Immersive and Virtual Reality Environment Computer-Assisted Personal Interview Digital Audio Workstation Dance2 the Music Parttime Art Education (dutch: deeltijds kunstonderwijs) Embodied Music Mediation Technology Fast Fourier Transform Graphical User Interface Heart as an Ocean Human-Computer Interaction Music Information Retrieval Multi-Level Sonification Tool New Interfaces for Musical Participation Survey (with waves for PaS ’04, PaS ’09 and PaS ’14) Parameter Complexity Index Policy Research Center (i.e. “Steunpunten beleidsvoorbereidend onderzoek”) Quantity of Movement Quality of Experience Rapid Iteration Testing Evaluation Research through Design “SoundField” Sync-in-Team Use Case Use Case Complexity Index Usability Evaluation User Experience Virtual Reality Virtual String “What Moves You”-survey
xxxvii
List of Figures
Figure 3.01 Figure 3.02 Figure 3.03
Figure 3.04
Figure 3.05 Figure 3.06 Figure 3.07 Figure 3.08 Figure 4.01 Figure 4.02 Figure 4.03 Figure 4.04 Figure 4.05 Figure 4.06 Figure 4.07 Figure 4.08 Figure 4.09 Figure 4.10
Figure 4.11 Figure 4.11
Schematic representation of most EMMTs’ basic mode of operation. Table showing classic setup-properties for most EMMTs. Participatory, user-centered and iterative development strategy enumerating the design stages and the design elements to be addressed during this process. Table summarizing the design guidelines discussed in §3.2: setupspecific elements, use case-related items, AI&VRE related requirements and usability-specific demands. Schematic representation of the adopted iterative design and evaluation strategy. Parameter Complexity Index (PCi) Use Case Complexity Index (UCCi) Scheme summarizing the guidelines, challenges, operational requirements and design strategy inherent in the conceptual model. Picture of the Perpendicam top video recording footage in EyesWeb. Schematic representation of the Perpendicam2 45° angle setup @ ISSSM ‘08. Picture showing Perpendicam2 participants interacting during social condition. Screenshot of the ISSSM 2008 Movement Annotation Max/MSP-patch. Graph showing results of the Perpendicam2 Annotations @ ISSSM ‘08. Graph showing the results of the manual QOM-analysis of Perpendicam2 @ ISSSM ‘08. Screenshot showing the Max/MSP-patch controlling “Beat it”. Picture showing three participants experimenting with the 2.0 version of "Beat it". Graph showing a MatLab plot example of total time in level for one of the groups in “Beat It”. Graph showing the autocorrelation analysis performed on the raw WiiMote-data obtained from one of the groups interacting with the “Beat It”-setup. Picture showing a group of participants and a conductor interacting and performing live with the “Dance2 the Music”-setup. Picture showing a group of participants and a conductor interacting and performing live with the “Dance2 the Music”-setup.
xxxix
Figure 4.12 Figure 5.01 Figure 6.01 Figure 6.02 Figure 7.01 Figure 7.02 Figure 7.03 Figure 7.04 Figure 7.05 Figure 7.06 Figure 7.07 Figure 7.08 Figure 7.09 Figure 8.01 Figure 8.02 Figure 8.03 Figure 9.01 Figure 9.02 Figure 9.03 Figure 9.04 Figure 9.05 Figure 9.06 Figure 9.07 Figure 9.08 Figure 9.09 Figure 9.10 Figure 9.11 Figure 9.12 Figure 9.13 Figure 9.14 Figure 9.15 Figure 9.16 Figure 9.17 Figure 9.18 Figure 9.19 Figure 9.20 Figure 9.21
xl
Picture showing the demonstrator of the IPEM EME & associated HOP sensors. Pie chart showing attitudes for technology-enabled artistic interaction in a cultural context. Picture showing “Heart as an Ocean” (HaaO), an artistic installation based on action-perception coupling using biometric feedback. Picture showing the Lament installation @ Music center Bijloke, Ghent. Software screenshot of the Max/MSP-patch controlling the “Musical Synchrotron”. Picture demonstrating the basic operation of the SIT setup. Close-up picture of the visualization presented to the participants. Picture showing participants playing SIT @ the 2008 IBBT open house, Ghent. Picture showing participants playing SIT @ the 2008 Science Fair, Mechelen. Picture showing participants playing SIT @ NEXT‘08 game convention, Ghent. Collage of the different interview settings Cross tabulation graph of comprehension vs. ability to describe set task. Table showing the results of the chi square-analysis. Compound picture showing interaction with the Virtual string prototype. Picture showing the multi-perspective view on the VS prototype. Picture showing a user operating the music-based framework implementation for interactive sonification. Schematic representation of the framework’s inner software and hardware workings Picture of the laptop-based demonstrator of “SoundField”. Control room of the “SoundField” installation. Table showing open-ended properties of the “SoundField” installation. Schematic Representation of “SoundField”’s elements and behaviors Schematic representation of the implementable scenarios. Table showing the thematic overview of all of the “SoundField” Use Cases. Figure depicting the chronology of the use case development Picture of the SoundObjects demonstration. Picture of the SoundFlock demonstration. Picture of the SoundSculpture 1 use case. Picture of the SoundSculpture 2 use case. Picture of the SoundSculpture 3 use case. Picture of the SoundScapes use case. Picture of the Soundtrails 1 use case. Pictures of the Soundtrails 2 use case (demo and canvas). Picture of the SoundPath use case. Picture of the SoundTracks use case. Picture of the SoundDance use case. Picture of the SoundMovie 3 use case. Table summarizing goals, features and participants for all “SoundField” use cases (2 pages).
Figure 9.22
Figure 9.23 Figure 9.24
Figure 9.25
Figure 9.26
Figure 9.27 Figure 9.28 Figure 9.29 Figure 10.01
Figure 10.02
Figure 10.03 Figure 10.04 Figure 10.05 Figure 10.06 Figure 10.07
Figure 10.08 Figure 11.01
Radar plots showing n° of participants, n° of visual/sonic parameters and total n° of controllable objects for all 22 use cases in the “SoundField” project (3 pages). Graph showing the intended focus of interaction for sonic and the visual parameters. Graph showing measure of complexity of the implemented sonic features compared to the general evaluation of sound (i.e. discernibility, accuracy, suitability and esthetics) for the individual use cases. Graph showing measure of complexity of the implemented visual features compared to the general evaluation of the visuals (i.e. discernibility, accuracy, suitability and esthetics) for the individual use cases. Graph showing the evaluation averages for both sound & visuals compared to how well the relation between the two was perceived (for individual use cases). Graph showing use case level of complexity compared to the overall evaluation of the social interaction indicators. Graph showing use case level of complexity compared to the overall assessment of the use case. Graph showing number of participants compared to Quality of Social Interaction. Evolution of digital participation (from top to bottom): [1] culture consumption by means of screens, [2] accessing cultural information, [3] acquisition of cultural products, and [4] creation of (cultural) content [source: PaS 2009 & 2014]. Digital participation rates over the last ten years presented by age cohort for [1] information, [2] product distribution and [3] cultural experience in Flanders and Brussels [source: PaS 2004, 2009 & 2014]. Pie chart showing distribution of data entries according to institution type (n=1230). Relative frequency of posts placed on social media by various types of institutions. Use of Facebook and Twitter by cultural organizations for promotion of events and interaction with followers/audiences. Table showing the competence model, skill, activity level and competence type. Graph showing the evolution of the most used interface for the three age categories included in the study (i.e. 9-11 years old, 12-13 years old, 14-16 years old). Table showing results obtained for eleven of the media literacy indicators included in the 2014 PaS. Differences in setup-properties between most traditional music mediation controllers and the development model used for “SoundField”.
xli
Table of Contents
Dankwoord .......................................................................................................................... xi Abstract and Keywords (EN) ............................................................................................xiii Keywords .......................................................................................................................... xvii Samenvatting en kernwoorden (NE) ............................................................................... xix Kernwoorden ...................................................................................................................xxiii Preface and Acknowledgements...................................................................................... xxv List of Abbreviations .................................................................................................... xxxvii List of Figures ............................................................................................................... xxxix Table of Contents ................................................................................................................. 1 Part 1 : .................................................................................................................................. 7 Introduction, Theoretical Framework & Methodology ..................................................... 7 1
Introduction .............................................................................................. 11
1.1
Art as a means of communication .......................................................... 11
1.2
Issues ......................................................................................................... 13
1.3
Research Context ..................................................................................... 15
1.4
Working Hypotheses ............................................................................... 17
1.5
Research Questions.................................................................................. 19
1.6 Main perspectives .................................................................................... 21 1.6.1 Installation art as an ecological testbed for research ............................................ 21 1.6.2 Shift from user-centered to participatory .............................................................. 22 1.7
Stylistic conventions, reference system and software ........................... 23
1.8
Overview of the chapters ........................................................................ 24
1
2
Theoretical Framework ............................................................................ 27
2.1 Theoretical Introduction to the Embodiment Paradigm ..................... 27 2.1.1 Status Quaestionis: musicology and embodiment ................................................ 27 2.1.2 The body as a mediator ......................................................................................... 29 2.2 State of the art on new interfaces for musical expression .................... 34 2.2.1 Art Technique Craftsmanship Technology Skill ............................. 34 2.2.2 Computer Music .................................................................................................... 35 2.2.3 Tactile interfaces, gestural controllers and haptic feedback ................................. 35 2.2.4 New interfaces for musical expression ................................................................. 36 2.2.5 Immersive Environments in Artistic Practice ....................................................... 38 2.3
User-centered procedures: an overview of applicable HCImethods (Usability, UI, UE and UX) ...................................................... 43 2.3.1 HCI and usability: standards, methods and metrics .............................................. 43 2.3.2 User-centered approaches for artistic interfaces ................................................... 44
2.4 State of the Arts in the Digital Design Procedures ............................... 47 2.4.1 Open-endedness .................................................................................................... 47 2.4.2 A user-inspired development strategy................................................................... 48 2.5
The cultural-creative sector, adoption and literacy ............................. 49
3
Methodology and Conceptual Model ....................................................... 53
3.1 METHODS ............................................................................................... 53 3.1.1 Applied experimentation ....................................................................................... 54 3.1.2 Interface comparison ............................................................................................. 54 3.1.3 Survey research ..................................................................................................... 54 3.1.4 Interview Analysis ................................................................................................ 55 3.1.5 Usability Inspection Methods ............................................................................... 55 3.1.6 Participatory Design Strategy and QoE Evaluation .............................................. 56 3.2 DESIGN GUIDELINES .......................................................................... 57 3.2.1 Setup...................................................................................................................... 57 3.2.2 Strategy ................................................................................................................. 58 3.2.3 Design guidelines .................................................................................................. 59 3.3 DEVELOPMENT .................................................................................... 62 3.3.1 Challenges ............................................................................................................. 62 3.3.2 Iterative Design Model.......................................................................................... 63 3.3.3 Conceptualization of the Evaluation Metrics ........................................................ 64 3.4
2
Conceptual Model .................................................................................... 67
Part 2: ................................................................................................................................. 69 Empirical Studies ............................................................................................................... 69 4
Initial Research Studies............................................................................ 73
4.1
“Perpendicam" ........................................................................................ 73
4.2
“Beat it” & “Dance2 the Music” ............................................................. 78
4.3
Rudimentary Usability & Mapping test with the EME @ ISSSCCM 2009 ........................................................................................ 82
4.4
Conclusion ................................................................................................ 84
5
What moves you ........................................................................................ 87
5.1
Motivation ................................................................................................ 87
5.2
Purpose ..................................................................................................... 88
5.3
Survey setup, structure and contents..................................................... 89
5.4
Results ....................................................................................................... 91
5.5
Interpretation and discussion ................................................................. 97
5.6
Conclusion ................................................................................................ 99
6
Audience perception of EMMTs: case studies with HaaO &
Lament
101
6.1
Heart as an Ocean.................................................................................. 101
6.2
Lament .................................................................................................... 103
6.3
Conclusions............................................................................................. 104
7
Interface Evaluations using the "Sync-in-Team" device ..................... 107
7.1
Setup Description ................................................................................... 108
7.2
Experiment locations and setup modifications ................................... 110
7.3
Interview circumstances........................................................................ 112
7.4
Interview Data........................................................................................ 113
3
7.5
Analysis and Results .............................................................................. 114
7.6
Evaluation ............................................................................................... 116
7.7
Conclusion .............................................................................................. 117
8
Usability Heuristics and Metrics: the Virtual String Study & the
Multi-Level Sonification Tool ......................................................................................... 121 8.1
Virtual objects as a mediator principle ............................................... 122
8.2
Virtual String ......................................................................................... 123
8.3
Multi-level Sonification Tool ................................................................ 124
8.4
Conclusion .............................................................................................. 127
9
Participatory and Iterative Design Study: “SoundField” ..................... 129
9.1
Introduction............................................................................................ 129
9.2
Prototypes and developmental strategy: ............................................. 131
9.3 Elements, Behaviors and Scenarios: .................................................... 133 9.3.1 Elements and Behaviors ...................................................................................... 134 9.3.2 Scenarios ............................................................................................................. 135 9.4 Use cases ................................................................................................. 137 9.4.1 Demonstration use cases ..................................................................................... 139 9.4.2 Sound manipulation use cases............................................................................. 141 9.4.3 Sonification tools ................................................................................................ 145 9.4.4 Choreography tool............................................................................................... 147 9.4.5 Theatrical use cases ............................................................................................. 148 9.4.6 Movie manipulation tools ................................................................................... 150 9.4.7 Educational and Therapeutic demonstrators ....................................................... 151 9.5 Evaluation ............................................................................................... 156 9.5.1 Post-demonstration and pre-interaction evaluation ............................................ 156 9.5.2 Post-interaction evaluation .................................................................................. 157 9.5.3 Complexity Indexes ............................................................................................ 157 9.6 Results ..................................................................................................... 158 9.6.1 Evaluations .......................................................................................................... 163 9.6.2 General tendencies: ............................................................................................. 169 9.7
4
Discussion and Conclusion .................................................................... 171
Part 3: ............................................................................................................................... 175 Epilogue, Discussion & Conclusion ............................................................................... 175 10
Epilogue .................................................................................................. 179
10.1 10.1.1 10.1.2 10.1.3
Digital Culture Participation and Audience Engagement ................. 180 E-culture and Digital Participation .......................................................... 180 Mapping of Digital Technology-Use in the Cultural Industries .............. 182 Discussion, Conclusion and Policy Recommendations........................... 186
10.2 10.2.1 10.2.2 10.2.3 10.2.4
Media Literacy as a Requirement ........................................................ 187 About Media Literacy .............................................................................. 188 Competence Models and Conceptualization ........................................... 189 Relevant Findings .................................................................................... 190 Discussion and Conclusion ...................................................................... 192
11
Discussion ............................................................................................... 193
11.1
Evaluation of the Research Questions ................................................. 193
11.2
Validity of the hypotheses ..................................................................... 198
11.3
Limitations of the final study and recommendations for future research................................................................................................... 200
12
Conclusions ............................................................................................. 203
12.1
Overview of results ................................................................................ 203
12.2
Accomplishments and value ................................................................. 208
12.3
Thoughts on the technological & academic aftermath ...................... 210
List of Publications .......................................................................................................... 213 Bibliography ..................................................................................................................... 215 Appendices ....................................................................................................................... 243
5
Part 1 :
Introduction, Theoretical Framework & Methodology
7
The first part of the thesis consists of three chapters that present the general state of affairs. The first chapter introduces the main topic, the research context, the artistic and scientific perspectives, the working hypotheses and research questions. The second chapter surveys the theoretical concepts that the methodologies and the empirical studies put to use over the remainder of part 1 and 2 of the thesis. The third chapter, provides the methodological framework, design guidelines, the model that was used for the development strategy and the conceptualization of the evaluation types.
9
1
Introduction
At the start of this chapter, a short introduction is given concerning the state of the arts for new interfaces for interactive artistic expression and the paradigm shift they have brought about for both the research community and the cultural-creative industries alike. Then, some of the inherent problems of associated technologies are discussed. Consequently, the context of the project is offered, the research topics in which the presented studies were performed are dealt with, and the current issues related to this research line are enumerated. Subsequently, the bibliographical and methodological conventions used in this thesis are briefly touched upon. Finally, a short overview and abstract is given of the chapters in this thesis.
1.1 Art as a means of communication Art has always been used as an instrument for communication. Traditionally, by employing prevailing laws of aesthetics and iconography imbedded and understood by the culture of origin, meaning is transmitted. Since the advent of conceptual art, the cultural norms of iconography and aesthetics have become broader - at times blurred - and art creation has in some instances become more intuitive and a less culturally constrained practice. It may be argued that because of this, the nature of art has become more openended (Sabbe, 1989), and given the great rise in ICT-use of the past decades, this element of openness has been further amplified by the apparent lack of tool definition and explicit purpose. This shift from art as a defined way of communication towards open expression has not seldomly resulted in an increased illegibility. This incomprehensibility itself can be considered an obstacle, yet, when combined with (hard to use) interfaces, such as those not uncommon in new media art, it becomes very problematic, especially for interactive art where the participation of the audience is key to the experience, for the audience to
11
discover the meaning of the art-piece (Manovich, 2001). Therefore, the goal of introducing an embodied foundation for making new media art is to instigate meaningful interaction, as well as to provide the content, the intention behind it and the function of the art-piece (Coussement et al., 2010). New technologies based on gestural devices that are connected with the human body can be developed; with a user-centered design strategy, media art may be regarded as the enabler of this methodology. The advent of these technologies has continuously opened new possibilities for sound synthesis and extended control beyond traditional interfaces, a trend that has made rapid advances and has seen exponential rises in technological resources and potential (Paradiso & O'Modhrain, 2003). Given these new possibilities, an artistic as well as academic aspiration has taken form to engage in a more spontaneous, natural and embodied interaction (Leman, 2008) with musical content through the use of gestural interfaces. Consequently, during the last decades, the emphasis in music research has gradually shifted from synthesis and analysis to mobile and tangible interfaces, various types of sensors and motion capture systems that can be put to use to support this musical expression (Fishkin, 2004). With this tendency, the study of gestural control and mapping strategies, namely how the capabilities of the human sensorimotor system can be linked to the sonic characteristics and parameters of the digital instrument (Miranda & Wanderley, 2006) has become the focus of study and the goal in the development of these interface-based instruments. HCI and usability studies are supplying the tools for evaluation (Wanderley & Orio, 2002; Birnbaum et al., 2005; Kiefer et al., 2008) and gradually, evaluation standards and guidelines are being developed that enable the comparative analysis of collaborative interfaces for musical expression (Blaine & Fels, 2003). In recent times, the far-reaching adoption and use of new media interfaces and tools has had a profound and complex impact on society, not in the least on the creative industries. A veritable metamorphosis can be discerned in the cultural and creative industries over the last decades. Partially because of this, human-computer interaction studies and usability studies have evolved into a well-established research field with set methodologies, protocols (Scapin & Law, 2007) and standards (Bevan, 2001). The field has been refined and practiced increasingly by organizations, research institutes and industry and usability studies have made considerable and invaluable contributions to the ergonomics of various types of human-computer interaction.
12
1.2 Issues Recent decades saw a significant rise in availability of new digital tools and technologies that support interactivity in a cultural context. In scientific and artistic research as well as creative industries, a growing interest for these technologies can be attested and efforts on the development of collaborative and interactive interfaces are being made. Some of these technologies have the potential of interfacing art in such a way that they enable new possibilities for audience interaction and artistic expression for performers and audiences alike. Cultural institutions do not always take an unambiguous stance towards these collaborative and interactive artistic technologies. Aside from the financial burden brought about by the considerable investments required for technologically enhanced interaction spaces, reliability and maintenance issues that pervade current state-of-the-art technological implementations can be mentioned here among the chief reasons. However, other obstacles hinder technology acceptance for the cultural sector and its audiences alike. These adoption issues can best be summarized as a dichotomy between the demonstrable benefits of technological availability, advances and the increased artistic opportunities on the one hand, and the problematic relation with readiness, eagerness, adoption and domestication of technology on the other. One of the aims of this thesis is to identify to what extent the adoption is hindered by technological and budgetary restrictions, as well as to discover other factors that contribute to the aforementioned adoption problems. An additional concern is to find out whether these adoption problems are mainly related to the cultural sector and its institutions, to target audiences, or to both. At present, not all types of new interfaces and associated new modes of interaction are exhaustively dealt with. Since the advent of gestural interfaces, a lot of new problems have arisen with applying traditional usability metrics and there is an apparent lack of standardization and consistency (Norman, 2011). In music research, and more specifically, in the development of new interfaces for musical expression and music technologies [(Poupyrev et al., 2001)] (Wanderley & Orio, 2002), this is the case even more so. Computer-based technologies that enable musical expression are a relatively new concept and the research field is in constant evolution and revolution. The rise of multi-, new and unstable media art, moreover, has brought about an evolution in the artistic landscape and the reality of the cultural sector. The changes pertain to various aspects of the sector and are noticeable in research and development departments of the creative industries, in policies and programming of cultural institutions and in the position of culture-enthusiastic audiences alike. These changed perspectives and new attitudes towards new media technologies have prompted the paradigm of embodiment to enter a broad scope of music research.
13
The topic of this research and of this thesis is therefore on a sociomusicological investigation into the cultural-creative application of embodied mediation technology in the music sector. This research project focusses on the sociomusicological aspects of these new technologies (i.e. through sociological studies and methodologies). Within this perspective, Embodied Music Mediation Technologies are studied, i.e. state-of-the-art musical expression tools, tangible interfaces, immersive virtual and augmented reality environments, collaborative controllers and multimedia devices that to a certain extent rely on the paradigm of embodiment. Within the exploratory scope of this thesis, the applicability and genuine application of these technologies in the cultural-creative sector (i.e. the broad scope of industry, universities, private-public intermediaries and cultural institutions) is assessed. Eventually, the goal is to work towards definition, opportunity ascertainment and eventual valorization of embodied music mediation technologies as well as assess properties of new media adoption, new media literacy, new media readiness and eagerness, within a regional scope and specifically in view of the cultural and artistic sector. However, numerous issues pertaining to the implementation of new gestural interfaces for computer-based musical expression remain to date unresolved and the more recent fascination from the research community for immersive, augmented and virtual reality environments, makes these issues even more substantial and problematic. First, the cultural and aesthetic goals related to the development of these technologies are very varied given the number of different practitioners and conceivable applications. Their purpose may range from fundamental academic research into fields such as performance, composition, dance, choreography, sound design, game development, interactive media installation art and education. Given the fact that in the context of design of electronic musical instruments the aspect of musical performance is often overlooked, goalassessment of these tools for musical expression is by no means straightforward and consequently this type of human-computer interaction to an extent eludes traditional usability heuristics and metrics and some of the formalized methods are not readily applicable. Therefore, more often than not, there is a discrepancy between the functionalities of certain technologies and the collaborative aspirations users may have for them. This apparent lack of standardization among the various purposes, interfaces and applications, consequently instigates confusion in users, cultural institutions and cultureenthusiast audiences. Moreover, there is no common set of standards in existence to evaluate these very divergent and multi-purposed designs and finally, given this multitude of possible uses and operation strategies, there is a problem pertaining to the broad audience’s understanding of how instruments function and are being played. As both audiences and cultural institutions are insufficiently acquainted and confronted with the possibilities latent in state-of-the-art music mediation technologies, both the sector and culture participants have a hesitant and rather unpositive attitude towards these technologies. Finally, to date, there are many reliability issues to be reported in most
14
technology-based systems that enable cultural expression. The combination of these issues hinders adoption and instigates a reluctance to deal with such systems (Paradiso & O'Modhrain, 2003).
1.3 Research Context Summarizing, there is myriad of problems pervading the development of embodied music mediation technologies: first, there is the issue of embodiment to consider, namely, how music interaction principles relate to the foundations of embodied interaction and music perception (e.g. links to the field of interactive sonification). Secondly, there are various different perspectives of user specificities to consider: a user can take on both the role of a performer or a member of the audience, both roles having different stances towards interaction. Thirdly, and directly linked to these user roles, there are usability issues and human computer interaction principles to consider, namely, what constituted good user experience and proper functioning of said devices and systems, as they determine the users' willingness to interact, technology acceptance and eventually adoption and domestication. As a fourth research topic, there is the issue of location to consider; highly technical immersive installations as well as low-key interactive applications can be set up in either lab contexts, performance spaces or in museum contexts. All require specific and manifestly different interaction principles and have specific technical requirements. The contents of this thesis originate in the goals and requirements of the EMCOMETECCA Metusalem project (i.e. Embodied Music Cognition and MEdiation TEchnology for Cultural and Creative Applications), awarded to the Institute for Psychoacoustics and Electronic Music (IPEM) of Ghent University. Within the scope of this project, the fundamental requirements were defined as three interlinked scientific goals. For the first scientific goal, which can be described as “fundamental empirical research", the aim was to define a set of relevant epistemological concepts concerning the theory of embodied music cognition and to organize a fundamental research roadmap concerning the link between psychoacoustic and sensory-motor fundamentals that underpin the mechanics of embodiment in artistic praxis. For the second goal, “system development", the aim was to generate a set of state-of-the-art technologies (i.e. interfaces, devices and platforms) and application domains that can simultaneously support embodied interaction with musical content, as well as the research endeavors defined in the first scientific goal of the project. The third scientific goal, dubbed “useroriented approaches”, went into user-related issues concerning the interaction with and implementation of the technologies defined in the second scientific goal. This thesis focusses on the research questions defined in this last scientific goal. 15
This research line was initially described as “a sociomusicological research into the application of embodied music mediation technologies to the cultural-creative sector”. The research aims of this work package pivoted around four basic concepts, all of which the studies contained in this thesis deal with to some extent. These concepts can be summarized as 1) sociomusicology, 2) taxonomy, 3) usability and 4) applicability. What is understood by sociomusicology within the scope of this thesis, is that within the paradigm of music sociology, all music listeners and producers are considered potential audiences (and more specifically participants, test-subjects and technology users). Personality traits and musical preferences are considered, but in essence all participants are regarded as equals, inconsiderate of their musical background, their performative skills, stylistic predilections and gestural idiosyncrasies. Consequently, within the scope of this thesis, the ambition is on the one hand to cater for the general technological aspirations, while on the other hand making allowances for individual specificities. Concerning the second concept, out of the specificities of various state-of-the-art interfaces for artistic expression, some general tendencies are deduced. What is understood with a capture-all term like "Embodied Music Mediation Technologies” (EMMTs), can go into many directions, e.g. tangible interfaces for musical expression, malleable and responsive artistic tools, virtual, immersive and augmented reality environments, collaborative controllers and interactive multimedia installations. The point is that in this plethora of devices, in broad lines the basic functionality is similar, irrespective of shape and size. Therefore, a taxonomy of these devices can be made to enhance our understanding of the mechanics of these interfaces. Out of this diversity of interfaces and application domains follows the necessity of the third concept, i.e. to generate some basic guidelines concerning optimal operation. As all usage of sensor-based music technologies basically entails some form of humancomputer-interaction, there is no reason to assume these technologies would be impervious to typical usability problems. The quality of interaction determines the success of use and the proneness for non-use of a given interface. Therefore, immediate affordance and seamless functionality are vital, and intelligibility of action-response mapping, transparency of sonification and optimal quality of experience are much desired qualities. Expanding on this, the fourth concept comes into play, as the context of use to a large extent influences and even determines the aforementioned experience. Whether or not an application is suitable for a certain purpose, is a function of the complexity of operation, of social variables and of the locality of use.
16
1.4 Working Hypotheses The use of embodied music mediation technologies has been pervading and dominating systematic musicology research for over a decade (e.g. the success of conferences such as NIME). With new media adoption and the diffusion of state-of-the-art technological devices heading towards nearly 100% penetration and adoption in young populations (Schuurman et al., 2011) and educational institutions rapidly catching up to this tendency (Birchfield et al., 2008; Tselios et al., 2008), inversely, cultural institutions as well as its audiences struggle with the acceptance of this technological evolution (Nielsén, 2009). This raises very pertinent and fundamental questions concerning technologies that employ embodiment, as well as obstacles that need to be overcome methodologically. The studies described in this thesis are timely, as they zone in on some of these existing lacunas in the broad multi-disciplinary domain of music research. There is no lack of interest in music mediation technologies from various academic fields. Since some of the various elements inherent in EMMTs and aspects relevant to the development AI&VREs are so strongly interlinked, the chapters describing the empirical studies take on a number of standpoints (i.e. research hypotheses) concerning EMMTs and AI&VREs, which, through an RtD-approach, are ascertained or disproven. Concerning the paradigm of embodiment, this thesis assumes that its core concept is universal and innate in humans, irrespective of sociodemographic or culturally defined characteristics, and is therefore particularly well suited and universally applicable in the design and development of gestural music interfaces. Concerning the basic operation strategies, this thesis assumes that EMMTs can foster a distinct (and more direct) interaction with musical content than traditional musical instruments, based on the principle of embodiment, more specifically through the means of the illusion of non-mediation. Concerning the validity of the interaction metaphors for cultural expression, this thesis assumes that the interaction strategies with the technology follow a set of patterns which are essentially natural, and therefore are intuitive to learn, relatively simple to master and extremely well-suited to be used in both social and interactive contexts. Concerning technology adoption of EMMTs, this thesis assumes the necessity of particular AI&VRE and EMMT-interfaces to be rigorously tested across various art practices and artistic application domains. 17
To assess the applicability of theoretical concepts such as flow, presence and affordance to AI&VRE and EMMTs, this thesis assumes parameters adhering to these concepts to be elaborately tested during lab-based and/or real-life interaction with these interactive music devices. This is instrumental in the examination and incorporation of social interaction patterns and gaming strategies into interactive music designs. Irrespective of the general applicability of the concept of embodiment, this thesis assumes the assertion of participants’ wants, needs and abilities can ameliorate (prospective) users' comfort and understanding of interface operation. Moreover, the thesis assumes that the vital information conveyed in the in-depth assessment of users' aspirations combined with this enhanced user-comfort and understanding during the operation of the interface can in turn help in improving the EMMT-based catering for various cultural practices and artistic goals. Concerning AI&VRE development methodology, this thesis presumes an RtDapproach to be beneficial for probing whether standardized HCI evaluation methods can be applied and geared towards EMMTs (as, even though the interaction metaphor is grounded in embodiment, the technology is in essence software-based and computer-driven. Moreover, as is also the case in other current Living Lab research, the thesis assumes that within development strategy, concepts like "iterative", "participatory" and "parallel design" will accelerate development, but need to be adopted from computer engineering into a predominantly artistic context. Concerning ecological validity, this thesis presupposes that the successful cultural implementation of the described technology greatly benefits from labbased interface assessment and scenario-based development, but can essentially only be achieved through in situ testing. Furthermore, the same goes for the design scenarios in which social interaction can occur: the collaborative modalities need to be examined in view of creating interactive opportunities within some of the EMMT designs, yet the testing the incorporation of social strategies can only be achieved through live interaction with the presented technologies.
18
By this stage, it is obvious that the topic at hand is beset on all sides by unpredictability. Apparently, "how and what musical parameters need to be addressed to further develop current embodied interactive music systems" is unclear at this point. "What information about prospective users is vital to prompt further development" is to date open to question. "Whether user-profiles really need to be established and how they should give rise to the definition of expressive goals in specific artistic forms" is at present debatable. "In what way usability affects spontaneous engagement, ameliorates quality of experience, enhances adoption and ultimately can facilitate ecological embedding in the cultural-creative sector" is at this stage still up for speculation. In what follows, a thematically organized enumeration of research questions will be given.
1.5 Research Questions The previous point lists several assumptions that are presupposed to make sense of the topic, and a set of postulations, which the thesis aims to empirically test throughout the different stages attested in the following research studies. Yet, many more fundamental questions lie at the basis of these hypotheses. In order to be able to evaluate current computer-based music mediation technologies, it is essential to establish 1) how undefined –or rather open-ended– artistic purpose and intent can be dealt with, 2) how non-task oriented development and flexible wielding strategies need to be anticipated, and 3) how meaningful social and interactive communication can be implemented in a functional yet customizable design. Furthermore, given the specific nature of these technologies, reassessment of a set of usability heuristics and metrics that either are suitable (e.g. self-reported metrics), that need to be readjusted (e.g. technology acceptance models) or that cannot readily be applied (e.g. performance metrics) is required. This entails standardization of design and assessment parameters, in order to be able to evaluate the qualities and usability problems of different types of immersive/collaborative/interactive music mediation systems and to a certain extent surpass the application-specific elements like, choice of input device, interfacing technology, feature extraction, mapping strategy and output. Conversely, amidst the vast abundance of interfaces that support technology-enabled artistic expression, to date the promise of a universally understandable and meaningful embodied interaction is yet to be delivered and cultural industries have not adopted (let alone implemented) these technologies to their full potential.
19
Therefore, in view of the project goals and prevalent problems, it becomes clear that both in terms of theoretical concepts and available methodological tracks, this research topic has very specific and interdisciplinary requirements, as its research questions come from the field of musicology, sociology, cultural studies, HCI and user research and deal with artistic vision, creative technologies, engineering, culture participation, interactivity, user experience, and so on. The following list of research questions serves as a basis for the following chapters. Concerning the concept of embodiment: Assuming the paradigm of embodiment is universal, innate, spontaneous and natural; what are the factors that generate the interaction problems with EMMTs? Concerning the basic operation of EMMTs: What is the state of the art concerning New Interfaces for Musical Expression and Embodied Music Mediation Technologies? What opportunities are there for categorization of such systems, devices and interfaces? What technological issues do EMMTs suffer from? Concerning interaction and adoption: Why is the adoption process (by the cultural sector) so difficult? What are the obstacles that are hindering user adoption? What is the cause of the problems pervading EMMT adoption? Which of the stakeholders require what means and/or tools? Concerning natural, social and interactive: What does more “natural” interaction constitute from the part of the end-user? How can social interaction be promoted during EMMT interaction? What factors hinder it? What constitutes interactivity in art? What enables it? What impedes it? Concerning artistic expression and application domains: How diffuse is the scope of artistic opportunities and application domains? What are the application-specific favorable circumstances for embedding? How viable are certain application domains within a given context? Can some broadly applicable parameters be identified for optimal functioning of the aforementioned devices? Concerning AI&VRE development methodology: What method(s) do we need to gather the relevant information from the participants (what questions, how to present them)? 20
How can aspirations, prerequisites and necessities of potential end-users ideally be identified? How can functional, artistic and aesthetic requirements be accommodated technically? What can be learned from earlier implementations (i.e. the transition from implementation/instance-based to iterative (RITE) design)? Can elements from traditional HCI and usability studies be applied to assess the functioning of EMMTs? Are there opportunities for HCI-standardization within the specific field of EMMT development? Concerning HCI evaluation methods applied to EMMTs: How can implementations ideally be evaluated (beyond the scope of singular implementations and across various application domains)? How can different design-elements and system/setup aspects be measured? How can (and should) feedback ideally be recorded and re-applied?” What UX-based metrics can be defined for the optimization of the functionality/usability of these devices?
1.6 Main perspectives 1.6.1 Installation art as an ecological testbed for research
In a laboratory context, experiments are designed to limit the number of variables in play. This allows for the systematic exclusion of confounding factors and generally makes data analysis more straightforward and the results more compelling. However, a laboratory setup often presents circumstances that are rather detached from real-life, which can make the experience unnatural for the participants, especially when dealing with interactive technologies. Consequently, although suited to research purposes such as prototype evaluation, experiments conducted in a lab-context do not convey the same information as tests done in more everyday cultural settings. As a result, there is a need to achieve a balance between the accuracy of hypotheses testing in experiments and the naturalness of the experience and situation. Science, especially cognitive science and research on human-computer and humanrobot interaction, uses interactive art as a testbed for studying action, perception, and
21
cognition (Seifert, 2008). In addition, new media exhibitions and (interactive) art can enable user-oriented research, serving as a valid and a genuine context in which to conduct user-oriented tests. The contextual, iconographic and aesthetic aspects of the art piece can guide the user with respect to the purpose of the experiment, without limiting the overall experience. Consequently, to meet the functional needs of a given artistic context and assess an artistic purpose of the technology, we advocate that experiments with slightly more advanced setups should be conducted in the cultural sector. New media festivals (e.g. Ars Electronica, Transmediale) form an excellent ecological testbed for this type of research.
1.6.2 Shift from user-centered to participatory
As such, the process of creating content for an interactive music application is often an exercise in obtaining a balance between the inspiration and the technical resources and limitations of the research team working on it. In practice, a music application is often created by a number of researchers who between themselves decide upon the constituent parts of that application, like the interface that will be used, feature extraction, mapping strategy, feedback, etc. (Wanderley & Orio, 2002). Using user-related information and feedback is as such not new to this type of development-process (Stowell, Plumbley & Bryan-Kinns, 2008), but to date it is integrated to its full potential or throughout all the development stages into the domain of systematic musicology. To have prospective users evaluate an application and collect their feedback is a procedure often practiced only after the main developmental work is finished. The collected feedback is then only important to make smaller adaptations to the pre-existing design. Given the flexible nature of embodied music mediation technology (Leman, 2008), this presents both tremendous opportunities as well as equally large problems. A broad range of cultural and creative goals are defined in European roadmaps and action plans (e.g. HORIZON 2020), but the importance and necessity of involving prospective end-users in the process of tool building is not stressed, and to date largely overlooked. There is a wide range of conceivable creative application domains, a cornucopia of technologies to choose from (Blaine & Fels, 2003) and a broad spectrum of cultural contexts where embodied music mediation technology could be introduced. Even so, at present, an outspoken viewpoint on how this technology can/should be embedded in the existing cultural landscape is lacking, and perspectives on viable deployment strategies are vague and isolated instances at best.
22
1.7 Stylistic conventions, reference system and software
The following conventions are used for the composition and formatting of the thesis. Writing and orthography adhere predominantly to the U.S. English dictionary (exceptions being some U.K. English terms and minor Dutch passages). In terms of indexing and numbering, the chapters prior to the body of the thesis are given roman numerals, the appendices are presented with an Arabic number. The thesis chapters and paragraphs are hierarchically numbered as follows: 1, 1.1, 1.1.1 etc. The referencing style predominantly follows the American Psychological Association (APA) style (6th edition). For in text references the following conventions are applied: (Last Name, XXXX) when the referenced work has only one author, (Last Name & Last Name, XXXX) when the referenced work has two authors, and (Last Name et al., XXXX) when the referenced work has three or more authors. Art pieces, music applications, demos, software platforms or interactive installations that were developed in the broader scope of the EMCOMETECCA-project are referred to as “Name” in the text. Footnotes are incorporated sparingly throughout the text, mostly limited to chapters 5 and 10, and as an elaboration on the presented survey results, or in reference to studies conducted outside the scope of this thesis. For the completion of the thesis’ projects, the following computer programs were used: for keeping a research log and the construction of the thesis chapters, Evernote was used primarily. As a referencing software and for keeping all academic source material organized, experiments were done with Endnote and BibDesk, but eventually Zotero was used. Software programming for demos and applications was done predominantly in Max/MSP/Jitter, but in the later stages of the research project SuperCollider, Java 3D and NetBeans were also used. For sound processing, aside from Ableton Live, Logic, GarageBand and Audacity were used primarily. Manual annotations of video were done in Annotation and ELAN, but for the most part, a custombuilt application developed in Max/MSP/Jitter was used. As the main presentation software, Keynote was used, but together with PowerPoint, it also served for the main program for formatting schemes, graphs and figures. All semi-fixed interviews and questionnaires were constructed using SONA or Qualtrics at first, and eventually in the open source LimeSurvey. As a local server environment, MAMP was used to test and run the surveys. For data analysis, some of the preliminary results were processed using MatLab, but the majority of the data was gathered, cleaned and processed in Excel, and analyzed SPSS and with the help of prof. dr. Andy Field’s book “Discovering Statistics Using SPSS” (2009).
23
1.8 Overview of the chapters In the first part of the thesis – covering three chapters – the general context of the thesis is given. In the first chapter, an introduction to the topic is given and the problems are defined. In the second chapter, a broader theoretical background is presented of the concepts that underpin the project, namely 1) the paradigm of embodiment and its connection to musicology, 2) a state of the arts concerning new interfaces for musical expression, 3) an introduction into HCI-usability and its application domain in systematic musicology, 4) an insight into user-centered digital design procedures, and 5) the challenges brought about by e-culture and digitization for the cultural-creative industries. Within that scope, concepts related to activity theory, ergonomics, user experience, flow, presence, goal assessment and game enjoyment are scrutinized, applied and operationalized in both task- and experience-oriented heuristics and metrics (developed and tested throughout the different experimental stages described in this thesis. In the third chapter, the state of the arts of the available methodologies related to the thesis’ endeavor is discussed and a set of literature-based design guidelines are enumerated. These design guidelines (on how to cater for musical interactivity and collaborative interfaces), describe a set of the interaction design caveats and the usability maxims. From this a conceptual model is deduced which will be fully deployed in the “SoundField” project (as described further on in Chapter 9), which consists of 1) applied experimentation, 2) interface evaluation, 3) interview techniques, 4) self-reporting and survey research, 5) usability evaluation of existing prototypes, and 6) human-computer interaction methods applied and attuned to embodied music mediation technologies. The following chapters – contained in part 2 of the thesis – give a quasi-chronological overview of how methodological concepts have gradually been applied throughout a number of case studies, aimed specifically at the exploration of the various aspects of the complex status quaestionis. In the fourth chapter, a series of application-based tests, predominantly revolving around interface evaluation, illustrate the complex relation between gestural interfaces and meaningful musical expression, advocating a more usercentered development approach to be adopted. In the fifth chapter, a multi-purpose questionnaire dubbed "What Moves You" is discussed, which aimed at creating a survey of the (prospective) end-users of embodied music mediation technologies. Therefore, it primarily focused on cultural background, musical profile and preferences, views on embodied interaction, literacy of and attitudes towards new technology and participation in digital culture. In the sixth chapter, the ethnographical studies that accompanied the exhibition of two interactive art pieces, entitled "Heart as an Ocean" & "Lament", are discussed. In these studies, the use of interview and questionnaire methodologies, in conjunction with the presentation and reception of interactive art pieces, are probed. In the seventh chapter, 24
the development of the collaboratively controlled music-game Sync-In-Team is presented, in which interface evaluation, presence, game enjoyment and goal assessment are the pivotal topics. In the eighth chapter, two usability studies are considered, that were conducted on prototype systems/interfaces, namely a heuristic evaluation of the "Virtual String" and a usability metrics evaluation on the "Multi-Level Sonification Tool". The findings of these two studies in conjunction with the exploratory studies performed in association with the interactive art pieces, finally gave rise to the “SoundField”-project, which is recounted in full throughout the ninth chapter, the integrated participatory design and evaluation method, presented in the conceptual model, is fully applied during the course of the “SoundField”-project, in which technological opportunities and ecological validity and applicability are investigated through user-informed development of numerous use cases. The third part of the thesis renders the final conclusions of this research project. The tenth chapter sets out with an epilogue in which a brief overview is given on how the state of the arts has evolved since the end of the project (as the research ended in 2012, but the research field has obviously moved on), and attempts to consolidate the implications of the research studies with some of the realities of the Flemish cultural-creative industries. Chapter eleven continues by discussing the strengths and weaknesses of the conceptual model throughout the various stages of the project. Also, it comprises the evaluation of the hypotheses, how the assumptions that were made held up, and how the research questions eventually could be assessed. Finally, in the twelfth and last chapter concludes with the most important findings of the project. Also, it discusses some of the implications on cultural production, artistic research policy and offers an outlook on future research beyond the scope of the “SoundField” project.
25
2
Theoretical Framework
As the previous chapter recounts the broad context in which underlying research studies were conducted and discusses the contours of the central thesis, this chapter delves into the concepts that underpin both the conceptual model and the experimental framework for the thesis. These concepts comprise the foundations of embodiment as well as nexus between embodiment and musicology and the application domains of the embodiment paradigm for EMMT development. Next, an overview is given of the state of the arts in new interfaces for musical expression by means of an anthology of relevant developments leading up to the starting point of this research project. Then, an overview of applicable HCI-methods and potential user-centered procedures concerned with EMMT & AI&VREdevelopment is presented, and finally, new media adoption and the cultural-creative industries are concisely discussed.
2.1 Theoretical Introduction to the Embodiment Paradigm 2.1.1 Status Quaestionis: musicology and embodiment Musicology has come a long way since Adler’s "Umfang, Methode und Ziel der Musikwissenschaft" first described the research field and in 1885 defined its subdisciplines – historical and systematic musicology. Since antiquity, the fundamentals of music have already been debated by philosophy and historically, music has received a great deal of attention from other scientific fields such as psychology, (cultural) sociology and pedagogy. Of late, even such disciplines as engineering and computer science have joined in on the action and started sharing a mutual interest for the subject of music. Therefore, being a scholar of current systematic musicology (Parncutt, 2007) and operating under the new musicological regime (Agawu, 1997) has somewhat necessitated the adoption of numerous ‘new’ methodologies and paradigms, traditionally associated
27
with different academic fields, other than those of music theory, music history and ethnomusicology (Clayton, Sager & Will, 2005), lest musicology itself would be overrun by its neighboring research fields, or become obsolete altogether. Sociology and musicology are great allies, because the interrogation and audience research methods (Roose, Lievens, & Waege, 2007) generally practiced, are particularly well-suited for both research fields and entail the same methodological considerations when for instance dealing with demographic information by means of internet questionnaires (Gosling et al., 2004) or survey evaluation tests (Presser et al., 2004). Within the fields, a great deal of attention is given to musical preferences (Litle & Zuckerman, 1986; Pearson & Dollinger, 2004), how these correlate to personality traits (Rentfrow & Gosling, 2003; Zweigenhaft, 2008) and how music preferences steer interpersonal relations (Rentfrow & Gosling, 2006) and self-perception (Ross & Shulman, 1973). The relation between cultural ‘habitus’ and musical preferences then seeks to explain such concepts as cultural omnivorousness and highbrow taste patterns (Van Eijck & Lievens, 2008), horizontal and vertical boundaries of cultural hierarchies (Berghman & Van Eijck, 2009), the impact of mediation (Hennion, 2003), participation levels (Lievens, Waege & Siongers, 2015), evidence-based and data-driven genre classifications and taxonomies (Pachet & Cazaly, 2000), as well as the attitudes, dispositions and social homogeneity of audiences (Roose, 2008). As musical identities (Hargreaves, Miell & MacDonald, 2002) are attributed more or less equal importance as categorical conventions such as musical styles and genres (Moore, 2001), it is logical that identification with certain types of music in adolescence (North, Hargreaves & O’Neill, 2000), the inverse thereof – e.g. symbolic exclusion of certain music styles (Bryson, 1997), group-based identities of music listeners (PrietoRodríguez & Fernández-Blanco, 2000) and the formative properties of music for personality, are also a point of interest, both to this research field, and this thesis. With research focusing on the importance of music in connection to personality traits, the fields of cultural sociology and music psychology are bridged. In reference to personality assessment – and for the purpose of this thesis, the fundamental principles of the “Big Five” (McAdams & Pals, 2006) personality correlates and demographic variables (González Gutiérrez et al., 2005) were investigated, evaluated (McAdams, 1992) and critically applied (Musek, 2007). Furthermore, opportunities were investigated to operate with other domains (Herzberg & Brähler, 2006) and abbreviated measures of the Big-Five personality dimensions (Gosling, Rentfrow & Swann, 2003). Yet, within the context of this thesis, music psychology has most merits with more fundamental considerations like the fundamental capacities for music (Jackendoff & Lerdahl, 2006), the analysis of music perception (Cross, 1998) and the mirror neurons that underpin the self-perception (Bem & McConnell, 1970) and interpersonal relations
28
(Gallese, Eagle & Migone, 2007) of social cognition and intentional attunement (Gallese, 2006) when dealing with musical stimuli. Moreover, the way in which developmental origins of musicality (Trehub, 2003) shape a musical language that functions as an entrainment and coalition signaling system (Hagen & Bryant, 2003) that forms the basis for human-system interfaces (Gill, 2007), is a key element in this discourse. Also, brain activation during music interaction (Seung et al., 2005), the audio–motor coordination (Baumann et al., 2007), and the audio–motor interactions in music perception and production (Zatorre, Chen & Penhune, 2007) that form the basis of stimulus–response (in)compatibilities (Dassonville et al., 2001), when dealing with musical mediation (Born, 2005), be it in the natural or digital world, are under scrutiny. Within that scope, music psychology is also involved with how digital devices can capture the deep bodily origins of the subjective and expressive control (De Preester, 2007) and emotional meaning in music of expressive body movement (Boone & Cunningham, 2001). Just like the innate propensities for speech, music performance (Carlson, et al., 1989) musicality (Seitz, 2005) and sonic communication (Feld, 1984, Dahlstedt & Nordahl, 2001) can be harnessed in music interaction design (Wilkie, Holland & Mulholland, 2010) by means of body movement. As music is believed to be a link between cognition and emotion (Krumhansl, 2002), emotional responses to sounds (Tajadura-Jiménez et al., 2010) can be interfaced through motion (Isbister, 2011), which in turn – and through mediation – can resonate in cyberspace (Tanzi, 2005). Thus, the body can convey non-verbal emotional (Boehner et al., 2007) and social interaction t(hr)o(ugh) multimedia systems (Camurri, 2009), and full-body motion can foster the expressive control of music and media (Castellano et al., 2007); this is what is roughly conveyed with the concept of embodiment.
2.1.2 The body as a mediator Derived from the phenomenology set forth by Husserl and further explored by MerleauPonty (Carman, 2000), the body is seen as the natural go-between for musical experiences and the physical environment. Opposite to the classical Cartesian divide, the body is thought to shape the mind (Gallagher, 2005), and from a psychometric standpoint (Longo et al., 2008), evidence for a neural basis for both experiences and social cognition can be found in the embodied experience (Gallese, 2005 & 2007). When the relation between music and the concept of embodiment is discussed in science (e.g. dimensions of body image and schema), this traditionally happened in a medical context (Gallagher, 2001). For instance, the embodied foundations that underpin the therapeutic aspects of musical experiences (Van der Schyff, 2013) or dance/movement therapy (Pylvänäinen, 2003), the benefits thereof for patients with autism spectrum
29
disorder (De Bruyn, Moelants & Leman, 2012) and the disembodied mental states of melancholia and schizophrenia (Fuchs, 2005) are debated among others. Embodiment as such serves as a vantage point for consciousness of the body and body image (De Preester & Tsakiris, 2009), the relation between that body and the technology used to represent it (De Preester, 2011), the kinaesthetic illusion of perception and action through technology (Kammers, Van der Ham & Dijkerman, 2006), the perception of one’s proper action in relation to the calibration of the self (Knoblich, 2002, Lackner & Di Zio, 2000), and the recognition of one’s own actions and the identification of proper performance (Repp & Knoblich, 2004). Embodiment is thus engaged with physical versus representational properties of animation (Hilti & Brugger, 2010) and how imitation of complex movements of the human body need to be represented (Schwoebel, Buxbaum & Branch Coslett, 2004). From this point on, it is clear that embodiment and human AR/VR-representations are akin in nature and closely linked to one another. Specifically within the scope of augmented and virtual reality environments and devices, there is a need to implement the things the body instinctively knows and is accustomed to, using embodied metaphors in hybrid physical digital environments (Antle, Corness, & Droumeva, 2009). As such, complex states of mind like simulation of injury, identity (Wainwright & Turner, 2004), intimacy (Fels, 2000 & 2004), entelechy (Seifert & Kim, 2007) and bodily selfconsciousness (Lenggenhager et al., 2007) have found their way into AI&VRE and embodiment research. Embodiment, multimodality and composition are convergent themes across HCI and education research using mixed-reality learning environments (Birchfield et al., 2008), so it would be only natural to assume that it would also move into the realm of artistic practice. Given this broader context, embodied music cognition research then has focused on the shift from traditional music perception to music (inter)action, as the act of music making (i.e. the musical gesture) is considered equally important as the produced and/or perceived sonic content itself. Technological mediators can extend the range and realm of the human body, like prosthetics increasing the operational range of the human body both in the natural or virtual world(s). In the paradigm of embodied music cognition, it is the human body that is seen as a natural mediator between musical experiences and the environment, rather than the sensor technologies that facilitate it. One of the goals of the mediation technology is to achieve the illusion of non-mediation, in which the subject views the mediation technology – traditionally the musical instrument, but of late increasingly sensor and motion capture technologies – as being part of the proper body. The study and exploration of an embodied way of interacting with musical content and the flexible relationship between musical experience and physical environment that these devices convey, have become an important topic in music research over the last decades. The use of new technologies means that musically relevant action strategies are not fixed, allowing mediation devices to function as a natural extension of the human body. As a
30
consequence, the guidelines for proper tool/instrument operation (e.g. instrument performance) become heavily subject-dependent and limited only by the operational range and constraints of the human body (Godoy, 2004) and the capturing technologies. The embodied music cognition paradigm (Leman, 2008) in turn implies that the human body is capable of conveying and deciphering musically encrypted information from one person to the next (Reybrouck, 2005 & 2006), and this corporeal encoding, decoding and multisensory integration (Maravita, Spence & Driver, 2003) of musical information, is ideally suited to be used in the context of algorithmic sound generation (Kim & Seifert, 2006). The logical bridge between the acoustic and the digital realm, is in that sense the human body, and musical organology as such needs to start taking the human sensorimotor system into account within that complex relationship (Magnusson & Mendieta, 2007). Anyhow, mediation technologies (e.g. sensors or motion capture devices) could theoretically allow the body to function as an ‘augmented’ mediator, without having to re-direct the energy of the human motor system onto an external instrument that only allows a fixed mode of musical interaction (Miranda & Wanderley, 2006). As such, employing gesture, schemata and stylistic-cognitive types, the body could be set to/in music (López Cano, 2003), provided the tools (Maravita & Iriki, 2004) would be available for whole body interaction (England, 2011), the embodied musical instruments becoming cognitive extensions (Magnusson, 2009) much in the same way as would be the case for augmented dancers (Legrand & Ravn, 2009). This human-machine interaction paradigm (Godoy & Leman, 2010) is made possible by the idea that action and perception are tightly coupled and embedded within sonic action- reaction-cycles (Leman, 2008b) that are linked with a repertoire of gestures and actions (referred to as human action-oriented ontology and affordances; Gibson, 1977). Closing the semantic gap, i.e. the discrepancy between the users’ high level cognitive representations and the low level measurable attributes in music (Leman, 2008b), can only be achieved by finding the appropriate correlations between corporeal articulations (gestures) and moving sonic forms (music). The musical communication process that results from these technologies ideally occurs between multiple participants and within an interactive environment in which social and musical interactions can take place. Within a supportive, engaging and communicative environment, which is not merely a catalyst for communication, an interactive augmented reality environment can take on the role of social agent (Leman, 2008). The independent and interactive effects of embodied-agent appearance and behavior on self-report, cognitive, and behavioral markers of co-presence in immersive virtual environments. (Bailenson et al., 2005) Accordingly, together with its technological mediator, the (human) body can be considered the instrument that expands the musical experience together with the technologically extended environment. It is an aspiration of both the scientific community and the cultural-creative sector to allow the embodied mind to express its musical intentions through an augmented environment for interactive sound creation. The goal is
31
to build an embodied music controller that is unhindered and unbound by the fixed interaction principles that are often prescribed by traditional interfaces. Theoretically, this opens up a whole range of possibilities for new and meaningful musical interactions, as well as opportunities for social interactions. This is because the participants of the social and musical communication process are no longer tied to stationary control or interface devices. However, this also raises a range of new questions (preferably not answered solely by its designers), as the technologies involve the prospective technology-users in decision-making processes such as mapping, sonification and social interaction strategies. Issues like expressiveness in the traditional musicians’ body movement (Dahl & Friberg, 2007), enactment in the listening body (Peters, 2010), physicality and feedback (Bahn, Hahn, & Trueman, 2001) and musical attunement (FinkJensen, 2007) are but of a few concepts embodied electronic music production would struggle with. A great deal of studies have dealt with auditory encoding of movement (Phillips-Silver & Trainor, 2007), the relation between action and sound in music-related body movement (Jensenius, 2007) and the neurobiological foundations for the instrumentalized body (Paillard, 1994). But it is the implementation of musical expressiveness which poses the biggest challenge in digital musical instrument design (Arfib et al. 2005). What constitutes the expressive gesture for interactive multimedia (Leman & Camurri, 2004), what design principles may be employed for expressive tangible interaction (Ross & Keyson, 2005), how can expressivity become part of a mapping strategy for physical interfaces (Merrill & Paradiso, 2005) or what mapping metaphors (Fels, Gadd & Mulder, 2002) need to be applied for that purpose, and ultimately, how convincing are the expressions of machines (Bartneck, 2001) and computers in support of musical expression (Bryan-Kinns & Mary, 2004), or can can they eventually become? Studies on simple musical gestures (Mion & D’Incà, 2006), on interface expressivity by means of player-based studies (Poepel, 2005) and on specific interfaces that promise to foster musical expression (Poupyrev et al., 2001, Camurri, Canepa & Volpe, 2007) are relatively common, yet the quality of the ‘E’ in NIME (Dobrian & Koppelman, 2006), to date remains a topic of controversy. Expressivity and the conveying of emotion aside, the question remains on what elements of body movement are to be considered meaningful gesture in dance or gesture in music, and which ones are ancillary. The “living gesture” has a momentary signification (Halton, 2004) and musical gestures comprise a combination of sound, movement and meaning (Godøy & Leman, 2009). Playing non-haptic instruments entails the software-based analysis of gesture for music (Nevile, Driessen & Schloss, 2003) and the recognition of “imagined” movements (Schwoebel, Boronat & Branch Coslett, 2002) or the mimicry of sound-producing gestures (Godøy, Haga & Jensenius, 2006). The basic spatiotemporal reference frames for repetitive dance/music patterns (Leman & Naveda, 2010) are what will eventually support interactive engagement in EMMT systems, and that necessitates the definition of what constitutes the meaningful gesture in
32
embodied musical interaction (Paine, 2004). As such, the gesture, whether it be executed in dance (Poesio, 2002) or is employed to exert gestural control of music (Wanderley, 2001) needs to be defined in a gesture description interchange format (Jensenius, Kvifte, & Godøy, 2006) of what the symbolic representation of the body in AI&VRE environments conveys in terms of meaning. This understanding of gesture is pivotal in understanding the relation between the resourceful action (Dourish, 1997), how it is to be understood and, as such, ideated or represented, reflected upon, evaluated in creative tasks such as composition (Coughlan & Johnson, 2006), and – in the case of EMMTs – correctly/acceptably mapped to sound. The gesture is to be understood as a symbolic action, bearing a semiotic signification in the composition of meaningful music interaction (Hamman, 1999). How these bodily representations need to be translated into an aesthetic perspective (Vickers, 2004), whether the body is to be understood as the sum of its online and offline representations (Tsakiris & Fotopoulou, 2008) and what elements in the distinct representations of the human body (Schwoebel & Coslett, 2005) are to be considered meaningful, is subject to much academic contention and debate at present. Yet, they are fundamental in the construction of apt mapping strategies, that are primordial in eventually allowing meaningful and realistic the full-body mediation of embodied music interaction. Additionally, there are the more extraneous factors of embodied interaction with mediation technologies and and AI&VRE-operation to consider, namely the components, functions, and modifiability of dynamic spatial cognition (Sekiyama, 2006) or how the mental motor imagery and the proprioception of body schema (Shenton, Schwoebel & Coslett, 2004) adapts the perception of self through tool use in timing-based models (Nabeshima, Lungarella & Kuniyoshi, 2005). The similarities between imitation and representation of action in reference to the dynamic body schema (Buxbaum, Giovannetti & Libon, 2000) are often studied using fMRIs (Chaminade, Meltzoff & Decety, 2005), and the human perception from the inside out (Holmes & Spence, 2006) heavily relies on the action-oriented representation of our peripersonal and extrapersonal space (Higuchi, Imanaka & Patla, 2006). That peripersonal space sets the boundaries of the proper body schema (Cardinali, Brozzoli & Farne, 2009) and delimits the awareness of personal space in collaborative music (Fencott & Bryan-Kinns, 2010). It may well be assumed that these are the same boundaries that beset the auditory peripersonal space in humans (Farnè & Làdavas, 2002) in multimodal AI&VRE interaction. A last noteworthy – and somewhat more promising – aspect of embodiment in multimodal virtual interaction, is that there appears to be evidence for the fact that there is a form of cross-modal congruency in humans, or cross- and multisensory contributions in the 3D representation (Spence et al., 2004). There is evidence that embodied interfaces for musical interaction allow for cross-modal experiences (Chapados & Levitin, 2008) and cross-sensory perception of emotions (Vines et al., 2006) in musical performances. Through the interaction with multisensory musical entertainment systems (Zhou et al.,
33
2004), tool-use may well change the multimodal spatial interactions between vision, aural cues and touch (Maravita et al., 2002), triggering neuronal mechanisms and perceptual functions of multisensory interactions in auditory cortex (Musacchia & Schroeder, 2009), which enables the convergence of multifarious perception (Driver & Spence, 2000), and multisensory integration (Spence & Squire, 2003) that could be described as a ‘synesthetic’ experience, or as Vines describes it, “music to the eyes” (Vines et al., 2011).
2.2 State of the art on new interfaces for musical expression 2.2.1 Art Technique Craftsmanship Technology Skill In Latin, the definition of the word for art (i.e. “ars”) comprises meanings that range from ‘art production’, over ‘skill’ and ‘technical prowess’, to ‘craftsmanship’ and the ‘ability to wield technology’. That definition resonates very well with modern-day instrument building. Technical advance has always been one of the important motors that drives the evolution of culture and cultural artifacts (Pacey, 1983), and music is anything but an exception to this; musical aptitude tends to inspire/require innovation and technological advances in turn instigate musical inspiration. New and innovative technologies have traditionally eagerly been embraced by musicians and music-related industries (Théberge, 1997), as technological improvements allow for the evolution in playing styles, and ultimately, musical genres. Arguably, the compositions that Beethoven, Chopin or Liszt left behind could not have originated on keyboards used at the time of Scarlatti, Bach or even Mozart. Similarly, the respective innovations made by Leo Fender, Lester William Polsfuss, Edward Van Halen and Steve Vai – most of them accomplished musicians themselves and each building on the accomplishments of the previous – revolutionized the instrument we have come to know as the electric guitar in such a way that it would be unrecognizable to and completely unplayable by the likes of Jean Baptiste ‘Django’ Reinhardt, only a (few) decade(s) earlier. The pace, scale and manner in which this is happening today is historically unprecedented, and the identification and transposition of authentic instruments into new musical practices and technologies (Impett, 1998) is becoming increasingly complex. There is a need for a new coherent terminology and model of instrument description and design (Kvifte & Jensenius, 2006), as the gradual appropriation of sensor-based technologies in the technological repertoire and extended electrophonic musical organology has instigated novel ways of interacting with music. As such, the last few decades have seen an exponential rise in the number of new interfaces that support unconventional types of interaction (Lähdeoja et al., 2010), and more importantly within 34
the scope of this thesis, embodied music cognition has been proposed as a research paradigm for gestural music technologies (Leman, 2008).
2.2.2 Computer Music There is a long-standing relation between computers, sound generation and music making (Pope, 1996) and real-time algorithmic composition had long been a phenomenon (Grisey, 1987). Yet, when additional processing power turned the compositional mechanisms of computational sonification and composition into a reality (Dean et al., 2006) around the turn of the twenty-first century, this former niche activity started playing in the big leagues. Regular off-the-shelf laptops combined with digital evolutions (Miranda & Al Biles, 2007) and the development of computer languages such as SuperCollider (McCartney, 2002) became sufficiently mature to see computers come into the picture as a serious means of music making (Shackel, 2000) alongside and equivalent traditional music instruments. Computing for musicians (Husbands et al., 2007) evolved into computer-mediated music production (Duignan, Noble & Biddle, 2010) and live humancomputer music-making (Stowell et al., 2009) and laptop performances (Reddell, 2003; Collins, 2003); even verifiable laptop orchestras (Trueman, 2007; Smallwood et al., 2008) came into existence. Experimental music composition evolved from technical to technological (Hamman, 2000; Hamman, 2002), and in the same time-frame, the internet which had been anticipated to play some role of importance in algorhythmic sound generation (Duckworth, 1999) came into play as a means to communicate network-based computing information (Vickers & Alty, 2002b) and for actually creating distributed sonic works (Traub, 2005). And this in turn spurred on a whole new range of cyber artists to explore the possible connections with ICT art and technology (Jacobson, 1992) as well as a new breed of new luthiers (Jorda, 2005; Bongers, 2007) to experiment with electronic musical instruments.
2.2.3 Tactile interfaces, gestural controllers and haptic feedback The advances in computer technology were accompanied by the advent of verifiable cornucopia of new technologies that would enable non-traditional interaction with both the computers themselves and the music generated by it. This constitutes the convergence of a range of alternate controllers and musical interfaces purposed for interactive entertainment (Blaine, 2005) such as non-command-based user interfaces (Nielsen, 1993), musical auditory interfaces (Leplâtre & Brewster, 1998), force-feedback hand controllers for musical interaction (Sinclair, 2007), optical tracking devices for music and dance performance (Paradiso & Sparacino, 1997), sensor-based instruments for musical performance practice (Tanaka, 2000), non-speech sound-enabling user interfaces (Absar 35
& Guastavino, 2008), sensing systems (Bellotti et al., 2002), wireless sensor interfaces and gesture-followers for music pedagogy (Bevilacqua et al., 2007), 3D-motion capture devices to enable motion analysis and music mapping (Bevilacqua, Ridenour & Cuccia, 2002), vibrotactile feedback devices (Birnbaum & Wanderley, 2007), tactual displays that grant access to the sound properties in electronic musical instruments (Bongers, 1998), tangible (Fishkin, 2004) and malleable interfaces (Kiefer, 2010a), and even wearable technologies to allow for musical engagement and sonic exploration (Schroeder & Rebelo, 2007). This plethora of input devices ranges from low-cost music controllers (Jensenius, Koehly & Wanderley, 2006) like sensors to be supplied to large groups of interactive participants at a reasonable cost (Feldmeier & Paradiso, 2007) over interfaces that constitute some form of hardware hacking for the purpose of electronic music creation and manipulation (Collins, 2009) of over-the-counter commercial sensor-enabled technologies – like Wiimotes (Kiefer, Collins, & Fitzpatrick, 2008a) to state-of-the-art custom-built wireless technologies (Kuyken et al., 2008) that can be used as musical controllers. This vast array of gesture-recognizing or feature-extracting devices are in turn used to enable various forms and degrees of interaction in multimedia art (Bongers, 2000). Some of the current trends in electronic music interfaces (Paradiso & O’Modhrain, 2003) comprise the use of electronic music controllers (Paradiso, 2003) for sensor-based music systems that enable large interactive surfaces (Paradiso et al., 2000) as well as the interaction with tangible hypermedia in augmented reality environments (Sinclair et al., 2002). Also, there is a focus on (wearable) wireless sensor network systems (Paradiso et al., 2010) that allow for the identification and facilitation of social interaction and that are capable of conveying dimensions of emotion in expressive musical performance (Vines et al., 2005). In some more elaborate cases – not unlike “SoundField”, combinations of stateof-the-art technologies are involved with component-based frameworks for the development of virtual musical instruments based on physical modeling (Tzevelekos et al., 2007), musical exoskeletons – i.e. motion capture suits (Collins et al., 2010) and sensor networks capable of registering social and group dynamics (Laibowitz et al., 2006). Overall, all given technologies are to some extent and in some shape or form involved with creating input devices that generate data streams from which relevant features can be extracted and to which suitable mapping techniques can be applied, granting somewhat intuitive control over sonic and/or visual output with the distinct purpose of facilitating generation, composition, editing and education of digital music (Kiefer, 2010b).
2.2.4 New interfaces for musical expression The technologies discussed have given rise to a profusion of tools and interfaces for musical expression, all working as digital instruments that work from the inside-out
36
(Harris, 2006), yet some having alternate focal points. A first group of interfaces comprises EMMTs that aim at staying true to the basic operation or stylistic conventions of traditional music instruments, like the real-time dynamics measuring violin performance “hyperbow” controller (Young, 2002), an expressive musical gesture measuring “Conductor’s Jacket” (Marrin & Picard, 1998), the virtual “SuperPolm MIDI violin” (Goto, 1999), adaptive and interactive orchestral conducting systems like “iSymphony” (Lee et al., 2006). Also, these interfaces go into the realm of musical automation like musical automata (Lewis,1999) and robotic musicianship (Weinberg & Driscoll, 2006), software-based tools for improvisation (Sayer, 2003; Sayer 2006) featuring musical style continuators (Pachet, 2003), continuous sound in tangible interfaces like “the spinotron” (Lemaitre et al., 2009) and motif design interfaces like “Auralisation” (Vickers & Alty, 2002). A second group of interfaces are basically a group of predominantly stand-alone sound producing tools, allowing low-threshold music interaction – mainly focused on children, fast learning curves, collaborative control and enjoyment. This group features such EMMTs as the “Beatbugs” (Aimi & Young, 2004), cube-shaped tangible interactive interfaces named “AudioCubes” (Schiettecatte & Vanderdonckt, 2008), wireless interactive performance “Eos Pods” (Bianciardi, Igoe, & Singer, 2003), free musicenabling leather instruments called “Children of Grainger” (Favilla & Cannon, 2006), “Playpens”, “Fireflies” and “Squeezables” (Weinberg, 2002), the wearable “Celeritas” (Torre et al., 2007), “Block jam” devices (Newton-Dunn, Nakano & Gibson, 2003) and sound toys for non-musicians like “Play!” (Robson, 2002). Also, there are collaborative controllers with interpersonal haptic feedback such as “OROBORO” (Carlile & Hartmann, 2005), the “KiOm” (Kapur et al., 2006) or collaborative networked music spaces for novices such as “JamSpace” (Gurevich, 2006). Examples can be given of more educational music gaming interfaces enabling social play (Isbister, 2010) or cooperative music lessons for novices, which include “CODES” (Miletto et al., 2009), “SoundBlocks” and “SoundScratch” (Harrison, 2005), and the evolution of the “Jam-O-Drum” multiplayer musical controller into the “Jam-O-Whirl” gaming interface “Jam-O-World” (Blaine & Forlines, 2002). Finally, there are a number of musical robots that support adaptive social composition systems like “ORB3” (Livingstone & Miranda, 2005), “RoboMusic” (Lund & Ottesen, 2008) and “RoboMusicKids” – a music education system with robotic building blocks (Nielsen, Brendsen & Jessen, 2008) that can be mentioned in this context. Moving away from the steep learning curve gaming interfaces, a third group of more complex and intricate network-based frameworks for collaborative development and performance avail themselves of similar technologies and comparable digital musical instruments (Malloch, Sinclair & Wanderley, 2008). These systems encompass platformbased networked music performance tools like “DIAMOUSES” (Alexandraki et al., 2008), real-time, distributed, voice-controlled collaborative instruments such as “Auracle”
37
(Ramakrishnan, Freeman & Varnik, 2004; Freeman et al., 2005) or interactive installations like the “Shannon portal” (Ciolfi et al., 2007). Also, physical sensing immersive environments like the “magic carpet” (Paradiso et al., 1997), multimedia composition and performance tools like “Soundium” (Müller et al., 2006), the interactive multimedia playground that is “Soundium 2” (Schubiger-Banz et al., 2003), interactive spatial sound and video artworks such as “Echology” (Deutscher et al., 2005), “interactive, stereographic, 3D audio, immersive virtual worlds like “Allobrain (Thompson et al., 2009), interaction-based integrated real and virtual world design such as “Roomware” (Streitz et al., 2001) and even pyroelectric sensor data sonification tools for interactive media events such as “Soundsense (Bower et al., 2005) deserve to be mentioned here. Then, as gestures and dance moves appear to be close allies, there are a lot of EMMTechnologies in which movement are dance are the focal point of the interaction and the technology (Ebenreuter, 2005). Building on the neural basis of dance movement (Brown, 2005), there are those interfaces that make observations and registration of dance (Calvo-Merino, 2004), that make automated annotation and retrieval of dance media objects (Ramadoss & Rajkumar, 2007), or that (re)interpret contemporary dance (Stevens & McKechnie, 2005). More actively using the dancing body as an input and resource, there are tools that enable composition for interactive dance (Toenjes, 2007), that create expressive dance environments (El-Nasr & Vasilakos, 2008), that allow performance of electronic music by dancing “dance jockey” (De Quay, Skogstad & Jensenius, 2011) or audience participation in a dance-club context, by means of collaborative creation of visuals (Kaiser, Ekblad & Broling, 2010); there even are augmented reality (“AR/DJ”) disk jockey systems (Stampfl, 2003, Berry et al., 2006). This overview could go on for several more pages, yet it suffices to refer to some surveys and anthologies that capture the essence of these technologies, describing a variety of elements and examples of the vast and ever-expanding profusion of computermusic interfaces (Pennycook, 1985). Most noteworthy are the digital musical instruments of the CIRMMT/McGill digital orchestra project (Pestova et al., 2009; Ferguson & Wanderley, 2010), the catalogues of the STEIM musical instrument designs (Ryan, 1991), the gestural sensor technologies used for the brain opera (Paradiso, 1999), the interactive virtual opera in the “virtualis project” (Bonardi & Rousseaux, 2002), and the digital and sonigraphical instrument anthologies of Blaine & Fels (2003) and Jordà (2003 & 2004), covering the ‘hall of fame’-interfaces ranging from the FMOL to the reacTable.
2.2.5 Immersive Environments in Artistic Practice Connecting a number of sensors to a computer or an interfacing platform (Bennett et al., 2007) does not constitute an EMMTs or AI&VREs as such; whether striving for musical
38
user interaction (Johnston, Candy & Edmonds, 2008) or virtual 3D instrument interaction (Mulder, Fels & Mase, 1999), as long as not even a vague purpose is defined, it is just one piece of expensive gear hooked up to other state-of-the-art bits and pieces. That is why most of the implementations discussed in the previous section generally have some form of objective, target audience or interaction focus. Beyond the evident relationship between embodiment and dance, augmented reality (Azuma, 1997) virtual worlds (Bainbridge, 2007) network systems for music and sonic art creation (Barbosa, 2003) are in existence. This new interaction paradigm made possible by the latter-day instruments makes for the fact that novel ways of interaction in musical multimedia performance can be explored (Bongers, 1999). Expressive performances can be modeled, analyzed and synthesized (Grindlay & Helmbold, 2006) with computerassisted content editing techniques for live multimedia manipulation (Arisona et al., 2008), and interactive performance with electronics-based virtual musical instruments (De Oliveira Rocha, 2008) as such has become an interactive performance practice with its own mannerisms, technological characteristics and issues (Goto, 2000). The psychological process (Juslin, 2003) at the basis of expressive audio synthesis, namely turning gestural performances into sounds (D’Inca & Mion, 2006), including what structural information needs to be conveyed (Vines et al., 2004) and what emotional coloring (Juslin & Sloboda, 2010) is latent in computer-controlled music performance (Bresin & Friberg, 2000) pivots on a system’s ability to recognize and interpret perceived intention (Corness, 2008). Aside from individual differences in music performance (Sloboda, 2000), musical prowess (Lotze et al., 2003) and the choice of media and task and its effects on user performance (Mennecke, Valacich & Wheeler, 2000), articulation rules (Bresin, 2001), gesture control and sound spatialization are the elements that make up the musical performance (Marshall, Malloch & Wanderley, 2009) and give it a sense of “liveness” (Croft, 2007). As such, immersive environments in which augmented media performances with interactive technologies (Wong, 2009) live music performance (Kaltenbrunner, Geiger & Jordà, 2004) mediated expression and audience participation: (Freeman, 2008) occurs, become performing spaces (Rebelo, 2003). As such, the rise and tentative integration of interaction with computer-based (Lewis, 1999) or computer-mediated alternate ‘realities’ (Barbatsis et al, 1999) have become a mainstay in artistic research and a staple of experimental art performance. Embodied interaction increasingly takes place in hybrid digital environments that combine physical, augmented and virtual reality elements. Such an immersive environment draw on the ontological metaphor that music is movement, and the bodily knowledge (Antle et al., 2009) that underpins how we effect movement in a musical manner is thus functionally employed for the actual performance of music. Consequently, the spatial production of sound and performance of music (Rebelo, 2003) through the use of human unencumbered movement (Paine, 2002) can occur through the exploitation of the functional similarities
39
between the spatial representations in real and virtual environments alike (Williams, 2007). The previously stated central concepts that form the basis for the functionality of these hybrid digital environments, are predominantly the human capacity for embodiment, but also, the sense of immersion. The clearly stated goal of the mediation technology to achieve the illusion of non-mediation, and the interacting subject to view the mediation technology as a part of the proper body, is taken one step further in the context of AI&VREs, as the (human) body itself can be considered a part of the instrument that is the technologically enhanced environment. These concepts allow the human body to become the instrument itself within a system that is capable of algorithmic sound generation (Kim & Seifert, 2006) and in artistic human-computer interaction (Seifert, 2008). However, this occurs only when a human is in a way absorbed by this interaction. This immersion in the virtual environment allows a person to experience the environment through interaction with it (Lipscomb, 2004) and enables an immersive performance (Sawchuck, 2003). So, as the embodiment paradigm allows collaborative (Turk, 2008) and corporeal exploration of musical creativity in a collaborative virtual environment (Barrass & Barrass, 2006), it enables the investigation of how new, transformed and augmented modes of social interaction (Bailenson et al., 2004/2005) and collaboration (Billinghurst & Kato, 2002) occur within virtual environments. It necessitates the investigation of how to deal with human factors (Stanney, 2003), social inhibitions (Hoyt et al., 2002) and interpersonal limitations that occur through the intrusion of a peripersonal environment (Diniz et al, 2012) and interpersonal distance (Bailenson et al., 2001 & 2003) in technologically informed performance environments (Schroeder, 2006) and augmented reality systems (Livingston, 2005). And finally, it allows to take into account the fundamental and technological limitations (Kyriakakis, 1998) given the complexity of non-hierarchical interactive music environments (Lewis, 2000) that use the body as an instrument, the ecological considerations of real and unreal experiences of auditory-visual virtual environments (Larsson, 2001) as well how to apply ergonomics in user-informed virtual environments (Aarts, 2005) applications (Wilson, 1999). Summarizing, immersion in distributed performances (Sawchuk et al., 2003; Chew et al., 2004) and interactive public sound art installations (Birchfield et al., 2006; Bandt, 2006) have the potential to blur the boundaries of the eye, ear, space and time, and remap multimodal musical interaction spaces by tool use (Berti & Frassinetti, 2000; Bongers & van der Veer, 2007; Emmerson, 1998). Virtual, augmented, immersive and mixed reality environments (Stanney, 2002) form the new paradigm for interacting with computers, as they are able to interface between the real and virtual worlds (Mackay, 1998). The designs, implementations, and application domains are quite all-encompassing, as they range from multisensory (Calvert, Spence & Stein, 2004) interaction spaces for artistic purposes (Frisoli & Camurri, 2006), dance (Bevilacqua et al., 2007), music and multimedia (Camurri & Ferrentino, 1999), to virtual work places of the future (Wilson &
40
D’Cruz, 2006), methodological tools for social psychology (Blascovich et al., 2002), therapeutic goals (Ellis & van Leeuwen, 2009) and intelligent learning environments (Cook, 1998). A lot of potential, but also, a lot of additional issues stem from the fact that AI&VREs assume a form of interactivity, collaborative control, group interaction and social computing (Musser, Wedman & Laffey, 2003) and allow for cooperative creative exploration (Turk et al., 2009) and control. Collaborative group playing (Weinberg, 2003) in virtual environments has its foundations in a digitally transformed social interaction, in which a decoupling occurs between the representation of behavior and form (Bailenson et al., 2004). The (virtual) object-focused interaction in collaborative virtual environments (Hindmarsh et al., 2000) makes for a shared augmented reality (Billinghurst & Kato, 2002) – or rather – multiple realities (Fraser et al., 2000) and joint musical experiences (Blaine & Fels, 2003). The externalization of musical creativity in virtual or augmented environments (Barrass & Barrass, 2006) and music-making as a collaborative activity (Martin, 2006) assumes cooperative multi-user gestural interactions (Morris et al., 2006). Modeling and mapping those onto multi-user instruments (Jordà, 2005) and into perceptual audio features (Mion et al., 2010) so that they enhance user interaction and express both the collective and individual behavior (McPhail, 2006) merging virtual and real for face-to-face collaboration (Tan et al., 2000) is daunting enough in and of itself. In order to prevent confusion, it also entails designing constraints for those performing with digital musical systems (Magnusson, 2010) to safeguard the transparency in the actionreaction couplings. Furthermore, that is not taking the complex concept of interactivity into the equation, i.e. what it means to interact with – rather than responsive – reactive technological agents (Kaptelinin & Nardi, 2006; Kwastek, 2008; Salminen, 2009), and coping with how humans react to agent-like machines, and what users retain from such an interaction process (Quiring, 2009). For a portion of this thesis (i.e. chapters 8 and 9), the experimental frame of reference also ventures into the realm of interactive sonification (Hermann, Hunt, & Neuhoff, 2011; Hermann & Hunt, 2005) and spectromorphology (Smalley, 1997). This entails the shaping of multisensory actions through tool manipulation (Farnè, Iriki & Làdavas, 2005) into shape-form acousmatic images (Smalley, 2007). This implies crossmodal space, crossmodal attention (Spence & Driver, 2004), through a timbral space-mapping (Seago, Holland & Mulholland, 2005). As such, music-based interactive sonification (Diniz, Demey & Leman, 2010; Weinberg & Thatcher, 2006) by means of spatialized tool-use (Farnè, Serino & Làdavas, 2007) enables the multisensory representations of space (Legrand et al., 2007), which by means of data sonification and sound visualization (Kaper, Wiebel & Tipei, 1999), yields aural and semi-tangible access into geographical data patterns (Zhao, Plaisant & Shneiderman, 2005). The resulting sounding object (Rocchesso & Fontana, 2003), can be analyzed by criteria based on Denis Smalley’s timbral theories (Hirst et al., 2002); the interpretation is given musical signification 41
through acousmatic listening (Atkinson, 2007). This exemplifies the importance of interaction and direct relation between manipulation (Fernström & McNamara, 2005) for interactive sonification (Hunt & Hermann, 2004) to ensure real-world correspondences (Barrass, Whitelaw & Bailes, 2006) between the applied gestural control and the resulting sound (Couturier, 2006). Examples of usability research on the nexus of EMMTs, AI&VREs and interactive sonification are not widespread, although the first examples of human-computer interaction design examples of interactive sonification instruments or agents date back a solid decade (Fernström & Brazil, 2004; Fernström, Brazil & Bannon, 2005). Concerning the use of AI&VREs as embodied music mediation technologies, the penultimate concept of mapping, which entails feature extraction of human movement qualities for sound analysis or synthesis (Pedersen & Alsop, 2012), is a particularly complex one. Music can as such be generated from motion by means of synthesis engines (Hunt & Wanderley, 2002) that turn performers’ motion parameters into sound for interactive performances through transdomain mappings (Ng, 2004). How listeners critically assess their motion in reference to the applied musical parameters (Eitan & Granot, 2006) determines the congruency between gesture and sound, as well as the quality of the mapping (Bencina, Wilde, & Langley, 2008) and its suitability to control parameters like musical instruments (Levitin, McAdams & Adams, 2002; Goudeseune, 2002). As far as mappings for cooperative music environments are concerned, there is a substantial difference between interfaces for ‘musical activities’ and interfaces for actual ‘musicians’ (Miletto et al., 2009). A final concept that can greatly affect creativity in relation to musically innovative tools (D’Arcangelo, 2001 & 2002), is the context in which interaction occurs. Accounting for the interaction space or context within presence and interaction design research, requires specific methods and tools (Kaptelinin, Nardi & Macaulay, 1999). Tactical usercentered and participatory design is already being applied in order to understand both the context of design (Svanæs & Gulliksen, 2008) and the ‘community computing’ contexts (Merkel et al., 2004). Also, specific methodologies and toolkits are in place to assess context-specific usability guidelines for interface design (Henninger, 2000) and to supporting the rapid prototyping of context-aware applications (Dey, Abowd & Salber, 2001). To date, great questions arise from how musical context can be aptly taken into account in parameter mapping research for interfaces for musical expression (Loughlin, 2009), as it should – eventually – be an integral part in the development of mobile (Driver & Clarke, 2008) context-aware computing methods (Moran & Dourish, 2001; Dourish, 2001).
42
2.3 User-centered procedures: an overview of applicable HCI-methods (Usability, UI, UE and UX) 2.3.1 HCI and usability: standards, methods and metrics As far as the concept of usability goes, it is that part of human computer interaction (Sharples, 1994) involved with the quality of use (Bevan, 1995a), the quality in use (Bevan, 1999) and meeting user needs through means of usability engineering (Nielsen, 1994b) and by applying usability inspection methods (Nielsen, 1994c). Concerning roots and trends (Hartson, 1998), the research field is concerned with how interfaces work (Nielsen, 1994a) measuring (Bevan, 1995) and standardizing the quality of use and integrating operational environment considerations into evaluation methods (Aborg et al., 2003). In the wake of its conception, international standards for HCI and usability (Bevan, 2001a) were developed to ensure quality in use for all (Bevan, 2001b), and a range of universally applicable usability inspection methods (Mack & Nielsen, 1994) and usability measurement tools (Bevan, 2006) were developed, to provide strategies and guidance concerning usability and accessibility (Bevan, Petrie, & Claridge, 2007). Akin to a sort of Fitts’ law for the design of as well as a research tool for human-computer interaction (MacKenzie, 1992), heuristic evaluation (Nielsen & Molich, 1990; Hvannberg, Law & Lérusdòttir, 2007), GOMS interface analysis techniques (John & Kieras, 1996), MEMS (Kamalian, Takagi & Agogino, 2004), think-aloud protocols, questionnaires and interviews (Donker & Markopoulos, 2002), the cognitive walkthrough method (Wharton et al., 1994) – as well as more streamlined versions thereof (Spencer, 2000) – focus groups (Langford & McDonagh, 2004), narrative analysis (Franzosi, 1998), discourse analytic method (Talja, 1999 ; Yagi, 1999) discourse markers (Kyratzis & Ervin-Tripp, 1999), verbal protocol methods (Ryan & Haslegrave, 2007), the application of universal design resources (Law et al., 2008) and other elicitation methods (Strickler, 1999) have been either applied or developed to locate and report existing usability problems. Moreover, the research field has a long-standing relationship with both activity theory and ergonomics. From its basic concepts and applications (Kaptelinin, Kuutti & Bannon, 1995), the study of ergonomics as the definition of professional needs and opportunities (Wilson, 1997), in human-computer interaction (Nardi, 1996), the research field of activity theory involved with usability research has branched out into the study of both physical and virtual tools, design, groupware (Fjeld et al., 2002) and even music education (Welch, 2007) of late. The usability research field has meanwhile matured (Jokela et al., 2006) into a set of integrated engineering support tools (Andre et al., 2001). Usability testing (Wichansky, 2000) currently comprises as set of evolving methods, guidelines and patterns (Henninger, 2001), revolving around user-centeredness and empirical assessments (Jokela, 2004). The 43
current practice of measuring usability (Hornbæk, 2006) generally entails the work of usability professionals (Gulliksen, Boivie & Göransson, 2006), that constantly scrutinize and refine the available methods (Hanington, 2003). As such, socio-cognitive engineering (Sharples et al., 2002) has become a human-centered “technomethodology” (Dourish & Button, 1998) capable of dealing with the uncertainties (Nunes, 2010) that arise from interacting with humans, and even those of specific needs groups – e.g. children (Markopoulos & Bekker, 2003).
2.3.2 User-centered approaches for artistic interfaces At present, usability-oriented and user-centered engineering have become widely accepted in industry and research (Mao, 2005). Developers have come to understand that the active involvement of users is indispensable to bridge the gap between users' intentions and actual design practice. This participatory design strategy (Kensing, 1998) is generally conducted alongside an iterative process that incorporates elements such as task-centered design, user-inquiries, prototyping, heuristic evaluation, user testing and evaluation (Woods, 1996). Also in game development, the appropriation of usability heuristics, evaluation of player enjoyment, characterization and measurement of user experiences and iterative design are also widely acknowledged (IJsselstein, 2008; Desurvire, 2009). Additionally, with the growing involvement with ICT-technologies, systematic musicology has also gradually come to adopt elements of usability research, applying HCI-methodology for evaluating musical controllers (Kiefer, Collins, & Fitzpatrick, 2008b), collaborative development and evaluation techniques for digital musical instrument development (Malloch, Sinclair & Wanderley, 2007), discourse analysis for expressive musical interfaces (Stowell, Plumbley & Bryan-Kinns, 2008), heuristic evaluation for virtual reality applications (Sutcliffe & Gault, 2004), and user interface evaluation for intuitive interaction with multimedia (Oh, 2004) The growing importance of usability engineering incentivized the fact that attention also be payed to experiential factors that aid or hinder interaction. One element to discuss in this context is that boredom matters (Anderson, 2004), boredom hinders immersion and effectively prevents a good player experience (Nacke & Lindley, 2008). Closely linked is the fact that cognitive complexity can be a seriously distorting factor in human-computer interaction (Rauterberg, 1996), and that “too many notes” or too high levels of complexity (Lewis, 2000) can result in error making (Fyans, Gurevich & Stapleton, 2009), detrimental to good interactions or experiences. Also, there appears to be a delicate balance between affordances and constraints in screen-based musical instruments (Magnusson, 2006). Aside from technological limitations of immersive audio systems (Kyriakakis, 1998), EMMT-user might experience similar gestural constraints as real
44
instrumental performers. (Mulder, 2000), and there are decisive constraints in electronic musical instruments (Gurevich, Stapleton & Marquez-Borbon, 2010). However, these constraints may be turned into creative resources for interaction design (Mose Biskjaer & Halskov, 2014), and taking a turn for the positive, usability tests focusing on enjoyment and fun in interaction might be of equal, if not greater importance (Blythe et al., 2004). This is a function of how pleasantness can be derived from the bodily experience (Rozendaal & Schifferstein, 2010) or the extrapolation of that experience into digital game enjoyment (IJsselsteijn et al., 2008) and/or the field of EMMTs. Anyhow, there is great importance in the usability of EMMTs and AI&VREs to be attributed to (the audience interpretation of) enjoyment levels (Glass & Stevens, 2005). More specifically, to the fact that shared fun is double fun in terms of player enjoyment (Gajadhar, de Kort & IJsselsteijn, 2008), as the social settings in which EMMTinteraction might occurs, is not altogether dissociate from the player enjoyment in games (Sweetser & Wyeth, 2005). Similarly to enjoyment, the ease of use (Vredenburg, 1999) is a major factor in the overall wellbeing of the perceived interaction with technologies. In crafting participation, both an ecology and experience is configured (Heath et al., 2002) that approximates the player experience of digital games (Van den Hoogen, IJsselsteijn & de Kort, 2008; Poels, de Kort & Ijsselsteijn, 2007). It is one of the great current challenges in the development of today’s ict-environments (De Marez & De Moor, 2007) that these human-centered models for interactive multimedia (Borchers & Mülhauser, 1997) try to capitalize on the optimal flow-like (Csikszentmihalyi & Hunter, 2003) experience in a socio-spatial context (De Kort & Ijsselsteijn, 2008), applying methods for evaluating gameplay experience (Nacke, Drachen & Göbel, 2010). The evaluation of accessibility, usability and user experience (Petrie & Bevan, 2009) have become integral parts of user experience evaluation methods (Vermeeren et al., 2010), and they are consequently being applied in EMMT and AI&VRE development, as they are particularly well-suited to capture performers’ intentions and steer the listeners’ experiences (Gabrielsson & Juslin, 1996) of auditory-visual virtual environments through ecological acoustics (Larsson, Västfjäll & Kleiner, 2001). The sense of immersion in virtual environments hinges on the quality of multisensory perception and the integration of visual and auditory information into musical experiences (Hodges, Hairston & Burdette, 2005), similar to the importance of musical scores in video gaming experiences (Lipscomb & Zehnder, 2004). Even though quantitative and qualitative approaches borrowed from HCI are increasingly being applied to the evaluation of musical controllers and realistic live human-computer music-making (Wanderley & Orio, 2002; Kiefer et al., 2008; Stowell et al. 2009), HCI methodologies are too date not always readily applicable to the multidisciplinary field of computer music, and more specifically, to technology-informed installation art (Höök, 2003). This is on the one hand due to the fact that the apportionment of different media has a direct and substantial effect on users' performance
45
(Mennecke, 2000) and perception (Vines et al., 2006). On the other, multi-media environments rarely offer genuine opportunity for quality of performance evaluation, since the emphasis is generally more on the experience of operating a system than it is on the execution of a task. As a result, usability engineering of virtual environments and virtual reality applications (Sutcliffe, 2004) does differ considerably from traditional HCI-usability techniques, because traditional usability principles do not consider characteristics unique to VE systems, such as the design of way-finding and navigational techniques, object selection and manipulation, as well as integration of visual, auditory and haptic system outputs. The first goal is to identify the criteria that drive effective VE system design (Stanney, 2003). There is a wide variety of different artistic purposes these applications may be used for, a whole range of different sensors, cameras and tangible interfaces that may be employed, many options for feature extraction and consequently many conceivable mapping strategies, and finally, vast opportunities for multi-mediated output, all of them subject to artistic intention and aesthetic predilection. As a consequence, usability metrics that focus on task completion and errors (e.g. efficiency, effectivity (time-related) tests, learnability (task success over time)) are all applicable only to a certain extent. Apparently, in relation to technology-informed art production, usability evaluation methods are not readily available, so it is necessary to establish criteria for evaluating and customizing usability evaluation methods (Hartson; 2000) to suit this purpose. Themes like embodiment, multi-modality, composition and HCI (and education) converge in mixed-reality environments (Birchfield, 2008). Not only the designing but also the evaluation has become the focus of research, whether the multimedia platforms focus on gaming (Jurgelionis, 2007) or on interaction with virtual musical instruments (Johnston, 2008). Increasingly attention is given to multi-user gestural interactions and cooperative gestures (Morris, 2006) and to the evaluation and incorporation of human factors in augmented reality systems through visual, auditory and tactile cues and tasks (Livingston, 2005). However, in the case of interfaces aimed at artistic and musical expression, it is not sufficient to measure task performance alone. There is a shift from functional efficiency to overall user experience as in interactive artworks (Deutscher, 2008). The participant 's experience is key in the development of workable and functioning systems, therefore, there is a genuine need for a broadly applicable method that provides ample opportunity for recording user-feedback, that reconciles user-related requirements and usabilitycentered evaluation methods in a use case-driven engineering process (Seffah, 2001). Within an under-developed meta-design environment, participants with a compulsion for artistic expression are enabled to set out design guidelines throughout an end-userfocused development process (Fischer, 2004) in which modifications and additions can constantly reevaluated, reiterated and improved. On the one hand, this procedure
46
safeguards the possibility of a structured contextual approach towards universal accessibility (Stary, 2000), on the other is very malleable to cater for specific groups (Markopoulos, 2003). And finally, this strategy enables the employment of social creativity for the definition of esthetic, interactive, social and functional requirements, to help define the majority of conceivable opportunities and problems early on. Therefore, in the scope of this thesis, the choice is made to evaluate quality of experience in controlled testbed-oriented living lab settings (De Moor et al., 2010), so that participant sensation of interactive artworks can be studied using laboratory-based methods (Deutscher, 2008), in context that feels ecological and through experiences that are perceived as genuine. These performance ecosystems (Waters, 2007) ideally create a living lab (Schuurman et al., 2010) setting in which ICT-innovation can be forstered irrespective of user typologies, allowing for ecological approaches to musical interaction and creation (Gurevich & Treviño, 2007), so that multisensory perception and integration can have ecological validity (De Gelder & Bertelson, 2003).
2.4 State of the Arts in the Digital Design Procedures 2.4.1 Open-endedness In the previous sections, the fact that paradigm of embodied music cognition assumes that everyone can make use of the same natural mediator, i.e. the human body was discussed. As a result, developers aim to employ gestures that bear an obvious connection to musical features and that are universally understandable, regardless of cultural background, and use them as the controls of an interactive music tool that ideally can be operated by most users. This creates the need for a novel multi-modal HCI (Kim, Youn & Hong, 2007) in which the evaluation of sensors as input devices for computer music interfaces (Marshall & Wanderley, 2006) can be measured and tested for user interface consistency across end-user applications (Satzinger & Olfman, 1998). Various frameworks are in existence for the evaluation of digital musical instruments (O’Modhrain, 2011), and methods like CASSM to test utility and conceptual fit (Blandford et al., 2008) or the KTH-rule system for musical performance as proposed by Friberg, Bresin & Sundberg (2006) already partially deal with the problem, but fall a bit short when establishing usability for interactive music applications that use embodied mediation technology (Deweppe, Lesaffre & Leman, 2009). There are many issues and challenges for the effective combination of hybrid usability methods in evaluating educational and artistic applications of ICT (Tselios, Avouris & Komis, 2008) and ongoing debates on what criteria to consider when evaluating usability evaluation 47
methods. (Hartson, Andre, & Williges, 2003). Evaluating and implementing strategies for embodied music interaction (Deweppe et al., 2010) requires the considerate borrowing of tools from HCI (Orio, Schnell & Wanderley, 2001) comparison of suitable techniques (Jeffries, Miller, Wharton, & Uyeda, 1991) and assessment of the criteria that enable evaluation of physical modelling schemes for music creation (Castagne & Cadoz, 2003). However, what constitutes musically meaningful interactions is not a fixed principle and may be considered to depend both on the corporeality of the individual user and their cultural background. Aside from that, personal sensibilities also need to be taken into account in evaluations of what makes sense in interactive art (Höök, Sengers, & Andersson, 2003) and there is the evaluator effect in usability evaluation methods (Hertzum & Jacobsen, 2001) to consider, even without taking into account user diversity and the difficulties in achieving effectiveness in design for varied user groups (Stary, 2001).
2.4.2 A user-inspired development strategy The drawback of the fact that these technologies are much more versatile than more traditional musical instruments, is that the high degree of variability regarding possible manipulation strategies makes the connection with human body movement considerably less straightforward. Since, independently of the interface that is used, mappings can be based on interaction patterns that are considered musically meaningful by users, the operation of these tools is at risk of being highly interpretable by each and any individual user. This highlights the need for prospective users to be systematically consulted to evaluate and assess their involvement with these tools, during the development and testing, to ultimately ensure optimal usage. Implementation, improvement and concerns about user-centered design are by no means new (Gasson, 1999), but at the turn of the century, user-centered design practice (Vredenburg et al., 2002) did evolve into a more fundamental requirement in system development (Keinonen, 2008). Instead of dictating end-users how to make sound use of a type of technology, user characterizations, user requirements and user activities were charted (e.g. the KESSU 2.2 model by Jokela (2008)) and developers genuinely started embracing diversity in user needs (Khalid, 2006) and considering users’ concerns as universal design resources (Law, Jaeger & McKay, 2010). Designing for became designing with through informant and interactive development methods (Scaife et al., 1997) and user needs analysis has become one of the main requirements in engineering (Lindgaard et al., 2006). Various forms of user-centeredness and associated methods (Iivari & Iivari, 2011) try to achieve optimal impact of user-oriented design in development (Veryzer & Borja de Mozota, 2005) and the current state of user-centered
48
design practice (Mao et al., 2005) is such that it seriously takes user modeling into account in human–computer interaction (Fischer, 2001). The need for user-involvement stems from the fact that new gesture-based technologies have a high degree of flexibility and adaptability, and therefore user-inspired implementation strategies aim at testing various aspects of tools and frameworks to elucidate what developments are viable, and what features are to be considered undesirable. As such, user-centered design has become one of the normative policies of technology development (Garrety & Badham, 2004) about which various manifestos have been written, no longer solely focusing on the involvement of end-users for product development (Fischer et al., 2004; Fischer, Nakakoji & Ye, 2009), but throughout the design process. As soon as the importance of user-driven and user-centered development had become apparent to the field of HCI and usability engineering (Nielsen, 1994b), userinvolvement started going beyond user-centered (Fischer, 2003) and evolved into participatory design (Asaro, 2000). Various quantitative and qualitative approaches came to fruition (Clemmensen, 2004) to deal with the issues and concerns of user-spired modeling (Kensing & Blomberg, 1998). End-users gradually were incorporated more early on in a focused collaboration process (Keikonen, Jääskö & Mattelmäki, 2008) revolving around creative practice-based interaction design (Barrass, 2008), and those user co-creators (Banks & Humphreys, 2008) as such became part of a culture of participation. (Fischer, 2009). As for the conceptual framework of this thesis applied in the “SoundField”-project, the implemented design strategy revolved around the following concepts: in terms of a collaborative and iterative design strategy, “SoundField” aimed at comparing and reconciling usability-centered and use case-driven requirements engineering processes (Seffah, Djouab & Antunes, 2001). To understand the productive and structural interactions with the digital tools that enabled the creative activities in a longitudinal perspective (Coughlan & Johnson, 2009) an optimal interaction with the user-pool was propagated in a scenario-based (Carrol, 1999) and usability-based parallel design strategy (Nielsen & Faber, 1996), in order to improve both the system itself and the resulting participatory artworks (Birringer, 2005).
2.5 The cultural-creative sector, adoption and literacy The exponential rise of new media at the turn of the century radically and fundamentally changed the outlook of the creative economy and the cultural sector (Emmerson, 2000). Various national (De Voldere et al., 2006) and international studies (e.g. the KEA- report, as well as various Unctad, PEW, and Unesco studies) investigate this changed reality and 49
the impact thereof on the cultural and creative economy. Pivotal in these studies are the diversification of the actors in the concentric model of the cultural industries (Throsby, 2008; Pratt, 2005) and the definition of challenges and opportunities for the creative economy in various regions (Nielsén, 2009). Simultaneously, a culture of control came into existence in the media space (Dourish, 1993) and the triple articulation –i.e. press, content, carrier, platform and even context – of media technologies (Courtois et al., 2012) have forced the cultural sector to rethink their business model, especially their communication and dissemination strategies (Vom Lehn, Heath, & Hindmarsh, 2005). Aside from digital art (Paul & Werner, 2003), a culture of participation through old and new media (Sonck & de Haan, 2012) pervades the creative industries at present, and concert venues, museums and galleries alike are required to exhibit interactivity and collaboration in order to stay relevant (Vom Lehn, Heath, & Hindmarsh, 2001). This gives rise to the creation of a digital culture (Gere, 2002) that beyond artistic experiences, assumes access, interactivity, and authenticity (Hand, 2008) to which we will come back in chapter 10. Another central trend to line of research (Tomasello, Lee & Baer, 2009), is the various trends and perspectives of new media adoption and acceptance that can be discerned (Dillon & Morris, 1996). This is because, as much as the creative industries need to keep up with technological innovations – like the EMMTs and AI&VREs that form the central theme of the thesis, it is the eventual audience that decides on whether to appropriate (Dourish, 2003) or domesticate (Haddon, 2006) the emerging interactive technologies (Gaye et al., 2006) or not (Satchell & Dourish, 2009), thus choosing to become part or abstain from a community that propagates cultural agency (Dourish, Anderson & Nafus, 2007) – i.e. humanity 2.0 (Fuller, 2012) – or not. A substantial part of the studies involved with this type of research focus on (inter)national and regional differences in adolescents' media use (Roe, 2000; Schuurman, Courtois & De Marez, 2011), on distinguishing the differences between old and new media usage for young populations (Johnsson-Smaragdi et al., 1998), or on specificities of new media literacy (Brown, 1998; Brown 2008) and audio-visual media consumption with specific age groups (Adoni & Nossek, 2001). The fact that new media attitudes differ significantly by age, family dynamics, geographical factors, social status, economic position and education, and consequently, general tendencies are very hard to convey by probing very specific populations. Also, gender divides in ICT attitudes (Broos, 2005) and other psychological factors greatly influence ICT adoption (Broos & Roe, 2006), making a good overview of the multi-literacies that are relevant to the underlying form of digital production (Erstad, Gilje & de Lange, 2007) difficult to capture, as the digital divide is much more than a merely generational phenomenon. To date, the systematic exclusion of some of the variables can result in a realistic assessment of different profiles of new media adoption (Schuurman et al., 2011), but when considering the impact of and attitudes towards new media in the different segments of the cultural-creative economy, a substantial amount of different viewpoints
50
(sociology, psychology, economy and marketing, human computer interaction and informatics, etc.) are required to get a good perspective. The rest of this thesis will come back at length to the key elements discussed in this section, and will involve itself with the supposedly relevant personality assessments, musical identities (Rentfrow & Gossling, 2003) cultural profiles (Roose, 2008) and collaborative end-user customization (Bentley & Dourish, 1995), in order to gain insights in the technology’s stakeholders and into viable embedding strategies to cope with the complex nature of a rapidly changing cultural-creative landscape.
51
3
Methodology and Conceptual Model
In this chapter, we discuss what working methodologies were applied throughout the research project, how they evolved, and how eventually an amalgamated methodological model was ascertained. We first take a closer look at the various methodological elements that were used throughout the different stages of the research project and to what end they were investigated and incorporated. Throughout the application of these methods, a number of recurrent concepts could be defined. These are consequently presented as a set of literature-based design guidelines (featuring user-related, device-related and context-related elements) for collaborative and/or interactive music interfaces. In the subsequent conceptual model that is derived from these methodological considerations and design guidelines, the most important concepts are summarized and in turn operationalized as a number of assessment parameters. First, the user-centered and iterative development cycle that results from the joining of different research methods is covered. Secondly, the employed heuristics and metrics for EMMT assessment are presented and finally, the applied parameter spaces and indexes are presented.
3.1 METHODS At the beginning of this project, the work plan encompassed a state of the arts of EMMT development (cf. Chapter 2.2). At this point in the project, it became clear that most of the ongoing EMMT-related studies had either an outspoken psychological approach (focusing on perceptual and/or behavioral effects of EMMT interaction) or were prone to a predominantly technical standpoint (with a focus on the implemented interfaces and computational aspects of certain device). This instigated the choice to commence research from the vantage point of interface-based experimentation.
53
3.1.1 Applied experimentation During this first stage, a few empirical studies were organized, which utilized a combination of post-interaction questionnaires, interviews and video-annotations. This collection of video annotation (of gestural communication and entrainment), quantitative and qualitative material yielded only peripheral and partial results, indicative for the limitations of an unpremeditated RtD-approach. Organizing tests around a given set of devices by varying a preselected isolated musical parameter has time and time again proven to be a sure-fire way to obtain results in musicological research. It is a relatively efficient way to achieve meaningful (i.e. statistically significant) results; however, as seems to be the case in many research examples, these results appear to be very much prone to a plethora of confounding factors (e.g. what musical parameter is under investigation, the overall musical stimulus that is employed, the interface used, the mapping strategy that is implemented, etc.) and it is difficult to properly interpret results. Moreover, more often than not, results do not supersede the level of the individual test case, and are inept to appraise EMMT-qualities.
3.1.2 Interface comparison In a second stage of research, a limited number of system-related parameters (e.g. usability heuristics, self-attested experiences of flow and presence, perceived affordances, goal assessment and enjoyment) were probed surrounding a given EMMT in a relatively rudimentary manner. The test consisted of an interaction session with the device, where in between tests the gestural interface (i.e. the input device) was changed. For the purpose of evaluation, a short post-interaction questionnaire was developed for making a crude heuristic usability assessment of the aforementioned parameters. In this manner, relatively basic, yet identical assessments of UX, flow and presence and QoE could be made with the different iterations of the same EMMT. Methodologically, this set of tests moved into the realm of CAPI and fixed interview techniques (built around said questionnaire).
3.1.3 Survey research The third stage of this research project attempted to make a survey of the various types of current and future EMMT users. The majority EMMT-relevant parameters (i.e. personality traits, socio-demographic information, cultural background and training, music and dance training and preferences, digital literacy and eagerness to interact with state-of-the-art technologies) were defined 54
and bundled in a dedicated online-survey to identify relevant properties of prospective users, and to investigate and establish potential user-profiles and eventual user-groups the technology might cater for. The survey also would ideally serve as a pool from which test subjects could be recruited for further experimentation.
3.1.4 Interview Analysis A fourth stage of the research was concerned with a number of in situ tests revolving around interactive arts installations in a live museal context. These tests availed themselves of finished applications that were based on the paradigm of embodied music cognition and heavily drew upon audience interaction for their basic functioning. The actual studies focused on the self-reporting of the perception of transparency, affordance and enjoyment on the one hand, and the semi-fixed interview-based evaluation of a number of usability parameters on the other. As the user-feedback was directly channeled into the (re-)development of the installations, this study also functioned as a testbed for a design process based on the integration of user-generated recommendations for EMMTs.
3.1.5 Usability Inspection Methods The fifth stage of this research project focused on investigating manageable wielding strategies for AR interfaces, in order to engender meaningful embodied interaction, in the first place with the technology itself, and more importantly with the accessible musical content. A number of elements were adopted from HCI-studies and standard usability evaluation methods to mold a number of EMMT-specific performance test. For this purpose, a set of task-based and experience-oriented heuristics and metrics were devised to probe functional and appropriate action-reaction and perception-action strategies. Concerning the used musical parameters (i.c. pitch, interval and chord), a number of tests were devised to assess whether the interaction with the virtual content was sufficiently comprehensible as well as flexible (i.e. affording an adequately clear and fine-grained maneuvering space to the user).
55
3.1.6 Participatory Design Strategy and QoE Evaluation As the set aim of the research project was to aid the introduction of embodied music mediation devices into multiple segments of the cultural-creative sector and to facilitate more global, ecological usage of these systems, various methods were explored to identify what elements of interactive music systems ensure this transferability, and what constitutes ecological viability. The different instances of the ensuing research project are thus concerned with user-related aspects that need to be addressed to make this transition possible. Throughout the various stages of the research described in the subsequent chapters (cf. infra), the need for a multi-method approach became increasingly clear. Therefore, most of the methods from the previous stages were eventually brought together in the final project (see Chapter 9), which on the one hand explored the boundaries of the adaptability of the interactive music systems (and their suitability for specific artistic practices), and the adaptability to its user on the other. This resulted in an artistic incubator that was organized as a Living Lab setting, in which a number of EMMT-customized HCIapproaches were joined together. In various stages of the participatory development process, focus group sessions and interviews were organized with the (prospective) test subjects and as an evaluation methodology for interactive interfaces in support of creative practices EMMT-related HCI and UX investigations were operationalized as post-interaction questionnaires. Suitability of different methods for either more task-oriented or more experience-oriented issues had been established within the prior research stages, yet the choice to work with an ecosystem for open innovation entailed a shift from singular interface-based experimentation towards iterative UX-based exploration. Essentially, this means that the methodologies applied throughout this research project underwent an evolution from predominantly user-centered to fundamentally user-driven. The rationale behind it is that a more eclectic approach is needed to eventually facilitate the aim of transition from a lab context into a real music environment. Moreover, this also dictates that the research paradigm, that was initially strictly empirical in nature, shifted to one that is manifestly participatory. More specifically, instead of working with singular approaches to tackle an individual problem, the adoption of a conglomerate methodological model was necessitated, which joins together qualitative and quantitative approaches derived from human-computer interaction, interaction design, usability studies and user experience applied to the field of systematic musicology.
56
3.2 DESIGN GUIDELINES As was stated in the previous section, some EMMT-related challenges require a pluriform methodological framework that combines elements that are qualitative in nature, as well as more quantitative approaches. To cope with the multifarious data and the specific requirements thereof, it appears to be advisable to work along the lines of a design strategy that fosters a number of setup-related guidelines, as well as a design strategy, design-specific guidelines and usability guidelines (see figure 3.04). Since the beginning of the NIME conferences, publications have steadily surfaced in which attempts were made to make an anthology of interface types and systems, as well as to summarize and compare efficient interaction strategies. These often enumerate the beneficial properties of those systems, which are considered to be the "best practices". In what follows, an effort is made to synthesize these warnings and recommendations (Leman et al., 2011)
3.2.1 Setup In terms of setup, EMMTs traditionally have the following properties: they conceptually consist out of an 1) input device (e.g. a custom-built interface, a sensor technology, an augmented instrument or a motion capture technology) which through a cable or wireless protocol (e.g. Wi-Fi or Bluetooth) is linked to a 2) computer. Within the dedicated software, the 3) unrefined data emitted by the device is collected, and by means of 4) data processing, relevant information is extracted.
input device (e.g. sensor)
computer
data
feature extraction
mapping
Output
Figure 3.01: Schematic representation of most EMMTs’ basic mode of operation.
These relevant cues can in terms be 5) mapped onto the desired 6) output devices (figure 3.01). The user can then evaluate the outcome and choose to either persist or change the behaviors that were recorded by the input device (Miranda & Wanderley, 2006).
57
Furthermore, most systems feature a limited set of controllable parameters (which are deliberately kept open-ended and variable) and a set of fixed parameters. These parameters may include 1) artistic practice, 2) cultural purpose/goal, 3) number of participants, 4) number of (virtual) elements, and 5) element behaviors on the one hand, 6) the used technological interface(s), g) the basic operations, 7) the core mapping strategies and 8) the sonic and visual output on the other. Most systems feature some degree of flexibility, but the number of controllable parameters is normally (and intentionally) kept quite small, to ensure that between input and output, there would be some recognizable features (figure 3.02). Figure 3.02: Table showing classic setup-properties for most EMMTs.
Aspect Cultural Practice
Traditional MMC operation Semi-fixed or Fixed
Artistic Purpose Number of participants Elements / Scenarios Sonic output Visual output Technologies Basic operation Basic mapping
Fixed Fixed - Variable N/A Fixed - Variable N/A - Fixed - Variable Fixed Fixed Fixed
This is also related to the fact that the data generally serves a double purpose (i.e. feeding gestural information to the software, and being used for post-hoc statistical analysis). This will be dealt with more at length in the discussion section (Chapter 11).
3.2.2 Strategy The advocated design strategy (cf. §3.1.6.) consists of a participatory, user-centered, iterative and interaction design-based approach that can help overcome some of the reported challenges (see Chapter 1). The participatory design method (Kensing, 1998) ensures the user-centered system design is done in a pragmatic, functional way, to relate to cognitive properties of the users (Roth, 2002) and to define 1) the goals and constraints in the application domain, 2) the range of tasks practitioners (want to) perform, 3) the strategies that can be made available to perform these tasks, 4) the factors that add to task complexity, and 5) the tools that can be employed to achieve goals more effectively. The iterative design approach in turn may employ different implementation strategies for a technology and within a setup built up out of interrelated modular components,
58
throughout various prototyping stages it allows changes within a given setup, whilst safeguarding strong elements from previous versions. This direct contact with and feedback from users is beneficial to 1) discover the artistic aspirations, 2) accommodate users to fulfill this purpose using the technology, 3) test use-cases, 4) discuss and define problems, and 5) evaluate and attune the used technologies to meet the requirements formulated in the earlier stages of development. Moreover, a strategy based on an in situ incubator may be advisable to provide insight in viable operation modalities, to instigate user-inspired goal assessment of the technologies and to enhance user-acceptance. This design methodology is used to hint at solutions of 1) how can these reported issues with digital instruments, augmented and virtual reality environments be (partially) overcome, 2) in what way user-centered and user-inspired development can enhance the applicability and acceptance of digital instruments and immersive environments in the cultural sector, and 3) whether the presented development strategy is a valid and “cost-efficient” design methodology (figure 3.03). Figure 3.03: Participatory, user-centered and iterative development strategy enumerating the design stages and the design elements to be addressed during this process.
Design stage Preparatory and participatory user-defined and user-inspired design ideas
Collaborative & Iterative design In situ incubation for improving literacy and adoption
Design element application goals task range performance strategies task complexity factors interactive (virtual) tools revised artistic aspiration goal accomplishment use case testing problem definition evaluation and attunement reported issues applicability valence & cost-efficiency
3.2.3 Design guidelines The literature mentions numerous guidelines, which are commonly used for the general design (but also in part for the evaluation) process. These guidelines are built up from a collection of 1) setup-related elements, 2) use case-related elements, 3) parameters related 59
to the development of immersive environments, and a selection of 4) general usability criteria.
3.2.3.1 Setup related guidelines The setup-related elements (Ulyate & Bianciardi, 2002) that are considered is that a given setup should have 1) a simple and entertaining manipulation strategy, as well as a 2) selfexplanatory mapping strategy. Moreover, 3) responsiveness is considered more important than resolution, and each use case is required to offer 4) immediate and continuous reward, as well as an 5) identifiable and as much as possible 6) an aesthetically coherent response.
3.2.3.2 Use Case related guidelines More specific design choices (Blaine & Fels, 2003) can be based on a number of userrelated, feedback-related and device-related considerations. First, the user-related aspects relate to 1) the objective and focus of the application (i.e. the goal and the implementation strategy), 2) the level of expertise that is required (i.e. balancing skills and challenges to prevent either boredom or frustration), 3) the required level of physicality (i.e. the amount of movement required to effect change between users and between user and interface) and 4) the user interaction (i.e. the level of social interaction the application allows or assumes). Secondly, the feedback-related considerations pertain to 5) the used media (i.e. visual and/or sonic feedback) and 6) the controllable musical range (which is generally closely related to the maximum number of participants, the level of control, and based on the users’ ability to discern their proper input in what is presented in terms of feedback). Thirdly, the device-related elements bear on 7) the choice for interface type(s) and the interface interaction (i.e. what sensors are used and how they can actively or passively be controlled), 8) the location and context (i.e. whether the application is for instance presented as a secure immersive environment or in a more public context) and 9) the technology ‘s scalability (i.e. the flexibility in the actual size and number of possible participants within a given application).
3.2.3.3 Parameters related to immersive environments The assessment of the virtual environment technology (Bowman et al., 2002) relies on the following (generic and non-application specific) parameters. First, the setup aimed at implementing 1) natural actions and engagement with the system, 2) compatible with the user-defined task with a 3) close relation between action and representation. Secondly, 4) coordination, navigation and orientation was kept clear and easy, using 5) realistic and discernible feedback and offering 6) faithful viewpoints. Thirdly, the setup aspired to create a genuine 7) sense of presence. And finally, it provided 60
the opportunity 8) to support learning and 9) to incorporate turn-taking (to safeguard the possibilities for gaming purposes).
Figure 3.04: Table summarizing the design guidelines discussed in §3.2: setup-specific elements, use case-related items, AI&VRE related requirements and usability-specific demands.
Guidelines
Element Simplicity
Setup related guidelines
Velocity Validity
User Use case related guidelines
Feedback
Device
Action
AI&VRE related guidelines
Sensory
Experience
Challenge
Usability guidelines
Skill
Stability
Desired Quality easy manipulation strategy self-explanatory mapping responsiveness > resolution immediate and continuous reward identifiable response aesthetically coherent response goal and focus level of expertise level of physicality level of (social) interaction used media controllable range interface types location and context size natural action and engagement compatible with user-defined task relation action / representation coordination/navigation/orientation discernible/consistent/predictable faithful viewpoints sense of presence support learning and mastery opportunity for turn-taking visibility discoverability operational feedback non-destructive actions and error-control consistency reliability scalability
61
3.2.3.4 Usability guidelines As the development revolves around gesture-based processes in a unidirectional communication between human and computer, a number of fundamental principles of interaction design are examined (Norman, 2010). First, matters of 1) visibility (or latent affordances) and 2) discoverability are evaluated. Secondly, the system's 3) consistency, 4) reliability and 5) feedback with operations needs to be considered. Thirdly, opportunities of 6) non-destructive operations need to be reviewed and finally, the issue of 7) scalability is to be assessed.
3.3 DEVELOPMENT 3.3.1 Challenges This section joins together the aforementioned methodological considerations and the elaborate set of guidelines of the previous section. However, the broadness of the theoretical framework and the wide variety of methodologies and guidelines described in the previous section give rise to four challenges that pervade the development of collaborative immersive environments aimed at artistic expression. First, the transparency of the implemented technologies is a pivotal element in end-user acceptance, so in parallel with users' reluctance to engage in overly complex and intricate system setups, it is of the utmost importance to incorporate the end-users in the decisionmaking throughout the design process (to safeguard system transparency and the users’ willingness to interact). Secondly, artistic immersive environments are not by definition task-oriented, but they generally do harness an imbedded manipulation strategy to serve a prescribed artistic intent or purpose. Within this model, we strive to keep that purpose as flexible and openended as possible, so in this respect, the end-users' experience takes precedence over goal assessment, and quality of experience should be advocated to substitute task-driven metrics. Thirdly, the lack of standardization in both operation strategies and application domains exemplifies the need for a framework that at least enables a form of systematized evaluation for user-participation, acceptance and satisfaction with the technology, to establish a common ground between different (conceivable) interfaces.
62
And fourthly, traditional usability evaluation methods are lacking when dealing with immersive collaborative interactive design. Evaluation parameters, heuristics and metrics should be reconsidered and adjusted when deciding on input devices, feature extraction, interfaces, mapping strategies and suitable outputs to elucidate some of the technology's assets, to assess possible goals and to trace technology-adoption issues early on.
3.3.2 Iterative Design Model When summarizing the above enumeration of guidelines, a conclusion can be drawn that from various research fields and in different academic literatures, a utopian system is described which largely fosters the same commendable aspects and ideal elements. An EMMT (irrespective of it being based around physical or virtual control) should have a self-evident affordance as well as a distinct purpose (whether that be a type of artistic performance, revalidation, educational, recreational, etc.), which becomes immediately obvious upon interaction. The implemented mapping it utilizes to link gestural interaction to sonic and/or visual output should be self-explanatory and intuitive. Its assemblage benefits from being modular, to allow for minor changes to the setup whilst safeguarding strong elements. Its basic operation should initially be simple and fun, but incorporating a pathway towards expert performance prolongs the willingness to interact before boredom emerges. Responsiveness always surpasses the importance of fine-grained resolution. Its acceptability is closely linked to a system of reward (e.g. output), and this reward should be immediate and continuous. Finally, its response to actuated gestures should be identifiable as the result of one’s own input, and should ideally be aesthetically coherent. To account for the great variability in purpose and artistic content, a conceptual model is required that enables a rigid evaluation procedure that is able to establish quality of use and experience whilst being flexible enough to cope with large setup-related variability (i.e. superseding the specific use-cases or cultural/artistic practices). On the basis of these previous descriptions, the decision was made to work with a conceptual living lab-based development model for the cultural(-creative) sector that is a variation on the iterative design cycle defined by Gabbard et al. (2002), which is to say that the underlying model follows the same basic design steps. This implies that the conceptual model starts from a top-down framework in which bottom-up use case development is organized in five stages, namely use cases featuring 1) a demonstration, 2) an interrogation, 3) the implementation of the user-requested interaction objects and features, 4) an interaction session and 5) a questionnaire-based evaluation (see figure 3.05). The first stage in the iterative use case development process (i.e. the presentation and confrontation with an abstract demonstrator version of the system) is there to familiarize the potential users with the technology on the one hand, and to ensure an informed
63
consent prior to the collaborative process on the other. The second stage, existing predominantly out of semi-fixed interview questions and presented as a focus group session, informs the assessment of artistic purpose, the recommendation for interaction strategies, for artistic practice, the number of participants and the desired number of implemented objects. The findings of these sessions are then summarized into an implementation summary, which states the use case-specific requirements for both the software engine, the sonic and the visual core. The implementation is a relatively fast and streamlined process (which in practice never exceeded 24 hours). The last two stages of the process then consist of the presentation of and interaction with the collaboratively designed user interface to the focus group participants, as well as the post-interaction usability and UX-evaluation. Essentially, on the level of the individual use case development, on the level of the software and system's architecture, as well on the level of whole (iterative) design trajectory, this constitutes an RtD (Research-through-Design) process, normally associated with Lean or Agile software development, and additionally some forms of extreme programming (not completely dissociate from the implementation strategy discussed in Chapter 9).
[1]Demonstration
[5]Evaluation
[4]Interaction
[2]Interrogation
[3]Implementation
Figure 3.05: Schematic representation of the adopted iterative design and evaluation strategy.
3.3.3 Conceptualization of the Evaluation Metrics This section covers the contents that were dealt with and the information that was gathered during the different stages of the development process. In the iterative design cycle, three information and feedback gathering instances can be defined, i.e. 1) the postdemo and pre-interaction Focus Group sessions, 2) the post-interaction survey contents, and 3) the post hoc complexity indexes that were applied.
64
3.3.3.1 Focus Group sessions Meeting the basic functional and aesthetic requirements was done by means of an intense, short, yet effective focus group session with the participants who had observed or interacted with the demo versions of a given presented system. The questions were subdivided as follows: 1) group information, 2) aesthetic task and artistic purpose, 3) demo evaluation 4) feature implementation 5) action-reaction couplings, and 6) behaviors and scenarios. The first set of questions, in which basic group information was gathered, dealt with the group’s familiarity with new media technologies as well as the participants’ training, experience and level of accomplishment within one or several art forms. The second topic probed the aesthetic tasks and artistic purpose the group envisioned for the system. In this set of questions, an inquiry was made into the participants’ basic understanding of the operation of the platform (in which they were invited to describe their actions and how the machinery interpreted those). Moreover, participants were asked to describe the end to which they would like to use a specific setup, the amount of concentration to execute the set task, as well as the level accuracy and control, which their actions would be translated into sound and visuals with. The third FG set of questions inquested the participants’ appreciation of the features latent in the demo versions of the system; essentially, the quality of the visuals and sounds would be assessed, as well as changes that would need to be made and the ease with which the connection between certain actions and sonic and visual elements could be discerned. The fourth part of the FG session would in turn investigate the envisioned features to be implemented into the specific use case. The optimal number of simultaneous participants would be ascertained, the basic gestural means of operation would be defined, the importance, conditions and social interaction patterns for the UC under development would be defined, and a crude first assessment would be made of demo features which would need to be changed, as well as features that would need to be added to the application. The fifth part of the FG would build further on this and focus on how the sonic and visual features and their action-reaction couplings with the gestural input should manifest themselves. Basically, the functionality of spheres, strings and planes would be investigated, as well as the way in which, within the given the expressed purpose, their sonic and visual properties should be aligned with the gestural input. The optimal number of spheres, strings and/or planes would be discussed with the participants to create a UC which is both sufficiently challenging, as well as sufficiently interesting (i.e. balancing the number of affordances latent in the interaction space to prevent boredom or frustration to set in). Also, the best gesture-to-sound mapping options would be discussed (e.g. 'what type of sound goes well with what type of interaction’, ‘what about properties like
65
appearing or disappearing, collision and proximity’ and ‘what about percussive and/or continuous sound, pitch, panning’). The visuals would be dealt with as a function of this, and essentially the fifth section of the focus group would render the answer to the question: "What type of movements goes with what type of sound and what type of visuals”. Finally, the sixth and last part of the FG would have a double function: first, it would aim at helping speed of the UC development process along, by recapitulating the chosen element behaviors and proposing implementable scenarios (cf. 9.3.2 – ”SoundField” Scenarios–). And secondly, a discussion about other and future uses of EMMTs would conclude each of the focus group sessions.
3.3.3.2 Post-interaction Survey Upon interaction with a number of the UCs, a selection of the active participants would be questioned about their experience by means of a questionnaire. Aside from a concise socio-demographic and musical background query, this survey was specifically designed to probe a set of usability parameters and contain a basic QoE interrogation. Concerning the usability parameters, evaluations were made of the 1) sonic parameters, 2) visual parameters, and 3) how both interrelate. Concerning the QoE, elements of 4) social interaction, 5) ease of manipulation, and a set of 6) use case specific implementations were evaluated. In terms of both the sonic and visual parameters, a) discernibility, b) accuracy, c) applicability, and d) aesthetics were rated; in terms of the interrelation of sound and visuals, d) synaesthesia, e) complementariness, and f) main focus of interaction were rated. As the majority of the UCs drew upon social interaction, for this section of the survey g) in-world perception, h) quality of communication, i) support, and j) incentive were evaluated. Related to tool manipulation k) accuracy and l) goal completion functionality were assessed. Finally, for experience, m) entertainment levels and n) overall appearance were rated.
3.3.3.3 Indexes After the experimental stages were completed, a set of indexes were partly devised, partly deduced to ascertain in what way the multifariousness of setup parameters could account for the variance in the aforementioned parameter ratings.
Figure 3.06: Parameter Complexity Index (PCi)
66
The first index (see figure 3.06) gave a measure for the level of parameter complexity of both the interface and UC-specific setup. This index captured the total number of controllable sonic and visual parameters divided by the total number of manipulatable objects (excluding the virtual objects that were fixed in the room and could be triggered by proximity).
Figure 3.07: Use Case Complexity Index (UCCi)
The second index (figure 3.07) gave a measure for the level of the total use case complexity (ascertaining the action potential for the same setup and features, but also taking into account a social dimension. This index calculated the total number of controllable sonic and visual parameters divided by the total number of manipulatable objects (excluding the virtual objects that were fixed in the room and could be triggered by proximity), and multiplied by the number of actively involved participants (i.e. the number of possibilities for action control open to the different users involved within a use case).
3.4 Conceptual Model By this point, it should be clear that the development of EMMTs and AI&VREs is by no means a straightforward matter; aside from the intricacies associated with creating and embedding an open-ended installation within the convoluted context of the cultural sector, it is in itself a complex interplay of guidelines, goals, challenges, design strategies, evaluation methods and caveats that has to be made allowance of. The conceptual model was constructed partially through trial and error, yet mostly through elaborate experimentation and extensive literature review. It was conceived for a number of urgent reasons, first of which, is out of necessity: as the fourth chapter will indicate, working destitute of a proper design methodology is a guaranteed approach to set up for failure, with resulting EMMTs that feature plenty of gratuitous and undesirable qualities and unwanted proneness to confusion and failure. Secondly, it allows for some form of standardisation, comfort and direction throughout a predominantly open-ended and extremely daunting design and evaluation process. Thirdly and finally, as it sets forth 67
quite a few options, rules and strategies, it embellishes an otherwise lean design process with a sense of ease and agility, as the model dictates what concerns need to be addressed in what stage of the process, and what further course of action needs to be taken given a specific outcome. In sum, the conceptual model comprises the methodologies, the design guidelines, the iterative development model, the stated challenges, the desired qualities and a somewhat standardised evaluation procedure, all into one consolidated RtD-approach. In the heavily abridged schematic representation below (fig. 3.08), the pivotal elements and concepts – dealt with at length throughout this chapter – are enumerated once more. Figure 3.08: Scheme summarizing the guidelines, challenges, operational requirements and design strategy inherent in the conceptual model.
•SETUP: simple, fast & valid •DESIGN: user-based, feedback- & device-driven •AI&VRE: action-oriented, sensory rich & experience-enabling •USABILITY: challenging, skillencompassing & stable
•Prototyping & DEMO •User INTERROGATION (FGs) •Desiderata IMPLEMENTATION •User Testing & INERACTION •Post-hoc EVALUATION (UEquestionnaire & Indexes)
Elements & Guidelines
AI&VRE Challenges
Design Strategy & Evaluation Method
Operational Goals
•Transparent & engaging CONTROL •Flexible, non-task oriented, yet clear PURPOSE •Need for systematised EVALUATION strategies •Early identification of ADOPTION issues through adaptable heuristics & metrics
•Clear AFFORDANCE & purpose •Intuitive MAPPING •MODULAR construction •SIMPLE, fun + learning curve •RESPONSE > Resolution •Immediate & Continuous REWARD •Identifiable & Aesthetic OUTPUT
And so, this third chapter and the first part of the thesis draws to a close. As we have dealt with the broader cultural and research context, the theoretical concepts that underpin the subject matter and the methodologies which enable both the development and evaluation of EMMTs, the following section will zone in on the lab tests that instigated the conceptual model, and the various research studies throughout which the model was gradually applied and refined.
68
Part 2:
Empirical Studies
69
The second part of the thesis presents an overview of how the methodological concepts discussed in the third chapter of the thesis were implemented within various empirical studies. The six chapters are presented in their logical developmental order, although some of the studies in actuality ran concurrently. The fourth chapter presents a series of interface evaluation tests, eliciting the need for a more user-centered development strategy. The fifth chapter discusses the “What Moves You”-questionnaire, directed at identifying EMMT-opportunities as well as desirable end-user traits. The sixth chapter features the reception and perception of two interactive art pieces. The seventh chapter focuses on interface evaluation as well as presence, game enjoyment and goal assessment of a collaboratively controlled music-game. The eighth chapter considers heuristic evaluation and application of usability metrics to two EMMT prototypes systems/interfaces. Finally, the ninth chapter recounts how the findings of the previous chapters have eventually inspired the “SoundField”-setup, in which the conceptual model and integrated participatory design and evaluation approach are applied throughout the entire duration of the project, probing the technology’s artistic potential and ecological validity through elaborate iterative use case testing. All of the chapters contained in this second part of the thesis examine aspects of the complex status quaestionis and probe parts of the hypotheses formulated in the introduction.
71
4
Initial Research Studies
In this chapter, the early studies are described, both implementations and tests, that exemplify some of the fundamental problems encountered in the development and implementation of embodied music mediation devices. In this stage of the PhD-project (i.e. spring 2008), swift development of multiple and pluriform systems with various sensors was still the key aim. Achieving an intelligible, functioning, enjoyable and concrete setup appeared to be the logical outcome of simple trial and error, and experimental gold was to be struck eventually by mere effort and sheer determination. As such, the user-centered paradigm was left unexplored and by no means a priority yet. During the design process, however, apparent flaws and repetitive mistakes instigated the diligent adoption of some methodologies affiliated with HCI, usability and user-oriented design approaches, as these only gradually but steadily became more of a necessity, than a luxury. Design flaws inherent in these early design attempts highlighted the need for a generative user-centered strategy for designing embodied mediation technologies. These applications provided the groundwork for the different procedures contained within the conceptual framework (described in Chapter 3.3 and developed throughout the experiments described in the ensuing chapters), the foundations for the adoption of a profoundly user-oriented design and evaluation methodology, as well as the eventual modular development approach of the interactive installation that was implemented during the “SoundField”project (see Chapter 9).
4.1 “Perpendicam" The initial step in this research project constituted two tests involving the impact of social interaction on music-induced movement. During both a student experiment (i.e. within the context of the 2007-2008 Systematic Musicology class, part of the musicology master 73
curriculum at Ghent University) and an experiment conducted during an intensive summer program (i.e. the 2008 International Summer School for Systematic Musicology) the impact of social circumstances (or absence thereof) on quantity of movement (QoM) and on extraction index, both features of the EyesWeb software (Camurri et al., 2010) was measured. The customized EyesWeb-patch that was used, was developed by dr. Luiz Naveda. Also, qualitative assessments of the quality and quantity of social interaction were established by means of manual annotations.
Figure 4.01: Picture of the “Perpendicam” top video recording footage in EyesWeb.
Figure 4.02: Schematic representation of the Perpendicam 45° angle setup @ ISSSM 2008.
The experimental design of the first test consisted of a group of 6 participants (3 female, 3 male) who were asked to engage in a form of spontaneous free form dancing in a ± 30 square meters. The stimulus that was used consisted of a genuine 12-minute club
74
mix of a selection of 5 (at the time) popular dance songs. The participants were aged between 14 and 16 years old and a total of four groups (i.e. 24 test subjects) was tested.
Figure 4.03: Picture showing Perpendicam2 participants interacting during social condition.
The video footage (see Figure 4.01) was processed in the EyesWeb software, making use of some of the available patches (i.c. extraction indexes and blob-detection) to assess the proximity of the participants to one another and to ascertain the QoM. No conclusive results could be drawn from the resulting data.
Figure 4.04: Screenshot of the ISSSM 2008 Movement Annotation Max/MSP-patch.
In between the first and the second tests, a number of modifications were made to the setup. First, the camera angles were changed from two perpendicular cameras filming the whole group on a vertical and horizontal axis, to the capturing of the individual participants in a 45° angle (Figure 4.02). In the iteration of the experiment, the choice was made to work with smaller groups of participants (i.e. 3 test subjects) and less familiar 75
music styles were chosen as a stimulus (i.c. 30 second excerpts of ethnomusicological field recordings). Also, the ten music excerpts were randomized and presented twice to all participants, respectively in a solitary and a social condition (Figure 4.03). This would technically allow us to measure the impact of the social versus the individual condition, as well as the impact the acquaintance with the other participants would have in relation to the quantity of movement. The extraction index was dropped from the experimental design, and a qualitative assessment for quantity of movement, in this instance inspired by body schema theory, was manually annotated and automatically time-stamped during video-playback, using a customized Max/MSP/Jitter-patch, developed by Pieter Coussement (Figure 4.04).
Figure 4.05: Graph showing results of the Perpendicam2 Annotations @ ISSSM ’08.
Additional information, which aside from socio-demographic information and musical background, was concerned with acquaintance, self-perception of activity and social interaction, and acquired by means of questionnaires and short interviews. Some invaluable lessons were learned during these experiments. Most importantly, the qualitative assessments were in essence more valuable to the experiments than the actual data that was gathered. The adjustments that were made to the experimental setup had quite profound influences on the outcome. First, the changes in the setup considerably
76
improved the ambiance of the experiment and had a profound influence on the quality and quantity of the social interaction between participants (Figure 4.05). Since in these exploratory stages the research focus was still heavily on modalities of dance – and since dancing within the presented context could be considered an expression of enjoyment – making the participants feel at ease (i.e. not manifestly confronted with a lab setting) proved to be a very important asset.
Figure 4.06: Graph showing the results processed in MatLab of the manual QOM-analysis of Perpendicam2 @ ISSSM ’08.
More importantly, the setup changes made in between the two test-sessions to a certain extent had an impact on the goal of the experiment. Whereas in the first test the focus was predominantly on the quantity of movement and extraction (see Figure 4.01), in the second test the attention shifted somewhat towards the quality of interaction between the participants (as it proved to be the key component in the quantity of movement). Accordingly, the outcome was severely altered by the setup changes that were made. For example, the change in camera angle had quite a negative influence on the footage for the purpose of quantitative analysis in EyesWeb, yet, it made the footage all the more useful to make the manual annotations in Max/MSP (see Figure 4.06). So, from this perspective, the setup change also instigated a necessary focus shift from quantitative to qualitative assessments. Even though the analysis of the quantitative results obtained in the first test did not yield any particularly useful findings, it still opened up some previously unexplored perspectives for this research.
77
4.2 “Beat it” & “Dance2 the Music” Simultaneously to these tests, development was started on an embodied music mediation device (much in the same vein as the various NIME-applications discussed in the theoretical framework). The main focus of this design was on responsive gestural interaction with a music-controlling system and much less on user-related elements. For this purpose, some preliminary tests were done in the context of a number of small applications, called “Beat it” and “Dance2 the music”.
Figure 4.07: Screenshot showing the Max/MSP-patch controlling "Beat it".
The first application presented three participants with separate instrument sections of the same music track. A patch was created in Max/MSP that allowed each of the participants that was handed a Nintendo WiiMote to trigger an instrument section linked to it (figure 4.07). This was done by rhythmically shaking the device to a click-track in a vertical axis, much in the same way as tapping the beat and similar to waving a baton. When a participant got synchronized with the beat within a pre-established accuracy level (i.e. a six BPM threshold) for more than three consecutive beats, the prerecorded track associated with that particular WiiMote was activated. Accordingly, when a participant stopped moving or the participant 's synchronization surpassed the accuracy threshold, the feedback stopped (until three correct beats were registered again). Based on the size of the jerk and the intensity of movement, three different sound-levels could be triggered by each of the participants. That way, they could 78
play at low sound Intensity levels at 60%, at a medium sound intensity level at 80% and at a high sound intensity level at 100% of the maximum volume. In order to incorporate a gaming aspect, an opportunity for reward was built in by means of a bonus track. When all three participants entrained and attuned the intensity of their movements (thus conforming their sound intensity levels), the bonus track was triggered.
Figure 4.08: Picture showing three participants experimenting with the 2.0 version of “Beat it”.
A first experiment with “Beat It” was done on the 24th of April 2008 with a group of adolescents aged between 16 to 18 years old. Analysis of the synchronization data as well as a non-formalized post-interaction evaluation showed that the application showed considerable flaws. The main problems pertained to the chosen threshold (which was too narrow) and the sound intensity levels, which were not sufficiently contrasting to easily discriminate. Thus, in a second experiment, run on the 15th of May 2008 with a group of students with ages ranging from 19 to 23, the threshold was increased to 10 BPM (five above and five below the actual tempo) and the sound intensity levels were respectively put at 100%, 120% and 140% (figure 4.08). The general evaluation of this project exemplified a number of design mistakes, not uncommon in the development of embodied music mediation technologies. First, no trial phase was included in the experiment, in which participants could freely interact with the system, assess the levels and thresholds and familiarize themselves with the sounds assigned to their controller. As a consequence, participants had to discern the basic operation as well as their role during the experiment and while interacting (with the system and each other). Moreover, there is a dire need for visual feedback when operating 79
this type of technology. With participants solely relying on aural cues, at best aided by the screen-representation of the software-patch, discernibility is low and confusion is imminent (see Figure 4.09 in reference to the difficulty to entrain). This made the gameaspect increasingly difficult to grasp, and consequently, as soon as all three participants would correctly entrain with each other, even more confusion would ensue (see Figure 4.10 for glitches due to the entrainment effect). This resulted in a strange recurrent phenomenon, where practically as soon as all three participants conformed to the same sound intensity level, thus triggering the bonus track, at least one of the participants would stop moving entirely.
Figure 4.09: Graph showing a MatLab plot example of total time in level for one of the groups in “Beat It”.
The results of the movement analysis were inconclusive at best, and during post interaction interviews, a large number of test-subjects reported that over the course of the game they had felt either confused, bored or frustrated with the system. This faulty application aptly demonstrated the need for more user-involvement during the development, as most of the reported issues could have been anticipated solely by demoing a prototype to a small number of test users during the development.
80
Figure 4.10: Graph showing the autocorrelation analysis performed on the raw WiiMote-data obtained from one of the groups interacting with the “Beat It”-setup.
On the 22nd of June 2008, a demonstration, called “Dance2 the Music”, was given for an open day with roughly the same system that allowed four participants to simultaneously control a number of musical parameters of prerecorded tracks (figure 4.11). As the participants were elementary school children, the interaction was somewhat simplified and more controlled. The different sound intensity levels were omitted and the thresholds were increased even further. The metronome-track was substituted by a genuine rhythm-section (i.e. drums and bass) and the controllable tracks were distinctly different instrument groups (e.g. keyboards, guitars, percussion and a brass section). A researcher took up the role of conductor who prompted the children to activate their tracks either separately, in groups, or all together. A lot of these changes vastly improved the experience for the users. Not only did this demonstration have a significantly reduced task complexity, the distinctly different tracks gave the participants the opportunity to efficiently discern the audio assigned to the different controllers. Moreover, there was an enhanced experience because of the simplified and more straightforward action-reactioncoupling. Finally, the incorporation of visual cues (albeit only by means of a conductor giving instructions) safeguarded the gaming element and improved game enjoyment.
81
Figure 4.11: Picture showing a group of participants and a conductor interacting and performing live with the "Dance2 the Music"-setup.
In these preliminary tests, the importance of the developmental considerations greatly outweighed the implications of the actual embodied music mediation system and the quantitative results of the movement data analysis. This instigated the idea that there was a lack of proper evaluation techniques for gesture-based unstable media art technologies. The present development also illustrated the necessity of a user-inspired development process for embodied music mediation devices, as a far-reaching user-oriented approach would help to elucidate some of the requirements connected to the contextual application of these technologies.
4.3 Rudimentary Usability & Mapping test with the EME @ ISSSCCM 2009
A third preparatory stage for this research project consisted of setting up a studentexperiment for the 2009 ISSSCCM (i.e. the International Summer School for Systematic, Cognitive and Comparative Musicology). The study itself was not executed within the scope of the thesis, yet the experiment proposal and guidelines were completely based on some of the prior experiences from the “Synch-in-Team”-study (which will be discussed in the following chapter) and contained a lot of the ideas discussed in the theoretical framework. The experiment pertained to a wide variety of features connected to new interfaces for musical expression (i.c. it also entailed a small flow and presence assessment study), but essentially, it pivoted around a rudimentary interface evaluation. Based on literature
82
review of related studies and the findings of the two previous exploratory experiments, one of the goals was to establish whether solely changing the input device of an application would constitute any improvements in control and usability of a given application. Correspondingly, the other aim was to assess how the changes in the physical interaction with the sensor device would influence the embodiment schema. Finally, this pilot study would attempt to incorporate user-feedback for improving the usability of a pre-existing application (by making small changes to the adjustable setup).
Figure 4.12: Picture showing the demonstrator of the IPEM EME & associated HOP sensors.
The setup consisted of an existing music controller system, namely the Max/MSP-based application EME, developed at IPEM (Cornelis, O., Demey, M. & Leman, 2008). This application allowed users to control a set of predetermined musical parameter (e.g. intensity, timbre), to control filters and/or to trigger MIDI-events. The control of the expressive musical features was achieved by means of both a gyroscope and a 3-axis accelerometer, which would translate actions that mimic interaction of more conventional controls (e.g. a button pushed or a knob turning), as well as more abstract corporeal articulations (e.g. music intensity connected to the shaking of an arm) to musical stimuli (Figure 4.12). The system was originally operated with custom-built HOP-sensors (Kuyken et al., 2008), but in this study, different input devices with comparable specifications (e.g. Wii remote with the adjacent Wii Motion Plus or the iPod Touch) could also be used to give similar expressive control over the identified parameters within the same software setup and through a basically unaltered software mapping. (The affordances latent in the setup were those of a virtual knob being turned, a filter being controlled, or more intensity being activated.) Experiment participants would then be asked to operate with both interface devices. After having managed the application with
83
both input devices a number of usability metrics would then be applied (i.c. task analysis) to assess how fast/well a certain task could be performed and to establish which of the input devices yielded the best affordance and control (i.e. the optimal usability for the setup). By means of an ensuing questionnaire, the participants would also be interrogated about their experience of flow and presence and their enjoyment during the interaction. The follow-up questions would then deal with: 1) the difference they had perceived between both devices, 2) which of the configurations yielded the best control to perform the set tasks, and finally, 3) in which of the experimental conditions the sounds were more easily manipulatable than the other. During the execution of the actual experiment, a lot of variables from this initial proposal were changed. As the experiment eventually only used one device (i.e. the HOP sensor) with two different mappings, the whole idea of interface evaluation was omitted. Aside from that, no iterations were done and prior to the experiment, no genuine task was defined, which made task analysis altogether impossible. Moreover, the HOP sensors performance was reported to be lacking and the main focus of the questionnaire was on flow and presence during the interaction. Consequently, as no genuine interface evaluation was conducted, the remainder of the evaluation did not surpass an appraisal of the device that was used and the differences in quality of interaction were essentially not assessed beyond the scope of user experience.
4.4 Conclusion In summary, these early setups did prove to be successful in terms of interface development as they had reliability, sensor, mapping and sonification problems to deal with, not unlike some other NIMEs described in the literature (cf. supra). Because of this, the experiments that were done with these devices did not render any conclusive results. Also, in terms of system development, this procedure of developing “one of”devices could have gone on for several more years without actually resulting in any more useful or usable - let alone ecologically viable - applications. During the tests conducted with “Perpendicam”, the “Beat it” application and the ensuing “Dance 2 the Music”, it became apparent that attempting to test elements of flow and presence in conjunction with testing embodiment parameters as well as game features, scattered the attention of the studies all too much. Had the research aim been more attuned to only one of the above topics, the outcome might have been more easily understandable setups and more manageable musical parameters. Moreover, there was a large proneness to the stimuli and musical parameters being used in the various applications. Essentially, these setups to a large extent were experimental lab setups disguised as games. 84
Additionally, all the aforementioned applications and tests suffered from severe hard- and software (reliability) problems. Hence, given the faulty software, the at times difficult to understand and static gameplay, ambiguous stimuli, unclear controllable parameters and dispersed research focus, a number of decisions were made to re-adjust goals for the research project. First, application development would no longer be one of the research endeavors, but would be organized within an interdisciplinary collaboration with a proper software development team. Secondly, the necessity for end-user expectations and capacities in the pre-development stage became apparent to ensure the application would fit the artistic goals of the test audience as well as the eventual adoption by the user (and by extrapolation, the sector). Thirdly, these informed design choices would need to be reinforced by educated guesses and selective trial and error in a design strategy that would use iterations to accommodate rapid interface adjustments in small testbeds. This could then eventually evolve into an artistic incubator (e.g. the “SoundField”-project). Still, the early projects proved to be useful efforts as the nonfulfillment of goals set forth in the above projects can be considered essential to the origination and evolution of the previously presented methodological and developmental framework. The outcome of these pilot studies was often considerably altered by relatively small changes made to the setup and the developmental considerations that arose from the setup changes were found to be more interesting than the results of the data analysis and the quantitative assessments. Moreover, as the quality of interaction with the various setups proved to be quite hard to establish and the evaluations performed did not exceed a merely functional assessment of the device(s) used, the lack of proper evaluation techniques for gesture-based unstable media art technologies became apparent. And finally, with the prospect of having to meet requirements of a contextual application of these technologies outside of the lab, the necessity of a user-involvement and a user-inspired development process for embodied music mediation devices became manifest. This chapter dealt with a number of rather unfulfilling setups and seriously flawed iterative design tests, the examples do elucidate a number of the pitfalls associated with EMMT-development. First, scarcely concealed research setups make for relatively poor musical game designs, and secondly, some form of user-involvement from the early development stages onward may well be advantageous in preventing confusion, quandaries, boredom or errors in the eventual design. As this chapter recounts some of the early attempts of EMMT-creation, simultaneously a survey was constructed to zone in on what constitutes some of the necessary capacities and relevant convictions of prospective AI&VRE- and EMMT-users. Therefore, the following chapter will deal with musical background, training and preferences, in a quest to define provisional user types for EMMTs.
85
5
What moves you
Whereas the previous chapter discussed some early conjectural setups and trial experiments, this chapter takes a step away from the actual music mediation devices and interface evaluations. Partially overlapping in terms timing, another study zoned in on what prospective users would bring to the table before the actual interaction. As such, in the form of an online survey, a sociomusicological study was conducted to gaining more insight in the personalities and attitudes of the future participants. This study would deal with culture-enthusiasts' (in)formal musical training, personal background, as well as music and dance predilections. On the one hand, this could enable the potential definition of a set of profiles for EMMT-users, on the other hand, it could in a later stage of development eventually also serve as a well-documented panel of eventual test subjects.
5.1 Motivation The exponential rise of new media at the turn of the century radically and fundamentally changed the outlook of the creative economy and the cultural sector. Various national (De Voldere et al., 2006) and international studies (e.g. UNESCO, UNCTAD & KEA) investigate this changed reality of cultural and creative economy. Pivotal in these studies are the diversification of the actors in the concentric model of the cultural industries (Throsby, 2008) and the definition of challenges and opportunities for the creative economy in various regions (Nielsén, 2009). Central topics to this line of research are the various trends and perspectives of new media adoption that can be discerned. Traditionally, studies focused on (inter)national and regional differences in adolescents' media use (Roe, 2000), on distinguishing the differences between old and new media usage for young populations (Johnsson-Smaragdi et al., 1998), or on specificities of new media literacy and audio-visual media consumption with specific age groups (Adoni &
87
Nossek, 2001). More recently, also gender divides in ICT attitudes (Broos, 2005) and other psychological factors that influence ICT adoption (Broos & Roe, 2006) are being considered, pointing out that the digital divide is more than a merely generational phenomenon. An easily overlooked aspect is that new media attitudes differ significantly by various socio-demographic variables like gender, age, family dynamics, geographical factors, social status, economic position and education, and consequently, general tendencies are very hard to convey by probing specific populations (Lievens & Waege, 2011b). So, to date, only the systematic exclusion of some of the variables can result in a realistic assessment of different profiles of new media adoption (Schuurman et al., 2011). Moreover, when considering the impact of and attitudes towards new media in the different segments of the cultural-creative economy, a substantial amount of different viewpoints (sociology, psychology, economy and marketing, human computer interaction and informatics, etc.) are required to get a good perspective. Personality assessments, musical identities (Rentfrow & Gossling, 2003) and in a broader sense cultural profiles (Roose, 2008) of the stakeholders are key elements to gain some insight into the complex nature of a rapidly changing cultural-creative sector.
5.2 Purpose In respect to the mission statement put forth in the description of the fourth work-package of the Emcometecca-project, shortly after finishing the Sync-in-Team-study in early 2009, work was started on a survey of the Flanders cultural landscape. Moreover, to work on useful developmental strategies for embodied music mediation technology, a good deal of a priori knowledge about future users, the context in which they will be used and the purpose they should serve, is vital. Therefore, complementary to the usability studies, other methods are needed to provide this information about a target audience and to base eventual developmental strategies. For this part of the project, one of the ideas was to organize a type of panel discussion between the various stakeholders, in which cultural institutions, creative industries, research departments and prospective end-users could collaboratively assess opportunities and the viability of embodied music mediation technologies within the cultural sector. From the part of the end-users, this was to be done by means of a multi-purpose questionnaire, in which relevant viewpoints concerning embodied music mediation technology could be probed efficiently and on a large scale. On a considerably smaller scale and with a very specific and exploratory purpose, attuned to embodiment and audience adoption of embodied music mediation technologies, the presented survey crudely follows some of the same basic research lines as the Flanders 88
participation survey (Lievens & Waege, 2011) and the IBBT Flanders Digimeter (De Marez & Schuurman, 2010). The purpose was to (1) get an overview of a representative end-user portion of the stakeholders, to (2) gain insight into the relevant convictions and beliefs in relation to the development of embodied music mediation tools, and to (3) identify possibilities for efficient and effective embedding of the technology. In parallel with this survey of potential end-users' cultural profiles, artistic education, cultural background and new media behavior, an attempt was done to create an annotated database/bank/pool of potential prospective users among culturally active audiences. Eventually, based on key components of the related work, a questionnaire was developed, entitled "what moves you". In addition to an exhaustive overview of personality traits and cultural predilections, the questionnaire aimed at offering a comprehensive overview of new media knowledge, usage, literacy and attitudes, as well as it made an attempt to probe self-reported movement and dance strategies, associated with embodied music mediation technologies.
5.3 Survey setup, structure and contents "What Moves You" was an online questionnaire. Various different platforms were tested (e.g. Sona Systems, Survey Monkey, Qualtrics and Googledocs). The questionnaire was eventually implemented using the open-source LimeSurvey-system to gather relevant information of a wide range of possible prospective users. This software, though relatively cumbersome in terms of installation, yielded the most options for data export and featured the most elaborate variety of question types. It ran simultaneously in two languages, had 61 (multi-item) questions and five questionnaire sections, covering seven topics: (1) demographic information, (2) personality traits, (3) background and education in music and dance, (4) cultural and musical preferences, (5) movement and dance as a response to music, (6) dance and dance-music preferences, (7) acquaintance, usage and attitude towards new technologies. Aside from gender and age, the demographics section dealt with current studies or professional occupation and highest achieved education level. This section also gave the opportunity to sign up for a wide range of future experiments. Finally, a personality assessment (of both individual and social personality traits) was presented using a 7-point Likert scale, based on the "Big Five Questionnaire" (Rentfrow & Gossling, 2003). The second section of the questionnaire probed artistic education, namely the respondents' background in music and dance. First, the duration and types of formal music education were investigated. Then, respondents were asked whether they mastered a 89
music instrument, and if so, which instrument(s) they played, what their best instrument was, how long they had been playing, what their average practice time was, what level they had achieved, whether or not they made their own music, and if so, what different types of music production they engaged in. The third part of this section asked about dance education: whether or not people had received a form of dance education, and if so, what type(s) of training, for how long they had taken dance classes, the types of training and the types of dance, how they evaluated their own performance, and if applicable, what formal level they had achieved. In the third section of the questionnaire, the respondents' cultural and musical profile was established. To assess cultural preferences, to determine the culture-enthusiasts' appreciation and attendance-frequency, an exhaustive list of different types of cultural events was used. Using a similar, not overly specialized list of different music styles, inquiries about the appreciation and the frequency of active listening to different music genres were made. Moreover, to investigate how broad or specialized their musical tastes were, they were asked to attest their favorite music genre, and finally, to estimate the average listening behavior, participants were asked how many hours per week they spent both actively (i.e. listening) and passively (i.e. hearing) involved with music. In the fourth and fifth section, bodily responses to musical stimuli are evaluated. In the first part of the section, an analysis is done of subtle body movement that occurs as a response to musical cues. The frequency, situation and manner in which it occurs are examined, as well as the repetition and the emphasis on different parts of the body. In the second part, the social settings and incentives of dancing in response to music are quantified, as well as the duration of and reasons for dancing. In the third and last part of this section, dance preferences are estimated: in what styles do people like to dance and with what frequency do these given dance styles occur. Moreover, in the sixth section of the questionnaire, the respondents were asked to make an appraisal of the frequency in which they dance in response to different music styles and what music styles they prefer to dance to. In the seventh section of the questionnaire, habits and attitudes concerning new technologies were probed. The respondents were asked to estimate their active involvement with new technologies, with computers and with computer-related appliances. Then, they were asked to judge the purpose and the frequency of their internet usage (disregarding the time spent on the internet for professional purposes). In a next part, acquaintance, typology and manner of gaming was probed (Schuurman et al., 2008). Then, both the personal perspective and the acquisition attitude towards new (digital) technologies was determined. Finally, in reference to new possibilities for interaction made available by state-of-the-art technologies, the respondents were asked about their opinion towards audience participation in cultural events, both in terms of attitude, manner and extent.
90
The online questionnaire was dispersed with the aid of a number of different organizations active in the cultural landscape of Flanders, with the goal of screening and recruiting diverse culture-enthusiasts with different musical and cultural preferences for experiments. The survey ran in collaboration with Ghent University (http://www.ugent.be/), Muziekcentrum Vlaanderen (http://www.muziekcentrum.be/), Kunstencentrum Vooruit (http://vooruit.be/), Muziekcentrum Bijloke (http://www.debijloke.be/) and the Institute of BroadBand Technology (http://www.ibbt.be/en), which later became iMinds and currently operates under the IMEC-umbrella. Numerous other stakeholding organizations in the Flanders cultural creative sector were contacted (e.g. Sociaal Fonds voor de Podiumkunsten, Belgian Entertainment Association, Clubcircuit, Federatie Muziekfestivals in Vlaanderen, Muziekmozaïek, Overleg Kunsten Organisatie, Poppunt), yet none of those offered their collaboration to this project. Finally, a terminal running the survey was set up for the period of two weeks at the Bijloke Music venue to entice some of the concertgoers to partake in the questionnaire. This recruitment approach, combining elements of targeted and to a lesser extent snowball sampling, eliminated to a large degree the randomization of participants – which was not a key objective of the survey anyway. Conversely, it provided a far better way to investigate how a specific part of the population (i.c. predominantly young culture-enthusiasts prone to display a high degree of outdoor culture-participation) likes to interact with a certain type of technology within a given context. The questionnaire ran in a test phase in the fall of 2009, and the survey was active from late 2009 to spring 2011. In this period over five hundred (n=512) respondents participated, of which close to 400 (n=381) completed the entire questionnaire. Given the fact that there is no clear indication of the amount of people that were contacted throughout the various outreaches by the organizations listed, it is neigh impossible to accurately calculate a response rate ratio; although realistically, based on the limited sample size, it would be improper to assume that the obtained results would be suitable to infer any claims that would apply to the general population.
5.4 Results The first section of the questionnaire revolved around basic socio-demographic elements, interest in musicological experiments and personality assessments. About two thirds of the respondents were female (69,8%) and also two thirds of the respondents were in the 16 to 25 age-category (67,2%). Less than 20% of all respondents were over 36 years old. Correspondingly, close to 60% of the population were full-time students. As education level is concerned, close to 30% of all respondents had obtained a Master degree, over 91
50% had finished their Bachelors, and given the fact that over 60% of all respondents were students, it can be attested that the education level of the respondents was overall relatively high. A next set of questions – and essentially the main reason the institute was invested in organizing a survey in support of the EMCOMETECCA-project – investigated the respondents’ willingness to participate in future research experiments of the systematic musicology department. Respondents were asked whether they would be willing to partake in future research (to which 25% answered negative), and, if so, whether that involvement would be limited to online questionnaires (32,6%), or they would also volunteer for lab tests (42%). Within that 74,6% that indicated being interested in future participation, respondents were most enthusiastic about future experimentation revolving around e-health applications, interactive (E)MMTs, digital music education tools and fundamental embodiment and movement research, and slightly less interested in collaborating on future developments concerning musical gamification, NIMEs and MIR. The last part of the socio-demograpic section of the questionnaire, i.e. the personality assessment, probed both individual and social elements of the participants’ personality. Respondents were asked to self-report on 16 individual and 15 social character traits using a 7-point Likert-scale. All the response distributions of the summative scale roughly followed a pattern approximating a normal probability distribution, even though most deviated in some way. Overall outcomes for “melancholic”, “lazy”, “extraverted”, “carefree”, “competitive” and “impulsive” were all negatively skewed (i.e. left-tailed) and platykurtic (i.e. sub-Gaussian). Responses for “shy (individually)”, “introverted”, “moody (socially)”, had a positively skewed (i.e. right-tailed) and platykurtic (i.e. sub-Gaussian) outcome. The measured results for “originality”, “relaxed”, “curiosity”, “energetic (both individually and socially)”, “enthusiastic (both individually and socially)”, “wellorganized”, “ingenious”, “persistent”, “artistic”, “sophisticated”, “creative”, “talkative”, “relaxed”, “assertive”, “sociable”, “cooperative” and “inspirational” were all negatively skewed (i.e. left-tailed) and leptokurtic (i.e. super-Gaussian). The plotted area for items “quiet”, “shy (socially)” were positively skewed and leptokurtic. Finally, answers for “moody (individually)” had a mesokurtic and unskewed distribution pattern. The deviations from the normal distribution apparently seem to be grouped around the same five categories – and their sub-categories – found in the Big Five, namely extraversion (containing traits like enthusiasm and assertiveness), neuroticism (covering volatility to withdrawal), conscientiousness (with such elements like industriousness or orderliness), agreeableness (dealing with such qualities like compassion or politeness) and openness to experience (ranging over openness to intellect). Moreover, the amount of skewness and kurtosis appear to be correlated with the fact that the qualities described in the character traits listed, are perceived by the respondents as desirable properties or not – not uncommon in personality research. Also, no demonstrable proneness to central tendency could be attested in this part of the survey (which is to a certain extent indicative
92
of the fact that most of the respondents were not altogether unfamiliar with questionnaire research, and/or took the completing of it to heart). However, without going into the intricacies of multivariate models, factor analysis and multi-dimensional or hierarchical clustering techniques associated with this type of research, no further assertions have been made at this time on whether the attempts on simplifying the pre-existing personality assessment models were at all successful. Though interesting in and of itself – i.c. to ascertain whether this rather comprehensive and rudimentary version of the “Big Five”-questions could result in comparable analyses and similar parameters as the original – the personality assessment would only come to be relevant within a further stage of research (i.e. when the survey respondents would serve as potential participants for testing actual EMMT-applications). Since this utilization of the survey eventually proved to remain suppositional, and since such an additional research goal would vastly overshoot the purpose of this thesis, this part of the questionnaire was not processed in much more detail, and no further comparisons were made. The second section of the questionnaire dealt with various forms of art education. Approximately two thirds (60,4%) of all respondents had received some form of musical education, about half (49,9%) received some form of dance education; almost a quarter (23,1%) received 10 years or more of formal music education; numbers for dance education are significantly lower. Of those who had received a formal or dance education, duration and type(s) of lessons were probed1. As musical instruments were concerned, respondents were asked about whether or not they actively played any (49% did), and if so, how many and which they played2, what the chief instrument was, the amount of years they had been involved with them (27,5% had over ten years of music training), weekly practice regimen (only 10% playing less than an hour weekly), and the self-perceived level that had been achieved. Furthermore, subjects were asked about the making of their own music, and, if so, in what way(s) they were involved in their own music production.
1
Concerning both music and dance education, respondents were asked whether they were autodidact (M:10,2% / D: 1%) or had followed private (M: 17,8% / D: 6,5%) or formal part time art education (M: 53% / D: 11%), learned from books (M: 13,4% / D: 0,5%), magazines (M: 4,5% / D: 0,2%), instructional videos (M: 2,9% / D: 0,8%), conservatory (M: 4,7% / D: 0,25%) and/or university (M: 8,4%). 2 The list presented included: Violin, Viola, Cello, Double Bass, Bass Guitar, Acoustic Guitar, Electric Guitar, Piano, Harp/Harpsichord, Organ, Synthesizer/Keyboard, Computer, Recorder, Flute, Oboe, Clarinet, Saxophone, Trumpet, Trombone, French Horn, Tuba, Drums, Percussion, Vocals, Groovebox, or other (full results in appendix 2)
93
Concerning dance lessons, respondents were additionally asked how long they had taken classes (with about 40% having taken over a year’s worth of lessons), what formal level they achieved and how they evaluated their own ability. The next section of the questionnaire probed the cultural and musical preferences of the target population. Concerning cultural preferences, respondents were asked about their appreciation (on a seven-point Likert-scale) and average attendance frequency3 regarding a number of broadly defined cultural genres, namely “cinema”, “opera”, “dance performance”, “musical”, “theater”, “comedy”, “classical concert, “jazz concert”, “pop concert”, “music festival”, “museum (permanent collection)”, “new media exhibition/demonstration/performance”, “art exhibition (temporary collections)” and “lecture”. Concerning musical preferences, respondents were asked to indicate how much they liked listening (on a seven-point Likert-scale) to “Classical Music”, “Pop music”, “Dance”, “Hip Hop / Rap”, “Electronic Music”, “Gothic”, “Country”, “Soul/R&B”, “Jazz”, “World Music”, “Rock”, “Latin”, “Metal”, “Folk”, “Alternative”, “Blues” and “Soundtrack”. Also, they were asked to specify with an indicative relative frequency4 how often they actively listened to said genres. Moreover, respondents were requested to specify a favorite music genre and asked to estimate how many hours per week they spent both actively and passively involved with music. To this, answers included “none” (A: 8,1 % / P: 1,6 %), “between 1 and 5 hours” (A: 40,1 % / P: 12,9 %), “between 6 and 10 hours” (A: 21,3 % / P: 14,7 %), between 11 and 15 hours (A: 14,2 % / P: 15,5 %), “between 16 and 20” (A: 8,1 % / P: 15,5 %), or “more than 20 hours per week” (A: 7,9 % / P: 39,9 %). The fourth and fifth section of the questionnaire dealt with movement – as a less conscious predominantly passive-listening – response to music, and dance – as the deliberate active-listening equivalent. As far as “moving in response to music” goes, only 1% of the respondents says never to engage in this reflex, whereas about two thirds said to do it frequently; for “dance in response to music”, the threshold appeared to be a bit higher, but also here, it appeared to be a well-accepted, widespread phenomenon. The response to these questions appears to hint at the universal applicability of an embodied music cognition paradigm – as the musical stimulus effectively instantiates a bodily response in the majority of respondents.
3
Average frequency options: [1] never, [2] less than once a year, [3] approximately once per year, [4] more than once per year but less than once per month, [5] approximately once per month, [6] more than once per month, [7] approximately once per week, and [8] more than once per week 4 Relative frequency options: [1] never, [2] rarely, [3] occasionally, [4] frequently, [5] all the time
94
Then, the (relative frequency4 of the) occurrence of both types of bodily reactions to musical stimuli were enquired further into, in terms of typology and intensity of motion, repetition and focus on specific body parts. In terms of location, the social preferences5 and situations6 in which movement occurred as a response to music were inquested, and specific to conscious dancing, the triggering mechanisms/incentives7 as well as the motivation/reason8 to keep engaged in the activity. Related to the themes of dance and motion, the sixth section then zoned in on the stylistic preferences for movement and dance styles. First, the types of dance styles9 respondents potentially partake in were scrutinized in terms of enjoyment (on a 7-point Likert scale) and occurrence (in terms of relative frequency4). Then, dance preferences were linked to the music styles discussed in section three of the questionnaire (cf. supra) and also assessed in terms of enjoyment and occurrence. The seventh and last questionnaire section centered around the use of and knowledge about computational technologies. First, active involvement with modern day technologies was under scrutiny, namely average amount of non-work computer use, with half a percent indicating not being involved at all, 17% keeping computer usage limited to one hour per day, 54,9% of respondents averaging between 1 and 3 hours daily, 21,8% reaching between 4 and 6 hours per day, and 5,8% surpassing the 6 hours per day. Next, a number of specific computer uses were probed, namely use of the internet (98,4%), gaming (19,2%), graphic (17,9%) and music applications (26,5%). Also, the use of a
5
Social preferences for dancing/moving in response to music: [1] alone, [2] with your partner, [3] with a dance-partner, [4] with a group of friends, [5] with anybody 6 Situational or locative preferences for dance/move: [1] at a party (e.g. with friends, family, ...), [2] at a music concert (as a member of the audience), [3] at a public dance event (e.g. a techno party), [4] at home, [5] in a bar, [6] in a dance school, [7] in the car, bus, train, ..., [8] during work, [9] during shopping, [10] during working out 7 Trigger or incentive to engage in dancing: [1] a particular piece of music you like very much, [2] a certain style of music you like very much, [3] a certain style of music you particularly like dancing to, [4] a dance style you particularly like, [5] other people that inspire/invite you to dance, [6] the fact that you simply like to dance 8 Motivation for staying on the dance floor: [1] a particular piece of music you like very much, [2] a certain style of music you like very much, [3] a certain style of music you particularly like dancing to, [4] a dance style you particularly like, [5] other people that inspire/invite you to dance, [6] the simple fact that you like to dance 9 Dance styles: [1] Social/Club Dance (like in a disco), [2] Participation Dance (e.g. "Conga line"), [3] Ceremonial Dance, [4] Folk Dance, [5] Ballroom Dance, [6] Latin Dance, [7] Swing Dance, [8] Street Dance (e.g. "Break Dance"), [9] Performance Dance (e.g. "ballet"), [10] Novelty/Fad Dance (e.g. "Macarena"), [11] Free Dance, [12] Competitive Dance
95
number of computer-related appliances was investigated (e.g. USB, Bluetooth, FireWire and external gaming devices, mp3-players, digital (photo)cameras or smartphones).10 Next, approximate use frequency11 and purpose12 of free-time internet usage were asked about, as well as the type of internet connection available to the respondents at their homes. Also, affinity with games and gaming was examined, namely the manner in which games were accessed13 as well as the types of games14 that were played regularly. The penultimate set of questions delved into attitudes towards new (digital) technologies and the acquisition thereof (i.e. comparable to profiles like early adopters, lead-users, etc.) with regards to new technologies.
Figure 5.01: Pie chart showing attitudes for technology-enabled artistic interaction in a cultural context.
10
Given the fact that only 6,8% of respondents indicated to ever have used smartphones at the time, it should come as ample proof that the underlying results have become a bit outdated, and that digital evolutions take place at an ever-increasing pace. For reference, smartphones appeared first in Digimeter 2010, clocking in at 23,5%, and have since steadily evolved to 73,9% in Digimeter 2016 (De Marez et al., 2010-2016). 11 Approximate free-time usage of the internet: [1] Monthly, [2] Weekly, [3] Almost daily, [4] Less than 1 hour a day, [5] Less than 3 hours a day, [6] More than 3 hours per day, [7] More than 6 hours a day 12 Potential purpose for internet usage: [1] E-mail, [2] Internet-banking, [3] Online Gaming, [4] Blogs, [5] Online shopping, [6] Downloads (music/movies), [7] Online work (VPN), [8] Chatting, [9] Browsing, [10] Online social networks 13 Manner computer games are accessed in: [1] In the computer’s operating system, [2] Using a CD-Rom/DVD, [3] Online games, [4] Combination of local and online, [5] Local Area Network (LAN), [5] Using a console 14 Types of games: [1] Strategy games, [2] First person shooter games, [3] Racing games, [4] Knowledge games, [5] Music games, [6] Fantasy games, [7] Social Games / Party Games, [8] Flash games (on the internet), [9] Retro-games, [10] Games in the operating system of your computer, [11] Sports games, [12] Action/Adventuregames, [13] Role playing games, [14] Multi-player online games, [15] Simulators, [16] Fighting games (16)
96
And lastly, the final part of the survey probed attitudes towards audience participation in cultural events, and the preferential way in which respondents would like to do this. Bearing a lot of importance to this project, the results indicate that not even young people were particularly eager for technology-enabled artistic interaction or culture-participation at the time of the interrogation. Concerning participation in artistic context, results suggested only 10% to be sincerely enthusiastic about the prospect of technology-enabled interaction. About 40% of the participants said not be opposed to the idea, but had reservations with the limitations of audience participation. Essentially, close to half the respondents had a hesitant or downright negative attitude towards it (figure 501).
5.5 Interpretation and discussion Summarizing, of all respondents, close to 70% were female. Respondents under 35 years old made up 85% of the total, of which 67% were aged between 16 and 25. Over half the respondents were still studying, most of which at a bachelor or master level, so in combination with the results of the question concerning the highest obtained degree, it can be assessed that the respondents were overall highly educated. Over 60% of the respondents had received a formal music education, of which over 40% more than four years. About half the respondents actively played musical instruments, and about 20% of all respondents created and produced music of their own. In an overview of receptive cultural events, the overall outcome was that nearly all of the respondents engaged in cultural activities about once a month, with the emphasis on activities such as musea, pop concerts and theater performance and a smaller representation for activities such as opera, classical concerts and dance performances. About half the respondents said to attend a new media performance or exhibition at least once a year. Practically all respondents were regularly engaged with a computer (most also with other digital devices). Moreover, aside from work or studies, 71% spent more than an hour per day involved with a computer or laptop. A quarter of the respondents also said to be actively involved with computer-based music applications, a fifth of the respondents used applications for graphic design, and equally, one fifth of the respondents regularly engaged in computer games. Concerning the attested profiles in new media technologies and the respondents’ attitudes towards innovation, about 10% of the respondents considered themselves to be highly enthusiastic lead-users, whereas about a third of the respondents took an indifferent or negative stance towards new technologies. In terms of technology acquisition, less than 10% were early adopters and close to half of the 97
respondents reported to generally wait quite long before they (wanted to) adopt a new technology. All of these and other results pertaining to the full descriptive analyses and the associated graphs of the full questionnaire are presented at length and in more detail in Appendix 2. At the time of execution, this research project was not yet sufficiently apprised of longitudinal inquiries such as the PaS ’04 & ‘09, the Apestaartjaren studies, the multiple waves of the Digimeter, the SCV-survey or the MICT-gaming survey. Rather than availing itself of the methodologies and data of these rather prestigious but also (fairly) expensive studies – some of which were instantiated roughly around the same time of this questionnaire, some predating it by several years – the WMY-survey purposefully elected to cover the selected fields in one all-encompassing study. Needless to say, that such an ambitious, comprehensive and convoluted attempt was inherently set-up for (at least partial) failure. As such, the tool itself might have yielded good results, but the survey was too broad in extent, large in scope and missed direction, in the sense that it aimed at too many things at once, and executed none of them particularly well. Basically, it investigated elements that should have been covered in five differently geared and properly delineated questionnaires. As its prime intended purpose was to recruit prospective technology-users and the application domains were – at the time of launch – insufficiently concrete and still adequately undefined (i.e. the Emcometecca-project might still have ventured into several different directions, e.g. the development of music games, search engines for music, DAWs for music production, tools for musical sports applications, performances employing interactive music technology, development of educational music applications and/or fundamental research on music and movement), the strategy is essentially justifiable, even though the execution was flawed and yielded only little of the aspired results. Even though eventually most of these research topics would resurface in the “SoundField”-project (covered in Chapter 9), they could not be properly nor exhaustively dealt with within the encompassment of one questionnaire. It probably would also have been interesting to perform multivariate analyses on some of the variables, though solely from the viewpoint of fundamental and/or methodological research (e.g. how results obtained for “type of future research in which respondents would like to partake” correlate to the “way in which they would like to technologically interact”, juxtaposed with some of the categorized independent variables from the sociodemographic section). This was dispensed with, as it would constitute great additional efforts – in an already much delayed project, but was essentially of very little consequence to the rest of the thesis. The obvious problems being the unrepresentative scope of the sample, unsuitable for broad extrapolations concerning the general population that prevent further interpretation of more intricate statistical inference (beyond the
98
descriptive analyses presented), as it would contain little more than rough and unsubstantiated conjectures at best. Additionally, there is the fact to consider that such a fully annotated database and pool of potential test subjects had no scientific or real-life impetus anymore, as in effect no further experimentation was planned off the basis of the WMY-survey once it was completed. This is due to the fact that it was eventually not used as the recruiting tool that it was initially designed to be in the broader Emcometecca-context. A few other research projects (i.c. the D-Jogger use case and a music perception experiment revolving around the impact of the bass drum on dance movement) availed themselves the mailing list obtained by the study, but had very little interest in the questionnaire results themselves. And so, as even the underlying research project was moving in an altogether different direction, further analysis was dispensed with. Finally, even though it was no longer of any particular use for the rest of the project, a methodological consideration can be deduced from the execution of this survey. Though it did not deliver the anticipated results within the scope of the larger project, a similar questionnaire – using a drastically reduced list of more attuned questions and topics, running on the same tried and tested open source platform (i.e. LimeSurvey) was implemented in other, comparable projects (i.c. Sound Caterpillar) with more promising results, proving the chosen approach to be not 100% properly executed, but essentially suitable and effective.
5.6 Conclusion This survey and the execution thereof certainly has substantial merits, but also some quite severe limitations and shortcomings. On the one hand, it covers and offers information on most of the socio-demographic and relevant action-oriented ontological attributes prospective EMMT-users might possess, on the other hand, in respect to an assessment of the broad scope of the cultural industries, it doesn’t render a representative image of the prospective audiences, due to the insufficiently large sample and additionally the inadequate sampling across various age groups. This makes all results obtained somewhat ambiguous to interpret, and any of the potential claims prone to considerable reservations. In respect to the Emcometecca project, the survey could have been an apt medium for recruiting well-documented test subjects for experiments, and prospective users could have been contacted for specific studies based on any number of parameters out of the resulting database. The questionnaire itself bears no blame for the fact that no such continuation was in the eventual planning of the project anymore.
99
The results, in sum, zone in on a predominantly young sample of highly educated culture-enthusiasts, most of which had received some form of artistic training and most of which had demonstrable media skills, were asked about their opinions concerning technological advances in a receptive cultural context and the possibility of technologyassisted participation. The majority of respondents expressed concerns or were overtly unenthusiastic about that prospect. In this chapter, focus was on the properties encountered in future EMMT-users, more precisely, on what impact musical background, training, preferences, necessary capacities and relevant convictions may have on the development and functioning of prospective AI&VREs and EMMTs. The following chapter deals less with the internal properties and motives, but looks towards the prevalent perceptions of culture participants confronted with actual EMMTs in a receptive environment.
100
6 Audience perception of EMMTs: case studies with HaaO & Lament
As much as the previous study strived to ascertain user types and some of the more general opinions of the eventual end-users as a means of understanding the audiences' broad attitudes towards new technologies, it was an inadequate instrument to probe the genuine reactions end-users may have when being confronted with an actual EMMTsetup. An online survey simply cannot suffice to understand of how audiences respond to real-life examples of interactive art systems. Therefore, while the questionnaire was launched to make a survey of a broad scope of potential end-users of cultural embodied technologies, two smaller exploratory in situ tests were performed, focusing on the genuine experience of culture-enthusiasts when interacting with live artistic installations. In this chapter, two consecutive tests are described that were done with different artistic installations of dr. Pieter Coussement (Coussement et al., 2008; Coussement et al., 2010).
6.1 Heart as an Ocean The first installation, “Heart as an Ocean” (figure 6.01), is a new media piece that is based on the artistic use of participants' auditory senses and their bodily response to sound (Coussement et al., 2008). It was meant to be experienced individually and drew on a strong action-perception-coupling between the user and the installation, which was established using the biometric feedback-loop of an EEG-signal, sonified as waves crashing onto a shore. The artist’s hypothesis was that people would become more at ease by listening to the sound, which was a direct consequence of their own brain and heart activity. In the artist's view, ‘The Heart as an Ocean’ was conceived not only as a means to communicate between artist and public, but also to communicate on a broader social level among the public itself. In a broader context, exhibiting with ‘The Heart as an
101
Ocean’ also functioned as an experimental setting in which new forms of interactivity were explored, more particularly, in the context of media installations and with new sensor technologies. The media piece explored the fundamentals of meaningful interaction by examining to what extent the physiology of the human body can be both sensor and actuator in an art context.
Figure 6.01: Picture showing “Heart as an Ocean” (HaaO), an artistic installation based on action-perception coupling using biometric feedback.
As the media piece was designed to reflect the state of mind of any person who interacts with the installation, it was possible to develop an individually responsive installation that would also allow researchers to produce measurable outcomes. To complement the objective measurements, a semi-fixed interview was prepared to evaluate user-experience within the context of the artwork, immediately after interaction. In addition to standard questions on demographic information and cultural background, the questions relevant to the installation and the participants' experience dealt with 1) comprehension, 2) perceived purpose, 3) evaluation, 4) operation and 5) sensory perception within the installation. Since there was no definable task, usability metrics and heuristics were not applicable in this context. Even though most of the participants were quite willing to partake in the interviews, the results were relatively inconclusive. This was due to three reasons. The first reason is that the questions that were asked did not befit the interview technique. The questions assumed a quite unambiguous response, yet because of the interview technique, respondents quite often took great liberty to nuance their responses, made analogies and quite often responded only in part to the questions that were asked. A second reason can be found in the unencumbered atmosphere brought about by the artistic context, where people would walk out on the interview, half way through. And the third and last reason can be found in the fact that, given the individual nature of the interaction with the installation, it was apparently quite awkward for a considerable number of the interrogated to report on their personal experiences. These 102
issues demonstrate how difficult it was to successfully gather relevant user feedback in the context of a finished art piece and within a non-secluded art environment.
6.2 Lament Unlike the previous installation, the second installation, “Lament” (figure 6.02), was not designed to be experienced individually, but provoke participants to actively and collaboratively experience a sonic environment. As an iconographic identity, the installation consisted out of five megaphones, suspended from the ceiling, spread throughout a hall, and modified to suite the purpose of for the installation. In designing the visual identity of this installation, a minimalistic approach was taken, where everything that was visible, had a clear and understandable function. Additionally, all associations made when thinking of megaphones, favor interaction. Because of this, the public could easily connect to the installation, albeit from a more outside position of the installation, since the megaphone microphones pointed outwards.
Figure 6.02: Picture showing the “Lament” installation @ Music center Bijloke, Ghent.
The auditory identity, however, has a somewhat different, more ambiguous nature. When entering the room, the audible content is not directly connected to the visual. There appears to be only a subtle singing sound, which has a very distinct tone quality and gives a peaceful impression. Although very monotonous in nature, the sound is continuously changing, both in volume and timbre, in response to the public moving through the space. Each of the megaphones reacted individually to what was nearest to them and responded 103
to the sound the public made by softly ‘singing’. When a participant spoke directly into one of the megaphones, the preset amplitude threshold was crossed, triggering a second level, in which that particular megaphone ‘sung’ out loud. Each megaphone had an individual voice, which led to participants investigating the different megaphones. The whole installation acted as a macro-organism engaging with the environment by listening to the surrounding sound, and to attract participants, the megaphones' affordance (its function is immediately clear from its form) was utilized. With its detached nature in reference to the visual identity, this is catalytic for a more involved interaction from the public. The controls on the megaphones were overridden and the amplification and volume control of both the input (microphones) and output (speakers) was controlled by Max/MSP-software, using the finite state machines paradigm (Coussement et al., 2010). Next to the installation, a computer terminal enabled the public to report on their appraisal of the artwork and to evaluate their experience by means of an online questionnaire. A first set of questions pertained to the participants’ awareness of the installation, the functional and aesthetic properties of the artwork, the responsiveness, the interactive nature of the art-piece and the assessment of the interaction. The second part of the questionnaire asked about demographic information, cultural activity and background, and specific familiarity with new media and installation art. In one week time, approximately 4000 visitors attended the exhibition at music center Bijloke in Ghent, Belgium. Conversely, no more than fifty attendees took the time to fill out the questionnaire. This made is quite apparent that very few people were interested to spontaneously comment on their experiences, even though the possibility to do so had been made clear by means of posters and signs put up in the exhibition space.
6.3 Conclusions Considering the outcome of these pilot studies, no indispensable scientific information was gained from either of the events. However, methodologically, both studies had significant merits into research on artistic interactive applications. In the first instance, the information that was expected, was required to be concise and precise. Conversely, the experience-based feedback that was gathered was elaborate, rich and colorful. With the aim of reporting back to the artist on the experiences with the interactive installation of the participants, this was valuable feedback. Alternately, for extracting broad assessments on the global experience of participants interacting with the installation, it proved to be ill suited.
104
In the second instance, different problems arose. Using a formalized online questionnaire left less to chance, yet this time, it was the level of response was the major issue. Even though well-advertised, the questionnaire only got through to a limited number of very dedicated culture-enthusiasts. The majority of the festival-visitors to whom "Lament" was presented, didn't care enough to make the commitment of filling out the questionnaire. To some extent, this confirms some of the earlier findings of the "What Moves You"-questionnaire, namely that, inconsiderate of age, an even a more than averagely culturally active audience is not (yet) particularly eager to interact with interactive installations or new interfaces for musical expression. Moreover, from the limited number of questionnaires that were completed, the questions pertaining to additions and changes that could be made to the setup, remained largely unanswered. This in turn indicated that, inversely to the previous test, the incorrect interrogation technique was used to obtain the desired information. In both instances, it was clear that merely exhibiting a finished art piece in a museum context was not an efficient way of conducting research on interactivity, or of further developing a new interface-based artistic installation. Additionally, the above experiments illustrate that, concerning interrogation of culture-enthusiasts, interview and questionnaire strategies are complementary. Both are indispensable to get parts of the feedback that is vital to inspire further development, but the inquiry form is key in view of required information. In sum, this chapter dealt with the not so straightforward and perhaps not so amiable reception and perception of EMMT-based art installations in an ecological setting, revealing and exemplifying such issues like the opacity of affordances of new and unstable media art. The next chapter will discuss another research stage, partially overlapping in timing, that gives a more successful example of an interface evaluation study (than the example given in Chapter 4), namely the iterative design study centered around the "Synch-in-Team" application.
105
7 Interface Evaluations using the "Sync-in-Team" device
This chapter takes a step away from the factual (cf. Chapter 5) and contextual (cf. Chapter 6) considerations concerning EMMT-use, and delves back into an interface evaluation study revolving around the “Sync-in-Team”-application (Deweppe et al., 2009; Leman et al., 2009; Leman et al., 2010). It constitutes the first serious attempt at combining some of the experiences discussed in Chapter 4 and to make a more systematic user-experience evaluation of an application under development during multiple experiment and demonstration sessions of a collaborative music game. More specifically, the research aimed at encompassing how prospective users could and wanted to use the technology, how well they experienced control and what they appreciated about the prototype. Within a system that had a fixed purpose but a relatively flexible setup, the goal was 1) to evaluate the properties and qualities of the presented setup, 2) to gain insight in the shortcomings participants felt the setup had, and 3) to gather feedback on changes and additions that should be made.
Figure 7.01: Software screenshot of the Max/MSP-patch controlling “Musical Synchrotron”.
The method that was applied to record these user-evaluations, was a structured interview technique that relies on a pre-established set of questions. A random set of participants, 107
who played the prototype of the music game, was requested to take part in such an interview. These interviews were recorded on video, so that the entire feedback could be processed afterwards. The objective of this study was to establish how well the feedback could be recorded and applied to the improvement of a given iteration of the setup. In this project, the combination of interviews and discourse analysis (Talja, 1999) thereof were investigated, to assess how well they were suited to obtain relevant user-feedback of the participants of the setup, in this particular case, the “Sync-in-Team” collaborative music game (Leman et al., 2009; Seif El-Nasr et al., 2010). The approach probed how well feedback regarding this application could be recorded and applied in future developments and what additional user-related methods should be merged with it to obtain a strategy that would allow embedding of mediation technologies based on embodied music cognition within existing cultural contexts. In this study, the problem at hand is addressed by focusing on the users who under different conditions and on different locations played the social music game “Sync-inTeam”, based on the “Musical Synchrotron” (Demey, Leman, De Bruyn, Bossuyt & Vanfleteren, 2008), which was also developed at IPEM (Figure 7.01).
7.1 Setup Description The setup that was used was a collaborative music game, in which participants were divided into two competing teams of two players. By dancing synchronously with the musical beat and with their team partner, they controlled a visual feedback, projected on the ground.
Figure 7.02: Picture demonstrating the basic operation of the “SIT”-setup.
108
The participants wore a belt onto which a wireless motion sensor was attached. Before the game started, participants were instructed to dance to the beat of the music and simultaneously try to synchronize with the other team member. Every time the game was played, five forty-second music excerpts were played as the musical stimulus the participants had to synchronize with. In the basic setup, the five excerpts were taken from relatively unknown electronic music songs. More music sets were introduced to the game in the following setups. The acceleration of each participant’s dancing was measured with a sensor they wore on the hip. These measurements controlled the visualization of the synchronization level (Figure 7.02).
Figure 7.03: Close-up picture of the visualization presented to the participants.
To determine team players’ synchronicity, the tempo was extracted using an FFT and was compared with the BPM of the music excerpts. The extracted tempos were evaluated and resulted in an increase (when synchronized) or decrease (when not synchronized) of the global score of a team. That score was visualized by the projection on the dance floor (Kaiser, Ekblad & Broling, 2007). The visual feedback (Figure 7.03) consisted of two colored dots. Whenever a team performed well, the colored dot of that team expanded. When a team did not synchronize well to the music and/or to one another, the dot that represented their score decreased to its original size and shape. So, the precision they performed the set task with, translated into the visualization. The team that performed the task best over a short span of time, got the bigger visualization and the team that performed the task best throughout the entire game, won.
109
7.2 Experiment locations and setup modifications The experiments the music game was tested in, were executed in four stages and four different settings, which also implies four different interview situations. Each experiment had the same basic components as described earlier, although the particular setting and the state of the art of the development of the game yielded slightly different conditions. A first series of experiments was done in lab setting at Ghent University, over the course of three days. A basic structure of the game with two conditions was used and evaluated. First, in an individual condition (De Bruyn, Leman, Demey, Desmet & Moelants, 2008), the teams were given the chance to experiment in the music game with their teammate and in the second social condition, the teams played in competition with each other. In this early developmental stage, the visual feedback was projected on the floor and the dots representing the teams remained stationary. Also, only the aforementioned set of five musical excerpts was used in both the individual and the social session. Nearly all participants for this series of experiments were university students.
Figure 7.04: Picture showing participants playing SIT @ the 2008 IBBT open house, Ghent.
For the second series of experiments, the game was moved to the then Institute of Broadband Technology (IBBT), currently iMinds, in Ghent for a public open house day. The introduction round (i.e. individual condition) was omitted from the course of the experiment/game, the visualization was projected on the floor in its definitive, dynamic form and the participants were given the opportunity to choose from different sets of musical excerpts. The sets the participants could choose from were electronic music, Latin, classical music, hip hop, RnB and rock music. Their choice could be made either to
110
improve their own score or to negatively influence the performance of the other team. For this second series of experiments, every game contained two sets of five music excerpts in the genres chosen by the two teams and presented to them in a random order. The length of the excerpts was reduced to approximately thirty seconds per excerpt. This series of experiments had the largest demographical variation, as the participants were recruited from the visitors of the open house day (figure 7.04).
Figure 7.05: Picture showing participants playing SIT @ the 2008 Science Fair, Mechelen.
The third series of experiments took place at a three-day Science Fair, (Dutch: Wetenschapsfeest) biannually organized in Mechelen by the Flemish Government. This fair typically targets children and adolescents, but also adults (parents). For this occasion, a set of children-songs was added to the list of music categories to choose from and the number of sets was doubled, so that the teams could also choose to compete in the same music style. Aside from that, no fundamental changes were made to the setup of the game (figure 7.05).
111
The last series of experiments was done at NEXT08, a two-day national gaming convention in Brussels. In that setup the feedback was projected on a screen, and more music sets were made available for the participants to choose from (figure 7.06).
Figure 7.06: Picture showing participants playing SIT @ NEXT’08 game convention, Ghent.
7.3 Interview circumstances Given the different types of locations (e.g. lab versus semi-public and full-public places), efforts were done to stay as close as possible to the basic predetermined structure of the interviews. For the first interviews, carried out at the IPEM lab, we had the opportunity to interview all of the participants individually and with a more semi-structured approach, so that it was possible to elaborate and ask follow-up questions based on some of the answers given by the respondents.
Figure 7.07: Collage of the different interview settings
112
During the second stage, that is the experiments at the IBBT, a random selection of both winning and losing dance-couples was invited to give feedback on their experience with the game and on their performance. Due to time limitations, teams were interviewed instead of individual participants. Therefore, it was necessary to more rigidly adhere to the predetermined interview structure. The interviews at the third location, the Science Fair, were done in a manner similar to the one utilized at the IBBT, with the only exception that large percentages of the public (and by consequence of the participants) were children and adolescents. At NEXT08, the noisy environment made it impossible to conduct interviews. Some participants did evaluate changes made to the game and setup, but this feedback could not be recorded (figure 7.07).
7.4 Interview Data The central objectives of the questions asked concerned the users’ experience of affordance (Gibson, 1979), flow (Csikszentmihalyi, 1990) and presence (Biocca, Jin & Choi, 2001). Specifically for this game, questions were incorporated to investigate the usability (Nielssen, 1994 & Bevan, Kirakowski & Maissel, 1991), elements of goal composition (Nielsen, 1994) and game enjoyment (Ijsselsteijn et al, 2008). The questionnaire was designed to obtain as much nuanced information as possible: questions were open-ended and a ‘yes’ or ‘no’ answer was not an option. A first section investigated how clear the objective of the game was to the participants and how successful their used strategy was. To establish how well the participants understood the set task, they were asked to describe the task they had to perform to achieve the goal of the game. A second set of questions investigated how hard the participants had to concentrate to perform the task and how they experienced control over the visualization. In the third section of the questionnaire, participants were asked to evaluate the musical and visual features of the game both aesthetically and as a means to play the game. Additionally, the participants were asked whether they could think of suggestions, changes or additions that would make the game more appealing to them. Other questions dealt with the creativity in the performed task, what type of movements achieved the best results, if there was an evolution in their performance throughout the game, what elements the participants focused on most, what strategies were used to improve the team’s performance and eliminate the competition. Finally, some questions were asked about the making and correcting of errors, the challenge the participants experienced and the gratification (or frustration) they felt after playing the 113
game. Some of these questions had to be taken out of the interview structure after the first series of interviews due to time restraints; others were dropped because the response was already implicitly present in the answers provided to some of the other questions. When respondents had answered multiple items to one question, they were asked to specify one response; in most cases, this was the first and most spontaneous answer they had provided. The video-registration files of the first three locations were processed into accurate transcriptions of the interviews. After the transcription, the results from the interviews were paraphrased, catalogued and quantified for statistical analysis.
7.5 Analysis and Results Forty-five interviews were carried out at the first three locations, of which two could not be used due to technical difficulties with the recording equipment or problems with the execution of the experiment, so the discourse analysis was performed on forty-three interviews. Approximately one third of the interviews were done at each of the locations and thirty-five percent of the interviews were done with individual participants, the others were team interviews. The team interviews were treated in the same way as the individual ones. Of all the respondents, half had won the game, half had lost.
Figure 7.08: Cross tabulation graph of comprehension vs. ability to describe set task.
114
In ninety-five percent of the interviews, the goal of the game was said to be (very) clear, and this was confirmed by the results of letting the participants describe the task they had to perform (although during the interviews, sometimes the task was confounded with the used strategy). In only twenty percent of the cases, respondents stated to have needed very little attention to perform the relatively easy task and for nearly half (46,5%) of the interviewed, performing the task required their constant attention. In approximately seventy percent of the interviews, respondents experienced a good control most of the time over their visualization, whereas twenty-three percent experienced little and seven percent of the participants experienced no control at all. A chi square-test showed a significant difference between the participants' level of experienced control over the game, by means of the resulting visualization, and whether they had been able to describe the set task (figure 7.08 and 7.09). The explanation for this could be that, although most subjects understood the goal of the game and the set task rather well, they were not able to describe –let alone– perform the task properly, hence did not get a good visualization. Moreover, no such significant relation could be found between the chosen music genres or the outcome of the game (winning/losing). Figure 7.09: Table showing the results of the 2-analysis.
All of the interviewed stated to have enjoyed the visualization, so it was not possible to see a rise in the appreciation by changing the visualization from stationary to orbiting. The participants were already enthusiastic about this aspect of the game in the first stage of the development, and in the later stages, they had no grounds for comparison. However, in nearly a third of the cases, respondents did not find the visualization sufficiently clear as a feedback on their performance. In three of the interviews, the respondents suggested to incorporate an actual score into the game. From the evaluation of the changes made to the game setup based on the participants’ feedback, we could see a significant relation between introducing choice of music genres into the game and the appreciation for the music to which subjects had to dance. This relation could not be demonstrated between the three different setups (with the increased choice of music between the second and the third setup). The question whether the music 115
was well suited as a tool to perform the task, did not provide conclusive data either; the reason for this could be either related to the fact that the participants had to agree upon a certain music style to play the game as a team or the fact the teams also had to compete in the music style the other team had chosen.
7.6 Evaluation The feedback we recorded allowed us to further improve the setup and to verify whether the changes that were made, had improved the game or not. This process can easily be reiterated in a structured way and with the incorporation of some of the suggested changes and additions to the setup, it can result in a more user-friendly setup and a more enjoyable game. More importantly, these suggestions give us a clear indication of what participants would like this technology to do. In this particular case, even though the respondents were generally quite enthusiastic about the game, in all but fourteen percent of the interviews participants did suggest changes that would make the game more enjoyable to them. Only a quarter of the respondents could not think of any further additions that could be made to further improve the game. Additionally, only twelve different answers were recorded to both these questions, which implies that large sections of the interviewed had the same view on how the game should be improved. In a third of the interviews, respondents confided that they would like to play this game with more (teams of) contestants (Feldmeier & Paradiso, 2007). In more than thirty percent of the interviews the suggestion was made to incorporate more (realistic) light-effects in the setup. These answers do not only give insight into on how participants evaluated the game, but they also indicate that, as a number of participants apparently did not perceive the experience to be realistic enough, what environment the game should be set in. Although the results of this approach still only apply to the actual setup, they give a good idea of what could make the game more enjoyable and realistic. Overall, using interviews to record feedback about a given application is a relatively cost-effective way to pinpoint what things that are lacking from it in an early stage of the development, to find out what changes and additions should be taken into account for improvement. Yet in a more advanced stage of development, the interview-strategy becomes too time-consuming and the advantage of a flexible interview-structure, which allows for follow-up questions, diminishes the more crystallized and application becomes. Moreover, for researchers working on the hard- and software development of embodied music mediation technology, the presented approach can hardly be considered a solid foundation to base more fine-grained modifications on. Other tests (e.g. user profiling, 116
usability heuristics and metrics) are required to give more attuned results once the preliminary stages in the development are over. Another shortcoming with using the described strategy as a developmental guideline is that it only allows evaluations to be made on an existing system and therefore it cannot take other prospective participants or different possible implementation context into account.
7.7 Conclusion Using a selection of heuristics adapted from and interaction design methodology, some initial problems with the tool’s operation (relating to the algorithms, hardware and visualizations) could be located and remedied. Also, initial inquiries were made that dealt with the properties of different input devices in order to establish the best use of this technology. After the initial design and evaluation stage, further usability testing was done in combination with actual participant testing of the application. A first set of tests was done in the lab, after which the game was presented to the general public during three events. The users’ interaction was recorded on video and after playing the game, a random selection of people that had interacted with the gameprototype, were interviewed. The interviews consisted of a pre-established set of questions that were presented after the game to the contestants in a semi-structured way. The topics that were addressed in this investigation related to the users’ experience of affordance, usability and game enjoyment (Ijsselsteijn et al., 2008). The interviews investigated how transparent the objective of the game was to the participants and how successful their used strategy was. Furthermore, the participants were tested on how well they understood set tasks, assessed strategies and goals, and how easily they could achieve them. The participants were also asked questions to probe the level of concentration required to perform the task, the level of control they experienced, and to evaluate the musical and visual features of the game, both aesthetically and functionally. Follow-up questions dealt with the creativity, focusing on the representation, gaming strategies, making and correcting of mistakes, the amount of challenge that was experienced, and the gratification (or frustration) they felt after playing the game. Finally the respondents were asked to make recommendations for changes or additions to make the game more appealing. To improve the affordance and usability of the application, an interface evaluation test was performed with three different input devices during the public events, where the application was shown to establish what interface had the best properties and yielded the best control. The user-experience evaluation of the prototype was an apt means to address flaws and strengths in the design, all of them pointed out by users, as users were given an 117
active role in the design process. The improvement of a setup thus became an integral part of this iterative feedback design process. Additionally, this study showed that the constructive implementation of user-feedback, in parallel with user-evaluation research was very useful in aiding the development of interactive or collaborative embodied music gaming applications from an early stage onwards. Finally, this methodology proved to be sufficiently flexible for it to be adapted towards an evaluation of comparable applications. To recapitulate, the Sync-in-Team was a successful test for doing troubleshooting on an already largely finished interface and a good Interface Evaluation test. This benchmark test was a good starting ground for further developing the user-oriented work package of the EMCOMETECCA-project, yet its findings were relatively limited, as they were confined to the setups used in this specific study. The three instantiations of the setup, though they can be considered “iterations”, remained quite sequential in nature. Still, throughout the iterations, a largely comparable/identical technical setup was used, rendering comparable results. Also in this case, in the results a proneness to the stimuli used in the setup can be distinguished, aside from the interfaces being compared. So, in conclusion, this study had a valuable outcome. As the results of the interview analysis could be put to good use, combining it with other user-centered methodologies and other possible contexts for embedding, would greatly benefit and accelerate future development of embodied music applications. However, development of contextdependent applications cannot dispense with good insights into prospective and eventual users. There is a real need to combine this user- and context-oriented development process with sensible and thorough usability testing by well-documented users. The incorporation of their feedback into the development process is a laborious approach, which is however more likely to deliver readily applicable findings and more goal-attuned interfaces. Therefore, this is an essential part of the strategy to obtain plausible and realistic implementation scenarios to allow cultural embedding of embodied music applications and ensure the ecological viability. As the previous chapter predominantly discussed the ad hoc assessment of finished art pieces, and the chapters before that focused either on the evaluation of the interfaces themselves, or on the crucial properties of the prospective users, this chapter zoned in on 1) the importance of interim “end” user evaluations of a working prototype, 2) the necessity of an iterative aspect of the design process, and 3) the importance of flexibility concerning the implementation context. The changes that were being implemented between the different test cycles and that were a result of these three key aspects, all proved to be mostly beneficial to the eventual design of the system and further illustrated the importance of a thoroughly thought through development and evaluation procedure. The following chapter follows up on this line of sequential development and involves itself with the methods associated with HCI. However, this development occurs in the
118
context of a user-centered, and – more importantly – a lab-based control room. The ensuing experiment pertains to a technology that partially instigated the “SoundField”platform, yet from its early conception, the system was only tangentially related to genuinely artistic practice, and zones in on 3D virtual object mediation. Within the confinement of this thesis, it rendered the perfect opportunity for testing a number of HCImetrics and heuristics to be explored.
119
8 Usability Heuristics and Metrics: the Virtual String Study & the Multi-Level Sonification Tool
The studies organized in combination with the exposition of “The Heart as an Ocean” and “Lament” were valuable, but as the presented installations were to a high degree completed art pieces, there was little room for further experimentation, development or improvement. Similar problems can be attributed to the iterative “Sync-in-Team” (SIT) study, where all changes to the setup had to be made essentially outside of the actual presentation context. Also, most of the evaluation methods up to this point were quite time-consuming and essentially post-hoc; this approach might be ideal for the purpose of well-documented research, but is insufficiently fast-paced for real-world GUI-development. Also, as the SIT shows, up to this point, most of the changes made to a setup are predominantly consequences of contextual factors (i.c. changes in setup-location), rather than the result of the evaluation analyses. In this chapter, the focus of the development strategy is on speed: quickly isolating problems in the design, rapidly adjusting, aiding and furthering usability, and swiftly moving on with an improved prototype. In order to assess the mediation methodology that would satisfy the adaptability and scalability requirements of user-centered interfaces, an altogether different lab-based type of usability study was required. As part of a different work package of the Emcometeccaproject, this study primarily focused on the conceptual bridge between sound objects (Chion, 2000), motor constraints of the human body (Godoy, 2004) and embodied mediation technology (Leman, 2008). However, the purely functional study of objectoriented software paradigm with its focus on the mediating role of the body through virtual entities (Mulder, 1997), yielded itself perfectly to serve as a basis for applying a set of usability heuristics and metrics to the representational mediators, multilevel mapping layers and immersive natural communication tools latent in a pre-existing augmented reality environment (Diniz et al., 2010).
121
8.1 Virtual objects as a mediator principle A close relationship between user and mediation technology is expected to provide a solid ground for the application of user-oriented methods. Experiments like "Beat it", "Sync-inTeam" and the exploratory studies performed in combination with "The Heart as an Ocean" and "Lament" zeroed in on parts of acclaimed HCI methodologies and usability elements. However, they do not investigate the full scope of usability research. In all of the above cases, the usability evaluations relied solely on qualitative post-hoc assessments by participants, without the inclusion of performance-based metrics. The following two tests are in essence the early prototypes of the “SoundField”-installation (Chapter 9). In order for the platform used in the “SoundField”-project to function, a quantitative and heuristic evaluation of the systems features was required. Therefore, simultaneously with the previous tests of the artworks, the development of a software framework was initiated, aiming at providing an adaptable platform for deploying user-centered interfaces. Although providing insights into the relationship between (task) performance and user appreciation, the exploration of features through use cases confined to a specific domain proved to be too restricting. More specifically, this limitation was present in the interaction features’ assessment and in the structuring process, a fundamental condition for the adaptability adherent to a rapid prototyping requirement. Therefore, a need for a broader, more ecological context that would allow the extraction of features based on demand and experimentation was identified. This strategy would result in a more focused and comprehensive bottom up approach, while guiding the top down background and mediation paradigms preliminary decisions. This new development context was created in “SoundField” (which will be the focus of the following section).
Figure 8.01: Compound picture showing interaction with the Virtual string prototype
122
8.2 Virtual String In the “Virtual String”, the suitability of the mediation principle described above was assessed. Additionally, it tested both the design and the performance of the framework’s preliminary implementation. The use case involved “playing” a virtual string object suspended in an augmented reality environment, using two virtual objects that are controlled by the user’s hands (figure 8.01). The physical space was mapped onto the virtual world in which a cylindrical object representing the sound producing string was positioned in the middle. Through 3D positioning, orientation and by means of optical tracking, the users could “pluck” the string when it collided with the users’ tracked “hands”, represented through rectangular virtual objects (see figure 8.02).
Figure 8.02: Picture showing the multi-perspective view on the VS prototype.
The initial assessments were drawn from the users’ comments that originated during and followed the interaction sessions. Participants were asked to perform simple tasks during which they were asked to describe their actions. As such, their comprehension of the tool could be rated, the comfort of the interaction could be examined and usability issues in the design could be detected. A set of simple heuristics and metrics were applied during and after the interaction (figure 8.03). Users were asked to describe their actions, interpret the aural and visual cues, perform a set of pre-defined time-based tasks (to probe efficiency and accuracy), rate their performance and the system's responsiveness and comment on their experiences. 123
The information obtained from this evaluation provided some specific further implementation guidelines regarding the configuration and morphology of the virtual object in the following study.
Figure 8.03: Picture showing a user operating the music-based framework implementation for interactive sonification.
8.3 Multi-level Sonification Tool The previous prototype was developed as a proof of concept prototype. In the following music-based interactive sonification system, a more far-reaching target was to apply this technology to a specific domain, namely, to use interactive sonification for sound-based multivariate exploration. Consequently, based on the feedback from the virtual string project, we developed a new and more advanced interactive sonification system that targeted the interactive exploration of multivariable data through non-speech audio communication (Diniz et al., 2010). The goal of this implementation was to enable users to access multiple information levels in non-speech sound-based communication (Diniz et al., 2010) and to investigate, through the application of concepts used in electroacoustic musical composition, the possibility of establishing a unified context between individual sound streams that are exposed simultaneously through time. The design was based on the incorporation of theoretical concepts into the software architecture (such as sound objects, corporeal motor composition, embodied music cognition). As in the previous use case, the inspection process was conducted through the interaction with virtual objects in an immersive 3D environment. This extended the study 124
of the relationship between the users’ embodied behavior and the virtual entities, and the different sound levels. The present system’s conceptual design was based on the combination of two main metaphors: the virtual inspection window, providing access points to the variables and their values belonging to a given dataset, and the virtual microphone, which implemented a sonic inspection tool controlled by the user’s hand. As such, the architectural framework was based on the functional division of modalities into individual branches around a virtual-scene representation. Each independent virtual object belonging to the virtual inspection window functioned as a sound source. The sonic representation of its value was activated through collision detection, when the inspection volume of the virtual microphone intersected the virtual objects. Moreover, two additional sonic representations of the relationship between the data’s values were made available, interval and chord, in order to provide a sonic feedback of proportions in the data. In addition, the position of the activated objects in the inspection volume of the virtual microphone influenced the respective output in terms of auditory relevance. Following a top-down approach, it was composed of abstract managing cores and their respective elements per modality – visual, auditory and human input – which provided a unified, scalable interface between the elements that constituted a particular iteration of the system. The concrete low-level functionality was then implemented via a bottom-up procedure through the use of adequate external software libraries. In summary, the adopted interface paradigms can convey multiple perspective views between different levels in the form domain by representing the evolution of the sound object/data in time and in structure, and by establishing and comparing different groupings of N variables. As a result of this encapsulation, the concrete implementation of the virtual worlds and their modal representation and manipulation could be either refined or substituted according to the desired performance, access or functional needs. As such, we concluded that the adopted approach could help to close the semantic gap between the user and data through sound. Finally, a preliminary user test was conducted. Although revealing some issues concerning the ad hoc configuration, the task performance improvement as well as the overall positive feedback concerning the base methodology was encouraging. To investigate whether the platform functioned as envisioned, two basic user tests were performed. These consisted of both observations, documenting users’ appraisal and feedback regarding the interface. In the latter test, a measurement of their performance was taken while conducting an experiment with a set of predefined tasks. The methods used were adopted from the field of HCI and usability studies and based on techniques such as heuristic evaluation and cognitive walkthrough. The initial and exploratory investigation was performed by a small number of evaluators and consisted on free interaction with the prototype while all the three levels of sonification were activated. They were asked to inspect its basic operation and to 125
comment on it. No further instructions were given at this time. Most users found the interface to be quite responsive and its operation to be intuitive. However, only a small percentage of the test-subject mentioned the different levels in the sonification, so the purpose of these different levels had to be clarified. Other problems that were reported with the prototype included the fact that movements of the inspection tool parallel to the screen were visualized a certain inclination, which complicated the interaction with the virtual objects. This was due to the use of a perspective based visualization. Finally, some of the users complained about problems concerning their depth-perception within the visual representation. The addition of the second screen, which conveyed depth perception, improved the spatial awareness of the users and other suggestions that were made by the participants were considered for further implementation. The actual interface, however (i.e. the MoCap-system), was not substituted. For the second test, a more thorough evaluation of the platform was made, and users were required to perform a series of tasks in which they evaluated the different levels of sonification and the relations between them (Figure 8.03). The aim of this test was to investigate whether perceptual abilities of the participants were increased by the different sonification levels or not. The proposed tasks were comprised of the exploration of a predefined one-dimensional dataset through the use different combinations of the sonification levels. Within each test, the user would be asked to find a certain relation present in the presented dataset with only the lowest level of sonification after which the same user would be asked to repeat this task using the first level combined with one of higher degrees of sonification to explore a differently ordered representation of that same dataset. The test that focused on the combination of the level 0 and level 1 sonification did not show a rise in effectiveness for any of the participants. This test proved to be problematic because of the fact that a number of different relationships within the dataset (i.e. intervals) were sonified, and it was too difficult for participants that did not have any specific musical training, to discern which. The combination of the level 0 and level 2 sonification, however, in which the test subjects were required to find a set of relations (i.e. chord) did yield good results. The addition of the level 2 sonification proved to be very valuable in the discerning task, and it improved the performance of every participant (i.e. the time consumed in performing the task). After exploration of the interface and performance of the set tasks, participants were required to evaluate the human interface they had used in terms of performance, maneuverability and precision much in the same way as the initial evaluation. Moreover, participants were asked to comment on the completeness of the improved visual output (in order to find the requested relations and to interact with the virtual array) and on the sonification output in terms of distinguishability, information carrying potential and aesthetic qualities. Additionally, a number of remarks and requests were recorded that are being considered for further implementation. A first issue that was raised was that the 126
different sound levels (i.e. the different sound-relations within the data) should be made selectable. Directly related to that, one of the respondents pointed out the need for a test with non-prepared dataset. Admittedly, this was a just critique, which on the other hand indicates that the plat- form functioned properly as means to evaluate the dataset, because otherwise, the test-subject would not have been able to notice the fact that the values had been retained and only their order had been changed. Finally, some of the test-subjects suggested a number of changes that should be made to the inspection tool, namely the fact that a rectangular as opposed to a square base of the inspection tool would allow for more accuracy in the exploration, and that making the edges of the pyramid-shape visible would enhance the perception concerning the orientation of the tool and thus the precision in the exploration task.
8.4 Conclusion The overall evaluation of the latter prototype revealed that all participants were capable to understand and properly operate this platform. Concerning the sonification output, the users reported being able to perceive the different sound-levels and could discern the information that was conveyed by them, although the execution of all aspects set forward in the tasks, was at times problematic. Furthermore, performances were considerably improved by the use of the different sonification levels. More importantly, the addition of the multiple viewing perspectives and ‘camera angles’, requested during the evaluations by some of the early users, had a similar positive and elucidatory effect. In that respect, the findings of this case study of initial user testing and user-informed iterative design, was quite promising in view of further development and as a test case for parts of the conceptual model. In essence, this chapter consolidates a lot of the problems encountered in the earlier, more exploratory stages of the research project. Finally, in practice testing parts of the implementation of the conceptual model and delving into some of the ideas set forth therein, and carrying out real-time user-suggested modifications in the actual context of use, with the emphasis on an agile work flow, and immediate improvement of the overall usability, not to mention tentatively applying HCI-methods, like usability inspection methods, metrics and heuristics (e.g. cognitive walkthroughs and time-based task performance). The following chapter consolidates most of the methods described in the previous chapters, and within the “SoundField” project, brings together methods borrowed from
127
Agile development, HCI-heuristics, sociological survey methodology, iterative design, etc. in an open-ended collaborative design project.
128
9 Participatory and Iterative Design Study: “SoundField”
In the previous chapter, the last of the methodological considerations concerning the conceptual model were put into place, namely the iterative design based methodology around a prototype, and unhesitatingly making instant adjustments, while testing it with predetermined usability metrics. In this chapter, most of the lessons learned throughout the previous chapters come into place. The “SoundField”-project comprises a bottom-up development, paired with a topdown infrastructure, an open-ended, user-centered and collaborative design strategy combined with rapid and iterative prototyping and development, and an integrated evaluation component inherent in the development. This is the project in which lessons are learnt from mistakes made in the past, and working methodological components from the earlier studies are consolidated and implemented into the experimental application of the conceptual model.
9.1 Introduction “SoundField” is an immersive augmented reality environment that makes an embodied interaction with visual and sonic content in an immersive environment possible. It is in essence an unfinished black box artistic installation based on a flexible, multi-modal enabled java-framework which functions as a testbed for different modes of sonic, visual an social interaction, usability (and interface) evaluation and use-case development. It can accommodate a variable number of participants and it presents a space in which sound can be created, influenced, manipulated or explored by means of movement in an augmented reality environment. Technically, the framework consists of a flexible platform with a 10 infrared camera array and a quadrophonic speaker system. The cameras constantly track
129
the position of the participants, enabling the user to experience corporeal control over the sonic and visual environment and creating a forum for the exploration of new modes of creative sound and music interaction. As the subjects move through the room, they effect change in the projected images and the produced sound. The information from cameraarray is gathered, sent to the java-framework and parsed to Max/MSP in combination with Ableton Live for sound and visuals (figure 9.01). The mechanical setup is, both in terms of content and interaction, in essence openended. In collaboration with the users, we reflect on the framework, on its purpose and on its (possible) contents, on improving ergonomics, etc. Using a multi-view orthographic projection and a combination of infrared motion tracking and custom user-interface devices, “SoundField” provides real-world access to objects existing in a virtual reality. The levels of interaction are derived from theories relating to geometrical forms set forth in Kandinsky's treatise "on point and line to plane", and are accordingly represented in the various accessible levels of the installation. This
Figure 9.01: Schematic representation of the framework’s inner software and hardware workings
The larger the number of participants and the more social interaction, the better the interaction with the installation becomes. As a consequence, the installation was devised in such a way that the different levels of human computer-interaction can only be accessed when participants interact and collaborate in it. As such, “SoundField” functions as a catalyst for social interaction, more than merely as an interactive human computer interface. The levels of interactivity are defined by increasing spatial dimensions: from a
130
single point in space, over to a line, which connects two points, then to a surface, where three points are interconnected. Additionally, each of these dimensions corresponds to an equivalent aural dimension, where sounds gain in complexity throughout the increased dimensionality.
Figure 9.02: Picture of the laptop-based demonstrator of “SoundField”.
9.2 Prototypes and developmental strategy: A first demo was presented during an artistic summer school in July 2011 in Dworp, Belgium, which brings together people from all ages and a great diversity in artistic fields (figure 9.02). This provided a representative and varied test audience, similar to the eventual users. The participants were asked to interact with a computer-based mock-up demonstrator of the eventual platform, enabling only computer mouse-based interactions. After an exploration and interaction with the demonstrator, participants were gathered into small focus groups based around artistic practices. During half-open interview sessions, the envisioned immersive environment setup was explained and questions were asked pertaining to functionality and goal assessment, tool operation, social interaction, sonic and visual components and aesthetic properties. At the beginning of October 2010, a technically fully deployed version of the demonstrator that had been shown during the summer school was tested by means of a heuristic evaluation (figure 9.03). After resolving some of the reported issues, a similar combination of demonstrations and focus 131
group sessions was done in collaboration with the instructors of WISPER (a cultural organization that provides workshops and courses in dance, literature, music, theater, sculpture and audiovisual arts). During both of the above sessions, ideas emerged that inspired either implementation strategies or interaction scenarios or strategies, some of which crystallized into specific use cases. Finally, from October 2010 to May 2011, in close collaboration with the Flanders “Youth and Music”-program, the “SoundField”installation was presented to the - albeit relatively specialized - broader audience.
Figure 9.03: Control room of the “SoundField”-installation.
During this period, several fully functional implementations of the technology were developed according to a method, analogous to the formative user-centered evaluation cycle of Gabbard et al. (1999). After a short introduction and demonstration of the platform, a small group of users were asked to interact with the abstract demo-version. Then another demo was presented where the platform took up the role of an active agent. During these interactive demo-sessions, participants were invited to describe their actions, much in the way that an interaction design session is conducted (Barrass, 2008) and consecutively were asked questions about their performance. They were also encouraged to ask questions to improve their understanding of the installation. After the demo-session, people were asked to participate in a focus group session that was conducted with a number of participants ranging from 5 to 20 participants. Even though the main purpose of the demonstrators was to show participants what the framework was capable of, it was also an intention of this session to record the spontaneous ideas and associations of the participants. Consequently, the artistic background of the coinciding workshops, the aspirations expressed by the participants, their questions, remarks, findings and feedback gave a clear direction to each one of the use cases (figure 9.04).
132
Furthermore, due to the experience the focus group participants had gathered during the interaction session, their expectations concerning clarity, ease of use, control and aesthetic properties could also be discussed in a focus group. In this manner - within the scope of the possibilities inherent in the framework - prior to the rapid development process, the developers had already gathered a very thorough insight into what needed to be implemented into the system. In this way, and to an extent achievable through rapid prototyping, the requested features were incorporated and presented to the same groups of users. Another run of exploratory tests and interaction sessions was conducted, after which the overall presentation, the specifically implemented features of the given use case and the experience of the users were recorded. Finally, to be able to incorporate the developers’ experiences in each of the consecutive use cases, the same procedure was applied throughout all iterations. Figure 9.04: Table showing open-ended properties of the “SoundField” installation.
Aspect Cultural Practice
“SoundField” operation Variable
Artistic Purpose Number of participants Elements / Scenarios Sonic output Visual output Technologies Basic operation Basic mapping
Variable Variable Variable Variable Variable Fixed Fixed Fixed
So, in summary, the participatory design and iterative development process described above, which included 1) demonstration, 2) interrogation, 3) implementation, 4) interaction and 5) evaluation (except for those use cases solely devised for demonstration purposes) was carried out throughout the entire duration of the project.
9.3 Elements, Behaviors and Scenarios: To be able to achieve results with rapid prototyping, the “SoundField”-platform is based on a system that uses a) malleable elements with selectable element-behaviors and b) semi-defined scenarios were devised.
133
9.3.1 Elements and Behaviors First, the individual ‘elements’ and their ‘behaviors’ are discussed. The basic levels of operation are constructed around three types of virtual objects with a customizable set of properties. This means that all three element classes have a set of adjustable characteristics, that can be implemented into the framework (if necessary) given the specific requirements of a given use case. The number of elements and the combination of behaviors is then adjusted based on the requested level, type and purpose of social interaction within that given use case. Spheres are the first and lowest level of interaction within the augmented reality environment that focus on the most basic elements, (i.e. orb-shaped virtual objects). Sphere-object behaviors can only have a limited number of properties. They are either fixed or mobile. Fixed spheres have a static location within the environment. The visual and sonic representations of spheres are either a direct result of the 3D-positions of the rigid bodies that control the respective spheres, or they can be the result of the proximity or collision between (combinations of) fixed and mobile spheres. The second level of interaction constitutes strings. Strings are commonly the result of two (fixed or mobile) spheres. This makes the properties of string-behaviors slightly more complex. Strings can be fully fixed (when the string is a result of two fixed spheres), semi-fixed (when the string is a result of a fixed and a mobile sphere) or free (when the string is a result of two mobile spheres). Additionally, in the case of semi-fixed of free strings, strings can be made stable or breakable. In the case where they are breakable, the proximity between two spheres is central in the appearing and disappearing of strings. Strings are activated by collision between a mobile object and the string. The distance between the position of the two spheres that delineate the string, controls the pitch of the string. The third and highest level of interaction is concentrated around planes. Planes are established through a constellation of minimally three spheres (or strings). The behavior of planes is essentially the same as the strings. The activation is also based on collision between a mobile object and the string and the total surface controls the pitch of the object. Planes can also be stable or (dis)appearing. In the latter case, they are also the result of the proximity of the three constituent spheres. As a consequence of this structuring, a single user can only access basic (i.e. low) levels of interaction, based around position, collision (with another object) and/or the proximity of a sphere (in reference to another object). A single user can reposition that sphere by using his movements to trigger sounds and visuals in their own right, or can even get involved in a communication with the platform itself. But the artistic possibilities are limited. Access to the second level, represented by a line connecting two spheres, can
134
only be granted when at least two persons occupy the interaction space. An additional premise is that the two users are in essence required to interact: the string is represented either sonically, visually or both, but the sonic/visual properties conveyed in this second level can only be instantiated when the two users interact by repositioning the spheres close enough for them to react. Accordingly, accessing the third level (i.e. a surface mimicking the properties of a membrane or drum skin), requires a similar interaction of minimally three participants (see Figure 9.05).
Figure 9.05: Schematic Representation of SoundField’s elements and behaviors.
9.3.2
Scenarios
The basic behaviors described above, can be presented individually, but more often, they are bundled into specific combinations of premeditated sets of elements of fixed/free behaviors. These combinations were dubbed ‘scenarios’. From a user’s viewpoint, these scenarios convey a metaphor for the latent action possibilities and afforded levels of (social) interaction in the interaction space. From the developers’ side, they also entail a development-strategy that allows the rapid implementation of sets of element-behaviors into the platform. As such, they can be considered pre-coded building blocks (with a flexible number of virtual objects and with semi-defined behaviors) that can be swiftly modified. Each scenario constitutes specific behaviors of elements (objects, strings, planes) and focuses on how the different elements can be acted upon given a specific combination of string-, object- and/or plane-responses. Five of these quickly adaptable, combinable and quickly implementable scenarios were devised. The choice to use a 135
(combination of) scenario(s) was based on the user-expectations discussed during the focus group sessions. After the focus group session, a rapid analysis was made of the purposes that were expressed, of the requirements for visual and sonic features that needed to be implemented and of the modes of social interaction that needed to be facilitated. Based on this analysis, the prototyping procedure started with establishing which interaction scenarios needed to be implemented in the given use case.
Figure 9.06: Schematic representation of the implementable scenarios.
A first scenario is simply called ‘spheres’. This scenario constitutes the basic means of navigation in the interaction space and its basic operation revolves around the 3D-position of a (number of) sphere(s). Its purpose can be to control simple position, to calculate proximity or to detect collision in the interaction. All use cases use at least one spherescenario. The second scenario (still with a relatively low level of interaction) was dubbed ‘necklace’. It relates to the position or proximity of two dynamic objects in reference to one another and form the basis for all string behaviors. A third was labeled ‘fence’. This scenario constitutes the creation or presence of a number of static objects, strings or planes in the interaction space which can be acted upon by means of a sphere. A fourth scenario was called ‘umbrella’. In this scenario, a large number of static or dynamic objects are all individually connected to one central object. The fifth and final scenario, dubbed ‘tug of war’, is a variation to the previous scenario. Here, two groups of static or dynamic interconnected objects are linked through one central dynamic object that is part of both groups.
136
This very versatile combination of flexible element numbers and behaviors with adaptable and joinable scenarios was a prerequisite to be able to deal with the different artistic practices, functional requirements and aesthetic properties expressed in the focus groups and to successfully incorporate these user-informed adaptations in the use case development (see Figure 9.06).
9.4 Use cases Each time the platform was put to use in such a way that it would serve an artistic purpose within a given cultural practice, that specific crystallization or iteration would be defined as a use case. All use-cases were developed in roughly the same manner (and most of them as part of an artistic workshop). After the demonstration and exploration session, ideas were collected in a group-discussion for possible interaction-strategies, sounds and visual aspects (see Figure 9.07). Ensuing the brainstorm-session, time was taken to collect the ideas that had arisen during the interaction and focus group sessions and how they would be implemented (based on the building blocks described in the previous section). Figure 9.07: Table showing the thematic overview of all of the “SoundField” Use Cases.
Theme/Purpose Demonstration Sound Manipulation
Use Cases SoundObjects, SoundFlock, SoundSculpture 1 & 2, SoundStage, SoundGrid SoundSculpture 1, 2 & 3, SoundTrails 1 & 2, SoundPath, SoundTracks, SoundMaze 1, 2, 3 & 4, SoundGrid
Movie Manipulation
SoundMov(i)Es 1, 2 & 3, SoundStage
Choreography
SoundPath, SoundDance, SoundStage
Theatrical performance SoundScene, SoundTales, SoundStage Education / Therapy
SoundMaze 1, 2, 3 & 4, SoundGrid
This, in turn, gave rise to particular use cases. Each of these customized versions of the platform was presented back to the participants. After any task performance, interaction session and/or artistic performance, the (implemented) features of that given use-cases
137
were evaluated by means of a standardized evaluation questionnaire that was applied throughout the project. It is important to note that none of the use cases dealt with the combination of traditional instrument performance combined with an interactive augmented reality environment. The musical interaction in all instances came from direct interaction of participants with only the “SoundField”-platform itself. The focus was on how this augmented reality environment functioned as a tool for meaningful collaborative or interactive musical expression (Miranda & Wanderley, 2007), not on how it could work as a technologically enhanced canvas fostering more traditional modes of music performance.
Figure 9.08: Figure depicting the chronology of the use case development.
The relatively new element in the presented approach is that, even though a wide range of cultural practices were explored, the development process and the basic modes of operation were always kept practically identical. This was done with the explicit goal of establishing the highest possible level of standardization between the use cases throughout the entire project, to ensure there would be a common ground to base the evaluation on. It also enabled participants of different use cases to immediately understand the functionality of different iterations and to identify with the rendered output, no matter what use case was presented. This inter-use case conformity of properties was safeguarded across all iterations of the platform. Next, a thematically organized overview of the use cases that were developed over the course of the project will be presented. The descriptions are not exhaustive and grouped into the cultural practices that were explored. A short qualitative description will be given with every group of use cases (figure 9.08). 138
9.4.1 Demonstration use cases The demonstration use cases were presented to all the users of the “SoundField”framework. The goal was to illustrate the basic elements and features of the system, to explain the different levels of operation and to exemplify possibilities (and restrictions) of what the installation was capable of. The non-representational character of these demonstrations on the one hand ensured that the demonstrations were simple enough, on the other hand, that participants would be stimulated to come up with new ideas for purpose, sonic and visual content.
9.4.1.1 SoundObjects A first setup was called SoundObjects. This was the demonstrator use case that was presented to all the users of the “SoundField”-framework prior to the collaborative design process. Its goal was to illustrate the basic elements and features of the system, to explain the different levels of operation, and to exemplify possibilities (and restrictions) of things the installation was capable of. The demonstration required the involvement of five participants. Participants (each of them equipped with one rigid body) were asked to explore the room, at first without any further explanation. The positions and the collaboration between the participants activated properties in the visual and sonic environment. The visual representation of five objects (four sonic objects and one activation object that triggered collision) corresponded with the position of the five participants in the room.
Figure 9.09: Picture of the SoundObjects demonstration.
139
Through interaction with each other, they could control the sonic properties of both the low (sphere) and higher level sound objects that formed a line (a string that interconnected two of spheres) and a triangular plane (a membrane that connected three of the spheres) in the visual representation. The activation object was then used to trigger the sound of the string (that produced a pitch based on the distance between the two spheres) and the membrane (whose pitch was based on the constantly changing surface between the three spheres). The spheres produced their own sound based on their position in the room. The visual representations (i.e. simple geometrical shapes) and sounds (all of which were based on filtered noise synthesis) were deliberately kept quite abstract. This was done intentionally, on the one hand to keep the demonstration simple enough, on the other hand to stimulate the participants to come up with new ideas for purpose, sonic and visual content.
9.4.1.2 SoundFlock The second demonstrator that preceded most of the FG sessions, was presented in addition (but also in contrast) to the first demo; the goal was to show that with a limited yet similar number of basic features, a completely different virtual environment could be created. The demo accommodated one participant that was given rigid body and was asked to move with it through the room. The participant’s movement was translated to the virtual space as a ‘flock’ of visual entities, programmed to closely follow the projection of the handheld infrared-object. Depending on the quantity of movement of the participant, the flock stayed near the participant’s actual position or followed it with some delay.
Figure 9.10: Picture of the SoundFlock demonstration.
The virtual movement of the entities in the flock (not the movement of the rigid body itself) in turn was responsible for the sonic outcome (i.c. granular synthesis). Through the properties of the participant’s movements - erratic or fluent, slow or fast - an interactive communication between participant and representation became apparent. This exemplified
140
the flexibility of the platform as well as an easily understandable example of the level at which sound and visual representations could be made interactive (i.e. a result of the collaboration between man and machine), comparable to the swarm granulator developed by Blackwell & Young (2004).
9.4.2 Sound manipulation use cases This series of use cases accommodated the simultaneous interaction of up to five participants. These iterations aimed at real-time 3D-manipulation of sound parameters. At first, only spatially controlling a single filter, yet, as soon as the properties of three axes in space were available, other parameters (such as pitch, playing speed, timbre, dynamics, etc.) were incorporated.
9.4.2.1 SoundSculpture 1 SoundSculpture 1 was the first user-inspired use case of the “SoundField”-framework, which aimed at realtime 3D-manipulation of a sound, much in the same way as a sculpture is molded into the desired shape from a block of marble, much in the same way as the virtual sound sculpting interface of Mulder & Fels (1998). The use-case accommodated the simultaneous interaction of up to five participants that could manipulate and control sound parameters.
Figure 9.11: Picture of the SoundSculpture 1 use case.
At first, only spatially controlling a single filter, yet, as soon as the properties of three axes in space were available, other parameters (such as pitch, playing speed, timbre, dynamics, etc.) were incorporated. The participants could activate and alter the musical parameters of the sound attributed to the rigid body by moving around in the space. The location in the interaction space determined the amount of sound-effects that was used. By moving the infrared markers over the length-, width- and heigth-axes of the room, the 141
sounds could be manipulated, while the sonically molded objects were simultaneously represented in the virtual space. Just like with an actual raw material, the artistic expression consisted of giving the virtual sound object the desired shape.
9.4.2.2 SoundSculpture 2 The two (very similar) variations that ensued were dubbed SoundSculpture 2. Both variations of SoundSculpture 2 used the same visualization and modifiable sonic variables, i.e. playing direction and speed, timbre and amplitude. The basic presentation of the previous use case was applied with the addition of a waveform-rendering that was incorporated in the visualization, based on the total amplitude. For the first variation, sounds that were produced during the actual creation of sound-producing sculptures in an artistic workshop, were recorded. Much in the same way as in the previous use case, the sounds that were used were considered the raw material. In this respect, the ensuing sound manipulation sessions were considered the meta-sculptures. After the recording, the sounds were uploaded into the software, played back and looped. In the second variation, the same basic setup was used to do an experiment in which four 8-bar instrumental songexcerpts were played back in a loop. Both setups, however, proved to be extremely difficult to control, as it was practically unachievable to properly synchronize constituent excerpts. As a consequence, at this stage in the development, no post-interactionevaluations were performed yet.
Figure 9.12: Picture of the SoundSculpture 2 use case.
9.4.2.3 SoundSculpture 3 SoundSculpture 3 continued along the lines of the previous experiment. This setup functioned with basically the same constituent elements, i.e. four song-samples with variable playing speed, timbre and amplitude. However, playing direction was eliminated
142
as a feature, yielding a lot more fine-grained spatial control over playing speed. Also, the looped samples were kept considerably shorter. Finally, as it only seemed to confuse participants, the amplitude-based waveform-rendering was removed from the visualization. These modifications made the use case a lot more enjoyable to the participants. This use case was the first in this line of development in which, through spatial sound manipulation, a sonic outcome was produced that could be labeled as enjoyable.
Figure 9.13: Picture of the SoundSculpture 3 use case.
9.4.2.4 SoundScapes SoundScapes is the last variation of the platform that functioned similarly to the previous use cases. As it directly followed a sound production workshop, this use case revolved around the ambient sounds that the participants had created and recorded themselves during a sound interaction workshop.
Figure 9.14: Picture of the SoundScapes use case.
143
The use case gave 4 participants the opportunity to spatially mold the created soundscapes further. It is in respect to the basic operation mode in more than one way comparable to the SoundSculpture use-cases. Yet, given the goal of the workshop, very little emphasis was put on the visual representations, so, solely for referencing position, the basic JAVA-3D representations were used.
9.4.2.5 Soundtrails 1 The SoundTrails 1 use case started from essentially the same basic idea as the aforementioned SoundSculpture use cases. In this use case, however, the emphasis was very much on the visual representation, so a lot more attention could be given to the development thereof, with the explicit purpose of solving some of the problems pervading the previous iterations. This use case also presented essentially a three-dimensional sonic canvas. A group of four participants spatially controlled playing speed, timbre and amplitude of synthesized sample-based sounds, while a three-dimensional visual representation of each of the rigid bodies lined out the trajectory of the movements made by the participants. The color of the representation changes perpetually in a fixed order and after an interval of few seconds, the representation of the path began to fade and eventually disappeared after about five seconds. The motion characteristics were conveyed in the sonic content (i.e. the more intense the movements, the more intense the sounds were rendered). Whether in this use-case the sonic or visual representation of the rigid bodies got more attention, was entirely dependent on the participant.
Figure 9.15: Picture of the Soundtrails 1 use case.
144
9.4.2.6 Soundtrails 2 Towards the end of the project and during a sound interaction workshop, SoundTrails 2, an adaptation of the previous use-case, was reiterated (and modified) on popular demand. The use-case still contained a three-dimensional audiovisual canvas, but some of the features were altered and improved based on participants’ requests. The sonic content remained largely unaltered and was made somewhat less distorted. This version of the use-case, however, offered the opportunity of interaction to only two participants. The contrast of the colors was made a lot brighter and no longer continuous. The colors did no longer appear in a fixed order, but were randomized and a motion threshold was incorporated for activation of outlining the trajectory. A final change in this use case was that the created trails didn't disappear after an interval of five seconds. Once a trail surpassed the activation threshold, the multi-colored trail appeared and remained visible. In this way, each participant was able to make a three-dimensional action painting. Every three minutes, however, the canvas was cleared and the action painting could start afresh. The changes that were implemented between the first and the second version of SoundTrails were applauded by the participants, but it is worth noting that due to these changes, the iteration lost much of its social significance.
Figure 9.16: Pictures of the Soundtrails 2 use case (demo and canvas).
9.4.3 Sonification tools This set of use cases got to the heart of the project, namely, gathering data from one source – in this case a person’s spatiotemporal cues – and conveying that information to a second person by means of sound either simultaneously or diachronically. Both use cases contained similar sonic elements, albeit with slightly different outer appearances, and served similar purposes.
145
9.4.3.1 SoundPath The first of the sonification use case contained elements of the previously described sound manipulation tools, yet SoundPath (Diniz et al., 2011) comprised two basic stages in its execution. First, an array of intervals was scattered in space by a first participant, who performed part of a free-form choreography in the room. The distribution of pitches and intervals was connected to the amount of movement made at that given point in time and the full trail that was set out in the room represented the full movement pattern. Then, an other participant was given two devices that enabled the inspection of that path. The system made it possible for the second participant to inspect individual sonic grains of the movement, larger portions of the trail that was set out or even the overall properties of the choreography. The size of the portion that was scrutinized was controlled by means of a second device that regulated the variable inspection scope of this virtual microphone. Each of the users was explained that by reaching out with their arms, the range of the virtual inspection tool (microphone) would increase, while exploring the choreography with a closed posture would allow a fine-grained control and detailed scope of a portion of the movement. Additionally, the users were informed that a low-pitched sound signaled points in the beginning of the sequence and - when progressing over the trail - the pitch would become higher near the end. In this fashion, the intricacies of the whole routine could be extrapolated solely from pitch and 3D-location, and the choreography could completely be reconstructed by the second participant, without the interference of the first.
Figure 9.17: Picture of the SoundPath use case.
9.4.3.2 SoundTracks The previous use case was very much applied to a specific purpose for individual users, but had no real artistic value per se. SoundTracks was a use case that combined elements of SoundPath on the one hand and the SoundTrails 1 use case on the other. This setup required two participants and the visual representations were somewhat brighter and more 146
contrasting, but were in essence the same as in the SoundPath use case. The sonic elements were kept practically identical, but alternated from high to low and low to high pitches.
Figure 9.18: Picture of the SoundTracks use case.
Also in terms of basic operation, the setup was quite similar, as one of the participants was responsible for setting out the trail and one participant had the exploration tool. The main difference, however, is that both participants collaborated, so the act of creating and exploring of the trail coincided. Furthermore, to avoid a “cluttered” virtual environment, the created paths were set to disappear after a five second interval (much in the same way as in the SoundTrails 1 use case).
9.4.4 Choreography tool Whereas the sonification examples delved deep into transferring information by means of spatiotemporal sonic cues, the following example took a comparable vantage point, which was then applied to an actual performance setting, and developed to be use as an actual music-making device.
9.4.4.1 SoundDance SoundDance was a use case completely based around a singular workshop that focused on spontaneous, non-narrative choreography by an instructor, working together with a class of students. The objective of this use case was to provide an abstract sonification of the dance-improvisations. Five participants at a time, three of which controlled the interactive elements, experimented with abstract synthesized sounds that were triggered by collision of fixed strings and free objects. By passing through interconnecting virtual strings suspended between each of the dancers, the participants did not only interact with one another, but also with the constantly changing virtual and sonic environment. Through their
147
choreography, they created an ambient soundscape, so in this use case, the dancing bodies could be considered an autonomous musical instruments in themselves.
Figure 9.19: Picture of the SoundDance use case.
9.4.5
Theatrical use cases
In not all instances, the interactive environment offered by the “SoundField”-installation was used as the direct focal point of the action or attention. In the following three instances, it served as a technologically enhanced backdrop to enable augmented dramatic performances.
9.4.5.1 SoundScene In the first of the use-cases to fulfill a dramatic purpose called SoundScene, the “SoundField”-framework functioned as the backdrop in a drama workshop, creating a (predominantly) sonic environment for theatrical performance. The number of actors was not limited as such, yet the number of participants controlling the sonic environment did not exceed three. Visually, the scene remained basically unaltered, as the visual representations only showed one blue and two pink dynamic spheres, serving either as sound manipulation objects in their own right or as an indicator for the proximity of the controller device to any given sound source (visually represented as static yellow spheres) in the sonically enhanced space. During the performance, sonic elements latent in the environment were triggered and reshaped, depending on what the theatrical setting required. The sonic elements of the performance included the looped playback of a number of monologues (recorded with the actors during rehearsals) and soundscapes (which were recorded during exercises in the drama workshop). The loops were uploaded into the four static and two free sphere objects. The panning and amplitude could be controlled by the two dynamic spheres, the
148
activation or blend of the four static sample-based elements could then be controlled by the remaining controller object. The overall outcome was a mix of the dramatic performance with an antique choir-like theatrical backdrop that was presented to the audience.
9.4.5.2 SoundTales In an adaptation of this use case, SoundTales, the goal was also to offer a customizable sonic environment in which participants could collaborate in the context of a drama workshop. In this use case, however, less emphasis was on pre-scripted performance, more emphasis was put on improvisation. Unlike the previous use case, the created sonically augmented stage did not have any prerecorded monologues or similar narrative elements, but served solely to offer sound effects and background noises that would provide the setting that could inspire or support the narrative line in a number of improvised scenes. The four static elements of the previous use case were reiterated but made invisible. The dynamic sound elements were omitted from this setup, but the number of controller objects was doubled. Visually, only the four controller objects were represented in the environment and sonically, a range of different soundscapes and sound effects were loaded into the controller objects of which panning and amplitude was triggered based on the proximity to the four invisible and static sound stations. Within this workshop, the installation functioned both as a context as well as an enabler for the purpose of improvisation.
9.4.5.3 SoundStage In a last use case that focused on theater, dubbed SoundStage, the “SoundField”platform functioned as an augmented multi-media performance stage for musical and theatrical improvisation and free choreography. The goal set by the artists was to create a mixed-media performance, staged within the augmented reality environment, in which live-generated camera-footage would be projected on the sidewalls and live-sound (generated on a soundboard, captured with two contact microphones) would be manipulated in real-time using the virtual elements. Two participants manipulated the two dynamic spheres, which controlled playback speed, amplitude and spatialization of the buffered live-audio. It was a very impressive performance in itself, yet no post-interaction evaluation was recorded and this use case was only considered for demonstration purposes. First, the multi-media performance used the “SoundField” platform only as a context in which the actual performance could take place and as a technical environment that could accommodate projection. Secondly, from the very beginning of the project, the decision was made not to do real-time manipulation of traditional instrumentperformance. And finally, the actual platform played only a minor and supporting role in the overall outcome of this performance.
149
9.4.6
Movie manipulation tools
For yet another set of use cases, the “SoundField”-installation was employed as a tool for real-time manipulation of streaming video-content, allowing instantaneous control over a set of live-editing parameters to a variable group of users. The SoundMov(i)es were a set of spatial movie-editing use cases that were developed throughout three iterations. All three iterations were presented as separate performances at the ending of different videoworkshops. For the larger part of the workshops, the participants worked together in small groups on short movies (or movie excerpts) that focused primarily on visual and sonic textures, less on narrative lines or dialogue. When the short videos were ready, in each of the three iterations the participants were given the opportunity to further modify/process their work in collaboration with the platform.
9.4.6.1 SoundMovie 1 The basic idea, throughout all three use cases, was that the spatiotemporal position of the participants controlled a number of parameters of how the sonic and visual playback of the movie-material was merged. In each of the iterations, 5 participants controlled the parameters of four simultaneously rendered videos. The location of four of the controller devices controlled the properties of the four individual videos through 3D-position. The other device regulated the overall blend of the four movies in one montage that was presented on the side-projections. In SoundMovie 1, only the sound pressure level of the four movies was mapped onto the height-axis.
9.4.6.2 SoundMovie 2 Throughout all three iterations, the controllable parameters of the visualizations were basically kept the same. The transparency/opacity of the visual representations were mapped to the width-axis of the room, the color saturation to the height-axis and the playback speed and direction onto the length-axis of the room. The main differences between the iterations manifested themselves in the sonic parameters that were changed between the first and second use case. In SoundMovie 2, the playback speed and direction of the sound were mapped to length axis and 3D-panning (by means of the controller device) was incorporated.
9.4.6.3 SoundMovie 3 Finally, in SoundMovie 3, the playback direction of sound and visuals was dropped to yield more fine-grained control over playback speed. Also, in this last iteration, the blended image was projected directly onto the controller device to enable faster adjustments and better, more fine-grained control.
150
Figure 9.20: Picture of the SoundMovie 3 use case.
9.4.7
Educational and Therapeutic demonstrators
A number of use cases and iterations probed the idea of implementing the “SoundField”setup in a more applied fashion rather than as a reactive, interactive or unstable art system. Because a number of workshop-teachers, working for field organizations in education, revalidation and health care, were curious whether the platform could be utilized to accommodate particular purposes, a set of applications was developed to assess the educational or therapeutic potential and tentatively establish the merit of using the “SoundField”-platform in these specific contexts. The requested setups targeted a number of specific (needs) groups – namely educational gaming for children and youngsters, and revalidation and relaxation for people with developmental disabilities. The resulting use cases were developed in close collaboration with the workshopteachers, as none of the core-members in the development-team had sufficient skills in pedagogy and psychology and were by no means properly trained in medicine to autonomously deal with the set requirements. So, as these highly-specific use cases were essentially exploratory and demonstratory in nature, they were devoid of an explicit artistic goal, so, even though highly relevant and interesting, they were not considered for evaluation. Even though remote usability evaluations with disabled people are essentially possible (Petrie et al., 2006), the participants for whom the setup was intended were not technically involved in the development process and fell outside of the target population of the setup and associated development strategy. Therefore, the decision was made not to include them, which not prevent reporting on these use cases.
151
9.4.7.1 SoundMazes The SoundMaze iterations were a group of exploratory and demonstrator use cases. Essentially, the idea for the first two iterations was based on a research project called “Sound Caterpillar”, that focusses on social music perception and musical development in children for training and assessment of sound discrimination skills, targeting perceptionaction abilities (De Bruyn et al., 2009). The latter two iterations were designed in collaboration with four groups of high school children (age fourteen to seventeen). All the SoundMaze iterations focused on the pitch and timbre discrimination abilities as well as ability to recombine musical elements or excerpts. The four use cases were designed using the same basic elements. Little focus was on the visual aspect, as the room only represented twelve static, rectangular and monochrome walls. Two dynamic spheres, manipulated by two participants, could be navigated through the 12 wall-maze. Throughout the different iterations, the 12 sonic walls produced a given sound upon collision with one of the two spheres. The first SoundMaze focused on pitch discrimination. In this use case, a set of randomized different or identical pitches were presented to groups of two users. The presented pitches were eleven two-second samples of synthesized sounds, that were less than an octave and at least a semi-tone apart. Three different combinations were used for this experiment. First, ten different pitches and one pair of two identical pitches was uploaded, then, a test was done with six different pitches and three pairs of two identical pitches, and finally, six differently-pitched pairs of two identical pitches were presented. Contrary to expectations, the fewer identical pairs were presented, the easier participants found it to find the identical pitches. In SoundMaze 2, the exact same setup was used, the only difference being that this use case focused on randomized timbres. Accordingly, the set task was to find the same instruments, as all twelve sound walls presented the same pitch in this use case. Furthermore, the same three combinations of different and identical sounds were implemented. In this use case, all sounds (max. eleven) were identically pitched samples with a duration of 2 seconds. Inversely to the pitch-based use case, participants appeared to find the timbre discrimination task easier the fewer timbres were presented. Based on comments made by some of the participants of the two previous use cases, two iterations were devised that had a more musical and practical focus. Both the SoundMaze 3 and SoundMaze 4 used the same setup of a visual representation of twelve walls coupled to a variety of samples, also activated through collision. The purpose of the SoundMaze 3, however, was to compose a tune, using individual pitches that were scattered throughout the room. In SoundMaze, the goal was to recreate a song using a number of song excerpts, which needed to be triggered in a correct order. This specific development focused on the exploration of potential educational opportunities, but in essence, the “SoundField”-platform did not provide more than a mere
152
context to enable that exploration. Therefore, given the variety of contents and tasks throughout the iterations, but also, since the use cases were not instigated by the participants themselves (but requested by their teachers), and finally, since focus group sessions were not a part of this development, the SoundMaze iterations were not considered for evaluation.
9.4.7.2 SoundGrid Similar to the previous use case, a simplified version, dubbed SoundGrid, consisted of an similarly organized sonic environment, in which six large virtual surfaces with a stringlike collision behavior, were suspended. These orthogonally positioned walls could be triggered by four dynamic spheres. The samples loaded into the walls were ambient padsounds, which, upon collision, faded in and out over the course of eight-seconds. This use to a certain extent established the therapeutic potential of the “SoundField”-platform (comparable to what is known as a "snoezelen room”) for a group of patients with developmental disabilities.
153
Figure 9.21: Table summarizing goals, features and participants for all “SoundField” use cases.
Use Case
Purpose
Participants
demonstration social interaction
Sound Objects
5
demonstration
Sound Flocks Sound Sculpture 1 Sound Sculpture 2
Sound Sculpture 3 Sound Scapes Sound Trails 1 Sound Trails 2 Sound Scapes Sound Trails 1 Sound Trails 2
1 demonstration sound manipulation demonstration sound manipulation
sound manipulation
sound manipulation
sound manipulation spatial 3Dsketching sound manipulation spatial 3Dsketching sound manipulation
sound manipulation spatial 3Dsketching sound manipulation spatial 3Dsketching choreography data sonification
Sound Path
154
Visuals
Sounds
1 activator cube 4 connected spheres 1 string 1 membrane (JAVA 3D) 1 controller object 3D Boid algorhythm (Max/MSP/Jitter) 5 colored spheres (Max/MSP/Jitter)
spheres: XYZ filtered noise string: pitch (collision) membrane: pitch (collision) (Max/MSP) flock: granular synthesis (Max/MSP/Jitter)
spheres necklace tug of war
spheres: XYZ filtered noise (Max/MSP/Jitter) spheres: X/Y/Zcontrolled playback of song-excerpts
spheres
1 dynamic sphere (1 flock)
5 dynamic spheres
5 4 dynamic spheres
4 4 dynamic spheres
4 4 dynamic spheres
4 2 dynamic spheres
2 4 dynamic spheres
4
4 colored spheres 4 sphere-trails (5”) (Max/MSP/Jitter) 2 colored spheres 2 sphere-trails (3’) (Max/MSP/Jitter) 4 colored spheres (JAVA 3D)
4 dynamic spheres
4 2 dynamic spheres
2
2
4 colored spheres (JAVA 3D)
4 dynamic spheres
1
4 colored spheres amplitude-based waveformrendering (Max/MSP/Jitter) 4 colored spheres (Max/MSP/Jitter)
4
choreography sound detection sound manipulation
Sound Tracks
Elements 1 dynamic collision object 4 dynamic spheres 1 dynamic line 1 dynamic plane
(1 sphere-creation object 1 sphere-based path 1 inspection tool 1 scope-modifying tool 1 sphere-creation objects 1 sphere-based path 1 inspection tool 1 scope-modifying tool
4 colored spheres 4 sphere-trails (5”) (Max/MSP/Jitter) 2 colored spheres 2 sphere-trails (3’) (Max/MSP/Jitter) 1 path of n spheres 1 virtual microphone scope collision and scope variation (JAVA 3D)
1 path of spheres (5”) 1 virtual microphone scope collision and scope variation (JAVA 3D)
(Max/MSP/Jitter) spheres: X/Y/Zcontrolled playback of samples (Max/MSP/Jitter) spheres: X/Y/Zcontrolled playback of soundscapes (Max/MSP/Jitter) spheres: X/Y/Zcontrolled sample-based synthesis (Max/MSP/Jitter) spheres: X/Y/Zcontrolled sample-based synthesis (Max/MSP/Jitter) spheres: X/Y/Zcontrolled playback of soundscapes (Max/MSP/Jitter) spheres: X/Y/Zcontrolled sample-based synthesis (Max/MSP/Jitter) spheres: X/Y/Zcontrolled sample-based synthesis (Max/MSP/Jitter) Path: position, index, velocity and acceleration FM synthesis for creation AM synthesis for activation (SuperCollider & Max) Path: position, index, velocity and acceleration FM synthesis for creation AM synthesis for activation (SuperCollider & Max)
Scenario
spheres umbrella
spheres
spheres
spheres
spheres
spheres
spheres
spheres
spheres
spheres fence
spheres fence
Sound Mov(i)es 1
Sound Mov(i)es 2
Sound Mov(i)es 3
Sound Dance
Sound Scene
Sound Tales Sound Maze 2 Sound Maze 3 Sound Maze 4 Sound Grid
video editing sound manipulation
video editing sound manipulation
video editing sound manipulation
choreography sound manipulation
theater performance augmented performance
theater performance sonically enhanced stage demonstration educational timbre discrimination Demonstration (educational) sound composition demonstration educational sound organisaition demonstration therapeutic sound manipulation
1 controller object (proximity) 4 dynamic spheres
1 controller cube 4 malleable videos 2 side-projections (Max/MSP/Jitter)
1 controller object (proximity) 4 dynamic spheres
1 controller cube 4 malleable videos 2 side-projections (Max/MSP/Jitter)
1 controller object (proximity) 4 dynamic spheres
1 controller mix 4 malleable videos 2 side-projections (Max/MSP/Jitter)
1 dynamic collision object 4 dynamic spheres 1 dynamic line
1 activator cube 2 connected spheres 1 string (JAVA 3D)
3
1 proximity control sphere 2 dynamic spheres 4 static spheres
1 blue sphere 2 pink spheres 4 yellow spheres (Max/MSP/Jitter)
4
4 proximity control spheres 4 static spheres
4 spheres (- 4 invisible statics) (Max/MSP/Jitter) 2 dynamic spheres 12 static walls (Java 3D) 2 dynamic spheres 12 static walls (Java 3D) 2 dynamic spheres 12 static walls (Java 3D) 4 dynamic spheres 6 static walls (Java 3D)
5
5
5
3
2
2
2
2
2 dynamic spheres 12 static walls
2 dynamic spheres 12 static walls
2 dynamic spheres 12 static walls
2 dynamic spheres 8 static walls
4 XYZ controlled amplitudes
spheres necklace
(Max/MSP/Jitter)
4 spheres: X/Y/Z controlling playing direction/speed, amplitude & panning (Max/MSP/Jitter)
4 spheres: X/Y/Z controlling playing speed, amplitude & panning (Max/MSP/Jitter)
spheres: XYZ filtered noise string: pitch (collision) (Max/MSP)
XYZ amplitude, freq & pan proximity-based activation (Max/MSP/Jitter) proximity-based activation of XYZ amplitude & panning (Max/MSP/Jitter) (collision) 2” samples (12 pitches) (Max/MSP/Jitter)
spheres necklace
spheres necklace
spheres necklace
spheres fence
spheres fence
spheres fence
(collision) 2” samples (12 pitches) (Max/MSP/Jitter)
spheres fence
(collision) 2” samples (12 pitches) (Max/MSP/Jitter)
spheres fence
(collision) soundscape-samples (8”) (Max/MSP/Jitter)
spheres fence
After this substantial presentation section, describing in detail of the use cases that were developed throughout the project, we proceed with a section on how the use cases were evaluated.
155
9.5 Evaluation In this section, we report on how the user-informed design decisions were systematically recorded. Also, the methods are discussed that were used to evaluate the user-centered construction of the use cases. At two distinct instances in the iterative process and by means of both quantitative and qualitative assessments, relevant user-feedback was gathered. The contents of the evaluation comprise the systematic evaluation of the parameters described in the conceptual model, and additionally, the two indexes are presented. The methods that make up the elements for the evaluation strategy discussed in chapter 3, are inspired by the dimension spaces for musical devices (Birnbaum et al., 2005), and in terms of AI&VRE-specific usability evaluations, the user-centered prototyping (Lundberg, 2010), evaluation criteria (Gabbard, Hix & Swan, 1999), classification system (Stanney et al., 2003), partial methodology and surveys (Bowman, Gabbard & Hix, 2002) were derived from other studies on the nexus of system design and artistic research. The applied system of indexes is inspired by Rauterberg (1996). A selection of relevant graphs is included throughout the remainder of this chapter, and all graphical representations of the “SoundField” evaluation results are bundled in Appendix 4.
9.5.1 Post-demonstration and pre-interaction evaluation First, we present the structure and the role of the focus group sessions within the iterative design approach. After the two demonstrations (section 4.4.1) that were presented to all visitors/participants, a group session was organized in which the necessary information was acquired to base the design of the use cases on. In a semi-structured group interview (which left a lot of room for spontaneous thought and free association), the prospective users’ requirements and the following contents were discussed. First, the participants were invited to give feedback on their interaction with the platform (e.g. discuss how well they understood its functionality, rate the functionality or accuracy of the platform). Secondly, information was acquired concerning the actual purpose of the group's workshop. This allowed the identification of cultural practice, artistic goals and aspirations. From that perspective, further information was gathered concerning all the variables in the setup, i.e. the sonic and visual features that were to be implemented, assignable and befitting scenarios, the social situation and interface-interaction. In some cases, an additional demonstration of existing use cases would be given. Based on the demonstrations, additions and changes would be discussed.
156
9.5.2 Post-interaction evaluation
After a whole iteration was concluded and if the use case was sufficiently developed, the results of the user-informed development were recorded using a post-interaction evaluation questionnaire. The same questionnaire was used for all the use cases and predominantly employed a six-point Likert scale. For each of the evaluated use cases, a randomly selected sample of between five and twelve evaluators were recruited (from different groups of participants, if possible) that had participated in all user-oriented stages of the development cycle. The contents consisted of two parts. In the first part, information was acquired concerning the participant, namely demographic information, artistic activity, cultural background, (art) education and technology acquaintance. The second part aimed at the full evaluation of all aspects and implementations of the use case. The first two sections pertain to the evaluation of sonic and visual parameters (i.e. discernibility, accuracy, applicability and aesthetics). The third section covers the relation between sonic and visual elements (i.e. discernibility, synesthesia, complementariness) as well as the personal focus of system-interaction (i.e. whether the dominant aspect of the use case were its visual or its sonic features). The fourth section (if applicable) dealt with social interaction (i.e. perceived level of social interaction, quality of communication, system support for social interaction and stimulation for social interaction). And finally, the fifth and last section dealt with tool manipulation and general use case evaluation (i.e. experience, enjoyment, accuracy, appearance, and functionality).
9.5.3
Complexity Indexes
Since the majority of the use cases had a considerable set of common elements and features, an indexing system (Rauterberg, 1996) was generated to represent the levels of complexity for every use case. These levels of complexity are a measure of the affordances latent in the augmented reality environment of a given use case when a participant is confronted with it (irrespective of the personal difficulties a user might have with the technology). Four separate indexes were used, namely 1) a level of sonic complexity, 2) a level of visual complexity, 3) a level of parameter complexity and 4) a level of use case complexity. The level of sonic complexity constitutes the number of affordances for sonic expression contrasted with the highest count recorded for all 22 use cases. Accordingly, the level of visual complexity is calculated in the same way, but for the affordances for visual expression. The level of parameter complexity represents the
157
total number of controllable sonic and visual parameters divided by total number of manipulatable objects. And finally, the level of use case complexity introduces the total number of controllable parameters multiplied by number of actively involved participants and divided by total number of manipulatable objects (so as a function of the impact of the level of social complexity). These four indexes are then transformed to a percentage by putting the highest value recorded for each of the indices at one hundred percent. As such, the resulting percentages can be used as a comparative measure. In essence, the level of complexity in itself has no bearings on the quality of the use case, but it is a straightforward way of quickly grasping how complicated use cases are compared to another. As the indices are derived from objectively measurable features latent in each individual use case, a visual representation of the composing parts is given in figure 9.22. The way in which these parameters are presented is closely linked to other NIME-studies using dimension spaces for musical devices (Birnbaum et al. 2005) and/or collaborative musical performance (Hattwick & Wanderley, 2012).
9.6 Results A number of general and more specific findings can be reported about the evaluations recorded over the course of the “SoundField”-project. Concerning the results of the focus group sessions, no further details will be discussed at this point. Aside from the obvious (i.e. the fact that the individual use cases were developed in a certain way because of these sessions), the transcripts of these sessions are of little significance for the evaluation of the use cases. The demographic information gathered in the questionnaires showed that of all respondents (n=89), the majority (80%) was aged between 18 and 30 years old. Three quarters of the respondents was female. A vast majority said to have a genuine cultural interest (with only 9 % of the respondents reporting they only annually attended some form of cultural manifestation). On the other hand, more than half the respondents (55%) only engaged in a quite limited range of relatively low-threshold cultural activities (e.g. cinema, pop concerts and music festivals). Half the participants had gotten a form of art education and of those, the majority had studied for over five years. Finally, only a negligible number of respondents reported to annually visit a new media performance or exhibition, yet over three quarters (76%) said to be actively involved (outside of work and school) with new (media) technologies on a daily basis, the majority (59% of all respondents) even for more than one hour of free-time involvement per day. In reference to the use case evaluation part of the questionnaire, rather than going into a lengthy discussion about the implemented elements and focusing on the specific details of 158
all specific use cases, the focus will be on larger discernible tendencies throughout the entire development process. Furthermore, it has to be noted that only the fully developed use cases were evaluated (as the demonstration use cases and first stages of the sound manipulation use cases were not formally evaluated). First, an exploratory analysis of the descriptive statistics was done (among others using a chi square and spearman rank tests). However, these provided quite erratic, unsystematic, sporadic and only marginally interesting significant relations between mostly isolated parameters (for the most part in individual and non-related use cases), so the decision was made to average the 6-point Likert-scale values per use case and to transform them into percentages. In this way, the results of both individual and groups of use cases could more easily be qualitatively compared. As a rule of thumb, when elements were rated 50% or lower, they were considered very non-satisfactory. When the rating surpassed 66%, they were considered acceptable for the majority of participants. Only elements rated 80% or above were considered very good elements, suited for further development. We continue by presenting a number of tendencies. We proceed with the results of a number of specific feature evaluations. Afterwards, a number of general tendencies will be discussed.
159
160
161
Figure 9.22: Radar plots showing n° of participants, n° of visual/sonic parameters and total n° of controllable objects for all 22 use cases in the “SoundField” project (see Appedix 4 for more detailed and individual versions of these Radar plots, as well as all graphs of the “SoundField” evaluations).
162
9.6.1
Evaluations
9.6.1.1 Intended, Implemented and Perceived Focus of Interaction
This part of the results deals with how the intended focus of interaction, conveyed in the cultural praxis and artistic goal expressed in the focus group sessions, compared to what was implemented and how the users perceived it. The intended focus of interaction deals with the use case focus being on either visual or sonic components. Social factors are ignored here as all use cases aim at serving a social purpose (with the exception of SoundPath). The implemented focus of interaction compares the number of visual with the number of sonic affordances, presented as a percentage of the maximum number of affordances of either type implemented during the project (see Figure 9.23). Finally, the perceived focus indicates the element users had to concentrate on most. visual parameters
sonic parameters
100%
100%
100% 90% 80%
75%
71%
70%
93% 88%
88%
86%
93% 86%
86%
86%
75%
63%
60% 50%
50%
50%
30%
50% 43% 38%
38%36%
40%
50%
29% 21%
20% 10% 0%
Figure 9.23: Graph showing the intended focus of interaction for sonic and the visual parameters.
In the sound manipulation use cases, the intended focus in both SoundSculpture3 and SoundScapes was predominantly on sound. In both the SoundTrails, the visuals gained in importance, but the emphasis was predominantly on sonic elements. This is in line with what was implemented. Participants on average also reported most of their attention had
163
gone to the sonic features with the exception of the SoundTrails1 use case, where both visual and sonic elements were rated as equally important. In the theatrical use cases SoundScene and SoundTales, the intended focus obviously has the emphasis sonic features, which is properly reflected both in the implementation and user evaluation of visual and sonic characteristics. In the choreography and sonification use cases, the intended focus of interaction was on sound, yet in the implementation (based on the users’ requests), the visual features for both SoundDance, SoundPath and SoundTracks were more developed than the sonic ones. The user evaluations for both SoundDance and SoundPath reported sonic and visual features to be equally important. In SoundTracks, the stress was said to be on the sonic content. In the movie manipulation the intended focus of interaction was on visual elements. Even though this was the case in the implementation of SoundMovie1, in the two iterations of this use case, the sonic affordances increased exponentially and in the case of SoundMovie2 even outranked the visual elements. In SoundMovie1 and SoundMovie3 users explicitly reported the visual elements as the dominant feature whereas in SoundMovie2 users said to have paid equal attention to both. sonic parameters
evaluation all sonic features
100% 100%
93% 86%
90%
86% 81%
77%
80% 70%
93%
69% 65%
71% 70%
77% 69%
61%
65%
86% 86% 79%
76%
61%
60% 50%
43% 36%
40% 30%
29% 21%
20% 10% 0%
Figure 9.24: Graph showing measure of complexity of the implemented sonic features compared to the general evaluation of sound (i.e. discernibility, accuracy, suitability and esthetics) for the individual use cases.
164
9.6.1.2 Evaluation of sonic and visual parameters This part of the results deals with the measure of complexity of the implemented sonic features compared to the general evaluation of sound, namely discernibility, accuracy, suitability and esthetics (see Figure 9.24). Likewise, this measure for the implemented visual features will be compared to the general evaluation of the visuals (see Figure 9.25). visual parameters
evaluation all visual features 100%
100% 88%
90% 80%
75%76%
79%
88% 75%
70% 70%
63%
60% 50%
60%
52%
69%
67%
65% 60%
59%
56% 50% 50%
58% 50%
50% 40%
38%
38%
30% 20% 10% 0%
Figure 9.25: Graph showing measure of complexity of the implemented visual features compared to the general evaluation of the visuals (i.e. discernibility, accuracy, suitability and esthetics) for the individual use cases.
Concerning the sound manipulation use cases, SoundSculpture3, SoundTrails1 and SoundScapes featured the same sonic affordances, yet only SoundScapes was rated well due to the fact that the other ones were not aesthetically pleasing. In the case of SoundTrails2, the changed social condition and reduced number of affordances compared to SoundTrails1 vastly improved the evaluation of its sonic properties. Correspondingly, we see the same tendency in the evaluation of the visual features. In the theatrical use cases, an increase in acceptance can be seen from SoundScene to SoundTales because of a reduction of sonic affordances. However, this could be due to the fact that the SoundTales use case implemented more discrete and recognizable sonic content. The evaluation of the visual contents of these both use cases is only marginally interesting, since they were intentionally underdeveloped.
165
In the choreography use case SoundDance, despite the easily understandable sonic features (i.e. low level of sonic complexity), low levels are recorded for all the constituent elements of the sonic evaluation. Additionally, the visual components get an even poorer rating, even though the visual affordances offer ample opportunity for interaction. One possible explanation for this may be that, given the fact that the performers' head movement was captured, it hindered their ability to simultaneously enact (i.e. generate) and perceive their artistic output (Leman, 2008). Another plausible reason might be that, since the use case wasn't overly complex, it offered insufficient challenge to the participants, which in turn made the contents no longer appealing (Anderson, 2004). AVG S&V
RELATION SOUND & VISUALS
100% 90% 80%
74% 70%
78% 78%
73% 73% 70% 70% 69% 65% 64%
70% 60%
67% 64%
59% 56%
73% 71% 63% 61%
73% 60%62%
65%
50% 50% 40% 30% 20% 10% 0%
Figure 9.26: Graph showing the evaluation averages for both sound & visuals compared to how well the relation between the two was perceived (for individual use cases).
Concerning the sonification use cases, a doubling of affordances (analog to the increase in number of participants) appears to positively influence the evaluation for both sonic and visual features. In the movie manipulation use cases, an increase of implemented affordances appears to be only negatively influencing the appraisal of the sonic features. In the assessment of the visuals, we see the same tendency. A first hypothesis is that, in essence, the “SoundField” platform is not particularly well-suited for this purpose, as accuracy and suitability values appear to be more problematic than the esthetic qualities and the discernibility. On the other hand, as the two iterations of the SoundMovie1 use
166
case became increasingly convoluted, it might just be that these two use cases were too complicated to fully apprehend. And in closing, figure 9.26 shows how well the relation between sounds and visuals was perceived compared to how good they were rated overall.
9.6.1.3 Implications of social factors on use case complexity and on use case evaluation This section of the results deals with the quality of social interaction and how social factors further influenced the complexity of the use cases (see Figure 9.27). Accordingly, we look at how the use case-complexity (parameter complexity combined with the social factors) influenced the general appraisal of the use cases (see Figure 9.28). Use Case Level of Complexity
evaluation social interaction 100% 95%
100% 90% 80% 80%
79%
77%
73% 62%
60%
40% 30% 20% 10%
76% 70%
67%
70%
50%
79% 76% 76%
54%
59% 48%
41%
56% 51%
38% 29%
29% 19%
0
0%
Figure 9.27: Graph showing use case level of complexity compared to the overall evaluation of the social interaction indicators.
Three of the sound manipulation enabled four participants to interact with one another. In addition, SoundSculpture3, SoundTrails1 and SoundScapes featured the same number of sonic and visual affordances. SoundTrails2, that accommodated only two participants, correspondingly had fewer sonic and visual affordances. The SoundTrails2 got an adequate rating for social interaction potential and the features of the use case were rated as quite satisfactory in the final evaluation. In the chronology of the other sound manipulation use cases, a decline in the evaluation of social factors and overall use case
167
quality could be discerned between SoundSculpture3 and SoundTrails1. The use case evaluation then vastly improved in the SoundScapes use case, the evaluation of social interaction improved less significantly. Since interaction strategies and controllable parameters remained mostly unaltered between these iterations, the reason for the fluctuation in the evaluations is linked with the used raw material that enabled the interaction. Use Case Level of Complexity
USE CASE Rating 100%
95%
100% 90% 80%
78%
82%
81%
30% 20%
76% 67%
60%
40%
79% 72%
67%
70%
50%
79% 72%
76% 76% 73%
76%
65%
54% 48% 41%
38% 29%
29% 19%
10% 0%
Figure 9.28: Graph showing use case level of complexity compared to the overall assessment of the use case.
Concerning the choreography and sonification use cases, the number of actively involved participants was relatively low (for SoundDance, SoundPath and SoundTrails respectively three, one and two participants). In SoundDance, the number of sonic and visual affordances per participant was quite low, for SoundPath they were moderately high and for the SoundTracks use case they were the highest of all the recorded use cases. On the account of the low numbers of the participants, the use case complexity was quite reasonable for all three use cases. In spite of this social transparency and the reduced complexity, the evaluation of social factors for SoundDance was not satisfactory, for SoundTrack it was acceptable but by no means spectacular. For SoundPath, which was operated by individual participants, this measure was not applicable. Oppositely, the overall evaluation of the use case functionality for all three use cases was positive. Although quite similar in terms of properties as SoundDance, the theatrical use cases display quite different results. The number of actively involved participants for both SoundScene and SoundTales is quite moderate (namely three). Also, the number of
168
affordances was quite limited and, as stated earlier, the visual elements were kept as basic as possible. Inverse to the low amounts of parameter complexity, we see the highest levels for the relation between sonic and visual elements in these use cases and for the evaluation of the sonic properties. The levels of use case complexity are relatively low compared to other use cases, yet, the evaluations of the social interaction and of the use case evaluation are among the highest that were recorded throughout the project. The recognizable nature of both use cases' sonic material, the sparseness of visual elements and the very understandable activation thresholds may have largely contributed to these positive evaluations. And finally, in reference to the movie manipulation use cases, another trend can be observed. SoundMovie2 and SoundMovie3 have a large number of actively involved participants and large numbers of both sonic and visual affordances, which results in relatively high numbers of parameter complexity. Given the high number of participants, the social dimension of these use cases results in the highest levels of use case level of complexity. The evaluation of social interaction in SoundMovie2 is rather poor, yet in SoundMovie3 it is vastly improved. In reference to the use case evaluations, in spite of the obviously convoluted nature of the representations, all three SoundMovie use cases are rated quite positively.
9.6.2 General tendencies: However difficult it is to make assessments that are valid for all use cases, there are some broad findings that can be presented. We start by an assessment of the interaction design principles (Norman, 2010) stated earlier (see section 5.2). In terms of visibility, the (proper) interaction strategy was immediate and self-explanatory in all use cases and concerning the properties of the visual representation, evaluations on average slightly higher than the similar parameters for sonic representations. Concerning consistency, the latent affordances and visual cues appeared to be easily understood and the basic operation (i.e. object behaviors and implemented scenarios) was successfully kept practically identical throughout the project. On the subject of scalability, there are two main findings to be reported. On the one hand, the system performed as envisioned, making the transition from a computer-based demonstrator multiple differently sized setups (defined by the capacity of the camera array) successful. Moreover, the inclusion of external controller devices (e.g. I-CubeX, WiiMote) did not considerably effect the basic operation of the system. On the other hand, in use cases that made the system available to large numbers of participants through the highest number affordances (e.g. SoundMovie3), the repercussions on accuracy, discernibility and esthetic appraisal were that obvious that the resulting confusion outweighs the opportunities for artistic expression. With respect to reliability, even though the platform (Diniz et al.,
169
2012) functions properly, due to technological restrictions (e.g. limitations of the trackable surface and number participants, limitations of trackable number of virtual objects due to occlusion, and crashes of the Motion Capture system), the reliability of the entire system is still questionable. Regarding discoverability, there is a visual prevalence to be reported, even though most use cases predominantly focus on sonic parameters. Discoverability was acceptable mainly as a consequence of the clear visual feedback. This is likely to be due to the fact that, upon entering the immersive environment, the visual cues were already present, whereas the sonic elements required interaction to be explored. And finally, in connection with discernibility, the visual elements appears to be satisfactory, yet the number of controllable sonic parameters as well as the social complexity of the use case seem to have a negative influence. Subsequently, when considering all evaluated use cases, there are a number of verifiable tendencies that can be reported. A first set of tendencies is linked to the relationship between the intended, the implemented and the perceived focus of interaction. In principle, the intended, implemented and perceived focus conform rather well (i.e. the expressed goal of the use case, the implementation and the evaluation thereof correspond, notwithstanding some exceptions). The fact that in some cases the subservient feature was quite excessively developed, could not pull focus away from that intended purpose. Moreover, whether or not a use case was found to be well-suited for the expressed purpose (i.e. functionality), appears to be related to the fact that the use case was perceived as entertaining, much more than to the accuracy or to the esthetic qualities. And finally, whether or not participants found features esthetically pleasing and the mapping suitable, appears to be linked more to the manipulatable contents of the specific use cases than to the actual controllable parameters. A second set of tendencies is related to the relationship between sonic and visual content and action. On average, the evaluation scores of the visual feedback were consistently better than those for the sonic features, aside from the cases in which visual features were underdeveloped (i.e. the theatrical use cases). The appraisal of sonic and visual discernibility and accuracy appear to be closely linked together. Whether the relationship between gestural action and the resulting sounds and visuals could easily be discerned, contrarily seemed to coincide with the participants' assessment of sonic and visual components being complimentary to one another. Moreover, the feeling of synesthesia (or how well the senses worked together) corresponded quite accurately with the combined evaluation of sounds and visuals. However, there is an exception to this, namely the movie manipulation use cases, where the users' appraisal of the relation between sounds and visuals was dominated by the users’ poor evaluation of the visual parameters. A third set of tendencies is associated with the assessment of quality of use and experience compared to levels of complexity and the impact of social factors. First, accuracy and discernibility (and hence the appraisal of the use case quality) decrease as
170
the number of controllable sonic parameters goes up. Furthermore, the quality of social interaction appears to be associated with how suitable participants consider a use case to be for communication (once again, with the exception of the movie manipulation use cases. Then, there appears to be no demonstrable effect between use case complexity and the perceived quality of experience, yet both in number of affordances and participants, there appears to be a threshold of participants that can be involved with the same basic task. Moreover, the ideal number of participants throughout this project appears to be between three and five; when a use case has less than three participants, users perceive the social dimension as tedious, whereas, when five or more participants are simultaneously involved with the use case, real discoverability and discernibility issues start to appear. And finally, as the social complexity of a use case goes up, the quality of social interaction goes down (see Figure 9.29).
5 4 3 2 1 0
N of participants
quality of social interaction
Figure 9.29: Graph showing number of participants compared to Quality of Social Interaction.
9.7 Discussion and Conclusion There are a number of relatively new and different elements to this approach in comparison to similar types of development on the nexus between art and science. A first novelty to consider, is the fact that this project aimed to be rigorously open-ended with a flexible style of extreme programming. This open-endedness can be attested by the fact that that one of the few things that was imposed, was the immersive/interactive
171
environment itself. External elements could be introduced into the system in a broad range of cultural practices, and a lot of the features were developed collaboratively and from the ground up with the workshop-participants. As such, 22 use cases in 7 different instructional/educational/artistic realms could be created. Secondly, it is not entirely common that a team is simultaneously working on the improvement of the software, the implementations in the artistic genres and cultural fields, and the development of sonic and visual representations. The development of all these elements ran more or less concurrently (and iteratively) with the pre-interaction FGs, and the post-interaction evaluations, making this a very hands-on and real-time project. A third and last minor unconventionality is the fact that, contrary to typical lean development, the use cases and their iterations are rather well-documented. The SF-evaluations show that, irrespective of the majority of the culture participants’ initial and attested dismissive convictions, test subjects generally experienced sounds and visuals, the relation between them, the social interactions and the enjoyment overall to be quite positive – opposite to what was suggested by the mixed results obtained with the “WMY”-survey, the “HaaO”- and “Lament”-installations. Three reasons can be found for this shift in attitude. First, the active involvement and user-centered design aided in eliminating the opacity of unstable media installations that was discussed in chapter 6. By granting the audience insight in the inner workings and incorporating them in the eventual operations and functionality, the potential for hesitancy, confusion or fear all but dissipates. Secondly, the intuitiveness of the system vastly reduces the proneness for error-making and, as a consequence, the boredom or frustration that may ensue from it. And thirdly, intentional audience involvement manifestly reduced the threshold for participation in this project. By actively inviting a few groups of participants, end-user enthusiasm snowballed into the reality that more groups of culture-enthusiasts requested workshops revolving around the “SoundField”-system, and a larger and broader degree of participation then was originally envisioned, was achieved. This approach of offering the opportunity to partake in a participatory development project, all but nullified sociodemographic predetermination for participation, and prompted the fact that “SoundField” vastly exceeded expectations in terms of the specific population cohort it was intended to appeal to – both elements being quite uncommon to participation research in the cultural industries. So, as this type of project has shown the potential for swaying larger potential audiences then anticipated to get involved, the procedure described in the conceptual model may well be a gateway to broader and more diverse participation. A regrettable oversight in the project, is that practically none of the developments were seen through to their full extent, so the full potential of the system cannot be attested based on the underlying results. The short span of the artistic workshops combined with the emphasis of the project being on the collaborative nature of the development, brought about the fact that insufficient attention was given to the iterative aspect of the design approach (a couple of notable exceptions being the SoundSculpture and SoundMov(i)e
172
iterations). Over the course of the entire project, more consideration was given to the scope and artistic range of the system, rather than to depth of a singular development. Even though the decision to focus on breadth and amount is essentially justifiable, the absence of a more thoroughly refined use case does tarnish the otherwise overall quite positive outcome of the project. So, contrary to the prevalent rather conservative attitude in large parts of the artistic sector, strong properties of this project like “almost real-time modification” and “the potential for in-house customization, evaluation and adjustment”, could make this type of system, besides innovative, an interesting, worthwhile and perhaps even cost-efficient addition to the catalogue of a number of experimental and technology-eager institutions. Technological evolution has always been a natural ally for the cultural and creative industries at the forefront of artistic and broader societal innovation. With growing, increasingly media-literate audiences showing new-found enthusiasm for the potential of the technology and commercially available VR-technologies catching up to the challenges set forth in the project, this realistically is one direction wherein the industry could evolve. That by no means implies that the sector as a whole and its programming should adapt to this new – almost – reality, but by means of more AI&VRE artistic incubation projects, the public opinion might well be catching up to the notion that it will eventually be acceptable for (interactive) EMMT-technologies to be presented and embedded in preexisting cultural contexts, either technologically expanding typical programming or experimentally augmenting performance, much in the same way as Björk showcased with the ReacTable as early as 2007 (Jordà, 2003). The experiment described in this study effectively provides the keystone for the conceptual model, bringing together all the methodological elements discussed, tried and tested throughout the previous chapters into one consolidated and collaboratively designed art project.
173
Part 3:
Epilogue, Discussion & Conclusion
175
The third part of the thesis renders the final conclusions of this research project. The tenth chapter sets out with an epilogue in which a brief overview is given on how the state of the arts has evolved since the end of the project (as the research ended in 2012, but the research field has obviously moved on), and attempts to consolidate the implications of the research studies with the realities of the Flemish cultural-creative industries and media literacy levels of Flemish youths. Chapter eleven continues by discussing the strengths and weaknesses of the conceptual model throughout the various stages of the project. Also, it comprises the evaluation of the hypotheses, how the assumptions that were made held up, and how eventually the research questions can be assessed. Finally, the twelfth and last chapter concludes with the most important findings of the project. Also, an overview of the implications on cultural production, artistic research policy and an outlook on future research beyond the scope of the “SoundField”-project are offered.
177
10
Epilogue
It is not opportune to leave research results lingering for several years in academia; especially not in an industry that has unofficially – and perhaps somewhat begrudgingly – adopted the phrase “publish or perish” as a motto. So, naturally one would assume that when the report of a four-year research project is about five years overdue, there are bound to be negative repercussions on the validity of the body of work; quod non! The substantial detainment could have conveyed a reasonable threat to the timeliness of the results, the validity of the initial findings might have been compromised by the delay, or the significance of the project might even essentially have been nullified altogether. Conversely, when comparing the research project to more current similar academic efforts, both the technologies used and the accompanying studies do not appear all that much dated and have aged surprisingly well. Moreover, there is also some redemption to be found in the fact that during the ensuing olympiad, additional research could be conducted that was in more than one way beneficial to the initial cause of the thesis; not only was there ample opportunity to investigate other stakeholders and aspects of the cultural-creative industries during this period, it created the possibility to second guess the actual technological (r)evolutions that had actually taken place in both the sector and the broad socio-cultural context. Also, an all-important additional focus could be given to crucial competences like media literacy and the ins and outs of current forms of technology-informed audience participation (e-culture), all of which are essentially integral parts of the methodological jigsaw (cf. supra) that was discussed in the introduction, but that were at the time of the execution of the studies in no shape or form core to the focus of the research project. And finally, because a lot of the steps in the initial research process were taken in a somewhat erratic time frame, this chapter tries to offer a bird’s eye view on the subject by elaborating on the broader context. It does this by briefly discussing a number of studies from various other fields of sociology and communication science, namely cultural studies, cultural sociology, e-culture, cultural policy research and media literacy studies.
179
10.1 Digital Culture Participation and Audience Engagement This first two studies discuss the use of digital communication devices for cultural purposes. The Strategic note for the Vision on the Arts (2015) from Flemish Minister Sven Gatz (as well as those of his predecessors Joke Schauvlieghe and Bert Anciaux) see(s) digitization in the cultural and creative industries as a crucial factor in the development of a dynamic, efficient, accessible and internationally competitive sector. The minister refers in various strategic and operational objectives of the Vision Statement for the Policy of his legislature (2014-2019) to the positive impact of digitization of culture, but also, emphasis is in part on the added value digitization can entail for the institutions themselves. Research center iMinds-SMIT-VUB (in collaboration with CEMeSo-VUB) therefore analyzed the properties of the digital resources and online accommodations of the Flemish and Brussels cultural sector within the context of the research lines for e-culture and digitization of the third policy research center (i.e. PRC1) of the Flemish government. Also, the main findings for e-culture participation degree from the side of the Flemish culture audiences over the last three PRCs and coinciding PaS-waves (respectively from 2004, 2009 and 2014) are discussed in short in this chapter. This first section deals with the evolution of the online participation degree of Flemish culture participants, the following section summarizes the adoption of online and social media by the various segments of the cultural industry.
10.1.1 E-culture and Digital Participation The profile of the online culture consumer is made up by information concerning sociodemographic properties and involvement with e-culture. This term harnesses five factors: [1] actively accessing information about culture, [2] the online acquisition or purchasing of cultural products or services, [3] experiencing cultural content or performances by means of screens, [4] the creation of one’s own (cultural) content by using digital technologies, and [5] the use of mobile devices for producing or accessing (information about) cultural artefacts. Needless to say, this definition has undergone a number of slight alterations since it was coined around the turn of the century, due to the fact that the
1
These studies are presented on December 4, 2015 during the Participation Survey seminar organized by the Department of Culture, Youth, Sports and Media. The policy summaries are published subsequently by: http://www.steunpuntcultuur.be/
180
digital technologies, that enable e-culture, themselves have changed considerably in terms of functionalities during this period (bLievens, Siongers & Waege, 2015; Deweppe, 2016).
Figure 10.01: Evolution of digital participation (from top to bottom): [1] culture consumption by means of screens, [2] accessing cultural information, [3] acquisition of cultural products, and [4] creation of (cultural) content [source: PaS 2009 & 2014].
While the figures for traditional outdoors cultural participation have stayed roughly the same over the last five to ten years, the numbers for digital culture participation during that same period show that a dramatic increase can be attested for the second consecutive wave in a row (i.e. in between PaS ’04 and PaS ’09, and between PaS ’09 and PaS ’14).
Age
Culture Information
Cultural Products
Cultural Experience
2004
2009
2014
2004
2009
2014
2004
2009
2014
15-17
73,1%
87,8%
88,0%
61,3%
72,2%
96,4%
46,3%
58,8%
97,1%
18-34
53,4%
81,9%
89,4%
34,4%
69,5%
93,2%
25,3%
32,3%
97,3%
35-54
30,7%
66,9%
74,7%
15,7%
50,6%
70,2%
14,2%
14,8%
90,4%
54-65
15,7%
41,5%
48,6%
5,5%
28,9%
39,7%
6,6%
8,0%
87,6%
65+
3,7%
14,6%
27,0%
0,5%
8,7%
16,6%
1,1%
11,5%
84,5%
2845
3146
3949
2845
3146
3949
2845
1816
3131
N
Figure 10.02: Digital participation rates over the last ten years presented by age cohort for [1] information, [2] product distribution and [3] cultural experience in Flanders and Brussels [source: PaS 2004, 2009 & 2014].
181
This increase of e-culture participation lies somewhere between 6% and 34%, depending on the e-culture element measured (see Figure 10.01). Moreover, factors such as gender and education play an ever smaller role in the digital participation behavior, and also the differences between the age levels slowly decline (see Figure 10.02). Finally, as is the case with other forms of participation as well, the data presented increasingly pertains to so-called cultural omnivores. Within their general consumption patterns, their cultural habitus is becoming bigger (i.e. they participate more frequently and more intensively), it is becoming broader (i.e. they participate more eclectically) and less selective (i.e. the distinction between high and low cultural repertoires is dissolving).
10.1.2 Mapping of Digital Technology-Use in the Cultural Industries This second study discusses the use of online information, reservation and promotion channels used throughout the organizations of the Flemish cultural industries. During the last two decades, most cultural institutions gradually adopted and incorporated digital technologies in their daily operation and communication strategies. Alongside the core business of providing opportunities for classical outdoors cultural participation (i.e. the visiting of a receptive venue by an audience for the real-time and offline experience of an artistic event), the systematic advance of a parallel online offer by means of various digital platforms can be observed. Initially, these digital cultural resources were little more than a digital addendum to the physical institutions (the digitized aspect consisting for the most part of providing additional information (date, description, ticket availability, etc.) of the offline cultural activities in the program. Through the increased and enhanced capabilities for online participation (in terms of production, distribution, presentation, archiving and consumption of cultural products), institutions are nowadays able to disseminate their collections or programs better and to a larger (target) audience, and through the use of digital technologies, even able to achieve an improved, more profound and more actively involved experience of the arts. Comparable to similar international studies2, research center iMinds–SMIT–VUB conducted a study in which the use of digital communication and promotion channels by arts organizations in Flanders and Brussels was examined. Additionally, between
2
These include NESTA report of the British Arts and Humanities Research Council (http://artsdigitalrnd.org.uk/wp-content/uploads/2013/11/DigitalCulture_FullReport.pdf) and a study by the American PEW Research Centre (http://www.pewinternet.org/files/oldmedia/Files/Reports/2013/PIP_ArtsandTechnology_PDF.pdf). In the latter study, the status, content and adoption rate of digital technologies and communication channels at 1258 National Endowment-agency favored cultural organizations from the United States mapped.]
182
September 2013 and May 2015 online communication channels, tools and activity for different types of organizations in the cultural and creative industries, were mapped. Various online repositories of the overarching associations (such as Muziekcentrum.be, BAMart, VTi, and FARO) provided the basis for this dataset, which was built specially for this study and included a total of 1458 institutions. Of those entries, the 1230 officially registered and locally subsidized institutions were retained within eight segments of the cultural sector for the analysis (see Figure 10.03).
Figure 10.03: Pie chart showing distribution of data entries according to institution type (n=1230).
In terms of geographical spread, 25,6% of the organizations mapped in the dataset are from Antwerp, 11,2% are from Limburg, 17,3% are East-Flemish organizations, 13% are from Flemish Brabant, 14,8% are West-Flemish and 13,6% are from Brussels.
10.1.2.1 Use of websites by Cultural Institutions About 79% of the organizations manage their own unique website, about 15% have a page on a website hosted by the city or town, and close to 7% of the institutions has neither. In terms of information offered on those websites, about 93% of organizations provide contact information and more than 80% provide ticket information and an events calendar. About two thirds of the institutions maintain a blog, give a history of the organization and put social media links on their sites.
183
10.1.2.2 Media and Online Content The results show that only 58% of the organizations incorporate (links to) pictures on their website. Nearly a third allows for online ticketing and a quarter host or link to video content. The number of institutions offering or linking to audio streaming services or (paying) downloads for this sample is below 5%.
10.1.2.3 Adoption and Use of Social Networks Nearly 80% of all cultural organizations already used Facebook-accounts3 at the time of the data-gathering. For Twitter4, this number was around 40%, for YouTube-accounts the number dropped to about 30%. Other channels5 – at the time of the mapping – remained well below 10%. Just under 70% of organizations had Facebook-accounts with a range of 500+ users; 60% of institutions have at least 250 followers on Twitter.
Figure 10.04: Relative frequency6 of posts placed on social media by various types of institutions.
3
93% of the Theater venues, 84% of the Culture centers, 82% of the Festivals, 78% of Music institutions, 77% of Heritage institutions, 72% of Cinema and Film organizations, 61% of Museums and 54% the Amateur organizations 4 57% of the Festivals, 51% of the Theater venues, 43% for Cinema and Film organizations, 38% of Music institutions, 32% of Museums, 28% of the Culture centers, 13% of Heritage institutions and 9 % of Amateur organizations 5 Flickr, Instagram, Vimeo, Google+, Foursquare, Linkedin, Pinterest, SoundCloud, MySpace, Wordpress, Tumblr, Vines, Snapchat, Vi.be ... 6 Absolute frequency extrapolated from a reference period spanning approximately two months to four relative categories: [1] rarely (x ≤annually), [2] sporadically (x ≤monthly), [3] occasionally (x ≤weekly), regularly (x > weekly)
184
Overall, the frequency and intensity which organizations employ their social media outlets with, varies greatly. Generally, the larger institutions that have an active and multichannel social media presence, eagerly try to take advantage of this cheap and effective resource. Over 80% of organizations with (an) account(s) on social media regularly put up posts; more than 50% of the institutions publish information concerning their activities on a(n at least) weekly basis (see Figure 4). Also, for the promotion of events, sharing of news or spreading of audio(-visual) content, Facebook is the favorite platform for the majority of the organizations; close to 90% of the organizations active on social media make fairly regular use of it for this purpose, and according to 30 specialists – interviewed in the context of this study – active in various artistic fields, it is quickly shaping up to be the prime channel for direct target audience communication. As far as interaction and participation with audiences and the broad public is concerned, in nearly 50% of cases, no more than 5 likes are returned on news posted by the organizations. In about 95% of the cases, less than 5 user/follower reactions can be documented to posts initiated by the organization. Also, from the part of the institutions themselves, social media usage is rather passive and conservative. Some 70% of the organizations never interacts actively with its followers; barely 1% do so regularly (see Figure 5). And finally, providing authentic digital cultural content (e.g. online performances or exhibitions), is a practice common to barely 2% of the institutions.
Figure 10.05: Use of Facebook and Twitter by cultural organizations for promotion of events and interaction with followers/audiences.
185
10.1.3 Discussion, Conclusion and Policy Recommendations When cross-referencing some of these results to other international studies, the Flemish figures for e-culture, new and social media use are rather conservative and low in comparison. Some substantial regional differences aside (e.g. whether or not an organization has a website varies between 70% and 84.5%, solely based on the region of Flanders), the Pew study on “Digital technologies for Arts Organizations” shows higher adoption levels for new and social media across the board. Granted the fact that a different methodology was used (i.e. telephone survey), this should fully cover the fact that only 1% of the cultural institutions in the United States do not have their own website, whereas in Belgium that figure is as high as 7% (21% if not being the sole proprietor of the website is also included). Also, the results for both studies were obtained roughly in the same time-frame – the Pew even predating the mapping by a few months – and targeted both the same types of institutions, with the PEW-study covering approximately 3000 NEA-endowed organizations, and underlying study nearly half as much. For both studies, the obvious conclusion is that Facebook appears to be most prominent social network, yet for the Pew-study adoption rates for various social media platforms appear to be well above the figures for the local study (i.e. on average 20% higher for organizations in the United States then the Flemish ones), and also the diversity of different social media outlets is not at all comparable. In sum, the following findings can be postulated: first, the last fifteen years have seen a spectacular increase for the use of digital technologies in the cultural industries, both on the supply side as in terms of consumption: the steady rise of adoption rates for ICTapplication use – both from the side of the sector as from the side of its audiences – will undoubtedly continue over the following decade(s), permeating all levels and niches of the cultural and creative industries. Secondly, e-culture participation has risen significantly in comparison to traditional out-house culture participation: although the "digital divide" raises new challenges and barriers, the digital counterpart of a cultural experience is growing in importance, and arguably might well be more accessible to a larger part of the population. Side-effects of this evolution include the importance of cultural hierarchy moving to the background, and the difference between 'high' and 'low' art forms being of lesser importance in a digital environment. An increase in cultural participation, therefore, does not necessarily indicate a widened and intensified cultural palette for all, though increased accessibility and availability are an accommodating factor in lowering the ‘high’ culture threshold. Thirdly, on a local level, the adoption of digital (communication) technologies within the cultural sector, is a slow and conservative process (when compared to an international context). Representatives from different fields of the sector indicate that this hindrance is due mainly to a lack of training, vision, incentives and structural governmental funding. 186
With the exception of the large and well-established ones, most organizations are unable to allocate sufficient resources to deploy specialized personnel solely to facilitate an optimal e-culture supply and/or social media presence. Policymakers could play a significant role in overcoming that impediment. Fourthly, according to experts in the sector, the current detainment in the full e-cultural dissemination and disclosure of the materials produced by the Flemish cultural sector, is mainly due to the lack of a useful, transparent and balanced legal framework for the release of predominantly copyrighted content. Since e-culture can be a stepping stone towards the more traditional forms of culture participation, a facilitating role in this matter could be foreseen for the policymakers and government. Fifthly and lastly, the sector – at present – avails itself insufficiently of digital communication tools for the purpose of audience interaction. In spite of a vast range of additional possibilities (e.g. decisions concerning programming, crowdsourcing, etc.) social media opportunities are at present not yet well established in large parts of the sector. For now, the use of social media by the cultural industries remains largely focused on content that is passively consumed, rather than participatory. In other words, for now social media are a highly popular communication tool, but as an enabler for two-way interactions with the audience, at present, the cultural sector has yet to take the next step.
10.2 Media Literacy as a Requirement As the first two studies of this chapter dealt with an alternative and digital reality for the cultural industries, namely the impact of digitization giving way to the advent and growing importance of e-culture, the exponential rise of digitized forms of participation and the adoption of new and social media technologies from the institutional and the audience side alike, this second part of the chapter is concerned with media literacy levels of the Flemish population. A few excerpts are presented from two studies done within the framework of the PRC Media of the Flemish government, namely one study that focused predominantly on youngsters and the evolution and development of critical and strategic literacy, and a section of the PaS 2014, which zones in on a number of general media competences.
187
10.2.1 About Media Literacy Media is a container term that covers a vast number of concepts; within that pluriform set of meanings, three main topics are conveyed – the so-called triple articulation of media (Hartmann, 2006), defined as follows: 1) Media as a category in the workforce (e.g. journalists, editors, etc.) and the encompassing professional sector associated with it; 2) Media as the collection of written and audiovisual information, communication and entertainment sources; 3) Media as the repertoire of platforms, channels and devices through which these sources and contents can be consulted. The concept of media literacy harnesses a set of skills that need to be developed or attained in order to be able to work with state-of-the-art digital technologies, and by extrapolation to be fully included in a consequentially digitized society. Current debates on media focus predominantly on comprehension, utilization and possession of media outlets and devices (Hobbs, 1998; Livingstone, 2004), and on a more practical level on communication and creation through media, on media-multitasking and on the strategic abilities to use certain sources or devices (Van Deursen, van Dijk & Peters, 2011). Most studies apply a tool-based and/or activity-based explanatory model for measuring such competences. Notable, mainly quantitative data collections in this field, include the “Digimeter”-reports (De Marez et al, 2009-2015), the Participation Survey (aLievens, Siongers & Waege, 2015) and the SVR study (Moreas & Pickery, 2011). Generally, demonstrable correlations are sought between the proficiency certain tasks can be performed with and the socio-demographic properties of the executant – such as education or age cohort (Picone & Deweppe, 2015), as a basis for policy recommendations on einclusion. As such, multivariate analyses and projections can be made on the magnitude and evolution of the "digital divide", and the digitally excluded profiles of the population (Molina, 2003; Mariën & Vleugels, 2011). The Research Council of the Flemish Government synthesized the working definition in 2011 as follows: "Media literacy is the ability to access materials from different sources, to identify it, to understand it, to interpret it and evaluate it, and to communicate through them and produce content. Literacy as such is a continuum: it refers to the degree in which people have learned to achieve their goals in a digital way, to develop their knowledge and potential, and to participate fully in their community and in their social environment through media" (Moreas & Pickery, 2011 – translated from Dutch).
188
10.2.2 Competence Models and Conceptualization Somewhat rephrased, media literacy then encompasses the set of knowledge, attitudes and skills needed to participate actively, effectively and critically in a complex, fundamentally changing and increasingly media-driven world. To measure this in practice, various international studies operate on the basis of competence models. Knowledge-center “Mediawijs” is currently involved in the international comparison of various models of 27 EU-member states, as well as the construction of a consolidated “internationally valid” model, but in the absence of such a consolidated model, our work on “Critical and Strategic Media Literacy in Flemish Youth” was based primarily on Van Deursen & Van Dijk (2009a). Our model, additionally derived from the long-standing Dutch media knowledge center Mediawijzer.net7, took into account 10 competences, each defined as 5 levels of proficiency (ranging from complete neophyte to expert – more elaborately described within each competence). Furthermore, the competences can be divided into various categories based on the required interpretation: a first categorization considers the action involved (i.e. use, communication, comprehension or reflection), the second considers four types of media skills (operational, functional, critical and strategic), and finally, another categorization considers an activity level (Passive, Active, Reactive or Interactive). Figure 10.06: Table showing the competence model, skill, activity level and competence type.
N° Competence Skill type Activity level Action Passive Use 1 Orientation in media-environment Operation Operation Active Use 2 Operating devices Function Active Use 3 Information management Function (Inter)active Communication 4 Content creation Function (Inter)active Communication 5 Social Networking Service Use Critical Passive Comprehension 6 Awareness Medialization Critical Passive Comprehension 7 Discernment Media-production Critical Reactive Reflection 8 Consciousness Media-coloration Strategic (Re)active Reflection 9 Intentional Media-use/choices Strategic Passive/Reactive Reflection 10 Self-reflection Media-usage [based on the Van Deursen & Van Dijk (2009a) and MediaWijzer.net7 models]
7
Competence model “Mediawijzer.net” (2013, May) Retrieved from: https://www.mediawijzer.net/vanmediawijzer-net/competentiemodel/
189
10.2.3 Relevant Findings The first study quantitatively and qualitatively documented the media habits and skills of 60 children between the ages of 9 and 16, gender-balanced and equally distributed over 3 age groups (i.e. 9-11, 12-13 and 14-16, conform the EU-Kids online guidelines). Despite its relatively low n-value, the study rendered some interesting and previously abstruse insights. Without going into much of the detail – as it is not the main focus of this thesis, the age groups comprised in this study cover the most important formative years in the media skill development and adoption process in children and adolescents. Along with the age differences conveyed in the three groups, heavily dissociate and transitional media profiles and repertoires can be attested over this 7 year period. First, an obvious difference in repertoire can be distinguished between the youngest and oldest group. Whereas a classical device (i.c. TV) is the dominant interface in the first group, more multi-purposed interfaces like computer (for the middle age group) and smartphones (for the oldest group) become the prime media creation/consumption devices (Figure 10.07). Also notable is the fact that once over the age of eleven, there is a large and rapid increase in both media time spent per day and media repertoire, and from the age of twelve onward, simultaneous use of multiple different media devices start occurring (more) frequently (i.e. media-multitasking).
Figure 10.07: Graph showing the evolution of the most used interface for the three age categories included in the study (i.e. 9-11 years old, 12-13 years old, 14-16 years old).
190
Summarizing, twenty-first century youngsters display [1] a complex, varied and systematic media repertoire (with an explicit interest in innovative features), [2] demand to have ubiquitous access and instantaneous results to their digital actions, and [3] showcase a confident, audacious, efficacious and strategic interaction with their media devices. When compared to other age cohorts in the broad population, the youngsters surveyed in this small sample exhibit a much higher degree of media literacy than their older counterparts. Though very restricted in terms of space in the 70’ CAPI-questionnaire, the 2014 Participation Survey (aLievens, Siongers & Waege, 2015; Deweppe et al., 2015) also probed a number of media literacy skill indicators, albeit in a very applied and concise way (Figure 10.08). Figure 10.08: Table showing results obtained for eleven of the media literacy indicators included in the 2014 PaS.
(n=3949) Find online info Tape a TV-program Buy tickets online Share a photo on SNS Download an app Make an online call Make an online post Make reservations Upload a video Maintain a blog Adjust privacy settings
Don’t know (how) 19,1% 29,5% 33,8% 45,4% 48,1% 45,9% 50,5% 54,7% 60,8% 72,2% 44,1%
Know / Could 7,3% 6,4% 13,5% 9,6% 8,8% 11,5% 13,8% 11,7% 12,6% 14,3% 10,2%
Acquired Skill 73,6% 64,1% 52,8% 45,0% 43,0% 42,6% 35,8% 33,6% 26,6% 13,4% 45,7%
This singular item scale question presented a number of everyday media practices to the respondents, with the tasks described covering a range of the competences – and levels thereof – discussed earlier in the competence model. This question was presented to a representative sample of the Flemish population (n=3949) aged 16 to 85 years old, with an equal distribution for socio-demographic traits (such as gender, education level, relative income, etc.).The question probed both familiarity and proficiency. Some more obvious conclusions include the fact that no significant effect was found for gender, but that media skills do correlate with education level and age. However, more notable is the fact that, even though they are under the age to be included in the scope of the PaS ’14, the skillset of the age cohort interrogated in the much smaller PRC-study is more widespread, prevalent and advanced than that of the broad population.
191
10.2.4 Discussion and Conclusion There are three reasons to address the concept of media literacy within the context of this thesis. First, it is a reality for the cultural industries that in the foreseeable future, audiences and culture enthusiasts are (going to be) the so-called “millennials” and “digital natives”. They generally display more advanced technological savvy and skills, but also raise certain innovational expectations, and therefore pose new challenges to the sector. Additionally, as it is the age group in which frequent and multifarious media-usage is most prevalent, it to some extent vindicates the fact that the experiments in chapters 4, 5, 7 and 9 predominantly focused on this demographic cohort – even though mostly due to chance. Secondly, the media literacy levels and competence models discussed in this section, apply not only to youngsters, but to the general public. For technologies to be successful, the mere constructional properties (e.g. user-friendliness or GUI-quality) of the technology are not sufficient to guarantee adoption; some basic digital competences of the prospective users are also an indispensable part in that equation. Thirdly, and building upon that notion, familiarity with the functioning of digital technologies is pivotal in the successful ecological embedding of EMMTs in the cultural sector – as was discussed in both the fifth and ninth chapter. The willingness of users to interact assumes a certain level of media literacy, in order to foster the eventual acceptance of the technology. Moreover – as the sixth, seventh and eighth chapter demonstrated with varying and arguable success – affordance and clarity are key in achieving this. The test users’ ability to read the affordances latent in the EMMT or AI&VRE systems presented, may well be the single most important and indispensable element in the pathway towards adoption, and the all-important concept of media literacy has gone to some extent unnoticed throughout the studies in part two of the thesis. This chapter considered three additional studies outside the scope and timeframe of the initial research project, tackling some of the shortcomings in the core studies of the thesis, and putting them in a larger frame of reference. In these studies, [1] insights concerning the cultural-creative industries, [2] facts and figures concerning the nature of the audience’s proneness for e-culture participation and [3] the indispensable properties of media literacy are divulged. These three elements point out acute hindrances in the initial research plan, put up some serious reservations concerning the initial time frame of the project, but at the same time achieve putting the project in a much needed broader academic context, and arrive at hinting at the obstacles that need to be overcome to eventually successfully embed EMMT-based immersive AI&VREs within pre-existing artistic contexts.
192
11
Discussion
11.1 Evaluation of the Research Questions A set of postulations were formulated at the beginning of this thesis to be empirically tested throughout the different stages and research studies, namely to establish for current computer-based music mediation technologies 1) how undefined – or rather open-ended – artistic purpose and intent can be dealt with, 2) how non-task oriented development and flexible wielding strategies are to be handled, and 3) how meaningful social (interactive) communication can be fostered in functional and customizable design. Also, astute questions concerning the impact of standardization on design and assessment parameters (e.g. usability heuristics, self-reported metrics, technology acceptance models and performance metrics) were pondered. These research questions tackled in the opening chapter, were to some extent all dealt with in the chapters of part 2 of the thesis, namely the experimental studies. Based on those results, a number of findings, insights and estimations are presented. Concerning the concept of embodiment: “Assuming that the paradigm of embodiment is universal, innate, spontaneous and natural; what are the factors that generate the interaction problems with EMMTs?”: The main problem with this issue is based primordially around the feature extraction part of the EMMT’s functionality. The whole mapping of any given EMMT is based around the fact that – within a certain resolution – some elements out of the raw data stream of a sensor device are considered meaningful. These elements in turn trigger events which are noticeable in the sonic/visual output. The whole issue can in essence be traced back to the fact that “what is considered meaningful” by 1) the sensor itself, 2) the mapping that is applied to the extracted data-features and 3) the users. Users widely differ, preferring one mapping over the other, and what is considered the threshold for meaningful movement may vary from one user to the next. In sum, embodiment
193
may be innate and spontaneous, but extrapolating its universal properties to the world of the (sensor) technologies that support it, is too big a leap in reasoning – for now anyways. Concerning the basic operation of EMMTs: “What is the state of the art concerning New Interfaces for Musical Expression and Embodied Music Mediation Technologies?”: This is dealt with in the second chapter; suffice to say that over the sixteen years the NIME-conference has been annually taking place, a verifiable plethora of interfaces, systems, augmented instruments and such have been conceived, all with their own properties and merits, but as of yet, there is no all-encompassing theorem on “what works and what doesn’t”; trial and error, it appears, although now with an ever-growing element of educated guessing to boot. “What opportunities are there for categorization of such systems, devices and interfaces?”: There is very little merit in categorizing all manners in which new modes of music interaction can occur; especially since most of these efforts are at real risk to get outdated as soon as a new technology is available, or as soon as a new NIME, ICAD or ICMPC or ESCOM conference is around the corner. “What technological issues do EMMTs suffer from?”: The list is long and not very distinguished; (small) problems pervade the tentative assembly of basically every EMMT, and it is the delicate balance between the elements discussed in Figure 3.01 and the users, that determine the quality of operation and quality of interaction. One can only be hopeful that sensors, MoCap and VR-technologies are ever improving, so that they will one day be up to the daunting task of serving the user properly. Concerning interaction and adoption: “Why is the adoption process (by the cultural sector) so difficult?”: There are many conceivable motivations: budgetary restraints, reliability issues and sheer lack of interest in the technology, are among the chief reasons. Confronting users, audiences and organizations alike with expressive, exciting, functional, affordable and accessible technologies, might be the only way to sway them. “What are the obstacles that are hindering user adoption?”: Here, a more definitive answer can easily be provided: lack of familiarity and media literacy issues, to a lesser extent reluctance for engaging with interactive art. As of yet, there are guidelines to ensure at least the functional part of the equation; once the societal barriers can be raised, the technology might come to full fruition. “What is the cause of the problems pervading EMMT adoption?”: A combination of the problems stated in the previous three answers: they are based around reliability issues, adoption hindrances and societal challenges. All three 194
domains have been steadily evolving over the last decade, but the challenges that remain within those stated domains, are also dire to overcome. “Which of the stakeholders require what means and/or tools?”: There are a lot of needs to cater for: from the part of the organizations and from the part of the users alike. Most have been dealt with at length throughout the thesis, yet it is striking the good balance between them, that will eventually result in success. Concerning natural, social and interactive: “What does more “natural” interaction constitute from the part of the enduser?”: Even though we are striving for “more natural modes of interaction”, at no point in any of the experiments did sensor-wearing test subjects perceive the interaction to be more natural than for example the physical interaction between an instrumentalist and his violin. The idea is a commendable one, but to date, we are still mostly dealing with the illusion of a more natural interaction, rather than technologies that are in actuality able to foster more spontaneous embodied interaction. “How can social interaction be promoted during EMMT interaction? What factors hinder it?”: There is definitely a merit in making social interaction a prerequisite in the functioning or gameplay (examples). There are, however, also two limits to it: 1) having multiple people who have the exact same expressive range, is a certain path to confusion and frustration; the threats that have to be prevented at all cost. Moreover, 2) from the moment users can no longer physically perceive all the people anymore (which in our case occurred around 5-6 participants) – much in the same way the occlusion of the infra-red reflective markers was detrimental to the proper functioning of the motion capture device – the confusing outcome results in the withdrawal from both social and interface interaction among the participants. “What constitutes interactivity in art? What enables it? What hinders it?”: The ongoing debate here is between what constitutes a technology to be responsive and interactive, responsive being the instance where the interface effectively replicates a certain behavior each time an identifiable command is given. Interactivity implies that the system gets the command, interprets it, and computationally formulates a response, taking up the role of an active agent. All of the use cases were aptly responsive, in only one of the use cases did we come close to interactivity. It belongs to the realm of more advanced scientific fields such as computer learning, way beyond the scope of this and similar theses. Concerning artistic expression and application domains: “How diffuse is the scope of artistic opportunities and application domains?”: The possibilities are limitless; the biggest challenge is to put the right elements 195
together. As long as the human imagination can come up with challenges that more or less fit the available features, they could have been implemented and the team could have continued indefinitely – assuming proper funding. “What are the application-specific favorable circumstances for embedding?”: Speaking solely from personal experience, the availability of a somewhat concealed interaction space appears to be one of the most important assets. To forster unrestrained interaction, participants need to be familiar with the prototype, be at ease with the technology and be comfortable to employ their bodies as the natural interface – which they will not do with strangers watching. “How viable are certain application domains within a given context?”: This is entirely dependent on the circumstances: as they are still very much in an experimental stage, the project rendered a positive estimation within a number of application domains at best, but further ecologically embedded research and context-specific development would be needed in order to – even provisionally – guarantee a viable system. “Can some broadly applicable parameters be identified for optimal functioning of the aforementioned devices?”: Yes; the design guidelines in the conceptual model offer a systematic overview of developmental considerations that can genuinely increase the usability of a given setup. Even so, applying the conceptual model will always be a difficult equilibrium to strike between some of the elements inherent in its guidelines. Concerning AI&VRE development methodology: “What method(s) do we need to gather the relevant information from the participants (what questions, how to present them)”: The relevant questions were dealt with at length in chapter five. In terms of method, this is a very difficult trade-off; on the one hand, this development requires a lot of information on any given number of topics; on the other hand, self-reported measures can only go so far, and the amount of information required from prospective users cannot be harnessed within one survey alone, so a multimethod approach will always be advisable (combination of interviews / focus groups and questionnaires). “How can aspirations, prerequisites and necessities of potential end-users be identified?”: Ideally, these need to be dealt with in an interview setting, and after the prospective users had the chance to interact with a tangible prototype or demonstrator, as was practiced in the “SoundField”-project. “How can functional, artistic and aesthetic requirements be accommodated technically?”: The benefit of working in a small multi-disciplinary team on an interdisciplinary project, is that the idiom and jargon from the different scientific fields can be easily translated within the internal communication of 196
the team-members. The relevant desiderata coming out of the focus group sessions can as such be parsed quickly to both the artistic and computational side of the development. “What can be learned from earlier implementations (i.e. making the transition from implementation/instance-based to iterative (RITE) design)?”: There is essentially nothing particularly wrong with the way in which the interfaces in chapter four were constructed; they only functioned poorly in conjunction with humans. And as they were more prone to obvious mistakes than those systems that were constructed within a collaborative and – more importantly – iterative perspective, it to a certain extent proves that researchers are better off not trying time and again to re-invent the wheel, but build upon the merits of earlier peers. “Can elements from traditional HCI and usability studies be applied to assess the functioning of EMMTs?”: They are most useful, but in most instances not readily applicable. Moreover, if they are suitable to assess interaction qualities, then it is generally only up to a certain level. Though the interfaces and informatics might be similar, the goal and functions of the technologies by no means are. “Are there opportunities for HCI-standardization within the specific field of EMMT development?”: Once again, the metrics and heuristics used in this project are useful and applicable, but only measure a basic level of usability. Further research in more application domains and with different systems could advance a larger degree of standardization as such. Concerning HCI evaluation methods applied to EMMTs: “How can implementations ideally be evaluated (beyond the scope of singular implementations and across application domains)?”: Without assuming that the evaluation method displayed in chapter nine is in any way normative for future developments, it is however particularly well-suited for this endeavor. It is insufficiently developed to serve as any basis for standardization, but the ideas that underpin it, are decent enough to merit further scrutiny. “How can different design-elements and system/setup aspects be measured?”: In reference to the construction of the applied metrics, heuristics and indices, the third and ninth chapter go into the evaluation methodology at length. “How can and should feedback ideally be recorded and re-applied?”: Though the evidence gathered during the “SoundField”-project would suggest that the method can be used much in the same way, rendering the same basic conclusions, no such efforts were done within the scope of this project, due to time and budgetary restraints (even though the replicability of research is one of the cornerstones of the scientific method).
197
“What UX-based metrics can be defined for the optimization of the functionality/usability of these devices?”: A number of the studies performed, sought to address and assess these metrics (such as (game) enjoyment, clarity, understanding, affordances, etc.). Though not technically covering the same grounds as the circumscribed meaning of “User Experience” within the confinement of Communication Science research, the more comprehensive definition used for this project did just that: define metrics which could eventually elucidate what steps should be taken to improve the overall usability.
11.2 Validity of the hypotheses At the beginning of this thesis, ten working hypotheses were formulated on the basis of the research questions; essentially, a number of standpoints taken concerning EMMTs and certain aspects relevant to AI&VRE development, that through an RtD-approach, could be ascertained or disproven. In this section, those standpoints are briefly compared to the outcome of the experimental studies, and in doing so, appraised in terms of worth. “Concerning the paradigm of embodiment, this thesis assumes that its core concept is universal and innate in humans, irrespective of sociodemographic or culturally defined characteristics, and is therefore particularly well-suited and universally applicable in the design and development of gestural music interfaces.”: Partially true; as a theoretical concept, there is little debate about its innate status; however, assuming that universality to be the ideal foundation to base interface design is a bit of a stretch in reasoning, since it presupposes the same properties to hold true for the sensor technologies involved; quod non? “Concerning the basic operation strategies, this thesis assumes that EMMTs can foster a distinct (and more direct) interaction with musical content than traditional musical instruments, based on the principle of embodiment, more specifically through the means of the illusion of non-mediation.”: Sadly, for the time being, and to a large part due to usability problems pervading mobile interfaces and tangible technologies, this not a reality (yet). “Concerning the validity of the interaction metaphors for cultural expression, this thesis assumes that the interaction strategies with the technology follow a set of patterns which are essentially natural, and therefore are intuitive to learn, relatively simple to master and extremely well-suited to be used in both social
198
and interactive contexts.”: Granted the fact that most of the interaction strategies were intentionally kept easy (throughout the thesis), this statement is passable. “Concerning technology adoption of EMMTs, this thesis assumes the necessity of particular AI&VRE and EMMT-interfaces to be rigorously tested across various art practices and artistic application domains.”: From a user’s perspective, this has been done with some reasonably successful outcomes. From larger societal or institutional viewpoints, this has yet to be achieved. “In order to assess the applicability of theoretical concepts such as flow, presence and affordance to AI&VRE and EMMTs, this thesis assumes parameters adhering to these concepts to be elaborately tested during lab-based and/or real-life interaction with these interactive music devices. This is instrumental in the examination and incorporation of social interaction patterns and gaming strategies into interactive music designs.”: A lot of these statements can be attested by results obtained, although most of the chapters dealt with flow, presence and affordance only parenthetically and with insufficient depth. “Irrespective of the general applicability of the concept of embodiment, this thesis assumes the assertion of participants’ wants, needs and abilities can ameliorate (prospective) users' comfort and understanding of interface operation.”: There is sufficient evidence to accept this hypothesis; chapters 5 through 10 all in part indicate that a well-rounded profile of (prospective) users as well as user-informed customization greatly benefit the eventual outcome. “Moreover, the thesis assumes that the vital information conveyed in the indepth assessment of users' aspirations combined with this enhanced usercomfort and understanding during the operation of the interface can in turn help in improving the EMMT-based catering for various cultural practices and artistic goals.”: Partially true; the project did cater for a diverse crowd as well as to a range of different artistic practices, but this was only to a small extent due to the methodologies set forth predominantly in chapter five. “Concerning AI&VRE development methodology, this thesis presumes an RtDapproach to be beneficial for probing whether standardized HCI evaluation methods can be applied and geared towards EMMTs (as, even though the interaction metaphor is grounded in embodiment, the technology is in essence software-based and computer-driven).”: Partially true; there is evidence in the results to support this. A set of HCI-borrowed metrics and heuristics were effectively applied, yet the assumption goes a bit too far, in that it doesn’t take 199
the users’ perspective sufficiently into perspective, and puts too much trust in the infallible nature of the technologies used. “Moreover, as is also the case in other current Living Lab research, the thesis assumes that within development strategy, concepts like "iterative", "participatory" and "parallel design" will accelerate development, but need to be adopted from computer engineering into a predominantly artistic context.”: There is sufficient evidence to accept this hypothesis, although it should be noted that the Living Lab methodology was not followed rigorously. “Concerning ecological validity, this thesis presupposes that the successful cultural implementation of the described technology greatly benefits from labbased interface assessment and scenario-based development, but can essentially only be achieved through in situ testing.”: There is some evidence to support this hypothesis, but as no real-world examples have effectively been recorded, there is too little evidence to support this claim fully. “Furthermore, the same goes for the design scenarios in which social interaction can occur: the collaborative modalities need to be examined in order to create interactive opportunities within some of the EMMT designs, yet the testing the incorporation of social strategies can only be achieved through live interaction with the presented technologies.”: There is more than sufficient evidence to accept this hypothesis.
11.3 Limitations of the final study and recommendations for future research As is always the case for a thesis, not all issues were aptly resolved and some questions remained unanswered after the project had ended. For the final study, a great deal of importance can be attributed to the raw sonic material used for sonifications; it is very much akin to how the feature extraction from raw sensor data determines the way in which spatio-temporal cues are mapped onto action-relevant cues. Whether or not these are perceived as the right sonification/mappings or not, is up for debate, and more often than not, there is no unambiguous solution. The same goes for gameplay, music and dance preferences; one can only try to find common grounds for most users, but there is no clear answer on what constitutes an accurate mapping or sonification.
200
Accordingly, no progress was made in defining the proper action to sound mapping strategies. In all of the cases, results were to a large extent dependent on the users, material and the afforded interaction of the specific use cases. So, it seems that understanding and accommodating the users’ expectations is paramount. In terms of scalability, the transition from laptop-based demonstrators to full scale motion capture-enabled platforms may have been a successful one, yet surpassing the number of five participants resulted practically always in an unsatisfactory and chaotic output. Clearly, in relation to actively incorporating of participants, there still is a problem in equally dividing possibilities for expression whilst keeping the proper contribution discernible in the overall output, and at the same time maintaining headway to advanced level and safeguarding challenge. Regarding what amount of AI&VRE-features should be controllable by whom, there are definite limits to the amount of sonic and visual information that can be socially, collaboratively or interactively processed, and this will be one of the biggest challenges for future developments. Noting that the emphasis of the whole project was on designing for a broad scope of potential purposes, much more focus was put on breadth and range than on the in-depth development of one single finished and flawless application. As such, quite a few use cases can be considered not overly functional and some of them were merely developed for the purpose of demonstration; as was already stated, this was the main oversight in an otherwise enriching and prosperous project. The design strategy resulted in a wide range of interesting use cases and unanticipated additional ideas, that, through further development, could become viable applications for both artistic and non-artistic applications. With the average media literacy being on the rise, chances for success are a lot better than they were nine years ago, should the cultural industries be interested in pursuing this endeavor. And that is perhaps the biggest threat to the project: all the variables in the equation need to be favorable for this project to take off.
The limitations aside, future research should focus on the further implementation of VRtechnologies, as those currently seem to be the low-hanging fruits in this line of research: vast improvement on experience quality, with relative ease in terms of the implementation, and at a reasonable cost. More in depth and especially more attuned questionnaires should focus on relevant action-oriented properties of end-users, the correlation between flow, presence, affordance and (game) enjoyment, and finally, necessary media literacy parameters, relevant to this type of projects. At present, there are some under-investigated opportunities for the “SoundField”system to be employed in therapeutic and educational applications, yet it would require trained professionals (pedagogists and psychologists) to get involved in that part of the program. One of the foreseeable application domains could for instance be within a context of serious game development in the field of music theory education. 201
Finally, and perhaps most importantly, stakeholder interviews with relevant organizations in the cultural-creative industries should elucidate if organizations – and if so, which ones – in the sector are at present prepared to engage with the technology; otherwise, possible eventual efforts will indefinitely remain idle.
202
12
Conclusions
12.1 Overview of results The first part of the thesis presented the general state of affairs. An introduction and state of the arts for new interfaces for interactive artistic expression was given and the paradigm shift they could entail for both the research community and the cultural-creative industries alike. The inherent problems of the associated technologies and academic context of the project were discussed, after which the challenges of this research line of the Emcometecca-project were enumerated. The second chapter delved into the concepts that underpin both the conceptual model and the experimental framework for the thesis. These concepts comprised the foundations of embodiment and the nexus between embodiment and musicology and the application domains of the embodiment paradigm for EMMT development. Then, an overview of the state of the arts in new interfaces for musical expression was given by means of an anthology of the relevant developments leading up to the starting point of this research project. Also, an overview of applicable HCI-methods and potential user-centered procedures concerned with EMMT & AI&VRE-development was presented, to conclude with a concise discussion on e-culture and the repercussions of new media adoption for the cultural-creative industries. Then, the methodologies that were applied throughout the research project were covered, and how an amalgamated methodological model was ascertained throughout the different stages of the research project (what they investigated and how they were incorporated). Some recurrent concepts were defined and a literature-based overview was rendered on the design guidelines (the user-related, device-related and context-related elements) for collaborative and/or interactive music interfaces. The subsequent conceptual model harnessed the most important concepts and operationalized them as a set of assessment parameters, resulting in both a user-centered and iterative EMMT development and an overview of the employed heuristics, metrics, parameter spaces and indexes for EMMT and AI&VRE assessment.
203
In part 2, the research trajectory is described, starting with the early studies of the project that exemplify the fundamental problems encountered in the implementation of embodied music mediation devices. With swift development of multiple and pluriform, varied sensor-enabled systems still being the key aim, achieving intelligible, functioning, enjoyable and concrete setups proved to be more complicated than originally anticipated. Trial, error and determination proved to be insufficient to achieve scientific – let alone artistic – success. A number of unfulfilling setups and seriously flawed design tests aptly demonstrated the pitfalls in EMMT-development, namely having a couple of scarcely concealed research setups masked as music games, making for bad experiments and poor games alike. The recurrent flaws instigated the adoption of HCI, usability and useroriented methodologies and prompted the user-centered paradigm to be explored, more out of a necessity, than luxury. In the following chapter, but in the same timeframe, a survey was issued to assess the necessary capacities and relevant predilections of prospective AI&VRE- and EMMTusers, covering musical and dance background, training and preferences, to attempt defining provisional user types for EMMTs, and generally, what prospective users would bring to the table before the actual interaction. This would provide for potential definition EMMT-user profiles as well as a documented panel of eventual test subjects. Upon completion, some of the limitations of survey-research and the execution of this specific questionnaire became apparent. Despite covering a good deal of relevant sociodemographic and action-oriented ontological information about the prospective EMMTusers, it didn’t arrive at rendering a representative image of either the broad scope of the cultural industries or its core audiences, due to an insufficiently large sample and inadequate sampling across various age groups. Moreover, the resulting database remained largely untouched throughout the eventual planning of the Emcometeccaproject, making the results less relevant in view of the further continuation of the project. The results did however capture the fact that a predominantly young sample of highly educated culture-enthusiasts expressed genuine concern about the prospect of technologyassisted participation in a receptive cultural context, an issue that could have a great deal of importance in the functioning and the future adoption of AI&VREs and EMMTs, let alone the ecological embedding thereof in the cultural industries. The sixth chapter, in turn looked less at the internal properties and motives, but towards the prevalent perceptions of culture participants confronted with actual EMMTs in a receptive environment, as online survey simply does not suffice to understand of how audiences respond to real-life examples of interactive art systems. Two smaller exploratory in situ tests focused on the genuine experience of culture-enthusiasts when interacting with live artistic installations, i.e. two artistic installations of dr. Pieter Coussement (Coussement et al., 2008; Coussement et al., 2010). Results of these partially ethnographic, partially interview-based studies confirm the not so straightforward nor entirely amiable reception/perception of EMMT-based art installations in an ecological
204
setting. Results suggest a great deal of opacity in the affordances of said new and unstable media art pieces, although the prevalent attitudes were not downright derisive either. The next chapter took a step away from the factual (cf. Chapter 5) and contextual (cf. Chapter 6) considerations concerning EMMT-use, and delved back into an interface evaluation study revolving around the “Synch-in-Team”-application (Deweppe et al., 2009; Leman et al., 2009; Leman et al., 2010). This study constituted the first serious attempt at combining a more systematic userexperience-based evaluation with multiple iterative design stages of a collaborative music game development. Within a system that had a fixed purpose but a relatively flexible setup, the goal was 1) to evaluate the properties and qualities, 2) to quickly establish the shortcomings, and 3) to gather feedback on possible changes that should be implemented. Encompassing 4) how prospective users wanted to use the technology, 5) if they experienced sufficient control and 6) whether they appreciated the prototype, was the additional information required to foster prototype improvements in between the three different test cycles. Therefore, these interim “end” user evaluations of the functional prototype all proved to have substantial merits within the iterative aspect of the design process. And a final lesson learnt during this study, is the importance of flexibility concerning the implementation context. The eighth chapter followed up on this line of sequential development and got more involved with HCI-methods. Occurring mostly in the context of a user-centered and labbased control room, the development of a system focusing on 3D virtual object mediation created ample opportunity for investigating and successfully testing a number of HCImetrics and heuristics. Moreover, the evaluation methods applied up to this point had been quite time-consuming and post-hoc, but the improved approach made the evaluation method – even though not particularly ideal for well-documented research purposes – all the more suitable for fast-paced GUI-development, quickly isolating problems in the design, rapidly adjusting, aiding and furthering usability, and swiftly moving on with an improved prototype (unlike the much slower phased approach practiced in the SIT-study). As such, this study consolidated a lot of the problems encountered in the earlier stages. In practice testing parts of the conceptual model, carrying out real-time user-suggested modifications in the actual context of use, immediately improving the overall usability. The ninth and last of the chapters discussing the experimental research for this thesis, consolidated the methodological considerations concerning the conceptual model, and bringing together extreme programming, HCI-heuristics, sociological survey methodology, iterative design, etc. in an open-ended collaborative design project, called “SoundField”. The project itself entailed a bottom-up development, paired with a topdown infrastructure, an open-ended, user-centered and collaborative design strategy combined with rapid and iterative prototyping and development, and an integrated evaluation component, experimentally applying all the elements of the conceptual model.
205
Among the new and different elements in the design strategy, is the fact that this project aimed to be rigorously open-ended with a flexible style of extreme programming. This open-endedness can be attested by the fact that that one of the few things that was imposed top-down, was the immersive/interactive environment itself. External elements could be introduced into the system in a broad range of cultural practices, and a lot of the features were developed collaboratively and from the ground up with the workshopparticipants (Figure 11.01). The 22 resulting use cases in 7 application domains were made possible only by simultaneous efforts on software-improvement of the software, artistic sonic and visual implementations and constant interrogation and evaluation of and by the eventual user group, making this a very hands-on and real-time project. The “SoundField”-evaluations show that test subjects generally experienced sounds and visuals, the relation between them, the social interactions and the enjoyment overall to be quite positive. The audience being granted insight in the inner workings of the operations and functionality and the active involvement in a user-centered design format, immensely aided in eliminating the opacity of unstable media installations (discussed in chapter 6). Also, as the active involvement of participants reduced the threshold for participation in this project, and end-user enthusiasm eventually snowballed, practically nullifying socio-demographic predetermination for participation, the procedure described in the conceptual model may well be a gateway to broader and more diverse participation. Figure 11.01: Differences in setup-properties between traditional music mediation controllers (see Fig 3.02) and the development model used for “SoundField” (Fig 9.04).
Aspect
Traditional MMC operation
“SoundField” operation
Cultural Practice
Semi-fixed or Fixed
Variable
Artistic Purpose Number of participants Elements / Scenarios Sonic output Visual output Technologies Basic operation Basic mapping
Fixed Fixed - Variable N/A Fixed - Variable N/A - Fixed - Variable Fixed Fixed Fixed
Variable Variable Variable Variable Variable Semi-Fixed Fixed Fixed
The fact that almost none of the developments were seen through to their full extent, so that the full potential of the system cannot be attested based solely on the underlying results, is a regrettable oversight and shortcoming within this project. The fact that more consideration was given to the scope and artistic range of the system, rather than to depth
206
of a singular development was however a justifiable decision, and the absence of a thoroughly refined use case does genuinely blemish the otherwise overall quite positive outcome of the project. In sum, this study effectively provides the keystone for the conceptual model, bringing together all the methodological elements discussed, tried and tested throughout the previous chapters into one consolidated and collaboratively designed art project. Moreover, it shows a potentially viable and cost-efficient approach for technology-eager institutions to engage in the in-house customization, evaluation and adjustment of some compelling (interactive) EMMT-technologies to be embedded in pre-existing cultural contexts. With growing, increasingly media-literate audiences showing new-found enthusiasm for the potential of the technology and commercially available VRtechnologies catching up to the challenges set forth in the project, this realistically is one direction the industry could evolve in. And finally, the third part of the thesis kicked off with the introduction of three complementary studies, technically outside the scope of the research project. The delay in the finalization of the thesis created the opportunity to investigate some of the actual evolutions taking place in both the cultural sector and the broad socio-cultural context, and some necessary focus could be given to crucial competences like media literacy and technology-informed culture participation (i.e. e-culture). In doing so, chapter ten tried to have a bird’s eye view on the subject matter by elaborating on the broader context, taking findings from other scientific disciplines (e.g. communication science, cultural studies, cultural sociology and policy research) into account. The first two studies mentioned, dealt with the digital reality impacting the cultural industries, namely the advent and growing importance of e-culture, the exponential rise of digitized forms of participation and the adoption of new and social media technologies from the institutional and the audience side alike; the second part of the chapter was concerned with media literacy levels of the Flemish population. The three studies thus presented [1] insights concerning the cultural-creative industries, [2] facts and figures concerning the nature of the audience’s proneness for e-culture participation and [3] some insights into the properties of media literacy. The studies point out a manifest inconsistency in the initial research plan, and put up some serious reservations concerning the initial time frame of the project, hinting at the obstacles that need to be overcome to eventually successfully embed EMMT-based immersive AI&VREs within pre-existing artistic contexts.
207
12.2 Accomplishments and value The “SoundField”-project aimed at the exploration of three different PhD-projects and research lines: [1] the applicability of a multi-modal enabled platform for interactive sonification (Diniz et al., 2012), [2] a better understanding of interactivity in art (Coussement et al., 2012), and [3] on defining opportunities for artistic practice and deployment of immersive technologies in the cultural sector. The submission of this thesis, is the last piece of a computer science / artistic research / sociomusicology triptych, about nine years in the making. The presented methodologies and associated conceptual model have proven to be an effective way of rapidly establishing opportunities and identifying usability problems with a multi-purpose immersive environment. Over a period of four months and with the combined effort of a large group of culture-enthusiasts and an interdisciplinary team of researchers, twenty-two functioning instantiations of the “SoundField”-system have successfully been presented. Strategy-wise, this project was entirely based around participatory and iterative design, as it relied heavily on the indispensable and active incorporation of users, participants and artists over the course of the project. In combination with the in situ-work in an artistic incubator, it created the opportunity for the developers to get a clear idea on goal assessment, which in turn enabled the creation of user-molded use cases. The iterative development process allowed for user-informed design adjustments and finally, the evaluation method created an opportunity for analyzing the quality of the presented developments and assessing the acceptance of the technology. A customized set of design guidelines, usability heuristics, complexity indexes and evaluation parameters was developed which resulted in a formative procedure that was both practical and efficient. Moreover, it offered a realistic alternative for task-based usability and efficiency metrics, it constituted an adaptation of standardized human computer-interaction heuristics for gesture-based interfaces, and it made a valuable contribution to the quality of experience measurement of immersive / augmented / virtual reality environments. This design procedure, the user-informed decisions and the post-interaction feedback inspired the following observations: A first observation is that enjoyment is the dominant factor in the end result. Accuracy, discernibility, visibility, suitability, focus and even functionality are subordinate to whether or not the experience of interaction is perceived as entertaining. This perception is closely linked to the amount of challenge that is offered by the system and the amount of attention and skill that is required: this is a delicate balance to achieve! If a challenge is too easy, users will get bored. If it is too difficult, frustration ensues. Balancing challenge and skill while providing sufficient opportunity for reward ensures 1) a system that is 208
considered functional, 2) an experience that is perceived as pleasant, and 3) an interaction that is regarded as meaningful. An additional remark to be made in this context is that, no matter how well skills and challenges are balanced, how well expressed purpose and implemented artistic practice correspond, how well the controllable parameters are attuned and how well sonic and visual elements can be combined, the quality of the raw material that accommodates the interaction is of vital importance to the success of the use case. A second observation mad during the “SoundField”-project is that the quality of actionto-sound-couplings and action-to-vision-couplings are often linked to the esthetic and synesthetic qualities of a given use case. Moreover, visual cues are generally more easily understood than sonic ones. A third observation is that in nearly all instances where the operation of the “SoundField” framework was examined, accuracy and discernibility appeared to be joint properties. Additionally, in most cases, accuracy and discernibility are inversely proportionate to the number of sonic affordances latent in a use case. And a fourth observation is that the quality of social interaction is linked how suitable a use case is perceived to be to accommodate social interaction, and, within the scope of this project, the ideal social complexity was between three and five participants. Correspondingly, even though there was no direct correlation between use case complexity and quality of experience, confusion as a result of social complexity and unsuccessfully implemented features were most often responsible the negative assessment of use cases. Finally, this project was a testbed for technological development, for interactivity in art and for goal assessment and technology adoption in the cultural sector. A clear and exhaustive definition of all opportunities for embodied music mediation technologies in the cultural sector has not been given, but on the other hand, couldn’t have been within the scope of this thesis. The “SoundField”-project was conducted with a small team, over a limited timeframe and on a small but manageable scale. Its main contribution is that a number of conceivable uses for the technology have been dealt with, some observations pertaining to implementation have been made and a functional developmental strategy has been presented. Both on a societal, a technological and on an institutional level a lot of work remains to be done and a lot of issues remain open to debate, as at the end of this thesis, they should be. Still, similar initiatives may eventually be the way, albeit one small but valuable step at a time, towards sectoral understanding and audience-acceptance of artistically employed embodied music mediation tools.
209
12.3 Thoughts on the technological & academic aftermath The timeliness of the research project described in part two of the thesis was at best debatable; with most of the relevant technological development being done between 2005 and 2011, the sensor and motion capture devices used in this project (i.e. HOP-sensors, WiiMote, i-CubeX OptiTrack, etc.) paled in comparison to those that would have been available had the project commenced five to ten years later (e.g. Kinect, Leap Motion or Oculus Rift + Touch). These disadvantages in the technological state of the arts combined with the fact that immersive environments as well as multimodal-enabled systems were quite novel and at the time, seriously complicated thingsd. The fact that the project balanced on the nexus of art, computer science, musicology and social sciences, did not aid the opportunities for academic dissemination, as appropriate A1-journals were scarce at the time, and due to the project’s profound interdisciplinary nature, there would always be a problem with at least one pillar of the project being outside or only tangentially related to the scope of potentially relevant journals. This project did by no means spearhead the examined type of participatory and multidisciplinary development of immersive environments, but the number of comparable and contemporary initiatives were quite limited. And moreover, the fact that funding in the cultural sector is tight as it is and the PRC-study on digitization in cultural industries shows that as late as end 2015 still less than 4% of cultural organizations were employing online resources for anything other than passive communication and social networking (e.g. genuinely digital content or workshops revolving around state-of-the-art technologies similar to those described in the experiments described in chapters four to nine), it should not come as a surprise that the cultural sector has had an apprehensive and sceptic stance towards initiatives like “SoundField”. Work on the research for this thesis commenced in early 2008; all studies discussed in part two of the thesis – chapters four to nine – were conceptualized, rolled-out, instantiated, conducted and finished between early 2008 and early 2012. This might beckon the question: why the additional five years and the bonus studies in chapter ten – even though the presented studies do not make up the core of the research aspirations? The fact of the matter is that prior to 2008, when the work packages, methods and procedures for the Emcometecca-project were penned down, embodied music mediation technologies were already a technology which had come to – what was considered at the time – full fruition and had been established as a research field for a reasonable span of time. Meanwhile, the impact of corporeal technologies (e.g. Wii, Wiimote Plus and later Wii Balance Board and Kinect) on the gaming industry was quite profound. Even though nobody had the ability to accurately predict the precise direction digitization was going to 210
unfold in over the next few years, it could only be assumed that the combination of the aforementioned factors would also have a thoroughgoing influence on the cultural sector. In reality, as it turns out, the gyroscopes and accelerometers that were used throughout most of the studies in this thesis, have indeed become a quite common and a quite intrinsic part of our everyday life – and by extrapolation of the cultural sector: yet not at all in the way that the project could have foreseen. It is the widespread use of mobile consumer devices that has predominantly shaped the predicted technological revolution of the last decade, and much more than embodied per se, it has been a social one. The problem, therefore, with the early research stages and planning of the entire study, is that it largely coincided with the advent of smartphones and social media (i.c. Facebook). Essentially, it means that this project got its predictions for necessary future developments pretty much in the ballpark of what eventually happened. But the way in which audiences and institutions in the cultural-creative industries in actuality availed themselves of these new and exciting technologies during the same time frame in which these research studies took place, turned out to be quite dissociate from what was anticipated. Whether it is a condition other theses suffer from or not, is essentially a moot point; yet the five additional years of research were most welcome to support a much-needed different point of view, and even a paradigm shift to some extent. The switch to an altogether different faculty and research field aided the realization of this notion, and helped envisage some of the counterintuitive results from part two, through a different frame of reference. It does however permit me to say that we were probably not barking up the wrong tree, but we were barking a bit too early on: 1) a higher penetration of digital technologies in the cultural industries, 2) a more pervasively high degree of media literacy throughout the population, and 3) the timely availability of a number of more reliable and more familiar consumer electronics (e.g. smartphones and VR-helmets), would have made, if not all, than at least a great deal of difference. The additional studies in chapter ten are beneficial to accommodate a bird’s eye view on the developments in part two, and to some extent further legitimize the bottom-up and top-down elements latent in the conceptual model. They facilitate an all-important alternative and bigger picture, in research fields unbeknownst to me at the time. It is probably not all that uncommon for hindsight to come into the academic equation, but it is quite a nuisance when it partially nullifies the outcome of the initial PhD-research. Taking an additional research period to ponder upon these fundamental research questions is one way to compensate for some methodological insecurities and for second-guessing the original outcomes – although for future reference and as far as posterity is concerned, I do not recommend it as a strategy.
211
List of Publications
Deweppe, A. (2016). Digitale Cultuurparticipatie: Wie? Wat? En hoe? Lasso: Publicate van het Brussels Kunstenoverleg en Brussels Netwerk voor Cultuurparticipatie en Kunsteducatie. Deweppe, A., Diniz, N., Coussement, P., & Leman, M. (2015). Addressing engagement issues for interactive music technologies through a participatory design approach with ‘“SoundField”’. Journal of Arts & Communities, 7 (1-2), pp. 45-61. Deweppe, A. & Picone, I. (2015). Participatie in Vlaanderen 2: Eerste analyses van de Participatiesurvey 2014. In: Lievens, J., Siongers, J. & Waege, H. (eds.). Leuven, Belgium: Acco, pp. 155-179. Picone, I. & Deweppe, A. (2015). Participatie in Vlaanderen 2: Eerste analyses van de Participatiesurvey 2014. In: Lievens, J., Siongers, J. & Waege, H. (eds.). Leuven, Belgium: Acco, pp. 183-207 Deweppe, A., Picone, I., Segers, K. & Pauwels, C. (2015). Participatie in Vlaanderen 1: Basisgegevens van de Participatiesurvey 2014. In: Lievens, J., Siongers, J. & Waege, H. (eds.). Leuven, Belgium: Acco, pp. 115-140 Deweppe, A., Picone, I., Segers, K. & Pauwels, C. (2015). Participatie in Vlaanderen 1: Basisgegevens van de Participatiesurvey 2014. In: Lievens, J., Siongers, J. & Waege, H. (eds.). Leuven, Belgium: Acco, pp. 141-163. Deweppe, A., Diniz, N., Coussement, P. & Leman, M. (2014). Engaging Audiences through a Participatory Design Approach with the Interactive Music Installation “SoundField”. Proceedings of the 2014 Electronic Visualisation & the Arts conference (EVA), 8-10 July, London. In: Ng, K., Mcdaid, S. & Bowen, J. (eds.). Swindon, UK: British Computer Society Learning & Development Ltd, (Electronic Workshops in Computing (eWiC)) pp. 1-8. Picone, I., Deweppe, A., Glorieux, I., Minnen, J. & Van Tienoven, T-P. (2014). The MOTUS platform as a way of Contextualising Media Use Practices Paper presented at the International Association of Time Use Research 2014 Conference - Time Use - From Theory to Praxis, 30 July - 1 August, Turku, Finland. Deweppe, A., Coussement, P., Diniz, N. & Leman, M. (2014). Addressing Engagement Issues for Interactive Music Technologies through a Participatory Design Approach using the
213
“SoundField”-platform Abstracts of the 2014 International Perspectives on Participation and Engagement in the Arts conference [IPPEA], 20-21 June, 2014. Utrecht: The Netherlands.. pp. 29-30. Van Dyck, E., Moelants, D., Demey, M., Deweppe, A., Coussement, P., & Leman, M. (2013). The impact of the bass drum on human dance movement. Music Perception, 30(4), pp. 349–359. Correia Da Silva Diniz, N., Coussement, P., Deweppe, A., Demey, M., & Leman, M. (2012). An embodied music cognition approach to multilevel interactive sonification. Journal on Multimodal User Interfaces, 5 (3-4), pp. 211–219. Deweppe, A., Correia Da Silva Diniz, N., Coussement, P., & Leman, M. (2011). A methodological framework for the development and evaluation of user-centered art installations. (R. Timmers & M. Woodhouse, Eds.) Journal of Interdisciplinary Music Studies, 5(1), pp. 19–39. Deweppe, A., Correia Da Silva Diniz, N., Coussement, P., Lesaffre, M., & Leman, M. (2010). Evaluating and Implementing Strategies for Embodied Music Interaction (pp. 20–22). Presented at the CIM10. Van Dyck, E., Moelants, D., Demey, M., Coussement, P., Deweppe, A., & Leman, M. (2010). The impact of the bass drum on body movement in spontaneous dance. In: S. Demorest, S. Morrison, & P. Campbell (Eds.), Proceedings of the 11th international conference on music perception and cognition (ICMPC 11) (pp. 429–434). Presented at the 11th International conference on Music Perception and Cognition (ICMPC 11), Seattle, WA, USA: University of Washington. Correia Da Silva Diniz, N., Deweppe, A., Demey, M., & Leman, M. (2010). A Framework for music-based interactive sonification. 16th International Conference on Auditory Display, Proceedings. Presented at the 16th International Conference on Auditory Display (ICAD-2010). Leman, M., Lesaffre, M., Nijs, L., & Deweppe, A. (2010). User-oriented studies in embodied music cognition research. Musicae Scientiae, 14 (2), pp. 203–223. Deweppe, A., Lesaffre, M., & Leman, M. (2009). Discerning user profiles and establishing usability for interactive music applications that use embodied mediation technology. Triennial conference of the European society for the cognitive sciences of music conference, 7th, Proceedings. Presented at the 7th Triennial conference of the European Society for the Cognitive Sciences of Music Conference (ESCOM 2009).
214
Bibliography
Aborg, C., Sandblad, B., Gulliksen, J. & Lif, M. (2003). Integrating work environment considerations into usability evaluation methods—the ADA approach. Interacting with Computers, 15(3), 453–471. Absar, R. & Guastavino, C. (2008). Usability of non-speech sounds in user interfaces. In: Proceedings of the International Conference on Auditory Display (ICAD’08) (pp. 1–8). Agawu, K. (1997). Analyzing music under the new musicological regime. The Journal of Musicology, 15(3), 297–307. Aimi, R. & Young, D. (2004). A new beatbug: Revisions, simplifications, and new directions. In: Proceedings of the International Computer Music Conference. Alexandraki, C., Koutlemanis, P., Gasteratos, P., Valsamakis, N., Akoumianakis, D., Milolidakis, G. & Kotsalis, D. (2008). Towards the implementation of a generic platform for networked music performance: the DIAMOUSES approach. In: Proceedings of the ICMC 2008 International Computer Music Conference (ICMC 2008) (pp. 24–29). Anderson, B. (2004). Time-stilled space-slowed: how boredom matters. Geoforum, 35(6), 739– 754. Andre, T. S., Rex Hartson, H., Belz, S. M. & McCreary, F. A. (2001). The user action framework: a reliable foundation for usability engineering support tools. International Journal of Human-Computer Studies, 54(1), 107–136. Antle, A. N., Corness, G. & Droumeva, M. (2009). What the body knows: Exploring the benefits of embodied metaphors in hybrid physical digital environments. Interacting with Computers, 21(1-2), 66–75. Arfib, D., Couturier, J.-M. & Kessous, L. (2005). Expressiveness and digital musical instrument design. Journal of New Music Research, 34(1), 125–136. Arisona, S. M., Müller, P., Schubiger-Banz, S. & Specht, M. (2008). Computer-Assisted Content Editing Techniques for Live Multimedia Performance. In: Transdisciplinary Digital Art. Sound, Vision and the New Screen (pp. 199–212). Springer. Asaro, P. M. (2000). Transforming society by transforming technology: the science and politics of participatory design. Accounting, Management and Information Technologies, 10(4), 257–290. Atkinson, S. (2007). Interpretation and musical signification in acousmatic listening. Organised Sound, 12(02), 113–122. Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355–385. Bahn, C., Hahn, T. & Trueman, D. (2001). Physicality and feedback: a focus on the body in the performance of electronic music. In: Proceedings of the International Computer Music Conference (pp. 44–51). Bailenson, J. N., Beall, A. C., Loomis, J., Blascovich, J. & Turk, M. (2004). Transformed social interaction: Decoupling representation from behavior and form in collaborative virtual environments. PRESENCE: Teleoperators and Virtual Environments, 13(4), 428–441.
215
Bailenson, J. N., Blascovich, J., Beall, A. C. & Loomis, J. M. (2003). Interpersonal Distance in Immersive Virtual Environments. Personality and Social Psychology Bulletin, 29(7), 819–833. Bailenson, J. N., Swinth, K., Hoyt, C., Persky, S., Dimov, A. & Blascovich, J. (2005). The independent and interactive effects of embodied-agent appearance and behavior on selfreport, cognitive, and behavioral markers of copresence in immersive virtual environments. Presence: Teleoperators and Virtual Environments, 14(4), 379–393. Bainbridge, W. S. (2007). The Scientific Research Potential of Virtual Worlds. Science, 317(5837), 472–476. Bandt, R. (2006). Sound installation: Blurring the boundaries of the Eye, the Ear, Space and Time. Contemporary Music Review, 25(4), 353–365. Banks, J. & Humphreys, S. (2008). The Labour of User Co-Creators: Emergent Social Network Markets? Convergence: The International Journal of Research into New Media Technologies, 14(4), 401–418. Banks, J. & Potts, J. (2010). Co-creating games: a co-evolutionary analysis. New Media & Society, 12(2), 253–270. Barbosa, Á. (2003). Displaced soundscapes: A survey of network systems for music and sonic art creation. Leonardo Music Journal, 13, 53–59. Barrass, S. (2008). Creative practice-based research in interaction design. Computers in Entertainment, 6(3), 1–17. Barrass, S. & Barrass, T. (2006). Musical creativity in collaborative virtual environments. Virtual Reality, 10(2), 149–157. Barrass, S., Whitelaw, M. & Bailes, F. (2006). Listening to the mind listening: an analysis of sonification reviews, designs and correspondences. Leonardo Music Journal, 16, 13–19. Bartneck, C. (2001). How convincing is Mr. Data’s smile: Affective expressions of machines. User Modeling and User-Adapted Interaction, 11(4), 279–295. Baumann, S., Koeneke, S., Schmidt, C. F., Meyer, M., Lutz, K. & Jancke, L. (2007). A network for audio–motor coordination in skilled pianists and non-musicians. Brain Research, 1161, 65–78. Bellotti, V., Back, M., Edwards, W. K., Grinter, R. E., Henderson, A. & Lopes, C. (2002). Making sense of sensing systems: five questions for designers and researchers. In: Proceedings of the SIGCHI conference on Human factors in computing systems: Changing our world, changing ourselves (pp. 415–422). Bem, D. J. & McConnell, H. K. (1970). Testing the self-perception explanation of dissonance phenomena: On the salience of premanipulation attitudes. Journal of Personality and Social Psychology, 14(1), 23–31. Bencina, R., Wilde, D. & Langley, S. (2008). Gesture≈sound experiments: process and mappings. In: Proceedings of the 8th International Conference on New Interfaces for Musical Expression (NIME 08) (pp. 197–202). Benford, S., Magerkurth, C. & Ljungstrand, P. (2005). Bridging the physical and digital in pervasive gaming. Communications of the ACM, 48(3), 54–57. Bennett, P., Ward, N., O’Modhrain, S. & Rebelo, P. (2007). DAMPER: a platform for effortful interface development. In: Proceedings of the 7th international conference on New interfaces for musical expression (pp. 273–276). Bentkowska-Kafel, A., Cashen, T. & Gardiner, H. (2009). Digital Visual Culture: Theory and Practice (Vol. 3). Intellect Books. Bentley, R. & Dourish, P. (1995). Medium versus mechanism: Supporting collaboration through customisation. In: Proceedings of the Fourth European Conference on ComputerSupported Cooperative Work (ECSCW’95) (pp. 133–148). Berghman, M. & Van Eijck, K. (2009). Visual arts appreciation patterns: Crossing horizontal and vertical boundaries within the cultural hierarchy. Poetics, 37(4), 348–365.
216
Berry, R., Makino, M., Hikawa, N., Suzuki, M. & Inoue, N. (2006). Tunes on the table. Multimedia Systems, 11(3), 280–289. Berti, A. & Frassinetti, F. (2000). When far becomes near: Remapping of space by tool use. Journal of cognitive neuroscience, 12(3), 415–420. Bevan, N. (1995a). Measuring usability as quality of use. Software Quality Journal, 4(2), 115– 130. Bevan, N. (1995b). Usability is quality of use. Advances in Human Factors/Ergonomics, 20, 349– 354. Bevan, N. (1999). Quality in use: Meeting user needs for quality. Journal of Systems and Software, 49(1), 89–96. Bevan, N. (2001a). International standards for HCI and usability. International journal of humancomputer studies, 55(4), 533–552. Bevan, N. (2001b). Quality in use for all. User interfaces for all. Concepts, methods, and tools, 353–368. Bevan, N. (2006). Practical issues in usability measurement. Interactions, 13(6), 42–43. Bevan, N., Petrie, H. & Claridge, N. (2007). Tenuta: strategies for providing guidance on usability and accessibility. In Universal Access in Human-Computer Interaction. Applications and Services (pp. 20–27). Springer. Bevilacqua, Frederic, Guédy, F., Schnell, N., Fléty, E. & Leroy, N. (2007). Wireless sensor interface and gesture-follower for music pedagogy. In: Proceedings of the 7th international conference on New interfaces for musical expression (pp. 124–129). Bevilacqua, Frédéric, Naugle, L. & Valverde, I. (2001). Virtual dance and music environment using motion capture. In Proc. of the IEEE-Multimedia Technology And Applications Conference, Irvine CA. Bevilacqua, Frédéric, Ridenour, J. & Cuccia, D. J. (2002). 3D motion capture data: motion analysis and mapping to music. In: Proceedings of the workshop/symposium on sensing and input for media-centric systems. Bianciardi, D., Igoe, T. & Singer, E. (2003). Eos Pods: Wireless Devices for Interactive Musical Performance. In: Proceedings of the Fifth Annual Conference on Ubiquitous Computing. Billinghurst, M. & Kato, H. (2002). Collaborative augmented reality. Communications of the ACM, 45(7), 64–70. Birchfield, D., Phillips, K., Kidané, A. & Lorig, D. (2006). Interactive public sound art: a case study. In: Proceedings of the 2006 conference on New interfaces for musical expression (pp. 43–48). Birchfield, D., Thornburg, H., Megowan-Romanowicz, M. C., Hatton, S., Mechtley, B., Dolgov, I. & Burleson, W. (2008). Embodiment, Multimodality, and Composition: Convergent Themes across HCI and Education for Mixed-Reality Learning Environments. Advances in Human-Computer Interaction, 2008, 1–19. Birnbaum, D., Fiebrink, R., Malloch, J. & Wanderley, M. M. (2005). Towards a dimension space for musical devices. In: Proceedings of the 2005 conference on New interfaces for musical expression (pp. 192–195). Birnbaum, D. M. & Wanderley, M. M. (2007). A systematic approach to musical vibrotactile feedback. In: Proceedings of the International Computer Music Conference (ICMC) (Vol. 2, pp. 397–404). Birringer, J. (2005). Interactivity: user testing for participatory artworks. International Journal of Performance Arts & Digital Media, 1(2), 147–173. Blackwell, T. & Young, M. (2004). Swarm granulator. Applications of Evolutionary Computing. 399–408. Blaine, T. (2005). The convergence of alternate controllers and musical interfaces in interactive entertainment. In: Proceedings of the 2005 conference on New interfaces for musical expression (pp. 27–33).
217
Blaine, T. & Fels, S. (2003). Collaborative musical experiences for novices. Journal of New Music Research, 32(4), 411–428. Blaine, T. & Forlines, C. (2002). Jam-O-World: evolution of the Jam-O-Drum multi-player musical controller into the Jam-O-Whirl gaming interface. In: Proceedings of the 2002 conference on New interfaces for musical expression (pp. 1–6). Blandford, A., Green, T. R., Furniss, D. & Makri, S. (2008). Evaluating system utility and conceptual fit using CASSM. International Journal of Human-Computer Studies, 66(6), 393–409. Blascovich, J., Loomis, J., Beall, A. C., Swinth, K. R., Hoyt, C. L. & Bailenson, J. N. (2002). Immersive virtual environment technology as a methodological tool for social psychology. Psychological Inquiry, 13(2), 103–124. Blythe, M. A., Overbeeke, K., Monk, A. F. & Wright, P. C. (2004). Funology: from usability to enjoyment (Vol 3.). Springer. Boehner, K., DePaula, R., Dourish, P. & Sengers, P. (2007). How emotion is made and measured. International Journal of Human-Computer Studies, 65(4), 275–291. Bonardi, A. & Rousseaux, F. (2002). Composing an Interactive Virtual Opera: The Virtualis Project. Leonardo, 35(3), 315–318. Bongers, A. (2000). Interaction in multimedia art. Knowledge-Based Systems, 13(7), 479–485. Bongers, A. J. (1998). Tactual display of sound properties in electronic musical instruments. Displays, 18(3), 129–133. Bongers, B. (1999). Exploring Novel ways of interaction in musical performance. In: Proceedings of the 3rd conference on Creativity & cognition (pp. 76–81). Bongers, B. (2007). Electronic musical Instruments: Experiences of a new Luthier. Leonardo Music Journal, 9–16. Bongers, B. & van der Veer, G. C. (2007). Towards a Multimodal Interaction Space: categorisation and applications. Personal and Ubiquitous Computing, 11(8), 609–619. Boone, R. T. & Cunningham, J. G. (2001). Children’s expression of emotional meaning in music through expressive body movement. Journal of Nonverbal Behavior, 25(1), 21–41. Borchers, J. & Mülhauser, M. (1997). Musical Design Patterns: An Example of a HumanCentered Model of Interactive Multimedia. In: Proceedings of the IEEE International Conference on Multimedia Computing and Systems’ 97 (pp. 63–70). Born, G. (2005). On musical mediation: ontology, technology and creativity. Twentieth-Century Music, 2(1), 7–36. Bower, J. E., Lindroth, S., Burchett, J., Feller, S. D., Brady, D. J. & Brady, R. (2005). Soundsense: Sonifying pyroelectric sensor data for an interactive media event. In: Proceedings of the International Conference on Auditory Display (pp. 113–120). Bowman, D. A., Gabbard, J. L. & Hix, D. (2002). A survey of usability evaluation in virtual environments: classification and comparison of methods. Presence: Teleoperators and Virtual Environments, 11(4), 404–424. Bown, O., Eldridge, A. & McCormack, J. (2009). Understanding Interaction in Contemporary Digital Music: from instruments to behavioural objects. Organised Sound, 14(02), 188– 196. Bresin, R. (2001). Articulation rules for automatic music performance. In: Proceedings of the International Computer Music Conference (pp. 294–297). Bresin, R. & Friberg, A. (2000). Emotional coloring of computer-controlled music performances. Computer Music Journal, 24(4), 44–63. Brown, A. R. (2008). Popular Music Cultures, Media and Youth Consumption: Towards an Integration of Structure, Culture and Agency. Sociology Compass, 2(2), 388–408. Brown, J. A. (1998). Media literacy perspectives. Journal of communication, 48(1), 44–57. Brown, S. (2005). The Neural Basis of Human Dance. Cerebral Cortex, 16(8), 1157–1167. Bryan-Kinns, N. & Mary, Q. (2004). Computers in Support of Muscial Expression. Citeseer.
218
Bryson, B. (1997). What about the univores? Musical dislikes and group-based identity construction among Americans with low levels of education. Poetics, 25(2), 141–156. Buxbaum, L. J., Giovannetti, T. & Libon, D. (2000). The Role of the Dynamic Body Schema in Praxis: Evidence from Primary Progressive Apraxia. Brain and Cognition, 44(2), 166– 191. Calvert, G. A., Spence, C. & Stein, B. E. (2004). The handbook of multisensory processes. MIT press. Calvo-Merino, B. (2004). Action Observation and Acquired Motor Skills: An fMRI Study with Expert Dancers. Cerebral Cortex, 15(8), 1243–1249. Camurri, A. (2009). Non-verbal full body emotional and social interaction: a case study on multimedia systems for active music listening. Intelligent Technologies for Interactive Entertainment (pp. 9–18). Camurri, A., Canepa, C. & Volpe, G. (2007). Active listening to a virtual orchestra through an expressive gestural interface: The Orchestra Explorer. In: Proceedings of the 7th international conference on New interfaces for musical expression (pp. 56–61). Camurri, A. & Ferrentino, P. (1999). Interactive environments for music and multimedia. Multimedia systems, 7(1), 32–47. Camurri, A., Volpe, G., Vinet, H., Bresin, R., Fabiani, M. & Dubus, G., (2010). User-centric context-aware mobile applications for embodied music listening. User Centric Media (pp. 21–30). Cardinali, L., Brozzoli, C. & Farne, A. (2009). Peripersonal space and body schema: two labels for the same concept? Brain topography, 21(3-4), 252–260. Carlile, J. & Hartmann, B. (2005). OROBORO: a collaborative controller with interpersonal haptic feedback. In: Proceedings of the 2005 conference on New interfaces for musical expression (pp. 250–251). Carlson, R., Friberg, A., Frydén, L., Granström, B. & Sundberg, J. (1989). Speech and music performance: Parallels and contrasts. Contemporary Music Review, 4(1), 391–404. Carman, T. (2000). The body in Husserl and Merleau-Ponty. Philosophical topics, 27(2), 205– 226. Carrol, J. M. (1999). Five reasons for scenario-based design. In System Sciences, 1999. HICSS32. Proceedings of the 32nd Annual Hawaii International Conference on (p. 11–pp). Castellano, G., Bresin, R., Camurri, A. & Volpe, G. (2007). Expressive control of music and visual media by full-body movement. In: Proceedings of the 7th international conference on New interfaces for musical expression (pp. 390–391). Chaminade, T., Meltzoff, A. N. & Decety, J. (2005). An fMRI study of imitation: action representation and body schema. Neuropsychologia, 43(1), 115–127. Chapados, C. & Levitin, D. J. (2008). Cross-modal interactions in the experience of musical performances: Physiological correlates. Cognition, 108(3), 639–651. Chew, E., Zimmermann, R., Sawchuk, A. A., Kyriakakis, C., Papadopoulos, C., François, A. & Volk, A. (2004). Musical interaction at a distance: Distributed immersive performance. In: Proceedings of the Fourth Open Workshop on Integration of Music in Multimedia Applications (MusicNetwork) (pp. 15–16). Ciolfi, L., Fernstrom, M., Bannon, L. J., Desbpande, P., Gallagher, P., McGettrick, C. & Shirley, S. (2007). The shannon portal installation: Interaction design for public places. Computer, 40(7), 64–71. Clayton, M., Sager, R. & Will, U. (2005). In time with the music: The concept of entrainment and its significance for ethnomusicology. In: European meetings in ethnomusicology, 11, 3– 142. Clemmensen, T. (2004). Four approaches to user modeling—a qualitative research interview study of HCI professionals’ practice. Interacting with Computers, 16(4), 799–829.
219
Coleman, G. W., Hand, C., Macaulay, C. & Newell, A. F. (2005). Approaches to auditory interface design-lessons from computer games. In: Proceedings of the 2005 International Conference on Auditory Display. Collins, N. (2003). Generative music and laptop performance. Contemporary Music Review, 22(4), 67–79. Collins, N, Kiefer, C., Patoli, M. & White, M. (2010). Musical exoskeletons: Experiments with a motion capture suit. In: Proceedings of New Interfaces for Musical Expression (NIME), Sydney, Australia. Collins, N. (2009). Handmade electronic music: the art of hardware hacking. New York: Routledge. Cook, J. (1998). Mentoring, metacognition and music: Interaction analyses and implications for intelligent learning environments. International Journal of Artificial Intelligence in Education, 9, 45–87. Cornelis, O., Demey, M. & Leman, M. (2008). EME: a Wireless Music Controller for Real-time Music Interaction. In ARTECH 2008 4th International Conference on Digital Arts (pp. 212–215). Corness, G. (2008). Performer Model: Towards a Framework for Interactive Performance Based on Perceived Intention. In: Proceedings of the International Conference on New Interfaces for Musical Expression$}$ (pp. 265–268). Coughlan, T. & Johnson, P. (2006). Interaction in creative tasks; Ideation, Representation and Evaluation in Composition. In: Proceedings of the SIGCHI conference on Human Factors in computing systems (pp. 531–540). Coughlan, T. & Johnson, P. (2009). Understanding productive, structural and longitudinal interactions in the design of tools for creative activities. In: Proceedings of the seventh ACM conference on Creativity and cognition (pp. 155–164). Courtois, C., Mechant, P., Paulussen, S. & De Marez, L. (2012). The triple articulation of media technologies in teenage media consumption. New Media & Society, 14(3), 401–420. Couturier, J.-M. (2006). A model for graphical interaction applied to gestural control of sound. In: Proceedings of the international conference on Sound and Music Computing, Marseille. Creem-Regehr, S. H., Willemsen, P., Gooch, A. A. & Thompson, W. B. (2005). The influence of restricted viewing conditions on egocentric distance perception: Implications for real and virtual environments. Perception, 34(2), 191–204. Croft, J. (2007). Theses on liveness. Organised Sound, 12(1), 59. Cross, I. (1998). Music analysis and music perception. Music Analysis, 17(1), 3–20. Csikszentmihalyi, M. & Hunter, J. (2003). Happiness in everyday life: The uses of experience sampling. Journal of Happiness Studies, 4(2), 185–199. D’Arcangelo, G. (2001). Creating contexts of creativity: musical composition with modular components. In: Proceedings of the 2001 conference on New interfaces for musical expression (pp. 1–4). D’Arcangelo, G. (2002). Creating a context for musical innovation: a NIME curriculum. In: Proceedings of the 2002 conference on New interfaces for musical expression (pp. 1–4). D’Inca, G. & Mion, L. (2006). Expressive audio synthesis: From performances to sounds. In: Proceedings of the 12th International Conference on Auditory Display (pp. 182–186). Dahl, S. & Friberg, A. (2007). Visual perception of expressiveness in musicians’ body movements. Music Perception: An Interdisciplinary Journal, 24(5), 433–454. Dahlstedt, P. & Nordahl, M. G. (2001). Living melodies: Coevolution of sonic communication. Leonardo, 34(3), 243–248. Dassonville, P., Lewis, S. M., Zhu, X.-H., Ugurbil, K., Kim, S.-G. & Ashe, J. (2001). The Effect of Stimulus–Response Compatibility on Cortical Motor Activation. NeuroImage, 13(1), 1–14.
220
De Bruyn, L., Moelants, D. & Leman, M. (2012). An embodied approach to testing musical empathy in participants with an autism spectrum disorder. Music and Medicine, 4(1), 28–36. De Gelder, B. & Bertelson, P. (2003). Multisensory integration, perception and ecological validity. Trends in cognitive sciences, 7(10), 460–467. De Kort, Y. A. & Ijsselsteijn, W. A. (2008). People, places, and play: player experience in a socio-spatial context. Computers in Entertainment (CIE), 6(2), 18. De Kort, Y. A., IJsselsteijn, W. A. & Poels, K. (2007). Digital games as social presence technology: Development of the Social Presence in Gaming Questionnaire (SPGQ). In: Proceedings of PRESENCE 2007: The 10th International Workshop on Presence (pp. 195–203). De Marez, L. & De Moor, K. (2007). The challenge of user- and QoE-centric research and product development in today’s ict-environment. Observatorio (OBS*), 1(3). Pp? De Moor, K., Ketyko, I., Joseph, W., Deryckere, T., De Marez, L., Martens, L. & Verleye, G. (2010). Proposed framework for evaluating quality of experience in a mobile, testbedoriented living lab setting. Mobile Networks and Applications, 15(3), 378–391. De Oliveira Rocha, F. (2008). Performance with Electronics (Doctoral dissertation). Schulich School of Music, McGill University, Montreal, Canada. De Preester, H. (2007). The deep bodily origins of the subjective perspective: Models and their problems. Consciousness and Cognition, 16(3), 604–618. De Preester, H. (2011). Technology and the body: the (im)possibilities of re-embodiment. Foundations of science, 16(2-3), 119–137. De Preester, H. & Tsakiris, M. (2009). Body-extension versus body-incorporation: Is there a need for a body-model? Phenomenology and the Cognitive Sciences, 8(3), 307–319. De Quay, Y., Skogstad, S. & Jensenius, A. (2011). Dance Jockey: Performing Electronic Music by Dancing. Leonardo Music Journal, 21, 11–12. Dean, R. T., Whitelaw, M., Smith, H. & Worrall, D. (2006). The mirage of real-time algorithmic synaesthesia: Some compositional mechanisms and research agendas in computer music and sonification. Contemporary Music Review, 25(4), 311–326. Desurvire, H. & Wiberg, C. (2009). Game usability heuristics (play) for evaluating and designing better games: The next iteration. Online Communities and Social Computing. 557–566 Deutscher, M. C. (2008). Participant experience studies of interactive artworks: an investigation of laboratory-based methods used to study Echology. Deutscher, M., Hoskinson, R., Takashashi, S. & Fels, S. (2005). Echology: an interactive spatial sound and video artwork. In: Proceedings of the 13th annual ACM international conference on Multimedia (pp. 937–945). Deweppe, A., Diniz, N., Coussement, P., Lesaffre, M. & Leman, M. (2010). Evaluating and Implementing Strategies for Embodied Music Interaction. In: Proceedings of the 10th Conference on Interdisciplinary Musicology (pp. 19–21). Sheffield, UK. Deweppe, A., Lesaffre, M. & Leman, M. (2009). Discerning user profiles and establishing usability for interactive music applications that use embodied mediation technology. Dey, A. K., Abowd, G. D. & Salber, D. (2001). A conceptual framework and a toolkit for supporting the rapid prototyping of context-aware applications. Human-computer interaction, 16(2), 97–166. Dillon, A. & Morris, M. G. (1996). User acceptance of new information technology: theories and models. Diniz, N. C. D. S., Demey, M. & Leman, M. (2010). An Interactive Framework for music-based interactive sonification. In: Proceedings for the conference on Interactive Sonification (ISon 2010) (pp. 1–4). Stockholm, Sweden. Dobrian, C. & Koppelman, D. (2006). The’E’in NIME: musical expression with new computer interfaces. In: Proceedings of the 2006 conference on New interfaces for musical expression (pp. 277–282).
221
Donker, A. & Markopoulos, P. (2002). A comparison of think-aloud, questionnaires and interviews for testing usability with children. In: People and Computers XVI Memorable Yet Invisible (pp. 305–316). Springer. Dourish, P. (1993). Culture and control in a media space. In: Proceedings of the Third European Conference on Computer-Supported Cooperative Work 13–17 September 1993, Milan, Italy ECSCW’93 (pp. 125–137). Dourish, P. (1997). Accounting for system behaviour: Representation, reflection and resourceful action. Computers and Design in Context, MIT Press, Cambridge, MA, USA, 145–170. Dourish, P. (2001). Seeking a foundation for context-aware computing. Human–Computer Interaction, 16(2-4), 229–241. Dourish, P. (2003). The appropriation of interactive technologies: Some lessons from placeless documents. Computer Supported Cooperative Work, 12(4), 465–490. Dourish, P., Anderson, K. & Nafus, D. (2007). Cultural Mobilities: Diversity and Agency in Urban Computing. In C. Baranauskas, P. Palanque, J. Abascal & S. Barbosa (Eds.), Human-Computer Interaction – INTERACT 2007 (Vol. 4663, pp. 100–113). Springer Berlin Heidelberg. Dourish, P. & Button, G. (1998). On “technomethodology”: Foundational relationships between ethnomethodology and system design. Human-Computer Interaction, 13(4), 395–432. Driver, C. & Clarke, S. (2008). An application framework for mobile, context-aware trails. Pervasive and Mobile Computing, 4(5), 719–736. Driver, J. & Spence, C. (2000). Multisensory perception: beyond modularity and convergence. Current Biology, 10(20), 731–735. Duckworth, W. (1999). Making music on the web. Leonardo Music Journal, 9, 13–17. Duignan, M., Noble, J. & Biddle, R. (2010). Abstraction and activity in computer-mediated music production. Computer Music Journal, 34(4), 22–33. Ebenreuter, N. (2005). Dance movement: a focus on the technology. IEEE Computer Graphics and Applications, 25(6), 80–83. Eitan, Z. & Granot, R. Y. (2006). How Music Moves: Musical Parameters and Listeners Images of Motion. Music perception, 23(3), 221–248. El-Nasr, M. S. & Vasilakos, A. V. (2008). DigitalBeing–using the environment as an expressive medium for dance. Information Sciences, 178(3), 663–678. Ellis, P. & van Leeuwen, L. (2009). Confronting the Transition: Improving Quality of Life for the Elderly with an Interactive Multisensory Environment–A Case Study. In: Universal Access in Human-Computer Interaction. Addressing Diversity (pp. 210–219). Springer. Emmerson, S. (1998). Aural landscape: musical space. Organised Sound, 3(2), 135–140. Emmerson, S. (2000). Music, electronic media, and culture. Ashgate Publishing. England, D. (2011). Whole body interaction. London; New York: Springer. Erstad, O., Gilje, Ø. & de Lange, T. (2007). Re‐mixing multimodal resources: multiliteracies and digital production in Norwegian media education. Learning, Media and Technology, 32(2), 183–198. Farnè, A., Iriki, A. & Làdavas, E. (2005). Shaping multisensory action–space with tools: evidence from patients with cross-modal extinction. Neuropsychologia, 43(2), 238–248. Farnè, A. & Làdavas, E. (2002). Auditory peripersonal space in humans. Journal of Cognitive Neuroscience, 14(7), 1030–1043. Farnè, A., Serino, A. & Làdavas, E. (2007). Dynamic size-change of peri-hand space following tool-use: determinants and spatial characteristics revealed through cross-modal extinction. Cortex, 43(3), 436–443. Favilla, S. & Cannon, J. (2006). Children of Grainger: Leather instruments for free music. In: Proceedings of the 2006 conference on New interfaces for musical expression (pp. 370– 375). Feld, S. (1984). Communication, music, and speech about music. Yearbook for Traditional Music, 16, 1–18.
222
Feldmeier, M. & Paradiso, J. A. (2007). An interactive music environment for large groups with giveaway wireless motion sensors. Computer Music Journal, 31(1), 50–67. Fels, S. (2000). Intimacy and embodiment: implications for art and technology. In: Proceedings of the 2000 ACM workshops on Multimedia (pp. 13–16). Fels, S. (2004). Designing for intimacy: Creating new interfaces for musical expression. Proceedings of the IEEE, 92(4), 672–685. Fels, S., Gadd, A. & Mulder, A. (2002). Mapping transparency through metaphor: towards more expressive musical instruments. Organised Sound, 7(2), 109–126. Fencott, R. & Bryan-Kinns, N. (2010). Hey man, you’re invading my personal space! privacy and awareness in collaborative music. In: Proceedings of the 10th International Conference on New Interfaces for Musical Expression (NIME 2010) (pp. 198–203). Ferguson, S. & Wanderley, M. M. (2010). The McGill Digital Orchestra: An interdisciplinary project on digital musical instruments. Journal of Interdisciplinary Music Studies, 4(2), 17–35. Fernström, M. & Brazil, E. (2004). Human-computer interaction design based on interactive sonification-hearing actions or instruments/agents. In: Proceedings of the 2004 International Workshop on Interactive Sonification (pp. 1–4). Fernström, M., Brazil, F. & Bannon, L. (2005). Hci design and interactive sonification for fingers and ears. IEEE Multimedia, 12(2), 36–44. Fernström, M. & McNamara, C. (2005). After direct manipulation—direct sonification. ACM Transactions on Applied Perception, 2(4), 495–499. Field, A. P. (2009). Discovering statistics using SPSS. London: SAGE. Fink-Jensen, K. (2007). Attunement and bodily dialogues in music education. Philosophy of Music Education Review, 15(1), 53–68. Fischer, G. (2001). User modeling in human–computer interaction. User modeling and useradapted interaction, 11(1-2), 65–86. Fischer, G. (2003). Meta-design: Beyond user-centered and participatory design. In: Proceedings of HCI International (pp. 88–92). Fischer, G. (2009). End-user development and meta-design: Foundations for cultures of participation. In: End-user development (pp. 3–14). Springer. Fischer, G., Giaccardi, E., Ye, Y., Sutcliffe, A. G. & Mehandjiev, N. (2004). Meta-design: a manifesto for end-user development. Communications of the ACM, 47(9), 33–37. Fischer, G., Nakakoji, K. & Ye, Y. (2009). Metadesign: Guidelines for supporting domain experts in software development. Software, IEEE, 26(5), 37–44. Fishkin, K. P. (2004). A taxonomy for and analysis of tangible interfaces. Personal and Ubiquitous Computing, 8(5), 347–358. Fjeld, M., Lauche, K., Bichsel, M., Voorhorst, F., Krueger, H. & Rauterberg, M. (2002). Physical and virtual tools: Activity theory applied to the design of groupware. Computer Supported Cooperative Work, 11(1-2), 153–180. Franzosi, R. (1998). Narrative analysis-or why (and how) sociologists should be interested in narrative. Annual review of sociology, 517–554. Fraser, M., Glover, T., Vaghi, I., Benford, S., Greenhalgh, C., Hindmarsh, J. & Heath, C. (2000). Revealing the realities of collaborative virtual reality. In: Proceedings of the third international conference on Collaborative virtual environments (pp. 29–37). Freeman, J. (2008). Extreme sight-reading, mediated expression, and audience participation: Realtime music notation in live performance. Computer Music Journal, 32(3), 25–41. Freeman, J., Varnik, K., Ramakrishnan, C., Neuhaus, M., Burk, P. & Birchfield, D. (2005). Auracle: a voice-controlled, networked sound instrument. Organised Sound, 10(3), 221. Friberg, A., Bresin, R. & Sundberg, J. (2006). Overview of the KTH rule system for musical performance. Advances in Cognitive Psychology, 2(2-3), 145–161. Frisoli, A. & Camurri, A. (2006). Special Issue Editorial: Multisensory interaction in virtual environments. Virtual Reality, 10(1), 2–3.
223
Fuchs, T. (2005). Corporealized and disembodied minds: a phenomenological view of the body in melancholia and schizophrenia. Philosophy, Psychiatry & Psychology, 12(2), 95–107. Fuller, S. (2012). Preparing for life in humanity 2.0. Palgrave Pivot. Fyans, A. C., Gurevich, M. & Stapleton, P. (2009). Where did it all go wrong? A model of error from the spectator’s perspective. In: Proceeding of the Conference on New Instruments for Musical Expression (NIME). Gabbard, J. L., Hix, D. & Swan, J. E. (1999). User-centered design and evaluation of virtual environments. Computer Graphics and Applications, IEEE, 19(6), 51–59. Gabrielsson, A. & Juslin, P. N. (1996). Emotional expression in music performance: Between the performer’s intention and the listener’s experience. Psychology of music, 24(1), 68–91. Gajadhar, B. J., de Kort, Y. A. & IJsselsteijn, W. A. (2008). Shared fun is doubled fun: player enjoyment as a function of social setting. In: Fun and games (pp. 106–117). Springer. Gallagher, S. (2001). Dimensions of embodiment: Body image and body schema in medical contexts. In: Handbook of phenomenology and medicine (pp. 147–175). Springer. Gallagher, S. (2005). How the body shapes the mind. Cambridge University Press. Gallese, V. (2005). Embodied simulation: From neurons to phenomenal experience. Phenomenology and the cognitive sciences, 4(1), 23–48. Gallese, V. (2006). Intentional attunement: A neurophysiological perspective on social cognition and its disruption in autism. Brain research, 1079(1), 15–24. Gallese, V. (2007). Before and below “theory of mind”: embodied simulation and the neural correlates of social cognition. Philosophical Transactions of the Royal Society: Biological Sciences, 362(1480), 659–669. Gallese, V., Eagle, M. N. & Migone, P. (2007). Intentional attunement: Mirror neurons and the neural underpinnings of interpersonal relations. Journal of the American Psychoanalytic Association, 55(1), 131–175. Garrety, K. & Badham, R. (2004). User-centered design and the normative politics of technology. Science, Technology & Human Values, 29(2), 191–212. Gasson, S. (1999). The reality of user-centered design. Journal of End User Computing, 11(4), 3– 13. Gaye, L., Holmquist, L. E., Behrendt, F. & Tanaka, A. (2006). Mobile music technology: report on an emerging community. In: Proceedings of the 2006 conference on New interfaces for musical expression (pp. 22–25). Gere, C. (2002). Digital culture. Reaktion Books. Gill, S. P. (2007). Entrainment and musicality in the human system interface. AI & Society, 21(4), 567–605. Gillespie, T. (2010). The politics of “platforms.” New Media & Society, 12(3), 347–364. Glass, R. & Stevens, C. (2005). Making sense of contemporary dance: An Australian investigation into audience interpretation and enjoyment levels. Sydney: University of Sydney. Glinert, E. M. (2008). The human controller: usability and accessibility in video game interfaces. Massachusetts Institute of Technology. (thesis?) Godøy, R. I., Haga, E. & Jensenius, A. R. (2006). Playing “air instruments”: mimicry of soundproducing gestures by novices and experts. In: Gesture in Human-Computer Interaction and Simulation (pp. 256–267). Springer. Godøy, R. I. & Leman, M. (2009). Musical gestures: Sound, movement, and meaning. Routledge. González Gutiérrez, J. L., Jiménez, B. M., Hernández, E. G. & Puente, C. P. (2005). Personality and subjective well-being: Big five correlates and demographic variables. Personality and Individual Differences, 38(7), 1561–1569. Gosling, S. D., Rentfrow, P. J. & Swann, W. B. (2003). A very brief measure of the Big-Five personality domains. Journal of Research in Personality, 37(6), 504–528. Gosling, S. D., Vazire, S., Srivastava, S. & John, O. P. (2004). Should We Trust Web-Based Studies? A Comparative Analysis of Six Preconceptions About Internet Questionnaires. American Psychologist, 59(2), 93–104.
224
Goto, S. (1999). The aesthetics and technological aspects of virtual musical instruments: The case of the SuperPolm MIDI violin. Leonardo Music journal, 115–120. Goto, S. (2000). Virtual musical instruments: Technological aspects and interactive performance issues. IRCAM, Trends in Gestural Control of Music. Goudeseune, C. (2002). Interpolated mappings for musical instruments. Organised Sound, 7(2), 85–96. Grindlay, G. & Helmbold, D. (2006). Modeling, analyzing, and synthesizing expressive piano performance with graphical models. Machine Learning, 65(2-3), 361–387. Grisey, G. (1987). Tempus ex Machina: A composer’s reflections on musical time. Contemporary Music Review, 2(1), 239–275. Gulliksen, J., Boivie, I. & Göransson, B. (2006). Usability professionals—current practices and future development. Interacting with Computers, 18(4), 568–600. Gurevich, M. (2006). JamSpace: designing a collaborative networked music space for novices. In: Proceedings of the 2006 conference on New interfaces for musical expression (pp. 118– 123). Gurevich, M., Stapleton, P. & Marquez-Borbon, A. (2010). Style and constraint in electronic musical instruments. In: Proceedings of the 2006 conference on New interfaces for musical expression (Vol. 10). Gurevich, M. & Treviño, J. (2007). Expression and its discontents: toward an ecology of musical creation. In: Proceedings of the 7th international conference on New interfaces for musical expression (pp. 106–111). Haddon, L. (2006). The contribution of domestication research to in-home computing and media consumption. The Information Society, 22(4), 195–203. Hagen, E. H. & Bryant, G. A. (2003). Music and dance as a coalition signaling system. Human nature, 14(1), 21–51. Halton, E. (2004). The living gesture and the signifying moment. Symbolic interaction, 27(1), 89– 113. Hamman, M. (1999). From symbol to semiotic: Representation, signification, and the composition of music interaction. Journal of new music research, 28(2), 90–104. Hamman, M. (2000). From technical to technological: Interpreting technology through composition. Proceedings of the 2000 Coloquium on Musical Informatics. Hamman, M. (2002). From technical to technological: The imperative of technology in experimental music composition. Perspectives of New Music, 40(1), 92–120. Hand, M. (2008). Making digital cultures: Access, interactivity, and authenticity. Ashgate Publishing, Ltd. Hanington, Bruce. (2003). Methods in the making: A perspective on the state of human research in design. Design Issues, 19(4), 9–18. Hargreaves, D. J., Miell, D. & MacDonald, R. A. (2002). What are musical identities, and why are they important. Musical identities, 1–20. Harris, Y. (2006). Inside-out instrument. Contemporary Music Review, 25(1-2), 151–162. Harrison, J. (2005). SoundBlocks and SoundScratch: tangible and virtual digital sound programming and manipulation for children. Massachusetts Institute of Technology. Hartmann, B., Doorley, S. & Klemmer, S. R. (2008). Hacking, mashing, gluing: Understanding opportunistic design. Pervasive Computing, IEEE, 7(3), 46–54. Hartson, H. R., Andre, T. S. & Williges, R. C. (2003). Criteria for evaluating usability evaluation methods. International Journal of Human-Computer Interaction, 15(1), 145–181. Hartson, R., H. (1998). Human–computer interaction: Interdisciplinary roots and trends. Journal of Systems and Software, 43(2), 103–118. Hattwick, I., & Wanderley, M. M. (2012). A Dimension Space for Evaluating Collaborative Musical Performance Systems. New Interfaces for Musical Expression (12), 21–23. Heath, C., Luff, P., Vom Lehn, D., Hindmarsh, J. & Cleverly, J. (2002). Crafting participation: designing ecologies, configuring experience. Visual Communication, 1(1), 9–33.
225
Henninger, S. (2000). A methodology and tools for applying context-specific usability guidelines to interface design. Interacting with computers, 12(3), 225–243. Henninger, S. (2001). An organizational learning method for applying usability guidelines and patterns. In: Engineering for Human-Computer Interaction (pp. 141–155). Springer. Hennion, A. (2003). Music and mediation: towards a new sociology of music. The cultural study of music: A critical introduction, 80–91. Hermann, T. & Hunt, A. (2005). Guest Editors’ Introduction: An Introduction to Interactive Sonification. Multimedia, IEEE, 12(2), 20–24. Hermann, T., Hunt, A. & Neuhoff, J. G. (2011). The sonification handbook. Logos Verlag. Hertzum, M. & Jacobsen, N. E. (2001). The evaluator effect: A chilling fact about usability evaluation methods. International Journal of Human-Computer Interaction, 13(4), 421– 443. Herzberg, P. Y. & Brähler, E. (2006). Assessing the Big-Five personality domains via short forms. European Journal of Psychological Assessment, 22(3), 139–148. Higuchi, T., Imanaka, K. & Patla, A. E. (2006). Action-oriented representation of peripersonal and extrapersonal space: Insights from manual and locomotor actions. Japanese Psychological Research, 48(3), 126–140. Hilti, L. M. & Brugger, P. (2010). Incarnation and animation: physical versus representational deficits of body integrity. Experimental brain research, 204(3), 315–326. Hindmarsh, J., Fraser, M., Heath, C., Benford, S. & Greenhalgh, C. (2000). Object-focused interaction in collaborative virtual environments. ACM Transactions on ComputerHuman Interaction (TOCHI), 7(4), 477–509. Hirst, D. & others. (2002). Developing Analysis Criteria Based on Denis Smalley’s Timbre Theories. In: Proceedings of the Australasian Computer Music Conference 2002 (pp. 43– 52). Hobbs, R. (1998). The seven great debates in the media literacy movement. Journal of communication, 48 (1), 16-32. Hodges, D. A., Hairston, W. D. & Burdette, J. H. (2005). Aspects of multisensory perception: the integration of visual and auditory information in musical experiences. Annals of the New York Academy of Sciences, 1060(1), 175–185. Holmes, N. P. & Spence, C. (2006). Beyond the body schema. Human body perception from the inside out. Oxford University Press, Oxford, 15–64. Höök, K., Sengers, P. & Andersson, G. (2003). Sense and sensibility: evaluation and interactive art. In: Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 241–248). Hornbæk, K. (2006). Current practice in measuring usability: Challenges to usability studies and research. International journal of human-computer studies, 64(2), 79–102. Hoyt, C. L., Blascovich, J. & Swinth, K. R. (2003). Social inhibition in immersive virtual environments. Presence: Teleoperators and Virtual Environments, 12(2), 183–195. Hsu, W. (2006). Managing gesture and timbre for analysis and instrument control in an interactive environment. In: Proceedings of the 2006 conference on New interfaces for musical expression (pp. 376–379). Hunt, A. & Hermann, T. (2004). The importance of interaction in sonification. Computer Music Journal, 5(6), xx–xx Hunt, A. & Wanderley, M. M. (2002). Mapping performer parameters to synthesis engines. Organised Sound, 7(2), 97–108. Husbands, P., Copley, P., Eldridge, A. & Mandelis, J. (2007). An introduction to evolutionary computing for musicians. In: Evolutionary Computer Music (pp. 1–27). Springer. Hvannberg, E. T., Law, E. L.-C. & Lérusdóttir, M. K. (2007). Heuristic evaluation: Comparing ways of finding and reporting usability problems. Interacting with computers, 19(2), 225–240.
226
Iivari, J. & Iivari, N. (2011). Varieties of user-centeredness: an analysis of four systems development methods. Information Systems Journal, 21(2), 125–153. IJsselsteijn, W., van den Hoogen, W., Klimmt, C., de Kort, Y., Lindley, C., Mathiak, K. & Vorderer, P. (2008). Measuring the experience of digital game enjoyment. In: Proceedings of Measuring Behavior (pp. 88–89). Impett, J. (1998). The identification and transposition of authentic instruments: Musical practice and technology. Leonardo Music Journal, 21–26. Isbister, K. (2010). Enabling social play: A framework for design and evaluation. In: Evaluating User Experience in Games (pp. 11–22). Springer. Isbister, K. (2011). Emotion and motion: games as inspiration for shaping the future of interface. Interactions, 18(5), 24–27. Jackendoff, R. & Lerdahl, F. (2006). The capacity for music: What is it, and what’s special about it? Cognition, 100(1), 33–72. Jacobson, L. (1992). CyberArts: exploring art & technology. San Francisco: Miller Freeman. Jeffries, R., Miller, J. R., Wharton, C. & Uyeda, K. (1991). User interface evaluation in the real world: a comparison of four techniques. In: Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 119–124). Jensenius, A. R., Kvifte, T. & Godøy, R. I. (2006). Towards a gesture description interchange format. In: Proceedings of the 2006 conference on New interfaces for musical expression (pp. 176–179). Jensenius, A. R.. (2007). Action-sound: Developing methods and tools to study music-related body movement. Oslo, Norway. Jensenius, A. R., Koehly, R. & Wanderley, M. M. (2006). Building low-cost music controllers. In: Computer Music Modeling and Retrieval (pp. 123–129). Springer. John, B. E. & Kieras, D. E. (1996a). The GOMS family of user interface analysis techniques: Comparison and contrast. ACM Transactions on Computer-Human Interaction, 3(4), 320–351. Johnston, A., Candy, L. & Edmonds, E. (2008). Designing and evaluating virtual musical instruments: facilitating conversational user interaction. Design Studies, 29(6), 556–571. Jokela, T. (2004). Evaluating the user-centeredness of development organisations: conclusions and implications from empirical usability capability maturity assessments. Interacting with Computers, 16(6), 1095–1132. Jokela, T. (2008). Characterizations, Requirements, and Activities of User-Centered Design—the KESSU 2.2 Model. In: Maturing Usability (pp. 168–196). Springer. Jokela, T., Siponen, M., Hirasawa, N. & Earthy, J. (2006). A survey of usability capability maturity models: implications for practice and research. Behaviour & Information Technology, 25(03), 263–282. Jorda, S. (2005). Digital Lutherie (PhD Thesis). Pompeu Fabra University. Jordà, S. (2003). Sonigraphical instruments: from FMOL to the reacTable. In: Proceedings of the 2003 conference on New interfaces for musical expression (pp. 70–76). Jordà, S. (2004). Digital instruments and players: part I—efficiency and apprenticeship. In: Proceedings of the 2004 conference on New interfaces for musical expression (pp. 59– 63). Jordà, S. (2005). Multi-user instruments: models, examples and promises. In: Proceedings of the 2005 conference on New interfaces for musical expression (pp. 23–26). Juslin, P. N. (2003). Five facets of musical expression: A psychologist’s perspective on music performance. Psychology of Music, 31(3), 273–302. Juslin, P. N. & Sloboda, J. A. (2010). Handbook of music and emotion: Theory, research, applications. Oxford University Press. Kaiser, G., Ekblad, G. & Broling, L. (2010). Audience participation in a dance-club context: Design of a system for collaborative creation of visuals. Nordes, (2).
227
Kaltenbrunner, M., Geiger, G. & Jordà, S. (2004). Dynamic patches for live musical performance. In: Proceedings of the 2004 conference on New interfaces for musical expression (pp. 19–22). Kamalian, R., Takagi, H. & Agogino, A. M. (2004). Optimized design of MEMS by evolutionary multi-objective optimization with interactive evolutionary computation. In: Genetic and Evolutionary Computation (pp. 1030–1041). Kammers, M., Van der Ham, I. & Dijkerman, H. (2006). Dissociating body representations in healthy individuals: differential effects of a kinaesthetic illusion on perception and action. Neuropsychologia, 44(12), 2430–2436. Kaper, H. G., Wiebel, E. & Tipei, S. (1999). Data sonification and sound visualization. Computing in science & engineering, 1(4), 48–58. Kaptelinin, V., Kuutti, K. & Bannon, L. (1995). Activity theory: Basic concepts and applications. In: Human-computer interaction (pp. 189–201). Springer. Kaptelinin, V. & Nardi, B. A. (2006). Acting with technology. Mit Press. Kaptelinin, V., Nardi, B. A. & Macaulay, C. (1999). Methods & tools: The activity checklist: a tool for representing the “space” of context. Interactions, 6(4), 27–39. Kapur, A., Tindale, A., Benning, M. & Driessen, P. (2006). The KiOm: A Paradigm for Collaborative Controller Design. In the Proceedings of the International Computer Music Conference (ICMC). Keikonen, T., Jääskö, V. & Mattelmäki, T. (2008). Three-in-one user study for focused collaboration. International Journal of Design, 2(1), 1–10. Keinonen, T. (2008). User-centered design and fundamental need. In: Proceedings of the 5th Nordic conference on Human-computer interaction: building bridges (pp. 211–219). Kensing, F. & Blomberg, J. (1998). Participatory design: Issues and concerns. Computer Supported Cooperative Work, 7(3-4), 167–185. Khalid, H. M. (2006). Embracing diversity in user needs for affective design. Applied Ergonomics, 37(4), 409–418. Kiefer, C. (2010a). A malleable interface for sonic exploration. In: Proceedings of New Interfaces for Musical Expression. Kiefer, C. (2010b). Input devices and mapping techniques for the intuitive control of composition and editing for digital music. In: Proceedings of the fourth international conference on Tangible, embedded, and embodied interaction (pp. 311–312). Kiefer, C., Collins, N. & Fitzpatrick, G. (2008a). Evaluating the wiimote as a musical controller. In: Proceedings of the 2008 International Computer Music Conference (ICMC). Kiefer, C., Collins, N. & Fitzpatrick, G. (2008b). HCI methodology for evaluating musical controllers: A case study. In: Proceedings of NIME (Vol. 8). Kim, J. H. & Seifert, U. (2006). Embodiment: The body in algorithmic sound generation. Contemporary Music Review, 25(1-2), 139–149. Kim, J. H. & Seifert, U. (2007). Embodiment and agency: Towards an aesthetics of interactive performativity. In: Proceedings of the 4th Sound and Music Computing Conference (pp. 230–237). Kim, J.-H., Youn, J.-S. & Hong, K.-S. (2007). Toward a novel multi-modal HCI: fusion architecture using confidence score and fuzzy value. In: Knowledge-Based Intelligent Information and Engineering Systems (pp. 246–253). Knoblich, G. (2002). Self-recognition: Body and action. Trends in cognitive sciences, 6(11), 447– 449. Krumhansl, C. L. (2002). Music: A link between cognition and emotion. Current Directions in Psychological Science, 11(2), 45–50. Kuyken, B., Verstichel, W., Bossuyt, F., Vanfleteren, J., Demey, M. & Leman, M. (2008). The HOP sensor: wireless motion sensor. In: Proceedings of the International Conference on New Interfaces for Musical Expression (pp. 229–231).
228
Kvifte, T. & Jensenius, A. R. (2006). Towards a coherent terminology and model of instrument description and design. In: Proceedings of the 2006 conference on New interfaces for musical expression (pp. 220–225). Kwastek, K. (2008). Interactivity–A Word in Process. In: The Art and Science of Interface and Interaction Design (pp. 15–26). Springer. Kyratzis, A. & Ervin-Tripp, S. (1999). The development of discourse markers in peer interaction. Journal of pragmatics, 31(10), 1321–1338. Kyriakakis, C. (1998). Fundamental and technological limitations of immersive audio systems. Proceedings of the IEEE, 86(5), 941–951. Lackner, J. R. & Di Zio, P. A. (2000). Aspects of body self-calibration. Trends in cognitive sciences, 4(7), 279–288. Laibowitz, M., Gips, J., Aylward, R., Pentland, A. & Paradiso, J. A. (2006). A sensor network for social dynamics. In: Proceedings of the 5th international conference on Information processing in sensor networks (pp. 483–491). Langford, J. & McDonagh, D. (2004). Focus groups: supporting effective product development. CRC press. Larsson, P., Västfjäll, D. & Kleiner, M. (2001). Ecological acoustics and the multi-modal perception of rooms: real and unreal experiences of auditory-visual virtual environments. In Proceedings of the Conference on Auditory Display (pp. 245–249). Larsson, P., Västfjäll, D. & Kleiner, M. (2004). Perception of self-motion and presence in auditory virtual environments. In: Proceedings of 7th annual workshop of presence (pp. 252–258). Larsson, P., Västfjäll, D., Olsson, P. & Kleiner, M. (2007). When what you hear is what you see: Presence and auditory-visual integration in virtual environments. In: Proceedings of the 10th Annual International Workshop on Presence (pp. 11–18). Law, C. M., Jaeger, P. T. & McKay, E. (2010). User-centered design in universal design resources? Universal Access in the Information Society, 9(4), 327–335. Law, C. M., Yi, J. S., Choi, Y. S. & Jacko, J. A. (2008). A systematic examination of universal design resources: part 1, heuristic evaluation. Universal Access in the Information Society, 7(1-2), 31–54. Lee, E., Karrer, T. & Borchers, J. (2006). Toward a framework for interactive systems to conduct digital audio and video streams. Computer Music Journal, 30(1), 21–36. Lee, E., Kiel, H., Dedenbach, S., Grüll, I., Karrer, T., Wolf, M. & Borchers, J. (2006). iSymphony: an adaptive interactive orchestral conducting system for digital audio and video streams. In CHI’06 extended abstracts on Human factors in computing systems (pp. 259–262). Legrand, D., Brozzoli, C., Rossetti, Y. & Farne, A. (2007). Close to me: Multisensory space representations for action and pre-reflexive consciousness of oneself-in-the-world. Consciousness and cognition, 16(3), 687–699. Legrand, D. & Ravn, S. (2009). Perceiving subjectivity in bodily movement: The case of dancers. Phenomenology and the Cognitive Sciences, 8(3), 389–408. Lemaitre, G., Houix, O., Visell, Y., Franinović, K., Misdariis, N. & Susini, P. (2009). Toward the design and evaluation of continuous sound in tangible interfaces: The spinotron. International Journal of Human-Computer Studies, 67(11), 976–993. Leman, M. (2008). Embodied Music: Cognition and Mediation Technology. The MIT Press. Leman, M. & Camurri, A. (2004). Musical content expressive gesture interactive multimedia. In: Proceedings of the Conference on Interdisciplinary Musicology (CIM04). Graz, Austria: R. Parncutt, A. Kessler & F. Zimmer (Eds.). Leman, M., Demey, M., Lesaffre, M., van Noorden, L. & Moelants, D. (2009). Concepts, Technology, and Assessment of the Social Music Game. In: Computational Science and Engineering, 2009. CSE’09. International Conference on (Vol. 4, pp. 837–842).
229
Leman, M. & Naveda, L. (2010). Basic gestures as spatiotemporal reference frames for repetitive dance/music patterns in Samba and Charleston. Music Perception: An Interdisciplinary Journal, 28(1), 71–91. Lenggenhager, B., Tadi, T., Metzinger, T. & Blanke, O. (2007). Video ergo sum: manipulating bodily self-consciousness. Science, 317(5841), 1096–1099. Leplâtre, G. & Brewster, S. A. (1998). Perspectives on the design of musical auditory interfaces. International Journal of Computing Anticipatory Systems, 4, 227–239. Levitin, D. J., McAdams, S. & Adams, R. L. (2002). Control parameters for musical instruments: a foundation for new mappings of gesture to sound. Organised Sound, 7(2), 171–189. Lewis, G. E. (1999). Interacting with latter-day musical automata. Contemporary Music Review, 18(3), 99–112. Lewis, G. E. (2000). Too many notes: Computers, complexity and culture in voyager. Leonardo Music Journal, 10, 33–39. a Lievens, J., Siongers, J., & Waege, H. (2015). Participatie in Vlaanderen 1: Basisgegevens van de Participatiesurvey 2014. Acco. b Lievens, J., Siongers, J., & Waege, H. (2015). Participatie in Vlaanderen 2: Eerste Analyses van de Participatiesurvey 2014. Acco. Lindgaard, G., Dillon, R., Trbovich, P., White, R., Fernandes, G., Lundahl, S. & Pinnamaneni, A. (2006). User Needs Analysis and requirements engineering: Theory and practice. Interacting with Computers, 18(1), 47–70. Lipscomb, S. D. & Zehnder, S. M. (2004). Immersion in the virtual environment: The effect of a musical score on the video gaming experience. Journal of physiological anthropology and applied human science, 23(6), 337–343. Litle, P. & Zuckerman, M. (1986). Sensation seeking and music preferences. Personality and individual differences, 7(4), 575–578. Livingston, M. A. (2005). Evaluating human factors in augmented reality systems. IEEE Computer Graphics and Applications, 25(6), 6–9. Livingstone, D. & Miranda, E. (2005). ORB3–musical robots within an adaptive social composition system. In: Proceedings of the International Computer Music Conference. Livingstone, S. (2004). Media literacy and the challenge of new information and communication technologies. The Communication Review, 7 (1), 3-14. Longo, M. R., Schüür, F., Kammers, M. P., Tsakiris, M. & Haggard, P. (2008). What is embodiment? A psychometric approach. Cognition, 107(3), 978–998. López Cano, R. (2003). Setting de body in music. Gesture, schemata and stylistic-cognitive types. In: International Conference on Music and gesture. Lotze, M., Scheler, G., Tan, H.-R. ., Braun, C. & Birbaumer, N. (2003). The musician’s brain: functional imaging of amateurs and professionals during performance and imagery. NeuroImage, 20(3), 1817–1829. Loughlin, D. (2009). Interfaces for musical expression: the importance of musical context in parameter mapping research. (thesis) University College London. Lund, H. H. & Ottesen, M. (2008). RoboMusic: a behavior-based approach. Artificial Life and Robotics, 12(1-2), 18–23. Lundberg, J. (2010). Guidelines for developing an interactive multimedia prototype: based on comparison of low-and-high-fidelity prototypes in usability testing. Royal Institute of Technology. Stockholm, Sweden. Mack, R. L. & Nielsen, J. (1994). Usability inspection methods. Wiley & Sons. Mackay, W. E. (1998). Augmented reality: linking real and virtual worlds: a new paradigm for interacting with computers. In: Proceedings of the working conference on Advanced visual interfaces (pp. 13–21). MacKenzie, I. S. (1992). Fitts’ law as a research and design tool in human-computer interaction. Human-computer interaction, 7(1), 91–139.
230
Magnusson, T. (2006). Affordances and constraints in screen-based musical instruments. In: Proceedings of the 4th Nordic conference on Human-computer interaction: changing roles (pp. 441–444). Magnusson, T. (2009). Of epistemic tools: Musical instruments as cognitive extensions. Organised Sound, 14(02), 168–176. Magnusson, T. (2010). Designing constraints: Composing and performing with digital musical systems. Computer Music Journal, 34(4), 62–73. Magnusson, T. & Mendieta, E. H. (2007). The acoustic, the digital and the body: A survey on musical instruments. In: Proceedings of the 7th international conference on New interfaces for musical expression (pp. 94–99). Malloch, J., Sinclair, S. & Wanderley, M. M. (2007). From controller to sound: Tools for collaborative development of digital musical instruments. In: Proceedings of the 2007 International Computer Music Conference, Copenhagen, Denmark. Malloch, J., Sinclair, S. & Wanderley, M. M. (2008). A network-based framework for collaborative development and performance of digital musical instruments. In: Computer Music Modeling and Retrieval. Sense of Sounds (pp. 401–425). Springer. Mao, J.-Y., Vredenburg, K., Smith, P. W. & Carey, T. (2005). The state of user-centered design practice. Communications of the ACM, 48(3), 105–109. Maravita, A. & Iriki, A. (2004). Tools for the body (schema). Trends in cognitive sciences, 8(2), 79–86. Maravita, A., Spence, C. & Driver, J. (2003). Multisensory integration and the body schema: close to hand and within reach. Current Biology, 13(13), 531–539. Maravita, A., Spence, C., Kennett, S. & Driver, J. (2002). Tool-use changes multimodal spatial interactions between vision and touch in normal humans. Cognition, 83(2), 25–34. Mariën, I., & Vleugels, C. (2011). Van digitale kloof naar digitale inclusie. Tijdschrift voor Communicatiewetenschap, 39(4), 104. Markopoulos, P. & Bekker, M. (2003). On the assessment of usability testing methods for children. Interacting with computers, 15(2), 227–243. Marrin, T. & Picard, R. (1998). The “Conductor’s Jacket”: A Device for Recording Expressive Musical Gestures. In: Proceedings of the International Computer Music Conference (pp. 215–219). Marshall, M. T., Malloch, J. & Wanderley, M. M. (2009). Gesture control of sound spatialization for live musical performance. In: Gesture-Based Human-Computer Interaction and Simulation (pp. 227–238). Springer. Marshall, M. T. & Wanderley, M. M. (2006). Evaluation of sensors as input devices for computer music interfaces. In Computer Music Modelling and Retrieval (pp. 130–139). Springer. Martin, P. J. (2006). Musicians’ Worlds: Music-Making as a Collaborative Activity. Symbolic Interaction, 29(1), 95–107. Maynes-Aminzade, D., Pausch, R. & Seitz, S. (2002). Techniques for interactive audience participation. In: Proceedings of the 4th IEEE International Conference on Multimodal Interfaces (p. 15). McAdams, D. P. (1992). The five-factor model in personality: A critical appraisal. Journal of personality, 60(2), 329–361. McAdams, D. P. & Pals, J. L. (2006). A new Big Five: fundamental principles for an integrative science of personality. American Psychologist, 61(3), 204. McCartney, J. (2002). Rethinking the computer music language: SuperCollider. Computer Music Journal, 26(4), 61–68. McPhail, C. (2006). The crowd and collective behavior: Bringing symbolic interaction back in. Symbolic interaction, 29(4), 433–463. Mennecke, B. E., Valacich, J. S. & Wheeler, B. C. (2000). The effects of media and task on user performance: A test of the task-media fit hypothesis. Group Decision and Negotiation, 9(6), 507–529.
231
Merkel, C. B., Xiao, L., Farooq, U., Ganoe, C. H., Lee, R., Carroll, J. M. & Rosson, M. B. (2004). Participatory design in community computing contexts: Tales from the field. In: Proceedings of the eighth conference on Participatory design: Artful integration: interweaving media, materials and practices-Volume 1 (pp. 1–10). Merrill, D. & Paradiso, J. A. (2005). Personalization, expressivity, and learnability of an implicit mapping strategy for physical interfaces. In: Proceedings of the CHI Conference on Human Factors in Computing Systems, Extended Abstracts (pp. 2152–2161). Miletto, E. M., Pimenta, M. S., Bouchet, F., Sansonnet, J.-P. & Keller, D. (2009). Music creation by novices should be both prototypical and cooperative-lessons learned from CODES. In: Proceedings of the XII Brazilian Symposium on Computer Music. Miletto, Evandro Manara, Flores, L. V., Pimenta, M. S., Rutily, J. & Santagada, L. (2007). Interfaces for musical activities and interfaces for musicians are not the same: the case for codes, a web-based environment for cooperative music prototyping. In: Proceedings of the 9th international conference on Multimodal interfaces (pp. 201–207). Mion, L. & D’Incà, G. (2006). Analysis of expression in simple musical gestures to enhance audio in interfaces. Virtual Reality, 10(1), 62–70. Mion, L., D’Incà, G., de Götzen, A. & Rapanà, E. (2010). Modeling expression with perceptual audio features to enhance user interaction. Computer Music Journal, 34(1), 65–79. Miranda, E. R. & Al Biles, J. (2007). Evolutionary computer music. Springer. Miranda, E. R. & Wanderley, M. M. (2006). New digital musical instruments: control and interaction beyond the keyboard. Middleton, Wisconsin: A-R Editions. Molina, A. (2003). The digital divide: The need for a global e-inclusion movement. Technology Analysis & Strategic Management, 15 (1), 137-152. Moore, A. F. (2001). Categorical conventions in music discourse: style and genre. Music & Letters, 82(3), 432–442. Moran, T. P. & Dourish, P. (2001). Introduction to this special issue on context-aware computing. Human–Computer Interaction, 16(2-4), 87–95. Moreas, M.-A., & Pickery, J. (2011). SVR-Studie: Mediageletterdheid in een digitale wereld. Claes Printing. Morris, M. R., Huang, A., Paepcke, A. & Winograd, T. (2006). Cooperative gestures: multi-user gestural interactions for co-located groupware. In: Proceedings of the SIGCHI conference on Human Factors in computing systems (pp. 1201–1210). Mose Biskjaer, M., & Halskov, K. (2014). Decisive constraints as a creative resource in interaction design. Digital Creativity, 25(1), 27–61. Mulder, A. (2000). Towards a choice of gestural constraints for instrumental performers. Trends in gestural control of music, 315–335. Mulder, A. & Fels, S. (1998). Sound Sculpting: Manipulating sound through virtual sculpting. In: Proceedings of the 1998 Western Computer Graphics Symposium (pp. 15–23). Mulder, A. G., Fels, S. S. & Mase, K. (1999). Design of virtual 3D instruments for musical interaction. In Graphics Interface (Vol. 99, pp. 76–83). Müller, P., Schubiger-Banz, S., Arisona, S. M. & Specht, M. (2006). Soundium’s Design Tree: Supporting Multimedia Composition and Performance. In: Proceedings of Digital Art Weeks, 6. Musacchia, G. & Schroeder, C. E. (2009). Neuronal mechanisms, response dynamics and perceptual functions of multisensory interactions in auditory cortex. Hearing research, 258(1), 72–79. Musek, J. (2007). A general factor of personality: Evidence for the Big One in the five-factor model. Journal of Research in Personality, 41(6), 1213–1233. Musser, D., Wedman, J. & Laffey, J. (2003). Social computing and collaborative learning environments. In: Advanced Learning Technologies, 2003. Proceedings. The 3rd IEEE International Conference on (pp. 520–521).
232
Nabeshima, C., Lungarella, M. & Kuniyoshi, Y. (2005). Timing-based model of body schema adaptation and its role in perception and tool use: A robot case study. In Development and Learning, 2005. In: Proceedings. The 4th International Conference on (pp. 7–12). Nacke, L., Drachen, A. & Göbel, S. (2010). Methods for evaluating gameplay experience in a serious gaming context. International Journal of Computer Science in Sport, 9(2), 1–12. Nacke, L. & Lindley, C. (2008). Boredom, Immersion, Flow-A pilot study investigating player experience. In: Proceedings of the 2008 IADIS Gaming conference: Design for engaging experience and social interaction, (25–27). Nardi, B. A. (1996). Activity theory and human-computer interaction. In: B. A. Nardi (Ed.), Context and consciousness: Activity theory and human-computer interaction (pp. 7–16). Nevile, B., Driessen, P. & Schloss, W. (2003). A new control paradigm: software-based gesture analysis for music. In: Communications, Computers and signal Processing, 2003. PACRIM. 2003 IEEE Pacific Rim Conference on (Vol. 1, pp. 360–363). Newton-Dunn, H., Nakano, H. & Gibson, J. (2003). Block jam: a tangible interface for interactive music. In: Proceedings of the 2003 conference on New interfaces for musical expression (pp. 170–177). Ng, K.C. (2004). Music via Motion: Transdomain Mapping of Motion and Sound for Interactive Performances. Proceedings of the IEEE, 92(4), 645–655. Nielsen, J, Brendsen, N. K. & Jessen, C. (2008). RoboMusicKids–music education with robotic building blocks. In: Proceedings of the 2008 Second IEEE International Conference on Digital Games and Intelligent Toys Based Education, (pp. 149–156). Nielsen, J. (1993). Noncommand user interfaces. Communications of the ACM, 36(4), 83–99. Nielsen, J. (1994a). As they may work. Interactions, 1(4), 19–24. Nielsen, J. (1994b). Usability engineering. Access Online via Elsevier. Nielsen, J. (1994c). Usability inspection methods. In: Conference companion on Human factors in computing systems (pp. 413–414). Nielsen, J. & Faber, J. M. (1996). Improving system usability through parallel design. IEEE Computer, 29(2), 29–35. Nielsen, J. & Molich, R. (1990). Heuristic evaluation of user interfaces. In: Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 249–256). North, A. C., Hargreaves, D. J. & O’Neill, S. A. (2000). The importance of music to adolescents. British Journal of Educational Psychology, 70(2), 255–272. Nunes, I. L. (2010). Handling Human-Centered Systems Uncertainty Using Fuzzy Logics. The Ergonomics Open Journal, 3, 38–48. O’Modhrain, S. (2011). A framework for the evaluation of digital musical instruments. Computer Music Journal, 35(1), 28–42. Oh, S. (2004). User interface for intuitive interaction with multimedia. Journal of Individual Differences, 31(10), 1356–1363. Orio, N., Schnell, N. & Wanderley, M. M. (2001). Input devices for musical expression: borrowing tools from HCI. In: Proceedings of the 2001 conference on New interfaces for musical expression (pp. 1–4). Pachet, F. & Cazaly, D. (2000). A taxonomy of musical genres. In: Proceedings of the conference on Content-Based Multimedia Information Access (RIAO) (pp. 1238–1245). Pachet, F. (2003). The continuator: Musical interaction with style. Journal of New Music Research, 32(3), 333–341. Paillard, J. (1994). The instrumented and the instrumentalized body: neurobiological foundations. In: Lewis, H. G. & Sigaut, F. (1994) Culture and the uses of the body. Fyssen, Paris. Paine, D. G. (2004). Gesture and musical interaction: interactive engagement through dynamic morphology. In: Proceedings of the 2004 conference on New interfaces for musical expression (pp. 80–86). Paradiso, J. A. (1999). The brain opera technology: New instruments and gestural sensors for musical interaction and performance. Journal of New Music Research, 28(2), 130–149.
233
Paradiso, J. A. (2003). Dual-use technologies for electronic music controllers: a personal perspective. In: Proceedings of the 2003 conference on New interfaces for musical expression (pp. 228–234). Paradiso, J. A., Gips, J., Laibowitz, M., Sadi, S., Merrill, D., Aylward, R. & Pentland, A. (2010). Identifying and facilitating social interaction with a wearable wireless sensor network. Personal and Ubiquitous Computing, 14(2), 137–152. Paradiso, J. A. & O’Modhrain, S. (2003). Current Trends in Electronic Music Interfaces. Guest Editors’ Introduction. Journal of New Music Research, 32(4), 345–349. Paradiso, J., Abler, C., Hsiao, K. & Reynolds, M. (1997). The magic carpet: physical sensing for immersive environments. In: CHI’97 extended abstracts on Human factors in computing systems: looking to the future (pp. 277–278). Paradiso, J., Hsiao, K.-Y., Strickon, J. & Rice, P. (2000). New sensor and music systems for large interactive surfaces. In: Proceedings of the International Computer Music Conference (ICMC) (pp. 277–80). Paradiso, J. & Sparacino, F. (1997). Optical tracking for music and dance performance. Optical 3D Measurement Techniques 4(1), 11–18. Parncutt, R. (2007). Systematic musicology and the history and future of western musical scholarship. Journal of interdisciplinary music studies, 1(1), 1–32. Paul, C. & Werner, C. (2003). Digital art. Thames & Hudson London. Pearson, J. L. & Dollinger, S. J. (2004). Music preference correlates of Jungian types. Personality and Individual Differences, 36(5), 1005–1008. Pedersen, M. & Alsop, R. (2012). An approach to feature extraction of human movement qualities and its application to sound synthesis. In: Proceedings of the 2012 ACMC conference (pp. 62–68). Pennycook, B. W. (1985). Computer-music interfaces: a survey. ACM Computing Surveys, 17(2), 267–289. Pestova, X., Donald, E., Hindman, H., Malloch, J., Marshall, M. T., Rocha, F., Ferguson, S. & Wanderley, M. M. (2009). The CIRMMT/McGill digital orchestra project. In: Proceedings of the 2009 International Computer Music Conference. International Computer Music Association, San Francisco (pp. 37–41). Peters, D. (2010). Enactment in Listening: Intermedial dance in EGM sonic scenarios and the listening body. Performance Research, 15(3), 81–87. Petrie, H. & Bevan, N. (2009). The evaluation of accessibility, usability and user experience. In: Stepanidis, C. (ed.). The Universal Access Handbook. CRC Press. Petrie, H., Hamilton, F., King, N. & Pavan, P. (2006). Remote usability evaluations with disabled people. In: Proceedings of the SIGCHI conference on Human Factors in computing systems (pp. 1133–1141). Phillips-Silver, J. & Trainor, L. J. (2007). Hearing what the body feels: Auditory encoding of rhythmic movement. Cognition, 105(3), 533–546. Picone, I., & Deweppe, A. (2015). Media- en nieuwsrepertoire in Vlaanderen. In: Lievens, J., Siongers, J. & Waege, H. (eds.). Participatie in Vlaanderen 2: Eerste analyses van de Participatiesurvey. Acco. 183-207. Poels, K., de Kort, Y. & Ijsselsteijn, W. (2007). “It is always a lot of fun!”: exploring dimensions of digital game experience using focus group methodology. In: Proceedings of the 2007 conference on Future Play (pp. 83–89). Poepel, C. (2005). On interface expressivity: a player-based study. In: Proceedings of the 2005 conference on New interfaces for musical expression (pp. 228–231). Poesio, G. (2002). The Gesture and the Dance. Nineteenth Century Theatre and Film, 29(2), 40– 50. Pope, S. T. (1996). A taxonomy of computer music. Contemporary music review, 13(2), 137–145. Poupyrev, I., Lyons, M. J., Fels, S. & XXX XXX. (2001). New interfaces for musical expression. In CHI’01 extended abstracts on Human factors in computing systems (pp. 491–492).
234
Pratt, A. C. (2005). Cultural industries and public policy: An oxymoron? International journal of cultural policy, 11(1), 31–44. Presser, S., Couper, M. P., Lessler, J. T., Martin, E., Martin, J., Rothgeb, J. M. & Singer, E. (2004). Methods for testing and evaluating survey questions. Public opinion quarterly, 68(1), 109–130. Prieto-Rodríguez, J. & Fernández-Blanco, V. (2000). Are popular and classical music listeners the same people? Journal of Cultural Economics, 24(2), 147–164. Pylvänäinen, P. (2003). Body image: A tripartite model for use in dance/movement therapy. American Journal of Dance Therapy, 25(1), 39–55. Quiring, O. (2009). What do users associate with “interactivity”? A qualitative study on user schemata. New Media & Society, 11(6), 899–920. Ramachandran, V. S. (1998). Consciousness and body image: lessons from phantom limbs, Capgras syndrome and pain asymbolia. Philosophical Transactions of the Royal Society of London: Biological Sciences, 353(1377), 1851–1859. Ramadoss, B. & Rajkumar, K. (2007). Semi-automated annotation and retrieval of dance media objects. Cybernetics and Systems, 38(4), 349–379. Ramakrishnan, C., Freeman, J. & Varnik, K. (2004). The architecture of auracle: a real-time, distributed, collaborative instrument. In: Proceedings of the 2004 conference on New interfaces for musical expression (pp. 100–103). Ramos, G., Hinckley, K., Wilson, A. & Sarin, R. (2009). Synchronous gestures in multi-display environments. Human–Computer Interaction, 24(1-2), 117–169. Rauterberg, M. (1996). How to measure cognitive complexity in human-computer interaction. Cybernetics and Systems Research, 815–820. Rebelo, P. (2003). Performing space. Organised Sound, 8(02), 181–186. Reddell, T. (2003). Laptopia: The spatial poetics of networked laptop performance. Contemporary Music Review, 22(4), 11–23. Rentfrow, P. J. & Gosling, S. D. (2003). The do re mi’s of everyday life: the structure and personality correlates of music preferences. Journal of personality and social psychology, 84(6), 1236. Rentfrow, P. J. & Gosling, S. D. (2006). Message in a ballad: the role of music preferences in interpersonal perception. Psychological science, 17(3), 236–242. Repp, B. H. & Knoblich, G. (2004). Perceiving Action Identity: How Pianists Recognize Their Own Performances. Psychological Science, 15(9), 604–609. Reybrouck, M. (2005). Body, mind and music: musical semantics between experiential cognition and cognitive economy. Transcultural Music Review, 9. Reybrouck, M. (2006). Music cognition and the bodily approach: Musical instruments as tools for musical semantics. Contemporary Music Review, 25(1-2), 59–68. Reybrouck, M. (2012). Musical Sense-Making and the Concept of Affordance: An Ecosemiotic and Experiential Approach. Biosemiotics, 5(3), 391–409. Reynolds, M., Schoner, B., Richards, J., Dobson, K. & Gershenfeld, N. (2001). An immersive, multi-user, musical stage environment. In: Proceedings of the 28th annual conference on Computer graphics and interactive techniques (pp. 553–560). Robson, D. (2002). Play!: Sound toys for non-musicians. Computer Music Journal, 26(3), 50–61. Rocchesso, D. & Fontana, F. (2003). The sounding object. Firenze: Mondo estremo. Roose, H. (2008). Many-Voiced or Unisono?: An Inquiry into Motives for Attendance and Aesthetic Dispositions of the Audience Attending Classical Concerts. Acta Sociologica, 51(3), 237–253. Roose, H., Lievens, J. & Waege, H. (2007). The Joint Effect of Topic Interest and Follow-Up Procedures on the Response in a Mail Questionnaire: An Empirical Test of the LeverageSaliency Theory in Audience Research. Sociological Methods & Research, 35(3), 410– 428.
235
Ross, M. & Shulman, R. F. (1973). Increasing the salience of initial attitudes: Dissonance versus self-perception theory. Journal of Personality and Social Psychology, 28(1), 138. Ross, P. & Keyson, D. V. (2005). The case of sculpting atmospheres: towards design principles for expressive tangible interaction in control of ambient systems. Personal and Ubiquitous Computing, 11(2), 69–79. Rozendaal, M. C. & Schifferstein, H. N. (2010). Pleasantness in bodily experience: A phenomenological inquiry. International journal of design, 4(2), 55–63. Ryan, B. & Haslegrave, C. M. (2007). Developing a verbal protocol method for collecting and analysing reports of workers’ thoughts during manual handling tasks. Applied Ergonomics, 38(6), 805–819. Ryan, J. (1991). Some remarks on musical instrument design at STEIM. Contemporary music review, 6(1), 3–17. Salminen, K. (2009). Observations on human–technology interaction aspects in remote handling for fusion. Fusion Engineering and Design, 84(7-11), 1697–1701. Satchell, C. & Dourish, P. (2009). Beyond the user: use and non-use in HCI. In: Proceedings of the 21st Annual Conference of the Australian Computer-Human Interaction Special Interest Group (pp. 9–16). Satzinger, J. W. & Olfman, L. (1998). User interface consistency across end-user applications: the effects on mental models. Journal of Management Information Systems, 14(4), 167–193. Sawchuk, A. A., Chew, E., Zimmermann, R., Papadopoulos, C. & Kyriakakis, C. (2003). From remote media immersion to distributed immersive performance. In: Proceedings of the 2003 ACM SIGMM workshop on Experiential telepresence (pp. 110–120). Sayer, T. (2003). Software intervention in the process of improvisation. Digital Creativity, 14(2), 95–104. Sayer, T. (2006). A conceptual tool for improvisation. Contemporary Music Review, 25(1-2), 163–172. Scaife, M., Rogers, Y., Aldrich, F. & Davies, M. (1997). Designing for or designing with? Informant design for interactive learning environments. In: Proceedings of the ACM SIGCHI Conference on Human factors in computing systems (pp. 343–350). Schiettecatte, B. & Vanderdonckt, J. (2008). AudioCubes: a distributed cube tangible interface based on interaction range for sound design. In: Proceedings of the 2nd international conference on Tangible and embedded interaction (pp. 3–10). Schroeder, F. & Rebelo, P. (2007). Wearable music in engaging technologies. Ai & Society, 22(1), 85–91. Schubiger-Banz, S., Müller, S. & others. (2003). Soundium2: An interactive multimedia playground. In: Proceedings of the International Computer Music Conference, San Francisco. Schuurman, D., Courtois, C. & De Marez, L. (2011). New media adoption and usage among Flemish youngsters. Telematics and Informatics, 28(2), 77–85. Schuurman, D., De Moor, K., De Marez, L. & Evens, T. (2010). Investigating user typologies and their relevance within a living lab-research approach for ICT-innovation. In: System Sciences (HICSS), 2010 43rd Hawaii International Conference on (pp. 1–10). Schuurman, D., De Moor, K., De Marez, L. & Van Looy, J. (2008). Fanboys, competers, escapists and time-killers: a typology based on gamers’ motivations for playing video games. In: Proceedings of the 3rd international conference on Digital Interactive Media in Entertainment and Arts (pp. 46–50). Schwoebel, J., Boronat, C. B. & Branch Coslett, H. (2002). The man who executed “imagined” movements: evidence for dissociable components of the body schema. Brain and Cognition, 50(1), 1–16. Schwoebel, J., Buxbaum, L. J. & Branch Coslett, H. (2004). Representations of the human body in the production and imitation of complex movements. Cognitive Neuropsychology, 21(2-4), 285–298.
236
Schwoebel, J. & Coslett, H. B. (2005). Evidence for multiple, distinct representations of the human body. Journal of cognitive neuroscience, 17(4), 543–553. Seago, A., Holland, S. & Mulholland, P. (2005). Towards a mapping of timbral space. In: Proceedings of the Conference on Interdisciplinary Musicology. Seffah, A., Djouab, R. & Antunes, H. (2001). Comparing and reconciling usability-centered and use case-driven requirements engineering processes. Australian Computer Science Communications 23 (1), 132–139. Seif El-Nasr, M., Aghabeigi, B., Milam, D., Erfani, M., Lameman, B., Maygoli, H. & Mah, S. (2010). Understanding and evaluating cooperative games. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 253–262). Seifert, U. & Kim, J. H. (2007). Entelechy and embodiment in (artistic) human-computer interaction. In: Human-Computer Interaction. Interaction Design and Usability (pp. 929– 938). Springer. Seitz, J. A. (2005). Dalcroze, the body, movement and musicality. Psychology of Music, 33(4), 419–435. Sekiyama, K. (2006). Dynamic spatial cognition: Components, functions, and modifiability of body schema. Japanese Psychological Research, 48(3), 141–157. Seung, Y., Kyong, J.-S., Woo, S.-H., Lee, B.-T. & Lee, K.-M. (2005). Brain activation during music listening in individuals with or without prior music training. Neuroscience Research, 52(4), 323–329. Shackel, B. (2000). People and computers—some recent highlights. Applied Ergonomics, 31(6), 595–608. Sharples, M. (1994). An introduction to human computer interaction. Citeseer. Sharples, M., Jeffery, N., Du Boulay, J., Teather, D., Teather, B. & Du Boulay, G. (2002). Sociocognitive engineering: a methodology for the design of human-centered technology. European Journal of Operational Research, 136(2), 310–323. Shenton, J. T., Schwoebel, J. & Coslett, H. B. (2004). Mental motor imagery and the body schema: evidence for proprioceptive dominance. Neuroscience Letters, 370(1), 19–24. Shoemaker, G., Tsukitani, T., Kitamura, Y. & Booth, K. S. (2010). Body-centric interaction techniques for very large wall displays. In: Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries (pp. 463–472). Sinclair, P., Martinez, K., Millard, D. E. & Weal, M. J. (2002). Links in the palm of your hand: tangible hypermedia using augmented reality. In: Proceedings of the thirteenth ACM conference on Hypertext and hypermedia (pp. 127–136). Sinclair, S. (2007). Force-feedback hand controllers for musical interaction. (thesis) McGill University. Slater, M. (1999). Measuring presence: A response to the Witmer and Singer presence questionnaire. Presence: Teleoperators and Virtual Environments, 8(5), 560–565. Sloboda, J. A. (2000). Individual differences in music performance. Trends in cognitive sciences, 4(10), 397–403. Smalley, D. (1997). Spectromorphology: explaining sound-shapes. Organised sound, 2(2), 107– 126. Smalley, D. (2007). Space-form and the acousmatic image. Organised Sound, 12(01), 35–58. Smallwood, S., Trueman, D., Cook, P. R. & Wang, G. (2008). Composing for laptop orchestra. Computer Music Journal, 32(1), 9–25. Sodnik, J., Tomazic, S., Grasset, R., Duenser, A. & Billinghurst, M. (2006). Spatial sound localization in an augmented reality environment. In: Proceedings of the 18th Australian conference on Computer-Human Interaction. (pp. 111–118). Sonck, N. & de Haan, J. (2012). De virtuele kunstkar: cultuurdeelname via oude en nieuwe media. Den Haag: Sociaal en Cultureel Planbureau. Spence, C. & Driver, J. (2004). Crossmodal Space and Crossmodal Attention. Oxford University Press, Oxford.
237
Spence, C., Pavani, F., Maravita, A. & Holmes, N. (2004). Multisensory contributions to the 3-D representation of visuotactile peripersonal space in humans: evidence from the crossmodal congruency task. Journal of Physiology, 98(1), 171–189. Spence, C. & Squire, S. (2003). Multisensory integration: maintaining the perception of synchrony. Current Biology, 13(13), 519–521. Spencer, R. (2000). The streamlined cognitive walkthrough method, working around social constraints encountered in a software development company. In: Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 353–359). Stampfl, P. (2003). Augmented reality disk jockey: AR/DJ. In: Proceedings of the ACM SIGGRAPH conference (pp. 1–1). Stanney, K. M. (2002). Handbook of virtual environments: Design, implementation, and applications. Lawrence Erlbaum Associates Publishers. Stanney, K. M., Mollaghasemi, M., Reeves, L., Breaux, R. & Graeber, D. A. (2003). Usability engineering of virtual environments (VEs): identifying multiple criteria that drive effective VE system design. International Journal of Human-Computer Studies, 58(4), 447–481. Stanney, K. M., Mourant, R. R. & Kennedy, R. S. (1998). Human factors issues in virtual environments: A review of the literature. Presence: Teleoperators and Virtual Environments, 7(4), 327–351. Stary, C. (2001). User diversity and design representation: towards increased effectiveness in design for all. Universal Access in the Information Society, 1(1), 16–30. Stevens, C. & McKechnie, S. (2005). Thinking in action: thought made visible in contemporary dance. Cognitive Processing, 6(4), 243–252. Stowell, D., Robertson, A., Bryan-Kinns, N. & Plumbley, M. D. (2009). Evaluation of live human–computer music-making: Quantitative and qualitative approaches. International Journal of Human-Computer Studies, 67(11), 960–975. Stowell, D., Plumbley, M. D. & Bryan-Kinns, N. (2008). Discourse analysis evaluation method for expressive musical interfaces. In: Proceedings of the International Conference on New Interfaces for Musical Expression (NIME) (pp. 81–86). Streitz, N. A., Tandler, P., Müller-Tomfelde, C. & Konomi, S. (2001). Roomware: Towards the Next Generation of Human-Computer: Interaction based on an Integrated Design of Real and Virtual Worlds. In: Human-Computer Interaction in the New Millenium (pp. 551– 576). Addison Wesley. Strickler, Z. (1999). Elicitation methods in experimental design research. Design issues, 15(2), 27–39. Sutcliffe, A. & Gault, B. (2004). Heuristic evaluation of virtual reality applications. Interacting with computers, 16(4), 831–849. Svanæs, D. & Gulliksen, J. (2008). Understanding the context of design: towards tactical user centered design. In: Proceedings of the 5th Nordic conference on Human-computer interaction (pp. 353–362). Sweetser, P. & Wyeth, P. (2005). GameFlow: a model for evaluating player enjoyment in games. Computers in Entertainment, 3(3), xx–xx. Tajadura-Jiménez, A., Larsson, P., Väljamäe, A., Västfjäll, D. & Kleiner, M. (2010). When room size matters: Acoustic influences on emotional responses to sounds. Emotion, 10(3), 416. Talja, S. (1999). Analyzing qualitative interview data: The discourse analytic method. Library & information science research, 21(4), 459–477. Tan, D., Poupyrev, I., Billinghurst, M., Kato, H., Regenbrecht, H. & Tetsuani, N. (2000). The best of two worlds: merging virtual and real for face-to-face collaboration. In: Workshop on Shared Environments to Support Face-to-Face Collaboration. Tanaka, A. (2000). Musical performance practice on sensor-based instruments. Trends in Gestural Control of Music, 13, 389–405.
238
Tanzi, D. (2005). Language, music and resonance in cyberspace. Contemporary Music Review, 24(6), 541–549. Thompson, J., Kuchera-Morin, J., Novak, M., Overholt, D., Putnam, L., Wakefield, G. & Smith, W. (2009). The Allobrain: An interactive, stereographic, 3D audio, immersive virtual world. International Journal of Human-Computer Studies, 67(11), 934–946. Throsby, D. (2008). The concentric circles model of the cultural industries. Cultural trends, 17(3), 147–164. Tobin, S. (2013). Portable Play in Everyday Life; the Nintendo DS. Palgrave Macmillan. Toenjes, J. (2007). Composing for Interactive Dance: Paradigms for Perception. Perspectives of New Music, 28–50. Tomasello, T. K., Lee, Y. & Baer, A. P. (2009). “New media” research publication trends and outlets in communication, 1990-2006. New Media & Society, 12(4), 531–548. Torre, G., Fernström, M., O’Flynn, B. & Angove, P. (2007). Celeritas: wearable wireless system. In: Proceedings of the 7th international conference on New interfaces for musical expression (pp. 205–208). Traub, P. (2005). Sounding the net: Recent sonic works for the internet and computer networks. Contemporary Music Review, 24(6), 459–481. Trehub, S. E. (2003). The developmental origins of musicality. Nature neuroscience, 6(7), 669– 673. Trueman, D. (2007). Why a laptop orchestra? Organised Sound, 12(02), 171–179. Tsakiris, M. & Fotopoulou, A. (2008). Is my body the sum of online and offline bodyrepresentations? Consciousness and cognition, 17(4), 1317–1320. Tselios, N., Avouris, N. & Komis, V. (2008). The effective combination of hybrid usability methods in evaluating educational applications of ICT: Issues and challenges. Education and Information Technologies, 13(1), 55–76. Turk, M., Höllerer, T., Arisona, S. M., Kuchera-Morin, J., Coffin, C., Hoetzlein, R. & Overholt, D. (2009). Creative collaborative exploration in multiple environments. Aalborg University. Tzevelekos, P., Perperis, T., Kyritsi, V. & Kouroupetroglou, G. (2007). A component-based framework for the development of virtual musical instruments based on physical modeling. In Proceedings of the 4th Sound and Music Computing Conference, SMC’07 (pp. 30–36). Van den Hoogen, W., IJsselsteijn, W. & de Kort, Y. (2008). Exploring behavioral expressions of player experience in digital games. In: Proceedings of the workshop on Facial and Bodily Expression for Control and Adaptation of Games, ECAG 2008 (pp. 11–19). Van der Schyff, D. (2013). Emotion, Embodied Mind and the Therapeutic Aspects of Musical Experience in Everyday Life. Approaches: Music Therapy & Special Music Education, 5(1), xx–xx. a Van Deursen, A. J., & Van Dijk, J. A. (2009). Improving digital skills for the use of online public information and services. Government Information Quarterly, 26(2), 333-340. b Van Deursen, A. J., & Van Dijk, J. A. (2009). Using the Internet: Skill related problems in users’ online behavior. Interacting with computers, 21(5), 393-402. Van Deursen, A. J., van Dijk, J. A., & Peters, O. (2011). Rethinking Internet skills: The contribution of gender, age, education, internet experience, and hours online to mediumand content-related Internet skills. Poetics, 39 (2), 125-144. Van Deursen, A. J., & Van Dijk, J. A. (2014). The digital divide shifts to differences in usage. New media & society, 16 (3), 507-526. Van Eijck, K. & Lievens, J. (2008). Cultural omnivorousness as a combination of highbrow, pop, and folk elements: The relation between taste patterns and attitudes concerning social integration. Poetics, 36(2), 217–242. Vermeeren, A. P., Law, E. L.-C., Roto, V., Obrist, M., Hoonhout, J. & Väänänen-Vainio-Mattila, K. (2010). User experience evaluation methods: current state and development needs. In:
239
Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries (pp. 521–530). Vertegaal, R. (1994). An evaluation of input devices for timbre space navigation. (Ma Thesis) Deptartment of Computing, University of Bradford, United Kingdom. Veryzer, R. W. & Borja de Mozota, B. (2005). The Impact of User-Oriented Design on New Product Development: An Examination of Fundamental Relationships. Journal of Product innovation management, 22(2), 128–143. Vickers, P. (2004). External auditory representations of programs: Past, present, and future–an aesthetic perspective. In: Proceedings of the International Conference on Auditory Display (ICAD 2004), S. Barrass and P. Vickers, Eds. ICAD, Sydney. Vickers, P. & Alty, J. L. (2002a). Musical program auralisation: a structured approach to motif design. Interacting with Computers, 14(5), 457–485. Vickers, P. & Alty, J. L. (2002b). Using music to communicate computing information. Interacting with computers, 14(5), 435–456. Vignoli, F. (2004). Digital Music Interaction Concepts: A User Study. In ISMIR. Vines, B. W., Krumhansl, C. L., Wanderley, M. M., Dalca, I. M. & Levitin, D. J. (2005). Dimensions of emotion in expressive musical performance. Annals of the New York Academy of Sciences, 1060(1), 462–466. Vines, B. W., Krumhansl, C. L., Wanderley, M. M., Dalca, I. M. & Levitin, D. J. (2011). Music to my eyes: Cross-modal interactions in the perception of emotions in musical performance. Cognition, 118(2), 157–170. Vines, B. W., Krumhansl, C. L., Wanderley, M. M. & Levitin, D. J. (2006). Cross-modal interactions in the perception of musical performance. Cognition, 101(1), 80–113. Vines, B. W., Wanderley, M. M., Krumhansl, C. L., Nuzzo, R. L. & Levitin, D. J. (2004). Performance gestures of musicians: What structural and emotional information do they convey? In: Gesture-based communication in human-computer interaction (pp. 468– 478). Springer. Vom Lehn, D., Heath, C. & Hindmarsh, J. (2001). Exhibiting interaction: Conduct and collaboration in museums and galleries. Symbolic interaction, 24(2), 189–216. Vom Lehn, D., Heath, C. & Hindmarsh, J. (2005). Rethinking interactivity: design for participation in museums and galleries. Work, Interaction & Technology Research Group, King’s College London. Vredenburg, K. (1999). Increasing ease of use. Communications of the ACM, 42(5), 67–71. Vredenburg, K., Mao, J.-Y., Smith, P. W. & Carey, T. (2002). A survey of user-centered design practice. In: Proceedings of the SIGCHI conference on Human factors in computing systems: Changing our world, changing ourselves (pp. 471–478). Wainwright, S. P. & Turner, B. S. (2004). Epiphanies of embodiment: injury, identity and the balletic body. Qualitative Research, 4(3), 311–337. Wanderley, M. M. (2001). Gestural control of music. In: Proceedings of the International Workshop on Human Supervision and Control in Engineering and Music (pp. 632–644). Waters, S. (2007). Performance Ecosystems: Ecological approaches to musical interaction. EMS: Electroacoustic Music Studies Network. Weinberg, G. (2002). Playpens, Fireflies and Squeezables: New Musical Instruments for Bridging the Thoughtful and the Joyful. Leonardo Music Journal, 12, 43–51. Weinberg, G. (2003). Interconnected Musical Networks-Bringing Expression and Thoughtfulness to Collaborative Group Playing. Citeseer. Weinberg, G. & Driscoll, S. (2006). Toward robotic musicianship. Computer Music Journal, 30(4), 28–45. Weinberg, G. & Thatcher, T. (2006). Interactive sonification: Aesthetics, functionality and performance. Leonardo Music Journal, 16, 9–12. Welch, G. F. (2007). Addressing the multifaceted nature of music education: An activity theory research perspective. Research Studies in Music Education, 28(1), 23–37.
240
Wharton, C., Rieman, J., Lewis, C. & Polson, P. (1994). The cognitive walkthrough method: A practitioner’s guide. In: Usability inspection methods (pp. 105–140). Wichansky, A. M. (2000). Usability testing in 2000 and beyond. Ergonomics, 43(7), 998–1006. Wilkie, K., Holland, S. & Mulholland, P. (2010). What can the language of musicians tell us about music interaction design? Computer Music Journal, 34(4), 34–48. Willemsen, P., Gooch, A. A., Thompson, W. B. & Creem-Regehr, S. H. (2008). Effects of stereo viewing conditions on distance perception in virtual environments. Presence: Teleoperators and Virtual Environments, 17(1), 91–101. Wilson, J. R. (1997). Virtual environments and ergonomics: needs and opportunities. Ergonomics, 40(10), 1057–1077. Wilson, J. R. & D’Cruz, M. (2006). Virtual and interactive environments for work of the future. International Journal of Human-Computer Studies, 64(3), 158–169. Witmer, B. G. & Singer, M. J. (1998). Measuring presence in virtual environments: A presence questionnaire. Presence: Teleoperators and virtual environments, 7(3), 225–240. Wong, E. L. (2009). Augmenting Media Performance with Interactive Technology. In: Proceedings of the Generative Art Conference (pp. 283–293). Yagi, S. M. (1999). Computational discourse analysis for interpretation. Meta: Journal des traducteursMeta:/Translators’ Journal, 44(2), 268–279. Young, D. (2002). The hyperbow controller: Real-time dynamics measurement of violin performance. In: Proceedings of the 2002 conference on New interfaces for musical expression (pp. 1–6). Zatorre, R. J., Chen, J. L. & Penhune, V. B. (2007). When the brain plays music: auditory–motor interactions in music perception and production. Nature Reviews Neuroscience, 8(7), 547–558. Zhao, H., Plaisant, C. & Shneiderman, B. (2005). I hear the pattern: interactive sonification of geographical data patterns. In CHI’05 Extended Abstracts on Human Factors in Computing Systems (pp. 1905–1908). Zhou, Z., Cheok, A. D., Liu, W., Chen, X., Farbiz, F., Yang, X. & Haller, M. (2004). Multisensory musical entertainment systems. Multimedia, IEEE, 11(3), 88–101. Zweigenhaft, R. L. (2008). A Do Re Mi Encore. Journal of Individual Differences, 29(1), 45–55.
241
Appendices
The appendices will be available: digitally only https://www.dropbox.com/sh/orkb72c17fd9iti/AAB5jXMdmZj7zu1asgpnbJRMa ?dl=0 available: From March 6, 2017 – June 30, 2017 Appendices: Appendix 1: WMY questionnaire Appendix 2: WMY descriptive stats and results Appendix 3: Evaluation form of “SoundField” Appendix 4: Graphs of the “SoundField”-analyses Appendix 5: Lament question form Appendix 6: Lament results Appendix 7: Heart as an Ocean questions
243
