Audio Visual Patch in MaxMSP

Audio Visual Patch in MaxMSP Module

Interactive Music Performance & Composition

Lecturers

Dave Payling & Si Waite

Student

Josef Tot (15025232)

Date

15th August, 2016



Table of Contents Introduction .......................................................................................................................................... 3 Visuals ............................................................................................................................................... 3-5 Music ................................................................................................................................................. 5-6 Patching ............................................................................................................................................. 7-8 Outcomes & Future Improvements ........................................................................................................ 8 Conclusion ............................................................................................................................................ 8 References ..................................................................................................................................... 9 - 10



2

Introduction The aim of this audiovisual piece was to enhance a listener's experience by enhancing the minimalistic musical piece with visual means through synchronicity, symmetry and minimalism. Further it was goal to create a tool that enables the user to manipulate visuals in an easy manner so that they fit with the musical piece and visual movement or representation of the audio. The focus lies on translating mainly rhythmical and harmonic components of the musical piece in a tightly synchronized manner and aiding with symmetry to emphasize symmetrical fundamentals, which can also be found in music. This report will examine the pieces concepts and goals by examining the main idea of having a synchronised audiovisual piece first, then further explaining the realisation of the musical as well as the visual element via Ableton and MAXMsp objects. Lastly it will present future proposals and development suggestions.

Visuals In the search of having appealing visuals and complementing the music's syncopation and minimalistic characteristics, symmetry lies at the fundamental principle of many things in nature. The theory of why it is aesthetically pleasing lies outside of the scope of this assignment, however it can be found in a variety of forms in nature and one of them is music as later on described by Dr. Crist's explanation in the music section. Symmetry itself comes in a variety of forms such as reflective, rotational and translational symmetry. For this audio visual piece a reflective mirroring effect was chosen as it allows to create abstract forms of any kind of video material upon importing into the patch in a simple manner. Robert Coolman explains reflective symmetry as: “In general usage, symmetry most often refers to mirror or reflective symmetry; that is, a line (in 2-D) or plane (in 3-D) can be drawn through an object such that the two halves are mirror images of each other.“ (Coolman, 2015) Further to having a more abstract and appealing form of mirroring similar to a kaleidoscope the project focused on a double reflection with a fold over as illustrated in fig. 1 below:



Fig. 1

3

Further Illustration in Fig. 2 below shows the abstraction of any type of video material:



Fig. 2

The original demonstration is a video of ink dropping into water without any reflections and then applying reflective transformation to create an abstract symmetry. A side effect of this pixel processing is distortion as seen in the color, which again can be reverted by changing color settings within Max MSP.



Fig. 3

The underlying concept of “motional” symmetry can best be described using Fig. 3 above. In order for the video to have a seamless motion across time it must revert to its first state at the end of its timeline, so that it loops, similar to a bar loop in the musical piece. This allowed for a fluid transformation of motion so that not just the still frame was symmetrical but also its motion was symmetrical with the synchronised musical piece. Furthermore to create a seamless repetition it was important to sync the motion of an object that was in a video with either the beat or bar of a track. There are two main problems when it comes to syncing motion with beats or bars, one of them being that frames of a video are not always syncable with the duration of beat or bar (frames per beat). Meaning that if a frame overlaps the boundary of a beats ending it would run out of sync over a certain period of time. Therefore prior to syncing it is important to retrieve information about videos frames per second and calculate the correlation between

4

bpm and frames per second so that the frames would finish as a whole number within the beat or bar. There is an online tool available which made calculating this approach much easier, available at vjamm.com. Artists ideas of synchronicity between audio and visuals date back almost a century, with pieces like Walter Ruthmans “Lichtspiel Opus I” screened in 1921 as well as pieces like “An Optical Poem” by Oskar Fischinger. Up to date synchronicity is found in many audiovisual pieces most influential for this piece was the highly synchronised laser show by Robert Henke named “Lumière II”.

Music To create an aesthetic correlation between the two mediums, it is important to segment the musical piece in terms of its structure and then to assign segments and individual rhythms to parameters in such a way that the visuals represent the music in an appealing way. The structure of the musical piece can be determined by analysing its individual instruments, rhythms and arrangement of the track. As Wallace Berry author of the book Structural functions in Music suggests that: “The beat or underlying “metric level” is at the heart of most common musical pieces” (Berry, 1976). The same applies to the musical piece created in Ableton Live. Time signatures and tempo serve as “metric level”, therefore creating a repetitive pulse in time which entrains a listener, thus in a way stimulates the listener's biological movement (e.g. moving the head, or dancing to the fundamental beat). The goal in this project was to further aid in entraining the listener by adding the visual element, which is synchronised using MIDI triggers. Inspiration for the minimalistic music was drawn from artists such as Steve Reich and his “Six Pianos” piece as well as artists of newer generations such as Gaiser with his piece “Some Slip”, using sampled sounds within a digital audio workstation to create minimalistic dance music coined “minimal techno”. The minimalistic music piece created for this project shares several characteristics with the minimalist movement, such as frequent repetition, static rhythms and minimalist motifs and the use of simplistic instruments and samples. Timothy Johnson author of the book “Minimalism: Aesthetic, Style or Technique?” refers to minimalism in brief as: “In various ways these pieces seem to suspend time, gradually revealing a slowly unfolding process or focusing upon a minute musical detail. They move sharply away from the idea of requiring the listener to recognize goals and goal-directed motion and toward the notion.” (Timothy, 1994) Therefore focus on details evolving typically within one or two bars and their repetitions create an entraining experience in the listener. Additionally the “suspension of time” aids in that the listener does not necessarily recognize where the music might head or where it would end. Further aim was to enhance this minimalist characteristic, with symmetrical visuals.

5

Symmetry can be found in many forms and music can inhibit many aspects of it. The Symmetry in this project’s music can be seen as repetition of rhythm on a linear time scale. Using modern 4/4 based music it consists of beats of the same size, therefore beats in 4/4 have equal sizes resulting in symmetrical meter across linear time of a bar. A good explanation for this can be delivered quoting Dr. Crist: “A symmetrical meter consists of beats of the same size. For instance, all the beats in 4/4 are of equal size (1 quarter note). In 6/8 meter, the beat sizes are also equal (the dotted quarter note). However, 5/8 has two beat sizes. The first a quarter note, the second a dotted quarter note. 5/8 is therefore said to be asymmetrical because one of its beats is larger than the other.” (Crist, 2005) Further the musical piece features harmonic symmetry first appearing at 1:12 as visually illustrated in screenshot 1 below. The melodic percussion starting E4 D#4 D4 C#4 in bar 1 and then inverted and transposed 2 semitones to G4 G#4 A4 A#4 in bar 2.



Screenshot 1

The musical piece was made in Ableton using sample shots, triggered by MIDI. MIDI was sequenced using Abletons drum rack. Additional synthesizers were used to create soundscapes. The MIDI triggers for the video file switching, stopping, playing and jumping to frame 1 were created within Ableton, via corresponding tracks containing MIDI messages. 100 BPM were set as it created a perfect balance between the videos frame loop and time it took to complete 1 bar, subsequently not having to alter the original frames per second of the video.

6

Patching



Fig. 4

Further on MAXMsp was used to establish MIDI, mirroring effect and color controls to manipulate previously synced videos. Fig. 4 shows the basic concept and its two core elements synchronicity controlled via video playback parameters and symmetry controlled through video effects. The mirroring effect was implemented following a tutorial on Cycling74. The original recipe takes 4 different camera views of a 3D camera input and maps it onto 4 videoplanes. However instead of using camera inputs, this patch used the synced videos. This effect results in complex visual patterns, similar to a kaleidoscope. A quick overview of the main patching elements shows the concept of [jit.gl.pix], which lies at the core of the mirroring or symmetry effect: As [jit.gl.pix] is opened, further objects are needed to create color controls and the mirroring effect. The color controls require a [param] object to be set, in order to be manipulatable via a [pak] object outside of the subpatch. Therefore in order to create a RGB color controller the [param] object is assigned three values [param colours 0 0 0]. Where “colours” corresponds to what to control in the main patch, therefore [pak colours 0 0 0]. Further three [flonum] objects allow to manipulate the RGB range of given objects. In order to create the mirroring effect X, XY, Y reflections were patched using [sample @boundmode] objects, which sample matrices and allow the video inputs to fold over itself, once manipulating the XY axis through [param offset 0. 0.]. Further [* -1 -1], [* -1 1] and [* 1 -1] objects were created in order to foldover X, XY and Y, creating the mirror reflections. As to the MIDI controls, a [notein] object allowed to receive notes specified by further channel number and a name [notein “channelname” “channel number”]. Then specific MIDI values assigned in Ableton allowed to route note values to desired messages via the [route] object. In this case each of the videos needed 3 parameter messages to be sent to [jit.movie], in order to control start, frame1 and stop of a video file that has been loaded using the read message beforehand. Additionally in order to switch between videos, it was necessary to create a [switch] object, so that a user could switch through given videos, in this case 5 different clips. This was done by simply assigning 1 through 5 messages and patching them into the corresponding [jit.movie] objects. In essence this component of the MIDI controls, allowed to switch clips by sending MIDI

7

messages out of Ableton. Furthermore, to save processing power, the stop message enabled to pause the video, whenever the video has been switched, because the video would still run in the background, therefore straining the graphics card.

Outcomes & Future Improvements The combination of synchronicity, symmetry and minimalistic modern dance music allows for a greater entrainment of a listener, thus ultimately delivering a greater audiovisual experience where rhythmic metrics are complemented and amplified by the visuals. Besides the controls over the mirroring, switching, pausing, start objects and messages, there are plenty of other manipulatable parameters that are going to be considered in future development, such as color controls and further mirroring controls. A [pattrstorage] object for example allows a user to store different parameters. This control could be used in further development to experiment between the aesthetics of the music and visual piece, and once appealing parameters would be defined, the would be easy to load via MIDI triggers defined in Ableton. Additionally the foldover parameters could be manipulated via MIDI triggers in a sense that instead of having 4 reflections constantly a user could switch between various reflections of the XY axis. Further development of this patch and an integration into Max4live would allow to manipulate videos without having to constantly jump between Ableton and MAXMsp.

Conclusion The goal of having synchronicity and symmetry was fulfilled in a basic iteration and the patch allows to manipulate a various range of parameters within MAXMsp itself, using Ableton as control interface, with the aid of MIDI information. MIDI information was used to manipulate given parameters. The concept of synchronicity, symmetry and minimalism lie at the core of the audio visuals. However further research into the field of audio visual aesthetics would allow to adjust a broader range of parameters in order to achieve additionally more appealing influence to the listeners/viewers. Also further development in terms of user friendliness and advanced tools would enable more users to experiment with this tool.

8

References Anon (n.d.). VJamm - Support. [Online]. VJamm - Support. Available from: http://www.vjamm.com/support_av_bpm.php?lang=en.html. [Accessed: 15 August 2016]. Coolman, R. (2015). What Is Symmetry? [Online]. 4 June 2015. LiveScience. Available from: http://www.livescience.com/51100-what-is-symmetry.html. [Accessed: 13 August 2016]. Crist, T. (2005). Rhythm and Meter. [Online]. 29 April 2005. Available from: http://clt.astate.edu/tcrist/theory4/rhythmandmeter.pdf. [Accessed: 13 August 2016]. Johnson, T.A. (1994). Minimalism: Aesthetic, Style, or Technique? The Musical Quarterly. 4th Ed. Oxford University Press. Available from: [Accessed: 14 August 2016]. A.C.V. (2013). An Optical Poem (1938) - Classic Short Film. [Online]. 2013. YouTube. Available from: https://www.youtube.com/watch?v=they7m6yepo. [Accessed: 15 August 2016]. minusofficial (2011). [MINUS112] Gaiser - Some Slip. [Online]. 8 August 2011. YouTube. Available from: https://www.youtube.com/watch?v=iaf2snnmive. [Accessed: 15 August 2016]. pianomania88 (2011). Six Pianos, Steve Reich. [Online]. 18 January 2011. YouTube. Available from: https://www.youtube.com/watch?v=edke10yz_zs. [Accessed: 15 August 2016]. kenef3 (2011). Lichtspiel Opus I (1921) - Walther Ruttmann. [Online]. December 2011. YouTube. Available from: https://www.youtube.com/watch?v=ahzddmyfzn0. [Accessed: 15 August 2016]. brtvofficial (2016). Robert Henke: Lumière II Boiler Room Mexico x MUTEK MX Live Set. [Online]. 5 January 2016. YouTube. Available from: https://www.youtube.com/watch?v=7cmkzfkumr8. [Accessed: 15 August 2016]. Mollaghan, A. (2015). The visual music film. Springer. Benson, A. (2012). C74: Recipe 55: MirrorHouse. [Online]. 7 June 2012. Cycling 74 Recipe 55 MirrorHouse Comments. Available from: 9

https://cycling74.com/2012/06/07/recipe-55-mirrorhouse/#.vzdr6hurjty. [Accessed: 14 August 2016]. Anon (2013). Synchronicity = Simplicity? [Online]. September 2013. Third Eye. Available from: https://mythirdeyeworld.wordpress.com/ramblings/synchronicity-simplicity/. [Accessed: 14 August 2016]. Clayton, M., Sager, R. & Will, U. (2004). In time with the music: The concept of entrainment and its significance for ethnomusicology. [Online]. Available from: http://www.stanford.edu/group/brainwaves/2006/willintimewiththemusic.pdf. [Accessed: 14 August 2016]. Grierson, M. (2005). Audiovisual Composition. [Online]. Available from: http://www.strangeloop.co.uk/dr. m.grierson - audiovisual composition thesis.pdf. [Accessed: 14 August 2016]. Ribas, L. (2014). Sound and image relations: a history of convergence and divergence. [Online]. 2 June 2014. Sound and image relations: a history of convergence and divergence. Available from: http://divergencepress.com/journal/journalissue/tabid/85/id/13/sound-andimage-relations-a-history-of-convergence-and-divergence.aspx. [Accessed: 14 August 2016]. Berry, W. (1987). Structural functions in music. New York: Dover.

10