Reading Neural Net Artworks

Reading Neural Net Artworks Tim Barrass University of Melbourne [email protected]

Abstract This paper introduces a semiotic approach for analysis of neural network based generative art using concepts of agency, learning and emergence. These concepts are explored through self organising map ants (SOMA), which are neural net based agents that adapt as they interact to generate digital “drawings” and animations. Artefacts produced by variations of the SOMA algorithm are presented, and an analysis is made using the semiotic model. The approach presented here provides a comprehensive and systematic framework to analyse, describe and critically discuss neural net artworks.

keywords: semiotics, neural networks, agency, learning, emergence, analysis

1.

Introduction

Since the late 1980's, a diverse field of artistic practices has developed in which neural nets have been coupled with a variety of other media to generate, modulate, classify, compose, animate, activate, extrapolate and interpolate aesthetic materials in static and changing environments, often including human interaction [1]. However, to date there has been little critical awareness of artistic work with neural nets, and no vocabulary of concepts has been assembled to address the specifics of work in this field. The approach I present here characterises neural nets as artistic media in terms of emergence, learning and agency, which in turn form the basis of a model for semiotic analysis of works in the medium.

2.

Background: Systemic functional semiotics

Systemic-functional semiotics is an established approach to analysing texts which offers a way to read potentially any media, by considering the social functions of signs in relation to the systems in which they are realised. The approach has been applied to verbal text [2,3,4], traditional artistic media [5], multimedia [6,7], graphics [8], funerary architecture [9] and generative art [10]. In systemic-functional semiotic analyses every utterance, including sculptural, mathematical or musical signs, is considered as part and product of a social context of situation [4, pp. 12-14]. The range of meanings typically found in a particular social context is called the register, and is characterised in terms of the field (what is happening?), tenor (who are taking part?) and mode (what part is the language playing?). Along with the context of situation, systemic-functional semiotic analyses consider three main functions of texts. The tri-functional rationale is given by Michael Halliday, to whom the approach is largely due: Whatever we are using language for, we need to make some reference to the categories of our experience; we need to take on some role in the interpersonal situation; and we need to embody these in the form of text. [2, p. 29] The same functions are performed by texts irrespective of the media in which they are realised [5]. While the functions are supported by different systems in the codes of each art form, the following questions can be asked across media: presentational: what does it convey, what is it an image of, or what is its use? orientational: how does it invite attention, what's the emotional or stylistic slant? compositional: how do the parts fit together to produce a coherent text? Systemic-functional semiotics analyses specific systems of relationships within an artefact and how they support each function at various ranks, or levels [5]: work: the whole thing episode: a segment with its own story figure: participates in an episode member: contributes to a figure, like a body part, a machine part, or a piece of clothing

Reading an artwork requires the consideration and selection of dominant features from the permutations of functions and ranks. In semiotics, the dominant aspects of a text are those which are foregrounded, in such a way as to focus and unite a work [8, p.14].

3.

A systemic-functional semiotic model applied to neural net art

I have applied the systemic-functional semiotic model for reading neural net art. I started by defining the ranks in terms of neural net art as: work: the work as a whole, including the environment, material elements and participants neural network: the overall neural net architecture and algorithm layer: a categorisation of input, output and internal neurons unit: a neuron I then surveyed 19 existing neural net artworks and identified emergence, learning and agency as key characteristics [1]. Many of the artists working with neural nets emphasise a degree of unpredictability in the behaviour of their works [1, pp. 25-27], which the broader field of artificial life art is identified with emergence, a term used when local interactions between many elements give rise to what are observed as surprising global outcomes, which then impose order on subsequent local events in a self-modifying cycle [11,12,13,14,15]. Most, but not all, neural net artworks to date involve nets that learn, so that the interaction of a network with its environment affects not only the moment-to-moment behaviour but also the connective structure and hence the resulting “habits” of the network, even though the underlying algorithm doesn't change. There are many different kinds of supervised and unsupervised learning, and some models such as reinforcement learning and genetic adaptation which can be either [16,17,18,19]. As a consequence of emergence and learning, a neural net may cross the line from mere reactive functioning to exhibiting agency, as if driven by an internal agenda which gives consistency to what the network does. I have found three main views of agency described in the literature. The first of these is the systems view that agency is not an absolute property, but increases by degrees as a system self-organises [14,20].

The second view is the common English usage meaning that an agent acts on your behalf

according to a set of instructions and therefore embodies an artist's own goals in their work [21]. The third view comes from the social sciences, and considers agency as the interplay between parts of a system that includes people and technology, without central control [22]. I mapped the characteristics of emergence, learning and agency to the presentational, orientational and compositional semiotic functions to produce a medium-specific framework for analysis, presented in a tabular format in my thesis [1, Table 5]. This table allows you to look up questions to analyse the function of a characteristic, at a particular rank. For example, the analysis of the orientational function of agency, at the rank of work, contains the following points to consider: agents/patients: who is acting and who is being acted upon? primary/secondary: who is acting on behalf of whom? delegation: how are tasks delegated by primary to secondary agents?

responsibility: how is responsibility for events distributed amongst agents? goals: how are goals set? translations of competence: how do competences shift between agents? This is just one of the thirty-six permutations of functions, characteristics, and ranks in the table. Another example would be presentational function of learning at the rank of layer. This table provides a comprehensive and systematic framework to analyse, describe and critically discuss neural net artworks, based on the selection of dominant features from the matrix of permutations.

4.

Case study: self organising map ants (SOMA)

SOMA is an algorithmic system I developed which generates artefacts as the outcome of interactions amongst a population of neural net based agents that adapt as they modify a shared environment. It is a further exploration of a model presented in Laying down a path in walking [23], in which a simulated colony of “ants” leave “pheromone” trails which self-organise in a process of reinforcement comparable to the reinforcement of paths in neural systems [24,25]. The current work investigates: - a “connectionist aesthetic”, developing a “feel” for neural network-related characteristics of emergence, learning and agency discussed in this paper. - a combination of connectionism and social simulation to visualise the development of subjective dispositions in social models. This involves reciprocal causation whereby the global organisation of the system shapes the internal dynamics of the agents that comprise it. This is in contrast to much of the research in social simulation, which involves agents with fixed internal rules [26,27]. - the production of images of complex, continuously changing structures that exhibit a balance between cohesion and dissipation.

4.1

Implementation

The SOMA algorithm involves a nest with hundreds or thousands of ants and a drawing surface to register marks left by the ants as they move. Each ant has a feeler to sense nearby marks and a Kohonen self-organising map (SOM) [28] “brain” that translates the feeler data into an angle for the ant to turn at each step as it moves. Each ant produces a trail of its own unique colour so the mixed colour patterns that build up show the combined effects of different ants. Illustration 1 shows a schematic representation of the model. Each move involves the following steps: 1. The feeler senses surrounding marks and provides the data to the ant's SOM. 2. The SOM categorises the feeler data into one of a pre-set number of changing patterns and “learns” the pattern by adjusting its internal connections in response to the new data. 3. The ant turns according to the SOM categorisation and takes a step, leaving a trail. 4. Old trails slowly fade.

Illustration 1. SOMA ant schematic.

4.2

Results

The images in Illustration 2 are generated by the SOMA algorithm. Animations produced by the system can be seen on the accompanying DVD.

Illustration 2. SOMA artefacts.

4.3

Analysis of SOMA

I applied the semiotic model to analyse the SOMA algorithm and artefacts, and summarised the results in a table in my thesis [1, Table 9]. Below is an example analysis of the orientational function of agency at the rank of work: agents/patients: Each ant acts upon the environment. The environment acts upon each ant. primary/secondary: All agents act on their own behalf. delegation: Agents orient others by the traces they leave, “enlisting” each other in shared activities. responsibility: All agents have equal responsibility for events in the system. goals: The inbuilt goal of each agent is to develop an internal structure that reflects its inputs over time. translations of competence: An ant's competence is coded by the strength of connections in its neural network. These are translated between ants via the traces they leave in the environment. The dominant features foregrounded in reading SOMA depend on the social context in which the work is presented. The context here is giving a research paper at a conference (the field) of generative art experts (the tenor), to illustrate a way of reading neural network art (the mode). In this context, dominant features of the analysis are: the representational function of emergence at the rank of work: The patterns that emerge in the environment represent the cumulative interactions of the entire population. the orientational function of agency at the rank of work: The agents enlist each other in common projects, which are recognisable as persistent, dynamic structures in the environment. the organisational function of learning at the rank of neural network: The changing connections in the agents' neural networks determine how they respond to and leave traces in the environment. It is worth noting that the features of a work can be processes which may not be easy to discern by observing outputs alone. Consider the context of a gallery exhibition of SOMA animations for a general audience. There, the organisational function of learning could be dominant at the rank of work, observed as continual system-wide adaptation, rather than at the rank of neural network which is presented as an internal process in this paper. Analyses of different kinds of neural net artworks can be found in my thesis [1].

5.

Summary

This paper has introduced a framework for analysing neural network based art, by combining a systemic-functional semiotic approach with an understanding of neural nets as artistic media in terms of agency, learning and emergence. As an example for analysis, the author's self organising map ant (SOMA) algorithm has been described, and artefacts produced by the algorithm have been presented. An analysis of the creative work has been made in terms of the semiotic model for neural net art. The approach presented here provides a comprehensive and systematic framework to analyse, describe and critically discuss neural net artworks.

Acknowledgements This work was carried out as part of a masters thesis titled Neural Networks in New Media Art submitted to the University of Melbourne, 2005. I would like to thank my supervisor Dr. Peter Morse for developing discussions on semiotics, and Dr. Stephen Barrass for valuable feedback during the preparation of this paper.

References [1] Barrass, T. (2005). Neural networks in new media art. Master's thesis. University of Melbourne. [2] Halliday, M.A.K. (1976). Functions and universals of language. In G. Kress (Ed.), Halliday: System and function in language (pp. 26-31). London: Oxford University Press. (Original work published 1970). [3] Halliday, M.A.K. (1978). Language as social semiotic. London: Edward Arnold. [4] Halliday, M.A.K., & Hasan, R. (1985). Language, context and text: Aspects of language in a socialsemiotic perspective. Geelong: Deakin University. [5] O'Toole, M. (1994). The language of displayed art. London: Leicester University Press. [6] Lemke, J.L. (n.d.a.). Mathematics in the middle: Measure, picture, gesture, sign, and word. Retrieved May 10, 2002, from http://academic.brooklyn.cuny.edu/education/jlemke/papers/myrdene.htm. [7] Lemke, J.L. (n.d.b.). Multimedia semiotic analysis – Focal questions. Retrieved May 15, 2002, from http://academic.brooklyn.cuny.edu/education/jlemke/guides/ multimedia_semiotic_analysis_questions.htm. [8] Kress, G., & Van Leeuwen, T. (1990). Reading images. Victoria: Deakin University Press. [9] Morse, P. (1994-2002). Bodies in time: a sociosemiotics of funerary monuments and architecture. PhD Thesis, Murdoch University (Electronic version). Retrieved May 27, 2003, from http://www.petermorse.com.au/PMcomhtml/petermorse.html. [10] Riley, H. (2002). Generative art: Multi-modal meanings. Presented at Generative Art 2002, Milan. Retrieved May 14, 2003, from http://www.generativeart.com/papersGA2002/4.pdf [11] Cariani, P. (1991). Emergence and artificial life. In C.G. Langton, C. Taylor, J.D. Farmer & S. Rasmussen (Eds.), Artificial life II (pp. 775-797). Reading, Mass.:Addison-Wesley. [12] McCormack, J., & Dorin, A. (2001). Art, emergence and the computational sublime. In A. Dorin & J. McCormack (Eds.), Proceedings of Second Iteration, second international conference on generative systems in the electronic arts (pp. 67-81). Melbourne: CEMA. [13] Ronald, E., Sipper, M., & Capcarrere, M. (1999). Testing for emergence in artificial life. In D. Floriano, J.-D. Nicoud & F. Mondada (Eds.), Advances in artificial life: 5th European conference, ECAL'99 proceedings (pp. 13-20). Berlin: Springer. [14] Salthe, S.N. (1993). Development and evolution: Complexity and change in biology. Cambridge, Mass.: MIT Press. [15] Whitelaw, M. (2004). Metacreation: Art and artificial life. Cambridge, Mass.: MIT Press. [16] Wasserman, P.D. (1989). Neural computing. New York: Van Nostrand Reinhold. [17] Ritter, H., Martinez, T., & Schulten, K. (1992). Neural computation and self-organizing maps: An introduction (D. Barsky, M. Tesch & R. Kates, Trans.). Reading, Mass.: Addison-Wesley. [18] Rumelhart, D.E., Hinton, G.E., & McClelland, J.L. (Eds.) (1986a). Parallel distributed processing: Explorations in the microstructure of cognition. Vol.1: Foundations. Cambridge, Mass.: MIT Press. [19] Rumelhart, D.E., Hinton, G.E., & McClelland, J.L. (Eds.) (1986b). Parallel distributed processing: Explorations in the microstructure of cognition. Vol.2: Psychological and biological models.

Cambridge, Mass.: MIT Press. [20] Juarrero, A. (1999). Dynamics in action: Intentional behaviour as a complex system. Cambridge, Mass.: MIT Press. [21] Gell, A. (1998). Art and agency: An anthropological theory. Oxford: Clarendon Press. [22] Latour, B. (1999). Pandora's hope: Essays on the reality of science studies. London: Harvard University Press. [23] Barrass, T. (2001). Visualising the emergence of shared behavioural pathways in a crowd. In A. Dorin & J. McCormack (Eds.), Proceedings of Second Iteration, second international conference on generative systems in the electronic arts (pp. 67-81). Melbourne: CEMA. [24] Gordon, D.M. (2000). Close encounters. In J. Gleick (Ed.), The best American science writing (pp. 5765). NY: Harper Collins. [25] Chialvo, D.R., & Millonas, M.M. (c1995). How swarms build cognitive maps. In Steels, L. (Ed.), The biology and technology of intelligent autonomous agents (pp. 439-450). Berlin: Springer. [26] Gilbert, N., & Troitzsch, K.G. (1999). Simulation for the social scientist. Buckingham: Open University Press. [27] Liebrand, B.G., Nowak, A., & Hegselmann, R. (Eds.) (1998). Computer modelling of social processes. London: Sage. [28] Kohonen, T. (1995). Self-organizing maps. New York: Springer.